noamaanMulla-03/transformers-issues huggingface dataset:
llama tokenizer encode -> decode is not same
Author: pocca2048
Pull request: no
Labels: name: bug

### System Info

transformers==4.49.0

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer

to = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
txt = " !" # " .", " ,", " !"
encoded = to.encode(txt, add_special_tokens=False)
decoded = to.decode(encoded)

txt == decoded
```

### Expected behavior

txt should be same as decoded but does not work for " !", " .", " ,"

Comments:
Hi there 👋 You can ensure that encoded and decoded values are the same by setting `clean_up_tokenization_spaces=False`. For example:
```py
from transformers import AutoTokenizer

to = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
txt = " !" # " .", " ,", " !"
encoded = to.encode(txt, add_special_tokens=False)
decoded = to.decode(encoded, clean_up_tokenization_spaces=False)

txt == decoded
```

See https://github.com/huggingface/transformers/issues/35938#issuecomment-2654022379 for more information.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
<spam>
Author: suhaibo666
Pull request: no
Labels: name: bug

### System Info

test

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

xxx

### Expected behavior

xxxx

Comments:
Fix torch version guard at import
Author: zucchini-nlp
Pull request: yes
Labels: name: for patch

# What does this PR do?

Fixes #36888 and fixes #36906. As per title, the kwargs we are using (`flatten_with_keys_fn`) are available only from 2.3.0. We can also pin at 2.2.0 and pass it as positional argument, but then `flatten_with_keys_fn` will not be used in torch (from what I can see in torch repo)

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36907). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

hi thanks for fixing - is this a hotfix for 4.50.0?

@geraldstanje1 yes, we will do patch release later today :)
Fix typos
Author: omahs
Pull request: yes

Fix typos

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36910). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update `trainer_pt_utils.py` docstrings for consistency
Author: ethanknights
Pull request: yes

# What does this PR do?

Descriptions about `trainer_pt_utils.py` functions are updated for better consistency, with some typos addressed in their docstrings.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).




Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).
Fix pytorch defomr attn path
Author: qubvel
Pull request: yes
Labels: name: Vision

name: for patch

# What does this PR do?

Fix pytorch-base path DeformableAttention for RT-DETR, DeformatbleDetr, OmdetTurbo and GrounginDino. 

Broken with 
 - #36853
 
Export is **not** fixed in case `kernels` are installed, it will be fixed in a separated PR with additional functionality for `kernels` library.

cc @ArthurZucker for patch release

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36923). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
More precise comment
Author: ydshieh
Pull request: yes

# What does this PR do?

so we don't make contributors confused.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

Yes, LGTM!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36935). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add seed to data collator classes
Author: capemox
Pull request: no
Labels: name: Feature request

### Feature request

I opened an issue for setting a seed for `DataCollatorForLanguageModeling` which would help reproducibility in pretraining encoder-only transformers without setting a global seed at https://github.com/huggingface/transformers/issues/36357, with a PR that also got approved (https://github.com/huggingface/transformers/pull/36497). 

I see that there is scope to add this seed feature to other data collator classes, namely `DataCollatorForWholeWordMask` and `DataCollatorForPermutationLanguageModeling`, where there are pseudo-random functions involved in the collator.

### Motivation

This feature would help reproducibility in data preparation without setting a global seed as in `transformers.set_seed`. This has advantages as I have pointed out in https://github.com/huggingface/transformers/issues/36357.

### Your contribution

Since I was involved with the seed feature in `DataCollatorForLanguageModeling`, I'm very familiar with this idea and I'd love to make PRs adding this feature to the other two collators!

Comments:
Limit numpy version to <2.0.0
Author: Nicba1010
Pull request: yes

# What does this PR do?

Limits numpy version to <2.0.0 to prevent it being installed on numpy>=2.0.0 projects since it is clearly incompatible in some regards.

Fixes https://discuss.huggingface.co/t/unable-to-create-tensor-you-should-probably-activate-padding-with-padding-true-to-have-batched-tensors-with-the-same-length/98181

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

Hi @Nicba1010, I don't think we want to limit to Numpy < 2.0! Instead, I'll try to investigate the bug in that thread

Hi @Nicba1010, finally got around to this one, but I couldn't reproduce the bug because the snippet on the forums used a local model. Can you or anyone else give me some code to reproduce the error?
Mask2Former _init_weights
Author: tommiekerssies
Pull request: no
Labels: name: Core: Modeling

name: WIP

name: bug

name: Vision

### System Info

- `transformers` version: 4.45.2
- Platform: Linux-6.8.0-51-generic-x86_64-with-glibc2.39
- Python version: 3.11.9
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.4
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.4.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: yes
- GPU type: NVIDIA RTX A6000

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run _init_weights

### Expected behavior

The _init_weights method of Mask2Former has multiple problems. It initializes nn.Embeddings with an std of .02 (original Mask2Former code uses PyTorch's default init with std of 1.0). Similarly, the mask MLP is initialised wrongly with zero biases. Finally, another example of a problem is that the initialisation of the multi-scale deformable attention is overwritten by the branch for the Mask2FormerPixelDecoderEncoderOnly.

Comments:
cc @qubvel for Mask2Former, but I think we would accept a PR to fix this! If you make that PR, can you include links to the relevant lines in the original Mask2Former code so we can compare against the reference implementation?

Hi @tommiekerssies, thanks for opening the issue! `_init_weights` are not ideal for some vision models in transformers, we would appreciate PR to fix it!

hi @qubvel , i was having a look at this issue and wrote a quick fix . pushed in PR #35904 .feel free to give this a look when you find the time . i will address any comments you might have on the same . 

also thank you to @tommiekerssies for raising the issue .🤗

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

not stale , active PR in Progress .

saying this once more , can i am trying to work this through , dont mark it as stale .

Added WIP to avoid marking as stale

thank you for the tag .i'll build on this and get back to you on the same .
Disallow Offload to disk for gguf files
Author: MekkCyber
Pull request: yes

# What does this PR do?

Disallow offload to disk for gguf files. It was introduced in this pr https://github.com/huggingface/transformers/pull/35628. 
We can't have offload to disk in the case of gguf files because we need to modify the state_dict to be compatible with transformers.



Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36933). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
MacOs: register_pytree_node got an unexpected keyword argument 'flatten_with_keys_fn'
Author: CloseChoice
Pull request: no
Labels: name: bug

### System Info

Shap uses transformers in the test suite. With the release of 4.50.0 we saw failures in our pipelines on the MacOs runners (see [here](https://github.com/shap/shap/actions/runs/14009619930/job/39227771610#step:6:719)). Interestingly the error cannot be reproduced on any Windows or Ubuntu. Reverting to 4.49 solved the issue for us.

Since there was a recent change on related code in the transformers library [here](https://github.com/huggingface/transformers/pull/36850/files#diff-357aea2fcf1b51b2a8ad17c7d30044159b1fe1a3f6bd832cced70aca84c8be66R588), this code be the cause of the issue, though I couldn't figure out the exact reason.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run this on MacOs:


```python
import transformers

name = "hf-internal-testing/tiny-random-BartModel"

tokenizer = transformers.AutoTokenizer.from_pretrained(name)
model = transformers.AutoModelForSeq2SeqLM.from_pretrained(name)
```

or

```bash
transformers-cli
```

### Expected behavior

Should run properly, as in version 4.49.0

Comments:
Duplicate of https://github.com/huggingface/transformers/issues/36888. I opened a PR to fix it :)

@zucchini-nlp thanks for the quick fix.
Add Phi4 multimodal
Author: Cyrilvallez
Pull request: yes

# What does this PR do?



Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36939). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fixing _pre_quantization_dtype when torch_dtype is None
Author: MekkCyber
Pull request: yes

# What does this PR do?

Sets `_pre_quantization_dtype` to the default dtype (`float32`) when `torch_dtype` is set to None. It was causing the `casting_dtype` to be None for `bitnet` for example


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36930). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Export for Phi4-mini
Author: guangy10
Pull request: yes
Labels: name: Text

name: torch export

# What does this PR do?

As titled. Add support for Phi3-mini and Phi4-mini as they are sharing the same modeling code. It's requiring additional configure (rope_type = "default") to make it exportable, otherwise we will have to rewrite the modeling code, specifically, specialize or get rid of the control flow in `_longrope_frequency_update`.


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. https://github.com/huggingface/transformers/issues/33838
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @amyeroberts @qubvel


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

run-slow: phi3

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/phi3']
quantizations: [] ...

run-slow: phi3

cc: @tugsbayasgalan

> run-slow: phi3

FYI: this trigger by comment is only reserved for some `transformers` internal members.

run-slow: phi3

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/phi3']
quantizations: [] ...
Fix cuda index issue in cache allocator
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR fixes cache allocator function when querying the max-memory available on cuda devices. Currently, it doesn't work for torch <= 2.4.1 due to `torch.cuda.mem_get_info` raising an issue if we pass a device without an index. 


Traceback returned for a loading a model like gpt2 with device_map="cuda"

```python 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/marc/transformers/src/transformers/models/auto/auto_factory.py", line 573, in from_pretrained
    return model_class.from_pretrained(
  File "/home/marc/transformers/src/transformers/modeling_utils.py", line 272, in _wrapper
    return func(*args, **kwargs)
  File "/home/marc/transformers/src/transformers/modeling_utils.py", line 4474, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/home/marc/transformers/src/transformers/modeling_utils.py", line 4859, in _load_pretrained_model
    caching_allocator_warmup(model_to_load, expanded_device_map)
  File "/home/marc/transformers/src/transformers/modeling_utils.py", line 5870, in caching_allocator_warmup
    index = device.index if device.index is not None else 0
  File "/home/marc/anaconda3/envs/llama-export/lib/python3.10/site-packages/torch/cuda/memory.py", line 684, in mem_get_info
    device = _get_device_index(device)
  File "/home/marc/anaconda3/envs/llama-export/lib/python3.10/site-packages/torch/cuda/_utils.py", line 38, in _get_device_index
    return _torch_get_device_index(device, optional, allow_cpu)
  File "/home/marc/anaconda3/envs/llama-export/lib/python3.10/site-packages/torch/_utils.py", line 803, in _get_device_index
    raise ValueError(
ValueError: Expected a torch.device with a specified index or an integer, but got:cuda

```

cc @winglian 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36937). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`Helsinki-NLP/opus-mt-it-en` isn't on HuggingFace Hub
Author: KickItLikeShika
Pull request: no
Labels: name: New model

### Model description

I have found lots of Opus translation model on HuggingFace Hub but couldn't find Portuguese to English model, however the model already exists in Helsinki repo https://github.com/Helsinki-NLP/OPUS-MT-train/tree/master/models/pt-en#opus-2019-12-05zip

is that something can be added quickly? 

### Open source status

- [ ] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://github.com/Helsinki-NLP/OPUS-MT-train/tree/master/models/pt-en#opus-2019-12-05zip

Comments:
Hey! I guess you can convert the weights yourself using the script in `transformers`. The steps are

- Download the marian weights from Helsinki models repo for pt-en
- Install transformers + packages used to convert the weights (wget sentencepiece gitpython)
- Run the conversion with `python src/transformers/models/marian/convert_marian_to_pytorch.py --src pt-en-folder --dest converted`, which results in `converted/opus-mt-pt-en` folder with the weights converted to HF format

Can be used 
```python
from transformers import pipeline
p = pipeline("translation", model="/path/to/converted/opus-mt-pt-en", max_new_tokens=50)
p("Portuguese text to translate")
```

Hello @zucchini-nlp,

I was looking into this. However there is a problem with your answer. The EN to PT and PT to EN models are not in SentencePiece ".spm" format, but BPE. The files I have are "source.bpe" and "source.tcmodel", so your answer isn't working!

https://github.com/Helsinki-NLP/OPUS-MT-train/tree/master/models/en-pt
https://github.com/Helsinki-NLP/OPUS-MT-train/tree/master/models/pt-en

Do you know how to progress now?

Also, these files are kind of old. Is there a better option that Helsinki-NLP for PT to EN and EN to PT at the moment?

Thanks!
Ruff update
Author: d-kleine
Pull request: no

Currently, `transformers` uses the outdated `ruff-lsp`

https://github.com/huggingface/transformers/blob/09a309d27364204eb118d352f22483bdd9652a46/setup.py#L164

Please see [this discussion](https://github.com/astral-sh/ruff/discussions/15991). I think it would be good to update to native server as stable in [Ruff 0.5.3](https://github.com/astral-sh/ruff/releases/tag/0.5.3)

@Rocketknight1 

Comments:
cc @arthurzucker @cyrilvallez @ydshieh since `ruff-lsp` is being deprecated, we might have to update our dependencies and do a PR to QC the library with the new version of ruff!

I am open to a upgrade, but I have 0 knowledge what is "ruff-lsp" and I can't see the fact we are using "ruff-lsp" in the library.
@d-kleine Could you help me understand this better although it sounds like a basic thing 🙏 ?

There is a warning (not an error) when using the Ruff version that is specified in the `setup.py` file:

![Image](https://github.com/user-attachments/assets/a4d90200-a4e6-44fc-9c79-571294f24dde)

The current Ruff version (0.5.1) is falling back to using the deprecated `ruff-lsp` (**Python**-based) for now as it is too old to use the new native server (**Rust**-based). When updating to at least version 0.5.3, would make use of the new native server. As this new native server is Rust-based, it would benefit from improved performance and new features (which `ruff-lsp` doesn't have because it's Python-based).

This would mean updating `ruff` to at least version 0.5.3 or higher, and probably also updating the ruffs commands in the `pyproject.toml` file(s) matching with the new version.


Got it , thank you for sharing!

yeah sure, fine to update to latest rather! 

I will open a pr to update it
Update ruff to `0.11.2`
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix #36705

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers.

@ArthurZucker 

CI is ✅ with the new (dev) images and this PR, see [here](https://app.circleci.com/pipelines/github/huggingface/transformers/122912/workflows/1617e479-a6c1-44f1-9e33-53221f770271) and [here](https://app.circleci.com/pipelines/github/huggingface/transformers/122916/workflows/d4a3d950-372b-40ec-a3ab-cce79f006acf)

Once approved, I will build the usual images and push, then merge the PR

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36962). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update after #36962
Author: ydshieh
Pull request: yes

# What does this PR do?

Some files on main were not updated in my previous PR #36962 and cause failure after it was merge to main

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36965). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
update
Author: ydshieh
Pull request: yes

# What does this PR do?

The `check_code_quality` job had `ruff format tests src utils --check` but we need to add `examples` too!


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36972). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.