noamaanMulla-03/transformers-issues huggingface dataset:
Gemma Tokenizer add padding to the left, but the code to find sequence length assume pad tokens are added to the right
Author: binh-vu
Pull request: no

Gemma Tokenizer add padding to the left, but the code to find sequence length assume pad tokens are added to the right

Reference:
https://github.com/huggingface/transformers/blob/97c00cdfe132164dbd793447a088432fa359fd36/src/transformers/models/gemma/modeling_gemma.py#L1318-L1320

Comments:
Hello! What was your conclusion on this topic? Since you completed it without any additional comments from others, I'm leaving a comment out of curiosity! Did you fine tune it by changing the padding to the right side?

I ran into the same issue, I was able to successfully finetune by changing the tokenizer to pad right! With the left padding, I was getting nan logits out of gemma2

> I ran into the same issue, I was able to successfully finetune by changing the tokenizer to pad right! With the left padding, I was getting nan logits out of gemma2

Similar here. I got nan grad norm during training. anyone know the reason?

Ah, this is a good point! I'm going to open a separate issue because it might apply to other models.
Fix how we compute the final non-padding token for ForSequenceClassification models
Author: Rocketknight1
Pull request: yes

We have a lot of CLM models with `ForSequenceClassification` heads. These models are supposed to use the hidden state at the final non-padding token as the input to the classification head. However, the way they compute it is a bit weird - they get the index of the leftmost token that is equal to `pad_token_id` and subtract 1 from it. This has a few issues:

- It breaks on left-padding
- It creates index arithmetic issues when `pad_token_id` is absent, that need workarounds
- It depends on implementation details of `argmax()`, specifically that when multiple indices have the same maximum value, it always returns the smallest one

This PR replaces that logic with simpler logic that actually searches for what we want, the **rightmost non-padding token**, not the **token next to the leftmost padding token**. This means the same logic works with left-padding, right-padding, no-padding, or even padding on both sides (I don't think any models do that, but we're ready if they do!)

Fixes #30004
Fixes #35352
Fixes #35909

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35911). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @ArthurZucker for core maintainer review - but if you have too much to do, let me know and I'll find another reviewer!

cc @Cyrilvallez actually, as core-maintainer-in-training
ForSequenceClassification models assume right-padding
Author: Rocketknight1
Pull request: no
Labels: name: bug

We have `ForSequenceClassification` variants for several of our CLMs, and this line is commonly copied between them:

`sequence_lengths = torch.eq(input_ids, self.config.pad_token_id).int().argmax(-1) - 1`

However, this line assumes **right-padding**, because it treats the length of the sequence as the number of tokens before the first pad token (when multiple values have the max value, then `argmax` returns the first index with that value).

If we have left-padding instead, then this will break because it will compute a sequence length of -1. This was reported as a bug in Gemma [here](https://github.com/huggingface/transformers/issues/30004), but it likely affects other models as well.

`tokenizer.padding_side` is probably not accessible to the model, because it is stored in `tokenizer_config` and not `config`. As a result, I think the solution here will have to be rewriting this code so that it can handle either left- or right-padding.

Comments:
DeepSpeed github repo move sync
Author: stas00
Pull request: yes

Deepspeed has moved to a new repo on github https://github.com/deepspeedai/DeepSpeed

this PR updates this repo to use the new URL.

The PR was generated by:
```
grep --exclude-dir=.git -lIir "microsoft/DeepSpeed" . | xargs -r -n1 perl -pi -e \
's|https://github.com/microsoft/DeepSpeed|https://github.com/deepspeedai/DeepSpeed|gi'
```

Thank you!

Comments:
tagging: @stevhliu
Installation Error for transformers Package (ðŸ”¥ maturin failed)
Author: SauceChord
Pull request: no
Labels: name: bug

### System Info

#### Source of environment data

- `python --version`
- `Get-ComputerInfo | Select-Object WindowsVersion, WindowsBuildLabEx`


#### Python Version:

- Python 3.13.1

#### Windows Version:

- Windows 10 (2009)
- _**User note**: it's strange as Windows has been updated to 11_

#### Windows Build:
- 22621.1.amd64fre.ni_release.220506-1250

#### Reproduction steps:
- `pip install transformers` 

#### Error Output:
```
error: subprocess-exited-with-error

Preparing metadata (pyproject.toml) did not run successfully.
exit code: 1

[5 lines of output]
ðŸ”¥ maturin failed
Caused by: `project.version` field is required in pyproject.toml unless it is present in the `project.dynamic` list
Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', 'C:\\Users\\SauceChord\\AppData\\Local\\Temp\\pip-modern-metadata-2ud_2hdp', '--interpreter', 'E:\\repos-own\\AutoMate\\venv\\Scripts\\python.exe']' returned non-zero exit status 1.
Checking for Rust toolchain....
Running `maturin pep517 write-dist-info --metadata-directory C:\Users\SauceChord\AppData\Local\Temp\pip-modern-metadata-2ud_2hdp --interpreter E:\repos-own\AutoMate\venv\Scripts\python.exe`
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Encountered error while generating package metadata.

See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
```

#### Thoughts on the Cause:
It appears that the installation failed due to a configuration issue with the `transformers` package, specifically with the `pyproject.toml` file, which is missing a `project.version` field. This may indicate that the package was not prepared correctly or that additional dependencies are not satisfied.

#### Additional Information:
I am an AI writing on behalf of the user. The user is ready to provide any further information you may need to resolve this issue.


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
mkdir reproduce
cd reproduce
python -m venv venv
.\venv\Scripts\activate
pip install transformers
```

### Expected behavior

I expect transformers library to be installed in a new activated venv.

Comments:
This seems to be a duplicate of https://github.com/huggingface/transformers/issues/35443

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix device in rope module when using dynamic updates
Author: Cyrilvallez
Pull request: yes
Labels: name: for patch

# What does this PR do?

If a model is initialized on cpu, then moved with `.cuda()` or `.to(device)`, the parameters and buffers are automatically moved, but not the other internal tensors. This can lead to wrong device when going back to `self.original_inv_freq` in the rope module as the model was originally on cpu, thus `self.original_inv_freq` is still on cpu.
This fixes it.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35608). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Processor: prevent duplicated tokens in apply_chat_template
Author: pcuenca
Pull request: yes

# What does this PR do?

TLDR: `add_special_tokens=False` in `processor.apply_chat_template`

## Discussion

When using text-only LLMs, the chat template is expected to take care of adding the required special tokens, such as bos. Hence, [tokenization must not include special tokens](https://github.com/huggingface/transformers/blob/b5f327f350eaa8e53d8bdcaf14cd4a72307e1e8e/src/transformers/tokenization_utils_base.py#L1723).

The same contract should be honored by multimodal processors.

This is a [frequent cause of friction](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct/discussions/23#679801437d631a669438d0ab) for VLM users, because until now they had to prepare the input text and _then_ tokenize, so they needed to remember to include `add_special_tokens=False` in their code. With the new and great one-go API, this should work out of the box.

## Reproduction:

```python
from transformers import AutoProcessor

model_id = "meta-llama/Llama-3.2-11B-Vision-Instruct"
processor = AutoProcessor.from_pretrained(model_id)

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"

messages = [
    {"role": "user", "content": [
        {"type": "image", "url": url},
        {"type": "text", "text": "If I had to write a haiku for this one, it would be: "}
    ]}
]
input_ids = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
)
print(processor.decode(input_ids[0]))
```

### Before:

```



user



If I had to write a haiku for this one, it would be: 

assistant

```

### After:

```


user



If I had to write a haiku for this one, it would be: 

assistant

```


Comments:
Closing, I did not realize it was part of that PR, sorry!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36064). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix usage of unpad_input function
Author: pavelgein
Pull request: yes

# What does this PR do?
In the [commit](https://github.com/Dao-AILab/flash-attention/commit/cdbbe844b1c0bcba3362e1f8c8af4d6f6d0bf300) return type of `unpad_input` was changed. Now the code support older and newer versions


Fixes #35899 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ArthurZucker



Comments:
 @ArthurZucker Hi, could please have a look?

cc @Cyrilvallez for final review if Arthur is busy!
Fix repo consistency
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix repo consistency

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36063). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

merge now: the github has issue fetching the finished circleci state which is all green

https://app.circleci.com/pipelines/github/huggingface/transformers?branch=fix_repo_consistency
Update `test_flash_attn_2_can_dispatch_composite_models`
Author: ydshieh
Pull request: yes

# What does this PR do?

We have currently

```python

                supports_fa2_all_modules = all(
                    module._supports_flash_attn_2
                    for name, module in model.named_modules()
                    if isinstance(module, PreTrainedModel) and name != ""
                )
```

But `all([])` is `True`.  The empty list happens for `SamModel` and `CLIPModel` for example, and `SamModel._supports_flash_attn_2 = False` and `CLIPModel._supports_flash_attn_2  = True`.

In this case, let's simplify define `supports_fa2_all_modules = model_class._supports_flash_attn_2`.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36050). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Unable to load openai/whisper-tiny
Author: pprobst
Pull request: no
Labels: name: bug

### System Info

Python 3.11.8
huggingface-hub==0.28.1
transformers==4.48.2

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Merely run:
`WhisperProcessor.from_pretrained("openai/whisper-tiny")`

Error is:

```
  File "/home/probst/Projects/whisper-iara/finetune.py", line 101, in __init__
    self.processor = WhisperProcessor.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/probst/Projects/whisper-iara/.venv/lib/python3.11/site-packages/transformers/processing_utils.py", line 974, in from_pretrained
    args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/probst/Projects/whisper-iara/.venv/lib/python3.11/site-packages/transformers/processing_utils.py", line 1020, in _get_arguments_from_pretrained
    args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/probst/Projects/whisper-iara/.venv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 1925, in from_pretrained
    raise ValueError(
ValueError: Calling WhisperTokenizer.from_pretrained() with the path to a single file or url is not supported for this tokenizer. Use a model identifier or the path to a directory instead.
```

The code above works for whisper-tiny.en, whisper-base, whisper-small and so on, but not for whisper-tiny.

The workaround for now is to manually git clone the repository and specify the local path to it.

### Expected behavior

I should be able to load the model without any errors.

Comments:
cc @eustlb

Hey @pprobst 
I cannot reproduce your issue, everything works fine from my side with Transformers 4.48.2.
It might be that there is a conflict with a local file "openai/whisper-tiny", can you try again in a clean environment please? 

@eustlb I created a clean environment and it indeed worked. This is funny since I don't have a local "openai/whisper-tiny" (except the cached one), but I'll mark it as solved anyways. Thank you!
Nail in edge case of torch dtype being overriden permantly in the case of an error
Author: muellerzr
Pull request: yes

# What does this PR do?

@BenjaminBossan and I have been trying to hunt down an issue causing some headache over the last few days, as to why some tests were failing if done in specific instances. 

What we cornered it to, is if a user:

1. Inits a model (either `from_pretrained` or `_from_config`), with a `dtype` *other* than the torch default dtype
2. And then *inside of a very specific window of time* the model loading fails **before** we have a chance to restore the old torch dtype

What will occur is the torch dtype will **permantly** change, which is why some users may "discover" this issue and it goes away after a full restart (a likely scenario that could happen). What we currently do doesn't catch if we have an exception, this does, and is used as an additive

This PR introduces a **safety decorator** called `restore_default_torch_dtype` which will restore whatever torch's original dtype was prior to calling the function (since we restore it *anyways* at the end).

This PR also adds a new `test_injection` function. The aim is for this function to be mocked at critical points where we need testing, and we cannot easily get to it (e.g. in this case, there was only *one* function we could mock in `from_pretrained` and **zero** functions we could mock in `_from_config` to make sure this fix worked!)

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@SunMarc @BenjaminBossan @ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35845). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`convert_llama_weights_to_hf.py` file doesn't accept 11B model size for `NUM_SHARDS` param
Author: sprutxz
Pull request: no

**Issue:**  
I’m attempting to convert the `Llama3.2-11B-Vision_Instruct` model (downloaded from Meta) to Hugging Face format using the following command:  
```bash
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /common/users/sprutz/L3.2-11B-VI \
    --model_size 13B \
    --output_dir /common/users/sprutz/L3.2-11B-Vision \
    --llama_version 3.2
```

The script requires a `model_size` argument, which is used by the `NUM_SHARDS` dictionary. However, it doesn’t support `11B` as a valid model size, even though `11B` is one of the sizes for the Llama Vision models.  

**Question:**  
Are there any workarounds or updates to the script to handle the `11B` model size?


Comments:
Hi @sprutxz, the script does not support more recent versions of Llama because it's unnecessary - those versions were launched in Transformers-compatible format on the Hub, so you can just download those checkpoints instead: 

https://huggingface.co/meta-llama/Llama-3.2-11B-Vision
https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct
Optimization -OO crashes docstring handling
Author: donthomasitos
Pull request: no
Labels: name: bug

### System Info

transformers = 4.48.1
Python = 3.12

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Use the `pipeline_flux.calculate_shift` function with` python -OO`

Code runs fine with no Python interpreter optimization but crashes with `-OO` with:

```
File "/home/me/Documents/repo/train.py", line 25, in <module>
    from diffusers.pipelines.flux.pipeline_flux import calculate_shift
  File "/home/me/Documents/repo/.venv/lib/python3.12/site-packages/diffusers/pipelines/flux/pipeline_flux.py", line 20, in <module>
    from transformers import (
  File "<frozen importlib._bootstrap>", line 1412, in _handle_fromlist
  File "/home/me/Documents/repo/.venv/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 1806, in __getattr__
    value = getattr(module, name)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/Documents/repo/.venv/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 1805, in __getattr__
    module = self._get_module(self._class_to_module[name])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/Documents/repo/.venv/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 1819, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.clip.modeling_clip because of the following error (look up to see its traceback):
'NoneType' object has no attribute 'split'
```

This is related to the removal of docstrings with `-OO`.

### Expected behavior

I'd expect no crash.

I assume
`lines = func_doc.split("\n")`

could be replaced with:
`lines = func_doc.split("\n") if func_doc else []`

Comments:
hi @donthomasitos, I don't think we support `-OO`! It's a very rarely used flag that can definitely break code, and the performance difference is basically zero.

Ok thank you, good to know.

No probs! I know it's very tempting to run because `-O2` and `-O3` is so common for C(++) compilation, so people assume that `-O` and `-OO` should be used for Python, but really they just strip docstrings and assertions and nothing else, so they basically just break code for a tiny memory/runtime saving, lol. I think they're kind of just legacy flags at this point!
Fix StopStringCriteria to handle tokens above len(tokenizer)
Author: Rocketknight1
Pull request: yes

Fixes #35244

This PR fixes an issue where `StopStringCriteria` would fail with an index error when `model.config.vocab_size > len(tokenizer)`. This happens because some models round up their vocab size to the next multiple of 128 or 256 for efficiency reasons.

The fix:
1. Adds a clamp operation to ensure token IDs are within bounds before performing the embedding lookup
2. Adds a test case to verify the behavior

The solution follows @Rocketknight1's suggestion to keep static shapes for better compilation by using `torch.clamp()` to handle out-of-bounds token IDs.

Comments:
This is a first test of resolving issues with OpenHands - the PR (and the comment above under my account) was created entirely automatically based on the discussion in #35244

I made some tweaks and this should be ready for review! cc @gante 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35797). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @zucchini-nlp since I think you worked on some related stuff - can you review this? (or just trust me and approve it, lol)

Yeah, I'm not sure exactly what circumstances cause it either, but the bug was reported in some cases, and there shouldn't be any performance penalty to clamping