noamaanMulla-03/transformers-issues huggingface dataset:
Add more rigerous non-slow grad accum tests
Author: muellerzr
Pull request: yes

# What does this PR do?

Should be merged **after** https://github.com/huggingface/transformers/pull/35651 (tests will fail until I rebase).

This PR tweaks the Trainer grad_accum tests to:

* Use a much smaller CLM (`nickypro/tinyllama-15M`)
* Use much smaller data + chunk of the dataset (`max_length=16` + first 40 items)

This takes us on CPU from ~76s per test to **~6s per test** so we can run it as part of our PR CI and **squash grad accum issues before they get merged**


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@Rocketknight1 @ArthurZucker 

Comments:
To check the GA loss for the condition that loss_kwargs was not enabled, could we simply hack the trainer by setting `trainer.model_accepts_loss_kwargs=False` before `trainer.train()`?

@hiyouga if you notice, `diff_broken` does this already in both tests. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35668). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@muellerzr Yhh I get it. The existing test cases only ensure the "diff broken" is very off, but could we have a test case to ensure the models do not accept `loss_kwargs` (like Qwen2-VL) to be correctly trained when GA is enabled?
Fix test fetcher
Author: ydshieh
Pull request: yes

# What does this PR do?

After #35167, the `__init__` under each model directory don't import stuff from the python files inside it. The test fetcher has since trouble to get the right dependencies, and our CircleCI is sometimes not running any actual tests.

This PR uses `define_import_structure` to fix the issue

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36129). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

There is one failure 

> tests/repo_utils/test_tests_fetcher.py::TestFetcherTester::test_create_reverse_dependency_map - ValueError:KeyError:src/transformers/models/bert/tokenization_bert.py. From src/transformers/models/bert/__init__.py

I will adjust the test and merge.
skip `test_initialization` for `VitPoseBackboneModelTest` for now
Author: ydshieh
Pull request: yes

# What does this PR do?

Skip for now

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36154). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add git LFS to AMD docker image
Author: ivarflakstad
Pull request: yes

# What does this PR do?
Adds git LFS to the AMD docker image.
Should fix this failing test

> FAILED tests/models/auto/test_modeling_auto.py::AutoModelTest::test_dynamic_saving_from_local_repo - OSError: Looks like you do not have git-lfs installed, please install. You can install from [https://git-lfs.github.com/.](https://git-lfs.github.com/) Then run `git lfs install` (you only have to do this once).


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36016). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Warning 'The attention mask is not set'
Author: AlessandroSpallina
Pull request: no
Labels: name: bug

Having the same warning appearing in a closed pull request #33509 

### System Info

- `transformers` version: 4.47.1
- Platform: Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 0.27.0
- Safetensors version: 0.5.0
- Accelerate version: 1.2.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: yes
- GPU type: NVIDIA RTX 4000 Ada Generation Laptop GPU

### Who can help?

@ylacombe 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

code:
```python
        pipe = pipeline(
            "automatic-speech-recognition",
            model=self.model,
            torch_dtype=torch.float16,
            chunk_length_s=30,
            batch_size=24,
            return_timestamps=True,
            device=self.device,
            tokenizer=processor.tokenizer,
            feature_extractor=processor.feature_extractor,
            model_kwargs={"use_flash_attention_2": True},
            generate_kwargs={
                "max_new_tokens": 128,
            },
        )

```

warning:
```
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Whisper did not predict an ending timestamp, which can happen if audio is cut off in the middle of a word. Also make sure WhisperTimeStampLogitsProcessor was used during generation.
```

### Expected behavior

No warning

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Mllama fsdp
Author: blbadger
Pull request: yes

# What does this PR do?

Fixes # [35880](https://github.com/huggingface/transformers/issues/35880)

During the forward pass pixel values are sent do the device and datatype assigned to the patch embedding's weight rather than the device and datatype of the model. This prevents misassignment issues when training via FSDP, as the patch embedding's weight parameters are dispatched to the proper device and datatype before reference.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
@qubvel


Comments:
@qubvel my pleasure, thanks for the suggestion! I have committed it and updated the branch.
PaliGemma2 doesn't work with transformers v4.48.2
Author: koron
Pull request: no
Labels: name: Should Fix

name: bug

### System Info

- `transformers` version: 4.48.2
- Platform: Linux-6.12.8-200.fc41.x86_64-x86_64-with-glibc2.40
- Python version: 3.13.1
- Huggingface_hub version: 0.24.5
- Safetensors version: 0.5.2
- Accelerate version: 1.3.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.0a0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: yes
- GPU type: AMD Radeon RX 7600 XT


### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Run [example script in PaliGemma2 model card](https://huggingface.co/google/paligemma2-3b-pt-224#use-in-transformers) with transfomers v4.48.2
2. It failed with `TypeError: PaliGemmaForConditionalGeneration.forward() got an unexpected keyword argument 'last_cache_position'. (snipped)`

If I try the same thing with transformers v4.48.1 it works.

### Expected behavior

It should be successful.

---

This PR https://github.com/huggingface/transformers/pull/35681 which is released as v4.48.2, introduced `last_cache_position` kwarg to Gemma2 model.

As a side effect of this, the same  `last_cache_position` kwarg may be passed to PaliGemma2.
But PaliGemma2 doesn't know `last_cache_position` kwarg, and [the example script in model card of PaliGemma2](https://huggingface.co/google/paligemma2-3b-pt-224#use-in-transformers) fails to run with the following error:

```
TypeError: PaliGemmaForConditionalGeneration.forward() got an unexpected keyword argument 'last_cache_position'. Did you mean 'cache_position'?
```


Comments:
Hey! Thanks for clear reproducer, indeed the paligemma should be adopted to the latest changes in LM backbone. Would you like to open a PR?

Thanks @zucchini-nlp for the reply.

Sorry, but I'm not yet familiar enough with the transformers code to make a PR.


Oke, no problem, I will fix it :)

FYI. It's still working with 4.47.1.

4.48.3 should have fixed it!
Fixup another model + encoder decoder
Author: muellerzr
Pull request: yes

# What does this PR do?

For now not doing GA in encoder/decoder based models. This PR unblocks users

Also there was another model reported to need the fix.


Fixes https://github.com/huggingface/transformers/issues/36074


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36095). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Request more specific info from bug reporters when opening deepspeed issues
Author: Ben-Schneider-code
Pull request: no
Labels: name: Feature request

### Feature request

Hi!

I would like the bug reporters to be prompted (or have section to fill in the reports template) to provide `ds_report` info and `zero3` config when opening a bug report related to deepspeed integration (maybe it could be more general). Anything to make sure these bits of info are more likely to included upfront would make some of these issues much more actionable.

### Motivation

I've been looking at some deepspeed integration bugs lately (#28808,#29348,#31867), I noticed that often more deepspeed info has to be requested. I was wondering if some specific (and maybe **BOLDED**)  guidelines about what info to provide would go a long way when opening bug reports. I think a reminder to include `zero configs` and `ds_report` might be helpful. I believe this is particularily a pitfall for stuff that is often parsed in (configs, etc).  
Something like:

### Reproduction
Please provide a code sample that reproduces the problem you ran into. It can be a Colab link or just a code snippet.
If you have code snippets, error messages, stack traces please provide them here as well.
Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.
*If you are opening an issue related to one of the following please ensure the this info is included in your reproduction script:
Deepspeed - zero3 config, ds_report output,
Trainer - your trainer config file,
etc.*

@ArthurZucker @amyeroberts 

Comments:
Sounds good, would you like to open a PR to update the bug report here : https://github.com/huggingface/transformers/blob/e4c19d7a6c6e41ed1beb9290d610b9c92222058a/.github/ISSUE_TEMPLATE/bug-report.yml#L3 
?
Fix: Llama - adjust the rotary embedding dimensions.
Author: abdullahselek
Pull request: yes

# What does this PR do?

I noticed a small bug with llama-3.1-8B-Instruct where rotary embedding dimensions don't match q's last dimension. This PR  adjusts the rotary embedding dimensions if they don't match q's last dimension.

## Description of the fix

I use hidden_size=4096, num_hidden_layers=32, num_attention_heads=32 and num_key_value_heads=32 in my model configuration. When I try to get the model's input embeddings using input_ids, `apply_rotary_pos_emb` raises a runtime error because of unmatched tensor dimensions.

This fix checks if the last dimension of cos (and sin) matches the last dimension of q. If not, it slices cos and sin along their last dimension so that they have the same size as q. This is important because the mismatch (e.g., 128 vs. 32) causes the multiplication (q * cos) to fail.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

I think @ArthurZucker might be the best person to review this PR.

Comments:
Replace deprecated update_repo_visibility
Author: Wauplin
Pull request: yes

`update_repo_visibility` has been deprecated for a more generic `update_repo_settings` and will be removed in a future version. This PR replaces it in a test. I bumped `huggingface_hub` lowest version to 0.26 to be sure it's installed. Latest `huggingface_hub` version is 0.28.

Related to https://github.com/huggingface/huggingface_hub/pull/2502

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35970). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Lots of failing tests but I hardly see how it can be related to my PR :grimacing: 

@ArthurZucker I'm still getting an error but doesn't seem related to this PR:

```
FAILED tests/utils/test_import_utils.py::test_clear_import_cache - AssertionError: assert 'transformers.models.auto.modeling_auto' in {'PIL': <module 'PIL' from '/usr/local/lib/python3.9/site-packages/PIL/__init__.py'>, 'PIL.BmpImagePlugin': <module 'PIL.BmpImagePlugin' from '/usr/local/lib/python3.9/site-packages/PIL/BmpImagePlugin.py'>, 'PIL.ExifTags': <module 'PIL.ExifTags' from '/usr/local/lib/python3.9/site-packages/PIL/ExifTags.py'>, 'PIL.GifImagePlugin': <module 'PIL.GifImagePlugin' from '/usr/local/lib/python3.9/site-packages/PIL/GifImagePlugin.py'>, ...}
 +  where {'PIL': <module 'PIL' from '/usr/local/lib/python3.9/site-packages/PIL/__init__.py'>, 'PIL.BmpImagePlugin': <module 'PIL.BmpImagePlugin' from '/usr/local/lib/python3.9/site-packages/PIL/BmpImagePlugin.py'>, 'PIL.ExifTags': <module 'PIL.ExifTags' from '/usr/local/lib/python3.9/site-packages/PIL/ExifTags.py'>, 'PIL.GifImagePlugin': <module 'PIL.GifImagePlugin' from '/usr/local/lib/python3.9/site-packages/PIL/GifImagePlugin.py'>, ...} = sys.modules
```

Ok to merge ?

Yep! unrelated to your pr!
tokenizer_class: `LlamaTokenizerFast` becomes `LlamaTokenizer` after load + immediate save
Author: Qubitium
Pull request: no
Labels: name: bug

### System Info

I do not understand why but saving a loaded tokenizer changes the tokenizer class type. Unsure this is a usage error on my part of expected output from HF. 



### Who can help?

@ArthurZucker @itazap

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```py

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("DeepSeek/DeepSeek-R1-Distill-Qwen-7B")
tokenizer.save_pretrained('./tokenizer_copy')
```

`tokenizer_config.json`
* before `save`  `"tokenizer_class": "LlamaTokenizerFast" `
* after `save` `"tokenizer_class": "LlamaTokenizer"`

### Expected behavior

Tokenizer Class stays the same

Comments:
I don't understand why `Fast` has special code and name stripped here:

https://github.com/huggingface/transformers/blob/373e50e9703024eeb09f97945fdefde80acc2619/src%2Ftransformers%2Ftokenization_utils_base.py#L2459-L2461

I think the code should be using `isInstanceOf` and not string class name check?

Having a look, thanks! 

TBH this has not been touched in 4 years it seems. Main reason is that if you keep the non fast you can reload and does not really make a big difference

Duplicating PR notes here:

Some models like recently released `deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B` does not provide `vocab.json` but has native type of a `Fast` tokenizer class: `LlamaTokenizerFast`. If this bug is not fixed, the loading of the tokenizer will break when `use_fast=False` is applied on the fast-stripped and down-classed tokenizer. The current code assumes all `Fast` models have  incuded `vocab.json` for reasons I do not yet understand. 

Full Context: We found this bug after using GPTQModel to quantize various DeepSeek models and using EvalPlus to run benchmarks on it. EvalPlus always load slow tokenizer and pass `use_fast=False`.  Native models worked fine, qunatized models failed. We traced the bug to this tokenizer save() bug that had nothing to do with quantization nor did we modify the tokenizer so this state change is unexpected. 

EDIT: May we should just remove this  `Fast` down-cast all together?

Yep answered there, let's fix it
`DataCollatorForMultipleChoice` exists in the docs but not in the package
Author: bauwenst
Pull request: no
Labels: name: Documentation

name: Feature request

### Feature request

Move the `DataCollatorForMultipleChoice` implementation from the docs to `transformers.data.data_collator.py`.

### Motivation

The [`transformers` docs](https://huggingface.co/docs/transformers/tasks/multiple_choice) provide all collator code needed to run `ForMultipleChoice` fine-tuning for datasets like SWAG. The docs say that *"`transformers` doesn’t have a data collator for multiple choice, so you’ll need to adapt the `DataCollatorWithPadding` to create a batch of examples"*, but... why? Why not add the code given in the docs to the package? It's the same repo...

https://github.com/huggingface/transformers/blob/a06a0d12636756352494b99b5b264ac9955bc735/docs/source/en/tasks/multiple_choice.md?plain=1#L115-L160

### Your contribution

None, the code is already there.

Comments:
Hey @bauwenst! That seems like an oversight. Would you like to open a PR to fix the docs?
Move `DataCollatorForMultipleChoice` from the docs to the package
Author: bauwenst
Pull request: yes

# What does this PR do?
Fixes #34671 by moving the implementations for `DataCollatorForMultipleChoice` that appear in the docs (copied across four different languages) into the `transformers` package.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request) "Pull Request" section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@LysandreJik asked me to contribute this. @ArthurZucker may also be interested, and @stevhliu because the implementations came from the docs.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34763). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Could you just run the code quality tool to ensure that the code quality passes? You can install them with the following, from the root of your clone:
```
pip install -e ".[quality]"
```
And then run them with:
```
make fixup
```

> Could you just run the code quality tool to ensure that the code quality passes? You can install them with the following, from the root of your clone:

@LysandreJik I ran both `make style` and `make fixup` and have committed all the automatic changes. There is nothing else I can do on this PR, as far as I can see.

@ArthurZucker Yes, as I wrote before, I thought I had run everything I needed to (`make style` and `make fixup`). I don't know what else to do in order to complete this.

Thanks @bauwenst 🤗
[FEAT] Compatibility with dduf format from diffusers
Author: SunMarc
Pull request: yes

# What does this PR do ?

This PR adds the compatibility with dduf format from diffusers. 
For tokenizers, I only made it compatible with T5 and CLIP for now. 

Diffusers PR : https://github.com/huggingface/diffusers/pull/10037


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35093). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

T5 and CLIP are okay for now as they are the most used.