noamaanMulla-03/transformers-issues huggingface dataset:
https://huggingface.co/hf-internal-testing tiny random models need to be converted to safetensors
Author: sfc-gh-sbekman
Pull request: no

The problem is that many transformers CI tests rely on these tiny models and they are mostly `pytorch_model.bin` format, e.g. see: https://huggingface.co/hf-internal-testing/tiny-random-T5Model/tree/main
But `modeling_utils` has massively changed recently and uses a different code pass for `safetensor` model files - and most modern models use that. 

Which means transformers CI isn't testing the code properly.

For example a recent Deepspeed integration was broken because the tests use https://huggingface.co/patrickvonplaten/t5-tiny-random/tree/main but the code was doing something different for `pytorch_model.bin` files and thus a massive breakage introduced in https://github.com/huggingface/transformers/pull/36963 was missed. I looked at replacing it with  https://huggingface.co/patrickvonplaten/t5-tiny-random/tree/main but it has the same issue - doesn't have a .safetensor file.

You can see why the massive testing model update is needed if we want the tests to actually test:

https://github.com/huggingface/transformers/blob/0ef339ff1b63bb03a388c79bfbebec9085e10564/tests/deepspeed/test_model_zoo.py#L56-L90

cc: @ydshieh, @LysandreJik 

Comments:
Hi! 

I will update those repositories in transformers/tests/deepspeed/test_model_zoo.py so they have safetensors files.

I updated 22 repositories in `transformers/tests/deepspeed/test_model_zoo.py` that is in `hf-internal-testing`.

For example: https://huggingface.co/hf-internal-testing/tiny-random-t5-v1.1/tree/main

Those repositories contained files with old format that I couldn't detect what their architecture were, and I only load them with `AutoModel`.

Thank you very much, @ydshieh - that's great!

besides deepspeed tests I'd imagine many other transformers tests are impacted by this as well 

Yeah True. At some point, I made a script to create tiny model for pipeline testing in a more systematic way , and most of them contain safetensors

https://huggingface.co/hf-internal-testing/tiny-random-Gemma3ForConditionalGeneration/tree/main

Some earlier ones don't have safetensor, like

https://huggingface.co/hf-internal-testing/tiny-random-BertModel/tree/main

I will find the time to update the scripts, push safetensors, and use them to replace (at least some) tests that are still using the legacy repository that are not really maintained by me.

Thanks a lot, @ydshieh!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
fix docs serving typos.
Author: zhanluxianshen
Pull request: yes


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37936). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks review
enable xpu in test_trainer
Author: yao-matrix
Pull request: yes

128 PASSED, 27 FAILED for lacking of `optimizer_update_32bit` xpu op in BNB, the implementation in under going, once done, will pass after PR merged to bnb.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

@ydshieh, pls help review, thx.

cc @IlyasMoutawwakil 

the failed case seems not relevant to my PR

the failed case seems not relevant to my PR

@ydshieh , could we merge it? pls let me know if you have any other review comments. Thx very much.
Feat: Add ddp_param_to_hook_all_reduce option
Author: demoncoder-crypto
Pull request: yes

Addresses #36941, Tried to implement please let me know if its anything useful

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

cc @SunMarc 

Understood @SunMarc I will start to work on it
[vLLM] GPT2 with transformers backend
Author: ariG23498
Pull request: yes

I am pretty sure the merge resolution (#36943) removed the current changes.

You can run the following with `vLLM` to check the current changes.
```
python examples/offline_inference/basic/generate.py --model gpt2 --model-impl transformers --enforce-eager
```

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37975). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[offload] respect `max_memory` argument when factoring in unused reserved memory
Author: gante
Pull request: yes

# What does this PR do?

Our daily CI had >1000 new failures since #37920, on tests that use the `max_memory` argument in `from_pretrained`. This is because we now recapture unused reserved GPU memory, which caused us to go beyond the user-defined `max_memory`. This PR fixes it.

Affected tests, which set `max_memory`:
- `test_cpu_offload`
- `test_disk_offload_bin`
- `test_model_parallelism`

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37982). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
feat: Add encode, patch_token to RagTokenizer (fixes #35532)
Author: demoncoder-crypto
Pull request: yes

Addresses #35532. Please do let me know if the implementation is correct

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

cc @arthurzucker @itazap 

hello @demoncoder-crypto 🤗 Thanks for the proposal! Could you share more about the use cases or scenarios where this `patch_token` would be particularly beneficial for RAG models, as opposed to existing token like `[SEP]`? I haven't seen as many requests for a `patch_token` so just curious !
Force torch>=2.6 with torch.load to avoid vulnerability issue
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title, following the vulnerability report received. `torch.load` in unsafe even with `weights_only=True` for any version < 2.6

Whenever we do not have `weights_only=False` explicitly, either from user input or internally, we should raise an Error asking to upgrade torch.

This PR does not update the files in `examples/legacy`, as they are, as their name suggest, legacy examples

cc @SunMarc as well, how restrictive is it from the Trainer point of view?


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37785). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey, thanks for looking into this! Yes my only concern is indeed the trainer, which is why I want thoughts from Marc before merging 😉 But in general this is not as restrictive as it seem, as most people are already on torch >= 2.6, and we mostly use safetensors anyway (worst case, being prompted to upgrade torch is a 1 time thing)

For tests it's no matter, we run them with torch >= 2.6 anyway. About a wrapper, I choose not to do it because I think this one is more explicit, i.e. we still see the use of the torch primitive easily. A wrapper would require to replace each call as well anyway (except if patching the main `torch.load` but it's a very bad practice).

In general, we do need to patch every/most call, as otherwise people will keep flagging the library as unsafe, even if it's technically only local save, then local load of the same files 🙂

From Trainer point of view, this will only impact resuming training from a checkpoint (as we have to load the different states)  and a few other small features. So I think it should be good to merge. Let's see if we get any issues in the following days, not sure how restrictive this PR will be. 

Alright, merging then! Thanks for the input @SunMarc 🤗 Also got the green light from @LysandreJik offline 

Ic, thanks for the detailed response! That makes sense :+1: 

(and I agree wrappers are definitely less explicit)

@SunMarc are there any workarounds for resuming from checkpoint as it's currently failing our CI for that use case when loading the optimizer and scheduler.
<img width="1017" alt="Screenshot 2025-05-06 at 10 34 20 PM" src="https://github.com/user-attachments/assets/863822df-3d75-4fc5-a722-f555d5ea693b" />


> @SunMarc are there any workarounds for resuming from checkpoint as it's currently failing our CI for that use case when loading the optimizer and scheduler.

Not sure what would be the best solution, we can potentially add a flag so that we bypass the torch check. Or for this specific test, maybe require to have torch >=2.6. cc @Cyrilvallez
[chat template] separate jinja logic from tokenizers 
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/36713


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37602). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@Rocketknight1 , kind ping
remove duplicate code
Author: kaixuanliu
Pull request: yes

@amyeroberts, @qubvel pls help review.




Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37991). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update AMD docker with needed packages
Author: Cemberk
Pull request: yes

# What does this PR do?

The AMD Dockerfile is based on ubuntu base container we expect pytorch docker to have needed hip dependencies also added bitsandbytes and flash attention packages to fix failing tests these changes were discussed in a meeting. These changes are expected to solve 99/144 test cases posted in slack and solves 91/93 require_bitsandbytes tests  @ivarflakstad 



Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.
[CSM] tiny fix on generation
Author: eustlb
Pull request: yes

# What does this PR do?

Nit fix when `output_audio=True`.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38001). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix Qwen models export with torch 2.7
Author: guangy10
Pull request: yes

# What does this PR do?

Due the regression in `torch 2.7` reported in https://github.com/pytorch/pytorch/issues/150994, export with `strict=True` will fail. The workaround is to toggle the `strict=False` on 2.7.


Fixes #37876


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. #37876
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @gante @ydshieh

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

cc: @tugsbayasgalan
Add BLIP and GIT model support to image-to-text pipeline
Author: mayankdhingra02
Pull request: yes

# What does this PR do?
This PR adds support for using BLIP and GIT models with the `image-to-text` pipeline.

Both model types already had underlying compatibility in `ImageToTextPipeline`, but they were not formally registered in `SUPPORTED_TASKS`, which prevented `pipeline(..., model="Salesforce/blip-image-captioning-base")` from working directly.

### Changes:
- Updated `SUPPORTED_TASKS["image-to-text"]["pt"]` to include `"blip"` and `"git"` model types.

### Motivation:
This enables easier access to popular models like BLIP and GIT via the pipeline interface, improving usability for multimodal captioning tasks.

---

Fixes: #21110 

## Before submitting
- [x] I’ve tested the change locally with BLIP models and it works correctly.
- [x] I’ve read the [contributor guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request).

## Who can review?

@Rocketknight1 @zucchini-nlp (for visual-language model pipelines)


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Got it — thanks for the update! I’ll go ahead and close this PR then. @zucchini-nlp
make aya vision 5 integration tests pass on xpu
Author: yao-matrix
Pull request: yes

tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_forward
tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_generate_text_only
tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_generate_chat_template
tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_batched_generate
tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_batched_generate_multi_image

@ydshieh @IlyasMoutawwakil , pls help review, thx.


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

run-slow: aya_vision

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/aya_vision']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37990). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: aya_vision

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/aya_vision']
quantizations: [] ...
`image_transforms:pad` throws `ValueError` if the input contains a batch dimension
Author: sebasv
Pull request: no
Labels: name: bug

### System Info

```
- `transformers` version: 4.31.0
- Platform: macOS-15.3.2-arm64-arm-64bit
- Python version: 3.11.8
- Huggingface_hub version: 0.28.1
- Safetensors version: 0.5.2
- Accelerate version: 1.6.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO
```

### Who can help?

@amyeroberts @qubvel 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I ran into this issue using the `Mask2FormerImagePreprocessor` but a minimal reproducible example would be

```python
from transformers.image_transforms import pad
import numpy as np

boring_image = [[[0]]]
batched_boring_image = [boring_image]
pad(image=np.array(batched_boring_image), padding=((0,0),(0,0)))

# output

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (4,) + inhomogeneous part.
```

The problem is that the `pad` function expands the padding argument to `(0, (0,0),(0,0),(0,0),(0,0))`, ie the input for the batch dimension is a scalar, not a tuple.

What I don't understand is how I'm the first to hit this issue. This code has been stable for years, still if I do something as straightforward as 

```python
image = np.array([[[1, 2, 3]]])
mask = np.array([[[1, 1, 0]]])
print(mask.shape)
instance_id_to_semantic_id = {1: 1}
Mask2FormerImageProcessor.from_pretrained(
        "facebook/mask2former-swin-small-coco-instance", do_normalize=False, do_reduce_labels=False, ignore_index=0
    )(
        images=[image],
        segmentation_maps=[mask],
        instance_id_to_semantic_id=instance_id_to_semantic_id,
        return_tensors="pt",
    )

```

I already get this error. Am I doing something wrong here?

### Expected behavior

no exception thrown

Comments:
Fix `Optional` typing
Author: qubvel
Pull request: yes

# What does this PR do?

Make style CI green again

cc @ydshieh 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38018). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Print commit SHA on slack message for new model notification.
Author: ydshieh
Pull request: yes

# What does this PR do?

Print commit SHA on slack message for new model notification.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38019). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[CI] remove duplicated message on GH comment to run slow tests
Author: gante
Pull request: yes

# What does this PR do?

`This comment contains run-slow, running the specified jobs:` appears twice on the GH comment to trigger slow tests ([example](https://github.com/huggingface/transformers/pull/37922#issuecomment-2853876256)).

This PR removes one of the copies 🤗 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37970). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix device mismatch by moving num_items_in_batch to loss device in fixed_cross_entropy (#37886)
Author: iamsernine
Pull request: yes

Fixes: https://github.com/huggingface/transformers/issues/37886

This PR ensures that the num_items_in_batch tensor is moved to the same device as the loss tensor before performing division inside the fixed_cross_entropy function. This prevents runtime device mismatch errors when models are trained on non-default devices (e.g., CUDA).

🔧 Changes made:
Updated fixed_cross_entropy to move num_items_in_batch to loss.device before division when reduction is set to 'sum'.

This fix is particularly relevant for ForCausalLMLoss, where num_items_in_batch may be on a different device than logits or loss.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

cc @SunMarc!
Skip `test_push_to_hub_with_saves_each_epoch` for now
Author: ydshieh
Pull request: yes

# What does this PR do?

Need to check if it is staging hub issue or anything else.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38022). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.