noamaanMulla-03/transformers-issues huggingface dataset:
fix failing `test_sdpa_can_dispatch_on_flash`
Author: ydshieh
Pull request: yes

# What does this PR do?



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39259). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Good but:
```         if pixel_values is not None:
            vision_outputs: SamVisionEncoderOutput = self.vision_encoder(pixel_values, **kwargs)
>           image_embeddings = vision_outputs.last_hidden_state
E           AttributeError: 'tuple' object has no attribute 'last_hidden_state'
``` 

is related for sam!

@ArthurZucker Fixed, all good now

@ArthurZucker If you have time, let's see if we can merge this one before the end of today?

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: t5gemma
Update docstring for glm4v
Author: ddeellttaa
Pull request: yes

What does this PR do?
This PR updates the class docstring example for a multimodal model, addressing the following issues:

Corrects the input type: Changes the model input from a list to a tensor, aligning with model requirements.

Properly handles image input: Adds the image to the model input for correct multimodal inference.

Updates to processor usage: Replaces the use of tokenizer with processor for appropriate multimodal preprocessing.

These changes ensure that the documentation provides a runnable and accurate example for users working with multimodal models.


## Before submitting
- [✅ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ✅] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ❌] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [no need ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ no need] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: glm4v
Glm 4 doc
Author: zRzRzRzRzRzRzR
Pull request: yes

Improve the introduction documentation for GLM-4-0414 and GLM-4.1V.
Improve the pytest testing for GLM-4.1V.

Comments:
run-slow: glm4v

This comment contains run-slow, running the specified jobs: 

models: ['models/glm4v']
quantizations: [] ...

run-slow: glm4v

This comment contains run-slow, running the specified jobs: 

models: ['models/glm4v']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39247). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: glm4v

This comment contains run-slow, running the specified jobs: 

models: ['models/glm4v']
quantizations: [] ...

run-slow: glm4v

This comment contains run-slow, running the specified jobs: 

models: ['models/glm4v']
quantizations: [] ...

run-slow: glm4v

This comment contains run-slow, running the specified jobs: 

models: ['models/glm4v']
quantizations: [] ...


run-slow: glm4v



**[For maintainers]** Suggested jobs to run (before merge)

run-slow: glm4v

run-slow: glm4v

This comment contains run-slow, running the specified jobs: 

models: ['models/glm4v']
quantizations: [] ...

Tests green, merging
fix colpali mapping
Author: ManuelFay
Pull request: yes

# What does this PR do?


Fixes # (issue)

Fixes the issue of Colpali not working sincer the breaking changes with VLMs in 4.52.
Most models were converted, not the native HF colpali making the code in docs here fail: https://huggingface.co/docs/transformers/model_doc/colpali.

@yonigozlan @zucchini-nlp 


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: colpali

@yonigozlan This fails because of the modular that copies over the mapping to ColQwen2 (which does not need it).
I don't know how to fix this, but in this PR you already havce the problem identified, and the checkpoint mapping done so it should go quickly for you :)
[shieldgemma] fix checkpoint loading
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, reported by @ydshieh internally

(how many more models we have that load a VLM internally 🥲 )

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39348). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> (how many more models we have that load a VLM internally 🥲 )

😅 Not sure why I don't report at the day after the PR. I will check

(if you ever want to double check, but I think nothing remaining in the following report)

https://huggingface.co/datasets/hf-internal-testing/transformers_daily_ci/blob/main/2025-05-08/ci_results_run_models_gpu/new_model_failures.json

The reason why this `shieldgemma` was not reported at that day is: it's already failing before (gated repo), so not reported as `new failures`

Let me push a fix for the test and we could go

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: shieldgemma2

wait, the test needs a bit more fix. On it

all good ✅
[BLIP] remove cache from Qformer
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Qformer doesn't use cache and is not generative, this PR deprecates it for one version. We'll remove all usage of cache in non-generative models after 4.55 :)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39335). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: blip_2, instructblip, instructblipvideo

run-slow: blip_2, instructblip, instructblipvideo

This comment contains run-slow, running the specified jobs: 

models: ['models/blip_2', 'models/instructblip', 'models/instructblipvideo']
quantizations: [] ...
Deprecate AutoModelForVision2Seq
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, we'll remove it anyway in the end so let's start raising warnings. I already ask all new models to not use `Vision2Seq`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38900). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I don't remember all the discussions around this, but is ImageTextToText what will be fully future-proof? So we don't go with ForCausalLM in the end?

I don't think we'll remove `ImageTextToText` in the near future though there were discussions on unifying "AutoForMultimodal" under one umbrella mapping.  didn't yet start working on new auto class, but I think it will be a new auto-mapping that:

1. Unifies all generative model, either under one class or two classes (one for text and other for multimodality)
2. Main goal is to clean up garbage repetitive code and we don't plan to delete existing auto classes, otherwise it might break a lot of external libraries that depend on us. Until new auto is added, we'll have a surge of new VLMs added to the mapping which makes it harder to just remove it

Personally, I'm in favor of keeping `ImageTextToText` for a while as the recommended mapping

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto
Fix Lfm2 and common tests
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

cc @ydshieh 

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: lfm2

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39398). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove do_reduce_labels Argument from model initialization in run_semantic_segmentation_no_trainer
Author: eromomon
Pull request: yes

# What does this PR do?

When running ```run_semantic_segmentation_no_trainer.py``` from ```example/pytorch/semantic-segmentation``` using the command provided in the Readme file:

```bash
accelerate launch run_semantic_segmentation_no_trainer.py --output_dir segformer-finetuned-sidewalk --with_tracking --push_to_hub
```

An error is returned:
```bash
TypeError: SegformerForSemanticSegmentation.__init__() got an unexpected keyword argument 'do_reduce_labels'
```
In the Python script, the model is instantiated with the class ```AutoModelForSemanticSegmentation```, which does not have listed ```do_reduce_labels``` as a parameter. This causes the test to fail.


Proposed Change:
This PR proposes removing line 333 from ```run_semantic_segmentation_no_trainer.py``` to resolve the issue.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


@amyeroberts, @qubvel hi, I believe you could review this PR, as it pertains to the computer vision model field.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39322). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Passing do_reduce_labels to the AutoImageProcessor works fine. The PR has been updated  to reflect this change.

Hi @qubvel  I've requested your review for the latest PR update. Please let me know if you have any comments.
Fix: Add version check for timm to support mobilenetv5 models (fixes #39208)
Author: VIGNESH15103
Pull request: yes

### What does this PR do?

Fixes issue #39208: Adds a version check for `timm` to ensure support for models like `mobilenetv5_300m_enc`, which require `timm >= 0.9.10`.

### Motivation

The issue occurred because the `mobilenetv5_300m_enc` model name was not recognized in older versions of `timm`. This PR adds a conditional check to raise an informative `ImportError` if an unsupported version is detected.

### Changes

- Added a version check using `packaging.version` inside `TimmWrapperImageProcessor.__init__`.
- Raises an error if a `mobilenetv5` model is used with `timm < 0.9.10`.

### Issue Link

Closes #39208

### Reviewer Suggestion

@amyeroberts @qubvel (vision models)


Comments:
Following the feedback:

- I removed the version check from image_processing_timm_wrapper.py.
- Instead, I now catch RuntimeError during timm.create_model(...) inside both TimmWrapperModel and TimmWrapperForImageClassification, as suggested.
- Clear ImportError is raised when the architecture is unsupported, prompting users to upgrade timm.
- Applied ruff formatting to resolve CI failures.

**Kindly requesting a review and workflow approval when convenient. Thanks a lot!** 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: timm_wrapper

Hi! I've addressed all your feedback:

-  Wrapped model creation in a helper method as suggested.
-  Added specific exception handling.
-  Removed the redundant version check in the image processor.
-  Ran `ruff` formatting and all CI checks pass now.

Could you please take another look and let me know if there's anything else to improve? If everything looks good, it would be awesome if you could approve and merge it 😊
deepspeed typing Dict error
Author: cnut1648
Pull request: no

### System Info

- `transformers` version: 4.32.0.dev0
- Platform: Linux-5.10.179-171.711.amzn2.x86_64-x86_64-with-glibc2.26
- Python version: 3.9.16
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- Accelerate version: 0.22.0.dev0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sgugger 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Right now any command with `--deepspeed /path/to/json` will fail and throw the following error
```
--deepspeed: invalid Dict value
```
This is reported in https://github.com/huggingface/transformers/pull/24549#issuecomment-1613046347 but the merged fix https://github.com/huggingface/transformers/pull/24574 does not resolve this.
In fact the `deepspeed: Union[str, Dict]` field in `training_args.py` still raise the error when the `deepspeed` is passed as the string. This seems to be a limitation of python dataclass.

### Expected behavior

`deepspeed` flag should support string.

Comments:
Yes, this has been broken by #24550 again...

@sgugger Any chance this got broken once again on the latest version (4.53.2)?
:rotating_light: Totally rewrite how pipelines load preprocessors
Author: Rocketknight1
Pull request: yes

Pipelines load preprocessors in a very hacky way, by checking various mappings. This creates a lot of weird side-effects, such as models needing to be added to tokenizer mapping lists or else [they can't be used in pipelines](https://github.com/huggingface/transformers/pull/38802).

This PR overhauls everything. Every pipeline has attributes `_load_processor`, `_load_image_processor`, `_load_feature_extractor` and `_load_tokenizer`. These are set on the base `Pipeline` class and should be overridden by all subclasses. They have the following possible values:

- `True` (this preprocessor must be loaded, always try to load it and throw an error if we fail)
- `False` (this preprocessor is not used by the pipeline, don't try to load it)
- `None` (this preprocessor is optional, try to load it but continue even if loading fails)

This lets us delete a lot of mappings and simplify things!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38947). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

all the tests are green on my first commit this is terrifying

Tests seem to pass locally so cc @arthurzucker @cyrilvallez for core maintainer review!

@cyrilvallez you were totally right! The tests should have just checked if `load_tokenizer` was `True` or `None`. The block was being skipped entirely when a tokenizer string or tuple was passed in as a kwarg. I've refactored to fix it for all of the preprocessor types.

Merging!
Paligemma model card needs update
Author: punitvara
Pull request: no

Hi 

I found a minor problem with paligemma model card. How can I raise a PR to fix it ? I am first time contributor. I raised PR. Whom should I mention to review it ? 
https://huggingface.co/google/paligemma-3b-pt-896

Comments:
Hi @punitvara, you can ping @stevhliu for model card updates.

Thanks @Rocketknight1 I sent PR on model card. Let me see if I can find him on discord. 

Hi, feel free to ping me on the PR! :)

I pinged you on PR @stevhliu 

~Hmm I don't see it, would you mind sending me the link to it?~ Ah I see, it is on the Hub!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closed in this [PR](https://huggingface.co/google/paligemma-3b-pt-896/discussions/10)
Latest Transformers release causes CUDA out-of-memory errors during VisionLLM fine-tuning
Author: ruiite
Pull request: no
Labels: name: bug

### System Info

With verion of transfomers == 4.53.0 
Script: https://github.com/huggingface/smollm/blob/main/vision/finetuning/SmolVLM2_Video_FT.ipynb
This script for tuning SmolVLM2 on video undrestanding causing CUDA out of memory problem. 
While this script on the same hardware with transformers == 4.51.3 is working perfectly. 
Vision model was freezed during the training. Not using LORA or QLORA.

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce: 
1. Run script with transformers ==4.51.3 on GPU with 24 Gigabytes of memory with LORA = False, Qlora = False, SMOL = TRUE.
2. Run the same script with transformers==4.53.0.

### Expected behavior

With version of transformers of 4.53.0 you will get CUDA out of memory. While with 4.51.3 is not. 

Comments:
Must have been https://github.com/huggingface/transformers/pull/39147 which fixed it in patch release. SmolVLM in `4.53.0` stopped sampling frames automatically and broke BC

Since the issue is already resolved and included in latest patch release, you need to install from `4.53.1`. We can't change an already released past version :)
Use np.pad instead of np.lib.pad.
Author: rasmi
Pull request: yes

# What does this PR do?

`audio_utils.py` still uses `np.lib.pad` from numpy 1.X rather than `np.pad`. This fixes the single remaining call in `transformers` to use `np.pad`.


## Who can review?

cc: @ArthurZucker, @Rocketknight1


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39346). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
LlamaAttention forward function type hint is incorrect from new Branch
Author: ArkVex
Pull request: yes

Hi, this PR fixes a small issue in the LlamaAttention class. The return type in the forward method currently shows three values, but the function actually returns only two. This seems to have been missed during the attention refactor (possibly in PR https://github.com/huggingface/transformers/pull/35235).

I’ve updated the type hint to reflect the actual return values, just to avoid confusion for anyone reading or using the code. Let me know if any other changes are needed. Happy to help!

Comments:
@Rocketknight1 same error again


![image](https://github.com/user-attachments/assets/956a2cbf-e308-47ed-b9cb-994cb63236ed)
its saying this change is causing the issue...but this was the whole point for this pr

Hey @ArkVex , There are many models which inherit from llama modules. So, running `make fix-copies` should make the CI happy

![455077514-5c43c10a-e0bd-4bf1-9f2c-51a96534a46a.png](https://github.com/user-attachments/assets/f2a1999a-b9bf-4856-90e9-b455cb11e41a)

I did...but I don't know why it is throwing error 

It's working fine for me. Can't you see any changes in your branch after running. Might be that the changes have been applied and in the next run there is no diff. Otherwise install `pip install -e '.[quality]'` and then once again try it.

Hi @ArkVex, the problem is that `make fix-copies` detects changes by comparing against your **fork's** `main` branch and not **Hugging Face's** main branch. Because you've made the same changes on `main`, it doesn't think any files have changed in your PR. You should reset your `main` branch to match `huggingface/transformers@main`, and then pull the changes to your local machine's main branch. After that, everything should work.

If it helps, the reset command(s) may look something like:

```
git switch main
git remote add upstream https://github.com/huggingface/transformers.git
git reset --hard upstream/main
```

(X-Ref to link this to the last MR: https://github.com/huggingface/transformers/pull/38795)
Avoid registering pytree when using FSDP
Author: kaixuanliu
Pull request: yes

When using FSDP, this `register_pytree_node` operation will cost lots of extra memory. We found after this PR: https://github.com/huggingface/transformers/pull/35873, we cannot finetune 70b model using FSDP due to OOM issue.

Comments:
@SunMarc @ArthurZucker @IlyasMoutawwakil pls help review, thx!


@kaixuanliu do you mean this PR https://github.com/huggingface/transformers/pull/36311 where it was added ?
Do you have exact measures of how much it costs when FSDP is not used and when it's used, because the operation itself has nothing to do with FSDP.

@IlyasMoutawwakil ，Oh yes, it's 36311, not 35873. And for the extra memory, I made an experiment: I use 4 processes to do FSDP finetune  with llama2-7b model, and compare the maximum memory consumption for two configurations. Result shows w/ `register_pytree_node` operation, it will cost extra ~12GB memory for each card.

After consideration, I think actually we do not need KVCache at all during finetune/training stage. So will close this PR, and set `use_cache=False` explicitly in application code.
Fix typo in `/v1/models` output payload
Author: alvarobartt
Pull request: yes

# What does this PR do?

This PR fixes a typo affecting the output payload of the `/v1/models` endpoint in `transformers serve`.

## Before submitting

- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [ ] Was this discussed/approved via a GitHub issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker or @gante

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39414). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.