noamaanMulla-03/transformers-issues huggingface dataset:
Add Idefics2 Fast ImageProcessor
Author: sushmanthreddy
Pull request: yes
Labels: name: Vision

name: Processing

related #36978

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

@yonigozlan @Rocketknight1  if u get chance pls take look for this PR



cc @yonigozlan @qubvel 

Hi @sushmanthreddy, huge thanks for working on the PR! It would be super helpful for us if you could make sure quality tests and related image-processor tests all pass before the initial review 🤗 

@qubvel I have kept commit for the quality checks here . 


and below are screenshots of the tests passing screenshot of idefics2 model 

<img width="1301" alt="Screenshot 2025-03-28 at 8 55 42 PM" src="https://github.com/user-attachments/assets/7edfacc2-cf78-4895-8e0b-f4b786334e5d" />
<img width="1295" alt="Screenshot 2025-03-28 at 8 56 09 PM" src="https://github.com/user-attachments/assets/3853385c-1a42-40a7-b0c3-c492a154e55b" />


@qubvel I have run quality checks in my local machine ,they are running fine but here cli they are failing dont knew the reason, can someone pls help me?


@sushmanthreddy there might be something wrong in your .vscode/settings.json config or ruff config? Is your `Ruff: Line Length` set to 119 in the vscode settings?

BTW, the ruff was updated last week, make sure you have `ruff==0.11.2`, you can simply update as follows:

```
pip install -U ruff==0.11.2
```
Static cache should support indexing
Author: tugsbayasgalan
Pull request: yes

# What does this PR do?

past_key_values[0][0] is still a pretty common pattern to access kv cache and it is not compatible with StaticCache today. 

cc: @guangy10 


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers.

cc @gante 

@tugsbayasgalan it is intentionally not enabled on most `Cache` classes -- only `DynamicCache` has it, for backwards compatibility with the legacy format. `DynamicCache` and the legacy cache are equivalent, but the same can't be said of other `Cache` classes.

I'm leaning towards not adding this capability, unless there is an important problem this change solves :)

> @tugsbayasgalan it is intentionally not enabled on most `Cache` classes -- only `DynamicCache` has it, for backwards compatibility with the legacy format. `DynamicCache` and the legacy cache are equivalent, but the same can't be said of other `Cache` classes.
> 
> I'm leaning towards not adding this capability, unless there is an important problem this change solves :)

This is for executorch-optimum integration where we advertize a recipe to export a model to executorch device. StaticCache is used as a recipe, but when I try on the transformer models, lot of them have code like:
```
past_key_values[0][0]
```
which is not compatible with StaticCache. Are we going to rewrite the model key-value cache code in the long term to be compatible with HuggingFace provided classes? 
cc: @guangy10 

@tugsbayasgalan the pattern you described, `past_key_values[0][0]`, should only exist in legacy codepaths -- codepaths designed to handle the legacy cache format. If this is not the case, let me know :D

Yes, we are rewriting model internals to handle the `Cache` abstraction (which includes `StaticCache`). Some older models don't yet support the `Cache` format, we're slowly going through all models 🤗 Different caches have different update steps, and thus we need to have the abstraction layer to limit model-level cache-dependent code.

I'd like to avoid this indexing pattern mostly to contain long-term maintenance costs. The more functionality we add to our codebase, the bigger are our long-term maintenance costs. However, if the functionality unlocks new use-cases, we're open to it!
Refactor attention for SigLIP based models
Author: qubvel
Pull request: yes
Labels: name: Vision

name: Multimodal

# What does this PR do?

Refactor attention for SigLIP-based models to use updated attention_interface function instead of class-based approach

 - SigLIP
 - SigLIP2
 - Idefics2
 - Idefics3
 - SmolVLM

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36981). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: siglip, siglip2, smolvlm, phi4_multimodal

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/phi4_multimodal', 'models/siglip', 'models/siglip2', 'models/smolvlm']
quantizations: [] ...

Could you also update the tests, so

> FAILED tests/models/phi4_multimodal/test_modeling_phi4_multimodal.py::Phi4MultimodalIntegrationTest::test_audio_text_generation - ValueError: The repository for microsoft/Phi-4-multimodal-instruct contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/microsoft/Phi-4-multimodal-instruct.

could run ? 🙏 

@ydshieh do you have an idea how to fix it?

trust_remote_code=ture I guess?

hmmm, is it intended to have `trust_remote_code` for the test? I suppose the model is supported natively
cc @Cyrilvallez 

or better if we can use a repo. that is not using remote code. This model is in `transformers` so there might be such repositories.

But I don't know if we intend to test this specific one in the first place 

Hi @Cyrilvallez Do you know why the docstring and the tests are using [this repository](https://hf.co/microsoft/Phi-4-multimodal-instruct) that uses remote code?

The hub PR has not been merged yet, and I did not open it yet when I merged the model (so could not add a revision to the model loading), so they are expected to fail until it's merged! Sorry about that! You can skip them for now if blocking!

@ydshieh Just realized that the model is actually not touched in the PR (just two lines swapped in the  module init, so no need to run slow tests for it)
Avoid pipeline test failing related to Hub call
Author: ydshieh
Pull request: yes

# What does this PR do?

Try to avoid pipeline test failing by (potentially) reducing the number of calls to hub.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37170). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ruff format ?

> ruff format ?

I ignore ruff during the process. Only matter when it comes to the end :-)

Wait this one to merge:)

Hub received DOS...

Yes, I think the issue was that we were accidentally DOSing the Hub by repeatedly downloading this 10mb dataset! Maybe we should either:

1) Make a smaller dataset with only 1-2 samples and use this for the tests
2) Rewrite the tests that load from `librispeech_asr_dummy` so that they only read one sample from the Hub, instead of loading everything. I think a lot of tests call `dataset.sort()`, which might implicitly load the entire dataset. Can we make sure we only stream the one sample used in the test?

However, I think this is a good solution for now to get the CI green, and we can do something more complex afterwards!

Yeah, let's merge this one and see how it goes (with all PRs CI running), then decide what extra could be done too.
Fix meta state dict loading with quantizers
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

This is a temporary fix, while seeing if we can refactor hqq and b&b 4bits to work without the full state dict but only a given param one at a time.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37136). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Verified. Thanks for your fix.
[doc] Fix link for Quark quantization page
Author: BowenBao
Pull request: yes

# What does this PR do?

Fixes link to Quark page in quantization docs, the previous link shows 404.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

cc @SunMarc @MekkCyber for review


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37179). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
enable test_assisted_decoding_in_different_gpu UT on XPU
Author: yao-matrix
Pull request: yes

pytest -rA tests/generation/test_utils.py::GenerationIntegrationTests::test_assisted_decoding_in_different_gpu  **passed**


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

cc @ydshieh

Thanks!

@faaany Could you help having a double check? 🙏 As well as any comment if you have any.

LGTM
fix XPU UT error case brough by RNG difference btw XPU and CUDA
Author: yao-matrix
Pull request: yes

**Symptom**

`pytest -rA tests/generation/test_logits_process.py::LogitsProcessorTest::test_watermarking_processor` will assert out with below logs while running on XPU.

> elf.assertTrue((out[:, 1] == scores_wo_bias + watermark.bias).all())
> E       AssertionError: tensor(False, device='xpu:0') is not true

**Root cause**
`WatermarkLogitsProcessor` uses device RNG `self.rng = torch.Generator(device=device)` to generate random numbers, this leads to different generated random number even with same seed, since RNG is device-dependent. 
In this case, CUDA's generated greenlist_ids is:  `greenlist_ids: tensor([19, 10,  8,  9,  1], device='cuda:0')` while XPU's greenlist_ids is `greenlist_ids: tensor([10, 17,  5,  4,  3], device='xpu:0')`. But the UT only check id 0, which leads to failure on XPU

**How to Fix**
set to-be-checked `greenlist_id` on XPU to 3, as `greenlist_id = 3 if torch_device == "xpu" else 1`. 
 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

cc @gante @ydshieh
Different limits for saving only model weights and saving full checkpoints
Author: Tim-Siu
Pull request: no
Labels: name: Feature request

### Feature request

I wonder if it is possible to have different limits for saving full checkpoints and model weights.

Currently, only `save_total_limit` control the limits.

### Motivation

Sometimes people need intermediate weights for analysis, but saving full checkpoints take too much space.

`save_only_model` can be helpful, but we may still want the latest full checkpoint if we need to resume training.

In this case, it would be great to have seperate limits for saving full weights and checkpoints.

### Your contribution

If this feature is welcome, I can submit a PR.

Comments:
Thanks for the feature request! This makes a lot of sense, as saving full checkpoints takes up a lot of space, while model weights might still be useful for analysis.

I propose adding two separate arguments:

- save_checkpoint_limit for controlling full checkpoints
- save_model_limit for model weights

I'll work on implementing this and submit a PR soon. Let me know if you have any specific suggestions!
[chat-template] fix video loading
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

The video loading logic was broken after the last feature of `load_audio_from_video`. The condition should not be applied to loading videos, but only to audio from video. The CI didn't catch anything because we don't have `pyav` in general dependencies

The fix is to apply `if/else` to only small block when audio is loaded

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37146). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @ydshieh , is it possible to run `pyav` related tests in CI?

Hi, we can add `video` to

> transformers[sklearn,sentencepiece,vision,testing,tiktoken,num2words]"

inside

> docker/torch-light.dockerfile

Let me know what you think!

@ydshieh yeah, works for me, thanks 

ok, i will update and come back to you once done

I will push some changes to this PRs

Hi , there are

```
FAILED tests/models/qwen2_vl/test_processor_qwen2_vl.py::Qwen2VLProcessorTest::test_chat_template_video_custom_sampling - AssertionError: 14400 != 1
FAILED tests/models/qwen2_vl/test_processor_qwen2_vl.py::Qwen2VLProcessorTest::test_chat_template_video_special_processing - AssertionError: 1756800 != 1
```

See

https://app.circleci.com/pipelines/github/huggingface/transformers/124215/workflows/973d3524-2a9f-42d3-a30e-a79b42f0c3ec/jobs/1675196/parallel-runs/6?filterBy=ALL

We can keep the changes in .circleci/create_circleci_config.py so the CI will run with dev images. We will revert it before merge

Oh cool, thanks for finding broken test ❤️ 

Yeah, i think for CI it is better to submit a PR of its own, with core maintainer's approval as well. Should I revert that after making sure all is green?

The change in .circleci/create_circleci_config.py should be reverted before merge.

But I should push a commit to add `video` in `docker/torch-light.dockerfile` that should be kept (which is good to commit along this PR together).
Skip code `307` in `RequestCounter`
Author: ydshieh
Pull request: yes

# What does this PR do?

A tiny PR to avoid some tests to fail when we have code `307` in the http request, which means

> On the other hand, a 307 redirect indicates that the URL being requested by the user has moved to a temporary location, but will return.

(The CI was red because of this, but it become normal today, but this PR is trying to avoid this in the future).

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36953). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Let's merge to avoid some red CI with hub job failing
Add device workaround for int4 weight only quantization after API update
Author: jerryzh168
Pull request: yes

Summary:
We add the new torchao API support in hf transformers: https://github.com/huggingface/transformers/pull/36526 one thing that's missing is it does not account for int4 weight only quant config only works on cuda, this PR adds back the workaround

also updated the version requirement to > 0.9 temporarily so that we can use the torchao nightly before 0.10 is released, we should chagne this back before land

Test Plan:
local test: https://gist.github.com/jerryzh168/0e749d0dab40e2a62a7f2e48639f77b5 (we can setup deserialization test later when we can quantize a small model and host in a stable place like TinyLlama/TinyLlama-1.1B-Chat-v1.0)

Reviewers:

Subscribers:

Tasks:

Tags:


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

cc @sunmarc @mekkcyber

There is still a failing test on the CI [check_repository_consistency](https://app.circleci.com/pipelines/github/huggingface/transformers/123493/workflows/2350d667-f137-4b57-9f1b-4fc11cddf043/jobs/1667321?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link&utm_content=summary) - Failed

we recently updated the ruff version so this is probably the code_quality test is not passing. Can you please update your version of ruff @jerryzh168 ?
Try to avoid/reduce some remaining CI job failures
Author: ydshieh
Pull request: yes

# What does this PR do?

Similar to #37170, another try to avoid/reduce some remaining CI job failures

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37202). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

It's ready for an review. The [exmpale job](https://app.circleci.com/pipelines/github/huggingface/transformers/124472/workflows/69fda893-bdba-428e-ab44-35629135a6c8/jobs/1678197) has some issues, but it's from CircleCI side not our side.
Skip flaky test counting calls to Hub
Author: Rocketknight1
Pull request: yes

This test works locally for me but seems very flaky on the CI. I'm going to disable it for now until we can properly investigate why! Also going to remove the TF version entirely.

Comments:
Hi @Rocketknight1 Could you check, in your PRs (or contributors), if they have [this commit on main](https://github.com/huggingface/transformers/commit/ed95493ce05688447d15d9a82d2d70695290ecff) included? And if they are still flaky after having it included?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37205). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yes, the issue is fixed so this PR isn't necessary!
Add py.typed
Author: cyyever
Pull request: yes

py.typed is a marker file to support typing. See [pep 561](https://peps.python.org/pep-0561/). It's used to indicate that transformer has typing support in the source code and tools such as mypy and pylint can take advantage of that. Therefore, we don't need an additional pyi file for typing.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Added

I'm not sure about this - I think a lot of our library isn't actually fully typed, and type hints are "nice to have" rather than mandatory. Will adding a `py.typed` create lots of errors whenever our types are missing or inaccurate?

@Rocketknight1 No Python project has perfect typing in practice.Errors are generated when the user enables type inference. Without py.typed in this case, errors are triggered for each transformer symbol, and now we can at least provide correct types some symbols and leave ``Any`` for other symbols.
Pytorch and NumPy all use py.typed, however, their typing is clearly not 100% correct.

Seems okay to me in that case, but I'll defer to core maintainers since it affects the entire project, cc @arthurzucker @cyrilvallez @LysandreJik 

Merging lots of your PRs now the CI is green, thanks again @cyyever!
Merge tensor operations with device transfer operations
Author: cyyever
Pull request: yes

This PR merges tensor operations with device transfer operations to avoid additional operations.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

@Rocketknight1 Thank you.

Hi @cyyever, tests seem mostly resolved at this point but there's one actual error - I think the `.to()` in `image_processing_detr` is incorrect.

@Rocketknight1 Wait other test fixes before I can investigate.

@Rocketknight1 Fixed

Tests are passing now, so I'm going to merge. Thanks again for the PR!