noamaanMulla-03/transformers-issues huggingface dataset:
[WIP] Refactor attention modules in Bert-based models to use global attention functions
Author: Marcel256
Pull request: yes

# What does this PR do?
Refactoring of the Attention Modules in bert-based models to use global attention functions. 
Allows support of SDPA and flash attention for Bert models and potentially flex attention in the future

This is a first draft (Feedback appreciated)
Could solve:
#37105 
#29129
#27957

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
[quantization] Dequant fp8 when cuda or xpu not available
Author: MekkCyber
Pull request: yes
Labels: name: Quantization

# What does this PR do?

Instead of erroring when no cuda or xpu device is available, we dequantize the weights to make it possible to do inference on cpu

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42511). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: finegrained_fp8
Attempt to fix VLM gradient enabling
Author: molbap
Pull request: yes

# What does this PR do?

As per title. Linked to https://github.com/huggingface/peft/issues/2880.
Follows more or less closely the already existing implementations for idefics2-3 and smolvlm, trying to cover several types of VLMs (they are named differently across the lib.)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41993). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks, yes it's a far less brittle option. There's a few (really a few and hopefully should be 0 after v5) modules that were just `nn.Module`s instead of `PreTrainedModel` so they would be off the hook, other than these few exceptions should work out well, will push something like that today

> So I'm thinking that we could potentially make it works ootx for all MLLMS (audio/vision/omni) by checking for each `PreTrainedModel` within the model and then setting grads on that models' inputs (`model.get_input_embeddings()`)
> 
> We use similar trick when setting attention implementations and check for `PreTrainedModel`'s, so it could be a good option. WDYT?

Sorry I may misunderstand the proposed solution but this doesn't seem to solve the problem? In a VLM where I target a module in the vision stack I need to have the vision model's inputs require grads, not the language model's input (`get_input_embeddings`).

@githubnemo `model.get_input_embeddings()` will return the lowest module for each models, which is the same thing current PR does by recursing over modules. So setting grads on inputs of `model.get_input_embeddings()` will enable it not only for for vision-modules, but also for audio and other modalities

BTW, when working on smth else noticed that we have code like below which can be deleted after this PR

https://github.com/huggingface/transformers/blob/80134e6e663db962c30d0e45c54265b3141bc7c7/src/transformers/models/idefics2/modeling_idefics2.py#L1026-L1030

Yes, for all idefics/smolvlm there'll be no need for that. Should ship that today (finally)

Iterated a bit on that and hit a dead end on idefics2/3 code, back at it tomorrow!

Tests failing appear unrelated (I rebased on main), what do you think of the new method @zucchini-nlp ? Also @githubnemo I updated the test a tad, let me know

run-slow: bart, blip_2, idefics2, idefics3, qwen2_5_omni, qwen2_5_vl, qwen2_vl, qwen3_omni_moe, smolvlm, timm_wrapper

This comment contains `run-slow`, running the specified jobs: 

models: ["models/bart", "models/blip_2", "models/idefics2", "models/idefics3", "models/qwen2_5_omni", "models/qwen2_5_vl", "models/qwen2_vl", "models/qwen3_omni_moe", "models/smolvlm", "models/timm_wrapper"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19740755485)

✅ No failing test specific to this PR 🎉 !


**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bart, blip_2, idefics2, idefics3, qwen2_5_omni, qwen2_5_vl, qwen2_vl, qwen3_omni_moe, smolvlm, timm_wrapper
skip `MLukeTokenizerTest` temporarily
Author: ydshieh
Pull request: yes

# What does this PR do?

As it fails at this moment. Skip for now but let's try to bring it back as soon as we could .

 https://app.circleci.com/pipelines/github/huggingface/transformers/155465/workflows/2c0019c5-4390-47c0-92a6-96aa6b30ce56/jobs/2044080

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: mluke

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42520). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
another way of fix for #42400
Author: ydshieh
Pull request: yes

# What does this PR do?

This was originally #40617, which IMO a better approach than #42400 (despite not ideal neither).

With current form, the failure ratio is 0.1% instead of 1%. We can add `@is_flaky(max_attempts=2)` as in #40617 however.

Comments:
@Rocketknight1 if you are happy with this PR, I will apply the same changes to other places.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42509). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This works too! I think we should add `is_flaky` and **also** `set_seed` though. All of those will reduce the chance of rare flakiness further, so we might as well do them all together!

I will apply the changes, but refrain myself of adding `set_seed`. I understand it works, but this is not an integration test that has use randomness to perform generation (or forward) then compare against some predefined expected outputs.

Setting seed for such non-integration tests isn't really a good approach (and just playing some lottery).

Unfortunately, I don't have bandwidth for now to see if there is more intrinsic reason behind the failures (that is related to randomness of weights I agree).

However, with the changes in this PR + is_flaky, it works well (last PR: no failures in 3000 runs)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: ernie4_5_moe, minimax, mixtral, qwen2_moe, qwen3_moe
feat(cb): use context manager in `generate_batch`
Author: McPatate
Pull request: yes


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42190). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Diff is weird, I think you need to rebase. Then I can approve!


> Diff is weird, I think you need to rebase. Then I can approve!

quizz time: what command did I use to rebase without any conflict?
avoid `afmoe` job crashing
Author: ydshieh
Pull request: yes

# What does this PR do?

Avoid `afmoe` job crashing like

https://app.circleci.com/pipelines/github/huggingface/transformers/155470/workflows/1bff2732-c69b-4d34-b2ab-ffc45a80b559/jobs/2044143

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: afmoe

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42524). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix tied_weights for vlms
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

Minimum fix from https://github.com/huggingface/transformers/pull/42385

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: kyutai_speech_to_text, llava_next_video, llava_onevision

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42523). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix Qwen3OmniMoE weight init
Author: Rocketknight1
Pull request: yes

In #42456 we renamed `self.router` to `self.gate` in some layers but the `_init_weights` for `Qwen3OmniMoE` still refers to router. This PR fixes it!

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen3_omni_moe

today is not a smart day for me

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42531). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[TP plans] Fix some incorrects TP plans
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. See [here](https://github.com/huggingface/transformers/pull/41634#discussion_r2565560762) for the source

TODO CYRIL: models modified only apply norm on head_dim!! So should be fine without replication. But then nano_chat needs to remove useless replication, and let's check other models with norms to be sure

cc @vasqu 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42448). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: apertus, doge, nanochat

Only a bit weird one is `Cohere`, it has this `use_qk_norm` attribute but it defaults to false and the base models also doesn't use it. If it were to use it, we would need to rep, otherwise not. See https://github.com/huggingface/transformers/blob/352a2e0c120370f7db3b40caa4c60f0efca4e737/src/transformers/models/cohere/modular_cohere.py#L139-L146

Not sure if we just rep either way but left it for now

Thanks a lot for double checking @vasqu! For cohere, if the main checkpoints do not use the norm, let's not replicate as it's a huge performance drawback! 
Cannot approve my own PR, so just gonna merge it!
[Ministral 3] Add ministral 3
Author: patrickvonplaten
Pull request: yes
Labels: name: New model

This PR adds ministral 3

Comments:
Ministral when?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42498). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, ministral3
Fix empty tensor shape issue in DynamicCache for torch.compile
Author: yashwantbezawada
Pull request: yes

# What does this PR do?

Fixes #42027

This PR fixes a regression where torch.cat receives incorrectly shaped empty tensors during GPT2 model tracing with torch.compile, causing compilation failures.

## Background

The issue was introduced in commit dc11a3cbb2c (PR #39797) where empty cache tensors were initialized as 1D tensors with shape [0] using torch.tensor([]). When these are concatenated with 4D key/value tensors [batch_size, num_heads, seq_len, head_dim] along dim=-2, torch.compiles tracing fails with empty tensor errors.

## Changes

### Modified DynamicLayer.lazy_initialization()
- Changed from: torch.tensor([], dtype=..., device=...)  → shape [0] (1D)
- Changed to: torch.zeros((batch_size, num_heads, 0, head_dim), dtype=..., device=...) → shape [batch, heads, 0, dim] (4D)

### Modified QuantizedLayer.update()
- Applied same fix when resetting cache after quantization
- Ensures empty tensors have proper 4D shape matching key_states dimensions

## Testing

The fix ensures:
- torch.cat([empty_4d_tensor, key_states], dim=-2) works correctly
- Compatible with torch.compile tracing
- Maintains backward compatibility with eager mode
- Works for both DynamicLayer and QuantizedLayer caches

## Impact

- Fixes regression from v4.52.4 to v4.57.1
- Affects models using DynamicCache with torch.compile (GPT2, and others)
- No breaking changes to API or behavior

Comments:
I see that the CI tests are failing (tests_exotic_models, tests_generate, tests_torch), while code quality checks pass. I'm unable to access the detailed CircleCI logs to understand the specific test failures. 

The changes I made:
- Changed empty tensor initialization from `torch.tensor([])` (1D) to `torch.zeros((batch_size, num_heads, 0, head_dim))` (4D with 0 seq_len)
- Applied this to both `DynamicLayer.lazy_initialization` and `QuantizedLayer.update`

This approach ensures torch.cat works correctly in torch.compile mode by providing properly shaped 4D tensors.

Could someone help me understand what tests are failing and why? I'd be happy to adjust the approach if needed. I'm aware of PR #40328 which takes a more comprehensive approach to torch.compile + DynamicCache compatibility.

cc @huggingface/transformers

This is an update to a PR from @cyrilvallez, so I'll wait for him to approve it! 

How are you using torch.compile? In general, it should not really be used with DynamicCache, as the shapes will keep on changing each iteration

The original issue was about using a custom torch.compile backend with `aot_export_joint_simple` for tracing - not really standard generation with DynamicCache.

I noticed PR #40328 is doing the proper fix with symbolic shapes and mark_dynamic. That seems like the right long-term solution. Should I just close this one?

Hey @yashwantbezawada! Indeed the PR you linked will provide much better support for compile options! I'll go back into it asap when things will be calmer after the `v5` release! Feel free to close in the meantime!

Thanks for the update! Closing this one - looking forward to #40328.
`tokenizers==0.22.2-rc0`
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Attempt

Comments:
Cool nothing broke, closing the PR as its a minor release, does not need to force update!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42553). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Quantization] fix dequant when block size is none & static quantization
Author: MekkCyber
Pull request: yes

# What does this PR do?

This pr fixes the dequantization when no block size is provided by assuming it's a per tensor quantization. 
It also fixes the input scales not being expected in the dequantization case.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42545). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: finegrained_fp8
use dict everywhere for core model loading
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Make sure we use dict not list should make everyone's life easier

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42392). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

On it
Try to refactor api stuffs
Author: ArthurZucker
Pull request: yes

# What does this PR do?
As we discussed offline, trying to refactor the expectations: 
```python 
from transformers import LlamaTokenizer 
LlamaTokenizer()
```
-> gives an empty skeleton of a llama tokenizer
```
LlamaTokenizer.from_pretrained()
```
will convert any files online to `vocab` and `merges` while using `Llama` 's skeleton. 

```
from transformers import TokenizersBackend
TokenizersBackend(tokenizer.json)
 TokenizersBackend.from_pretrained()
```
`TokenizersBackend` will convert to `tokenizer.json` when possible and will follow what's online. (enforce what was serialized basically)

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: albert, auto, barthez, bartpho, bert, biogpt, code_llama, deberta_v2, layoutxlm, llama, markuplm, openai, pegasus, t5, xlm
dots1 expert bias routing still broken in v5.0.0rc0
Author: fjosw
Pull request: no
Labels: name: bug

### System Info

N/A

### Who can help?

I just had a look at v5.0.0rc0 and saw that expert bias routing still seems to be broken for the dots1 model as a consequence of the refactoring in https://github.com/huggingface/transformers/pull/40132.

I provided a potential fix in https://github.com/huggingface/transformers/pull/41663 @ArthurZucker @Cyrilvallez 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

N/A

### Expected behavior

N/A

Comments:
We'll fix that! cc @vasqu if you have time or me hehe!

Taking a look!

I'm closing this since #41663 provides a proper fix. Thx a lot (and sorry for all the delays)! We potentially might move the function elsewhere (within the router) but I'll leave that to a separate PR.
[Fix] dots1 expert bias routing
Author: fjosw
Pull request: yes

# What does this PR do?

This is a follow up to https://github.com/huggingface/transformers/pull/41647. I went over the model code again and noticed that the dots1 routing apparently also broke in https://github.com/huggingface/transformers/pull/40132. dots1 seems to be very close in architecture to Deepseek V3 but not explcitly dependent. Not sure if there is a more elegant way to fix this?

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc @MekkCyber
- kernels: @MekkCyber @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad 

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


Comments:
Reminder: I think the dots1 model is still broken on main @ArthurZucker, not sure if this PR is the best way to address this, though.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: dots1

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41663). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add missing truncate field for yarn rope
Author: SunMarc
Pull request: yes

# What does this PR do?
This PR adds "truncate" field in optional_keys for yarn rope. This is actually used and right now we get a warning them loading gpt-oss model. 
```
Unrecognized keys in `rope_parameters` for 'rope_type'='yarn': {'truncate'}
```


Comments:
@bot /style

Style bot fixed some files and pushed the changes.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42189). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
make `num_items_in_batch` optional in compute_loss_func
Author: SunMarc
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/36331

This PR fixes the breaking change that we have with custom loss function as we introduced `num_items_in_batch`. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36426). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I think the `num_items_in_batch` argument should be mandatory because leaving it as `None` when `compute_loss_func` is set could trigger the GA loss bug. A better way to fix [issue #36331](https://github.com/huggingface/transformers/issues/36331) might be to add more specific documentation for `compute_loss_func`.
Add spinquant quantization
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR adds spinquant quantization method compatible with spinquant model from meta. 

TODO:
- Fix kernels issues

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36141). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi there,
May I know if SpinQuant will be included as one quantization method for quantization_config? Thanks!

Hopefully ! I will come back to this PR once I'm done with other projects !
Add fp8 llama script
Author: SunMarc
Pull request: yes

# What does this PR do ?

This PR adds the script that was used to quantize the 405B llama in FP8. 

I'm not sure where to put it for now 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34884). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.