noamaanMulla-03/transformers-issues huggingface dataset:
🚨 [v5] Toggle the serialization format in processors
Author: zucchini-nlp
Pull request: yes
Labels: name: for_v5?

# What does this PR do?

As per title, in v5 we will start saving in the new nested format. The loading is still supported though for old-format configs

Also, we don't need an `optional_attributes` field that does not do what it is called. Since it is only `chat template`, let's just manually pop and set it. For audio tokenizers, it'd be nice to refactor a bit, I think we could try to move it in `self.attributes`


Fixes https://github.com/huggingface/transformers/issues/40447 by deleting ambiguously named `optional_attributes`


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41474). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Done, completely removed all legacy options for saving and in tests. CI is failing for unrelated reasons

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: align, auto, chinese_clip, clipseg, flava, granite_speech, layoutlmv2, layoutlmv3, markuplm, mgp_str, oneformer, owlvit, speech_to_text, speecht5, vision_text_dual_encoder, wav2vec2
🚨 [v5] Rename left traces of `past_key_value` in BERT-like models
Author: zucchini-nlp
Pull request: yes
Labels: name: for_v5?

# What does this PR do?

As per title, finalizes the work from https://github.com/huggingface/transformers/pull/41425 and renames `past_key_value` as `past_key_values` in all files without deprecation cycle

It was a find and replace, so tests might fail 😅  

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: autoformer, bart, bert, bert_generation, bigbird_pegasus, biogpt, blenderbot, blenderbot_small, blip, bridgetower, camembert, data2vec, decision_transformer, electra, ernie, flaubert

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41448). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Ran slow tests on Bert, BART, Ernie, GPT2, T5 and MusicGen comparing to main branch, all green ✅ 

Not sure how they got back here because I had definitely made the switch in https://github.com/huggingface/transformers/pull/39956 already... We need to be very careful when touching a lot of files otherwise we shoot ourselves in the foot!

Yep, might be simply issues with rebasing main. Happens a lot when everyone is making huge PRs 😄 

Yup, sorry that's likely on me with the vLLM refactor :cry: wasn't aware of that rename
🚨 [v5] Rendundant code in nested configs
Author: zucchini-nlp
Pull request: yes
Labels: name: for_v5?

# What does this PR do?

This PR deletes `from_xxx_config` and similar methods from everywhere and instead ensures that the config init can accept subconfigs of all formats (dict or object)

Along the way, I found that not all configs call super init in the end and thus they can't set recursively attn implementation on subconfigs. I added a test for it and fixed all cases

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41314). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[v5] Delete left traces of feature extractor
Author: zucchini-nlp
Pull request: yes
Labels: name: for_v5?

# What does this PR do?

After rebasing https://github.com/huggingface/transformers/pull/40793 and getting new changes on `TypedDict`, I found out that some traces from feature extractor were still left. Thanks for type validation we can catch inconsistencies easily :)

This PR goes over the deprecated properties as well and deletes them. Also it deletes `current_processor` which was also deprecated for v5 

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: blip, blip_2, chinese_clip, clip, clipseg, donut, flava, fuyu, git, idefics, layoutlmv2, layoutlmv3, layoutxlm, mgp_str, musicgen, nougat

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41321). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
🚨 [v5] Delete feature extractors used for vision
Author: zucchini-nlp
Pull request: yes
Labels: name: for_v5?

# What does this PR do?

As per title, let's clean up for v5


These were supposed to be deleted anyway and we had the warning logged for a long time, even before I joined. Feature Extractors are now reserved for audio models only

Comments:
Nice, IIRC there were a few utils in processing handling that involved using the `feature_extractor` keyword as well

Yeah, the tests are failing, I will do one more round of cleaning

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto

Done

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41174). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Reverse conversion
Author: ArthurZucker
Pull request: yes

# What does this PR do?
@Cyrilvallez we might want to change the loading as well to always prepare a dictionary, seems to flow more naturally! 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42243). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Refactor `tokenization` in `transformers`
Author: ArthurZucker
Pull request: yes
Labels: name: for_v5?

# What does this PR do?
The goal of this PR is to get rid of the old and biased "slow vs fast" definition, simplify the code, unbloat the API and have something simple and stable. 

We don't want to break stuff, so will try not to.

Related to #40938 

Comments:
I'm waiting for this feature to be merged so much!
draft removal of special and added
Author: ArthurZucker
Pull request: yes
Labels: name: for_v5?

# What does this PR do?
For v5! Long due cleanup

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40821). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: biogpt
add pin memory and block table
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Update CB to support block table

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39130). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This is on pause as the performance gains were not really noticeable as of now
Many small fixes for the CI
Author: remi-or
Pull request: yes

This PR fixes various test fails that make the CI redder than it needs:
- the test to export a model is red on the CI because `is_mlx_available` is not traceable, to avoid this we use the same approach that was used w/ `_is_torch_available`
- skips the `test_sdpa_can_dispatch_on_flash` test for `blt` and `deepseek_vl_hybrid` because it always uses an attention mask
- change the repo used by `dbrx` in the `test_model_from_pretrained` test which had an invalid config
- fix a shape error in `mvp` and `led`, test is still partially fixed but still fails because of a tied weight issues. I will work on a PR to fix the test once #42362 is merged
- fixed `update_config_headdim` which is called in FA test and failed when `num_attention_heads` is not an int but a list of int. Explained the logic in a comment, but we chose the head_dim in a way that ensures the new headdim is as large as possible (which is good for triton)

This PR touches a few models but always in small ways, so I grouped the changes. Not much to review in the end! 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42364). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: blt, dbrx, deepseek_vl_hybrid, led, mvp
[core] fix mxfp4
Author: MekkCyber
Pull request: yes

# What does this PR do?

This pr fixes MxFP4 quantization

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42382). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: mxfp4
Remove `rope_theta` attribute map for `ModernBert`
Author: hmellor
Pull request: yes

This attribute being present via `attribute_map` causes problems for vLLM's backward compatibility.

We have the following code snippet for backward compatibility with custom models which were written for Transformers v4 

https://github.com/vllm-project/vllm/blob/d9d342d214b8c13f71215318a6d9252cc4a5ca47/vllm/transformers_utils/config.py#L466-L472

```py
        # When Transformers v5 is installed, legacy rope_theta may be present
        # when using custom code models written for Transformers v4
        if (rope_theta := getattr(config, "rope_theta", None)) is not None:
            standardize_rope_params(config, rope_theta=rope_theta)
            rope_config_validation(config)
            # Delete rope_theta to avoid confusion in downstream code
            del config.rope_theta
```

The attribute map makes `rope_theta` readable but not deletable, which causes this block to raise an error. 

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: modernbert

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42421). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Actually, I'll fix this on the vLLM side
Initial migration guide
Author: LysandreJik
Pull request: yes

WIP (related to https://github.com/huggingface/transformers/issues/40822)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42386). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Let's merge it and iterate via PRs; it's easier to contribute to it while opening PRs if it is merged.
🚨 Image-text pipeline expects correctly formatted chat
Author: zucchini-nlp
Pull request: yes
Labels: name: for_v5?

# What does this PR do?

As per title, we can break for v5 and enforce a correct usage of chat templates. I will request review after adjusting tests

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42359). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @ebezzam as discussed internally. I didn't move the `Chat` anywhere in this PR, ig it's already a part of the PR for `text-to-audio`

Big fan of this cleanup!
SAM3 point mode is not supported yet?
Author: haofanwang
Pull request: no

In [SAM3 official example](https://github.com/facebookresearch/sam3/blob/main/examples/sam3_for_sam1_task_example.ipynb
), they also support point mode. But it seems that transforms has not supported yet?


Comments:
Hello @haofanwang! we do support point mode, but you'll have to use [Sam3TrackerModel](https://huggingface.co/docs/transformers/main/en/model_doc/sam3_tracker) (as defined in the [sam3 paper](https://ai.meta.com/research/publications/sam-3-segment-anything-with-concepts/)) to use point and boxes prompts in the same way they are used for SAM2 and SAM.
We decided to split the models that do PCS - Promptable Concept Segmentation (Sam3Model and Sam3VideoModel) and PVS - Promptable Visual Segmentation (Sam3TrackerModel and Sam3TrackerVideoModel), as architecturally they are completely separate models, sharing only a vision backbone.
🚨 Remove generic output_attentions warning
Author: Aravind-11
Pull request: yes

# What does this PR do?

Removed broad warning for output_attentions in generic.py. Specific backend warnings remain in their respective modules.

Fixes #41924


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you write any new necessary tests?


## Who can review?

@vasqu 

Comments:
Sorry for the delays, merging now!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42334). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix argument ordering bug in spawn_tp_materialize
Author: 3outeille
Pull request: yes

with new `weight loading`, there was wrong arg ordering which cause `RouterParallel` to not be called when doing `tp_plan=auto`

```python
def spawn_tp_materialize(thread_pool, tensor, sharding_method, tensor_idx, dtype=None) -> Future:
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42425). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Bart config doesn't need generation parameters
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, fixes slow tests for BART and MVP


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42337). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

slow bart test won't pass anyway due to weight tying, so skipping it for now 

run-slow: bart

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bart, mvp

This comment contains `run-slow`, running the specified jobs: 

models: ["models/bart"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19710024461)

✅ No failing test specific to this PR 🎉 !
Simplify and standardize processor tests
Author: yonigozlan
Pull request: yes
Labels: name: for_v5?

# What does this PR do?

Improve ProcessorTestMixin to standardize processor tests, especially the setup part.
Requires https://github.com/huggingface/transformers/pull/41633

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41773). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@Cyrilvallez This should be ready to merge!
@ydshieh The failing tests here are also failing on main:
`test_overflowing_tokens` is failing for `test_processing_udop`, `test_processing_layoutlmv2` and `test_processing_layoutxlm`. It looks like a missing repo on the hub?

Hi @yonigozlan 

For `LayoutLMv2ProcessorTest`, after the merged PR

[v5] 🚨Refactor subprocessors handling in processors (#41633)
the error is

commit: f065e402
```
FAILED tests/models/layoutlmv2/test_processing_layoutlmv2.py::LayoutLMv2ProcessorTest::test_overflowing_tokens - OSError: Unable to load vocabulary from file. Please check that the provided vocabulary is accessible and not corrupted.
```

Prior, it was 

commit: 91d250ef
```
FAILED tests/models/layoutlmv2/test_processing_layoutlmv2.py::LayoutLMv2ProcessorTest::test_overflowing_tokens - ValueError: Words must be of type `List[str]` (single pretokenized example), or `List[List[str]]` (batch of pretokenized examples).
```

Could you check this?

I will check what I can do once the error being back to the same error. They are failing for quite some time 😢 

@zucchini-nlp @Cyrilvallez Thanks for the reviews! Waiting on your approval to merge now :)

One thing to note, I had to add a config for layoutxlm to have all tests pass. This hides a deeper issues, basically we have a few models in the library that don't have a config (namely code_llama, gpt-sw3, granitevision and nougat), but this messes up completely the "SUBPROCESSOR"_MAPPING that are built as `_LazyAutoMapping` objects in `auto_factory.py`. `_LazyAutoMapping` defines a `_reverse_config_mapping` dict, but if a config is used for several models in `CONFIG_MAPPING_NAMES` in `configuration_auto.py`, all but one of the mapping will be overwritten.

Imo the solution would be to always have a config when adding a model, especially now with modular, no reason not to have that.

I'll open a PR to enforce that, unless you think there is another way to handle the issue.


**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, edgetam, gemma3n, glm46v, glm4v, glm4v_moe, layoutlmv2, layoutxlm, llava_next, llava_onevision, mgp_str, owlvit, sam, sam2, sam2_video, sam3
Clean bnb integration using weight converter
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR cleans bnb integration, we don't need to deal with missing keys.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42426). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@bot /style

Style fix is beginning .... [View the workflow run here](https://github.com/huggingface/transformers/actions/runs/19711168298).

@bot /style 

Style bot fixed some files and pushed the changes.

run-slow: bnb 

This comment contains `run-slow`, running the specified jobs: 

models: []
quantizations: ["quantization/bnb"]

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19711890964)

✅ No failing test specific to this PR 🎉 !
Sam3 release on v4.57.3
Author: yonigozlan
Pull request: yes

# What does this PR do?

This PR is open for anyone wanting to use SAM3 on a stable version of transformers and not the main branch that can be unstable as we are gearing up for v5.

Use this command to install with pip
```
pip install git+https://github.com/huggingface/transformers.git@0cece43fb2e0400ee856b8cefb1749f6da1cbfec
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42403). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, edgetam, edgetam_video, nougat, sam2, sam2_video, sam3
[torchao] fix safetensors and enable loading from sharded files 
Author: liangel-02
Pull request: yes

**Context**

This PR is a followup to #40735 and #41138. Previously, we enabled safetensors in torchao for one shard file. This PR fixes some errors introduced in #41138 and handles the case when checkpoints are sharded onto more than one file, including the edge case where a single quantized tensor (ie `Float8Tensor`) is sharded onto two different files (ie `qdata` on one and `scale` on another). 

**Summary**

If we are loading in a component of a tensor subclass in `create_quantized_param()` called by `_load_state_dict_into_meta_model()`, we add this as a new parameter into the model. Then after all parameters are loaded, we unflatten the `state_dict` and reassign the model parameters. 

**Testing**

Modified unit tests to test all tensor subclasses 
`python tests/quantization/torchao_integration/test_torchao.py -k TorchAoSafeSerializationTest`


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: torchao_integration

@SunMarc i rebased my pr and now am seeing this error due to [#41580](https://github.com/huggingface/transformers/pull/41580)

```  
File "/home/liangel/local/transformers/src/transformers/modeling_utils.py", line 4122, in from_pretrained
    model, missing_keys, unexpected_keys, mismatched_keys, offload_index, error_msgs = cls._load_pretrained_model(
                                                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/liangel/local/transformers/src/transformers/modeling_utils.py", line 4275, in _load_pretrained_model
    missing_keys, unexpected_keys, mismatched_keys, misc = convert_and_load_state_dict_in_model(
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/liangel/local/transformers/src/transformers/core_model_loading.py", line 621, in convert_and_load_state_dict_in_model
    raise ValueError("This quantization method is gonna be supported SOOOON")
ValueError: This quantization method is gonna be supported SOOOON
```

will there be follow up changes to support torchao/what changes would be needed? cc @jerryzh168 


Happy to help if you want to do the changes here, I think @SunMarc and @MekkCyber are gonna be helping as well making sure torchao is supported! 

We just merged a big [PR](https://github.com/huggingface/transformers/pull/41580) and all quantization methods are impacted, we will add back the support for those methods asap ! 

closing due to #41580 #42289
Any to any pipeline and auto-mapping
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Adds any-to-any as a pipeline and in auto classes so that we can have a single mapping for all multimodal models. The model mapping is almost same as image-text-to-text, with inclusion of audio-LLM and omni-LLM. I hope I added all audio models, but lmk if anything is missing from recent ones

Fixes https://github.com/huggingface/transformers/issues/40302 and fixes https://github.com/huggingface/transformers/issues/37794

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40884). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@merveenoyan if you have time to look on the docs section, your advice will be appreciated. Do you think there is anything we should add or highlight? I added basic functionality with examples for now

Oke, I think this one is ready now, as long as CI turns green

Thank you! Looking forward to this getting merged 🙏🏻 

Closes https://github.com/huggingface/transformers/issues/40302

@merveenoyan @ArthurZucker friendly bump on PR review!

Test failures not related!

Test failures not related, kind ping @ArthurZucker whenever you have time

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, align, altclip, aria, audioflamingo3, auto, autoformer, aya_vision, bark, beit, bit, blip, blip_2, blt, bridgetower, chameleon
Add audio-text-to-text task
Author: jackzhxng
Pull request: no
Labels: name: Feature request

### Feature request

Add `audio-text-to-text`, for models like Voxtral. https://github.com/huggingface/transformers/blob/main/utils/update_metadata.py#L71, or something like `multimodal-text-generation` works too.

@Rocketknight1 

### Motivation

Needed to enable Voxtral on Optimum ET cleanly

### Your contribution

N/A - I'd contribute but I'm not sure where the touch points are to add a task like this or if it's even okay to add a new task

Comments:
IMO `multimodal-text-generation` would be better in long term and it could support omni-modality models as well

Yeah, I would prefer `multimodal-text-generation` as well actually

Let me assign it to myself and start working on it, has been long pending in my TODO to support auto-mapping-for-multimodal

Hi @zucchini-nlp thanks for taking this up, would be super useful in Optimum ET to add this.

Also, ideally there's some sort of task that is able to capture all of the `<model>ForConditionalGeneration`s out there: Gemma3ForConditionalGeneration, VoxtralForConditionalGeneration, LlavaForConditionalGeneration etc.