noamaanMulla-03/transformers-issues huggingface dataset:
Add Qwen3-Omni model registration to AutoModel and AutoModelForConditionalGeneration classes
Author: samudraneel05
Pull request: no
Labels: name: Feature request

### Feature request

When attempting to load Qwen3-Omni using Auto classes:
```python
from transformers import AutoModel
model = AutoModel.from_pretrained(
    "Qwen/Qwen3-Omni-30B-A3B-Instruct",
    trust_remote_code=True,
)
```

I get an error:
```
ValueError:
Unrecognized configuration class <class 'transformers.models.qwen3_omni_moe.configuration_qwen3_omni_moe.Qwen3OmniMoeConfig'> for this kind of AutoModel: AutoModel.
Model type should be one of .......
```


Optimally, the model should load successfully through Auto classes, similar to other Qwen models. 

### Motivation

I am working towards integrating Qwen3-Omni in the Keras-Hub library

### Your contribution

I am happy to take this up, if anyone else is not interested in this issue.

Comments:
The model has no base model, i.e. a model that outputs raw hidden states before the generative head. We have `Qwen3OmniMoeForConditionalGeneration` model class available with `AutoModelForMultimodalLM`  for generation

Hi, thank you for responding @zucchini-nlp ! I understand.

On a separate note, while going through the codebase files of Qwen3-Omni-Moe, in [configuration_qwen3_omni_moe](https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen3_omni_moe/configuration_qwen3_omni_moe.py) , inside the class `Qwen3OmniMoeTalkerCodePredictorConfig(PreTrainedConfig):`, the `__init__` has `self.sliding_window`defined two times leading to issues when I'm trying to work with it. Line 584 has it as         `self.sliding_window = sliding_window`, and 591 as `self.sliding_window = sliding_window if self.use_sliding_window else None`.

Is this intended behaviour, and I am missing something, or is it a bug and only one instance needs to be there?

Reproduction script:
```
from transformers import AutoConfig

config = AutoConfig.from_pretrained(
    "Qwen/Qwen3-Omni-30B-A3B-Instruct",
    trust_remote_code=True
)
```

Result: `AttributeError: 'Qwen3OmniMoeTalkerCodePredictorConfig' object has no attribute 'use_sliding_window'. Did you mean: 'sliding_window'?`


Supposed to be fixed by https://github.com/huggingface/transformers/pull/43593

Fix was just merged! Closing as resolved

thank you for all the follow ups!
Fix accelerate integration import 
Author: SunMarc
Pull request: yes

# What does this PR do?

We need to move `get_module_size_with_ties` in transformers as the min version of accelerate 1.1.0 don't have this function. It was introduced in 1.2.0 actually . Another solution would be change the minimum version but I prefer not for now. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43732). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix scheduler initialization order
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR put the `lr_scheduler` initialization after the optimizer is prepared to deal with any modification that might happen after. This is mainly to deal with deepspeed as the optimizer is modified and starting with torch 2.10, checks are more strict. 

Fixes https://github.com/huggingface/transformers/pull/43692

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43711). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> checks are stricter with 2.10, but this does not change anything for 2.4?

Nope !
Fix dimensional mismatch in Mistral3MultiModalProjector for multiple vision feature layers
Author: arrdel
Pull request: yes

## What does this PR do?

Fixes #42585 

This PR fixes a dimensional mismatch bug in the Mistral3 multimodal projector when using multiple vision feature layers.

## The Problem

When `vision_feature_layer` is configured as a list (e.g., `[-1, -2]` for multiple layers), the vision features from different layers are concatenated along the last dimension in `Mistral3PreTrainedModel.get_image_features()` (line 156):

```python
hs_pool = [image_outputs.hidden_states[layer_idx] for layer_idx in vision_feature_layer]
selected_image_feature = torch.cat(hs_pool, dim=-1)
```

This results in a tensor with shape `(batch, hidden_size * num_feature_layers)`.

However, `Mistral3PatchMerger` was initialized with only `hidden_size`, not accounting for the concatenated features. This caused:
1. The norm layer to have incorrect dimensions
2. The merging layer to have incorrect input/output dimensions
3. A mismatch between patch_merger output and linear_1 expected input

## The Solution

- Updated `Mistral3PatchMerger` to accept a `num_feature_layers` parameter (defaults to 1 for backward compatibility)
- Adjusted the merging layer dimensions to properly handle concatenated features:
  - Input: `hidden_size * num_feature_layers * spatial_merge_size**2`
  - Output: `hidden_size * num_feature_layers`
- Updated the norm layer in `Mistral3MultiModalProjector` to match the concatenated input dimensions

This ensures the dimensional flow is consistent:
- Input to projector: `hidden_size * num_feature_layers`
- After norm: `hidden_size * num_feature_layers`
- After patch_merger: `hidden_size * num_feature_layers`
- Input to linear_1: `hidden_size * num_feature_layers` ✅

The fix maintains backward compatibility - when `vision_feature_layer` is a single integer, `num_feature_layers = 1` and the behavior is identical to before.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: mistral3

I don't want to make this fix just yet! The reason is that we don't really know if this is a 'real' parameter or not, so we don't know if we should fix the code to use it, or simply eliminate it altogether. Leaving it alone for now, but if anyone from Mistral can clarify that's be great!
Issue loading tokenizer using AutoTokenizer.from_pretrained in v5
Author: msubrayada
Pull request: no
Labels: name: bug

### System Info

There is an issue when loading the tokenizer from the model guillermoruiz/bilmaLAT and others in transformers v5.

when I run the code:
```
bilmaLAT = AutoTokenizer.from_pretrained('guillermoruiz/bilmaLAT')
bilmaLAT('hola')
```

the output is:
`
{'input_ids': [2, 76, 83, 80, 69, 3], 'attention_mask': [1, 1, 1, 1, 1, 1]}
`

When I use Transformers v4.55.0 the output is 
`
{'input_ids': [2, 14518, 3], 'attention_mask': [1, 1, 1]}
`
which is the correct output.

I think the problem is when loading the normalizing functions of the tokenizer.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
import transformers
from transformers import AutoTokenizer

bilmaLAT = AutoTokenizer.from_pretrained('guillermoruiz/bilmaLAT')
bilmaLAT('hola')
```

### Expected behavior

The output should be 

`{'input_ids': [2, 14518, 3], 'attention_mask': [1, 1, 1]}`

Comments:
cc @arthurzucker @itazap

hey @msubrayada ! `bilmaLAT's` [tokenizer_config.json file on the hub](https://huggingface.co/guillermoruiz/bilmaLAT/blob/main/tokenizer_config.json) should be updated to:
 `tokenzier_class: "PreTrainedTokenizerFast"` (instead of `RobertaTokenizer` ) 

since `bilmaLAT's` uses a WordPiece tokenizer and RobertaTokenizer is BPE. This flew under the radar in v4 while in v5 we respect this mapping more strictly! 

Thanks for the clear issue / reproducer BTW 👍 

The problem was solved. Thank you for your response. 🤗
Docs: fix Training step by removing tokenizer from trainer initialization
Author: nesjett
Pull request: yes

# Summary
This PR removes the deprecated tokenizer parameter from the Quicktour documentation examples.

As of the v5.0.0 release, the tokenizer argument was officially replaced from the Trainer constructor. Currently, the documentation example fails with a TypeError, preventing users from completing the introductory tutorial.

# Changes
Location: 
- docs/source/en/quicktour.md
- docs/source/en/tasks/language_modelin.md
- docs/source/en/tasks/masked_language_modeling.md

Action: Replaced tokenizer=tokenizer from the Trainer initialization by processing_class=tokenizer

Reasoning: The tokenizer is officially replaced by processing_class:

TypeError: Trainer.__init__() got an unexpected keyword argument 'tokenizer'

# Related Resources
Breaking Change Reference: [Transformers v5.0.0 Release Notes](https://github.com/huggingface/transformers/releases/tag/v5.0.0)

Affected Doc Page: [Quicktour - Training Step](https://huggingface.co/docs/transformers/quicktour?pipeline-tasks=text+generation)

Checklist
[x] This PR fixes a typo or improves the docs.

cc @stevhliu @SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43733). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Make the CI Green Round 3
Author: Rocketknight1
Pull request: yes

This time I'm focusing on `_get_component_class_from_processor` in the tests. I'm not sure if the way this operates can sometimes result in Tokenizer classes going missing.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43741). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Allow bi-directional attention for all models
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

Allow the `is_causal` kwarg and config attribute to make well-behaved decoder-only models act as encoders

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43705). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update torch minimum version to 2.4
Author: cyyever
Pull request: yes

# What does this PR do?

PyTorch 2.10 has been released. Looking forward to lift the PT minimum version again.

Comments:
I'm not sure we automatically bump the Torch version with new releases like we do with Python! cc @ydshieh @cyrilvallez, is that a policy?

Humm no indeed, we don't have something like "we support last 5 versions". Bumps are usually not correlated with a new version, but for other internal technical reasons

There is a (not-written) convention about  : if a torch version is released 2 years ago, we will drop it.

But torch 2.2 is out Jan 30, 2024, not yet fully 2 years old.

The PyTorch release time is not quite regular. I recall that 2.8.0 had taken half a year. IMO a two year policy is good..

@cyyever PyTorch does a release approximately every quarter now.

Given #41960 this may make sense even though 2.2 isn't at two years yet. There's overlap on this with #42135.

@Rocketknight1 wdyt?

OK for me

Actually, there are other instances of that functions without protection. They should use our internal `is_tracing` instead for security, I'm gonna update them.
We can still bump torch though, I don't mind 

cc @cyyever this got lost (my bad), and now it's time to bump to 2.4 instead of 2.3! Would you be okay with updating the version number here and then I can approve and merge it? If not I can make a separate PR

@Rocketknight1 More updates

Okay, now we just need to fix our own CI :sweat_smile: 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41307). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

well okay one test is failing, hold on :melting_face: 

@Rocketknight1 It is green

@cyyever there was another merge conflict today - if I can get it green again after that I'll merge!

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: albert, bert, clvp, deepseek_v3, depth_anything, diffllama, esm, falcon, gpt_neo, gptj, kyutai_speech_to_text, mimi, moshi, nemotron
Remove more PyTorch < 2.4 code
Author: cyyever
Pull request: yes

# What does this PR do?

This PR removes more old PT code.
## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc @MekkCyber
- kernels: @MekkCyber @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad 

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen2
Modernize CLIP modeling code 
Author: molbap
Pull request: yes

# What does this PR do?

As per title. Adds the nice recent modeling utils to CLIP. Was motivated to be able to use it in the looong-standing #33962 . 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41546). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @zucchini-nlp if you want to take a look!

run-slow: clip, metaclip_2

This comment contains run-slow, running the specified jobs: 

models: ['models/clip', 'models/metaclip_2']
quantizations: [] ...

run-slow: clip, metaclip_2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: clip, metaclip_2

This comment contains run-slow, running the specified jobs: 

models: ['models/clip', 'models/metaclip_2']
quantizations: [] ...

Hi there,

We recently encountered an issue with `transformers` v5.0.0rc1 that seems to be related to the changes in this PR.

Specifically, `CLIPVisionModel.forward()` now returns `hidden_states=None` even when `output_hidden_states=True` is passed. It appears that the `output_hidden_states` argument was removed from `ClipEncoder.forward` (or is not being utilized from `TransformersKwargs` correctly).

**Comparison:**
- **v5:** [modeling_clip.py#L485](https://github.com/huggingface/transformers/blob/08810b1e278938278c50153ee1edfd7a20a759da/src/transformers/models/clip/modeling_clip.py#L485)
- **v4:** [modeling_clip.py#L506](https://github.com/huggingface/transformers/blob/753d61104116eefc8ffc977327b441ee0c8d599f/src/transformers/models/clip/modeling_clip.py#L506)

We suspect this is a regression. Could you please check if this was intended?

Related
- https://github.com/huggingface/transformers/issues/43761
- https://github.com/linkedin/Liger-Kernel/pull/1061

Thanks for the report, having a look and yes it should work with `check_model_inputs`
Fix T5 v1.1 detection
Author: githubnemo
Pull request: yes

PR #41541 refactored `tie_word_embeddings` handling (among other things) which subtly broke detection of T5 v1.1 vs. original detection. As a consequence, decoder output scaling was always applied, regardless of T5 version.

This is resolved by using the correct value for `tie_word_embeddings`.

**Testing:**

This was not covered by the tests since the tests instantiate the config once and modify attributes on the config. This is problematic since all the decision logic is happening in `T5Config.__init__`. This was addressed by having a specific `get_config_v1_1` method that initializes the config as if it were coming from a v1.1 model (e.g., flan-t5).



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43681). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: mt5, t5
Fix pe uts
Author: YangKai0616
Pull request: yes

# What does this PR do?

According to PR https://github.com/huggingface/transformers/pull/43030, update test files `tests/models/pe_audio/test_modeling_pe_audio.py` and `tests/models/pe_video/test_modeling_pe_video.py`.

The changes to `src/transformers/models/pe_audio/modeling_pe_audio.py` were made because of the following error:
```
pytest -rA tests/models/pe_audio/test_modeling_pe_audio.py::PeAudioModelTest::test_model_parallelism

FAILED tests/models/pe_audio/test_modeling_pe_audio.py::PeAudioModelTest::test_model_parallelism - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!
```

@vasqu , @ydshieh please help review this PR. Thans!


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: pe_audio, pe_video
Fix Trainer resume_from_checkpoint to apply checkpoint conversion mappings
Author: antznette1
Pull request: yes
Labels: name: Code agent slop

# What does this PR do?

Fixes #43701

When resuming training via [Trainer.train(resume_from_checkpoint=...)](cci:1://file:///c:/Users/brass/OneDrive/Desktop/Work/clients/Anthonette/transformers/src/transformers/trainer.py:2070:4-2175:13), the checkpoint loading path now applies registered checkpoint conversion mappings (e.g. legacy key renames) instead of calling [model.load_state_dict()](cci:1://file:///c:/Users/brass/OneDrive/Desktop/Work/clients/Anthonette/transformers/src/transformers/modeling_utils.py:292:0-323:106) directly on the raw checkpoint keys.

This aligns the resume-from-checkpoint behavior with the regular model loading behavior and prevents key mismatch issues for checkpoints saved with legacy/converted parameter names.

## Motivation / context

Some models/checkpoints require key renaming through conversion mappings (including legacy mappings). Previously, the [Trainer](cci:2://file:///c:/Users/brass/OneDrive/Desktop/Work/clients/Anthonette/transformers/src/transformers/trainer.py:277:0-5348:9) resume path bypassed that conversion logic, which could lead to missing/unexpected keys and failed/incomplete restores.

## Changes

- Update [Trainer._load_from_checkpoint](cci:1://file:///c:/Users/brass/OneDrive/Desktop/Work/clients/Anthonette/transformers/src/transformers/trainer.py:2718:4-2877:60) to load via [convert_and_load_state_dict_in_model()](cci:1://file:///c:/Users/brass/OneDrive/Desktop/Work/clients/Anthonette/transformers/src/transformers/core_model_loading.py:966:0-1221:43) with a [LoadStateDictConfig](cci:2://file:///c:/Users/brass/OneDrive/Desktop/Work/clients/Anthonette/transformers/src/transformers/modeling_utils.py:162:0-186:44) when a conversion mapping exists.
- Add a regression test ensuring resume applies the conversion mapping:
  [test_resume_from_checkpoint_applies_checkpoint_conversion_mapping](cci:1://file:///c:/Users/brass/OneDrive/Desktop/Work/clients/Anthonette/transformers/tests/trainer/test_trainer.py:185:0-217:89).

## Tests

- `python -m pytest -k resume_from_checkpoint_applies_checkpoint_conversion_mapping tests/trainer/test_trainer.py`



Comments:
Fixed Models with PEFT adapters won't load from local checkpoints issue
Author: Nitin75408
Pull request: yes
Labels: name: Code agent slop

# What does this PR do?

Fixes issue #43746: when loading PEFT adapters from local directories, the code no longer overrides the local path with the hub path from the adapter config, preventing unnecessary hub downloads. Centralizes path resolution in resolve_peft_base_model_path() so AutoModel.from_pretrained(), pipeline(), and PEFT utilities behave consistently. Adds offline mode support: if local_files_only=True and a hub download would be required, it raises a clear error instead of silently downloading. Includes tests for local path preservation, offline mode, and pipeline usage.

<!-- Remove if not applicable -->

Fixes #43746 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc @MekkCyber
- kernels: @MekkCyber @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad 

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto