noamaanMulla-03/transformers-issues huggingface dataset:
[`FA`] Generalize fa config checks and fix flags
Author: vasqu
Pull request: yes

As per title, oftentimes we have somethin along `config._attn_implementation == "flash_attention_2"`. This is unideal and very faulty as other FA flavors will be ignored, e.g. kernels and FA3. This simply changes this to the generalized version everywhere. 
Additionally, fixed a few wrong flags that still used `_supports_flash_attn_2` (including within tests), relevant commit a231a2be7fa81445085ecb50371606aeb7a113a7 

Note that sam2 video, sam3 tracker video, edgetam video do not have any FA tests but they only use a mask with target guided attention which is not used per default, so we add a fallback similar to sam3 to use FA for other attention blocks.

Note that I changed pixtral slightly more due to it doing unnecessary movements at each attention layer. Not sure why the mask was properly done but not the pos ids.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43121). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: pixtral

This comment contains `run-slow`, running the specified jobs: 

models: ["models/pixtral"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/20759163961)

✅ No failing test specific to this PR 🎉 !


Sounds to me like it's quite important to have a unique entry-point for these cases, as otherwise we have the risk to have very surprising bugs/issues in just a few models because they simply miss some checks

Yes let me change the check to a separate fn, will probably go over more models then (which already do the proper `"flash" in ...` check)

Moved it to `utils.generic` (import utils doesn't fit IMO), but added this fn to all models now. Should make this way more maintainable

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: afmoe, altclip, auto, autoformer, bamba, bark, bloom, clipseg, codegen, deepseek_v2, deepseek_v3, edgetam, edgetam_video, ernie4_5_vl_moe, falcon, falcon_h1

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43121&sha=6c5129
[LWDETR] update model repos from stevenbucaille to AnnaZhang
Author: sbucaille
Pull request: yes

# What does this PR do?

LWDetr got merged but I forgot to replace the model repos in the docs to a more "official" one.
As discussed with original authors, the checkpoints are available under [AnnaZhang](https://huggingface.co/AnnaZhang) profile. I just sent them an email about the merge, they should pull the latest version of the weights soon.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

@Cyrilvallez @stevhliu 

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: lw_detr

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43266&sha=2ef87f

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43266). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Floating-point precision issues in iamge_processing_deepseek_vl.py
Author: frankenliu
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.57.1
- Platform: Linux-6.8.0-54-generic-x86_64-with-glibc2.39
- Python version: 3.12.12
- Huggingface_hub version: 0.36.0
- Safetensors version: 0.6.2
- Accelerate version: 1.11.0
- Accelerate config:    not found
- DeepSpeed version: 0.18.1
- PyTorch version (accelerator?): 2.9.0+cu128 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>

### Who can help?

@yonigozlan @molbap

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In image_processing_deepseek_vl.py line 164-182

        if input_data_format is None:
            input_data_format = infer_channel_dimension_format(image)

        height, width = get_image_size(image, input_data_format)
        max_size = max(height, width)

        size = get_size_dict(size, default_to_square=True)
        if size["height"] != size["width"]:
            raise ValueError(
                f"Output height and width must be the same. Got height={size['height']} and width={size['width']}"
            )
        size = size["height"]

        delta = size / max_size
        # Largest side becomes `size` and the other side is scaled according to the aspect ratio.
        output_size_nonpadded = [
            max(int(height * delta), self.min_size),
            max(int(width * delta), self.min_size),
        ]
when height=2522, size=384, run:
        
        height, width = 2522, 928
        size = 384
        delta = size / height
        out_size = [int(height * delta), int(width * delta)]
        print(f"out_size:{out_size}")
        new_out_size = [round(height * delta), round(width * delta)]
        print(f"new_out_size:{new_out_size}")
out_size:[383, 141]
new_out_size:[384, 141]

### Expected behavior

change 
        output_size_nonpadded = [
            max(int(height * delta), self.min_size),
            max(int(width * delta), self.min_size),
        ]
to
        output_size_nonpadded = [
            max(round(height * delta), self.min_size),
            max(round(width * delta), self.min_size),
        ]

Comments:
Qwen3-VL-Moe implementation is not sparse
Author: Maxwell-Lyu
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.57.5
- Platform: Linux-5.4.0-200-generic-x86_64-with-glibc2.31
- Python version: 3.10.19
- Huggingface_hub version: 0.36.0
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.8.0+cu128 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA RTX A6000

### Who can help?

@yonigozlan 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Any attempt to do inference using Qwen3VLMoe model.

### Expected behavior

Expect a sparse forward of experts simillar to Qwen3MoeModel.

Comments:
convert trocr to onnx
Author: cqray1990
Pull request: no
Labels: name: bug

### System Info

python -m \
    transformers.onnx \
  --model=./checkpoint \
    --feature=vision2seq-lm \
    hand-write-onnx --atol 1e-4

error:
 
usage: Hugging Face ONNX Exporter tool [-h] -m MODEL
                                       [--feature {causal-lm,causal-lm-with-past,default,default-with-past,masked-lm,question-answering,seq2seq-lm,seq2seq-lm-with-past,sequence-classification,token-classification}]
                                       [--opset OPSET] [--atol ATOL]
                                       output
Hugging Face ONNX Exporter tool: error: argument --feature: invalid choice: 'vision2seq-lm' (choose from 'causal-lm', 'causal-lm-with-past', 'default', 'default-with-past', 'masked-lm', 'question-answering', 'seq2seq-lm', 'seq2seq-lm-with-past', 'sequence-classification', 'token-classification')

trocr is not vision2seq-lm?

### Who can help?

transfoomers 4.15.0

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

python -m \
    transformers.onnx \
  --model=./checkpoint \
    --feature=vision2seq-lm \
    hand-write-onnx --atol 1e-4


### Expected behavior

python -m \
    transformers.onnx \
  --model=./checkpoint \
    --feature=vision2seq-lm \
    hand-write-onnx --atol 1e-4


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
How to run Phi4MultimodalProcessor
Author: wcrzlh
Pull request: no
Labels: name: bug

### System Info

transformers version: 4.57.1
python version: 3.9

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

[Phi4MultiModal example](https://huggingface.co/docs/transformers/model_doc/phi4_multimodal)

### Expected behavior

I just run [the example](https://huggingface.co/docs/transformers/model_doc/phi4_multimodal) but there is an error raised.

Comments:
Can you please provide more info, meaning what example you're running 🙏

> Can you please provide more info, meaning what example you're running 🙏

I use the example in huggingface transformers docs for testing:
```
from transformers import AutoProcessor

model_path = "microsoft/Phi-4-multimodal-instruct"

processor = AutoProcessor.from_pretrained(model_path)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"},
            {"type": "text", "text": "What is shown in this image?"},
        ],
    },
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
)
```

It would raise an error:
```
Value Error: Loading this model requires you to execute custom code contained in the model repository on your local machine. Please set the option `trust_remote_code=True` to permit loading of this model.
```

@wcrzlh Hi, I think you just need to add `trust_remote_code=True` when loading the processor, as suggested by the error message.

You can try changing the line to:


```python
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
```

Thx. I tried to add `trust_remote_code=True` and it works. But I notice there is different implementation between huggingface model hub and transformers.models.phi4. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add support for logging number of image tokens
Author: h-tonywu
Pull request: yes

# What does this PR do?

With a single `include_num_input_image_tokens_seen` flag in the `TrainingArguments` (here with `SFTConfig`), you can report the number of visual tokens seen by the model.

Example:

```python
from trl import SFTConfig

# Configure training arguments using SFTConfig
training_args = SFTConfig(
    # insert other arguments for training
    report_to="wandb",
    include_num_input_image_tokens_seen=True,  # new
)
```

**Result when training a SmolVL**

- Model seed: `HuggingFaceTB/SmolVLM-256M-Instruct`
- Dataset: `merve/vqav2-small`

![Screenshot 2025-07-04 at 15 23 04](https://github.com/user-attachments/assets/a36ca688-7e08-4af9-b1a5-58ea63117d0f)

**Why is it useful?**

When training VLMs, monitoring the number of processed visual tokens helps with knowing the proportion of image-related tokens that are seen by the model during training. Moreover, for VLMs with dynamic patching (e.g., Qwen2-VL) this can also provide a signal about the resolution of images ingested.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation). → TODO
- [ ] Did you write any new necessary tests? → there are no existing tests for logging metrics in the repository, so I'm not sure writing my own test is relevant.

## Who can review?

@SunMarc, @yonigozlan, @qubvel


Comments:
> Thanks for adding this feature! It would be super nice to have any test case 🤗 Other than that looks great

I'll try to look into this ASAP, thanks guys! 🤗 

@qubvel I've thought about an easy way to test my code but that might demand a bit more work. Here's what I had in mind:

Implementing a `BasicImageTextToTextModel` model similarly to `BasicTextGenerationModel`: https://github.com/huggingface/transformers/blob/18e0cae207a38d2c430b5fa08f9597312d1c1ab3/tests/trainer/test_trainer.py#L507-L518

Then we could use it to test if the visual token count is correct. Wdyt? (btw maybe this class could be helpful for the VLM refactoring @yonigozlan, or maybe it already exists!)

Moreover, could we implement these tests in another PR?

I would suggest adding some tests to this PR, otherwise, with any VLMs or trainer refactoring, we might break the feature and miss it (and we are doing a lot of refactoring at the moment to unbloat the code)
implemented mimo model
Author: Aaraviitkgp
Pull request: yes

Issue : https://github.com/huggingface/transformers/issues/42954

I have implemented mimo model,
I have just changed rope_type as it was default which was causing error so now it is none by default also changed and have made some changes in class MiMoV2FlashRotaryEmbedding, added files for mimo integration.

@ArthurZucker

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto

@ArthurZucker The setup and quality workflow seems to fail, I am not able to understand why it is coming. 

@Aaraviitkgp  hello, i found your transformers github, https://github.com/Aaraviitkgp/transformers, have four mimo-v2-flash implemented branch. 
which one is the trainable final version of mimo-v2-flash. 
can i use the origin repo https://huggingface.co/XiaomiMiMo/MiMo-V2-Flash model to train?
Thank you for sharing and look forward to your reply
[mimi] mimi conv cache edge case
Author: eustlb
Pull request: yes

# What does this PR do?

two things;
1. factorize the cache init so that it can be overwritten
2. handle the specific edge case with short hidden state: if lenght is < from padding, then the cache init should be taking into account in the padding states. This edge case is never matched with mimi but happens with other models that would use this cache approach (#40546 )


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43130). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: kyutai_speech_to_text, mimi
remove restriction for 8-bit bnb model when moving across devices
Author: SunMarc
Pull request: yes

# What does this PR do?

As per title ! 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43258). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@bot /style 

Style fix is beginning .... [View the workflow run here](https://github.com/huggingface/transformers/actions/runs/20963255910).

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bnb

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43258&sha=01decb
[loading] Fix Transpose Operation, and qwen3_vl_moe mapping
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. Transpose needs to be more general in order to be used in reverse mode in a chain of several operations.
Supersedes https://github.com/huggingface/transformers/pull/43201 as it needed more work on the Transpose Operation!

Will also help https://github.com/huggingface/transformers/pull/43227

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43307). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Actually I checked and the params on which `Transpose` are used are not part of the TP plan anyway. So i'd rather merge this PR now as it fixes the model and unblocks other PRs, then we can see in the future about the TP shard dim if the issue comes up 

Can we add a comment at least, TODO/FIXME or similar at least? This is quite important (although not used with TP atm)

Done. I think it's easier this way as it's not as straightforward as I thought it would be to always switch the shard dim if required

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43307&sha=9a937a
DeepSeek-V2-Lite-Chat repetitive generation when `_experts_implementation="eager"`
Author: yiliu30
Pull request: no
Labels: name: bug

### System Info


- `transformers` version: 5.0.0.dev0
- Platform: Linux-5.4.0-169-generic-x86_64-with-glibc2.35
- Python version: 3.11.10
- Huggingface_hub version: 1.3.1
- Safetensors version: 0.4.5
- Accelerate version: 1.10.1
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.9.0+cu128 (CUDA)
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>


### Who can help?

text models: @ArthurZucker @Cyrilvallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python


import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
import transformers

model_name = "deepseek-ai/DeepSeek-V2-Lite-Chat"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

with torch.no_grad(), torch.device(device):
    trust_remote_code = False
    tokenizer = AutoTokenizer.from_pretrained(model_name,trust_remote_code=trust_remote_code)
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype="auto",
        trust_remote_code=trust_remote_code,
        _experts_implementation="eager"
        )
    # generate some text
    inputs = tokenizer("Hello, my dog is", return_tensors="pt").to(device)
    outputs = model.generate(**inputs, max_new_tokens=20)
    print(tokenizer.decode(outputs[0], skip_special_tokens=True))
    
    
# Setting `pad_token_id` to `eos_token_id`:100001 for open-end generation.
# Hello, my dog isodododododododododododododododododododod

```

### Expected behavior

The output should be reasonable.

Comments:
This is a chat model and outputs may be garbage if you don't use the chat template correctly. Have you tried using `tokenizer.apply_chat_template()` to format the input, or checked that you get a reasonable output without `_experts_implementation="eager"`?

Hi @Rocketknight1 if we remove the `_experts_implementation="eager"`, the output looks good.

Confirmed the issue locally, and confirmed that chat templates don't fix it. It does seem like `_experts_implementation="eager"` is broken for at least some models!

@yiliu30 this should now be fixed on `main` and the fix will be included in the next release. Thank you for the bug report!
Updated Ben
Author: BenjaminBossan
Pull request: yes

As discussed internally.

Still not finished, running into this error now:

```
MixtralForCausalLM LOAD REPORT from: peft-internal-testing/mixtral-pre-v5-lora
Key                                                              | Status   | Details                                                                                    
-----------------------------------------------------------------+----------+--------------------------------------------------------------------------------------------
model.layers.{0, 1}.mlp.experts.base_layer.lora_A.default.weight | MISMATCH | Reinit due to size mismatch ckpt: torch.Size([8, 16, 1024]) vs model:torch.Size([64, 7168])
model.layers.{0, 1}.mlp.experts.base_layer.lora_B.default.weight | MISMATCH | Reinit due to size mismatch ckpt: torch.Size([2, 7168, 16]) vs model:torch.Size([1024, 64])
model.layers.{0, 1}.mlp.experts.lora_B.default.weight            | MISMATCH | Reinit due to size mismatch ckpt: torch.Size([8, 1024, 8]) vs model:torch.Size([3584, 64]) 
model.layers.{0, 1}.mlp.experts.lora_A.default.weight            | MISMATCH | Reinit due to size mismatch ckpt: torch.Size([8, 8, 3584]) vs model:torch.Size([64, 1024]) 
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43319). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43319&sha=32adc7
[bugfix] fix qwen3vlmoe loading
Author: JJJYmmm
Pull request: yes

# What does this PR do?

Fix the loading of qwen3vlmoe experts.

test script:

```
from transformers import AutoProcessor, AutoModelForImageTextToText

model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3-VL-30B-A3B-Instruct", torch_dtype="auto", device_map="auto")
```

before:
```
model.language_model.layers.{0...47}.mlp.experts.down_proj    | MISMATCH | Reinit due to size mismatch ckpt: torch.Size([128, 768, 2048]) vs model:torch.Size([128, 2048, 768])  
model.language_model.layers.{0...47}.mlp.experts.gate_up_proj | MISMATCH | Reinit due to size mismatch ckpt: torch.Size([128, 2048, 1536]) vs model:torch.Size([128, 1536, 2048])
```

The reason is that the official checkpoint of Qwen3VLMoe follows the shape `[num_experts, out_features, in_features]`, but in the latest code, these weights are transposed in the last two dimensions. This pr transposes them back during conversion.

The model now loads successfully after the fix. 🫡


Comments:
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43201&sha=a9b5dc

Similar issue as in #43227 where reverse mapping fails cc @Cyrilvallez 

@vasqu a quik fix that i found for now is to rename the tensor first and then transpose it.

I'll wait for #43227 to be merged and then apply the final fix for the transposed experts. 🫡

Closing as superseded by #43307!