noamaanMulla-03/transformers-issues huggingface dataset:
Replace VMap that Fails Torch.Export
Author: YifanShenSZ
Pull request: yes

The specific vmap used in causal mask construction would fail torch.export by
```log
torch._dynamo.exc.TorchRuntimeError: Dynamo failed to run FX node with fake tensors: call_function <built-in function getitem>(*(FakeTensor(..., size=(1, 8), dtype=torch.int64), (BatchedTensor(lvl=1, bdim=0, value=
    FakeTensor(..., size=(1,), dtype=torch.int64)
), BatchedTensor(lvl=3, bdim=0, value=
    FakeTensor(..., size=(8,), dtype=torch.int64)
))), **{}): got RuntimeError("vmap: It looks like you're calling .item() on a Tensor. We don't support vmap over calling .item() on a Tensor, please try to rewrite what you're doing with other operations. If error is occurring somewhere inside PyTorch internals, please file a bug report.")
```

The solution is to use a simple causal mask construction
```python
    q_indices = torch.arange(kv_length - q_length, kv_length)
    k_indices = torch.arange(kv_length)
    causal_mask = q_indices[:, None] >= k_indices[None, :]
```

cc @ArthurZucker @Cyrilvallez

Comments:
We have #41852 which should be available in the v5.rc0. I don't think we will backport for v4, as we keep this to patches only (I think).

For v4 you can check the `integrations/executorch.py` on that version. Back then we had a different implementation that avoided this by overwriting the attention/mask fns. 

Thx Anton! 🙏 So by `v5` you mean at release 4.57.5 we will default to the non-vmap mask creation and torch.export can work out of the box? 😆

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: llama4

Haha no, I really meant v5 as in major version :D see https://github.com/huggingface/transformers/releases/tag/v5.0.0rc0 

Close as we will get the fix in v5
fix(Qwen3VLProcessor): fix incorrect handling of return_metadata
Author: yiguanxianyu
Pull request: yes

This PR fixes the following: 

- ensures video metadata is correctly returned in `Qwen3VLProcessor` when `return_metadata=True` during `processor.apply_chat_template()`.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen3_vl

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42731&sha=bbbcbd
Add optional backend parameter to VideosKwargs
Author: yiguanxianyu
Pull request: yes

- Add optional backend and torchcodec_device parameter in VideosKwargs to allow users to specify the desired video processing backend.
- Fix device detection while reading videos with torchcodec.

Comments:
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42733&sha=2eb668
Fix tests quantization 
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR fixes some errors in our quantization tests. Let's have a green ci for 2026. The only two quantization method left would be bnb and autoawq but those will be fixed in follow-up PRs 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42703). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42703&sha=a90ebc

@bot /style 

Style bot fixed some files and pushed the changes.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: compressed_tensors_integration, hqq, mxfp4, quark_integration, torchao_integration
[kernels] make the module declaration implicit with decorator
Author: MekkCyber
Pull request: yes

# What does this PR do?

instead of setting the module explicitely and not using it, this pr simply uses a decorator to set the attribute which is cleaner

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42700). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: afmoe, apertus, arcee, aria, bamba, bitnet, cohere, cohere2, csm, cwm, dia, dots1, emu3, ernie4_5, ernie4_5_moe, evolla
fix tp
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR fixes the initialization of `parallelism_config` when using tp. Fixes https://github.com/huggingface/transformers/issues/42661

Comments:
@bot /style 

Style bot fixed some files and pushed the changes.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42704&sha=5b2ddd

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42704). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Why below line was added?

Requires accelerate>1.12.0 to use Tensor Parallelism.

`ParallelismConfig` comes from a recent version of accelerate
Update replace_with_ for quants methods to not use recursion
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR updates replace_with_xxx methods to not use recursion anymore. This is simpler and easier to read.  The only one left is hqq and awq. For awq, I will wait for the refactor, for hqq let's see. 

Comments:
run-slow: bnb

This comment contains `run-slow`, running the specified jobs: 

models: []
quantizations: ["quantization/bnb"]

run-slow: bnb

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/20036862780)

⚠️ No test being reported (jobs are skipped or cancelled)!


This comment contains `run-slow`, running the specified jobs: 

models: []
quantizations: ["quantization/bnb"]

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/20037004356)

✅ No failing test specific to this PR 🎉 !


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42711). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix failing `CodeGenModelTests`
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Fix failing [CodeGenModelTests](https://github.com/huggingface/transformers/actions/runs/19983883282/job/57315087774#step:14:1844). 

These 2 `["def hello_world():", "def greet(name):"]` are resulting in same length, with no padding for either one.
```py

# use different length sentences to test batching
sentences = ["def hello_world():", "def greet(name):"]
inputs = tokenizer(sentences, return_tensors="pt", padding=True)
```
So, modified to this
```py
sentences = ["def greetings(name):", "def hello_world():"]
```


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@Rocketknight1 @ArthurZucker 

Comments:
run-slow: codegen

This comment contains `run-slow`, running the specified jobs: 

models: ["models/codegen"]
quantizations: []

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42730). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> run-slow: codegen

Sorry!. Forgot to remove `breakpoint()` at 1 location. So, CI is not moving. Removed it now.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: codegen

No probs!

run-slow: codegen

> No probs!
> 
> run-slow: codegen

Thanks! I think you need to stop the [previous one](https://github.com/huggingface/transformers/actions/runs/20064797768/job/57551135726).

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/20064797768)

✅ No failing test specific to this PR 🎉 !
Remove redundant else in activations.py
Author: leaderofARS
Pull request: yes

# Remove Redundant Else Block in get_activation()

**Fixes #42621**

This PR removes a redundant `else` block in `get_activation()` in `src/transformers/activations.py`. The `else` was unnecessary because the `if` branch already returned, and keeping it went against the project's preferred early-return style.

## Previous Control Flow

```python
def get_activation(activation_string):
    if activation_string in ACT2FN:
        return ACT2FN[activation_string]
    else:
        raise KeyError(f"function {activation_string} not found in ACT2FN mapping {list(ACT2FN.keys())}")
```

## Updated Control Flow

```python
def get_activation(activation_string):
    if activation_string in ACT2FN:
        return ACT2FN[activation_string]
    raise KeyError(f"function {activation_string} not found in ACT2FN mapping {list(ACT2FN.keys())}")
```

## Summary

- Pure cleanup (no behavior change)
- Simplifies logic and matches HF style
- All style & repo checks pass


Comments:
[BUG] [v5] Models loaded with zero3 have weights that are all zeros or all ones.
Author: Jintao-Huang
Pull request: no

transformers 5.0.0.dev main branch

<img width="551" height="442" alt="Image" src="https://github.com/user-attachments/assets/e583235d-ed32-4a77-a50e-814fc46bd81d" />


Comments:
cc @sunmarc @arthurzucker

Thanks for the report @Jintao-Huang, do you have a simple reproducer ? 

Fixed by #42626
Fix AutoTokenizer fallback to TokenizersBackend when tokenizer class is unknown
Author: Abdennacer-Badaoui
Pull request: yes

When `AutoTokenizer.from_pretrained()` cannot determine the tokenizer class from config, it now attempts to load using `TokenizersBackend` directly before falling back to SentencePiece. This enables loading tokenizers that have `tokenizer.json` files (like Bloom models) even when the tokenizer class cannot be determined from the model configuration.

This fix will make all the failing tests in `test_pipelines_text_generation` pass, and possibly some others as well.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42735). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey! if you can rebase / check this should have been fixed AFAIK

but thanks for giving it a look

Yes, that's working now! 👌🏻
[FP8 Devstral 24B] Repro PR
Author: patrickvonplaten
Pull request: yes

This PR can be used as a repro to show that the FP8 weight scales are correct for: https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512

It seems like somehow the activation scales are blowing up?!

Comments:
Refactor MoE experts for FSDP/ZeRO-3 compatibility
Author: shuyingl
Pull request: yes

## Summary

Refactors DeepSeek V3.2 MoE expert implementation to be compatible with FSDP and DeepSpeed ZeRO-3 distributed training frameworks.

### Problem
The previous implementation stored expert weights as 3D parameter tensors (`nn.Parameter(torch.empty(num_experts, 2*inter_dim, hidden_dim))`), which:
- Cannot be efficiently sharded by FSDP/ZeRO-3
- Required manual Expert Parallelism (EP) logic with hardcoded `ep_size`/`ep_rank`
- Used explicit `.to(device)` calls that interfere with automatic device management

### Solution
- **New `DeepseekV32Expert` class**: Individual expert module with standard `nn.Linear` layers
- **`nn.ModuleList`-based experts**: Each expert can be independently wrapped/sharded by distributed frameworks
- **Removed manual EP logic**: Let FSDP/ZeRO-3 handle parameter sharding and gradient synchronization
- **Removed explicit device placement**: Allow frameworks to manage device placement automatically

### Usage

**With FSDP:**
```python
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp.wrap import ModuleWrapPolicy
from transformers import DeepseekV32ForCausalLM, DeepseekV32Expert

model = DeepseekV32ForCausalLM(config)
model = FSDP(model, auto_wrap_policy=ModuleWrapPolicy({DeepseekV32Expert}))
```

**With DeepSpeed ZeRO-3:**
```python
import deepspeed
ds_config = {"zero_optimization": {"stage": 3}, "bf16": {"enabled": True}}
model = DeepseekV32ForCausalLM(config)
model, optimizer, _, _ = deepspeed.initialize(model=model, config=ds_config)
```

## Test plan
- [x] All 15 MoE-specific tests pass (`test_distributed_moe.py`)
- [x] Tests verify `nn.ModuleList` structure and `nn.Linear` layer usage
- [x] Tests confirm no old EP-specific attributes remain
- [x] Gradient checkpointing tests pass
- [x] Forward/backward pass tests pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Comments:
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42755&sha=86ab37
[Quantization] Fixing some tests
Author: MekkCyber
Pull request: yes

# What does this PR do?

Fix some remaining tests 

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bnb, torchao_integration

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42763). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Only call `torch.autocast` if it will have an effect
Author: hmellor
Pull request: yes

Adds a context manager to replace direct use of `torch.autocast` that only autocasts if:
    
- `autocast` is already enabled in this context 
- Or this call to `maybe_autocast` has `enabled=True`

This prevents `autocast` being added to the graph when it is effectively a no-op.
Which makes graph splitting in `torch.compile` more flexible as it removes the requirement that partition IDs be monotonically increasing.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: afmoe, apertus, arcee, aria, bamba, bitnet, blt, chameleon, cohere, cohere2, csm, cwm, dbrx, decision_transformer, deepseek_v2, deepseek_v3

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42747). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[kernels] Fix kernel CI
Author: MekkCyber
Pull request: yes

# What does this PR do?

Fix the kernel CI by simply changing the import path of `lazy_import_flash_attention`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42764). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
add_generation_prompt=False in Tokenizer.apply_chat_template has no effect
Author: AndreiMuresanu
Pull request: no
Labels: name: Chat Template

### System Info

Hi, I am trying to apply a chat template to an input without the generation prompt. However, setting add_generation_prompt=False appears to have no effect.

### Who can help?

@ArthurZucker @Rocketknight1

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Minimal reproducible example:

```Python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

chat = [
		{"role": "assistant", "content": "Hello, how are you?"},
	]
formatted_chat = tokenizer.apply_chat_template(chats, add_generation_prompt=False, tokenize=False)
print(formatted_chat)
```

transformers==4.41.0

### Expected behavior

Observed Output:

```


assistant


Hello, how are you?

assistant



```

Expected Output:

```


assistant


Hello, how are you?

```

Comments:
Hi @AndreiMuresanu, I just tried to reproduce this and `add_generation_prompt=False` worked correctly for me. Can you retry your code, and make sure you're updated to the latest version of `transformers`?

I have also encountered this issue. 

model : ```Llama-3-8B-Instruct```
package: ```transformers==4.41.1```, ```tokenizer==0.19.1```

Can you try installing `transformers` from `main` with `pip install git+https://github.com/huggingface/transformers.git`? I can't reproduce this issue at all, and I'm curious if it's caused by an old cached version of the model, or some recent change, or something.

same problem

> same problem

Solved. Because I used the initial chat template, which was later updated.

See Fix chat template to add generation prompt only if the option is selected ([#9](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/discussions/9))

@iseesaw thanks for pointing that one out! It seems like this was an issue with the model's chat template, which has since been resolved, and the issue will be fixed if people redownload the tokenizer. Going to close this issue now, since it's not a bug in `transformers` itself.

im sick of this : Error: Kwargs ['add_generation_prompt'] are not supported by `MistralCommonTokenizer.apply_chat_template`
trying to make live streaming with the TTS model voxtral-mini-3b and serving it via vllm, but vllm require transformers version 4.49.x or something like that while here you are saying to solve it it should be ==4.41 , does anyone have idea what can i do? im new here
Support `BatchFeature` in `LengthGroupedSampler` for Multimodal compatibility
Author: npurson
Pull request: no
Labels: name: Feature request

### Feature request

I am currently fine-tuning a multimodal model (Qwen2.5-VL) using the official Trainer. The training fails during the dataset length inference step in `LengthGroupedSampler` because the code strictly checks for dict or `BatchEncoding`, but multimodal processors often return `BatchFeature`.

Specifically, the following check raises a ValueError:

https://github.com/huggingface/transformers/blob/471d7ce9abbb3bc1b3bab673367378f9dbc3caac/src/transformers/trainer_pt_utils.py#L471

https://github.com/huggingface/transformers/blob/471d7ce9abbb3bc1b3bab673367378f9dbc3caac/src/transformers/trainer_pt_utils.py#L531

### Motivation

As above.

### Your contribution

Simply adding `BatchFeature` in the type check.

Comments:
@npurson sounds good, do you want to submit a PR?
Only default `rope_parameters` to empty `dict` if there is something to put in it
Author: hmellor
Pull request: yes

`set(rope_parameters.keys()).issubset(self.layer_types)` will evaluate to `True` if `rope_parameters` is an empty `dict`.

This is a problem if `standardise_rope_params` is called on a config for a model which do not have RoPE because it will trigger `Case 2`. This method might be called on models which do not have RoPE in third party libraries (e.g. vLLM) in order to "upgrade" configs of custom models they use.

This PR makes `standardise_rope_params` harmless to call on configs with no RoPE at all.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42651). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42651&sha=d58675
DataCollatorWithFlattening only accepts labels as list
Author: romitjain
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.55.4
- Platform: Linux-5.14.0-284.73.1.el9_2.x86_64-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 0.36.0
- Safetensors version: 0.5.2
- Accelerate version: 1.12.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.8.0+cu128 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: no
- GPU type: NVIDIA A100-SXM4-80GB

### Who can help?

@ArthurZucker @Cyrilvallez 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
# test.py
import torch
from transformers import DataCollatorWithFlattening

def repro_tensor_labels():
    features = [
        {
            "input_ids": torch.tensor([1, 2, 3, 4]),
            "labels": torch.tensor([10, 11, 12, 13]),
        },
        {
            "input_ids": torch.tensor([5, 6, 7]),
            "labels": torch.tensor([14, 15, 16]),
        },
    ]
    collator = DataCollatorWithFlattening(return_tensors="pt")
    return collator(features)

def repro_list_labels():
    features = [
        {
            "input_ids": torch.tensor([1, 2, 3, 4]),
            "labels": [10, 11, 12, 13],
        },
        {
            "input_ids": torch.tensor([5, 6, 7]),
            "labels": [14, 15, 16],
        },
    ]
    collator = DataCollatorWithFlattening(return_tensors="pt")
    return collator(features)

print("labels as tensors")
try:
    batch = repro_tensor_labels()
    print("batch:", batch)
except Exception as e:
    print("got error:", repr(e))

print("\nlabels as list")
batch = repro_list_labels()
for k, v in batch.items():
    print(k, v)
```

```bash
python test.py
```

This gives

```
labels as tensors
got error: TypeError('can only concatenate list (not "Tensor") to list')

labels as list
input_ids tensor([[1, 2, 3, 4, 5, 6, 7]])
labels tensor([[-100,   11,   12,   13, -100,   15,   16]])
position_ids tensor([[0, 1, 2, 3, 0, 1, 2]])
```

### Expected behavior

The current implementation of DataCollatorWithFlattening only works with labels provided as lists.
Can we support labels as `torch.Tensor`?

I can create a PR for the fix.

Comments:
cc @vasqu here, I don't remember, is this expected to fail or not? Do we only accept list inputs?

Kinda expected as we do not expect a certain datatype beforehand, i.e. "primitive" python. It also does not hurt to support those cases tho, it's a nice qol improvement (seeing already #42620 adding this cc @Rocketknight1)

That was my logic too - I wasn't sure if it was supposed to work with tensors or not, but if we can easily support it without breaking other code then we probably should.
[Devstral 24B] FP8 is currently not working correctly
Author: patrickvonplaten
Pull request: no
Labels: name: bug

### System Info

Latest transformers "main".

### Who can help?

@SunMarc @MekkCyber 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

If you run this code snippet: https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512#transformers **with** `dequantize=False`, you will notice that it has infinite repetition issues.

If however you run the model from this PR: https://github.com/huggingface/transformers/pull/42744, you can see that everything works fine which leads to the conclusion that something funky is going on with the activation scales (maybe they give `inf` values somewhere?).

The same activation scales work for vLLM: https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512#vllm-recommended so there is probably something we can do inside transformers to fix it?

### Expected behavior

That FP8 works correctly

Comments: