noamaanMulla-03/transformers-issues huggingface dataset:
handle flash slow tests
Author: itazap
Pull request: yes

fixing broken slow / flash tests as we cannot pass the regular 4dim attention mask for flash

_update_causal_mask does not work with the patch cross attention , handling patch attn mask update for flash in the _prepare function

Comments:
run-slow: blt

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41072). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you @itazap . I will let @vasqu or @Cyrilvallez or @ArthurZucker to review this.

Searching `"flash" in self.config._attn_implementation` within the codebase, I can see this happens mostly within methods `def _update_xxx_mask`, like `def _update_cross_attn_mask`, `_update_causal_mask`, etc.

So probably it's better to do the same for this model, but I'm not super sure if this values a lot.


And thank you for use `run-slow`.

It turns out that your github username is not in `.github/workflows/self-comment-ci.yml`, so the CI is not triggered.

I will open a PR to add you, then you can rebase on main and use `run-slow` again 🙏 


Could you approve

https://github.com/huggingface/transformers/pull/41094

🙏 



Merged #41094. You can rebase and try `run-slow` again.

(or if you already try to run the tests with a SSH runner, including slow tests , it's fine)

run-slow: blt

This comment contains run-slow, running the specified jobs: 

models: ['models/blt']
quantizations: [] ...

This comment contains run-slow, running the specified jobs: 

models: ['models/blt']
quantizations: [] ...

run-slow: blt

This comment contains run-slow, running the specified jobs: 

models: ['models/blt']
quantizations: [] ...

run-slow: blt

This comment contains run-slow, running the specified jobs: 

models: ['models/blt']
quantizations: [] ...

run-slow: blt

This comment contains run-slow, running the specified jobs: 

models: ['models/blt']
quantizations: [] ...

run-slow: blt

This comment contains run-slow, running the specified jobs: 

models: ['models/blt']
quantizations: [] ...

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: blt

run-slow: blt

This comment contains run-slow, running the specified jobs: 

models: ['models/blt']
quantizations: [] ...
LFM2-VL PyTorch
Author: yukiarimo
Pull request: no

Hello!

Can you please share the full PyTorch implementation of the LFM2-VL model so it will be possible to not rely on transformers (and also would be helpful for building custom models)?

Thanks!

Comments:
Hey @yukiarimo , not sure what you mean by full implementation. The model is supported and all the related model/processing files are located in https://github.com/huggingface/transformers/tree/main/src/transformers/models/lfm2_vl 🤗 

Yeah, I saw that. What I meant is I'm looking for pure PyTorch implementation without relying on transformers!

Solved. Built custom VLM.
Use DP+FSDP device mesh dimensions for scaling loss with default value of average_tokens_across_devices: True
Author: winglian
Pull request: no
Labels: name: bug

### System Info

So with [this PR changing the default for how the train loss is scaled](https://github.com/huggingface/transformers/pull/39395) and with [accelerate now supporting composable parallelisms](https://github.com/huggingface/accelerate/pull/3682), we need to update these token scaling to only scale across the data parallel dimension groups/meshes and ignore any TP or CP meshes. 

https://github.com/huggingface/transformers/blob/ad6fd2da0e09233f757bdb0bab0b1ee2b931f33c/src/transformers/trainer.py#L3900-L3905
and
https://github.com/huggingface/transformers/blob/ad6fd2da0e09233f757bdb0bab0b1ee2b931f33c/src/transformers/trainer.py#L5362-L5371


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

in axolotl, remove the hardcoded `average_tokens_across_devices` setting to use the default True, set any of `context_parallel_size` or `tensor_parallel_size` and the training loss will be scaled up by the product of those values.

### Expected behavior

train loss should be similar to the values when using pure DDP.

Comments:
I'm looking into this and planning to open a PR with a fix for correctly scaling the loss and token count across only the data parallel group, excluding TP/CP meshes. Will update soon.


cc @sunmarc @S1ro1 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I think we tackled this bit in the cp pr https://github.com/huggingface/transformers/pull/40205. This should solve the issue no @winglian ? cc @S1ro1 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Is`_index_first_axis` functionally equal to FA2's `index_first_axis`
Author: A1waysBeenHere
Pull request: no

Ref to the #40002 . This PR consoliated this function into `transformers.modeling_falsh_attention_utils` and removed the `index_first_axis` function that directly copied from flash-attention repo. I am wondering if they are functionally equally. 

```python
# The shape is recovered to the original one at the end, and ref to the comment, 
# it actually equal to return input[indices]
class IndexFirstAxis(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input, indices):
        ctx.save_for_backward(indices)
        assert input.ndim >= 2
        ctx.first_axis_dim, other_shape = input.shape[0], input.shape[1:]
        second_dim = other_shape.numel()
        # TD [2022-03-04] For some reason torch.gather is a bit faster than indexing.
        # return input[indices]
        return torch.gather(
            rearrange(input, "b ... -> b (...)"), 0, repeat(indices, "z -> z d", d=second_dim)
        ).reshape(-1, *other_shape)


# The currently one, I don't think the shape are exactly same here, cuz no shape recoverd
def _index_first_axis(tensor, indices):
    """
    A local implementation of the PyTorch indexing operation `tensor[indices]` on the first axis,
    after flattening the first two dimensions of the tensor. This is functionally equivalent to
    FA2's `index_first_axis` and replaces the need to import it.
    """
    # The input tensor is expected to be of shape (batch, seq_len, ...). We flatten the first
    # two dimensions to get (total_tokens, ...) before indexing.
    reshaped_tensor = tensor.reshape(-1, *tensor.shape[2:])
    return reshaped_tensor[indices]

```

Comments:
Hey @A1waysBeenHere,

- short answer: Yes, they are equal. 
- long answer: The reshape is necessary due to torch gather consolidating different shapes (see the rearrange, repeat shapes). You can also see Tri Dao's comment in the original implementation (right before the gather): ["# TD [2022-03-04] For some reason torch.gather is a bit faster than indexing. # return input[indices]"](https://github.com/Dao-AILab/flash-attention/blob/add175637c5d54b74bc25372e49ce282d6f236fc/flash_attn/bert_padding.py#L15-L16) (assuming the first two dimensions are flattened, i.e. bsz, seq_len -> total_nnz)

Edit: Depending on how you used it, there may be a difference in the original shape. The new pure torch version expects a non-flattened tensor, while the other original one expects the flattened one already. This depends on the context of the `pad/unpad` functions. So it is more likely correct to say that they are equal under `unpad` and expect different initial shapes `(bzs, seq_len, ...)` vs `(total_nnz, ...)`

Since it was originally not meant to be public-facing, I'm open for a PR here so that the functions are really equal, i.e. the initial shapes are the same (should be only the reshape to be moved outside).
Add Zagros and ZagrosNext model architectures to Transformers
Author: humanprojectceo
Pull request: yes

What does this PR do?
This PR introduces two new model architectures, Zagros and ZagrosNext, to the Transformers library. The implementation includes:

Model architecture files (modeling_zagros.py and modeling_zagros_next.py) in src/transformers/models/zagros/ and src/transformers/models/zagros_next/.
Configuration files (configuration_zagros.py and configuration_zagros_next.py) for both models.
Comprehensive tests for both models in tests/models/zagros/ and tests/models/zagros_next/.
Updated documentation in docs/source/en/ with usage examples and details for Zagros and ZagrosNext.

Both models follow the standard structure and conventions of existing Transformers models, ensuring compatibility with the library's pipelines and utilities.
Motivation and Context

The Zagros and ZagrosNext models are designed to [briefly describe the purpose, e.g., "enhance performance on specific NLP tasks with novel architectural improvements"].
These models leverage standard Transformer conventions, making them easy to integrate into existing workflows.
The implementation is fully compatible with the Transformers library and supports all standard functionalities (e.g., training, inference, and pipeline integration).

Dependencies

No additional dependencies are required beyond the standard Transformers setup.
Tested with Python 3.9+ and PyTorch 2.0+.

Before submitting

 This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
 Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request)?
 Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
 Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
 Did you write any new necessary tests?

Who can review?
@ArthurZucker (for text models)@Rocketknight1 (for pipelines and library compatibility)@stevhliu (for documentation)
Thank you for reviewing!

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, zagros, zagros_next

hi @ZagrosLLMModel, we generally don't accept PRs for new architectures without significant pre-trained checkpoints! Are there any pre-trained Zagros models available?

> hi @ZagrosLLMModel, we generally don't accept PRs for new architectures without significant pre-trained checkpoints! Are there any pre-trained Zagros models available?

Hi, yes sir we have:
https://huggingface.co/darsadilab/zagros-1.0-quick
Modernbert fix
Author: remi-or
Pull request: yes

There is a bug in `modernbert` where using FA2 changes the shapes of the hidden states when compared to `eager`. 
This fixes it in the base model, and since it was already compensated for `ModernBertForMaskedLM` we cancel out the change there.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41056). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I now textually assure you that the test passes :smile: but you can check the changes made
I could not find one instance of the `ModernBertForMultipleChoice` model on the hub safe for the one pushed during the addition PR (no likes on it tho)

run-slow: modernbert

This comment contains run-slow, running the specified jobs: 

models: ['models/modernbert']
quantizations: [] ...

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: modernbert

@remi-or lets go dawg
CI Runners - move amd runners mi355 and 325 to runner group
Author: glegendre01
Pull request: yes

Move CI runners 325/355 from scale set targetting to runner group

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41193). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[XPU] Add MXFP4 support for XPU
Author: YangKai0616
Pull request: yes

# What does this PR do?

Add **MXFP4** quantization support for **XPU** to support loading **GPT-OSS** models in MXFP4 format on XPU.

1. Add MXFP4 support for XPU
2. Improve test file `tests/quantization/mxfp4/test_mxfp4.py` to support XPU testing


Comments:
It seems the CI error is due to an excessively high request frequency. 
@ydshieh , could you pls help review this PR? Thx!

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: mxfp4

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41117). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
llama3-8b
Author: 52566rz
Pull request: no
Labels: name: bug

### System Info

When running 
Setting `pad_token_id` to `eos_token_id`:None for open-end generation.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

When running 
Setting `pad_token_id` to `eos_token_id`:None for open-end generation.

### Expected behavior

When running 
Setting `pad_token_id` to `eos_token_id`:None for open-end generation.

Comments:
Hi! Could you please provide more details about the bug you’re encountering with llama3-8b?
There is no license in the readme
Author: vinitjain2005
Pull request: no

I will add the license in the readme file. I am a hacktoberfest contributer please assign these task to me.

![Image](https://github.com/user-attachments/assets/5f848308-9d78-41d6-b5ba-f2588d279cd5)

Comments:
The licence is contained in the `LICENSE` file in the repo root! `README.MD` also has a license at the top (in a markdown comment, so it doesn't render when viewed). I don't think we need anything else!
More typing fixes
Author: cyyever
Pull request: yes

# What does this PR do?

This PR adds types to parameters and remove unnecessary `noqa`.



Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, csm, dia, diffllama, fuyu, gemma, gemma2, gemma3, gemma3n, got_ocr2, llama4, longcat_flash, longt5, mistral3, mllama, nougat

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41102). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[v5] Remove `model_parallel` deprecated feature 
Author: SunMarc
Pull request: yes

# What does this PR do?
This PR removes deprecates code related to `model_parallel` feature in the modeling as well as in Trainer. This was a feature that was added only for a few models like t5 and gpt2 but eventually we switched to using device_map and tp now. 
For the models, we removes `is_parallelizable` and `model_parallel` attributes and some deprecated methods.
For Trainer, we remove a small section related to `model_parallel`. 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41166). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bloom, camembert, codegen, decision_transformer, falcon_mamba, gpt2, gpt_neo, gpt_neox_japanese, gptj, imagegpt, kosmos2_5, lilt, luke, mamba, modernbert_decoder, moshi
cache offloading check is incorrect
Author: mobicham
Pull request: no
Labels: name: bug

<a href="https://github.com/huggingface/transformers/blob/a579de7f5e00a9fdb1e9828aa3ab78385959f231/src/transformers/cache_utils.py#L766">This check</a> creates some issues with torch.compile. The type hint is <a href="https://github.com/huggingface/transformers/blob/a579de7f5e00a9fdb1e9828aa3ab78385959f231/src/transformers/cache_utils.py#L685">bool</a>, but in some cases, that offloading value is actually a cuda device. 


### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

For example, with Qwen3-omni, if you do this, offloading is a cuda device, which triggers that offloading check, which crashes torch.compile:
```Python
from transformers import StaticCache
past_key_values = StaticCache(model.thinker.config, max_model_len, device, compute_dtype)
print(past_key_values.offloading)
# `cuda:0`- should be False!
```

### Expected behavior

Static cache should not crash with torch.compile.

Comments:
cc @cyrilvallez who wrote that block, also see the PR at #41169!

You are passing a device directly as `offloading` argument, that's why! See the docstring of the class in your IDE, or the signature [here](https://github.com/huggingface/transformers/blob/main/src/transformers/cache_utils.py#L1024-L1071)!

My bad! I copy pasted from some old code when the offloading option was not available! Thanks!
`train_tokens_per_second` is wrong after continuing from checkpoint
Author: AmitMY
Pull request: no
Labels: name: Good First Issue

name: bug

### System Info

- `transformers` version: 4.55.4
- Platform: macOS-15.6.1-arm64-arm-64bit
- Python version: 3.12.2
- Huggingface_hub version: 0.34.3
- Safetensors version: 0.5.3
- Accelerate version: 1.9.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.7.1 (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

 @zach-huggingface @SunMarc

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When continuing `run_clm.py` from a checkpoint, it starts with insane tokens per second numbers, then slowly, eventually, goes down. 

<img width="359" height="284" alt="Image" src="https://github.com/user-attachments/assets/99a4bf33-b061-4475-80ed-4a5ac89ed098" />

<img width="235" height="47" alt="Image" src="https://github.com/user-attachments/assets/8778dc76-7a14-4283-9a7b-9851fdf28bf7" /> c

### Expected behavior

It should be calculated with the local training run information, not the global one. The numbers should roughly be the same since it is the same exact setup.

Comments:
@ArthurZucker @SunMarc is this an issue ?


Yep probably!

Okk can you assign this to me plz @ArthurZucker ?


Hey! We never assign issue @ArkVex just feel free to open a PR and link it here!
Bump hfh prerelease version
Author: Wauplin
Pull request: yes

From `1.0.0.rc1` to `1.0.0.rc2` (minor fix in `@strict` dataclass so shouldn't change anything).

Comments:
@ArthurZucker @LysandreJik very minor bump. Can it be fast-tracked / merged soon? (blocking diffusers CI because of dependency table check).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41175). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ok, it's good
Attention sinks are not applied correctly in `integrations.flex_attention`
Author: jonny-so
Pull request: no

The `score_mod` function passed to `flex_attention` should operate on the pre-softmax attention scores, but the snippet below appears to be applying the attention biases (`s_aux`) and computing the **_post_**-softmax scores. 

https://github.com/huggingface/transformers/blob/67097bf34055c55b886dc92014fd628c9a70e168/src/transformers/integrations/flex_attention.py#L275-L280

I don't think it is possible to apply (gpt-oss-style) attention sinks using the `score_mod` alone, but you can do it by passing `return_lse=True` to `flex_attention` and renormalising using the extra return value. If someone can point me to where unit tests for this code should live I'm happy to PR a fix.

Comments:
cc @cyrilvallez

Hey guys 🤗
I had some bandwidth and decided to give a try at solving this issue here: https://github.com/huggingface/transformers/pull/41083. I did not find any unit tests related to this and were not sure where to add any so I quickly crafted a small script to try out the old vs. new version and to prove that the edits should fix the issue.
@jonny-so and @Cyrilvallez could you please review when you get the time ? I would great appreciate both of your reviews!
Thanks!