noamaanMulla-03/transformers-issues huggingface dataset:
Update SAM/SAM HQ attention implementation + fix Cuda sync issues
Author: yonigozlan
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Update SAM and SAM HQ attention implementation, and fix two lines that were causing unnecessary cuda sync when profiling with torch profiler.
Mostly split from https://github.com/huggingface/transformers/pull/32317 as PR is becoming huge

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39386). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: sam, sam_hq

Thank you @vasqu ! Made the changes. Should be ready for a final review @Cyrilvallez :)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: sam, sam_hq
Fix placeholders replacement logic in auto_docstring
Author: yonigozlan
Pull request: yes

# What does this PR do?

The placeholders logic in auto_docstring seems to be broken currently, so that placeholders like {image_processor_class} are not replaced properly.
This PR simplifies greatly the logic and fixes the issue

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39433). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add muon and flash-muon optimizer
Author: kadirnar
Pull request: no
Labels: name: Feature request

### Feature request

Muon: https://github.com/KellerJordan/Muon
Flash-Muon: https://github.com/nil0x9/flash-muon
Paper: https://arxiv.org/pdf/2502.16982

### Motivation

An effective optimizer method to further accelerate LLM training. The Muon team has recently proven the importance of the muon optimizer in the LLM models they released.

<img width="2536" height="1016" alt="Image" src="https://github.com/user-attachments/assets/93ccc436-f66b-4970-be55-074e5a36fa3d" />

### Your contribution

I want to add the transformers library to this optimization. 

Comments:
DTensor issues when running Llama4ForConditionalGeneration with tensor parallel.
Author: czkkkkkk
Pull request: no
Labels: name: bug

### System Info

transformers version: 4.52.4
pytorch version: 2.6

### Who can help?

transformers version: 4.52.4
pytorch version: 2.6

When running Llama4 with tensor parallel, [torch.nn.Unfold used in llama4 ](https://github.com/huggingface/transformers/blob/v4.52.4/src/transformers/models/llama4/modeling_llama4.py#L1320) isn't compatible with DTensor. So I got this error:
`NotImplementedError: Operator aten.im2col.default does not have a sharding strategy registered.` Looks like it is because the latest transformers use replicate DTensor for layers without tp_plan but `Unfold` isn't compatible with DTensor.

To workaround this error, I manually changed the input tensor to regular Tensor.
```
device_mesh = hidden_states.device_mesh if isinstance(hidden_states, DTensor) else None
placements = hidden_states.placements if isinstance(hidden_states, DTensor) else None
hidden_states = hidden_states.to_local()
hidden_states = self.unfold(hidden_states)
hidden_states = DTensor.from_local(hidden_states, device_mesh, placements)
```

After the change, I got `AttributeError: 'BaseModelOutput' object has no attribute 'to_local'` when running `vision_model` https://github.com/huggingface/transformers/blob/v4.52.4/src/transformers/models/llama4/modeling_llama4.py#L1543.


### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Minimal script to reproduce the error
```py
# test.py
import torch
from transformers.models.llama4.modeling_llama4 import Llama4ForConditionalGeneration

if __name__ == '__main__':
    model_id = "meta-llama/Llama-4-Scout-17B-16E-Instruct"
    model = Llama4ForConditionalGeneration.from_pretrained(
        model_id,
        torch_dtype=torch.bfloat16,
        low_cpu_mem_usage=True,
        device_map="auto",
    )
    B = 1
    S = 128
    input_ids = torch.randint(0, 1000, (B, S))
    attention_mask = torch.ones((B, S))
    pixel_values = torch.randn((5, 3, 336, 336)).to(torch.bfloat16)
    model(input_ids=input_ids, attention_mask=attention_mask, pixel_values=pixel_values)
```

```sh
torchrun --nproc-per-node=8 test.py
```

### Expected behavior

Expect the program to finish successfully.

Comments:
Hi @czkkkkkk, Are you already working on this? If not, I'd be happy to take it.

Hi @marcndo, I am not working on it. Would be great if you can help resolve this issue. Thanks!

`Unfold` not being compatible with `DTensor` is something we should definitely be wary of cc @cyrilvallez

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[gemma3] support sequence classification task
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, we can't copy from llama or any other LLM because Gemma3 needs to obtain `text_config` params and needs to pass extra vision kwargs in `forward`. Thus the code was adapted from llama and the tests are green

Fixes https://github.com/huggingface/transformers/issues/36755

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39465). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, gemma3

Nope, we usually don't add when there's no official pretrained checkpoint
refactor: remove `set_tracer_provider` and `set_meter_provider` calls
Author: McPatate
Pull request: yes
Labels: name: for patch

# What does this PR do?

As we are a library, we shouldn't set an OT tracer or a meter, given you can only set one per application.

Removing calls to `set_{tracer|meter}_provider`.

Closes #39115, #39143 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39422). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

this needs to be released as a patch release

Yes sorry doing this today
Causal mask is not compatible with Qwen2-VL when using padding-free training
Author: hiyouga
Pull request: no
Labels: name: bug

### System Info

transformers @ 4.53.2

### Who can help?

@amyeroberts @zucchini-nlp 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

https://github.com/huggingface/transformers/blob/903944a411c35b8f7b2b51ada77a7c2a80e7fb88/src/transformers/masking_utils.py#L708-L713

Qwen2-VL's position ids has a shape of `[3, batch_batch, seq_len]`, while the current code cannot process the 3D position ids.

https://github.com/hiyouga/EasyR1/issues/413

### Expected behavior

There should be a special case for Qwen2-VL's 3D MRope

Comments:
+1, we have a PR that should stop passing 3D positions to the mask creating function in https://github.com/huggingface/transformers/pull/39282. Ideally we should check if pad-free FA2 works correctly when "text-only" position ids are passed, and if not fix it or explicitly skip pad-free FA2

I have it in my TODO list, unless the contributor from above PR is willing to work on it. If you have bandwidth, feel free to check it out :)

Agreed, thanks for your prompt response @zucchini-nlp 

I think the padding-free training for Qwen2-VL is not work properly in transformers, we have created a patch to make the padding-free training available for Qwen2-VL models


Some important changes like:

https://github.com/hiyouga/EasyR1/blob/52ba694ab5185a967d64801681961501468e6ed8/verl/models/transformers/qwen2_vl.py#L192

https://github.com/hiyouga/EasyR1/blob/52ba694ab5185a967d64801681961501468e6ed8/verl/models/transformers/flash_attention_utils.py#L57

cc @Kuangdd01
`transformers.utils.metrics` sets global `TracerProvider`
Author: harupy
Pull request: no

## Reproduction

```python
import opentelemetry.sdk.trace as trace_sdk
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, SpanExporter
from opentelemetry import trace

import transformers.utils.metrics
# This calls `trace.set_tracer_provider` 

provider = trace_sdk.TracerProvider()
processor = SimpleSpanProcessor(SpanExporter())
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
# This does nothing and emits a warning:
# Overriding of current TracerProvider is not allowed
```

Since `import transformers.utils.metrics` calls `trace.set_tracer_provider`, `trace.set_tracer_provider` in my script does nothing.

## Version

transformers >= 4.53.0

## Related files/lines/PRs

- https://github.com/huggingface/transformers/blob/ccf2ca162e33f381e454cdb74bf4b41a51ab976d/src/transformers/utils/metrics.py#L42
- https://github.com/huggingface/transformers/pull/38085

Comments:
This will break people's observability stacks @ArthurZucker 

Best practice for OpenTelemetry in libraries is to use the API to obtain tracers/meters, but leave all provider configuration to the application. This ensures users can set up telemetry as they need, and prevents conflicts.

Could you please update the library to:

- Remove the calls to `set_tracer_provider` and `set_meter_provider`

- Use `get_tracer` and `get_meter` to obtain tracers/meters from `from opentelemetry import metrics, trace`

This will make the library much more interoperable with users’ own observability stacks!

Thank you!



Yes, cc @McPatate if you can have. look!

Proof for @kosmalapg 's point: using `transformer` (the latest version) together with `logfire` will give out the infamous `Overriding of current TracerProvider is not allowed` warning.

Hi @ArthurZucker , I was wondering if you had an ETA for the next patch? I am pretty keen to get this fix released please 🙏 

Will do today very sorry
Fix the check in flex test
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. Otherwise it passes for some models where it should not, and it complains later.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39548). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: colpali, colqwen2, got_ocr2
Rename `_supports_flash_attn_2` in examples and tests
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, forgot to rename the attribute in tests

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39471). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, bamba, bark, clip, esm, kyutai_speech_to_text, modernbert_decoder, musicgen, musicgen_melody, siglip, siglip2, videomae, vit_mae, whisper
Enabling `average_tokens_across_devices` by default in Trainer
Author: MilkClouds
Pull request: no

Originally, https://github.com/huggingface/transformers/issues/34242 pointed out that GA loss calculation was wrong for device sum. So the fix https://github.com/huggingface/transformers/pull/34373 was merged into main but this feature is disabled by default, defaulting `average_tokens_across_devices` to False.

I think advantage of this PR is reproducibility/correctness and disadvantage is None, so there's no reason to disable it by default. (As long as the implementation of `average_tokens_across_devices` is correct!)

We may think about the case where this feature is strictly needed:
- assume in GPU 0, label was [1,-100,-100,....-100]
- assume in GPU 1, label was [1,1,1,...,1]

Then since the loss calculation treats two GPU as same weight, unstable(since it's from only single item) loss signal from GPU 0 will dominate(50%) much more than we needed.

Comments:
🚨🚨🚨 [Trainer] Enable `average_tokens_across_devices` by default in `TrainingArguments`
Author: Krish0909
Pull request: yes

Fixes #39392

This change improves loss calculation correctness for multi-GPU training by enabling proper token averaging across devices by default.

## What does this PR do?

Changes the default value of `average_tokens_across_devices` from `False` to `True` in `TrainingArguments`. This ensures more accurate loss calculation in multi-GPU training scenarios by properly averaging tokens across devices.

As noted in the original issue, this feature provides reproducibility and correctness benefits with no downsides, so there's no reason to keep it disabled by default.

## Before submitting
- [x] Was this discussed/approved via a Github issue? Yes - #39392
- [x] Did you make sure to update the documentation with your changes? Yes - updated docstring

## Who can review?
@zach-huggingface @SunMarc @qgallouedec (trainer maintainers)

Comments:
@qgallouedec, did you face any issues after changing this default in trl ? Happy to do it transformers otherwise 

@SunMarc Hey! Happy to help test this on the trl side as well if needed. Let me know if you’d like me to run any specific checks!

No. No issue that I'm aware of

Hey @SunMarc , I noticed the tests are failing on ci/circleci: tests_torch and run_tests, while the rest of the checks seem fine.
Given the scope of this PR is limited to changing the default argument with no functional impact beyond configuration, should I investigate and fix these failures or are these known unrelated issues and safe to merge as-is?
Also, the workflows are awaiting approval — could you please approve them if a full CI run is required?
Thanks!

> Given the scope of this PR is limited to changing the default argument with no functional impact beyond configuration, should I investigate and fix these failures or are these known unrelated issues and safe to merge as-is?
> Also, the workflows are awaiting approval — could you please approve them if a full CI run is required?
> Thanks!

No it's fine, i'll take care of merging this PR 

> > Given the scope of this PR is limited to changing the default argument with no functional impact beyond configuration, should I investigate and fix these failures or are these known unrelated issues and safe to merge as-is?
> 
> > Also, the workflows are awaiting approval — could you please approve them if a full CI run is required?
> 
> > Thanks!
> 
> 
> 
> No it's fine, i'll take care of merging this PR 

Thanks! 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39395). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Raise `TypeError` instead of ValueError for invalid types
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Fixes raising `TypeError` instead of ValueError when encountering invalid types

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ydshieh @Rocketknight1 

Comments:
Hi @Sai-Suraj-27 this makes sense, however you probably need to:

1) Rebase your PR to resolve the conflicts
2) `pip install -e .[quality]` followed by `make fixup` or `make style` to resolve code style issues!
3) Look at the CI failures - some tests actually check that the expected error is raised, so you'd need to change those tests from  `ValueError` to `TypeError`

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aria, clip, gemma3, grounding_dino, llava, llava_next, llava_next_video, llava_onevision, mllama, myt5, owlv2, owlvit, phi4_multimodal, phimoe, pixtral, qwen2_5_omni

> Hi @Sai-Suraj-27 this makes sense, however you probably need to:
> 
>     1. Rebase your PR to resolve the conflicts
> 
>     2. `pip install -e .[quality]` followed by `make fixup` or `make style` to resolve code style issues!
> 
>     3. Look at the CI failures - some tests actually check that the expected error is raised, so you'd need to change those tests from  `ValueError` to `TypeError`

Hi, @Rocketknight1. Thanks for the review. Made the required fixes, CI is green now. Sorry for the delayed response.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38660). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix Docstring of BarkProcessor
Author: st81
Pull request: yes

# What does this PR do?

This PR corrects the return type hint in the docstring of `BarkProcessor.__call__`. The current docstring incorrectly stated that the method returns a `Tuple(BatchEncoding, BatchFeature)`, but it actually returns a `BatchEncoding` object.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

- speech models: @eustlb
- Documentation: @stevhliu

Comments:
thank you @st81 for correcting this! (I'm jumping in as I'm working with @eustlb on speech models.)

Could you also a return type hint like so?
```python
def __call__(
        self,
        text=None,
        voice_preset=None,
        return_tensors="pt",
        max_length=256,
        add_special_tokens=False,
        return_attention_mask=True,
        return_token_type_ids=False,
        **kwargs,
 ) -> BatchEncoding:       # Adding this part
```
You'll also have to add the following import:
```python
from ...tokenization_utils_base import BatchEncoding
```

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bark

@ebezzam 

Thank you for your feedback and suggestions! I've added the commit to apply your suggestion.

Thank you, LGTM 👍 

cc @Rocketknight1 can you merge? (or additional thoughts?) I confirm that this PR corrects the docstring on what is the actual `BarkProcessor.__call__` output.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39546). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[WIP] try to relax the tie_weights method
Author: molbap
Pull request: yes

# What does this PR do?
as per title, trials to rework parts of inits and what is done at weight tying. Might be superseded by #39339 


Comments:
Fyi had to change something there in #39228 for TP support on tied weights, so might be good to keep an eye there as well 

Gotcha @vasqu , took a look, seems compatible IMO! This PR is mostly to run more easily the test suite around weights tying :laughing:  

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39555). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Fast image processor] refactor fast image processor glm4v
Author: yonigozlan
Pull request: yes

# What does this PR do?

Refactor fast image processor of glm4v to be consistent with the library standards, and use grouping by shape

cc @Cyrilvallez 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39490). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: glm4v
Provide clearer instructions on how to specify target language.
Author: khof312
Pull request: yes

In the current documentation, I believe there is an error in the sample code. While the language token of the input is specified as `<2pt>` (Portuguese), the returned translation is in Italian, so actually the language should be `<2it>`.

I didn't initially understand that this was a token for the target language, so I have added an explanation of this to the documentation as well as instructions on how to get the available languages based on the [code](https://huggingface.co/spaces/jbochi/madlad400-3b-mt/blob/main/app.py) in Juarez Bochi's demo space.

It would be good to check that I have retrieved the correct range of tokens (4-459) and not omitted any (there were some additional tokens that looked to refer to scripts instead of languages, but I was not sure).

If these changes are approved, then they should also be perpetuated to the documentation of the different model checkpoints from google and jbochi.


Comments:
Hi @khof312, "Eu amo pizza!" is Portuguese, not Italian. The reference to pizza is deceptive, I realize, but in Italian this would be "Amo la pizza!", or maybe "Io amo la pizza!".

Oops, you are right about the language, my mistake. In this case the <2pt> language code (line 56) need not be changed, but is it still worth adding documentation of how to find the respective language codes?

(I have changed the associated file accordingly to fix my error).

cc @stevhliu for the docs question!