noamaanMulla-03/transformers-issues huggingface dataset:
[qwen2-vl] fix vision attention scaling
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, after the refactor scaling was accidentally changed from `1/math.sqrt(head_dim)` to `math.sqrt(head_dim)`


Comments:
run-slow: qwen2_vl,qwen2_5_vl,qwen2_5_omni

This comment contains run-slow, running the specified jobs: 

models: ['models/qwen2_5_omni', 'models/qwen2_5_vl', 'models/qwen2_vl']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39043). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Ah, @zucchini-nlp , there is also 

> tests/models/colqwen2/test_modeling_colqwen2.py::ColQwen2ModelIntegrationTest::test_model_integration_test

which is failing due to #38930. But also failing with this PR.



previously at this block

```
        # Check if the maximum scores per row are in the diagonal of the matrix score
        self.assertTrue((scores.argmax(axis=1) == torch.arange(len(ds), device=scores.device)).all())
```
in `tests/models/colqwen2/test_modeling_colqwen2.py`

we have

```
tensor([[7.0820, 6.6836, 7.5547],
        [8.1797, 9.3516, 8.0312],
        [7.6641, 8.3359, 8.9922]], dtype=torch.float16)
```
but after #38930

```
tensor([[15.0703,  8.7422, 15.0312],
        [ 9.5078, 16.8906, 10.6250],
        [15.6484, 12.3984, 20.4688]], dtype=torch.float16)

```

@ydshieh the ColQwen test is failing for me even before the Qwen2-VL vision refactor, maybe it started failing due to other PR?

> @ydshieh the ColQwen test is failing for me even before the Qwen2-VL vision refactor, maybe it started failing due to other PR?

I will check again (inside CI runner, it's that PR causing problem. Do you check locally or inside a SSH runner?)

Only locally for now

hmm, it is passing now on main, and even on the commit of this merged PR.

Either I made some mistake when checking back then or something strange there. Sorry to bother.

We are ✅ 

when I say pass, I mean on A10
docs: Gemma 3n audio encoder
Author: RyanMullins
Pull request: yes

# What does this PR do?

Updating Gemma 3n docs and docstrings to clarify the relationship between the newly trained audio encoder used in Gemma 3n and the USM model from the original paper.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface, @SunMarc and @qgallouedec
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @paruby
All CI jobs with A10
Author: ydshieh
Pull request: yes

# What does this PR do?

I updated

https://huggingface.co/datasets/hf-internal-testing/transformers_daily_ci/commit/687b472ebd38c936c34205c0fda34e27bb7fca9a

so all model jobs are using A10 now. Let's also change everything to A10 ( CI triggered by comment, the job checking new failures etc.)

Probably to remove the `runner_map` stuff and its usage, but let's keep it for a while (maybe it would be useful at some point)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39119). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[chat] Split chat/serve (built on top of lysandre's PR)
Author: gante
Pull request: yes

# What does this PR do?

Adds the following to #38443

- Streaming tool calls
- Persistent KV cache across server calls
- BC `chat` generation parameterization
- docs

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39031). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Support Asynchronous Evaluation on Separate GPU in `Trainer`
Author: AmitMY
Pull request: no
Labels: name: Feature request

### Feature request

Add support for asynchronous evaluation in `transformers.Trainer`, ideally enabling evaluation to run in parallel to training — potentially on a separate GPU — without blocking the training loop.

Even ideally, to really utilize the hardware fully, one GPU should be training, the other one should be evaluating, constantly. That means, that it might take a few minutes to evaluate, and maybe save the checkpoint to disk, but afterwards, a new eval should start immediately with the latest model weights, in an infinite loop.

### Motivation

In my training scenarios, especially with:
**slow checkpointing (e.g. network-mounted disk, NFS, HDD)** and **multi-GPU machines with underutilized resources**
the current blocking evaluation step can create a significant bottleneck. It halts the training loop and prevents full utilization of the hardware.

<img width="562" alt="Image" src="https://github.com/user-attachments/assets/426629b0-3cbe-446c-95bd-26324d2620dc" />

I do not want to try to patch this on top:

* Forking Trainer manually and running evaluation with a subprocess
* Spawning external watchdog scripts for async eval
* Deep copying the model to avoid disk writes

This seems like a natural fit for `Trainer`, similar to how `save_strategy` and `log_strategy` are already modular.

### Your contribution

Some ideas (not prescriptive):

* `eval_async=True`: runs evaluation in a forked process/thread
* `eval_device='cuda:1'`: specify a device for async eval
* `eval_strategy="async_steps"`: triggers parallel eval on step intervals
* Provide a callback hook or scheduler for async checkpoint evaluation

Implementation Ideas

* Use `torch.multiprocessing` or `concurrent.futures.ProcessPoolExecutor` to fork evaluation subprocess
* Snapshot the model state (via `state_dict()` or full checkpoint) and transfer it (memory, pipe, or fast serialization)
* Respect user-specified `eval_steps`, `metric_for_best_model`, and `load_best_model_at_end` behavior

Comments:
cc @SunMarc 

@SunMarc Hey. I have been looking into this request and have made some progress. Just wanted to ask: 'Can i work on this issue?'


*Another idea would be to support `eval_strategy="async_available"` which would run the evaluation continuously, as often as possible, on the other GPU, regardless of 1000 or 5000 steps passing.  
But `async_steps` would already be a great improvement.
docs: correct two typos in awesome-transformers.md
Author: VladimirGutuev
Pull request: yes

# What does this PR do?
https://github.com/huggingface/transformers/blob/main/awesome-transformers.md?plain=1
This PR fixes two small typos in the **[awesome-transformers.md](https://github.com/huggingface/transformers/blob/main/awesome-transformers.md?plain=1)** list:

- **DALL·E Flow** section: changes  
  `Itt leverages DALL·E-Mega…` → `It leverages DALL·E-Mega…`  
- **Underthesea** section: changes  
  `We provides extremely easy API…` → `We provide an extremely easy API…`

Fixes #39101

## Before submitting

- [x] This PR fixes a typo or improves the docs  
- [x] I’ve read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request)  
- [ ] Tests are not affected (no code changes)  
- [ ] I’ve updated the documentation with my changes (N/A – markdown only)  
- [ ] I’ve added necessary tests (N/A)

## Who can review?
Documentation maintainers, feel free to take a look: @stevhliu  

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39102). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
docs: fix typos in awesome-transformers.md WIP
Author: VladimirGutuev
Pull request: no
Labels: name: bug

### System Info

Hello! 
There are two small typos in [awesome-transformers.md](https://github.com/huggingface/transformers/blob/main/awesome-transformers.md?plain=1#L289-L294):
1. DALL·E Flow section (**TYPO**)
https://github.com/huggingface/transformers/blob/ccf2ca162e33f381e454cdb74bf4b41a51ab976d/awesome-transformers.md?plain=1#L289-L294
2. Underthesea section (**GRAMMAR**)
https://github.com/huggingface/transformers/blob/ccf2ca162e33f381e454cdb74bf4b41a51ab976d/awesome-transformers.md?plain=1#L527-L531

### Who can help?

@stevhliu 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

## Reproduction

Steps to reproduce the typo in documentation:

1. Open the file `docs/source/awesome-transformers.md`.  
2. Scroll down to the **DALL·E Flow** section.  
3. Observe the sentence:
   > “DALL·E Flow is an interactive workflow … **Itt leverages** DALL·E-Mega…”  
4. Scroll further to the **Underthesea** section.  
5. Observe the sentence:
   > “… **We provides** extremely easy API to quickly apply pretrained NLP models…”  

### Expected behavior

## Expected behavior

The text should read with correct grammar:

- In **DALL·E Flow**:  
  > “… **It leverages** DALL·E-Mega…”

- In **Underthesea**:  
  > “… **We provide an extremely easy API** to quickly apply pretrained NLP models…” 

Comments:
Update BigBirdPegasus model card
Author: dross20
Pull request: yes

# What does this PR do?
This PR replaces the BigBirdPegasus model card with a new model card matching the format introduced in #36979.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

@stevhliu 

## Notes
- Omitted `attn_implementation="sdpa"` since `BigBirdPegasusForConditionalGeneration` doesn't support SDPA.

Comments:
Suggested changes have been applied! @stevhliu 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39104). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix to make vllm happy
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Delete the faulty check on multimodal tokens

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38530). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Whisper] update token timestamps tests
Author: eustlb
Pull request: yes

# What does this PR do?

Includes 3 fixes for token timestamps for Whisper:
1. nit due to `assertEqual` →  `torch.testing.assert_close`
2. update values for A10
3. unhandled edge case (see added comments in `generation_whisper.py`)


Comments:
you are on A10, it's good. But if you haven't trigger the whole whisper test suite, it's good to use `run-slow: whisper` 😃 to make sure.



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39126). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: whisper

This comment contains run-slow, running the specified jobs: 

models: ['models/whisper']
quantizations: [] ...

> test_whisper_longform_multi_batch_hard_prev_cond

it is already failing (not related to your work)

> test_small_longform_timestamps_generation 

it was passing (last Friday) but it was on T4, and now all jobs are using A10.

Maybe it's a good time for us to update values  those 2 tests in this PR (assuming the new outputs still make sense? Would you be up for it ?

Yep, taking care of it!

run-slow: whisper

This comment contains run-slow, running the specified jobs: 

models: ['models/whisper']
quantizations: [] ...

run-slow: whisper

This comment contains run-slow, running the specified jobs: 

models: ['models/whisper']
quantizations: [] ...

run-slow: whisper

This comment contains run-slow, running the specified jobs: 

models: ['models/whisper']
quantizations: [] ...

looks like we need a seed for 

> tests/models/whisper/test_modeling_whisper.py::WhisperModelIntegrationTests::test_whisper_longform_multi_batch_hard_prev_cond

Now all pass deterministically. Merge, thank you for working on this 🙏
🚨 Don't use cache in non-generative models
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

When working on https://github.com/huggingface/transformers/pull/38635, I found that there are some models which have `past_key_values` in their signature, even though they cannot generate. The reason is that models were all copying from Bert

This PR clean it up and changes the copy statement to Align model and adds support for new attention API in all those models

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38751). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yes, great cleanup! Ping me whenever it's ready and you want a review

@Rocketknight1 ready for review! One thing to note: I didn't deprecate `past_key_values` from kwargs and simply deleted, since it wasn't used anyway. Do you think we need a deprecation cycle or raise an error when cache related kwargs are passed?

@zucchini-nlp I think it's okay! I really hope people weren't passing `past_key_values` to non-generative models anyway :grimacing:

run-slow: align,wav2vec2,layoutlm,clap

run-slow: align,wav2vec2,layoutlm,clap

This comment contains run-slow, running the specified jobs: 

models: ['models/align', 'models/clap', 'models/layoutlm', 'models/wav2vec2']
quantizations: [] ...

This comment contains run-slow, running the specified jobs: 

models: ['models/align', 'models/clap', 'models/layoutlm', 'models/wav2vec2']
quantizations: [] ...

hey @Cyrilvallez do you have any comments to address? 

Yeah, I have the same question. On one side the code path should have never been used and isn't propagated from Base/Task models. On the other side, we never know if users found a way to exploit it by loading specific layers and re-using them

I can add proper deprecation if we think this is too aggressive and remove everything in the next 2-3 releases. A bunch of unused code path is just 😖 

Very nice! In the era of unbloating, let's remove as much as we can, and redirect users with on the hub code for the ones that still need this? 
We can keep this for 1 release tho! 

@ArthurZucker @Cyrilvallez added `deprecate_kwarg` until the next v4.54 release in all forward calls. The failing test is not related
switch default xpu tp backend to pytorch built-in XCCL from pytorch 2.8
Author: yao-matrix
Pull request: yes

@SunMarc @stevhliu , pls help review, thx very much.

Comments:
thx all, i've solved the conflicts, @stevhliu , could you pls help merge? Thx very much.

Thanks, I'll merge for you once #39068 is complete!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39024). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Actually sorry @yao-matrix, can you rebase your branch please to get the latest fix on `main`?

> Actually sorry @yao-matrix, can you rebase your branch please to get the latest fix on `main`?

done, thx @stevhliu
LlamaAttention forward function type hint is incorrect from new Branch 
Author: ArkVex
Pull request: yes

Hi, this PR fixes a small issue in the LlamaAttention class. The return type in the forward method currently shows three values, but the function actually returns only two. This seems to have been missed during the attention refactor (possibly in PR https://github.com/huggingface/transformers/pull/35235).

I’ve updated the type hint to reflect the actual return values, just to avoid confusion for anyone reading or using the code. Let me know if any other changes are needed. Happy to help!

Comments:
### @Rocketknight1 trying again


I think I shall close the fork are refork it again

Hey @ArkVex, thanks for fixing typehints! To fix repo-cosistency issue can you please run the following command and commit the changes?

```
python utils/check_modular_conversion.py --fix_and_overwrite
```

@qubvel still throwing the same error


Idk what is wrong in this...i did such minute change and it is troubling so much


@ArkVex, I hope you don't mind, I've applied the command above and pushed the changes

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38998). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[VLMs] support passing embeds along with pixels
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title + some clean up on generation tests

Didn't expect this PR to grow so large. Now all vision LLMs can accept `inputs_embeds` as an input along with `pixel_values`. The tests are all passing on my end

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38467). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Several fixes for Gemma3n
Author: Cyrilvallez
Pull request: yes
Labels: name: for patch

# What does this PR do?



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39135). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: gemma3n

This comment contains run-slow, running the specified jobs: 

models: ['models/gemma3n']
quantizations: [] ...

All good, @ydshieh for the slow tests! (IntegrationTests are still skipped, I will check them soon)

Do you guys know why the training loss is exceptionally high? I don't think it's due to the gradient accumulation - it does quickly decrease, but it's very weird

👀 no idea! 

Wait I mis-spoke - grad accumulation does in fact not work correctly.

But the losses are still suspiciously high
`LayoutLMv3TokenizerFast` doesn't pass all the params.
Author: sergiopaniego
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.54.0.dev0
- Platform: Linux-6.1.140-154.222.amzn2023.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.17
- Huggingface_hub version: 0.33.1
- Safetensors version: 0.5.3
- Accelerate version: 1.8.1
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.7.1+cu126 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: Tesla T4

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

This issue was reported via Discord [here](https://discord.com/channels/879548962464493619/1019883044724822016/threads/1389535268494901379). 

The code to reproduce is the following:

```python
 processor = LayoutLMv3Processor.from_pretrained("microsoft/layoutlmv3-base")

def preprocess(example):
    encoding = processor(
        image,
        example["words"],
        is_split_into_words=True,
        boxes=normalized_bboxes,
        word_labels=[label2id[l] for l in example["labels"]],
        truncation=True,
        padding="max_length",
        return_tensors="pt"
    )

tokenized_dataset = dataset.map(preprocess, remove_columns=dataset.column_names)
```



### Expected behavior

The previous code instantiates the fast tokenizer and generates the following error:

```bash
TypeError: LayoutLMv3TokenizerFast._batch_encode_plus() got an unexpected keyword argument 'is_split_into_words'
```

If using `use_fast=False` when creating, the issue doesn't appear. Looking into the code, it seems like there is a missing `**kwargs,` [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/layoutlmv3/tokenization_layoutlmv3_fast.py#L516) when using the fast tokenizer. The problem is also present in `tokenization_layoutlmv2_fast`. 

I've already managed to solve it and will create the PR shortly :)

Comments:
Closing as I found the answer here:
`            is_pretokenized=True,  # we set this to True as LayoutLMv3 always expects pretokenized inputs`
https://github.com/huggingface/transformers/blob/main/src/transformers/models/layoutlmv3/tokenization_layoutlmv3_fast.py#L537
Fix missing fsdp & trainer jobs in daily CI
Author: ydshieh
Pull request: yes

# What does this PR do?

It's missing, as one can see in 

https://huggingface.slack.com/archives/C06SZSGL2AF/p1750393112155819

It's caused by

Switch to use A10 progressively (#38936)

where we add an extra variable `runner_map`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39153). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.