noamaanMulla-03/transformers-issues huggingface dataset:
Add prefix sharing to continuous batching
Author: remi-or
Pull request: yes

This PR adds a prefix sharing mechanism to the continuous batching API like the one present in VLLM.
It only activates if the model is a full-attention model, as is the case in VLLM.

The mechanism has two main components:
- block hashing: each block in the cache, once it is filled up, is given a hash that depends on all the tokens in the sequence up to and including the ones in the block
- prefix detection: when starting prefill for a request, we first look for a prefix with KV cache already computed, and if such a prefix is found, we skip the KV computation for it, using references to completed blocks to save compute
- block de-reference: when a block is given a hash, we check that no other block shares the same hash. This ensures that each block sharing the same information is unique, and helps keep the cache size in control

What is missing from this PR: 
- [x] more documentations
- [x] checking the code again
- [x] edge case: if the prefix is the entire initial request, we still need to do a forward with the last token of the request
- [x] checks TODOs to adress: the ones left are for a PR down the line

The PR is draft until these are resolved but any early comment is welcome

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42094). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Maybe I will proof read again, but I am removing draft for now. Will address review soon.

Removed draft, ready to merge IMO

Benchmarks, on H100, with the last version of the example script that adds the `--add-prefix` option and sampling off by default
Command: `python examples/pytorch/continuous_batching.py --attn $attn -mp none --add-prefix --samples 500`
| Attention | Version  | Generated tokens | Duration (s) | Throughput (tok/s) | Total prefill length matched |
|-------------|------------|--------------------------|------------------|--------------------------|-------------------------------------|
| SDPA | With prefix sharing | 112033 | 90.47 | 1238.37 | 16096 |
| SDPA | No prefix sharing | 112223 | 89.95  | 1247.55 | - |
| SDPA | Main branch | 112223 | 90.31 | 1242.60 | - |
| Flash attention | With prefix sharing | 111494 | 24.76  | 4503.30 | 16096 |
| Flash attention | No prefix sharing | 112599 | 26.87 | 4190.22  | - |
| Flash attention | Main branch | 112599 | 27.10 | 4155.44 | - |

The throughput gain increases when the prefix length increases. For reference, the prefixes in the table above are of length 0, 40, 60 or 80 tokens (roughly). If instead we add a large system prompt, so all requests share a 2500 tokens prefix, the gap is more noticeable. Here are the number with flash attention:

```
No prefix sharing:    195.49 seconds for 122435 tokens. 626.29tok/s
W/ prefix sharing:    129.37 seconds for 121763 tokens. 941.17tok/s
```
Use `getattr` in `standardize_rope_params` because `rope_parameters` not always present
Author: hmellor
Pull request: yes

Some custom models written for v4 will specify `rope_theta` but not `rope_scaling`/`rope_parameters`. In this case, there was an attribute error when accessing `rope_parameters` in `standardize_rope_params`. This PR fixes that.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42593). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: correct typos in code comments
Author: arrdel
Pull request: yes

# What does this PR do?

Fixes three typos found in code comments across the codebase.

## Changes

- **modeling_utils.py** (line 4183): Fixed `avaoid` → `avoid` in TODO comment
- **trainer_utils.py** (line 927): Fixed `weigth` → `weight` in TODO comment  
- **convert_slow_tokenizer.py** (line 1174): Fixed `Templace` → `Template` in comment

## Before submitting

- ✅ This PR fixes typos in comments
- ✅ All checks passed (`make fixup`)
- ✅ No functional changes, only comment corrections

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42577). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix the FA2 logic in the longcat_flash model
Author: YangKai0616
Pull request: yes

# What does this PR do?

FA2 does not support MLA (i.e., cases where the dimensions of Q, K, and V heads are inconsistent), so skip this test.


<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc @MekkCyber
- kernels: @MekkCyber @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad 

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


Comments:
@vasqu , please help review. Thanks!

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: longcat_flash

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42549). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Quantization] Remove dequant fp8 config
Author: MekkCyber
Pull request: yes

# What does this PR do?

Remove quantization config when dequantizing on the fly

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: finegrained_fp8

Quickly testing on a failing script

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42596). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[XPU] Fix fp8 UT patch func
Author: YangKai0616
Pull request: yes

# What does this PR do?

When testing `tests/quantization/finegrained_fp8/test_fp8.py` on XPU, it reported the following error:
```
FAILED tests/quantization/finegrained_fp8/test_fp8.py::FP8QuantizerTest::test_dequantization_no_accelerator - AssertionError: False is not true
FAILED tests/quantization/finegrained_fp8/test_fp8.py::FP8QuantizerTest::test_dequantize_when_no_accelerator - ValueError: Expected a xpu device, but got: cpu
FAILED tests/quantization/finegrained_fp8/test_fp8.py::FP8QuantizerTest::test_quantizer_validation_no_accelerator - AssertionError: RuntimeError not raised
```
The root cause is that the `_patch_no_accelerator` method does not fully take effect on XPU.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc @MekkCyber
- kernels: @MekkCyber @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad 

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


Comments:
Hi @MekkCyber , please help review, thanks!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42584). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: finegrained_fp8
Add torch compile to CB
Author: remi-or
Pull request: yes
Labels: name: run-benchmark

This PR adds the option to wrap the GPU-side of continuous batching with `torch.compile`
It also enable the benchmarking of CB with a compile config, and simplifies how benchmarking handles everything related to compile by using a `transformers` `CompileConfig` object

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42516). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Benchmarks, on H100, with the last version of the example script that changes the `--compile` option
Command: `python examples/pytorch/continuous_batching.py --attn $attn --add-prefix --samples 500 --compile`

| Attention         | Version       | Generated tokens | Duration (s) | Throughput (tok/s) |
|-------------------|---------------|------------------|--------------|--------------------|
| SDPA              | Main branch   | 112033           | 90.47        | 1238.37            |
| SDPA              | No compile    | 112033           | 90.85        | 1233.11            |
| SDPA              | With compile  | 113318           | 82.89        | 1367.14            |
| Flash attention 2 | Main branch   | 111198           | 28.59        | 3889.45            |
| Flash attention 2 | With compile   | 111998           | 25.07        | 4467.54            |
| Flash attention 3 | No compile    | 112021            | 22.71       | 4933.46           |
| Flash attention 3 | With compile  | 112647           | 18.70       |  6023.4           |

The PR does not affect un-compiled workload and significantly speeds-up compiled workloads.
Allow `validation_fn` to be `None` in `validate_rope`
Author: hmellor
Pull request: yes

Custom models which specify `mrope` as their `rope_type` were failing to initialise because of an attribute error when attempting to get `_validate_mrope_rope_parameters`, which doesn't exist.

The lines following this `getattr` indicate that this is suppose to trigger a warning, not an error.

This PR fixes that.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42601). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add SDPA support for PatchTST model
Author: Furkan-rgb
Pull request: yes

## What does this PR do?

Adds SDPA (Scaled Dot Product Attention) support for the PatchTST model.

### Changes:
- Added `PatchTSTSdpaAttention` class using `torch.nn.functional.scaled_dot_product_attention`
- Integrated attention class selection in `PatchTSTEncoderLayer` based on `config._attn_implementation`
- Added `_supports_sdpa = True` to `PatchTSTPreTrainedModel`
- Fixed `test_modeling_patchtst.py` `_prepare_for_class` method for proper dynamic batch size handling

### Testing:
All 67 PatchTST tests pass (103 skipped as expected):
```
pytest tests/models/patchtst/test_modeling_patchtst.py
```

### Notes:
- The SDPA implementation falls back to eager attention when `output_attentions=True` since SDPA doesn't return attention weights
- Uses the standard attention implementation pattern from other models in transformers

Comments:
Seems like the documentation build is failing due to improperly closed </div> tag during the build. Unrelated to my changes.

> Sorry, we no longer support implementing seperate Attention classes for each attention flavor. There are some older models which haven't been refactored yet for this case.
> 
> Please take a look at models like Albert or Bert (a bit messy because it has additional enc-dec logic) which already implement this. The essence is to have
> 
> * One Attention class
> * Specific attributes like `is_causal`, `scaling`, `num_attention_heads` to directly reuse across flavors
> * Generic code around everything around the core attn mechanism, e.g. the projections and views/reshaping
> * The interface (`ALL_ATTENTION_FUNCTIONS`) that calls the underlying flavor for us

Thanks for the feedback. Took a look and updated accordingly. Much cleaner now. The existing PatchTSTAttention already uses ALL_ATTENTION_FUNCTIONS.

@Furkan-rgb I fixed the SLOW tests and explicitly testing with sdpa

run-slow: patchtst

This comment contains `run-slow`, running the specified jobs: 

models: ["models/patchtst"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19828799881)

✅ No failing test specific to this PR 🎉 !


run-slow: patchtst

💔 This comment contains `run-slow`, but unknown error occurred and [the workflow run](https://github.com/huggingface/transformers/actions/runs/19830749766) aborted!

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: patchtst

run-slow: patchtst

This comment contains `run-slow`, running the specified jobs: 

models: ["models/patchtst"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19830810458)

✅ No failing test specific to this PR 🎉 !


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42465). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sorry for the delays, merged now :hugs:
Every model forward() should have **kwargs
Author: Rocketknight1
Pull request: yes

As the name suggests, this adds a test to ensure that all model classes have `**kwargs` at the end of their `forward()` methods. It also updates all models that were failing the test.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: align, altclip, audioflamingo3, auto, autoformer, bark, bart, beit, big_bird, bigbird_pegasus, biogpt, bit, blenderbot, blenderbot_small, blip, blip_2

Closing and moving to a cleaner branch
Populate deprecated models
Author: Rocketknight1
Pull request: yes

Entirely the fault of my local env, ignore this!

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42604). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Align tie weights in Idefics
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: idefics

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42551). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
repo. consistency bot
Author: ydshieh
Pull request: yes

# What does this PR do?

repo. consistency bot

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42575). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Quality fix is beginning .... [View the workflow run here](https://github.com/huggingface/transformers/actions/runs/19871288296).

Quality fix is beginning .... [View the workflow run here](https://github.com/huggingface/transformers/actions/runs/19871857299).

Quality fix is beginning .... [View the workflow run here](https://github.com/huggingface/transformers/actions/runs/19887420911).

Quality fix is beginning .... [View the workflow run here](https://github.com/huggingface/transformers/actions/runs/19887581389).

Quality fix is beginning .... [View the workflow run here](https://github.com/huggingface/transformers/actions/runs/19887715160).

Quality fix is beginning .... [View the workflow run here](https://github.com/huggingface/transformers/actions/runs/19887853227).

Quality fix is beginning .... [View the workflow run here](https://github.com/huggingface/transformers/actions/runs/19889737894).

Quality fix is beginning .... [View the workflow run here](https://github.com/huggingface/transformers/actions/runs/19889888937).

Quality fix is beginning .... [View the workflow run here](https://github.com/huggingface/transformers/actions/runs/19890138794).

Quality fix is beginning .... [View the workflow run here](https://github.com/huggingface/transformers/actions/runs/19890194750).

Repo. Consistency fix runs successfully without any file modified.

Repo. Consistency bot fixed some files and pushed the changes.

Repo. Consistency fix runs successfully without any file modified.

Repo. Consistency bot fixed some files and pushed the changes.

Repo. Consistency fix runs successfully without any file modified.

Quality fix is beginning .... [View the workflow run here](https://github.com/huggingface/transformers/actions/runs/19897254288).

Repo. Consistency bot fixed some files and pushed the changes.

Repo. Consistency fix runs successfully without any file modified.

Repo. Consistency bot fixed some files and pushed the changes.

Repo. Consistency fix runs successfully without any file modified.

Repo. Consistency bot fixed some files and pushed the changes.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: altclip

Repo. Consistency fix runs successfully without any file modified.
Fix Ernie Moe Test
Author: vasqu
Pull request: yes

Flaky test: 
- `pytest tests/models/ernie4_5_moe/test_modeling_ernie4_5_moe.py -k test_load_balancing_loss --flake-finder --flake-runs=1000`
- Encounter 0 failures

Hope it's more robust again

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42595). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: ernie4_5_moe

I think the core changes are
- Properly propagate padding tokens within the ids https://github.com/huggingface/transformers/blob/eb3c0880a4d2b713b49da7763b129e199608ba56/tests/models/ernie4_5_moe/test_modeling_ernie4_5_moe.py#L115-L120
- And using the config pad token id, not some arbitrary number

I think either or both contributed to slightly different behavior compared to the other moe models
[pipeline] fix unwanted import failure
Author: eustlb
Pull request: yes

# What does this PR do?

FIxes #42499

torchcodec installation can be tricky and right now pipeline can fail (if torchcodec is installed) even if torchcodec is actually not required. The lazy import can be even laziest :)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42598). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add FastVLM from CVPR 2025
Author: camilla-deckard
Pull request: no
Labels: name: New model

### Model description

Apple recently released FastVLM, a new vision-language model introduced at CVPR 2025, which significantly improves on previous models in the LLaVA family.

The smallest FastVLM variant outperforms LLaVA-OneVision-0.5B, achieving:

- 85× faster Time-to-First-Token (TTFT)
- 3.4× smaller vision encoder

Its ultra-low latency enables real-time deployment on mobile and edge devices, as demonstrated in the official demo. Given its strong performance and efficiency, I believe FastVLM would be a valuable addition to this repo.

If you also think so, I'd love to contribute this model.

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

Paper: https://www.arxiv.org/abs/2412.13303
Official repo: https://github.com/apple/ml-fastvlm

Comments:
Hey @kamila-chay !

Yes, adding the model is already planned afaik. @ariG23498 had a nice inference script using transformers under [this comment](https://github.com/apple/ml-fastvlm/issues/1#issuecomment-2883616294). To support the model in core library we need to convert weights, and we're waiting for Apple team on that I think

Hi @zucchini-nlp! Thank you for the answer, I didn't see the thread. Glad to hear it's already planned. You can mention me here if you need community support for this model in the future (to add new components etc, not quite sure what will be needed)

Now that it is public from apple, I think we can start working on it

Collection: https://huggingface.co/collections/apple/fastvlm-68ac97b9cd5cacefdd04872e

That's great to hear! I'm still willing to contribute, @ariG23498 let me know if i can open a PR and work on it. I have some free time now so I can start right away 💯

@kamila-chay that is great to hear.

@zucchini-nlp is that okay with you?

Sure, thanks @kamila-chay !  I haven't personally looked at model arch, it might be compatible with an existing llava or llava-next model code. Can you first check that and if yes, adapt a conversion script

It makes long term maintenance easier if we don't need to add new model class 

Sure I will do that!

Hi @zucchini-nlp, @ariG23498,
I just wanted to check in to make sure we’re aligned. I noticed that remote code was added to the official Apple repos on the HF hub 2 days ago, and I wasn’t sure if you still plan to integrate this model into the core library in that case. There’s quite a bit of new code to bring in if we go that route, so I wanted to clarify what the plan is 😃 

I am fine with bringing the code in transformers, though let's ask @ariG23498 about what we have agreed on with Apple team. Just in case anything changed over the weekend :)

Hi @ariG23498,
any info on that?

We are also looking to enable this model on ExecuTorch - would love to see this model in Transformers! Any plans on integrating the HF hub modeling code into the main repo?

Hey @kamila-chay sorry for the delay. It looks like we are on!
You can start working on the FastVLM integration to transformers.

Looking forward to it @kamila-chay 🙏
Distributed training of SigCLIP
Author: zyk1559676097-dot
Pull request: no

https://github.com/huggingface/transformers/blob/v4.57.1/src/transformers/models/siglip/modeling_siglip.py#L983, here define how to compute sigclip loss. In sigclip, different tpu will exchange data with each other. I want to know how to train a model in this way.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.