noamaanMulla-03/transformers-issues huggingface dataset:
[`BLT`] Fix cache usage
Author: vasqu
Pull request: yes

The issue is/was that the cache wasn't utilized at all while several generation methods rely on it. This resulted in a mismatch between actual input and attention mask
- the mask was correctly expanded one by one
- after the prefill phase, the cache wasn't used resulting in kv always being of size 1

Why did it even work?
- because we silently cut the mask to fit the kv shape, i.e. a silent bug and another good reason why enforcing the correct shape is highly needed

The fix:
- apply self caches for both the encoder and decoder

Future considerations:
- add cross cache for both encoder + decoder --> not used atm (but also wasn't used before as far as I can see)

Comments:
run-slow: blt

This comment contains `run-slow`, running the specified jobs: 

models: ["models/blt"]
quantizations: []

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42188). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: blt

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19337450988)

⚠️ No test being reported (jobs are skipped or cancelled)!


This comment contains `run-slow`, running the specified jobs: 

models: ["models/blt"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19338389859)

✅ No failing test specific to this PR 🎉 !


**[For maintainers]** Suggested jobs to run (before merge)

run-slow: blt, mllama

run-slow: blt

This comment contains `run-slow`, running the specified jobs: 

models: ["models/blt"]
quantizations: []

run-slow: blt

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19363807493)

⚠️ No test being reported (jobs are skipped or cancelled)!


This comment contains `run-slow`, running the specified jobs: 

models: ["models/blt"]
quantizations: []

Fixes the normal fast tests; weird issue atm on main where it takes significantly more memory / it's not properly cleaned up. Locally the slow tests pass except for those that already didnt 

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19364467440)

✅ No failing test specific to this PR 🎉 !
Add pytree registration for static cache
Author: angelayi
Pull request: yes

* Refactored the existing pytree registration for DynamicCache to be a more generalized form
* Also fixed the existing pytree registration which triggers [lazy_initialization](https://github.com/huggingface/transformers/blob/9aab965b1e61d92d402809bd467c317ec464e560/src/transformers/cache_utils.py#L94-L95). This shouldn't be needed because we already have all the contents of the dynamic cache created?
* Added pytree support for StaticCache

cc @tugsbayasgalan 

Comments:
Fix: Correct loss normalization in training_step for multi-GPU training
Author: KaparthyReddy
Pull request: yes

# Fix: Correct loss normalization in training_step for multi-GPU training

## Description
Fixes #37474 

This PR corrects the loss aggregation logic in `Trainer.training_step` when training with multiple GPUs.

## Problem
When `num_items_in_batch` is provided (e.g., for token-level loss normalization), each device computes loss as:
```python
per_device_loss = sum_of_losses / total_items_across_all_devices

Comments:
This should already be fixed in https://github.com/huggingface/transformers/pull/40799

@KaparthyReddy can you cool it down with these PRs, or at least read issues carefully and only make PRs when you understand the issue and you've checked they're not duplicates? You've made like 6 of them and they're all obviously written by Copilot, so it's very hard to tell if any of them actually do anything useful!

Noted. I’ll proceed as I see fit.
Add position encoding interpolation to DeiT
Author: KaparthyReddy
Pull request: yes

# Add position encoding interpolation to DeiT

## Description
Adds position encoding interpolation to DeiT models, enabling the use of pretrained checkpoints on images with different resolutions than those used during training.

Contributes to #30579

## Changes
- ✅ Added `interpolate_pos_encoding` parameter to `DeiTModel`, `DeiTForImageClassification`, `DeiTForImageClassificationWithTeacher`, and `DeiTForMaskedImageModeling`
- ✅ Implemented `interpolate_pos_encoding()` method in `DeiTEmbeddings` class
- ✅ Properly handles both CLS and distillation tokens (unique to DeiT)
- ✅ Added comprehensive test `test_model_with_different_image_size`
- ✅ All tests passing

## Implementation Details
DeiT has two special tokens (CLS + distillation), unlike ViT which only has one (CLS). The interpolation method:
1. Separates the CLS/distillation token embeddings from patch embeddings
2. Interpolates patch embeddings to match new image size
3. Recombines CLS/distillation tokens with interpolated patches

## Testing
```python
# Example usage
model = DeiTModel.from_pretrained("facebook/deit-base-distilled-patch16-224")
# Use with 480x480 images instead of 224x224
outputs = model(large_images, interpolate_pos_encoding=True)

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: deit

This code agent PR has a lot of unnecessary changes (note the `head_mask` lines, etc.) Hard to review until it's made more compact! (Please don't just ask your code agent to fix that, lol, they tend to make it worse)

Acknowledged.
Refines quantization method handling in model loading
Author: cdutr
Pull request: yes

## What does this PR do?

Fixes a regression from #41580 where loading quantized models fails with `ValueError: This quantization method is gonna be supported SOOOON` for all quantizers except FP8 (e.g., BitsAndBytes 4-bit/8-bit, HQQ, EETQ, Quanto).

The issue: the refactored weight loading code only handled FineGrainedFP8HfQuantizer and raised an error for all other quantization methods. However, other quantizers don't need `quantization_operation` set during weight conversion—they handle quantization when parameters are assigned to modules.

The fix: Remove the error and add a clarifying comment explaining the difference between FP8 (quantizes during conversion) and other quantizers (quantize during parameter assignment).

Fixes #42215

## Review

@CyrilVallez (model loading) @SunMarc @MekkCyber (quantization)

Comments:
Oh, got it. I will close it then. Thanks for the reply
Much more efficient and clear weight initialization and tie weights
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

Follow-up to https://github.com/huggingface/transformers/pull/41580. 
This PR is focused on speed and efficiency, as well as clarity. It avoids the following issues/bottlenecks:

- No longer need to switch dynamically the class of the Parameters (which is hard to understand and may lead to issues with quantization etc, as it's not always clear if the memory stays the same or if we double it)
- Avoids one more loop over all the modules to switch class
- Avoids initializing the tied weights (which takes a lonnnngggg time for big embeddings, e.g for `"google/gemma-2-2b"`, the embedding has size `256k * 2304`, which is more than 1GB of data in float16 in a single parameter, and it takes more than 3-4s to initialize it with `normal_`, see profiling below), when we will overwrite them anyway later (with the tied weights)
- avoids one more loop over all modules for tied weights (get the correct names in advance)

<img width="1418" height="353" alt="Screenshot 2025-11-14 at 10 43 53" src="https://github.com/user-attachments/assets/38dfd69c-b5cd-4b0b-a6ee-167c55493a26" />

On that trace, the call to `Parameter.normal_` takes 3.4s, and is only the `lm_head` (the only "missing" weight), even though it's not truly missing because it's a tied weight which is being overwritten later!

So basically, the following snippet 

```python
from transformers import AutoModelForCausalLM
import torch
import time

model_id = "google/gemma-2-2b"
device = 1 

torch.cuda.synchronize()
t0 = time.time()
model = AutoModelForCausalLM.from_pretrained(model_id, device_map=device, dtype="auto")
torch.cuda.synchronize()
dt = time.time() - t0
print(f"Took {dt:.2f} s")
```

takes about 7s, when it was taking about 3s on main before https://github.com/huggingface/transformers/pull/41580 /with the old loading). After this PR, it takes 3s as well, effectively being as performant as before.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42191). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aimv2, albert, align, altclip, aria, audio_spectrogram_transformer, audioflamingo3, autoformer, bamba, bark, bart, beit, bert, bert_generation, big_bird, bigbird_pegasus
feat(processing_utils): support optional attributes in ProcessorMixin
Author: MengAiDev
Pull request: yes

- Add functionality to handle optional processor attributes
- Update save_pretrained and from_pretrained methods to include optional attributes
- Modify processor_config.json handling to accommodate optional attributes
- Improve error handling and logging for optional attribute loading

# What does this PR do?

Fixes #40447 

cc @ArthurZucker @amyeroberts, @qubvel @Rocketknight1

Comments:
gradient scaling occurs even though total gradient remains < max_grad_norm in trainer.py
Author: lorsonblair
Pull request: no

Even though gradients remain < max_grad_norm throughout training, the gradient still goes through a scaling process. For instance, I set max_grad_norm = 1, and grad_norm consistently remains <= 0.33. Because the trainer takes you through the grad clip process if max_grad_norm > 0 or not None, this operation always gets executed within torch's clip function: `clip_coef = max_norm / (total_norm + 1e-6)`. Is there a way to prevent this? Thanks.  



Comments:
Hi @lorsonblair, I'd like to work on this!
You're right that the scaling happens every time even when it's not needed. PyTorch skips the if clip_coef < 1: check to avoid CPU-GPU sync overhead, but that means we're doing unnecessary work when gradients consistently stay below threshold.

I'm thinking we could add an opt-in flag in TrainingArguments (maybe skip_unnecessary_grad_clip?) that checks the norm before clipping. Default would be False to keep things backward compatible. This would help especially for CPU training or stable gradient scenarios.

If this sounds okay,  I will be happy to benchmark the performance impact and submit a PR!

Thanks @vaibhavgarg230, this sounds good. It would be really helpful. Thanks.

cc @sunmarc I think

If it doesn't affect performance much, an option to keep printing the grad_norm, even though no clipping occurs, would be resourceful. I'm running some experiments where I'm plotting the grad_norm (change in) throughout training. Thanks again. 

Totally makes sense! So the plan would be:
 - Skip the actual clipping operation when grad_norm <= max_grad_norm to save compute
 - But still calculate and return the grad_norm value so it gets logged normally

This way the optimization doesn't break your existing workflow and you'd still see the grad_norm in your logs/plots even when no clipping happens. I'll make sure the implementation preserves this behavior.

Let me put together a PR with both features

Sounds good. Thank you! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
When using a custom model, it copies the code into Hugging Face’s cache directory.
Author: wzr0108
Pull request: no

```
    model = AutoModel.from_pretrained(
        model_args.model_name_or_path,
        trust_remote_code=True,
        torch_dtype=compute_dtype,
        device_map=device_map,
        # init_vision=True,
        # init_audio=False,
        # init_tts=False,
    )
```
`model_args.model_name_or_path=/mnt/241hdd/wzr/MiniCPM-V-CookBook/MiniCPM-V-4_5`
The code actually runs in `/root/.cache/huggingface/modules/transformers_modules/MiniCPM-V-4_5`.
This makes my debugging difficult.
Is there a way to run the code directly?

Comments:
I'd like to help investigate this issue. I'll start by reproducing the behavior locally and then examine the cache handling logic in `src/transformers/utils/hub.py`. Should have an update within a few days on what I find.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I think modules are generally cached, unfortunately! This is because the code paths for local and remote models on the Hub are designed to be as similar as possible. Running remote code "in place" for debugging, similar to like `pip install -e`, is an interesting idea though, but we'd have to think really hard about adding that in case it causes maintenance issues in other places

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Closed prototype PR – superseded by future implementation
Author: jackiehimel
Pull request: yes

# What does this PR do?

Adds SDPA and FlashAttention-2 support to LayoutLMv3 following the same pattern as other models. Fully backward compatible.

SDPA converts masks to boolean format. FA2 uses `_upad_input` for variable-length sequences and avoids redundant unpads. Both fall back gracefully when needed. FA2 is O(N) memory vs O(N²).

Fixes #35467

# Changes

- Added `LayoutLMv3SdpaAttention` using `torch.nn.functional.scaled_dot_product_attention`
- Added `LayoutLMv3FlashAttention2` with `flash_attn_func` / `flash_attn_varlen_func`
- Both inherit from `LayoutLMv3Attention`
- Fallback to standard attention when backends unavailable or `output_attentions=True` / relative position bias is used

# Testing

- 121 tests passed in `test_modeling_layoutlmv3.py`
- Manually verified forward passes with/without attention masks

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@vasqu @ArthurZucker @CyrilVallez

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: layoutlmv3

Thank you so much @vasqu for your detailed review and guidance! It is greatly appreciated. After looking more closely at the integration patterns and the BERT implementation, I realize this needs a more substantial refactor to match the current approach. Closing this for now but will revisit with a cleaner implementation later. Appreciate it!
Fix UnboundLocalError in RT-DETR loss when auxiliary_loss is False
Author: yashwantbezawada
Pull request: yes

## What does this PR do?

Fixes #42219 

This PR fixes an `UnboundLocalError` that occurs in `RTDetrForObjectDetectionLoss` when `config.auxiliary_loss` is set to `False`.

## Problem

The function `RTDetrForObjectDetectionLoss` in `src/transformers/loss/loss_rt_detr.py` always returns `auxiliary_outputs` at line 468, but this variable is only initialized inside the `if config.auxiliary_loss:` block (line 451). When `auxiliary_loss=False`, the variable is never initialized, causing:

```python
UnboundLocalError: variable 'auxiliary_outputs' referenced before assignment
```

## Solution

Initialize `auxiliary_outputs = None` before the conditional block (line 450) to ensure it always has a value when returned.

**Changes:**
- Added `auxiliary_outputs = None` initialization at line 450

This is a minimal, defensive fix that ensures the function works correctly regardless of the `auxiliary_loss` configuration.

## Testing

The fix ensures that:
- When `auxiliary_loss=True`: `auxiliary_outputs` is set to the computed values (existing behavior)
- When `auxiliary_loss=False`: `auxiliary_outputs` is `None` (fixed behavior)

## Additional context

The `auxiliary_loss` parameter defaults to `True` in `RTDetrConfig`, which is why this bug may not have been caught in standard testing scenarios. The bug only manifests when users explicitly set `auxiliary_loss=False`.

Comments:
Closing this PR due to branch divergence issues. The branch has accumulated 800+ changed files from main, causing unrelated test failures.

I'll create a new clean PR with only the RT-DETR fix shortly.

**Issue**: UnboundLocalError when `auxiliary_loss=False` in RT-DETR
**Fix**: Initialize `auxiliary_outputs = None` before conditional use
GLM-V update with new processor
Author: zRzRzRzRzRzRzR
Pull request: yes
Labels: name: New model

Support another processor handling method

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42122). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker is merging it today, I can't merge when CI is red :) 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, glm46v, glm4v, glm4v_moe
Add DeepseekVLV2 Model
Author: Player256
Pull request: yes

Fixes  #36111 

This is a draft PR for adding deepseek_vl_v2 to the transformers library.

Comments:
cc @zucchini-nlp

@Rocketknight1 could you help me out here. I think I am getting blocked. Could you take a look in my code. When I try to do an inference to test it out I get this error:
`ValueError: Cannot use apply_chat_template because this processor does not have a chat template.`

Hi @Player256, chat templates are stored with models, usually in a file called `chat_template.jinja`, but in older models it can be in `chat_template.json` or in an entry in `tokenizer_config.json`. When loaded, they're set to `tokenizer.chat_template` or `processor.chat_template`. They convert chats into token sequences. If you use `apply_chat_template` or you call `TextGenerationPipeline` or `ImageTextToTextPipeline`, it'll try to tokenize the chat using a chat template, and it'll fail if no chat template is present. Take a look [here](https://huggingface.co/docs/transformers/en/chat_templating) for more.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto
Repeated `bos_token_id` is added when using tokenizer
Author: Hermit-w
Pull request: no

The code I use is:
```python
from transformers import AutoTokenizer

messages = [
    {"role": "user", "content": "Hello!"},
    {"role": "assistant", "content": "Hi there!"},
    {"role": "user", "content": "How are you?"}
]

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x22B-Instruct-v0.1")
# first usage
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
decode_text = tokenizer.decode(inputs[0])
print(f"Decoded text: {decode_text}")
# second usage
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(prompt, return_tensors="pt")
decode_text = tokenizer.decode(inputs['input_ids'][0])
print(f"Decoded text: {decode_text}")
```
And the results are:
```txt
Decoded text: 

 Hello!
 Hi there!

 How are you?

Decoded text: 


 Hello!
 Hi there!

 How are you?

```
A repeated `
` is added into the input respectively.

I'm not sure whether this is an expected behaviour, but it does really confused me.

Comments:
Yeah that's expected . If you check the value for `prompt` its 
```


 Hello!
 Hi there!

 How are you?

```


`apply_chat_template` already has a bos token and then when we pass it to the tokenizer we add it again . In your second usage adding the keyword parameter `add_special_tokens = False` will give the same result .


> Yeah that's expected . If you check the value for `prompt` its
> 
> ```
> 

 Hello!
 Hi there!

 How are you?

> ```
> 
> `apply_chat_template` already has a bos token and then when we pass it to the tokenizer we add it again . In your second usage adding the keyword parameter `add_special_tokens = False` will give the same result .

I noticed that this model is using llama tokenizer instead of MistralCommonTokenizer class even when the mistral-common lib is installed.Wasn't yet able to figure out the fix.  Do you? Or shall I raise this issue. 

>I noticed that this model is using llama tokenizer instead of MistralCommonTokenizer class even when the mistral-common lib is installed.Wasn't yet able to figure out the fix. Do you? Or shall I raise this issue.


Not sure myself . But if you go to the model [page][url] the tokenizer_class is LlamaTokenizer (line 6184) . So I think it's most likely not an issue

[url]: https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1/blob/main/tokenizer_config.json  

Thanks for the reply. I have no problem now and I think we can close this issue.
For finetuning MBart-based model, setting decoder_start_token_id in model.config is NOT ENOUGH.
Author: Bmingg
Pull request: no
Labels: name: bug

### System Info

Context: finetuning a MBart model with run_translation.py
Easy fix is to set it in model.generation_config as well. Both worked outside of run_translation.py, but not setting this in run_translation.py causes validation/evaluation to fail miserably. 

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run run_translation.py to finetune with a validation file. 

### Expected behavior

BLEU scores should be terrible due to the bug. This makes it hard to monitor finetuning.

Comments:
can you assign this task to me?


> can you assign this task to me?
How do I do that?


If someone wants to make a PR to adapt `run_translation.py` to work with generation_config too, I think we could accept that! cc @gante 

Also, we don't generally do issue assignment - just say that you're going to make the PR and then make the PR. Just please, if you're going to use a code agent, use something better than the free tier of Copilot 😅 

I have worked on this issue and sent PR by working on run_translation.py 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.