noamaanMulla-03/transformers-issues huggingface dataset:
Add multimodal_chat sample
Author: Wovchena
Pull request: yes

# What does this PR do?
Add a sample covering kv cache management

Fixes https://github.com/huggingface/transformers/issues/43045


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
One possible way to address both the repeated image recomputation and the `past_key_values` / image-token mismatch is to treat the image as fixed context and cache its features outside the chat loop. The image is preprocessed once, its vision-related tensors are reused for every turn, and only the text inputs grow while updating `past_key_values`. This avoids recomputing vision features on each iteration and prevents image placeholder tokens from being trimmed during generation.

```python
# --- Preprocess image ONCE ---
vision_inputs = processor(
    images=image,
    return_tensors="pt"
).to(device)

# Cache vision-related tensors (keep them static)
cached_vision = {
    k: v for k, v in vision_inputs.items()
    if k != "input_ids"
}

past_key_values = None
generated_text = ""

for user_message in chat_history:
    # --- Text-only processing per turn ---
    text_inputs = processor(
        text=user_message,
        return_tensors="pt"
    ).to(device)

    # --- Merge cached vision + new text ---
    model_inputs = {
        **cached_vision,
        "input_ids": text_inputs["input_ids"],
        "attention_mask": text_inputs["attention_mask"],
        "past_key_values": past_key_values,
    }

    outputs = model.generate(
        **model_inputs,
        max_new_tokens=128,
        use_cache=True,
    )

    past_key_values = outputs.past_key_values
    generated_text += processor.decode(outputs[0], skip_special_tokens=True)
````

If the pattern is adopted, the scope of this PR becomes very clear and strong: it serves as a reference-quality multimodal chat example that demonstrates best practices for image caching and correct use of text-only KV caching in a multimodal setting, remember not to touch the core modules or api's in this pr.


Also this is a simple workaround, any other approach is warmly welcome.
Enable flake8-pie rules
Author: cyyever
Pull request: yes

# What does this PR do?

This PR enables all [flake8-pie](https://docs.astral.sh/ruff/rules/#flake8-pie-pie) rules in ruff. These rules are:
```
PIE790   Unnecessary pass statement
PIE794   Class field {name} is defined multiple times 
PIE796   Enum contains duplicate value: {value}  
PIE800   Unnecessary spread ** 
PIE804   Unnecessary dict kwargs  
PIE807   Prefer {container} over useless lambda 
PIE808   Unnecessary start argument in range  
PIE810   Call {attr} once with a tuple   
```

Comments:
Adding more ruff rules so cc core maintainer @arthurzucker

gentle ping core maintainers @cyrilvallez @arthurzucker, since it's adding more ruff rules!

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bert, bertweet, big_bird, clap, conditional_detr, convbert, detr, funnel, glm46v, glm4v, glm4v_moe, gpt_neo, gpt_sw3, instructblipvideo, internvl, layoutxlm
add s3tokenizer
Author: manmay-nakhashi
Pull request: yes
Labels: name: New model

name: Audio

# What does this PR do?

This PR integrates the **S3Tokenizer** (Self-Supervised Speech Tokenizer) into Hugging Face Transformers. S3Tokenizer is a state-of-the-art audio tokenization model that converts raw audio waveforms into discrete tokens at 25 tokens/second, making it suitable for speech-to-speech and text-to-speech applications.

## Key Features

- **Efficient Speech Tokenization**: Converts 16kHz audio to discrete tokens at 25 Hz (25 tokens/second)
- **FSQ Quantization**: Uses Finite Scalar Quantization (FSQ) with a vocabulary size of 6,561 (3^8)
- **Long Audio Support**: Automatically handles long audio sequences using sliding window with overlap

## Implementation Details

This integration includes:

- `S3TokenizerConfig`: Configuration class for model parameters
- `S3TokenizerModel`: Main model class for audio tokenization
- `S3TokenizerFeatureExtractor`: Feature extractor for audio preprocessing

implementation is based on the [xingchensong/S3Tokenizer](https://github.com/xingchensong/S3Tokenizer) repository.

## Usage
```
### Basic Audio Tokenization

import torch
from transformers import S3TokenizerModel, S3TokenizerConfig

# Initialize model with default config
config = S3TokenizerConfig()
model = S3TokenizerModel(config, name="speech_tokenizer_v2_25hz")
model.eval()

# Load and tokenize audio (16kHz sampling rate)
audio = torch.randn(16000)  # 1 second of audio

# Generate speech tokens
with torch.no_grad():
    outputs = model(input_values=audio, return_dict=True)

print(f"Speech tokens: {outputs.speech_tokens}")
print(f"Token count: {outputs.speech_token_lens.item()}")  # Should be ~25 tokens for 1s audio### Batch Processing
```
### Batch Mode
```
import torch
from transformers import S3TokenizerModel, S3TokenizerConfig

config = S3TokenizerConfig()
model = S3TokenizerModel(config)
model.eval()

# Batch of audio samples (different lengths are okay)
audio_batch = [
    torch.randn(16000),
    torch.randn(32000),
    torch.randn(48000),
]

with torch.no_grad():
    outputs = model(input_values=audio_batch, return_dict=True)

# outputs.speech_tokens: [batch_size, max_sequence_length]
# outputs.speech_token_lens: [batch_size]
```
@eustlb 

Comments:
cc @eustlb @ebezzam 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, s3tokenizer
[Feature Request] Support explicitly mapping Model IDs to local paths
Author: Gianthard-cyh
Pull request: no
Labels: name: Feature request

### Feature request

I would like to propose a feature that allows users to explicitly map a Hugging Face Model ID (e.g., Qwen/Qwen2.5-7B) to a specific local directory at runtime.


Conceptual Usage:

```Python

from huggingface_hub import register_local_mapping

# Explicitly tell the library: 
# "When 'Qwen/Qwen2.5-7B' is requested, look at this local path instead of the Hub."
register_local_mapping("Qwen/Qwen2.5-7B", "/abs/path/to/local/weights")

# ... 
# Later, deep inside a third-party library:
# This loads from the local disk, BUT the model config still retains the original ID name.
model = AutoModel.from_pretrained("Qwen/Qwen2.5-7B") 
```

This would allow developers to maintain the semantic identity of the model (vital for logic) while having full control over the physical storage (vital for offline/custom deployment), without needing to alter upstream code.

### Motivation

Currently, to load a model from a local directory, we are required to pass the directory path directly to from_pretrained. While this works for simple scripts, it causes friction in more complex setups, particularly with third-party libraries or immutable codebases.

There are two main challenges:

1. Many downstream tools (like Agent frameworks or RAG pipelines) use the model_id string to decide internal logic (e.g., specific chat templates, stop tokens, or tokenizer adjustments). If we are forced to pass a file path like /data/models/v1, the library loses context of what the model is, potentially breaking this logic.

1. In many integration scenarios, the from_pretrained call is deeply nested within a library. Users often cannot easily inject a local path without resorting to monkey-patching or modifying the library's source code.

It would be extremely helpful to have a lightweight registry or mapping mechanism in huggingface_hub / transformers.


### Your contribution

I'd like to submit a PR if this proposal is approved by the maintainers.

Comments:
Add efficient logits_to_keep masking to ZambaForCausalLM for constrained generation
Author: Aniketsy
Pull request: yes

#40984 

This PR updates the `ZambaForCausalLM` forward method to efficiently mask logits using the `logits_to_keep` argument. It supports both top-k and index-based selection, setting all other logits to `-inf` after projection.

Please let me know if my approach or fix needs any improvements . I’m open to feedback and happy to make changes based on suggestions.
Thankyou !

Comments:
@Rocketknight1 Could you please review these changes and let me know if any improvements are needed?

Hi @Aniketsy, I think this is a different issue to #40984! That PR was about not computing logits for certain tokens by skipping the LM head entirely. This PR masks certain logits for a single token, making it more like a `LogitsWarper` that is used in generation.

@Rocketknight1 Thanks for clarifying! I’ll look more into this, and I’d really appreciate any further guidance or pointers you can share.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: zamba
[i18n-<te>] Translating docs to <Telugu>
Author: hakunamatata1997
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

Let's bring the documentation to all the <languageName>-speaking community 🌐 (currently 0 out of 267 complete)

Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `<languageCode>` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `<languageCode>/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue. Please ping @stevhliu and @MKhalusova for review.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).

## Get Started section

- [x] [index.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/index.md) https://github.com/huggingface/transformers/pull/20180
- [x] [quicktour.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/quicktour.md) (waiting for initial PR to go through)
- [ ] [installation.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/installation.md).

## Tutorial section
- [ ] [pipeline_tutorial.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/pipeline_tutorial.md)
- [ ]  [autoclass_tutorial.md](https://github.com/huggingface/transformers/blob/master/docs/source/autoclass_tutorial.md)
- [ ]  [preprocessing.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/preprocessing.md)
- [ ]  [training.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/training.md)
- [ ]  [accelerate.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/accelerate.md)
- [ ]  [model_sharing.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_sharing.md)
- [ ]  [multilingual.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/multilingual.md)

<!--
Keep on adding more as you go 🔥
-->


Comments:
@stevhliu Reopen the issue as all the pages have not been converted in this language.

Assign this issue to me
Qwen3Moe models have bug during generation since they calculate unnecessary load_balancing_loss
Author: zhangchong25
Pull request: no
Labels: name: bug

### System Info

Qwen3Moe models will calculate load_balancing_loss during evaluation, which will cause bug during generation on the 2nd (and later) steps. 

The problem can be handled by modifying 

```python
if output_router_logits:
    aux_loss = load_balancing_loss_func(
    ...
```

to

```python
if output_router_logits and self.training:
    aux_loss = load_balancing_loss_func(
    ...
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Perform model.generate() using a Qwen3MoeForCausalLM model. 

### Expected behavior

Generate successfully. 

Comments:
cc @Rocketknight1 I only did it for the Qwen models but the pattern is the same on other MoE, should I add the fix here already?

I think there was a bug on the test for qwen3-moe, it was using eval instead of train for training tasks.

Hey, just following up on this - `aux_loss` should only be calculated when `output_router_logits` is `True`, but that defaults to `False` for all the `Qwen` models I checked. Is `model.generate()` setting that flag somehow? I'd like to understand how this bug is happening before we start fixing it.

yeah I was trying to replicate the bug now with `model.eval()` mode and the only way I managed to get a not None `aux_loss` was by directly passing`output_router_logits=True` on the model kwargs for the forward pass

In that case, I'm not sure this is a bug! Generating with `output_router_logits=True` (rather than generating without it and then doing a single forward pass with it at the end) is a strange thing to do, so it's not clear that we need to support it

> In that case, I'm not sure this is a bug! Generating with `output_router_logits=True` (rather than generating without it and then doing a single forward pass with it at the end) is a strange thing to do, so it's not clear that we need to support it

@Rocketknight1 
Hi Matt,
Thank you for noticing this issue. 
Router logits in generation is needed when doing interpretation studies. Please consider to support it for flexible use. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
TrainingArguments does not process fsdp arguments correctly
Author: quic-meetkuma
Pull request: no
Labels: name: bug

### System Info

`transformers` version: 5.0.0.dev0 (Added backend specific code only)
Platform: Linux-6.8.0-41-generic-x86_64-with-glibc2.39
Python version: 3.10.19
Huggingface_hub version: 1.0.0.rc6
Safetensors version: 0.6.2
Accelerate version: 1.10.1
Accelerate config: not found
DeepSpeed version: not installed
PyTorch version (accelerator?): 2.7.0+cpu (NA)
Using distributed or parallel set-up in script?: Not needed

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Code references are given.

### Expected behavior

In the _process_fsdp_args() function within the TrainingArguments class, the parameters intended for the FullyShardedDataParallelPlugin from the accelerate module are not mapped correctly. Additionally, some parameters are missing entirely.

The current implementation does not pass the necessary parameters from self.fsdp_config into fsdp_plugin_args as expected. Parameters should be mapped based on the example below, but this mapping is incomplete in the existing code.

Example: https://github.com/huggingface/accelerate/blob/b9ca0de682f25f15357a3f9f1a4d94374a1d451d/tests/tp/fsdp2_tp_preparation.py#L72

Comments:
cc @SunMarc 

Can you try the branch from main ? We should have fixed the issues regarding those args. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> Can you try the branch from main ? We should have fixed the issues regarding those args.

Checked on main. Looks like the mapping is properly managed. I will check and re-open issue if I still face any issues.
Qwen2VLImageProcessor not respecting Qwen3VL config
Author: opherlieber
Pull request: no
Labels: name: bug

name: Vision

### System Info

Qwen3VL models use Qwen2VLImageProcessor/Qwen2VLImageProcessorFast.
Qwen2VLImageProcessor seems to have a bug where it [ignores](https://github.com/huggingface/transformers/blob/703da867001c5e2c8322b2c3395e6c4a5ba31022/src/transformers/models/qwen2_vl/image_processing_qwen2_vl.py#L164) and always overrides the 'size' value from the config, which causes wrong processing for Qwen3VL.
Qwen2VLImageProcessorFast is ok

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoImageProcessor
p_fast = AutoImageProcessor.from_pretrained("Qwen/Qwen3-VL-30B-A3B-Instruct", use_fast=True)
p_slow = AutoImageProcessor.from_pretrained("Qwen/Qwen3-VL-30B-A3B-Instruct", use_fast=False)
height, width = 300,166
print(p_fast.get_number_of_image_patches(height, width, images_kwargs={}), p_slow.get_number_of_image_patches(height, width, images_kwargs={}))
````
Output: `264 180`


### Expected behavior

Qwen2VLImageProcessor should use the size from the config and return the same result as Qwen2VLImageProcessorFast

Comments:
Yeah, that is a bad pattern to overwrote tbh. We should only raise an error is the format doesn't match Qwen-VL and use whatever is loaded from config
Would you like to submit a PR @opherlieber? Feel free to tag me when you have a draft
Improve `_init_weights` tests
Author: Cyrilvallez
Pull request: yes

# What does this PR do?



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43108). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43108&sha=ed54bd
[v5] Remove deprecated tranformers.onnx
Author: echarlaix
Pull request: yes
Labels: name: for_v5?


* remove deprecated `transformers.onnx` for v5
* remove all `XxxOnnxConfig` from concerned models config files

Also following https://github.com/huggingface/transformers/pull/41214

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41700). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: albert, bart, beit, bert, big_bird, bigbird_pegasus, blenderbot, blenderbot_small, bloom, camembert, chinese_clip, clip, codegen, conditional_detr, convbert, convnext

I have a question regarding this MR and don't know where else to ask: what about all the models that rely on transformers.onnx? Is there a simple way to migrate?

I currently rely on some models that require transformers.onnx and they will probably not be updated and I want to evaluate my options.

@thilomichael check out optimum-onnx at https://github.com/huggingface/optimum-onnx and https://huggingface.co/docs/optimum-onnx/en/quickstart
Fix Qwen3-VL typos and clarify image/video/vision token descriptions
Author: stefgina
Pull request: yes

# What does this PR do?

hey, this PR fixes several small docstring issues in Qwen3-VL and adds a few clarifications:

- correction for the `videos` argument description ("image" → "video") in `Qwen3VLProcessor.__call__()`.
- fixes various other typos in the processing & configuration files.
- the description was wrong for `image_token_id`, `video_token_id`, and `vision_start/end_token_id`
  so I updated it to distinguish modality tokens from the vision segment delimiters properly.
 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@zucchini-nlp 
@yonigozlan 


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen3_vl

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43033). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
VLM reloaded config provide wrong model_type attribution
Author: xin3he
Pull request: no
Labels: name: bug

### System Info

transformers                4.57.3

After saving config and then reloading the saved config of "Qwen/Qwen2-VL-2B-Instruct", I observer there is a wrong value `model_type` present.
The `config.model_type` should be `qwen2_vl` but now is `qwen2_vl_text`.

### Who can help?

@zucchini-nlp @CyrilVallez

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

To reproduce, execute below code:

```
from transformers import Qwen2VLForConditionalGeneration, AutoConfig

# default: Load the model on the available device(s)
model = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-2B-Instruct", torch_dtype="auto", device_map="auto"
)

model.config.save_pretrained("./qwen_vl_test_model")
config = AutoConfig.from_pretrained("./qwen_vl_test_model")
print(config.model_type)  #qwen2_vl_text
print(config.to_dict()['model_type'])  #qwen2_vl
```


### Expected behavior

Below prints should have the same output: `qwen2_vl`
print(config.model_type)  #qwen2_vl_text
print(config.to_dict()['model_type'])  #qwen2_vl

Comments:
@xin3he it was fixed in https://github.com/huggingface/transformers/pull/41808 which should be included in the next patch release