noamaanMulla-03/transformers-issues huggingface dataset:
Qwen3-Next tokenizer fails to apply chat template for openai-style message inputs
Author: felixzhu555
Pull request: no
Labels: name: bug

### System Info

```
transformers env
- `transformers` version: 4.56.2
- Platform: Linux-5.4.0-1150-aws-fips-x86_64-with-glibc2.31
- Python version: 3.11.13
- Huggingface_hub version: 0.35.1
- Safetensors version: 0.6.2
- Accelerate version: not installed
- Accelerate config: not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.8.0+cu128 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA H100 80GB HBM3
```

### Who can help?

@ArthurZucker @itazap 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Running this code with [openai-style](https://github.com/openai/openai-python/blob/main/src/openai/types/chat/chat_completion_content_part_text_param.py) chat input
```
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-Next-80B-A3B-Instruct")

chat = [{'role': 'user', 'content': [{'type': 'text', 'text': 'hello world!'}]}]
print(tokenizer.apply_chat_template(chat, chat_template=None, tokenize=False))
```

yields
```

user



```
where we clearly see the actual message content is completely ignored by `apply_chat_template`.

Not sure if this is intentional, since the tokenizer for Qwen3-Next resolves to Qwen2TokenizerFast, which maybe just doesn't support this type of chat input?

### Expected behavior

I would expect the output to be 
```

user
hello world


```
i.e. the same as if the chat input is `[{'role': 'user', 'content': 'hello world'}]`.

Comments:
Hi @felixzhu555, when models don't support image inputs, it's common that they expect `content` to be a string rather than a list of text blocks. This isn't really a bug in Transformers - the chat template lives in the model repo, so you'd have to open a PR there if you want to fix it!
[generate] cache missing custom generate file
Author: gante
Pull request: yes

# What does this PR do?

Reorders code in `load_custom_generate` such that we load the `.py` file first, and check whether it is to be used later. This way, we can cache the result of the Hub request even when the file doesn't exist.

⚠️ Since we try to load `custom_generate` at model-loading time, this results in much fewer Hub requests in situations like CIs. Some downstream CIs were hitting rate limits, in part due to this feature's uncached requests prior to this change.

______________________

✅ ran `custom_generation` tests, including existing slow tests.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41216). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Using on a subset of PEFT tests, this significantly reduces the number of requests (by ~75%) :rocket:
Remove old Python code
Author: cyyever
Pull request: yes

# What does this PR do?

Remove code for python < 3.10

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: idefics3

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41226). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
:rotating_light: [`v5`] Remove headmasking
Author: vasqu
Pull request: yes
Labels: name: for_v5?

As per title, time to deprecate this

This includes anything in the docs, tests, and modeling
- `xxx head mask`
- `xxx_headmasking`
- `xxx_head_mask`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41076). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Currently, a lot of tests fail because of rate limits. Checked with everything before the rates change so should hopefully be good :crossed_fingers: 

Edit: works now

Any way you could fix all conflicts before review? 😬 Sorry, this is the cost of such big PRs, any other PR anywhere is likely to create conflicts 🥲

@Cyrilvallez no worries, getting used to it :sweat_smile: resolved the conflicts and updated the recent new models as well

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: albert, align, altclip, audio_spectrogram_transformer, autoformer, bark, bart
update code owners
Author: ydshieh
Pull request: yes

# What does this PR do?

Have to do this now

Comments:
@yonigozlan @molbap 

Please let me know if you want to be removed in some entries, otherwise give me a 👍 or ✅ so I am sure I can merge 🙏 .



Looks good to me 👍

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41221). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Unify is_torchvision_v2_available with is_torchvision_available
Author: cyyever
Pull request: yes

# What does this PR do?

The minimum recommended torchvision version for torch 2.2 is 0.17 .

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: deepseek_vl_hybrid, dpt, llava_onevision, owlv2, rt_detr, sam2_video, segformer

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41227). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix multi-video timestamp bug in Qwen-3-VL
Author: tim120526
Pull request: yes

# What does this PR do?

This PR fixes a timestamp placeholder error that occurs when multiple videos are included in a single text input.

In `Qwen3-VLProcessor`, frames are sampled from videos, and the relative timestamp placeholder `"<{curr_time:.1f} seconds>"` is constructed for each frame to indicate its corresponding time information. Previously, the code used the index `i` to retrieve the video metadata (`metadata = video_metadata[i]`), which only worked correctly for a single video input. When multiple videos were provided, the metadata of the first video was incorrectly used to generate timestamp placeholders for all the other videos, resulting in errors.

This PR replaces the index `i` with the correct video index `index` when retrieving metadata, ensuring that each video use the appropriate timestamp placeholders.

Hi @zach-huggingface @SunMarc,   I am just following up on this bug and the proposed fix.  It is a one-line change, and I would be happy to assist with any questions or further clarifications to help get it merged.

Thank you very much!


Comments:
> Thanks, we have a similar issue in [GLM4V](https://github.com/huggingface/transformers/blob/main/src/transformers/models/glm4v/processing_glm4v.py) with the index, can you include the fix as well?

Done. We have fixed the issue in `processing_glm4v.py`. 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, bloom, camembert, codegen, decision_transformer

> Thanks! Can you also make sure the commit history is correct and doesn't include unrelated commits and then run `make fix-copies`?
> 
> Btw GLM tests are failing `tests/models/glm4v/test_processor_glm4v.py::Glm4vProcessorTest::test_apply_chat_template_video_1`

Sorry, I closed the previous PR and opened a new one here: https://github.com/huggingface/transformers/pull/41229
. The GLM4V error has also been fixed.
Fix typing of train_args
Author: cyyever
Pull request: yes

# What does this PR do?

Some `Optional` typing is quite confusing because their default values is not `None`.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41142). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@Rocketknight1 I have fixed `fsdp` typing. Otherwise HfArgumentParser could generate `None` as its default value.
Fix sliding window attn mask
Author: remi-or
Pull request: yes

It was reporedt in #41184 that the sliding window attention mask has errors in CB. This PR fixes this and adds a test for the `build_attention_mask` function, so it does not happen again.


Comments:
@bot /style

Style bot fixed some files and pushed the changes.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41228). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Revert "Fix DeepSpeed mixed precision precedence over Accelerate defaults"
Author: SunMarc
Pull request: yes

Reverts huggingface/transformers#39856

Not the intended fix. This breaks the logic in deepspeed and makes things complicated to maintain. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41124). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[v5] More Training Args cleaning 
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR removes EvaluationStrategy as it is deprecated

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41131). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[v5] Remove tokenizer from Trainer
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR removes tokenizers from Trainer as it will be deprecated for v5. Since we removed it from the init, I think it should also make sense to remove the setter and the property method for tokenizer 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41128). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[v5] Remove deprecated prediction loop
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR removes the legacy prediction loop that was deprecated 4 years ago. I think that it should be fine to deprecate it even without any warning for v5 which it would have been better to have. I checked files on github and no one really seemed to use the legacy one. Also `SequentialDistributedSampler` was bound to be deprecated in v5. 

If we still prefer to have it for one version, happy to submit a PR for a deprecation msg. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41123). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[v5] Remove train kwargs
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR removes train kwargs. There were used for deprecated arguments but since we are removing them, we can also safely remove the kwargs. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41127). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Trainer] deprecate `num_train_tokens`
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR deprecates `num_train_tokens` as we already have `include_num_input_tokens_seen` that is better supported. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41165). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] Fix tp_plan
Author: stevhliu
Pull request: yes

Fixes https://github.com/huggingface/transformers/issues/41189 and removes mention of manual `tp_plan`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41205). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix qwen text config
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/41020 and ensures constency in Qwen-VL text config. Side note: prev we had flat dict structure in Qwen and for BC passed kwargs to `super()` and to `text_config`. This caused confusion in TRL which apparently resets some text attributes manually when training

In this PR, Qwen will set/get text related attributes only through text config. The attributes are obtainable from nested config as `config.text_config.vocab_size` and from root as `config.vocab_size` (BC)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41158). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Test failures are not related

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: glm4v, glm4v_moe, qwen2_5_vl, qwen2_vl
Responses API does not support input in (complex) format
Author: adner
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.56.0
- Platform: Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.34.4
- Safetensors version: 0.6.2
- Accelerate version: 1.10.1
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.8.0+cu128 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 3090

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am serving the **openai/gpt-oss-20b** model using `transformers serve` and is calling the experimental Responses API (/v1/responses). It works when the input attribute is a string, as in the example [here](https://huggingface.co/docs/transformers/main/serving#responses-api):

```
{
    "model": "openai/gpt-oss-20b",
    "stream": true,
    "input": "Tell me a three sentence bedtime story about a unicorn."
  }
```

But if the input is an array in the following format an exception is thrown:

```
{
"model":"openai/gpt-oss-20b",
"input":[{"type":"message","role":"user","content":[{"type":"input_text","text":"Tell me a joke!"}]}],
"stream":true
}
```

[Exception stack trace](https://github.com/user-attachments/files/22063435/stack.txt)

Note that the following also works currently:

```
{
    "model": "openai/gpt-oss-20b",
    "input": [
        {
            "role": "developer",
            "content": "Talk like a pirate."
        },
        {
            "role": "user",
            "content": "Are semicolons optional in JavaScript?"
        }
    ],
"stream":true
}
```

### Expected behavior

The expectation is that the Transformers Responses API should accept also this more complex format - in addition to the simpler formats that it already supports. Existing OpenAI Responses API clients (for example the [C# OpenAI SDK](https://github.com/openai/openai-dotnet)) serialize the messages to the more complex format, which means that these currently do not work with the Transformers Responses API.

Comments:
cc @gante I think! Maybe @lysandrejik?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
`BlenderbotForConditionalGeneration` errors out with `list index out of range`
Author: ariG23498
Pull request: no
Labels: name: bug

Running the following:
```py
from transformers import AutoTokenizer, BlenderbotForConditionalGeneration

mname = "facebook/blenderbot-400M-distill"
model = BlenderbotForConditionalGeneration.from_pretrained(mname)
tokenizer = AutoTokenizer.from_pretrained(mname)
UTTERANCE = "My friends are cool but they eat too many carbs."
print("Human: ", UTTERANCE)

inputs = tokenizer([UTTERANCE], return_tensors="pt")
reply_ids = model.generate(**inputs)
print("Bot: ", tokenizer.batch_decode(reply_ids, skip_special_tokens=True)[0])

REPLY = "I'm not sure"
print("Human: ", REPLY)

NEXT_UTTERANCE = (
    "My friends are cool but they eat too many carbs.
 
That's unfortunate. "
    "Are they trying to lose weight or are they just trying to be healthier?
 "
    "
 I'm not sure."
)
inputs = tokenizer([NEXT_UTTERANCE], return_tensors="pt")
next_reply_ids = model.generate(**inputs)
print("Bot: ", tokenizer.batch_decode(next_reply_ids, skip_special_tokens=True)[0])
```

Fails with 

```
IndexError: list index out of range
```

[Colab notebook](https://colab.research.google.com/gist/ariG23498/063c2888c41afebb71ca614278161445/scratchpad.ipynb)

Should work as expected. The code has been taken from the [official docs](https://huggingface.co/docs/transformers/en/model_doc/blenderbot#transformers.BlenderbotForConditionalGeneration)

Comments:
Pretty sure it's due to the cache being initialised with kwargs there, i.e. the number of initialized cache layers is wrong

cc @manueldeprada @Cyrilvallez 

having a look


Happens because `num_hidden_layers` is wrong in the config: It should be same as `decoder_layers` but it coincides with `encoder_layers`

@gante since you dealt with a similar problem in https://github.com/huggingface/transformers/pull/40277, do you have any suggestion other than opening a PR [in the Hub?](https://huggingface.co/facebook/blenderbot-400M-distill/blob/main/config.json)?

yes, working on it (#40553 ) :D

TL;DR we need one abstraction to pull the encoder sub-config (any modality, not just text) from the full config, to then initialize the encoder cache correctly. There were other name clashes too.
Fix white space in documentation
Author: cyyever
Pull request: yes

# What does this PR do?

As the title says.
## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface, @SunMarc and @qgallouedec
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41157). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@stevhliu Fixed
Video processor accepts single frames on cuda
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/40867 and allows input videos to be a "3D tensor on cuda device"

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41218). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Since Pavel left, I will be bugging you with reviews for vision @yonigozlan 😄 

> [why was is np if input is in torch](https://github.com/huggingface/transformers/pull/41218/commits/e78ce19f349e696320b3a3516b1829c3b94d8cf4)

Fair question 😂
[docs] update tips syntax
Author: mishig25
Pull request: yes

see https://github.com/huggingface/doc-builder?tab=readme-ov-file#tip

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41160). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker @ydshieh @LysandreJik could any of the maintainers run `make quality` and merge it? thanks a lot
Use math.log2
Author: cyyever
Pull request: yes

# What does this PR do?

This PR changes `math.log(XXX, 2)` to `math.log2(XXX)`, to make the invocation more clear.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: phi4_multimodal, swin2sr

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41241). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[V5] Remove deprecated transformers.onnx
Author: cyyever
Pull request: yes
Labels: name: for_v5?

# What does this PR do?

Remove the source files, doc and the dependencies.

Comments:
Thank you @cyyever ! 

Let me know if you need helps for the failing CI.

I added the `for_v5?` label.



@ydshieh I need to quick eat the chicken....at hand

@ydshieh The current CI failure is unrelated.

Thanks for the ping @ydshieh ! It makes sense to remove everything (`OnnxConfig` included), let me know if you need help on this @cyyever 

@echarlaix To remove `ONNXConfig`, how do we deal with the model using them?

We can handle `ONNXConfig` in a separate PR if we decide to go for it (which is more involved).

Thanks again!

@ydshieh I removed ONNX from docs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41214). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.