noamaanMulla-03/transformers-issues huggingface dataset:
Make DataCollatorMixin handle BatchFeature
Author: jp1924
Pull request: yes

# What does this PR do?

Modify so that if the batch returned from DataCollatorMixin is a dict, it is converted to BatchFeature and output.
In most learning and projects, instead of returning the value returned from processing_class performed in pt_call or np_call as is, there are many cases where an additional dict is made and returned, and there are many times when the functionality provided by BatchFeature cannot be used.
So, I tweaked the collator so that if the batch is a simple dict, it can be converted to BatchFeature.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@SunMarc


Comments:
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43430&sha=3fa345

@SunMarc 

Looking at this failing in the test, originally it should have gone through np's shape and returned as a tuple, 
but since it changed to pt's tensor, it's returning as a torch.Size object and causing an assert error? 

Hmn - interesting idea, but the specific object being returned can vary. It's sometimes `BatchEncoding` rather than `BatchFeature`, so I think this PR will be quite breaking. The cases when an extra key is added and a dict is returned may also break the batch assumptions for BatchFeature/BatchEncoding, so this definitely needs some extra testing!

@Rocketknight1 Hmm... Yeah, since the Collator is used in various areas, 
if I add it carelessly, it could cause quite a bit of confusion. 
It seems right not to add it separately. 
It would be better to make the user return it as the necessary BatchFeature.
Qwen3 tokenizer wrong offset_mapping
Author: contribcode
Pull request: no
Labels: name: bug

### System Info

transformers 4.53.2, Ubuntu 22.04.4, python 3.11.13

### Who can help?

@ArthurZucker and @itazap There must be a problem with the `offset_mapping` of Qwen3 `tokenizer`. The starting point in the text for each token, except the first and the last, is one position behind. I compared it with the BERT's `tokenizer`, which produces what is expected:

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
sample_text='A girl is styling her hair.'
bert_tokenizer = BertTokenizerFast.from_pretrained('google-bert/bert-base-cased')
bert_encoding = bert_tokenizer(
    text=sample_text, add_special_tokens=False, return_offsets_mapping=True
)
print(bert_encoding['offset_mapping'])
qwen_tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen3-Embedding-0.6B')
qwen_encoding = qwen_tokenizer(
    text=sample_text, add_special_tokens=False, return_offsets_mapping=True
)
print(qwen_encoding['offset_mapping'])
```

### Expected behavior

[(0, 1), (2, 6), (7, 9), (10, 17), (18, 21), (22, 26), (26, 27)]
[(0, 1), (1, 6), (6, 9), (9, 17), (17, 21), (21, 26), (26, 27)]

Comments:
Hi @ArthurZucker and @itazap 👋

Just following up to see if this tokenizer `offset_mapping` issue could be reviewed when you get a chance. It seems like the positions returned by the Qwen3 tokenizer are shifted compared to standard behavior (e.g., BERT), as detailed above.

Happy to help test a fix or provide more examples if needed!


The tokenizer is based of `gpt2`, and GPT2 tokenizer has been around for quite a while! Are you using `tokenizers` ? 

I am using the tokenizers of the `transformers` library.

`from transformers import AutoTokenizer, BertTokenizerFast`

I could use `AutoTokenizer` for both models and it should be the same.

<img width="530" height="100" alt="Image" src="https://github.com/user-attachments/assets/f175c9b1-6b57-4bc8-a945-e05ec9959a73" />
Given the difference in encodings, this is expected.

hi @contribcode, was this problem solved for you?

thanks in advance

It turns out this is not a problem; as ArthurZucker points out, it’s expected behavior. The Qwen tokenizer adds the `Ġ` character to every word that is preceded by a space. I handled this manually in the code.
Fix Pixtral with flash attention
Author: zucchini-nlp
Pull request: yes

# What does this PR do?
Fixes https://github.com/huggingface/transformers/issues/43441

Pixtral always has packed inputs because inputs images are of different lengths. However position ids was a 1D tensor causing an issue in FA2. Unsqueezing a bs of 1 helps to solve the issue

Also, tiny bug in `torch._check`. Having it as `def msg(): return msg` will not resolve back to a string due to naming, so I renamed it to `msg_callable`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43443). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@bot /repo

Repo. Consistency fix is beginning .... [View the workflow run here](https://github.com/huggingface/transformers/actions/runs/21287360165).

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: chameleon, fast_vlm, lighton_ocr, llava, llava_next, llava_next_video, paddleocr_vl, paligemma, paligemma2, perception_lm, pixtral, qwen2_5_vl, qwen2_vl, qwen3_vl_moe, video_llama_3, video_llava
[BUG] Ministral-3 fails with FlashAttention in Transformers v5 RC
Author: dysby
Pull request: no
Labels: name: bug

### System Info

**Description**

In Transformers `5.0.0rc3`, running `mistralai/Ministral-3-8B-Instruct-2512` with FlashAttention enabled results in an `IndexError` during the attention forward pass.

This issue does not occur when FlashAttention is disabled.

---

**Model**

* `mistralai/Ministral-3-8B-Instruct-2512`

---

**Environment**

```text
- GPUs: 1
- PyTorch: 2.6.0+cu126
- Transformers: 5.0.0rc3
```

---

**Stack Trace**

```text
File "modeling_pixtral.py", line 247, in forward
    attn_output, attn_weights = attention_interface(
                                ^^^^^^^^^^^^^^^^^^^^
File ".../transformers/integrations/flash_attention.py", line 73, in flash_attention_forward
    attn_output = _flash_attention_forward(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../transformers/modeling_flash_attention_utils.py", line 638, in _flash_attention_forward
    is_fa_with_position_ids = _is_packed_sequence(
                              ^^^^^^^^^^^^^^^^^^^^
File ".../transformers/modeling_flash_attention_utils.py", line 447, in _is_packed_sequence
    torch.arange(position_ids.shape[1], device=position_ids.device) + position_ids.min()
                 ~~~~~~~~~~~~~~~~~~^^^
IndexError: tuple index out of range
```

---

**Expected behavior**

* Model executes successfully with FlashAttention enabled.

---

**Actual behavior**

* Execution fails with `IndexError` during the attention forward pass.

### Who can help?

@zucchini-nlp 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

demo code from model doc
+
```python
model = Mistral3ForConditionalGeneration.from_pretrained(model_id, 
                           device_map="cuda", 
                           attn_implementation="flash_attention_2")
```


### Expected behavior

**Expected behavior**

* Model executes successfully with FlashAttention enabled.

---

**Actual behavior**

* Execution fails with `IndexError` during the attention forward pass.

Comments:
I see, issue with pixtral because it is always packed. Fixing it
Fix lm_head weight tying in Mistral3ForConditionalGeneration
Author: aswin00000
Pull request: yes

## Problem
Ministral-3-3B-Instruct-2512 generates gibberish because `lm_head.weight` is not tied to `embed_tokens.weight`.

## Root Cause  
The Ministral-3-3B config has `text_config.tie_word_embeddings=True`, and the checkpoint only stores `model.language_model.embed_tokens.weight` (no `lm_head.weight`).  
However, for Mistral3 the top-level `config.tie_word_embeddings` was never set, so the base tying logic that relies on `_tied_weights_keys` was effectively disabled and `lm_head.weight` was randomly initialized.

## Solution
Instead of overriding `tie_weights()`, set `tie_word_embeddings=True` by default in `Mistral3Config` so that the existing weight-tying mechanism (`_tied_weights_keys = {"lm_head.weight": "model.language_model.embed_tokens.weight"}`) is applied correctly to Mistral3-based VLMs.

## Changes
- Updated `src/transformers/models/mistral3/configuration_mistral3.py`:
  - Added a `tie_word_embeddings: bool = True` argument to `Mistral3Config.__init__`
  - Stored it as `self.tie_word_embeddings`

Comments:
Requesting review from @zucchini-nlp as discussed in the original bug report.
   ;;  https://github.com/huggingface/transformers/issues/43404
This PR fixes the weight tying issue that was causing gibberish output in Ministral-3-3B-Instruct-2512.
## Summary
- **Problem**: Mistral3 models generate gibberish due to improper weight tying
- **Root Cause**: Base class only checked `config.tie_word_embeddings` but multimodal models use `config.text_config.tie_word_embeddings`
- **Solution**: Added custom `tie_weights()` method that checks both locations

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: mistral3

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43429). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zucchini-nlp

I’m a junior engineer and this is one of my first OSS contributions

If this change looks good, If it’s not too much trouble, even a short written confirmation (here or by email) that this fix was useful would mean a lot to me as I’m building my career.

Thank you again for the reviews!  

aswinmraju2002@gmail.com

Thanks for the PR @aswin00000 ! LGTM at this point and I will merge it when CI turns green, the failing tests are just flake and not related :)

Ty both!
Wrong tokenizer decoder type in Transformers v5
Author: awni
Pull request: no
Labels: name: bug

### System Info

Wrong decoder type with `5.0.0rc1`.

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run this:
```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Llama-8B")
print(tokenizer.decoder)
```

You get:
```
Sequence(decoders=[Replace(pattern=String("▁"), content=" "), ByteFallback(), Fuse(), Strip(content=" ", start=1, stop=0)])
```

with Transformers v5.

And with Transformers 4.57.3 and earlier you get:

```
ByteLevel(add_prefix_space=True, trim_offsets=True, use_regex=True)
```

Is it expected that this changed?

### Expected behavior

The same decoder type as transformers 4.57.3

Comments:
+1

So the issue seems to be that in 4.57.3 we just loaded the  file which specifies the tokenizer . 
In v5 we seem to be just manually generating the tokenizer instead of trusting the file . And so we're overwriting the tokenizer for Llama 3 . 
I'm not sure what's the correct fix but a hackish fix is to check for vocab and then create the tokenizers with respect to that .
I have a implemented this fix but it does include iterating the vocab via itertools . 

Sounds great, did you make a PR with the fix?

the fix is in #42894 we default to not using llama anymore it was kind of a proxy

Thanks!!

Closing as well
bring back clean_up_tokenization_spaces to tokenizers backend
Author: itazap
Pull request: yes

bring back clean_up_tokenization_spaces to tokenizers backend!  for v5

https://github.com/huggingface/transformers/pull/42916
https://github.com/huggingface/transformers/issues/42913
https://github.com/huggingface/transformers/pull/42900
https://github.com/huggingface/transformers/issues/42898



Comments:
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43426&sha=780e67

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43426). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Outstanding issues / PR before we can release v5
Author: ArthurZucker
Pull request: no

- #42697 actual perf upgrade 
- #34919 default dtype
- #42513
- #42563 final form and break for tokenization
- #42558 gradient checkpointing refactor
- #42555 shard size default
- #42491 PEFT + MOE refactor
- #41388 default to fast image processors
- #42418 
- #42894
- #42564

Comments:
can we include https://github.com/huggingface/transformers/pull/43303 too?

And 

- https://github.com/huggingface/transformers/pull/42845

too ? It's a clean up PR to definitely get rid of `requests` (removing 1 required dependency now that we have httpx)

https://github.com/huggingface/transformers/pull/43325 might be intended to be included in v5 as well, cc @Rocketknight1 

- Tom Aarsen

Yes, good point! I'll try to get that merged ASAP
Fix whisper tests
Author: eustlb
Pull request: yes

# What does this PR do?



Comments:
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43291&sha=70ba3a

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43291). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: whisper

This comment contains `run-slow`, running the specified jobs: 

models: ["models/whisper"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/21286768482)

### Model CI Report

#### ❌ Failed tests

- [whisper](https://github.com/huggingface/transformers/actions/runs/21286768482/job/61270343053):
    tests/models/whisper/test_tokenization_whisper.py::WhisperTokenizerTest::test_combine_tokens_into_words
    tests/models/whisper/test_tokenization_whisper.py::WhisperTokenizerTest::test_decode_asr_with_word_level_timestamps
    tests/models/whisper/test_tokenization_whisper.py::WhisperTokenizerTest::test_skip_special_tokens_with_timestamps
    tests/models/whisper/test_tokenization_whisper.py::SpeechToTextTokenizerMultilinguialTest::test_offset_decoding



**[For maintainers]** Suggested jobs to run (before merge)

run-slow: whisper
`_get_num_multimodal_tokens` is not implemented for model `mllama`
Author: mrtpk
Pull request: no
Labels: name: bug

vLLM 0.11’s Transformers-backend expects the HF processor to implement a method called `_get_num_multimodal_tokens` which is [not implemented for mllama](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mllama/processing_mllama.py) in `transformers 4.57.1`.

Because of this, `vllm serve meta-llama/Llama-3.2-11B-Vision` fails on  `vllm 0.11.0`. It works on `vllm 0.10.2`.
The error is  `MllamaProcessor' object has no attribute '_get_num_multimodal_tokens`.

## Related 
https://github.com/vllm-project/vllm/issues/27198

### Who can help?

Tagging @yonigozlan @molbap @zucchini-nlp for input — happy to implement the method if no one’s on it yet, and I’d appreciate your guidance.

### Reproduction

```
from transformers import AutoProcessor
proc = AutoProcessor.from_pretrained("meta-llama/Llama-3.2-11B-Vision-Instruct")
print(hasattr(proc, "_get_num_multimodal_tokens"))  # should be True but not
```

### Expected behavior

Implement `_get_num_multimodal_tokens` as it is implemented for other models in `./src/transformers/models/` (like `gemma3`.

## Useful links
* https://huggingface.co/docs/transformers/main/en/transformers_as_backend#multimodal-models
* https://blog.vllm.ai/2025/04/11/transformers-backend.html

Comments:
@mrtpk hey 👋🏻 

The Llama-Vision does not support it on purpose. The model has a different fusion method compared to other VLMs, i.e. it uses cross-attention instead of merging image embeddings with text. Currently the vLLM backend doesn't support this type of multimodal models

We can implement the method in processor, so it can be used for inference/training. We can't however enable vLLM easily right now

Hello @zucchini-nlp,

Thank you very much for the prompt response. I understand why the support in the vLLM was dropped.
Nevertheless, I would love to contribute if it brings value.

@mrtpk thanks! I am not sure what changed we'd need to push cross attention VLMs into the vLLM currently. The helper fn `_get_num_multimodal_tokens` itself will not be needed, as the model does not add any placeholders

Also, the field has moved to merged embedding largely, so ig cross attention models will not be released much. If you would like to contribute to "VLMxvLLM", we are planning to add video support and it might need some help. I haven't had much time to work on it due ti low priority, but I'll tag you here if there are any community-open tasks :)

Absolutely, thanks! Looking forward to your tag.
Closing this one for now.
[SAM3 Video] get_text_features returns CLIPTextModelOutput but code expects tensor
Author: ncerny
Pull request: no

### System Info

  - `transformers` version: main branch
  - Platform: Linux (CUDA)
  - Python version: 3.11

  ### Who can help?

  @yonigozlan @amyeroberts

  ### Reproduction

  When using `Sam3VideoModel` with text prompts:

  AttributeError: 'CLIPTextModelOutput' object has no attribute 'shape'

  **Root Cause:** In `models/sam3_video/modeling_sam3_video.py` line ~590:

  ```python
  text_embeds = self.detector_model.get_text_features(...)

  Returns CLIPTextModelOutput, but code expects a tensor. Later at models/sam3/modeling_sam3.py:2330:

  if text_features.shape[0] == 1  # Fails - CLIPTextModelOutput has no .shape

  Fix

  -                )
  +                ).pooler_output
```

I can submit a PR if helpful.

Comments:
Sure, would be nice to have full repro but feel free to open a PR!

Related issue and caused by the same PR: https://github.com/huggingface/transformers/issues/43475. If you open a PR, pls include this one as well since it's a similar issue in the same model class
Fix Mxfp4 dequantize
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. Fix https://github.com/huggingface/transformers/issues/43317.
They move the devices without being asked to do it, resulting in OOMs when using with a device_map="auto". Also, no need to upscale the dtype to int64 as they are only used to indexing and it takes up more memory in constrained device_map="auto" settings

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43326). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43326&sha=918d64
Perception Encoder follow up PR
Author: eustlb
Pull request: yes

# What does this PR do?


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43492). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: pe_audio, pe_audio_video, pe_video
fix:auto tokenizer pretrainedconfig
Author: ansh-info
Pull request: yes

# What does this PR do?

This PR fixes a crash in `AutoTokenizer.from_pretrained` when `AutoConfig` fails and the code falls back to `PreTrainedConfig`. The fallback path incorrectly calls `.get()` on a config object, which raises `AttributeError` and breaks cached/tokenizer reload flows (notably with `trust_remote_code=True`).

The fix uses `getattr(config, "model_type", None)` instead, preserving the previous behavior for configs with a `model_type` while avoiding the crash in the fallback case. A regression test is included to cover the `PreTrainedConfig` fallback path.

Context:

- In some cached or auth-influenced runs with trust_remote_code=True,
    AutoTokenizer.from_pretrained can fall back to PreTrainedConfig.
- The current code calls config.get("model_type", None), which raises
    AttributeError because PreTrainedConfig is not a dict.
- This change uses getattr(config, "model_type", None) instead, keeping behavior
    intact for normal configs and fixing the crash.

Tests:

- uv run -m pytest tests/models/auto/test_tokenization_auto.py -k pretrained_config
- uv run -m pytest tests/models/auto/test_tokenization_auto.py (note: one flaky/
    auth‑dependent test can fail with 401 when not authenticated)

Files:

- src/transformers/models/auto/tokenization_auto.py
- tests/models/auto/test_tokenization_auto.py

Fixes # (issue)

#43272 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

- text models: @ArthurZucker  @Cyrilvallez 


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43289&sha=409913
Refactor-tokenization-more
Author: ArthurZucker
Pull request: yes

# What does this PR do?
 1. Update `TokenizersBackend` to simplify converting from sentencepiece. 
 2. Add the `model` attribute to guide the conversion. 
 3. Adds the `convert_from_spm` for special cases when the spm is not straight forward. 
4. Simplify init for TokenizersBackend to support back passing vocab file / merges file path.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42563). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: albert, auto, barthez, bert, big_bird, blenderbot, camembert, clip, clvp, code_llama, codegen, cohere, deberta, deberta_v2, distilbert, dpr