noamaanMulla-03/transformers-issues huggingface dataset:
Accuracy regression in DeepSeek-R1-Distill-Llama-8B after upgrading Transformers from 4.55 to 5.9
Author: xiaowangintel
Pull request: no
Labels: name: bug

### System Info

transformers: 4.55 vs 5.9.0
PyTorch: 2.13

### Who can help?

@ArthurZucker and @itazap

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction


```
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "deepseek-ai/DeepSeek-R1-Distill-Llama-8B"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
).cuda()

messages = [
    {"role": "user", "content": "What is 1+1? Answer briefly."}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.6,
        top_p=0.95,
        do_sample=True
    )
input_len = inputs["input_ids"].shape[-1]
response = tokenizer.decode(
    outputs[0][input_len:],
    skip_special_tokens=True
)
print(response)
```

### Expected behavior

v4.55 response:
```
First, I recognize that the question is asking for the sum of 1 and 1.

Next, I identify the numbers involved in the addition, which are both 1.

Then, I perform the addition: 1 plus 1 equals 2.

Finally, I conclude that the correct answer is 2.
</think>

**Solution:**

To find the sum of 1 and 1, follow these easy steps:

1. **Identify the numbers to add:**
   - First number: 1
   - Second number: 1

2. **Add the numbers together:**
   \[
   1 + 1 = 2
   \]

3. **State the final answer:**
   \[
   \boxed{2}
   \]
```
v5.9.0 response:
```
ĊIĠneedĠtoĠdetermineĠtheĠsumĠofĠ1ĠandĠ1.ĊĊAddingĠ1ĠandĠ1ĠgivesĠaĠtotalĠofĠ2.Ċ</think>ĊĊ**Solution:**ĊĊToĠfindĠtheĠsumĠofĠ1ĠandĠ1,ĠfollowĠtheseĠsimpleĠsteps:ĊĊ1.Ġ**IdentifyĠtheĠnumbersĠtoĠadd:**ĊĠĠĠ-Ġ**FirstĠnumber:**Ġ1ĊĠĠĠ-Ġ**SecondĠnumber:**Ġ1ĊĊ2.Ġ**PerformĠtheĠaddition:**ĊĠĠĠ\[ĊĠĠĠ1Ġ+Ġ1Ġ=Ġ2ĊĠĠĠ\]ĊĊ**FinalĠAnswer:**Ġ\(\boxed{2}\)
```

Comments:
The tokenizer_config.json of DeepSeek-R1-Distill-Llama-8B specifies LlamaTokenizerFast. However, in Transformers 5.9, the definition file for LlamaTokenizerFast was removed, causing the tokenizer to fall back to LlamaTokenizer by default. Will there be any future updates or maintenance for DeepSeek-R1-Distill-Llama-8B?

will be fixed by https://github.com/huggingface/transformers/pull/46091 will comment soon when its merged

Closing the issue, as the problem has already been fixed.
Add Susono model
Author: puwaer
Pull request: yes

# What does this PR do?

This PR adds **Susono**, a decoder-only causal language model, to the library.

Susono extends the Qwen3-Next architecture (hybrid full-attention + GatedDeltaNet linear attention, MoE feed-forward) with two additional components:

- **Engram** — a deterministic N-gram hash-lookup conditional memory injected at selected layers (`engram_layer_ids`). It reads `input_ids` and is disabled automatically when only `inputs_embeds` is provided.
- **mHC-Lite** — `mhc_num_streams` parallel residual streams combined per layer via a learnable convex combination of permutation matrices (spans the Birkhoff polytope without iterative Sinkhorn projection).

With `use_engram=False` and `use_mhc=False`, the model is functionally equivalent to Qwen3-Next. Other features: configurable QK-LayerNorm (`qk_layernorm`), an optional attention output gate (`attention_output_gate`), a MoE shared-expert gate with a learnable bias, and a `full_attention_interval`-driven hybrid layer schedule.

## Implementation notes

- Implemented with **modular transformers** (`modular_susono.py`), reusing Qwen3-Next for the shared backbone; only the Engram, mHC-Lite, GELU-MLP and the configurable attention are written out. `modeling_susono.py` is generated from the modular file.
- Registered in the auto classes (config, base model, causal-LM, sequence/token classification, question answering).
- Verified that the generated model state-dict keys exactly match a real 10B checkpoint (616/616 keys, no missing/extra).
- Pre-trained checkpoints and specialized variants are available for verification:
  - **Base model:** [puwaer/Susono-10B-A1B-Base](https://huggingface.co/puwaer/Susono-10B-A1B-Base)
  - **Instruct model:** [puwaer/Susono-10B-A1B-Instruct](https://huggingface.co/puwaer/Susono-10B-A1B-Instruct)
  - **Thinking model:** [puwaer/Susono-10B-A1B-Thinking](https://huggingface.co/puwaer/Susono-10B-A1B-Thinking)

## Before submitting

- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request)?
- [x] Was this discussed/approved via a GitHub issue or the forum? - [x] Did you make sure to update the documentation? (`docs/source/en/model_doc/susono.md`)
- [x] Did you write any new necessary tests? (`tests/models/susono/`)

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, susono

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46685)

> ⚠️ **Code quality check failed** — all test jobs were skipped. Fix the code quality issues and push again to run tests.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46685&sha=380394

Hey! We generally only want to add models to Transformers after they have significant usage. For personal models like this, it's probably better to upload the code to the Hub repo and use `trust_remote_code=True`!
docs: update apple silicon doc with safetensors `0.8.0` benefits
Author: McPatate
Pull request: yes

Lmk if that's good enough or if you need more deets!

cf https://github.com/huggingface/transformers/pull/46523

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46744)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46744). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Avoid NumPy 2.0 `__array__` copy-keyword deprecation in `create_mm_token_type_ids`
Author: qgallouedec
Pull request: yes

`ProcessorMixin.create_mm_token_type_ids` does `np.array(tokenizer_input)` on each row of `input_ids`. Several processors (e.g. Qwen2.5-VL, Gemma4) call it with `input_ids` already as a tensor, and `np.array(tensor)` triggers NumPy 2.0's deprecation because torch's `__array__` does not accept the `copy` keyword. This fires once per example, flooding logs in any VLM workload (e.g. SFT over an image dataset). (Processors that pass a plain list, e.g. Gemma3, are unaffected.)

Fix: convert non-list rows to a plain list (`.tolist()`) before `np.array(...)`, so the `__array__` path is never taken. Output is unchanged.

## MRE

```python
import warnings, numpy as np, torch

warnings.simplefilter("error", DeprecationWarning)
np.array(torch.tensor([1, 2, 3]))
# DeprecationWarning: __array__ implementation doesn't accept a copy keyword, ...
```

In practice it is reached via a VLM processor that passes tensor `input_ids` to `create_mm_token_type_ids`:

```python
import warnings
from PIL import Image
from transformers import AutoProcessor

warnings.simplefilter("error", DeprecationWarning)
img = Image.new("RGB", (32, 32))
proc = AutoProcessor.from_pretrained("trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration")
proc(text="


 hi", images=img, return_tensors="pt")
```

```
Traceback (most recent call last):
  File "/fsx/qgallouedec/transformers/repro.py", line 8, in <module>
    proc(text="


 hi", images=img, return_tensors="pt")
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/processing_utils.py", line 693, in __call__
    text_inputs["mm_token_type_ids"] = self.create_mm_token_type_ids(text_inputs["input_ids"])
                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/processing_utils.py", line 928, in create_mm_token_type_ids
    tokenizer_input = np.array(tokenizer_input)
DeprecationWarning: __array__ implementation doesn't accept a copy keyword, so passing copy=False failed. __array__ must implement 'dtype' and 'copy' keyword arguments. To learn more, see the migration guide https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword
Segmentation fault (core dumped)
```


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46827). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46827)
fix: update tokenizers version range to include stable 0.23.1
Author: kamran-nizamani
Pull request: yes

## Summary
This PR updates the `tokenizers` dependency version range to correctly include the stable 0.23.1 release while avoiding the non-functional 0.23.0 RC.

## Problem
The current constraint:
`tokenizers>=0.22.0,<=0.23.0`

incorrectly allows a version (0.23.0) that was never a proper stable release.

## Solution
Updated version constraint to:
`tokenizers>=0.22.0,<0.24.0`

## Why this change
- Ensures compatibility with stable tokenizers releases
- Avoids broken/RC version (0.23.0)
- Keeps forward compatibility within minor versions
- No breaking changes

## Impact
- No functional changes to transformers code
- Only dependency constraint improvement
- Safe and backward compatible

Comments:
Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/27979430077) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

We'll handle version bumps internally!
Enhance apply_chat_template with overloads for type checking
Author: boringcrypto
Pull request: yes

Added overloads for apply_chat_template method for type checking to support different return types based on tokenize and return_dict parameters.

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46840)

> ⚠️ **Code quality check failed** — all test jobs were skipped. Fix the code quality issues and push again to run tests.

Never mind I guess. It would be nice to not get type errors, but it seems you only accept improvements that are some form of "perfect". Since I don't have a degree in "transformers PR submission", there's not much more I can do.
Add Model Architecture for MiniCPM3
Author: bzantium
Pull request: no
Labels: name: New model

### Model description

This is a feature request to add a new model architecture for MiniCPM3, a powerful, lightweight language model.

MiniCPM3 is the third generation of the MiniCPM series. It demonstrates performance comparable to or exceeding many 7B-9B models, despite its smaller size. It excels in both English and Chinese, with advanced capabilities like function calling and code interpretation. The model is built on the Transformer architecture and introduces several innovative features to enable its impressive performance.

Adding a standard architecture for MLA dense models like MiniCPM3 would be a valuable addition to the transformers library.

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

https://huggingface.co/openbmb/MiniCPM3-4B
https://github.com/OpenBMB/MiniCPM/
https://arxiv.org/abs/2404.06395

Comments:
Hi @bzantium,

  I noticed PR #41116 has been inactive since October 2025. Are you still planning to continue
  working on it?

  If not, I'd like to pick this up and submit a fresh PR with MiniCPM3 support using the current
  modular model pattern (inheriting MLA attention from DeepSeek-V2 and config/MLP from Llama). I
  have a working implementation with tests ready.

  Let me know!
Fix VideoPrismForVideoClassification returning last_hidden_state as h…
Author: sharmax-vikas
Pull request: yes

## What does this PR do?

Fixes a bug in `VideoPrismForVideoClassification.forward()` where `ImageClassifierOutput.hidden_states`
was populated with `vision_model_outputs.last_hidden_state` (a raw `torch.FloatTensor` tensor) instead
of `vision_model_outputs.hidden_states` (`tuple | None`). The `attentions` field was also missing from
the return entirely.

Fixes #46829

This violates the `ImageClassifierOutput` type contract:

```python
hidden_states: tuple[torch.FloatTensor, ...] | None = None
attentions: tuple[torch.FloatTensor, ...] | None = None
```

The sibling class `VideoPrismVideoModel.forward()` already handles this correctly — the fix brings
`VideoPrismForVideoClassification` in line with it and with all other vision classification models
(BEiT, AST, etc.).

### Changes
- `modular_videoprism.py`: corrected hidden_states field, added attentions field
- `modeling_videoprism.py`: same fix in the generated file
- `test_modeling_videoprism.py`: updated create_and_check_model to assert hidden_states and
  attentions are None by default; updated 3 stale @unittest.skip reason strings

### Tests

---
**Checklist:**
- ☑ I confirm that this is not a pure code agent PR.
- ☑ Did you read the contributor guideline
- ☑ Did you write any new necessary tests

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: videoprism

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46830)

run-slow: videoprism

just sanity checking with slow ci

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28025878824)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/videoprism"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28025878824)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [0d2cff0e](https://github.com/huggingface/transformers/commit/0d2cff0ee8c6ae73ea1dd3eefb80ec9cfbb98e35) | workflow commit (merge commit) |
| PR | [d41d7611](https://github.com/sharmax-vikas/transformers/commit/d41d761138d80c8a4c9fb6ef14a9f72d79beb99c) | branch commit (from PR) |
| main | [43639365](https://github.com/huggingface/transformers/commit/4363936534ef47721b84a95bfdc758f056c3ee98) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46830). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Bug] VideoPrismForVideoClassification returns last_hidden_state tensor as hidden_states in ImageClassifierOutput
Author: sharmax-vikas
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 5.13.0.dev0 (main branch, local dev install)
- Platform: Linux (WSL2, kernel 6.6.114.1-microsoft-standard-WSL2)
- Python version: 3.14.4
- PyTorch version: 2.12.1+

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
import torch
from transformers import VideoPrismVisionConfig
from transformers.models.videoprism.modeling_videoprism import VideoPrismForVideoClassification

config = VideoPrismVisionConfig(num_labels=4)
model = VideoPrismForVideoClassification._from_config(config)
model.eval()

pixel_values = torch.randn(1, 2, 3, 288, 288)
with torch.no_grad():
    outputs = model(pixel_values_videos=pixel_values)

print(type(outputs.hidden_states))
# Output:  <class 'torch.Tensor'>

# Should be None or tuple[torch.FloatTensor, ...] per ImageClassifierOutput contract
assert outputs.hidden_states is None or isinstance(outputs.hidden_states, tuple), (
    f"FAIL: hidden_states is {type(outputs.hidden_states)}, expected None or tuple"
)
# AssertionError: FAIL: hidden_states is <class 'torch.Tensor'>, expected None or tuple
```

### Expected behavior

outputs.hidden_states should be None by default (when output_hidden_states=False),
matching the ImageClassifierOutput type contract:

    hidden_states: tuple[torch.FloatTensor, ...] | None = None
    attentions: tuple[torch.FloatTensor, ...] | None = None

Currently the model incorrectly returns vision_model_outputs.last_hidden_state
(a raw tensor) as hidden_states, and omits attentions entirely.

The sibling class VideoPrismVideoModel.forward() already handles this correctly
by passing vision_model_outputs.hidden_states and vision_model_outputs.attentions.

Comments:
Fix mask when generating beyond sliding window
Author: zucchini-nlp
Pull request: yes

# What does this PR do?


Slow test `test_generate_beyond_sliding_window` should have checked it. Found why it didn't fire and fixed the test, it was failing to compile on CI runners (fixed by disabling autocompile). And we had a "too long prefill length", so it didn't check when input slowly approaches max sliding length

cc @kashif 

Comments:
thanks! checking

run-slow: diffusion_gemma

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28020257896)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/diffusion_gemma"]
quantizations: []

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46839). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28020257896)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [e1bdeb94](https://github.com/huggingface/transformers/commit/e1bdeb9443cfae42480ba7d7da3fe06eb48d0700) | workflow commit (merge commit) |
| PR | [17919390](https://github.com/zucchini-nlp/transformers/commit/1791939060d814b879d1bcf8e63c6c8aea3eecf2) | branch commit (from PR) |
| main | [decfe6a7](https://github.com/huggingface/transformers/commit/decfe6a72ee7c47d52735db30c8bab704d0156e2) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


For the model tester, suggest adding a short cross-ref so the input length isn't changed without re-checking this test, e.g. on `seq_length`:

```python
seq_length=25,  # don't change without re-checking test_generate_beyond_sliding_window (sliding-window mask)
```

On the sanity check: I ran the released 26B-A4B checkpoint through block diffusion generation on this branch and it decodes coherent text, and static vs dynamic cache match bit for bit on the tiny model. The mask change looks good end to end.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: diffusion_gemma

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46839)
fix: raise `ValueError` for empty conversation in `apply_chat_template`
Author: sharmax-vikas
Pull request: yes

Fixes #46752

---

`apply_chat_template` crashes with an unhandled `IndexError` when passed an empty list because `conversation[0]` is accessed without checking if the list is empty first. This affects any model with a chat template (reproduced on Qwen2.5 and Gemma-2).

Added an explicit empty check that raises a clear `ValueError` with a helpful message instead.

- [x] I confirm that this is not a pure code agent PR.
- [x] Was this discussed/approved via a Github issue? Yes — #46752




@ArthurZucker @itazap (tokenizers)

Comments:
cool, we can also add a small test to assert the error is raised on an empty conv

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46753). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@itazap Added the test as requested - all 99 CI checks are passing. Could you approve when you get a chance? Thanks!

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46753)
[ROCm] gpt-oss: route FA3 to aiter-flash-attn, generate ROCm fixtures
Author: Abdennacer-Badaoui
Pull request: yes

On ROCm, gpt-oss now routes the FA3-style attention to `kernels-community/aiter-flash-attn` (`_compatible_flash_implementations`) and generates separate ROCm fixtures (non-distributed + `tp_size=2`). Configs that depend on `kernels-community/megablocks` are skipped because it doesn't ship a ROCm build for torch 2.10 (the version the AMD CI runs today) we have the ones for 2.11 and 2.12, and configs that hit `kernels-community/sonic-moe` are skipped since that kernel has no ROCm build yet. 
Fixes around 140 failing tests. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46837). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gpt_oss, openai_privacy_filter

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46837)

This PR allows multiple different names for the same argument (especially for sinks): https://github.com/huggingface/transformers/pull/45153.
Once it's merged, our `aiter-flash-attn` can keep using `sink` as the argument name.