noamaanMulla-03/transformers-issues huggingface dataset:
Add Parakeet-RNNT
Author: eustlb
Pull request: yes
Labels: name: New model

# What does this PR do?

This PR adds parakeet RNNT architecture by refactoring the TDT one already merged: TDT is an extension of RNNT, and it should be implemented as such in the lib.

original checkpoints:
- [nvidia/parakeet-rnnt-0.6b](https://huggingface.co/nvidia/parakeet-rnnt-0.6b)
- [nvidia/parakeet-rnnt-1.1b](https://huggingface.co/nvidia/parakeet-rnnt-1.1b)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46331). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: parakeet

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27349686713)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/parakeet"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27349686713)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [c8d4a898](https://github.com/huggingface/transformers/commit/c8d4a898e022fb891ae138821cd3adda7bdcdcdd) | workflow commit (merge commit) |
| PR | [ce3e6028](https://github.com/huggingface/transformers/commit/ce3e6028bb96779f3df703de3ac565ce2e2ee34a) | branch commit (from PR) |
| main | [6b862157](https://github.com/huggingface/transformers/commit/6b8621578d28e2585382f418e701c8cc289fcc04) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


https://app.circleci.com/pipelines/github/huggingface/transformers/177680/workflows/3c0b8544-7b23-49a9-ae21-0476dd5961a5/jobs/2350759 might be a real regression but I can merge afterwards, just ping me. CI is super flaky :cry: 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, parakeet

It was indeed! Now fixed, merging 🎉 thank you so much for the reviews @vasqu !!!! 🫶
Fix chat !save custom filename parsing
Author: Chinmay1220
Pull request: yes

## Summary
- fix the chat CLI `!save NAME` command so the advertised custom filename is accepted
- keep malformed save commands invalid by parsing the command token explicitly
- allow `save_chat("chat.json", ...)` to write into the current directory without calling `os.makedirs("")`

## Validation
- `PYTHONPATH=src .\.venv\Scripts\python.exe -m pytest tests\cli\test_chat.py -q`
- `.\.venv\Scripts\python.exe -m ruff check src\transformers\cli\chat.py tests\cli\test_chat.py`
- `.\.venv\Scripts\python.exe -m py_compile src\transformers\cli\chat.py tests\cli\test_chat.py`
- `git diff --check`

## Duplicate check
- Searched GitHub issues and PRs for `save_chat`, chat custom filename, `chat_history` save filename, and `!save` / `SAVE_NAME`; no matching existing issue or PR found.

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46567)
docs: 🌐 add Turkish translation for README file
Author: onuralpszr
Pull request: yes

# What does this PR do?

Translate Transformers to turkish language

cc @merveenoyan  for proof reading (please) for certain terms I kept english or using developer friendly term some of them we discuss previously in other translation doc cases so I try to careful as much as possible. Thank you in advance. 

Docs: @stevhliu (only thing I see I should add this to all of the other readme files as link) 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46312). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Make DiffusionGemma trainable
Author: kashif
Pull request: yes

Three small fixes so DiffusionGemma works with Trainer and SFT recipes.

`bos_token_id` was missing from the strict generation config, which crashed `align_special_tokens` and broke checkpoint reload. The forward now also returns the encoder's last hidden state (needed for the encoder AR co-loss used in the reference fine-tuning recipe) and accepts a per-example `self_conditioning_mask`. Gradient checkpointing is declared unsupported since checkpointed layers drop the encoder KV cache writes the decoder reads from, which silently broke training.

Inference is unchanged, all diffusion_gemma tests pass.

cc @gante

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46568). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

The changes LGTM 👍 

Good point, switched to Seq2SeqModelOutput / Seq2SeqLMOutput and dropped the custom classes.

Reverted the Seq2Seq output switch after a closer look: it renames hidden_states and attentions to decoder_*, which breaks anyone already reading the 5.11 outputs. The model specific subclasses keep the old field names and just add encoder_last_hidden_state.

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46568)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: diffusion_gemma
Fix Wav2Vec2 word delimiter special token handling
Author: nightcityblade
Pull request: yes

Fixes #46552.\n\nThis keeps the Wav2Vec2 CTC word delimiter from being treated as removable special token by special-token-aware APIs, matching the decode path behavior.\n\nChanges:\n- Preserve the word delimiter in convert_ids_to_tokens(..., skip_special_tokens=True).\n- Exclude the word delimiter from get_special_tokens_mask(..., already_has_special_tokens=True).\n- Add a regression test for delimiter masking/skipping.\n\nTests:\n- python -m pytest tests/models/wav2vec2/test_tokenization_wav2vec2.py -q

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46578)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: wav2vec2
Remove softmax_backward_data
Author: cyyever
Pull request: yes

# What does this PR do?

Remove softmax_backward_data and its references.

Comments:
run-slow: sew_d

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: sew_d

run-slow: sew_d

This comment contains run-slow, running the specified jobs: 

models: ['models/sew_d']
quantizations: [] ...

Even though the tests pass, I'm a bit wary - let's leave this one for now! I don't want to delete a function when I'm not totally sure why it's there; Chesterton's Function and all that
Remove upper version bound of pandas
Author: cyyever
Pull request: yes

# What does this PR do?

Let's skip pandas 2.3.0 and allow users to install newer versions.


Comments:
cc @ydshieh since I don't know why that version skip was added!

It is #38605. Maybe it works now but I don't want to spend the time to check. 

PR LGTM, thank you

The error at that time:

> Using Python 3.9.23 environment at: /usr/local
Resolved 47 packages in 3.41s
      Built transformers @ file:///root/transformers
  × Failed to build `pandas==2.3.0`
  ├─▶ The build backend returned an error
  ╰─▶ Call to `mesonpy.build_wheel` failed (exit status: 1)

      [stdout]
      + meson setup
      /root/.cache/uv/sdists-v9/pypi/pandas/2.3.0/VqR3nkX0wjeGwwtaucpZ_/src
      /root/.cache/uv/sdists-v9/pypi/pandas/2.3.0/VqR3nkX0wjeGwwtaucpZ_/src/.mesonpy-2kp275i8
      -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --vsenv
      --native-file=/root/.cache/uv/sdists-v9/pypi/pandas/2.3.0/VqR3nkX0wjeGwwtaucpZ_/src/.mesonpy-2kp275i8/meson-python-native-file.ini
      The Meson build system
      Version: 1.8.1
      Source dir:
      /root/.cache/uv/sdists-v9/pypi/pandas/2.3.0/VqR3nkX0wjeGwwtaucpZ_/src
      Build dir:
      /root/.cache/uv/sdists-v9/pypi/pandas/2.3.0/VqR3nkX0wjeGwwtaucpZ_/src/.mesonpy-2kp275i8
      Build type: native build
      Project name: pandas
      Project version: 2.3.0

      ../meson.build:2:0: ERROR: Unknown compiler(s): [['cc'], ['gcc'],
      ['clang'], ['nvc'], ['pgcc'], ['icc'], ['icx']]
      The following exception(s) were encountered:
      Running `cc --version` gave "[Errno 2] No such file or directory: 'cc'"
      Running `gcc --version` gave "[Errno 2] No such file or directory:
      'gcc'"
      Running `clang --version` gave "[Errno 2] No such file or directory:
      'clang'"
      Running `nvc --version` gave "[Errno 2] No such file or directory:
      'nvc'"
      Running `pgcc --version` gave "[Errno 2] No such file or directory:
      'pgcc'"
      Running `icc --version` gave "[Errno 2] No such file or directory:
      'icc'"
      Running `icx --version` gave "[Errno 2] No such file or directory:
      'icx'"

      A full log can be found at
      /root/.cache/uv/sdists-v9/pypi/pandas/2.3.0/VqR3nkX0wjeGwwtaucpZ_/src/.mesonpy-2kp275i8/meson-logs/meson-log.txt

      hint: This usually indicates a problem with the package or the build
      environment.
  help: `pandas` (v2.3.0) was included because `transformers[quality]`
        (v4.53.0.dev0) depends on `datasets` (v3.6.0) which depends on
        `pandas`

Exited with code exit status 1

CI is already on 

pandas==2.3.3

so we are good

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41677). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
cleaning _accumulate_local_param_grad
Author: 3outeille
Pull request: yes

We cant remove `_accumulate_local_param_grad` fully as a user might want to train in TP only (without using `grouped_gemm` for the experts) which involves `moe_tp_gate_up_colwise` thus `StridedShard` (thus need to stitch gradient manuall)



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46394). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46394&sha=aa4a65
[CB] [Minor] Add parameter to tune default compile level
Author: remi-or
Pull request: yes

# Summary

Currently, continuous batching exposes the parameter `use_default_compile_configs` to the users so that they may enable `torch.compile` easily. But that parameter enables compile with `max-autotune-no-cudagraphs` and `dynamic=False` for best performances. This makes for a long warmup, which is not great for tests or to iterate over benchmarks. 

This PR cleanly deprecates this parameter and instead exposes `default_compile_level` which is an integer parameter that controls default compile behavior. Level 0 means no compile, and level 1-3 activate compile, with increasing performance and warmup duration.  
If a user passes the old parameter, a warning is raised, and the level is set to 3, so that the behaviour is the same as with the old parameter.

## Performance

No behavioral change, so same perf. 

## Tests

TBD

## Review

Reviewed by claude + coumpound engineering.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46533). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46533)
[CI] Enable PR CI for all fork PRs via security gate
Author: ydshieh
Pull request: yes

## Summary

- Remove the `if:` condition on `pr-ci` that previously restricted runs to members/collaborators/owners
- Remove the `debug-conditions` job (no longer needed)

Fork PRs from external contributors now hit GitHub's "Approve and run" gate, which is automatically handled by `pr-ci-security-gate.yml` → `huggingface/transformers-test-ci` security gate workflow. The security gate checks for shell/process execution issues and high severity findings introduced by the PR before approving the run.

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46591)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46591). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
PP-OCRv6: update documentation and slow tests
Author: zhang-prog
Pull request: yes


Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46576)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: pp_ocrv6_medium_det, pp_ocrv6_small_det, pp_ocrv6_small_rec, pp_ocrv6_tiny_rec

run-slow: pp_ocrv6_medium_det, pp_ocrv6_small_det, pp_ocrv6_small_rec, pp_ocrv6_tiny_rec

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27406476173)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/pp_ocrv6_medium_det", "models/pp_ocrv6_small_det", "models/pp_ocrv6_small_rec", "models/pp_ocrv6_tiny_rec"]
quantizations: []

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46576). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27406476173)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [d769d155](https://github.com/huggingface/transformers/commit/d769d15590f11a8d4c3507d011ee5e0238a965be) | workflow commit (merge commit) |
| PR | [01688b74](https://github.com/zhang-prog/transformers/commit/01688b74f693fe034012bdc0001e1e1739cc327f) | branch commit (from PR) |
| main | [80141396](https://github.com/huggingface/transformers/commit/801413961815dad5f943b77bb83645c5ef7bcc82) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !
fix(serve): don't classify Gemma 4 closed prefilled thinking block as reasoning
Author: Anai-Guo
Pull request: yes
Labels: name: Code agent slop

## What

Fixes #46561. With `transformers serve` and a Gemma 4 model, a normal (thinking-disabled) chat completion comes back with empty `content` — the whole answer is misclassified as `reasoning_content`.

## Root cause

Gemma 4's chat template prefills an **empty, already-closed** thinking block at the end of the prompt when thinking is disabled, so the prompt tail is `<|channel>thought\n<channel|>` → `[..., 100, 45518, 107, 101]` where `start_ids = [100, 45518, 107]` and the trailing token `101` is the thinking **end** token (`<channel|>`).

`_starts_in_thinking()` matches `start_ids` at the tail while tolerating one trailing token (so genuine prefilled openers like DeepSeek-R1 / QwQ that emit `<think>\n` still match). The closing tag was accepted as that tolerated trailing token, so the heuristic wrongly reported `start_in_thinking=True`. Because the output contains no thinking markers, `parse_reasoning()` then takes the "prefilled opener truncated before close" branch and returns `content=""` with everything in reasoning.

## Fix

Thread the already-computed `end_id` into `_starts_in_thinking()` and reject the one-trailing-token match when that trailing token is the thinking end token — a closed prefilled block means the prompt is *not* left inside a thinking block. The `trailing == 0` (ends exactly on opener) and `trailing == 1` with a non-end token (DeepSeek-R1 / QwQ `\n`) cases are unchanged.

## Tests

Added `TestStartsInThinking` (CPU-only, no model download) covering: ends-exactly-on-opener, opener + whitespace trailing, the Gemma 4 closed-block case (returns `False`), the closed block without `end_id` (back-compat still matches), no-thinking tail, and batched `input_ids`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46589)

Please stop the random drive-by Claude PRs! We can run Claude Code ourselves if we need it!
Fix torchaudio version not tied to torch version in docker file 
Author: ydshieh
Pull request: yes

# What does this PR do?

Well, it is not automatically tied ... and we have to do it explicitly

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46594)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46594). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: correct 'overriden' typo in multiple files
Author: Rohan5commit
Pull request: yes

## What does this PR do?

Fixes a misspelling 'overriden' → 'overridden' in comments and docstrings across multiple files:
- `processing_utils.py`
- `image_processing_utils.py`
- `video_processing_ernie4_5_vl_moe.py`

## Checklist

- [ ] I have read the contributing guidelines.
- [ ] The title of this PR is formatted properly.
- [ ] This PR has a descriptive description.

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46584)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: ernie4_5_vl_moe
fix: correct 'embedings' typo in MobileBERT comments
Author: Rohan5commit
Pull request: yes

## What does this PR do?

Fixes a misspelling in MobileBERT comments: 'embedings' → 'embeddings'.

## Checklist

- [ ] I have read the contributing guidelines.
- [ ] The title of this PR is formatted properly.
- [ ] This PR has a descriptive description.

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46582)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: mobilebert
fix: correct 'depreciated' typo in BART deprecation warning
Author: Rohan5commit
Pull request: yes

## What does this PR do?

Fixes a misspelling in the BART deprecation warning: 'depreciated' → 'deprecated'.

## Before/After

- Before: `The class 'PretrainedBartModel' has been depreciated`
- After: `The class 'PretrainedBartModel' has been deprecated`

## Checklist

- [ ] I have read the contributing guidelines.
- [ ] The title of this PR is formatted properly.
- [ ] This PR has a descriptive description.

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46581)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bart
fix: correct 'lenghts' variable name typo in higgs_audio_v2 processor
Author: Rohan5commit
Pull request: yes

## What does this PR do?

Fixes a misspelled variable name: 'lenghts' → 'lengths' in the higgs_audio_v2 processor.

## Checklist

- [ ] I have read the contributing guidelines.
- [ ] The title of this PR is formatted properly.
- [ ] This PR has a descriptive description.

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46585)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: higgs_audio_v2
fix: correct 'ouputs' typo in continuous batching docstring
Author: Rohan5commit
Pull request: yes

## What does this PR do?

Fixes a misspelling in the continuous batching docstring: 'ouputs' → 'outputs'.

## Checklist

- [ ] I have read the contributing guidelines.
- [ ] The title of this PR is formatted properly.
- [ ] This PR has a descriptive description.

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46583)

Hey, these small typo fix PRs cause a lot of noise! For stuff like this, we'd prefer much larger PRs that fix all typos across multiple files. Reviewing and running the CI for every single typo is probably not worth it
fix: correct 'funcitonality' typo in RoPE utils
Author: Rohan5commit
Pull request: yes

## What does this PR do?

Fixes a misspelling in a warning message: 'funcitonality' → 'functionality'.

## Checklist

- [ ] I have read the contributing guidelines.
- [ ] The title of this PR is formatted properly.
- [ ] This PR has a descriptive description.

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46586)
Add regression tests for void segmentation maps (MaskFormer, Mask2Former)
Author: JohnnyWilson16
Pull request: yes
Labels: name: Code agent slop

# What does this PR do?

Fixes #30064.

Processing a segmentation map where every pixel is `ignore_index` ("void") used to crash with `ValueError: need at least one array to stack`. Current `main` handles this correctly — the conversion functions guard the empty-mask case and return empty `(0, H, W)` `mask_labels` with empty `class_labels` — but the behavior had no test coverage, so it could silently regress the same way it was silently fixed during the v5 image-processor rewrites.

This PR adds `test_call_with_void_segmentation_maps` to the MaskFormer and Mask2Former image-processing test suites, covering both the PIL and torchvision backends:

- a single all-void map (`ignore_index` everywhere, `do_reduce_labels=False`)
- a batch mixing a void and a non-void map, including padding of the empty `(0, H, W)` masks
- an all-background map with `do_reduce_labels=True` (background is mapped to `ignore_index`, so the map is void as well)

Removing the empty-mask guards from the processors makes the new tests fail with the original error, so they act as a real regression tripwire.

## Tests

```
python -m pytest tests/models/mask2former/test_image_processing_mask2former.py tests/models/maskformer/test_image_processing_maskformer.py
# 60 passed, 8 skipped
```

## Disclosure (agentic contributions)

Submitted by Johnny Wilson (@JohnnyWilson-Portfolio). AI tooling was used to verify the issue status against `main` and to draft these tests, per the repository's agentic-contributions guidelines; as the human submitter I take responsibility for the changed lines, and the test commands above were run locally on my machine. Coordination: https://github.com/huggingface/transformers/issues/30064#issuecomment-4683603833. Differentiation: #38532 (stale WIP) drops void samples from the batch and adds library code; this PR is tests-only and keeps void samples in the batch.

## Who can review?

@qubvel @Rocketknight1 (active on #30064)


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: mask2former, maskformer

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46571)