noamaanMulla-03/transformers-issues huggingface dataset:
🚨 [DiffusionGemma] Return router logits and load balancing loss
Author: kashif
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=46642)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46642)
<!-- ci-dashboard-badge:end -->

DiffusionGemma is a MoE model but it never surfaced the router logits, so downstream trainers couldn't add a load balancing loss on the experts.

This wires it up the same way as the other MoE models (qwen3_moe as reference):
- adds `output_router_logits` and `router_aux_loss_coef` to the text config
- the decoder router now returns its raw logits, which the recorder exposes as `router_logits`
- `DiffusionGemmaForBlockDiffusion` returns `router_logits` and the load balancing `aux_loss` when the flag is set

It also makes `gemma4` and `gemma4_unified` trainable with the MoE load balancing loss (they were missing it too), and diffusion gemma inherits the config and imports the loss func from mixtral through modular.

The loss is only over the decoder experts (the canvas denoising path, which is unpadded so no mask is needed). Off by default, so generation is unchanged.

Note: the gemma routers now return their raw logits as the first output instead of the post-softmax probabilities. The model output is identical (the decoder layer discards that value and the router module isn't exported), but if you import a router `Module` directly, its first return value changed.

Tests: added `test_output_router_logits` and a general `test_load_balancing_loss` in `test_modeling_common.py`. Full diffusion_gemma, gemma4 and gemma4_unified suites pass on CPU.

AI assistance was used.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46642). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Reworked as discussed: gemma4 (and gemma4 unified) are now trainable with the MoE load balancing loss, and the load balancing func is imported from mixtral instead of duplicated. Diffusion gemma inherits the config and the func through modular, so its generated files are unchanged. gemma4 / gemma4 unified / diffusion gemma tests pass (638), repo consistency is green.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46642&sha=f7e9d2

The failing test in CI looks relevant here, it is the newly added testcase 👀 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: diffusion_gemma, gemma4, gemma4_unified, kimi_k25

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46642)
**Latest run:** [29188430896:1](https://github.com/huggingface/transformers/actions/runs/29397077179)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 171,343 | **Failures:** 1 | **Duration:** 8h 24m

<!-- ci-dashboard-recap:end -->
fix(musicflamingo): Fix dtype mismatch with _keep_in_fp32_modules_strict override
Author: harshaljanjani
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47300)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47300)
<!-- ci-dashboard-badge:end -->

### What does this PR do?

→ Restores the `_keep_in_fp32_modules_strict` override on `MusicFlamingoForConditionalGeneration` so half-precision works again.
→ Regression from #45534; the split into Model + head kept the None override only on the base-model class so the head class now inherits `["embed_positions"]` from Voxtral (via AudioFlamingo3). As for whether any other models from the same PR are afflicted, AF3 has the [same regression](https://github.com/harshaljanjani/transformers/blob/main/src/transformers/models/audioflamingo3/modeling_audioflamingo3.py#L569) but is already being addressed in `#47258` so it's left untouched here; `GlmAsrForConditionalGeneration` inherits the same attr but has no `embed_positions` param at all (rotary embeddings) so it's a [harmless no-op there](https://github.com/harshaljanjani/transformers/blob/main/src/transformers/models/glmasr/modeling_glmasr.py#L523) (happy to add the same override for clarity if wanted).
→ Made sure this doesn't cause any regressions in `tests/models/musicflamingo/`

cc: @eustlb

**Before ([CI Failures](https://github.com/huggingface/transformers/actions/runs/29221731474/job/86728607936))**

<img alt="image" src="https://github.com/user-attachments/assets/109b1a59-a319-454f-a944-87e20e12323d" /><br>

**After**

<img alt="3" src="https://github.com/user-attachments/assets/e2493654-e4b6-4f9d-8c83-d08cd0b18c15" />

### Code Agent Policy
- [x] I confirm that this is not a pure code agent PR.

### Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [x] Did you fix any necessary existing tests?

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: musicflamingo

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47300)
**Latest run:** [29244887356](https://github.com/huggingface/transformers/actions/runs/29244887356)
**Result:** `success` | Grafana metrics are not available yet.

<!-- ci-dashboard-recap:end -->

Good day @eustlb! Just checking in to see if there've been any updates :)
cc: @Rocketknight1
fix: make mistral base check conditional to fix offline loading
Author: Killusions
Pull request: yes
Labels: name: for patch

Backport of a small part of #42528, the model_info call currently breaks offline loading

⚒️ with ❤️ by @siemens

Comments:
@vasqu I'm fine with doing these changes and adding a test, but should we really deviate from the upstream/main branch in a backport? See https://github.com/huggingface/transformers/pull/42528

And if we add a test we should also merge it to main

@vasqu Ah I think I misunderstood you, fixed

@vasqu Also added a test

cc @ArthurZucker @itazap 

@vasqu @ArthurZucker @itazap 

What's the timeline for this? Is there any change you still require before merging this?

Bump @vasqu @ArthurZucker @itazap

Hey! I'll see with @vasqu
Transformers' version 5 is out!
Author: LysandreJik
Pull request: no

Thank you all for your patience while we released RC0, RC1, RC2, and RC3.

We're looking forward to collaborating with you on a stronger, stabler, and faster foundation. Please keep the feedback coming in GitHub issues, this is only the beginning 🙌 

@huggingface/transformers-core-maintainers 

Comments:
Thank you for your hard work. It is a wonderful contribution to the community.

BTW, I did not observe the `Backwards Incompatible Changes` in the release notes: `https://github.com/huggingface/transformers/releases/tag/v5.0.0`. Where can I find this information? Best

Thanks for the great work! 

Will 4.57.6 be the last v4 release? For all new models being added to transformers, will they be only available in v5?

Yes!

> Thank you all for your patience while we released RC0, RC1, RC2, and RC3.
> 
> We're looking forward to collaborating with you on a stronger, stabler, and faster foundation. Please keep the feedback coming in GitHub issues, this is only the beginning 🙌
> 
> @huggingface/transformers-core-maintainers



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

will there be more bug fixes v4 releases? what do we do if we want to fix a bug that's in both v4 and v5?

We'll fix in v5! 

1 2 3 4 5 6 7 8 9+0+1(re pe at)
Pin kernels to compatible version in benchmark workflow
Author: tarekziade
Pull request: yes
Labels: name: run-benchmark

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47339)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47339)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

The benchmark workflow installed `kernels` unpinned, pulling the latest version (>=0.16) which is outside transformers' required range [0.15.2, 0.16.0). The subsequent `.[torch]` reinstall does not include kernels, so the stale version was never corrected. At runtime the benchmark runner enables kernels whenever the import succeeds, but transformers' `is_kernels_available()` enforces the version range and rejects it, raising `ValueError: Kernels are not available`.

Install kernels via the pinned `.[kernels]` extra so setup.py's `kernels>=0.15.2,<0.16` constraint governs the installed version.

https://github.com/huggingface/transformers/actions/runs/29391269980/job/87275127776

Comments:
<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47339)
**Latest run:** [29401878050](https://github.com/huggingface/transformers/actions/runs/29401878050)
**Result:** `success` | Grafana metrics are not available yet.

<!-- ci-dashboard-recap:end -->

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47339). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

fixed run https://github.com/huggingface/transformers/actions/runs/29405978811/job/87321421271?pr=47339
[Improvement] Re-structure sonic MoE kernel interface to use a Handle object
Author: remi-or
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47316)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47316)
<!-- ci-dashboard-badge:end -->

In #47126 and offline we talked about how we want to take care of loading kernels like `DeepGEMM`.

Notably, we can put everything related to loading and calling the kernel inside a singleton `Handle` object. This object would only keep track of a few state variable and provide a single entry point to access the kernel. This has the following advantages:
- we can keep track of the cached state without using `@lru_cache` which can be opaque and cause memory leaks
- failures are actually cached: `@lru_cache` does not cache exceptions, so the cache never worked in a machine / env that did not support `sonicmoe`
- we can reset the cached state of the handle whenever we want through a simple `reset` method
- we can check the availability of the kernel easily through a `sonicmoe_is_available` property, that accesses the same underlying cached kernel as the function call itself (quite convenient, as is shown in the reworked `_test_eager_matches_batched_and_grouped_inference`)

The downside of the Handle API are that transient failures are also cached (hence the `reset` method, but it is not called automatically) and it's beefier than `@lru_cache`. IMO this is an ok tradeoff, but I would love some feedback on this.

Since I was fixing a failure in `sonicmoe`, I thought this was a good occasion to try out the Handle API. The fail was an image failure, which is now guarded against.

I also added a test for sonicmoe + compile and it passes on B200. Also tested the sonicmoe regular tests on a qwen and they pass.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47316). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47316)
**Latest run:** [29320447311](https://github.com/huggingface/transformers/actions/runs/29320447311)
**Result:** `success` | Grafana metrics are not available yet.

<!-- ci-dashboard-recap:end -->

Lots of feedback, thanks! This was more of a rough PoC so lots to take in, and I did not plan to spend too much time on this, it was more because tests were crashing on main that I took the opportunity to look into it. Will update when I have the bandwidth! 

No worries, we also nudged the kernels team so maybe we can simplify a bit more in the future. It's a bit awkward to handle these kernels :cry:
[docs] TokenizersBackend fallback
Author: stevhliu
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47302)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47302)
<!-- ci-dashboard-badge:end -->

Adds docs to describe the routing behavior behind how a checkpoint falls back to `TokenizersBackend` when there is no dedicated tokenizer class, an incorrect tokenizer class is stored on the Hub, or overriding a specific checkpoint (see https://github.com/huggingface/transformers/pull/45936).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47302). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47302)
**Latest run:** [29270728204:1](https://github.com/huggingface/transformers/actions/runs/29270728204)
**Result:** `success` | **Jobs:** 2 | **Tests:** 24 | **Failures:** 0 | **Duration:** 1m 57s

<!-- ci-dashboard-recap:end -->
ci: cover xet as well (runtime error)
Author: tarekziade
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47338)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47338)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

The fallback to use a tempdir on read only cache needs to be extended for XET that produces Runtime errors not OS Errors

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47338). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

new peft run with the latest version https://github.com/huggingface/transformers/tree/ci-verify-46766-runtime

fix validated at https://github.com/huggingface/transformers/actions/runs/29430931604/job/87405764055

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47338)
**Latest run:** [29422275203:1](https://github.com/huggingface/transformers/actions/runs/29432353612)
**Result:** `success` | **Jobs:** 4 | **Tests:** 269 | **Failures:** 0 | **Duration:** 2m 11s

<!-- ci-dashboard-recap:end -->
special_tokens, 'list' object has no attribute 'keys'
Author: Ratinax
Pull request: no
Labels: name: bug

### System Info

In `tokenization_utils_base.py` line 1190, there is a `special_tokens.keys()` done. But `special_tokens` is typed as `List[str]`.

And when using it, I have `special_tokens` being equal to `['
NOTUSED', '
NOTUSED', '<unk>NOTUSED']` this lead to the error `'list' object has no attribute 'keys'`

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

use `AutoTokenizer.from_pretrained(model_path)` with a valid model_path and a valid content

### Expected behavior

not raising an error and working

Comments:
I would like to take this.

I’ll reproduce it on current `main`, then add a narrow load time normalization in `tokenization_utils_base.py` so list-valued `model_specific_special_tokens` is treated as `extra_special_tokens` (where lists belong), with a local `AutoTokenizer.from_pretrained(...)` regression test in `tests/tokenization/test_tokenization_utils.py`.

Could not reproduce this on v5.13.0: AutoTokenizer.from_pretrained("almanach/camembert-base") (which has the 
NOTUSED style tokens) loads fine. The only special_tokens.keys() call at current HEAD is in _set_model_specific_special_tokens, and every path into it guards against list values, so this looks version-dependent. Could you share your transformers version and the tokenizer_config.json / special_tokens_map.json of the failing model? Happy to dig further with that.

tried to reproduced it but i am not getting the same error.
🚨 Pass tp_plan from lm_heads
Author: molbap
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47253)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47253)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

Noticed currently the `lm_head`-specific tp plans are wiped in `PreTrainedModel` post-init (we just ignore them). This should fix it

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47253). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47253)
**Latest run:** [29107923987:1](https://github.com/huggingface/transformers/actions/runs/29107923987)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 170,565 | **Failures:** 1 | **Duration:** 15h 14m

<!-- ci-dashboard-recap:end -->

Merging we need it for release
Add Inkling model
Author: molbap
Pull request: yes
Labels: name: New model

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47347)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47347)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

Add new model from Thinking Machines!

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, bamba, falcon_h1, falcon_mamba, granitemoehybrid, inkling, jamba, lfm2, lfm2_moe, mamba, mamba2, nemotron_h, qwen3_5, qwen3_5_moe, qwen3_next, zamba

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47347)
**Latest run:** [29440654491:1](https://github.com/huggingface/transformers/actions/runs/29440903312)
**Result:** `failure` | **Jobs:** 2 | **Tests:** 2 | **Failures:** 2 | **Duration:** 8s

> **Code quality check failed**: test jobs were skipped. Fix the code quality issues and push again to run tests.

<!-- ci-dashboard-recap:end -->

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47347). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Small improvement
Author: Cyrilvallez
Pull request: yes

# What does this PR do?



Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: inkling

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47348). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] FlashAttention kernel fallback
Author: stevhliu
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47345)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47345)
<!-- ci-dashboard-badge:end -->

Better documents the FA kernel fallback detection

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47345). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47345)
**Latest run:** [29447956142:1](https://github.com/huggingface/transformers/actions/runs/29447956142)
**Result:** `success` | **Jobs:** 2 | **Tests:** 7 | **Failures:** 0 | **Duration:** 17s

<!-- ci-dashboard-recap:end -->
Fix PIL backend fallback when torchvision is unavailable
Author: ErenAta16
Pull request: yes

This PR fixes a regression where PIL-based image/video processors were incorrectly treated as requiring `torchvision`.

As a result, `AutoProcessor` / `AutoImageProcessor` could fail in environments without `torchvision`, even though a valid PIL fallback exists.

### What changed

- Updated import-structure backend resolution in `import_utils` so PIL modules do not inherit `torchvision` requirements:
  - `image_processing_pil_*`
  - `video_processing_pil_*`
- Added a normalization step to prevent `torchvision` from being attached to PIL-only modules during backend aggregation.
- Added regression tests for:
  - PIL import-structure backend correctness
  - Auto-backend fallback to PIL when `torchvision` is unavailable

Fixes #45042

### Tests

- `python -m pytest tests/utils/test_import_structure.py -k "pil_import_structure_does_not_require_torchvision" -q`
- `python -m pytest tests/models/auto/test_image_processing_auto.py -k "auto_backend_falls_back_to_pil_when_torchvision_is_unavailable or backend_kwarg_pil" -q`

Comments:
Thanks for the run. The current failure is in
`tests/models/prophetnet/test_modeling_prophetnet.py::ProphetNetModelTest::test_encoder_decoder_model_generate`,
which is outside this PR scope.

This PR only changes PIL backend import-structure/fallback logic:
- src/transformers/utils/import_utils.py
- tests/utils/test_import_structure.py
- tests/models/auto/test_image_processing_auto.py

Could we rerun `tests_torch` once? If the same ProphetNet test is consistently failing, I can help track it separately.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45060&sha=8260c8

Checked this against current main before following up. #45045 rewrote every `image_processing_pil_*` file, and as a side effect none of them contain the `TorchvisionBackend` string anymore (confirmed via code search, zero matches), which is exactly what `BASE_FILE_REQUIREMENTS`'s heuristic in this PR was working around. So the concrete bug (Gemma3ImageProcessorPil, and everything else that hit this path) is already fixed on main through #45045's approach, not this one.

The `BASE_FILE_REQUIREMENTS` string-matching heuristic itself is still there and still fragile in principle, so this PR isn't wrong, just no longer fixing an active problem. Closing in favor of #45045 rather than keeping a defensive-only PR open.