noamaanMulla-03/transformers-issues huggingface dataset:
Fix MoonshineStreaming right-window off-by-one in sliding_window_mask_function
Author: rodriguescarson
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47028)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47028)
<!-- ci-dashboard-badge:end -->

Fixes #46305

## Problem

`sliding_window_mask_function` uses a strict less-than for the right (future) context:

```python
right_mask = (dist < 0) & (-dist < right_window_size)
```

With `right_window_size=4`, a query attends to only **3** future frames, but the Moonshine v2 paper (arXiv:2602.12241) defines the right window as:

> "a right window of w_right=4 implies an algorithmic lookahead of 4×20 ms = 80 ms"

At 50 Hz, 80 ms = **4** future frames. Repro from the issue:

```python
fn = sliding_window_mask_function((16, 4))
[kv for kv in range(11, 20) if fn(0, 0, 10, kv)]
# current: [11, 12, 13]        (3 frames, 60 ms)
# paper:   [11, 12, 13, 14]    (4 frames, 80 ms)
```

## Fix

Use `<=` so `right_window_size` future frames are attended, matching the paper's definition of `w_right`. Applied to both `modular_moonshine_streaming.py` and the generated `modeling_moonshine_streaming.py` (identical one-token change, so the two stay in sync).

## Note for @eustlb

This aligns the mask with the paper's stated `w_right` semantics. If the released weights were instead trained/exported with the strict (`< `, 3-frame) mask, please advise — in that case the inference mask should match training and the *docs/param semantics* would be what needs clarifying rather than the mask. The change itself is verifiable by inspection against the issue's pure-function repro.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: moonshine_streaming

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47028)
**Latest run:** [28608617635](https://github.com/huggingface/transformers/actions/runs/28608617635)
**Result:** `failure` | Grafana metrics are not available yet.

<!-- ci-dashboard-recap:end -->

Heads up: the failing `Check repository consistency` job is unrelated to this change. `check_modular_conversion.py` reports the diff in **gemma4** (`modeling_gemma4.py` vs its modular source), which is a pre-existing mismatch on `main`. This PR only touches the two `moonshine_streaming` files with an identical one-token change, and moonshine passes the modular check. Happy to rebase once gemma4 is back in sync.

Withdrawing this PR to follow the project's contribution process — I should have coordinated on #46305 and waited for a maintainer to confirm the fix direction (paper semantics vs. released-weights behavior) before opening it. Happy to reopen if @eustlb/@keveman confirm on the issue that the right-window mask should match the paper's w_right. Apologies for the premature PR.
Fix RTDetrHungarianMatcher crash on infinite cost matrix
Author: AyushDas4890
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47016)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47016)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

Fixes #47000

`RTDetrHungarianMatcher` crashes with `ValueError: cost matrix is infeasible`
when the cost matrix contains infinite values. Under AMP, fp16 sigmoid
saturation gives probabilities of exactly 0.0 or 1.0, and the 1e-8 epsilon in
the focal class cost underflows in fp16, so the log() terms overflow to ±inf
before reaching `scipy.optimize.linear_sum_assignment`.

This applies the same guard that was merged for Mask2Former in #27897:
clamp the cost matrix to [-1e10, 1e10] before the assignment. `DFineLoss`
reuses this matcher, so D-FINE is fixed too.

Added a regression test that reproduces the failure with fp16 saturating
logits (both directions) and verifies the matcher returns valid indices.
The test fails without the fix and passes with it.

Disclosure: I used an AI assistant to help diagnose the root cause and draft
this fix myself; I have reviewed the changes and run the new test myself.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: rt_detr

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47016)
**Latest run:** [28638832663:2](https://github.com/huggingface/transformers/actions/runs/28638832663)
**Result:** `failure` | **Jobs:** 14 | **Tests:** 47,253 | **Failures:** 1 | **Duration:** 17h 29m

<!-- ci-dashboard-recap:end -->
Fix Mistral4 tests
Author: 3outeille
Pull request: yes

https://github.com/huggingface/transformers/pull/44825

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_44827). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: auto, mistral4

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/23265208893)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/auto", "models/mistral4"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/23265208893)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [b90678b2](https://github.com/huggingface/transformers/commit/b90678b29ae2402a88cb47a606ba8ce25e6cef11) | workflow commit (merge commit) |
| PR | [45f5bb7b](https://github.com/3outeille/transformers/commit/45f5bb7b1c4f568461071bb5eb065c596031f24e) | branch commit (from PR) |
| main | [21950930](https://github.com/huggingface/transformers/commit/21950930a657cdc446188d38b6979b90513bab41) | base commit (on `main`) |

### Model CI Report

❌ **[2 new failed tests from this PR](https://huggingface.co/datasets/hf-internal-testing/transformers_pr_ci/raw/4e22f3d42964cca28dc1d1b7cbbc8867a6641086/2026-03-18/runs/31236-23265208893/ci_results_run_models_gpu/new_failures_with_bad_commit.json)** 😭

- [mistral4](https://github.com/huggingface/transformers/actions/runs/23265208893/job/67645102154):
    tests/models/mistral4/test_modeling_mistral4.py::Mistral4ModelTest::test_flex_attention_with_grads (✅ ⟹ ❌)
    tests/models/mistral4/test_modeling_mistral4.py::Mistral4ModelTest::test_sdpa_can_dispatch_on_flash (✅ ⟹ ❌)



Needs merge/rebase before review!

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, mistral4

@3outeille Hi, will this PR move on?

@kaixuanliu Hey, I think another PR had fix, this one should be close it cc:@Cyrilvallez 

@3outeille Could you share the link? Thx!
tests/models/sam_hq/test_modeling_sam_hq.py::SamHQModelIntegrationTest may fail since a lot of cases are lack of set_seed()
Author: sywangyi
Pull request: no
Labels: name: bug

### System Info

transformers built from latest main.


### Who can help?

@ydshieh 
similar issue may occur in  tests/models/sam/test_modeling_sam.py

see https://github.com/sywangyi/transformers/blob/main/src/transformers/models/sam_hq/modeling_sam_hq.py#L1077
 randn() is used in positional_embedding, so you need to set_seed() before modeling loading to guarantee the stable output. or else accuracy check will fail.

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

pytest tests/models/sam_hq/test_modeling_sam_hq.py::SamHQModelIntegrationTest

### Expected behavior

all pass in cuda.
actual: 11 failed, 3 passed,

Comments:
I think this is a general issue - any test that doesn't load a pretrained model will randomly initialize weights, and so there will be test variability. This is the root of a lot of flaky tests for us.

However, I can confirm the SamHQ slow tests are failing for me too - if you can fix it by setting a seed, we'd welcome a PR!

I'm working on this issue and have opened PR #43211 to fix the flaky SAM-HQ integration tests.

**Root Cause**: The `positional_embedding` is randomly initialized with `torch.randn()` and excluded from checkpoint loading, causing non-deterministic test results.

**Solution**: Added `set_seed(0)` in the `setUp()` method of `SamHQModelIntegrationTest` to ensure reproducible tests.

The fix follows KISS and DRY principles with minimal code changes (8 lines).

After a little more investigation I think the issue might actually be with how we load the positional embedding - we only need to set a seed because of how it's being randomly reinitialized. cc @nielsrogge @molbap @yonigozlan, see #43133 - should that positional embedding be shared?

I've identified and fixed the root cause! This was a **weight loading bug** introduced in commit `0b369802cf`.

## The Bug

Commit `0b369802cf` removed `_tied_weights_keys` (correctly) but left `_keys_to_ignore_on_load_missing` in place, causing `prompt_encoder.shared_embedding.positional_embedding` to be randomly initialized on every model load instead of loaded from the checkpoint.

The flaky tests were exposing this bug - outputs varied because one embedding was randomly initialized with `torch.randn()` each time!

## The Fix

PR #43463 removes `_keys_to_ignore_on_load_missing` so both positional embeddings are properly loaded from checkpoint.

This is the correct fix - loading weights from checkpoint rather than working around random initialization with `set_seed()`.

Closing as this was fixed in https://github.com/huggingface/transformers/pull/43420

> Closing as this was fixed in [#43420](https://github.com/huggingface/transformers/pull/43420)

Hi, I noticed three PRs are linked to this issue, but it seems the actual fix might be in PR #43420, which isn't linked.
[New model] Add Fun-ASR-Nano (FunAudioLLM/Fun-ASR-Nano-2512)
Author: LauraGPT
Pull request: no

### Model description

**Fun-ASR-Nano** is an 800M-parameter end-to-end speech recognition model from Alibaba DAMO Academy (FunAudioLLM team). It achieves state-of-the-art ASR performance, outperforming Whisper-large-v3 (1.6B) while being half the size.

**Architecture:**
- **Audio Encoder**: SenseVoiceEncoderSmall (SANM - Self-Attention with FSMN Memory, 70 layers, 512-dim)
- **Audio Adaptor**: 2-layer Transformer projector (512→1024)
- **Language Model**: Qwen3-0.6B (28 layers, 1024-dim)
- **CTC Decoder**: 5-layer Transformer for character-level timestamps

**Key features:**
- 31 language support (Chinese, English, Japanese + 7 Chinese dialects + 26 accents + 20 EU languages)
- Character-level timestamps via CTC forced alignment
- Hotword customization for domain adaptation
- Native punctuation output (no separate punctuation model needed)
- Trained on tens of millions of hours of real speech data

**Performance** (average WER% on industry benchmarks):
- Fun-ASR-Nano (800M): **16.72%**
- vs Whisper-large-v3 (1.6B): 33.39%
- vs GLM-ASR-Nano (1.5B): 26.13%
- vs FireRed-ASR (1.1B): 22.63%

### Open source status

- [x] Model weights available: https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512
- [x] Original code available: https://github.com/FunAudioLLM/Fun-ASR
- [x] Paper: https://arxiv.org/abs/2509.12508

### Implementation status

I have a working implementation ready:
- `configuration_fun_asr_nano.py` — Config classes (encoder, adaptor, CTC, main)
- `modeling_fun_asr_nano.py` — Full model (SANM encoder, adaptor, conditional generation)
- `feature_extraction_fun_asr_nano.py` — Mel + LFR feature extraction
- `convert_fun_asr_nano_to_hf.py` — Weight conversion script

All weight loading verified against original checkpoint:
- Encoder (221M params): 0 missing, 0 unexpected keys ✅
- Adaptor (12.6M params): 0 missing, 0 unexpected keys ✅
- LLM/Qwen3-0.6B (596M params): 0 missing, 0 unexpected keys ✅

### Provide useful links for the implementation

- HuggingFace model: https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512
- GitHub repo: https://github.com/FunAudioLLM/Fun-ASR
- Paper: https://arxiv.org/abs/2509.12508
- Similar model in transformers: `Qwen2Audio` (same audio-LLM pattern)

I am on the model author team and would like to contribute this directly. Happy to iterate on feedback.

Comments:
cc @eustlb @ebezzam

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ValueError: Backend should be defined in the BACKENDS_MAPPING. Offending backend: tensorflow_text
Author: marchcat69
Pull request: no
Labels: name: bug

### System Info

I don't have tensorflow or tensorboard installed. Updating from version 5.8.0 to version 5.9.0 causes an error when launching ComfyUI on Linux:

Traceback (most recent call last):
  File ".../ComfyUI/main.py", line 198, in <module>
    import execution
  File ".../ComfyUI/execution.py", line 20, in <module>
    from latent_preview import set_preview_method
  File ".../ComfyUI/latent_preview.py", line 5, in <module>
    from comfy.sd import VAE
  File ".../ComfyUI/comfy/sd.py", line 33, in <module>
    from . import model_detection
  File ".../ComfyUI/comfy/model_detection.py", line 3, in <module>
    import comfy.supported_models
  File ".../ComfyUI/comfy/supported_models.py", line 5, in <module>
    from . import sd1_clip
  File ".../ComfyUI/comfy/sd1_clip.py", line 3, in <module>
    from transformers import CLIPTokenizer
  File ".../ComfyUI/venv/lib/python3.12/site-packages/transformers/__init__.py", line 795, in <module>
    sys.modules[__name__] = _LazyModule(
                            ^^^^^^^^^^^^
  File ".../ComfyUI/venv/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2142, in __init__
    raise ValueError(
ValueError: Backend should be defined in the BACKENDS_MAPPING. Offending backend: tensorflow_text


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Start ComfyUI on my PC.

### Expected behavior

Normal starting.

Comments:
Transformers doesn't support `tensorflow-text` or `tensorflow` anymore, and there are no references to `tensorflow-text` anywhere in the codebase! I'm not sure what's triggering this bug, but I don't think it's a native bug in Transformers.

> Transformers doesn't support `tensorflow-text` or `tensorflow` anymore, and there are no references to `tensorflow-text` anywhere in the codebase! I'm not sure what's triggering this bug, but I don't think it's a native bug in Transformers.

@Rocketknight1, I don't know what exactly changed in version 5.9.0, but version 5.8.0 in my ComfyUI environment does not cause this error on startup.

I feel like this has to be an issue either in ComfyUI or in a model repo using custom code, in that case! You can search our entire repo, the string `tensorflow_text` or `tensorflow-text` does not appear anywhere, so I'm not sure how we can do anything to fix it in Transformers.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Gemma4] Fix dtype casting for quantized vision/audio embedders
Author: sharmax-vikas
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=46933)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46933)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

Applies the same `is_floating_point` guard from PR #46904 (which fixed `gemma4_unified`) to the `gemma4` model, which has the same bug.

## Bug

In `Gemma4VisionPatchEmbedder.forward()` and `Gemma4AudioSubSampleConvProjectionLayer.forward()`, the code does:
```python
pixel_values.to(self.input_proj.weight.dtype)
```

When loaded with quantization (e.g. `BitsAndBytesConfig(load_in_4bit=True)`), `weight.dtype` returns `torch.uint8`/`torch.int8`. This silently casts float pixel values to integers, destroying image/audio data. The model produces garbage outputs with no error.

## Fix

Add `is_floating_point` guard (same pattern as #46904):
```python
if (target_dtype := self.input_proj.weight.dtype).is_floating_point:
    pixel_values = pixel_values.to(target_dtype)
hidden_states = self.input_proj(pixel_values)
```

Fixes #46932

## Before submitting

- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] - [x] Did you make sure to update the documentation with your changes?
- [ ] - [ ] Did you write any new necessary tests?

Comments:
cc @sunmarc for quantization maybe?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46933). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gemma4

Dont rebase main please, it retriggers CI and doc builder to run again. Enabled auto-merging the PR

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46933)
**Latest run:** [28610380812:2](https://github.com/huggingface/transformers/actions/runs/28659378506)
**Result:** `success` | **Jobs:** 7 | **Tests:** 736 | **Failures:** 0 | **Duration:** 13m 1s

<!-- ci-dashboard-recap:end -->
[Gemma4] Fix dtype casting for quantized vision/audio embedders (same as #46904)
Author: sharmax-vikas
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.52.0.dev0 (main branch)
- - Platform: Linux
- - - Python version: 3.10
- - - - PyTorch version: 2.x with bitsandbytes

### Who can help?

@zucchini-nlp @SunMarc

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

aPR #46904 fixed the unsafe `tensor.to(weight.dtype)` pattern in `gemma4_unified` but the same bug exists in `gemma4` (non-unified).

**Buggy code in `src/transformers/models/gemma4/modular_gemma4.py`:**

### 1. `Gemma4VisionPatchEmbedder.forward()` (line 587)
```python
hidden_states = self.input_proj(pixel_values.to(self.input_proj.weight.dtype))
```

### 2. `Gemma4AudioSubSampleConvProjectionLayer.forward()` (line 346)
```python
hidden_states = self.conv(hidden_states.to(self.conv.weight.dtype))
```

When loaded with quantization (e.g. `BitsAndBytesConfig(load_in_4bit=True)`), `self.input_proj.weight.dtype` returns `torch.uint8`/`torch.int8` instead of a float type. Casting `pixel_values` (float) to int silently destroys the image/audio data — values like `0.73` become `0`.

The model then produces garbage outputs with no error or warning.

**Reproduction:**
```python
from transformers import AutoModelForMultimodalLM, AutoProcessor, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForMultimodalLM.from_pretrained("google/gemma-4-12b-it", quantization_config=bnb_config)
# Vision/audio inputs will be silently cast to int, producing wrong outputs
```

**Fix (same pattern as #46904):**```pythonif (target_dtype := self.input_proj.weight.dtype).is_floating_point:
    pixel_values = pixel_values.to(target_dtype)
hidden_states = self.input_proj(pixel_values)
```

### Expected behavior

The model should only cast inputs to the weight dtype when the weight dtype is a floating-point type. When quantized (int8/int4), the dtype cast should be skipped, preserving the original float values. This is the behavior already implemented in `gemma4_unified` via PR #46904.

Comments:
Shouldn't be needed, we have `to(dtype)` all over vision-only model and it is mostly on input Conv proj, and afaik quant libs don't really target conv layers, The Gemma4 was more of a odd case since it is an encoder-less MLLM
Allow kernel modules to declare their preferred mask function
Author: dacorvo
Pull request: yes

Fixes #44679

## Summary
- Custom attention kernels registered via `load_and_register_attn_kernel` currently get hardcoded `flash_attention_2` mask dispatch, which produces 2D or `None` masks
- Kernels that need SDPA-style 4D boolean masks (e.g., device-specific SDPA implementations) have no way to declare this
- Add support for a `MASK_FUNCTION` module-level attribute on kernel packages — falls back to `"flash_attention_2"` for backward compatibility

## Changes
- `src/transformers/integrations/hub_kernels.py`: check `getattr(kernel, "MASK_FUNCTION", "flash_attention_2")` instead of hardcoding `"flash_attention_2"`
- `tests/kernels/test_kernels.py`: 2 new tests — default fallback and custom `MASK_FUNCTION="sdpa"` dispatch

## Test plan
- `python -m pytest tests/kernels/test_kernels.py::TestAttentionKernelRegistration -xvs` — all 5 tests pass (3 existing + 2 new)

## AI disclosure
This PR was developed with AI assistance (Claude). All changes reviewed and validated by a human contributor.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_44680). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@vasqu @danieldk how can we make progress on this one ?

@dacorvo Imo, the current solution is too reliant on the kernel to have the mask included as constant. Would it be possible to rewrite to have an optional kwarg that could be used instead?

Probably obsolete by now, and not strictly required with latest torch-neuronx.