noamaanMulla-03/transformers-issues huggingface dataset:
[serge] integration failure triage - 2026-07-08
Author: github-actions[bot]
Pull request: no

<!-- serge-triage-run:integration-failure-triage:2026-07-08 -->

Automated **integration-failure triage** for the daily CI window `2026-07-02 → 2026-07-08`.

This issue was generated by AI-assisted automation. The grouping, summaries, and recommended follow-up can be incomplete or misleading; verify the failures before acting.

Serge dispatched one task per failure group below — each opens or updates its own PR on a `serge/fix/itf-<fingerprint>` branch. This table is refreshed in place as Serge runs: a group links its `#<pr>` when opened, shows `🚫 no fix` when Serge found no safe change, `⚠️ task failed` on error, or `(pending)` while still running (a late PR links on the next nightly run).

## Dispatched failure groups

| Model | Error | Occurrences | PR |
| --- | --- | --- | --- |
| `whisper` | other — other (20), shape/size mismatch (2) | 22 | `serge/fix/itf-cae056ff1f13` (pending) |
| `whisper` | output_mismatch — list output differs (10), other (6), tensor values differ (4) | 20 | #47150 |
| `musicgen_melody` | output_mismatch — tensor values differ (16) | 16 | 🚫 no fix |
| `gemma` | output_mismatch — list output differs (14) | 14 | 🚫 no fix |
| `dac` | output_mismatch — tensor values differ (6), other (6) | 12 | 🚫 no fix |
| `edgetam` | import_or_config — other (12) | 12 | #47122 |
| `glm46v` | other — other (12) | 12 | 🚫 no fix |
| `minimax_m3_vl` | load_error — other (12) | 12 | #47123 |
| `diffusion_gemma` | output_mismatch — tensor values differ (6), list output differs (4) | 10 | 🚫 no fix |
| `gemma3` | output_mismatch — list output differs (8), other (2) | 10 | 🚫 no fix |
| `gemma3n` | output_mismatch — list output differs (10) | 10 | 🚫 no fix |
| `glm_ocr` | output_mismatch — list output differs (10) | 10 | 🚫 no fix |
| `deepseek_v32` | load_error — other (8) | 8 | 🚫 no fix |
| `falcon_mamba` | output_mismatch — other (6), list output differs (2) | 8 | #47124 |
| `grounding_dino` | output_mismatch — tensor values differ (6), other (2) | 8 | 🚫 no fix |
| `higgs_audio_v2` | output_mismatch — tensor values differ (8) | 8 | 🚫 no fix |
| `mllama` | output_mismatch — other (8) | 8 | 🚫 no fix |
| `moshi` | output_mismatch — other (8) | 8 | 🚫 no fix |
| `mpt` | other — other (8) | 8 | 🚫 no fix |
| `peft_integration` | other — other (8) | 8 | 🚫 no fix |

_Generated 2026-07-09T00:47:17+00:00._

Comments:
Superseded by #47227 — closing this stale nightly integration-failure triage issue. Open PRs stay linked from the current issue.
add paged attention tests support for XPU
Author: kaixuanliu
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47163)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47163)
<!-- ci-dashboard-badge:end -->

Comments:
<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47163)
**Latest run:** [29073350562:2](https://github.com/huggingface/transformers/actions/runs/29073350562)
**Result:** `success` | **Jobs:** 2 | **Tests:** 21 | **Failures:** 0 | **Duration:** 1m 35s

<!-- ci-dashboard-recap:end -->

@ydshieh pls help review, thx!
Fix DistributedConfig docstring for unimplemented sp_plan 
Author: 3outeille
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47237)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47237)
<!-- ci-dashboard-badge:end -->

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47237). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47237)
**Latest run:** [29070955320:1](https://github.com/huggingface/transformers/actions/runs/29074529571)
**Result:** `success` | **Jobs:** 2 | **Tests:** 24 | **Failures:** 0 | **Duration:** 1m 47s

<!-- ci-dashboard-recap:end -->
TP dtensor
Author: 3outeille
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47194)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47194)
<!-- ci-dashboard-badge:end -->

# Summary 

- Unified TP runtime in `distributed/tensor_parallel.py`: `model.train()` uses DTensor redistribute; `model.eval()` uses plain `all-reduce` / `all-gather`
- For testing purpose, only `Cohere2Moe` is able to easily switch between training (`dtensor`) & inference (plain `all-reduce`).  Other models (`Llama, Mistral, Mixtral, Qwen3`) stay on the legacy TP backend 
- `apply_tensor_parallel()` dispatches to the new DTensor backend or the legacy integrations/tensor_parallel path if `enable_expert_parallel=False` and `style string in base_model_tp_plan use the new ALL_PARALLEL_STYLES registry`
- **Loading/saving hooks**: DTensor `shard-on-read` loading; `gather_tp_state_dict()` for TP checkpoint save
- `tp_size` comes from the `device_mesh`, not from `distributed_config` field


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47194). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: cohere2_moe

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47194)
**Latest run:** [29078143863:1](https://github.com/huggingface/transformers/actions/runs/29078143863)
**Result:** `failure` | **Jobs:** 16 | **Tests:** 173,676 | **Failures:** 24 | **Duration:** 21h 47m

<!-- ci-dashboard-recap:end -->
Be more defensive with `remap_legacy_layer_types` for custom models
Author: hmellor
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47245)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47245)
<!-- ci-dashboard-badge:end -->

https://github.com/huggingface/transformers/pull/47174 unconditionally calls `setattr` on `layer_types` even if they are unchanged. This causes errors on models where `layer_types` is a read-only property (i.e. Plamo3).

This PR updates this logic so `setattr` is only called if the the names changed, so this will happen less often.

Comments:
> why do some even have them read only

Who knows 🤷 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47245). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47245)
**Latest run:** [29094121906:1](https://github.com/huggingface/transformers/actions/runs/29094121906)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 170,734 | **Failures:** 3 | **Duration:** 18h 49m

<!-- ci-dashboard-recap:end -->

Merging since it's important for vLLM and the failing tests are known!
Fix FA2 long-context perf regression from removed .item() sync (#46693)
Author: ammmanism
Pull request: yes

What does this PR do?
Fixes #46693 — FA2 long-context generation perf regression (5.2.0 → 5.4.0+, ~17% slower on large KV caches).
Supersedes #47215 (closed — that branch had picked up unrelated commits from other local branches, causing CI/security-gate noise unrelated to this fix). This PR is scoped to exactly 3 files.
Root cause
PR #42435 removed .item() calls on max_seqlen_in_batch in three places, leaving it as a 0-dim CUDA tensor instead of a Python int:

_get_unpad_data()
_unpad_input()
prepare_fa_kwargs_from_position_ids()

These tensors are passed directly into flash_attn_varlen_func(max_seqlen_q=..., max_seqlen_k=...). On bandwidth-saturated GPUs with large KV caches (RTX 6000 Pro noticeably more than RTX 4090), this adds kernel-dispatch/sync overhead per decode step that compounds over long generations.
Fix
Restored .item() at all three call sites, guarded by is_torchdynamo_compiling() so torch.compile graph capture isn't broken — tensor form preserved under compile, scalar form used in eager. Also updated _process_flash_attention_kwargs() to convert to int in eager mode consistently (previously only converted during tracing).
Verification

Regression test asserting scalar type in eager mode (tests/utils/test_flash_attention_utils.py)
Standalone microbenchmark isolating the mechanism (bench_fa2_unpad.py)
ruff check / ruff format --check clean, utils/check_copies.py passes

Relates to the investigation in #47134 (draft) — this should close the residual gap flagged there.
cc @vasqu @zucchini-nlp @andreasgoulas
Before submitting

 Did you read the contributor guideline and the Pull Request checks?
 Was this discussed/approved via a GitHub issue? Yes — #46693
 Did you write any new necessary tests?

Comments:
Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/29095868080) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

Thanks for the review and the correction on the compile-mode guard — makes sense, appreciate you catching that. Since #47134 is merged and already resolves the regression, closing this in favor of that fix.
Fix CI read-only cache failures by patching cached_files in conftest
Author: ydshieh
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47043)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47043)
<!-- ci-dashboard-badge:end -->

## Context

In CPU CI, the shared HF cache is read-only (pre-populated with most models). Tests that call `from_pretrained` with a model not yet in the cache — or one with updated files — hit `OSError: [Errno 30] Read-only file system` and fail with an opaque error.

**This is a temporary workaround.** We are in discussion with the infra team to evaluate whether a proper writable cache system is feasible. Until then, we want a lightweight fix that handles the common cases without over-engineering.

## Why not #46768

[#46768](https://github.com/huggingface/transformers/pull/46768) addressed this with a per-test `@use_temp_cache_if_readonly` decorator. We moved away from that approach because it is opt-in and contributors adding a new test can forget it.

## Why patch `cached_files` instead of `from_pretrained`

Patching `from_pretrained` directly would require importing several large classes (`PreTrainedModel`, `PreTrainedTokenizerBase`, `PretrainedConfig`, `ImageProcessingMixin`, etc.) into `conftest.py`, which we want to avoid. `transformers.utils.hub.cached_files` is the common code path that all `from_pretrained` implementations go through for hub downloads, so a single patch there covers everything with no class imports.

## What this does

Wraps `transformers.utils.hub.cached_files` in `conftest.py` with a try/except that catches `EROFS` only. On failure, a session-scoped tmp dir is created (lazily, once) and the call is retried with `HF_HUB_CACHE` and `HF_XET_CACHE` patched to it. All other tests continue to read from the pre-populated read-only cache as before.

This covers the common cases well enough for a temporary fix. Edge cases that don't go through `cached_files` are out of scope for now.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47043). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: vit

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47043)
**Latest run:** [29095910795:1](https://github.com/huggingface/transformers/actions/runs/29096046001)
**Result:** `success` | **Jobs:** 2 | **Tests:** 17 | **Failures:** 0 | **Duration:** 1m 22s

<!-- ci-dashboard-recap:end -->
[fix] fix requirements audio feature and proc
Author: eustlb
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47113)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47113)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

`@requires` and `requires_backend` were not used when necessary. Almost impactless for the user when the missing is `torch` since the error is pretty clear, but much more problematic for eg Voxtral when trying to call TranscriptionRequest with a cryptic not defined error.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audio_spectrogram_transformer, audioflamingo3, csm, dia, glmasr, higgs_audio_v2, musicflamingo, nemotron3_5_asr, voxtral, xcodec2

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47113). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47113)
**Latest run:** [28809807142:1](https://github.com/huggingface/transformers/actions/runs/28809807142)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 171,369 | **Failures:** 6 | **Duration:** 25h 38m

<!-- ci-dashboard-recap:end -->
Fix InputTokensDetails missing cache_write_tokens for openai>=2.34.0
Author: ydshieh
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47248)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47248)
<!-- ci-dashboard-badge:end -->

## What this does

`openai==2.34.0` added `cache_write_tokens` as a required field to `InputTokensDetails`. This causes `compute_usage` to fail with a `ValidationError` when running with `openai>=2.34.0` (CI currently uses `openai==2.45.0`).

## Fix

Passes `cache_write_tokens=0` when the installed openai version has that field, using `model_fields` to stay compatible with older versions (`openai<2.34.0`).

Comments:
the change seems to handle both versions (claude told me that and I 120% trust it).

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47248)
**Latest run:** [29096452736](https://github.com/huggingface/transformers/actions/runs/29096452736)
**Result:** `success` | Grafana metrics are not available yet.

<!-- ci-dashboard-recap:end -->

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47248). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
tests only for the top 10 download models
Author: 3outeille
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47244)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47244)
<!-- ci-dashboard-badge:end -->

# Summary

- Tensor parallel CI was previously ran for all causal llm models that had a `base_tp_plan` in their configuration file. To cut down on CI time,  we will limit ourselves to testing only the top 10 most download causal llm.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47244). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

_skip_if_ep_not_supported is defined but never called — the EP tests (test_ep_forward, test_ep_forward_from_pretrained) still use _skip_if_not_supported(expert_parallel=True). Either wire up the new method or remove it.

yeah merge conflict, was solving it

Otherwise lgtm, make sure you check the run page (on dashboard) and confirmed it works well .

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47244)
**Latest run:** [29095639770:1](https://github.com/huggingface/transformers/actions/runs/29096142551)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 170,706 | **Failures:** 2 | **Duration:** 13h 7m

<!-- ci-dashboard-recap:end -->
Revert "Trigger a scheduled run"
Author: ydshieh
Pull request: yes

Reverts huggingface/transformers#47209

It was just to trigger a manual scheduled run. We still want it to be run in the previous time.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47249). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Potential host synchronization in deepspeed_sp_compute_loss due to Python conditional on CUDA tensor
Author: hdimmfh
Pull request: no
Labels: name: bug

### System Info

- Transformers: 4.57.2
- PyTorch: 2.7.1+cu128
- DeepSpeed: 0.17.6
- CUDA: 12.8
- GPU: NVIDIA B300 SXM6 (2 nodes × 8 GPUs, 16 GPUs total)
- Interconnect:
  - Intra-node: NVSwitch
  - Inter-node: InfiniBand

### Information

While profiling a distributed training workload with Nsight Systems, I observed what appears to be repeated host synchronization inside `deepspeed_sp_compute_loss`. The relevant code is located in: `src/transformers/integrations/deepspeed.py`

```python
good_tokens_per_rank = torch.distributed.nn.functional.all_gather(good_tokens, group=sp_group)

total_loss = sum(
    losses_per_rank[rank] * good_tokens_per_rank[rank]
    for rank in range(sp_world_size)
    if good_tokens_per_rank[rank] > 0
)
```

Assuming `good_tokens_per_rank` contains CUDA tensors, the Python conditional appears to evaluate a CUDA tensor in Python, which may introduce an implicit host synchronization.

https://github.com/huggingface/transformers/blob/main/src/transformers/integrations/deepspeed.py
```python
if good_tokens_per_rank[rank] > 0
```

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Possible implementation

Would it make sense to avoid the Python conditional and perform the filtering entirely on the GPU?
For example, something along the lines of:

```python
good_tokens = torch.stack(good_tokens_per_rank)
losses = torch.stack(losses_per_rank)

mask = good_tokens > 0

# Preserve the original "skip zero-token ranks" behavior.
safe_losses = torch.where(
    mask,
    losses,
    torch.zeros_like(losses),
)

total_loss = (safe_losses * good_tokens).sum()
```

This could potentially avoid evaluating CUDA tensors in Python while preserving the same semantics. However, I'm not sure whether the current implementation intentionally skips zero-token ranks to avoid NaN propagation or for some other reason, so I'd appreciate your thoughts before attempting a PR.

### Expected behavior

The loss aggregation should avoid unnecessary host synchronization while preserving the current semantics. If this behavior is intentional, it would be helpful to understand the rationale. Otherwise, I'd be happy to submit a PR after discussing the preferred implementation.

Thank you!

Comments:
I would like to open PR for the GPU-side aggregation approach

cc @sunmarc I think
Tar-slip / zip-slip arbitrary file write → RCE ...
Author: geo-chen
Pull request: no
Labels: name: bug

### System Info

Hi, I've reported this on 8 June 2026 to huntr as guided in the security.md policy but have not gotten any response since: https://huntr.com/bounties/ae9336f0-5fc7-4c87-a1d1-67ead0ec6f38

can i share details here? 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

-

### Expected behavior

-

Comments:
I'm working on this issue

If this is about conversion scripts, those are always excluded from release builds! They often have to open insecure formats (e.g. pickle, older `torch` weights), so we cannot guarantee their security. They are intended for advanced developers only.

thanks for the response. that's a fair point on conversion script.
The other item I've reported in the same huntr report is about the path traversal in get_checkpoint_shard_files (src/transformers/utils/hub.py) - which isn't a conversion script, it's a core utility that ships in every release and is hit by the default AutoModel.from_pretrained() call on any local sharded-checkpoint directory (no conversion step involved). it's exploitable against the current transformers==5.13.0 release: a weight_map value of "../elsewhere/file.safetensors" in model.safetensors.index.json gets os.path.join'd with zero ../absolute-path check, and the loader opens/reads the file outside the model directory.