noamaanMulla-03/transformers-issues huggingface dataset:
Add Shared Vision–Language Negative Sampling Utility for BLIP / LLaVA / IDEFICS
Author: Aki-07
Pull request: no
Labels: name: Feature request

### Feature request

Add a unified vision-language negative sampling utility for contrastive learning. Support hard negative mining and in-batch negative sampling. Compatible with BLIP, LLaVA, and IDEFICS training pipelines. Modular design for easy integration into existing VLM frameworks

### Motivation

VLM training still relies on ad-hoc contrastive negative sampling code.
A common utility to batch image–text negatives (with optional hard-negative mining) would simplify research pipelines and encourage standardization.

### Your contribution

If given a approval, will start working on this
Contributor: [@Aki-07](https://github.com/Aki-07)

Comments:
Hard negative mining definitely feels out of scope! cc @sunmarc if you want it in the Trainer, but this feels like something that should probably live outside `transformers`

Hey @Aki-07, sorry for the slow reply here, we're catching up on the backlog.

Thanks for the thoughtful proposal, but as @Rocketknight1 noted, a contrastive negative-sampling utility (in-batch + hard-negative mining) really sits outside the scope of transformers: it's a training/data-pipeline concern tightly coupled to a specific loss and research setup, rather than a modeling primitive we maintain per-architecture. Adding it would introduce a new public API surface across the Trainer and several VLM model paths for a fairly niche use case, and we try to keep that footprint small since anything we ship is effectively permanent.

This is a great fit for a standalone library or an examples/recipe that others can build on, happy to point people to it if you publish one. Closing as out of scope, but thanks for offering to contribute!
Wire DistributedConfig through from_pretrained and save_pretrained (FSDP orchestration 2/3)
Author: 3outeille
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47353)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47353)
<!-- ci-dashboard-badge:end -->

## Summary

Part of FSDP orchestration stack (2/3). Replaces #46990. Depends on #47352.

- Wire `prepare_distribute_model` / `maybe_distribute_model` into `from_pretrained`
- Add FSDP gather and DCP (`distributed_checkpoint=True`) paths in `save_pretrained`
- **Breaking:** remove `tp_plan` / `tp_size` kwargs from `from_pretrained`; use `distributed_config=DistributedConfig(tp_size=N)` instead
- Migrate TP tests and docs to explicit `DistributedConfig`

## Stack

#47352 (1/3) → this PR (2/3) → 3/3

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47353). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: deepseek_v4, glm_moe_dsa, gpt_oss

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47353)
**Latest run:** [29468406141:1](https://github.com/huggingface/transformers/actions/runs/29470570537)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 171,595 | **Failures:** 161 | **Duration:** 14h 5m

<!-- ci-dashboard-recap:end -->
Add FSDP CI and end-to-end FSDP tests + save/load fsdp
Author: 3outeille
Pull request: yes

close for https://github.com/huggingface/transformers/pull/47357

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47354). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47354)
**Latest run:** [29470574675:1](https://github.com/huggingface/transformers/actions/runs/29473404695)
**Result:** `cancelled` | **Jobs:** 16 | **Tests:** 172,702 | **Failures:** 159 | **Duration:** 11h 20m

<!-- ci-dashboard-recap:end -->

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: cohere2_moe
Re-enable VITS model forward test
Author: IMvision12
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47105)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47105)
<!-- ci-dashboard-badge:end -->

This PR removes a stale skip from `VitsModelTest.test_model_forward`.

The VITS forward pass now runs successfully with the seeded tiny test config, so the test is re-enabled. The old assertion expected a hard-coded waveform shape, which is brittle because VITS predicts output duration dynamically. The test now validates the forward output contract instead:

- waveform output is 2D
- batch size is preserved
- `sequence_lengths` has one value per sample
- waveform length matches the maximum predicted sequence length

Verified with a direct seeded VITS forward run and `py_compile`.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: vits

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47105)
**Latest run:** [28783899000](https://github.com/huggingface/transformers/actions/runs/28783899000)
**Result:** `success` | Grafana metrics are not available yet.

<!-- ci-dashboard-recap:end -->

cc @ydshieh @tarekziade

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47105). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
extend tests/models/qwen3_next/test_modeling_qwen3_next.py::Qwen3Next…
Author: sywangyi
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47184)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47184)
<!-- ci-dashboard-badge:end -->

…ModelTest::test_can_use_device_map and tests/models/videomt/test_modeling_videomt.py::VideomtForUniversalSegmentationIntegrationTest::test_instance_segmentation_inference_bf16 to xpu


- Intel XPU: @IlyasMoutawwakil

Comments:
<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47184)
**Latest run:** [28991127008:2](https://github.com/huggingface/transformers/actions/runs/28991494378)
**Result:** `success` | **Jobs:** 2 | **Tests:** 21 | **Failures:** 1 | **Duration:** 2m 3s

<!-- ci-dashboard-recap:end -->

Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/29480833847) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen3_next, videomt
[CB] [Major] Add multimodal support to continous batching
Author: remi-or
Pull request: yes

This PR adds support for image and text models to continuous batching. 

Tests run, will add image and audio perf numbers later. 

This is more a beta versions, we are missing a bunch of features: CUDA graphs for encoder, multiple modalities at once, etc. But better start from here and build little by little IMO. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46376). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46376&sha=3da698

You might be affected by https://github.com/huggingface/transformers/pull/46405. Hopefully useful for CB since the image embeddings will be of same sizes after it's merged, with each Nth image emb obtainable as features[N]`

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46376)
**Latest run:** [29415017650:1](https://github.com/huggingface/transformers/actions/runs/29480995867)
**Result:** `failure` | **Jobs:** 2 | **Tests:** 19 | **Failures:** 1 | **Duration:** 1m 52s

> **Code quality check failed**: test jobs were skipped. Fix the code quality issues and push again to run tests.

<!-- ci-dashboard-recap:end -->
Fix sdpa prefill with position_bias
Author: Cyrilvallez
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47359)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47359)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

In some rare cases, sdpa can crop k/v (during prefill with StaticCache under some other conditions). In those cases, we need to crop position_bias as well

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47359). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47359)
**Latest run:** [29476276538:1](https://github.com/huggingface/transformers/actions/runs/29477303082)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 171,597 | **Failures:** 162 | **Duration:** 12h 40m

<!-- ci-dashboard-recap:end -->

There are no full text tests for now 🥲 so merging for now, and I'll add one when we look more closely at the tests for the model!
Fix assisted decoding for models with EncoderDecoder cache & OlmoHybrid
Author: Cyrilvallez
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47361)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47361)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

As per the title. `_assisted_decoding` now calls `activate_past_recording`, which needs to be overwritten for `EncoderDecoderCache` as otherwise we hit an AttributeError
ALso fixes OlmoHybrid

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47361). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47361)
**Latest run:** [29480818982:1](https://github.com/huggingface/transformers/actions/runs/29480818982)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 167,204 | **Failures:** 3 | **Duration:** 14h 39m

<!-- ci-dashboard-recap:end -->

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: lfm2_vl, minicpmv4_6, olmo_hybrid
ci: Handle empty GitHub token in CI run lookup
Author: tarekziade
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47362)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47362)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

Fixes errors when the token is an empty screen

```
Traceback (most recent call last):
  File "/home/runner/work/transformers/transformers/utils/collated_reports.py", line 217, in <module>
    upload_collated_report(job, report_repo_id, filename)
  File "/home/runner/work/transformers/transformers/utils/collated_reports.py", line 123, in upload_collated_report
    workflow_run_created_time = workflow_run["created_at"]
KeyError: 'created_at'
```

Comments:
Example of failures: https://github.com/huggingface/transformers/actions/runs/29477626344/job/87558209323
run with this fix: https://github.com/huggingface/transformers/actions/runs/29480225115/job/87563403234

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47362)
**Latest run:** [29479758474:1](https://github.com/huggingface/transformers/actions/runs/29480209540)
**Result:** `success` | **Jobs:** 4 | **Tests:** 264 | **Failures:** 0 | **Duration:** 2m 39s

<!-- ci-dashboard-recap:end -->

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47362). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Is this also used for run-slow? Can we cross check?

no it's unrelated

Then I have another report to make 😢 I've seen run-slow always failing to report at the end, so thought it mightve been related. (The tests run, just the report at the end fails)

> Then I have another report to make 😢 I've seen run-slow always failing to report at the end, so thought it mightve been related. (The tests run, just the report at the end fails)

I've double-checked, and in fact this will indeed also fix run-slow reports
Add FSDP CI and end-to-end FSDP tests + save/load fsdp
Author: 3outeille
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47357)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47357)
<!-- ci-dashboard-badge:end -->

## Summary

Part of FSDP orchestration stack (2/2). Replaces #46990. #47352 (1/2) → this PR (2/2)

- TODO: Add FSDP gather and DCP (distributed_checkpoint=True) paths in save_pretrained
- Add `FSDPTesterMixin` and `tests/test_fsdp_mixin.py` (load/save/generation)
- Add `base_model_fsdp_plan` to cohere2_moe (only model in this stack)
- Add dedicated `fsdp_ci` CircleCI job and pytest markers

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47357). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: cohere2_moe

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47357)
**Latest run:** [29477587778:1](https://github.com/huggingface/transformers/actions/runs/29483891689)
**Result:** `failure` | **Jobs:** 14 | **Tests:** 142,585 | **Failures:** 90 | **Duration:** 11h 6m

<!-- ci-dashboard-recap:end -->
[`FP8`] Bump kernels version
Author: vasqu
Pull request: yes
Labels: name: for patch

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47344)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47344)
<!-- ci-dashboard-badge:end -->

As per title, sometimes during auto device it can lead to NaNs otherwise. NOTE: It does not use the fused paths

Comments:
<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47344)
**Latest run:** [29431402626](https://github.com/huggingface/transformers/actions/runs/29431402626)
**Result:** `success` | Grafana metrics are not available yet.

<!-- ci-dashboard-recap:end -->

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47344). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Fix] Small FA-related test failures in CB
Author: remi-or
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47341)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47341)
<!-- ci-dashboard-badge:end -->

This PR fixes two flash-attention related issues in CB:
- block table is not supported for sliding window attention, yet the order of checks makes it so that a sliding attention model can turn on the block table fast path if it uses FA2. This was previously not an issue because FA2 disable the block table fast path. Shuffling two code blocks in the init fixes this.
- some tests, especially those with Flash attention and compile, can be flaky, when generating using CB or `generate`. For those tests, we allow the possibility of comparing CB to a reference computed not with the same dtype, attention implementation and compile, but a high-precision (fp32) eager reference. If the test passes, it means CB is in accordance with an fp32 eager model, which IMO is a safe baseline. The difference with a `generate` call with FA + compile can be explained in many ways: compile fuses kernel differently because of the static cache, FA acts differently because of the different batching scheme, etc... 

This fixes a total of 7 tests on `main`, which did not use to fail before, so this is kind of a mystery of how we got there. For static cache, it might be because of recent changes in the masking scheme, and for sliding window issue, I think the test was not run when we enabled sliding window, or got lost in the refactors (this part of the code, the cache init, moved quite a bit recently).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47341). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47341)
**Latest run:** [29470002098:1](https://github.com/huggingface/transformers/actions/runs/29486798533)
**Result:** `success` | **Jobs:** 15 | **Tests:** 171,597 | **Failures:** 158 | **Duration:** 11h 56m

<!-- ci-dashboard-recap:end -->
[CB] Automatically switch attention implementation to flash
Author: remi-or
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47330)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47330)
<!-- ci-dashboard-badge:end -->

This PR change the behavior of CB when it comes to `attn_implementation`.

Before, we only ensured that the attention was paged, which is necessary for CB. With this PR, it also checks if flash is available (version 3 then 2, either through the package or through `kernels`) and if it is, it automatically switch to a flash implementation. 
The process throws a warning to make this clear to the user, and also throws a warning if flash is not available.

The rationale behind this is that flash is supported on more and more devices (CUDA, ROCm, CPU, XPU, ...) and using CB without flash one of its bigger advantage, ie. batching ragged sentence together without paying a huge compute cost.  
Eager and SDPA use an attention mask, which has the size of `total_query_tokens * total_kv_tokens` (including the ones read from the cache). This mask is:
1. big 
2. long to build 
3. long to move around 
4. very big.

The biggest winners of this change are going to be run-of-the-mill users who start a model with a default attn implementation like SDPA. Instead of seeing minimal differences between `generate` and `generate_batch`, they will have access to the full potential of CB.

Added a test, it passes. There are some flaky test on main but these will be closed in a (soon-to-be-open) PR: non-flaky tests pass for this PR ✅ 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47330). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47330)
**Latest run:** [29483269699:1](https://github.com/huggingface/transformers/actions/runs/29483269699)
**Result:** `success` | **Jobs:** 15 | **Tests:** 140,829 | **Failures:** 0 | **Duration:** 14h 13m

<!-- ci-dashboard-recap:end -->
Add beats model
Author: eslam-ahmed43
Pull request: yes

## BEATs Audio Model

BEATs ([paper](https://arxiv.org/abs/2212.09058)) is a Microsoft audio model 
that's widely used in audio research (BirdCLEF, DCASE) but didn't have a 
native Transformers integration yet. This adds it.

**Includes:**
- `BEATsConfig`, `BEATsModel`, `BEATsForAudioClassification`
- `BEATsFeatureExtractor` — uses `torchaudio.compliance.kaldi` to match 
  the original fbank extraction
- Conversion script tested against `BEATs_iter3_plus_AS2M.pt` (max_diff = 0.0)
- AutoModel / AutoModelForAudioClassification integration
- Docs and tests

**Two things worth calling out:**

The `pos_conv` in the original uses weight norm. During conversion I reconstruct 
the effective weight from `weight_g` and `weight_v` directly, so the HF version 
loads standard parameters without needing weight norm at inference.

The GRU relative position gating (`grep_linear`) has a fixed output size of 8 — 
this isn't obvious from the paper and took some digging in the original code.

Comments:
Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/29475028031) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, beats

cc @ebezzam @eustlb