noamaanMulla-03/transformers-issues huggingface dataset:
Fix broken internal documentation links
Author: sezer-muhammed
Pull request: yes

## What does this PR do?

Fixes a number of broken internal documentation links.

| File | Before | After |
|---|---|---|
| `conversations.md` | `./model_doc/gpt-oss` | `./model_doc/gpt_oss` |
| `main_classes/quantization.md` | `../quantization` | `../quantization/overview` |
| `model_doc/granitevision.md` | `(SigLIP)` | `(siglip)` |
| `model_doc/idefics2.md` (×2) | `../quantization` | `../quantization/overview` |
| `model_doc/minimax.md` (×2) | `../quantization` | `../quantization/overview` |
| `model_doc/mixtral.md` (×2) | `../quantization` | `../quantization/overview` |
| `model_doc/xlm-roberta.md` | `../quantization` | `../quantization/overview` |
| `model_doc/seamless_m4t.md` | `./generation_strategies` | `../generation_strategies` |
| `model_doc/splinter.md` | `../tasks/question-answering` | `../tasks/question_answering` |
| `model_doc/switch_transformers.md` | `../quantization/bitsandbytes/` | `../quantization/bitsandbytes` |
| `community_integrations/transformers_as_backend.md` (×4) | `./add_new_model`, `./custom_models`, `./perf_infer_gpu_multi`, `./contributing#…` | `../…` |


## Before submitting

- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46945)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46945). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add Audio-Visual Flamingo model
Author: lashahub
Pull request: yes

This PR adds support for **Audio-Visual Flamingo (AVF)**, an open audio-visual large language model for joint understanding and reasoning over audio, images, and videos.

The paper, model weights, and project page will be released in May 2026.

In Transformers, AVF pairs a **SigLIP** vision tower with an **AF-Whisper** audio encoder and a **Qwen2.5-7B** causal language model, with separate projectors for visual and audio features. For joint video-audio inputs, AVF aligns synchronized visual and audio chunks, interleaves them along the time axis, applies **Constrained Rotary Time Embeddings (CRTE)**, and feeds the fused sequence to the language model. It also supports **Dynamic-S2** preprocessing for high-resolution images and sampled video frames.

This PR introduces:
- `AudioVisualFlamingoConfig`
- `AudioVisualFlamingoForConditionalGeneration`
- `AudioVisualFlamingoProcessor`
- Joint video-audio handling from a single container via `load_audio_in_video=True`
- Dynamic-S2 visual preprocessing
- Temporal audio-visual interleaving with CRTE
- Modeling, processing, tests, and documentation

Example usage once the checkpoint is public:

```python
from transformers import AudioVisualFlamingoForConditionalGeneration, AutoProcessor

model_id = "nvidia/audio-visual-flamingo-hf"

processor = AutoProcessor.from_pretrained(
    model_id,
    padding_side="left",
    use_fast=False,
    load_audio_in_video=True,
    num_video_frames=128,
    audio_chunk_length="max_3600",
)
model = AudioVisualFlamingoForConditionalGeneration.from_pretrained(
    model_id,
    device_map="auto",
    load_audio_in_video=True,
).eval()

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "video", "video": "video.mp4"},
            {
                "type": "text",
                "text": "Describe both the visual scene and the spoken or environmental audio content.",
            },
        ],
    }
]

inputs = processor.apply_chat_template(
    conversation,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(processor.batch_decode(outputs[:, inputs.input_ids.shape[1] :], skip_special_tokens=True)[0])
```

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audiovisualflamingo, auto

Hi @eustlb! Gentle ping on this PR 🙂 Whenever you have a chance, I'd really appreciate a review. I'm happy to address any feedback or make any requested changes. Thanks!
Fix auto-mappings registration for remote code & fixes a few custom code issues
Author: Cyrilvallez
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=46876)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46876)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

This PR solves a few issues:
- Currently, loading remote code with AutoModels was slightly wrong. https://github.com/huggingface/transformers/pull/38474 had applied a fix, but this led remote models with the same `model_type` as a native library model to not be detected by `is_remote_code`, because `_auto_class` was not correctly set. This PR fixes it, so any custom remote model class loaded via AutoModel will always set it, independently of `model_type` collisions with native lib mappings.
- Also, this PR adds a new `is_custom_code` method to `PreTrainedModel`, to detect whether the model is either remote code, or locally defined model (anything that it not imported from `transformers`. This is more comprehensive than `is_remote_code` (which I kept separate for completeness of having both), and is generally what we want to use for any exception paths we may use to help avoid breaking BC for remote code.
- Fixes a few instances of `is_remote_code`, where it should in fact have been `is_custom_code`
- Finally, this PR skips any conversion mapping that may be applied via `model_type`, if the model is a custom model that shares this `model_type` with a library member, which would lead to apply conversions to random custom architectures, which is obviously wrong (see https://github.com/huggingface/transformers/issues/46650 as well)

Also fixes https://github.com/huggingface/transformers/issues/46650

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46876)
**Latest run:** [28346211416:1](https://github.com/huggingface/transformers/actions/runs/28346211416)
**Result:** `failure` | **Jobs:** 14 | **Tests:** 70,236 | **Failures:** 0 | **Duration:** 19h 44m

<!-- ci-dashboard-recap:end -->

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46876). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto
Fix default interpolation to BICUBIC for ViT, EfficientNet, PVT
Author: raimbekovm
Pull request: yes

# What does this PR do?

Fixes incorrect default interpolation in ViT, EfficientNet, and PVT image processors. The original implementations use BICUBIC but HuggingFace defaulted to BILINEAR/NEAREST.

Changes:
- ViT: BILINEAR → BICUBIC
- ViTFast: BILINEAR → BICUBIC  
- EfficientNet: NEAREST → BICUBIC
- PVT: BILINEAR → BICUBIC

Verified against timm - all three use BICUBIC: https://github.com/huggingface/pytorch-image-models/blob/main/timm/data/transforms_factory.py#L75

Note: This is a breaking change for users relying on previous defaults.

Fixes part of #28180

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. #28180
- [ ] Did you make sure to update the documentation with your changes?
- [ ] Did you write any new necessary tests?

## Who can review?

@nielsrogge

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: efficientnet, imagegpt, layoutlmv2, layoutlmv3, pvt, segformer, vit

Quick clarification on the file changes:

The second commit updates 6 additional models because they copy ViT's resize method. After changing ViT from BILINEAR to BICUBIC, these copies needed to stay in sync.

This PR complements the ongoing interpolation fixes (#28180) by @lukepayyapilli for other models.

Let me know if any changes are needed!

This makes sense and aligns ViT preprocessing with expected defaults (e.g. TIMM). Since this changes a default behavior, it might be worth explicitly calling it out as a breaking change for users relying on the previous interpolation. A small regression test asserting the default resampling value could also help prevent accidental future changes.

Hey, did you check this against the original implementation in each case?

Yes, verified against original implementations.

BICUBIC confirmed (timm): ViT, EfficientNet, PVT

Reverted to BILINEAR: ImageGPT, Segformer, LayoutLMv2, LayoutLMv3 - these use BILINEAR in their original repos (OpenAI, MMSegmentation, Microsoft/Detectron2).

Pushed a fix, PR now only changes the 3 models that actually need BICUBIC.

cc @nielsrogge for review in that case!
Fix shape mismatch in KyutaiSpeechToText `generate()` last window
Author: jiqing-feng
Pull request: yes



## Summary

Generating with a `KyutaiSpeechToTextForConditionalGeneration` model crashes when the
streaming `audio_window_size` is larger than the number of encoded audio frames:

```
RuntimeError: The size of tensor a (200) must match the size of tensor b (118) at non-singleton dimension 1
```

## Root cause

In `prepare_inputs_for_generation`, the `audio_tokens` buffer is pre-allocated with
`audio_window_size` frames. Each window the Mimi codec re-encodes the corresponding
audio slice and writes it back via `audio_tokens.copy_(new_audio_tokens)`. When the
last (or only) window has fewer frames than `audio_window_size`, the codec emits a
shorter tensor than the buffer, so the exact-size `copy_` fails.

## Fix

Copy only the overlapping frames so the buffer and codec output stay aligned:

```python
n = min(audio_tokens.shape[1], new_audio_tokens.shape[1])
audio_tokens[:, :n].copy_(new_audio_tokens[:, :n])
```

Single-shot generation is unchanged; repeated generation no longer crashes.

With the default `audio_window_size` (estimated from the audio length) the same
mismatch shows up as an off-by-one when the estimate exceeds the codec frame count
by one; the repro sets a large window to make the mismatch deterministic.

## Reproduction

```python
import numpy as np
import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor

model_id = "kyutai/stt-2.6b-en-trfs"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, dtype=torch.bfloat16).eval()
model.generation_config.audio_window_size = 200  # > encoded frames -> buffer/codec mismatch

audio = np.random.randn(24000 * 5).astype(np.float32) * 0.01  # ~118 frames
inputs = processor(audio=audio, return_tensors="pt").to(model.device)

out = model.generate(**inputs)
print(f"generated {out.shape[-1]} tokens")
```

Before the fix it aborts with the shape mismatch above; after the fix it prints the
token count.


<!-- ci-dashboard-recap:start -->

---



Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: kyutai_speech_to_text
Optimize Ernie 4.5 VL timestamp rendering with cached overlays
Author: raimbekovm
Pull request: yes

# What does this PR do?

Optimizes timestamp rendering in Ernie 4.5 VL video processing by caching text overlays and using torch alpha blending instead of slow `torch->PIL->torch` conversion for each frame.

**Performance improvement:**
| Mode | Time/frame | Speedup |
|------|-----------|---------|
| Original | ~2.2 ms | 1x |
| Cold cache | 1.5 ms | 1.5x |
| Warm cache | 0.17 ms | **~13x** |

For 180 frames (default max): ~400ms → ~30ms

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)?
- [ ] Did you make sure to update the documentation with your changes?
- [ ] Did you write any new necessary tests?

## Who can review?

@yonigozlan @molbap

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: ernie4_5_vl_moe
Fix mask loss to ignore padding areas in object detection
Author: raimbekovm
Pull request: yes

# What does this PR do?

Fixes the TODO in `loss_for_object_detection.py:196` - mask loss now properly ignores padding areas.

When batching images of different sizes, `nested_tensor_from_tensor_list` pads smaller images. Previously, padding areas were included in `dice_loss` and `sigmoid_focal_loss` computation. Now we use the `valid` mask to exclude them.

**Changes:**
- Add optional `valid_mask` parameter to `dice_loss` and `sigmoid_focal_loss`
- Use valid mask in `loss_masks` method (both `loss_for_object_detection.py` and `loss_rt_detr.py`)
- AMP compatible (preserves input dtype)
- Backward compatible (`valid_mask=None` by default)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes?
- [ ] Did you write any new necessary tests?

## Who can review?

@molbap @ArthurZucker

Comments:
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43466&sha=a453a5
Is transformers 5.12.1 requires tokenizers<=0.23.0,>=0.22.0 really right?
Author: jrp2014
Pull request: no

https://github.com/huggingface/tokenizers/releases/tag/v0.23.1 says that:

tokenizers 0.23.1 is the first proper stable release in the 0.23 line — 0.23.0 only ever shipped as rc0 because the release pipeline itself was broken (Node side hadn't shipped multi-platform binaries since 2023, Python side was on pyo3 0.27 without free-threaded support). 0.23.1 is the version where everything actually goes out the door together: full Node multi-platform wheels for the first time in years, Python 3.14 (regular and free-threaded 3.14t), full type hints for every Python class, and a stack of measurable perf wins on the BPE / added-vocab hot paths.

There is no functional 0.23.0 published — we tag 0.23.1 directly so users don't accidentally pull a never-shipped version.

So we should have an upper bound of <0.23 or <0.24?

Comments:
Hi, I'd like to work on this issue.

I'll first verify the current dependency constraint and check whether `tokenizers==0.23.1` is compatible with the current test suite. If everything looks good, I'll prepare a PR with the appropriate version range update.

Thanks!


cc @itazap 

The GitHub dependabot wants to upgrade my application to allow 0.23.1.  My other bot identifies a couple of cases where things change in 0.23.1, but I am not in a position to judge whether they matter.

Might want to wait for https://github.com/huggingface/transformers/issues/45736 tokenizers to be updated for this!
Remove some unnecessary generate warnings
Author: Cyrilvallez
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=46955)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46955)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

As per the title. I'm getting very angry with those warnings that are emitted all the time when generating with a single sequence, even though it is completely harmless. I believe this floods the user with unnecessary warnings.
Technically, even with a batch size of 1 we could have `pad_token` inside the sequence, but this is highly improbable (the input is very likely corrupted in this case, as it's useless and could be unpadded), so I don't believe it's worth to keep the warning for those cases. I reckon currently we flood user with annoying warnings way more than this counter-argument

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46955)
**Latest run:** [28361149263:1](https://github.com/huggingface/transformers/actions/runs/28361149263)
**Result:** `success` | **Jobs:** 13 | **Tests:** 46,629 | **Failures:** 0 | **Duration:** 14h 43m

<!-- ci-dashboard-recap:end -->

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46955). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Regression in v5.x: `ProcessorMixin._load_tokenizer_from_pretrained` forces subfolder for non-primary sub-tokenizers, breaking repos that put tokenizer files at root
Author: HanfengLiao
Pull request: no

## Summary

`ProcessorMixin._load_tokenizer_from_pretrained` in v5.x unconditionally appends the
sub-processor attribute name as a subfolder when loading a non-primary tokenizer:

```python
# src/transformers/processing_utils.py, ~L1457-1467
is_primary = sub_processor_type == "tokenizer"
if is_primary:
    tokenizer = AutoTokenizer.from_pretrained(path, subfolder=subfolder, **kwargs)
else:
    tokenizer_subfolder = os.path.join(subfolder, sub_processor_type) if subfolder else sub_processor_type
    tokenizer = AutoTokenizer.from_pretrained(path, subfolder=tokenizer_subfolder, **kwargs)
```

This breaks loading of processors whose tokenizer files live at the repo root but whose
sub-processor attribute is named anything other than `tokenizer` — e.g.
`UniversalActionProcessor` from [`physical-intelligence/fast`](https://huggingface.co/physical-intelligence/fast),
which uses `bpe_tokenizer` as the attribute name.

Under transformers v4.x this case worked because the loader could fall back to root.

## Repro

```python
# transformers==5.2.0
from transformers import AutoProcessor
proc = AutoProcessor.from_pretrained("physical-intelligence/fast", trust_remote_code=True)
```

Fails with `ValueError: Couldn't instantiate the backend tokenizer ...`.
`AutoTokenizer.from_pretrained("physical-intelligence/fast")` (no subfolder arg) works,
confirming the issue is specifically the forced subfolder.

## Suggested fix

When the subfolder lookup returns no files (or all `.no_exist`), fall back to loading from
root with a deprecation warning. This restores v4.x behavior while preserving the v5 intent
of supporting multiple sub-tokenizers in subfolders.

The cleanest fix is probably to try root first and only escalate to subfolder if the
attribute-named subdirectory exists in the repo file listing.

Comments:
cc @arthurzucker @itazap but not sure how common this is

Duplicate of https://github.com/huggingface/transformers/issues/44987, and also see my reply in https://github.com/huggingface/transformers/pull/45627#discussion_r3147562059

TL;DR; imo can be dropped as remote code supported only with v4. Also, Pi0 is not supported in transformers, and we will add the FAST version soon after testing the Pi0 impl with lerobot team. We will also upload a converted processing file to the repos under `lerobot` org on the hub

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix CohereASR training loss double shift
Author: w-Jessamine
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=46947)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46947)
<!-- ci-dashboard-badge:end -->

## Summary

CohereASR right-shifts `labels` into `decoder_input_ids`, then used `self.loss_function` (`ForCausalLMLoss`) to compute the training loss. That loss shifts labels internally again, so training targets were effectively `labels[..., 1:]`.

This switches CohereASR to a plain `CrossEntropyLoss`, matching the Moonshine fix and seq2seq models that already align decoder inputs from labels before computing loss.

Fixes #46894.

## Tests

- `PYTHONPATH=src /tmp/transformers-pr-venv/bin/python -m pytest tests/models/cohere_asr/test_modeling_cohere_asr.py::CohereAsrModelTest::test_training_loss_no_double_shift -q`

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46947)
**Latest run:** [28335138925](https://github.com/huggingface/transformers/actions/runs/28335138925)
**Result:** `success` | Grafana metrics are not available yet.

<!-- ci-dashboard-recap:end -->

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: cohere_asr
Fix Florence2 training-loss double-shift (same pattern as Moonshine #…
Author: sharmax-vikas
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=46898)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46898)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

Fixes #46897

Fix double-shift training-loss bug in `Florence2ForConditionalGeneration` - the exact same pattern fixed in Moonshine (#46784) and CohereASR (#46894).

`Florence2ForConditionalGeneration.forward()` shifts labels right via `shift_tokens_right()` to create `decoder_input_ids`, then calls `self.loss_function()` which maps to `ForCausalLMLoss`. That function shifts labels again internally (`labels[..., 1:]`), so the model trains against `labels[..., 1:]` instead of `labels`.

The fix replaces `self.loss_function()` with plain `CrossEntropyLoss()`, matching the approach used in Whisper, Bart, and the now-fixed Moonshine.

Both `modular_florence2.py` (source of truth) and `modeling_florence2.py` (auto-generated) are updated.

## Who can review?

@zucchini-nlp @yonigozlan (multimodal/vision-language models)

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46898)
**Latest run:** [28365382183:2](https://github.com/huggingface/transformers/actions/runs/28365382183)
**Result:** `success` | **Jobs:** 2 | **Tests:** 19 | **Failures:** 0 | **Duration:** 2m 2s

<!-- ci-dashboard-recap:end -->

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46898). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: florence2