noamaanMulla-03/transformers-issues huggingface dataset:
[docs] Fix grammar in BLIP visual question answering example
Author: Charuhasini30
Pull request: yes

This PR fixes a small grammar issue in the BLIP documentation.

Changes:
- Replaced "how to visual question answering" with "how to perform visual question answering"

No functional changes.

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46813)
fix another flaky test
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

as per title, has been annoying lately

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46767). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46767)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: kosmos2_5, paligemma
[Windows] RTX 5070 Ti (Blackwell sm_120) - setup and deployment notes
Author: loongmiaow-pixel
Pull request: no

### Environment
- GPU: NVIDIA GeForce RTX 5070 Ti Laptop GPU (Blackwell, compute capability 12.0)
- Driver: 595.79 (CUDA 13.2)
- OS: Windows 11
- Python: 3.14
- transformers: [latest]

### Problem
transformers on RTX 5070 Ti requires workarounds:

- **TORCH_CUDA_ARCH_LIST=12.0** required for Blackwell
- Model loading may fail or fall back to CPU without explicit arch
- **CUDA_VISIBLE_DEVICES=-1** system env var blocks GPU detection

### Solution
```powershell
$env:TORCH_CUDA_ARCH_LIST = "12.0"
$env:CUDA_VISIBLE_DEVICES = "0"
$env:CUDA_MODULE_LOADING = "LAZY"
```

### Question
Would it be helpful to add RTX 50 series notes to the GPU documentation? I can contribute a PR with Windows troubleshooting.

Comments:
hmm, a compatibility issue? dunno whom to get for this one

@stevhliu , looks to be more docs related so imma leave it up to u, but feel free to get anyone else too


I don't think this is docs related! It's a surprising issue because I think most hardware that works with PyTorch should work with Transformers, and I don't think we've seen other people saying that 50XX cards don't work. Are you sure this isn't some weird local environment issue?

yeah, well, i didnt wanna sound like an alarmist of sorts so i was hoping it to be docs related and went in that line - see the last para in issue, am looking into the repro of it rn cus this better be a local issue 

tho there maybe breakage cus pytorch rolled out there updates just a few days ago? no, if that was the case would have come up earlier, if i get anything from those places then i will put it here

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Use cached added-token dicts in per-token decode loops
Author: ishan-1010
Pull request: yes

## What this fixes

Fixes #46396 (follow-up to #46323). Four slow tokenizers read the `added_tokens_encoder` / `added_tokens_decoder` properties inside the per-token loop of `convert_tokens_to_string`. Both properties rebuild and re-sort the full added-token map on every access, so decoding T tokens with N added tokens costs O(T * N * logN).

## The fix

Point those loops at the maintained `_added_tokens_encoder` / `_added_tokens_decoder` dicts instead, same as #46323 did for `convert_tokens_to_ids`. Membership and lookup don't depend on the property's sort order, so behavior is unchanged. Files: `byt5`, `myt5`, `dia`, `perceiver` tokenizers; 10 lines changed, attribute swaps only.

This is option (B) from #46396. The broader option (memoizing the property itself) is left for that issue since it has the cpmant-mutation and sort-order questions.

## Numbers

Decoding 2,840 mixed tokens with 4,000 added tokens (M1, CPU):

| tokenizer | before | after |
|---|---|---|
| ByT5 | 2,207 ms | 0.6 ms |
| Perceiver | 1,270 ms | 0.6 ms |

Output verified byte-identical (md5) before/after for both.

## Tests

```
python -m pytest tests/models/byt5/test_tokenization_byt5.py tests/models/myt5/test_tokenization_myt5.py tests/models/perceiver/test_tokenization_perceiver.py
# 78 passed, 107 skipped, 6 failed (same 6 fail on unmodified main; network/torch integration tests)
python -m pytest tests/models/dia/test_tokenization_dia.py
# 37 passed, 24 skipped
```

## Notes

Not a duplicate: no open PR touches these call sites (checked per the contribution policy). AI-assisted; I reviewed every line and ran the tests and benchmark above.


Comments:
cc @itazap

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46535). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: byt5, dia, myt5, perceiver
`added_tokens_encoder` rebuilds the entire added-token map on every access
Author: ishan-1010
Pull request: no


### Who can help?

@itazap

### Summary

Follow-up to #46323, which fixed the worst case: `_convert_token_to_id_with_added_voc` was reading the `added_tokens_encoder` property, which rebuilds and re-sorts the whole added-token map on every access. The property itself still rebuilds on every read, and a few call sites still go through it.

The property's docstring says the mapping is "cached ... in `self._added_tokens_encoder`", but it rebuilds and re-sorts on every call:

```python
return {k.content: v for v, k in sorted(self._added_tokens_decoder.items(), key=lambda item: item[0])}
```

That `_added_tokens_encoder` cache is maintained at every `_added_tokens_decoder` mutation site and is already what most of the file reads.

### Where it still bites

- ~47 call sites read the property, mostly `get_vocab()` (`vocab.update(self.added_tokens_encoder)`), called occasionally.
- Per-token: `byt5`, `myt5`, `dia`, and `perceiver` do `if token in self.added_tokens_encoder` inside `_convert_token_to_id`, so a rebuild happens per token.

### Two subtleties

1. The property returns a sorted map; the `_added_tokens_encoder` cache is insertion-ordered, so they aren't swappable 1:1 if sort order is part of the contract.
2. `cpmant` mutates the returned dict (`self.added_tokens_encoder.pop(...)`, `tokenization_cpmant.py:151`), which a shared cached dict would not tolerate.

### Possible directions

- **(A)** Memoize the sorted map, invalidated at the decoder-mutation sites (and handle the cpmant case). Fixes all call sites and matches the docstring.
- **(B)** Narrow: point the four per-token callers at the maintained `_added_tokens_encoder` cache, mirroring #46323.

Happy to open a PR for whichever direction you'd prefer.


Comments:
fix: resolve issue #39921
Author: KartavyaDikshit
Pull request: yes
Labels: name: Code agent slop

Resolves #39921

Improved error handling and aligned implementation details with best practices.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gemma3n

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46798)
fix: resolve issue #45698
Author: KartavyaDikshit
Pull request: yes
Labels: name: Code agent slop

Resolves #45698

Improved error handling and aligned implementation details with best practices.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46800)

> ⚠️ **Code quality check failed** — all test jobs were skipped. Fix the code quality issues and push again to run tests.
fix: resolve issue #44247
Author: KartavyaDikshit
Pull request: yes
Labels: name: Code agent slop

Resolves #44247

Improved error handling and aligned implementation details with best practices.

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46797)

> ⚠️ **Code quality check failed** — all test jobs were skipped. Fix the code quality issues and push again to run tests.
fix: resolve issue #17283
Author: KartavyaDikshit
Pull request: yes
Labels: name: Code agent slop

Resolves #17283

Fixed pipeline batch inference memory spike by properly releasing intermediate tensors during batch processing.

Signed-off-by: KartavyaDikshit <kartavyaniraj.dikshit2021@vitstudent.ac.in>

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46805)

> ⚠️ **Code quality check failed** — all test jobs were skipped. Fix the code quality issues and push again to run tests.
fix: correct phrase constraint state transition after fulfilled constraints
Author: KartavyaDikshit
Pull request: yes
Labels: name: Code agent slop

Resolves #19070

Phrase constraints (force_words_ids) were only appearing directly after input or at the end of generation. The issue was in how the constraint bank/state manages transitions after a phrase constraint is fulfilled. Added ForceWordsLogitsProcessor that replaces the deprecated constrained-beam-search approach with a local logits processor that properly handles state transitions.

Signed-off-by: KartavyaDikshit <kartavyaniraj.dikshit2021@vitstudent.ac.in>

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46815)

> ⚠️ **Code quality check failed** — all test jobs were skipped. Fix the code quality issues and push again to run tests.
fix: normalize VLM processor image input handling across models
Author: KartavyaDikshit
Pull request: yes
Labels: name: Code agent slop

Resolves #34545

Standardizes image input handling across VLM processors by adding URL loading, PIL Image support, and consistent nested/batched list handling. Added a `_normalize_images` utility in `image_utils.py` to reduce duplication.

## Changes

- **`image_utils.py`**: Added `_normalize_images()` utility that normalizes image inputs (URLs, paths, PIL, numpy, torch tensors, nested lists) into a consistent `list[list[...]]` format, loading URLs/paths via `load_image`.
- **`processing_paligemma.py`**: Fixed crash when passing URL strings or file paths by normalizing images before validation. Simplified the image structure logic since normalization guarantees a consistent format.
- **`processing_pixtral.py`**: Added explicit URL/image loading via `_normalize_images`. Removed unused `is_url`/`is_image_or_image_url` helper functions that were copied from Idefics2 but never used.
- **`processing_llava_next.py`**: Added explicit URL/image loading via `_normalize_images` for consistency with other processors.

## Processors not requiring changes

- **LlavaProcessor** / **Qwen2VLProcessor**: Use the base `ProcessorMixin.__call__` which already calls `fetch_images` via `prepare_inputs_layout`.
- **Idefics2Processor**: Already handles URLs via `load_image` in its custom `__call__`.
- **Idefics3Processor**: Already calls `self.image_processor.fetch_images(images)` in its `prepare_inputs_layout`.

Signed-off-by: KartavyaDikshit <kartavyaniraj.dikshit2021@vitstudent.ac.in>

Comments:
Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/27943320047) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: llava_next, paligemma, pixtral

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46814)

> ⚠️ **Code quality check failed** — all test jobs were skipped. Fix the code quality issues and push again to run tests.
fix: resolve issue #33165
Author: KartavyaDikshit
Pull request: yes
Labels: name: Code agent slop

Resolves #33165

Fixed model parallelism device map handling when tied weights span multiple devices.

Signed-off-by: KartavyaDikshit <kartavyaniraj.dikshit2021@vitstudent.ac.in>

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46806)
fix: resolve issue #35308
Author: KartavyaDikshit
Pull request: yes
Labels: name: Code agent slop

Resolves #35308

Fixed gradient checkpointing compatibility with FSDP on multi-GPU setups by properly handling activation recomputation wrapping.

Signed-off-by: KartavyaDikshit <kartavyaniraj.dikshit2021@vitstudent.ac.in>

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46804)
fix: distinguish eval OOM from train OOM in auto_find_batch_size
Author: KartavyaDikshit
Pull request: yes
Labels: name: Code agent slop

Resolves #32215

The auto_find_batch_size logic in Trainer previously only reduced the training batch size on OOM, even when the OOM occurred during evaluation. This fix catches OOM errors locally inside `_evaluate` and reduces `per_device_eval_batch_size` instead, preventing the error from propagating to `find_executable_batch_size` which would incorrectly reduce the training batch size.

Signed-off-by: KartavyaDikshit <kartavyaniraj.dikshit2021@vitstudent.ac.in>

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46809)
`apply_chat_template` crashes with `IndexError` when passed an empty list
Author: sharmax-vikas
Pull request: no
Labels: name: bug

### System Info

transformers: 5.13.0.dev0 (latest main, commit 2b422bde64)
Python: 3.14.4
OS: Ubuntu 25.04 (WSL2)

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
tokenizer.apply_chat_template([], tokenize=False)
```

Traceback:
```
File "tokenization_utils_base.py", line 3090, in apply_chat_template
    isinstance(conversation[0], (list, tuple)) or hasattr(conversation[0], "messages")
               ~~~~~~~~~~~~^^^
IndexError: list index out of range
```

Also reproduced on `google/gemma-2-2b-it` — same `IndexError`.

### Expected behavior

Should raise a clear `ValueError` (e.g., "conversation must not be empty") instead of an unhandled `IndexError`.

Root cause: In `tokenization_utils_base.py` line 3090, `conversation[0]` is accessed without checking if the list is empty:

```python
if isinstance(conversation, (list, tuple)) and (
    isinstance(conversation[0], (list, tuple)) or hasattr(conversation[0], "messages")
):
```

Fix: add `len(conversation) > 0` guard before accessing index 0.

Comments:
Raise clear ValueError for empty conversation in apply_chat_template
Author: punyamodi
Pull request: yes
Labels: name: Code agent slop

Fixes #46752

## What was wrong

Calling `apply_chat_template` with an empty list or tuple crashed with an unhandled `IndexError`:

```python
tokenizer.apply_chat_template([], tokenize=False)
# IndexError: list index out of range
```

This happens in `tokenization_utils_base.py` where `conversation[0]` is read to detect batched input, without first checking that the list is not empty.

## The fix

Add a small guard before that line. If the conversation is empty, we now raise a clear `ValueError` with a helpful message instead of crashing on an index access.

This matches how the rest of the method already reports bad input through `ValueError`.

## Checklist

- small, focused change (a few lines plus a test)
- added a test that checks both empty list and empty tuple raise `ValueError`
- normal single and batched conversations still work as before


Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46778)

blocked for raising an agent PR when one already existed, since that just spams maintainers!
fix(tokenization): guard against IndexError when apply_chat_template …
Author: Vitaliy-Pikalo
Pull request: yes
Labels: name: Code agent slop

When `apply_chat_template` is called with an empty list `[]`, line 3089 tries to access `conversation[0]` without first checking if the list is empty:

```python
if isinstance(conversation, (list, tuple)) and (
    isinstance(conversation[0], (list, tuple)) or hasattr(conversation[0], "messages")
):
```

This raises `IndexError: list index out of range`.

**Fix:** add `conversation and` before accessing `conversation[0]`, so Python's short-circuit evaluation prevents the index access on empty sequences.

**Reproduction:**
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
tokenizer.apply_chat_template([], tokenize=False)  # IndexError before fix, returns "" after
```

Fixes #46752.

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46789)
fix(tokenization): prevent IndexError in apply_chat_template on empty conversation list
Author: Hasnaathussain
Pull request: yes
Labels: name: Code agent slop

### What does this PR do?

Resolves #46752.

When calling `apply_chat_template` with an empty list (`[]`), the tokenization pipeline raises an `IndexError` because it attempts to check `isinstance(conversation[0], ...)` without validating if the list is populated.

This PR adds a `len(conversation) > 0` guard to prevent the crash in `tokenization_utils_base.py`, `processing_utils.py`, and `tokenization_mistral_common.py`. Also adds a regression test in `test_tokenization_utils.py`.

### Who can review?
@ArthurZucker @younesbelkada


Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46793)
fix: prevent IndexError in apply_chat_template with empty conversation list
Author: BB0813
Pull request: yes
Labels: name: Code agent slop

## Fix

Fixes #46752

### Problem

`apply_chat_template` crashes with `IndexError: list index out of range` when passed an empty list:

```python
tokenizer.apply_chat_template([], tokenize=False)
# IndexError: list index out of range
```

### Solution

Add a `len(conversation) > 0` guard before accessing `conversation[0]` in `tokenization_utils_base.py` line 3089.

### Changes

- `src/transformers/tokenization_utils_base.py`: Added length check before accessing `conversation[0]`

### Testing

- Empty list no longer raises `IndexError`
- Normal conversation lists continue to work correctly

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46794)

> ⚠️ **Code quality check failed** — all test jobs were skipped. Fix the code quality issues and push again to run tests.
Fix typo in MiniMax M3 VL documentation
Author: meghana243
Pull request: yes

# What does this PR do?

This PR fixes a typo in the MiniMax-M3-VL documentation.

The phrase:

"max-poolsthose per-key scores"

has been corrected to:

"max-pools the per-key scores"

This improves readability and clarity of the documentation.


Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46783)

Hey, because of the agent flood we're trying not to merge single typo PRs etc. right now, it's just too much maintenance burden! If you want to do a big typo sweep for a large group of files we might review/accept that
fix: resolve issue #43873
Author: KartavyaDikshit
Pull request: yes
Labels: name: Code agent slop

Resolves #43873

Improved error handling and aligned implementation details with best practices.

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46799)
fix: resolve issue #33985
Author: KartavyaDikshit
Pull request: yes
Labels: name: Code agent slop

Resolves #33985

Fixed tokenizer slow path regression where special tokens were incorrectly handled during batch encoding.

Signed-off-by: KartavyaDikshit <kartavyaniraj.dikshit2021@vitstudent.ac.in>

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46801)
[DiffusionGemma] Add DDIM and block refinement samplers
Author: kashif
Pull request: yes

Adds two more samplers for DiffusionGemma so people can try different denoising strategies without writing their own.

- `DiscreteDDIMSampler`: samples each canvas position from the exact posterior of the uniform corruption process. Works well with a constant temperature and around 20 steps.
- `BlockRefinementSampler`: commits an even share of the canvas per step, most confident tokens first, with optional editing of committed tokens.

The sampler is picked through `sampler_config`, same as the existing `EntropyBoundSampler`. Added unit tests for both, parametrized the generate test over them, and documented them in the model page.

Tests: `pytest tests/models/diffusion_gemma/test_generation_diffusion_gemma.py` and `test_modeling_diffusion_gemma.py -k test_generate` pass on CPU.

AI assistance was used.


Comments:
cc @gante  and @sayakpaul 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46595). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

running benchmarks as i type

It would be cool to measure both quality and tokens per forward! (I added tokens per forward as part of the output class in `generate`)

P.S.: In the first few commits I considered writing a base class for all samplers, since they will almost certainly share the random canvas initialization 🤗 

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46595)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: diffusion_gemma

closign in favour of having this on the diffusers side