noamaanMulla-03/transformers-issues huggingface dataset:
[`Auto`] Make processor subclasses overridable on load time
Author: vasqu
Pull request: yes

As per title, discovered in #39585 (went through a few versions until we found this :p)

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42912). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: vit,clip,detr,table_transformer,got_ocr2,whisper,wav2vec2,qwen2_audio,speech_t5,csm,gemma3,qwen2_5_vl,llava,smolvlm,gemma3n,qwen2_5_omni

This comment contains `run-slow`, running the specified jobs: 

models: ["models/clip", "models/csm", "models/detr", "models/gemma3", "models/gemma3n", "models/got_ocr2", "models/llava", "models/qwen2_5_omni", "models/qwen2_5_vl", "models/qwen2_audio", "models/smolvlm", "models/table_transformer", "models/vit", "models/wav2vec2", "models/whisper"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/20300023387)

✅ No failing test specific to this PR 🎉 !
local_files_only=True in pipeline() does not prevent model downloads in transformers==4.57.1
Author: nmahavishnu
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.57.1
- Platform: macOS-14.5-arm64-arm-64bit-Mach-O
- Python version: 3.13.2
- Huggingface_hub version: 0.35.3
- Safetensors version: 0.6.2
- Accelerate version: 1.5.2
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.9.0 (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Clear all local Hugging Face cache for the model using the official CLI:

```
huggingface-cli delete-cache
Follow the interactive prompts to remove all versions and revisions of the model you will test (e.g., bert-base-uncased).
```

Then, run:

```
python
from transformers import pipeline

a = pipeline("text-classification", model="bert-base-uncased", local_files_only=True)
```
Expected:
An OSError (or similar) should be raised saying that the files are missing.

Actual:
Files are downloaded from the Hugging Face Hub anyway, and pipeline loads successfully.


### Expected behavior

Expected:
An OSError (or similar) should be raised saying that the files are missing.

Actual:
Files are downloaded from the Hugging Face Hub anyway, and pipeline loads successfully.


Comments:
Hi @nmahavishnu, I don't think the `local_files_only` argument is directly passed through from pipeline to `from_pretrained()`. Try putting it in `model_kwargs`, or loading the model and tokenizer separately and then passing those objects to `pipeline()` instead of just strings.

I have submitted a fix for this in #42318.

The issue was that `local_files_only` was extracted from `kwargs` but missing from the `hub_kwargs` dictionary used during model loading. The PR ensures it is correctly propagated to `from_pretrained`.
Fix cuda index
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR fixes this issue https://github.com/huggingface/transformers/issues/42851. As we moved to multi-threaded loading by default, if the users passes `device_map="cuda"`, it will always load the model on cuda:0 as in threaded mode all the default devices reset to 0. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42897). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove tied weight keys Sam2Video
Author: yonigozlan
Pull request: yes

Not necessary anymore, and was crashing the model.

Fixes https://github.com/huggingface/transformers/issues/42837

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42840). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: edgetam_video, sam2_video, sam3_tracker_video

Could you approve @Cyrilvallez ?
Regression in SAM2 caused by changes in mark_tied_weights_as_initialized
Author: krzwaraksa
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 5.0.0.dev0 (commit hash: 37426b27bf51fb4cc9f7f7645a61d1b5dd28f20f)
- Platform: Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.11.0rc1
- Huggingface_hub version: 1.2.2
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.8.0+cu128 (CUDA)
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA GeForce RTX 4080

### Who can help?

@Cyrilvallez 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Script to reproduce:
```
from transformers import (
    Sam2VideoModel,
)

model = Sam2VideoModel.from_pretrained(
    "facebook/sam2.1-hiera-large",
    revision="665f8e2ad61cf5f53d65644ff27c8ee525124610",
)
```

Exception:
```
  File "/home/krwaraksa/code/SpatialAI2/repro_sam2.py", line 5, in <module>
    model = Sam2VideoModel.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/krwaraksa/code/SpatialAI2/sam3_venv/lib/python3.11/site-packages/transformers/modeling_utils.py", line 248, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/krwaraksa/code/SpatialAI2/sam3_venv/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3988, in from_pretrained
    model, missing_keys, unexpected_keys, mismatched_keys, offload_index, error_msgs = cls._load_pretrained_model(
                                                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/krwaraksa/code/SpatialAI2/sam3_venv/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4148, in _load_pretrained_model
    model.mark_tied_weights_as_initialized()
  File "/home/krwaraksa/code/SpatialAI2/sam3_venv/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4476, in mark_tied_weights_as_initialized
    param = self.get_parameter(tied_param)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/krwaraksa/code/SpatialAI2/sam3_venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 847, in get_parameter
    raise AttributeError("`" + param_name + "` is not an nn.Parameter")
AttributeError: `positional_embedding` is not an nn.Parameter
```

This is likely caused by this change: https://github.com/huggingface/transformers/pull/42270, in `mark_tied_weights_as_initialized()`. Before `AttributeError` was silently ignored, now it's getting thrown.

### Expected behavior

Model gets loaded. It works in transformers 4.57.3

Comments:
cc @molbap @yonigozlan @nielsrogge as well

Hello @krzwaraksa ! Thanks for raising the issue, it should be fixed here https://github.com/huggingface/transformers/pull/42840
Add fast image processor for  ViViT 
Author: msminhas93
Pull request: yes

## What does this PR do?

This PR implements a fast PyTorch image processor for ViViT (`VivitImageProcessorFast`) that extends the existing `BaseImageProcessorFast` class. The new processor provides optimized image processing capabilities specifically designed for video understanding tasks using the ViViT model.

## Key Features

- **Fast PyTorch-native implementation**: Leverages PyTorch operations for efficient video frame processing
- **Video structure handling**: Supports nested video inputs (batch of videos, each video is a list of frames)
- **Custom rescaling with offset**: Implements ViViT's unique rescaling behavior with optional offset parameter for [-1, 1] range
- **Channel dimension handling**: Automatic detection and conversion between different channel formats (CHW vs HWC)
- **Optimized preprocessing pipeline**: Sequential application of resize, center crop, rescale, and normalize operations
- **Batch processing support**: Efficient handling of multiple video inputs with proper tensor stacking
- **AutoImageProcessor integration**: Properly registered in the auto mapping system for seamless usage

## Before submitting

- [x] This PR implements a new fast image processor for ViViT
- [x] Did you read the contributor guideline, Pull Request section?
- [x] Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case. [Contributions Welcome] Add Fast Image Processors #36978
- [x] Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
- [x] Did you write any new necessary tests? Updated test suite to run against both slow and fast processors with 23/23 tests passing

## Who can review?

@yonigozlan

Comments:
Hi @yonigozlan! Could you please take a look. Thank you.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, vivit

Thank you for the comments! I've updated my PR @yonigozlan 

@yonigozlan If you could please look at the PR that'll be awesome

Thank you!
fix Dtensor and tensor mismatch for Col/RowRep
Author: 3outeille
Pull request: yes

same issue as https://github.com/huggingface/transformers/pull/42906

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42924). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[kernels] Add user_agent to track kernels metrics
Author: MekkCyber
Pull request: yes

# What does this PR do?

Refactors the call to `get_kernel` by adding a wrapper, to allow passing `user_agent` metrics in the future `kernels` release

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41689). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: mra, rwkv, sam3_video, yoso, mxfp4
Fix dtype quantizer
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR fixes code related to quantizer since `dtype=auto` default created some minor issues in our CI. We clean a bit of code related to dtype and fixes the tests as some of ours tests depends on the dtype of the model, which before where always on `torch.float32` if not specified. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42882). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: diffllama, falcon, falcon_mamba, gpt_neo, gptj, kyutai_speech_to_text, mimi, moshi, nemotron, fbgemm_fp8, finegrained_fp8, gptq, higgs
[docs] WeightConverter
Author: stevhliu
Pull request: yes

Adds docs for `WeightConverter` from #41580 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42636). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sound good! You might want to intro the weight converter with some docs about why it's needed and how dynamic weight loading works

I detailed it in the release notes: https://github.com/huggingface/transformers/releases/tag/v5.0.0rc0

> You might want to intro the weight converter with some docs about why it's needed and how dynamic weight loading works

reworked the intro to start with motivation and then brief explanation of how it works!

Nice, thanks for adding more concrete examples! I think that'll make it easier to understand.

Also big thanks for iterating with me on this, super cool learning about our new loading system ❤️ 

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42636&sha=626ca5
[docs] Expert parallelism
Author: stevhliu
Pull request: yes

updates the parallelism docs:

- distributed inference is now tensor parallelism so it is easier to find
- new expert parallelism guide

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42409). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

added a warning that `DistributedConfig` is experimental and may change!
[docs] optimization cleanup
Author: stevhliu
Pull request: yes

cleans up the Optimization section a bit to remove redundant content

- removes the hardware specific sections (GPU/CPU) since many of the same optimizations in these docs already exist
- give Optimum it's own section in the docs for more visibility and combines GPU/CPU usage for it
- give assisted decoding it's own section in the docs for more visibility as well and it's also not really a decoding strategy like the other search/sampling methods are
- also removes the [Optimizing inference](https://huggingface.co/docs/transformers/llm_optims) doc since it also has redundant content in it and will be replaced by the newer overview in #42538 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42827). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Improve BatchFeature (.to() works on lists/nested lists of tensors, auto-skip non array conversion)
Author: yonigozlan
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/42205

Also fixes processors handling videos crashing when return_metadata=True and return_tensors="pt" .

Cc @zucchini-nlp , for now I have hardcoded all the skip conversions. I think it's safer to do this for now instead of silently not converting any inputs to BatchFeature for which tensor conversion fails. The main issue is with arrays, where non ragged arrays would be converted to tensors, and ragged arrays silently wouldn't, which could cause some unexpected behavior.
Open to alternative solutions so happy to hear your thoughts!

Edit: Added auto-skip, except for ragged arrays

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: glm46v, glm4v, llava_next_video, perception_lm, qwen2_5_omni, qwen2_5_vl, qwen2_vl, qwen3_omni_moe, qwen3_vl, video_llama_3

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42884). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks, lgtm! The chat template tests seems to be failing and skipping mask conversion to tensor, can you check?

Yep thanks for catching that! The issue was that I was skipping converting empty lists/nested lists when we shouldn't
Support `.to(device)` or Device Aware Handling for Segmentation Labels in `EOMTImageProcessor`
Author: ariG23498
Pull request: no
Labels: name: Feature request

### Feature request

Enable device-aware handling of *segmentation label tensors* (specifically `class_labels` and `mask_labels`) returned by `EOMTImageProcessor`.

Currently, the processor outputs *lists of tensors*, which cannot be moved to a device using `.to(device)`, leading to device mismatches during training.

### Motivation

I’ve been fine-tuning *EOMT* on the `segments/sidewalk-semantic` dataset using the Transformers image processor.
Here is [the code](https://gist.github.com/ariG23498/f4eb5ce6fe57f474225f01741e035c24) reproducing the issue.


### Your contribution

Happy to help brainstorm and work on this if need be.

CC: @merveenoyan @NielsRogge @molbap 

Comments:
This is not limited to EOMTImageProcessor, can also be done for MaskFormer/Mask2Former/OneFormer.
Document new default shard size + dropped unsafe serialization
Author: Wauplin
Pull request: yes

Follow-up PR after https://github.com/huggingface/transformers/pull/42734 from @Cyrilvallez  (related to https://github.com/huggingface/transformers/issues/42555 and https://github.com/huggingface/transformers/issues/42556).

Let's document the changes introduced in #42734 in the `MIGRATION_GUIDE_V5.md`.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42904). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Quantization] rm _pre_quantization_dtype from quantization tests
Author: MekkCyber
Pull request: yes

# What does this PR do?

Removes tests related to `_pre_quantization_dtype` since it was removed in this pr : https://github.com/huggingface/transformers/pull/42882

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bnb, gptq, quark_integration

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42939). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update `param_element_size`
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR fixes how we calculate the param size for quantized models. This should be simpler to hack around !

I added some tests to check that it works for the following methods `bnb`, `finegrained_fp8`, `torchao`, `mxfp4`, `quanto`. 

cc @Cyrilvallez as you were concerned at some point. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42818). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Will have a last look next Monday!

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bnb, finegrained_fp8, mxfp4, quanto_integration, torchao_integration
Stop ContinuousBatchingManager.request_id_iter after terminal outputs
Author: antznette1
Pull request: yes

## Summary 
ContinuousBatchingManager.request_id_iter() could keep polling after a request had already reached a terminal state. This PR makes the iterator stop once it yields a terminal GenerationOutput (`FINISHED` or `FAILED`), and also exit when the generation thread is no longer alive and there are no more results to drain.

## Changes
- Stop iterating after emitting `RequestStatus.FINISHED` or `RequestStatus.FAILED` for the request.
- Add a unit test that verifies request_id_iter() terminates after a terminal output, even if the manager thread is still alive.

Comments:
> I feel like the requeue logic in get_result() is a bug waiting to happen.
> 
> I feel like something like this would make more sense, right?
> 
> ```python
> def request_id_iter(self, request_id: str):
>     while True:
>         if not self.is_running() and self.output_queue.empty():
>             return
>         result = self.get_result(request_id=request_id, timeout=0.1)
>         if result is not None:
>             yield result
>             if result.status in {RequestStatus.FINISHED, RequestStatus.FAILED}:
>                 return
> ```

Thanks for the suggestion — I agree the current “requeue” behavior in `get_result()` is a bit fragile (it can cause churn / unfairness if the queue has multiple request_ids and a consumer keeps asking for one specific id).
My change was intentionally scoped to make `request_id_iter()` terminate correctly once a terminal `GenerationOutput` is observed, without touching the broader queue semantics. But your proposed loop makes sense and is cleaner than my current `thread_alive/timeout=0` handling.
I’m happy to update `request_id_iter()` to follow the pattern you suggested:
```
def request_id_iter(self, request_id: str):
    while True:
        if not self.is_running() and self.output_queue.empty():
            return
        result = self.get_result(request_id=request_id, timeout=0.1)
        if result is not None:
            yield result
            if result.status in {RequestStatus.FINISHED, RequestStatus.FAILED}:
                return
```

Separately (or in a follow-up PR if you prefer), I can also take a look at improving/removing the requeue logic in `get_result()` itself (e.g. a per-request buffer or a different retrieval API), since that’s likely the more fundamental fix.

Let me know if you’d like me to adjust `request_id_iter()` in this PR and keep any `get_result()` redesign for a follow-up.

This modifies behaviour in a breaking way, we haven't documented but the iterator is mostly used in scenarios where you never want to stop inference. If you really want to kill the iterator for now you'll have to call `ContinuousBatchingManager.stop` in a separate thread, which will stop iteration.
Your current implementation, IIUC, exits at the first finished request, which completely defeats the purpose of using continuous batching.