noamaanMulla-03/transformers-issues huggingface dataset:
About Siglip2 feature selection
Author: MonolithFoundation
Pull request: no
Labels: name: Vision

### Feature request

Hi, glad to see Siglip2 support.

Wanna consult, since Siglip2 dynamic input (max_num_patches) have padding, does the output need to be selected?

For example, if we have max_num_patches=1024, but there is some padding due to image ratio keeping, the final output 1024 tokens will have some tokens is padding? How to remove them?

### Motivation

For downstream tasks such as img understanding.

### Your contribution

Would like contribute a PR if it's needed

Comments:
Hey @MonolithFoundation, thanks for the question. You can use `pixel_attention_mask` produced by `processor`, which indicates padded patches.

The mask has shape (batch_size, max_num_patches) with 0s for padded patches (tokens)

Hi, would like consult a little further:

I want using siglip2 as a MLLM vision backbone, I would like using 1024 as max_patches, but I got some issues:

- how could I shrink down the tokens feed into LLM? I tried used something like VLPatchMerger like in Qwen2VL, it learned something but not learned very well;
- If set maxpatches into a fixed number, such as 1024, everysingle image will actually being resized into that target size, how could this able to be dynamic? (therotically it's same as resize into a 512x512 size)



Hey @MonolithFoundation, Siglip2 vision position embeddings are resized to match the image size in "patches", also the image can be non-square and with any aspect ratio, e.g. 1024x256 or 256x1024 and that aspect ratio would be preserved.

https://github.com/huggingface/transformers/blob/41925e42135257361b7f02aa20e3bbdab3f7b923/src/transformers/models/siglip2/modeling_siglip2.py#L171-L227

Please use https://discuss.huggingface.co/ for the questions, we are trying ot keep issues only for bug reports and feature requests.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Skip FP8 linear tests
Author: MekkCyber
Pull request: yes

# What does this PR do?
Skips FP8 linear tests because FP8 is only supported on gpus with compute cabalitie >=9.0 (they pass locally)



Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37008). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Optimize tokenizer.decode() Performance for `List[int]` Inputs
Author: n0gu-furiosa
Pull request: no
Labels: name: Feature request

### Feature request

When calling `tokenizer.decode()` with a `List[int]` as `token_ids`, the method appears to be significantly slower than necessary due to redundant `to_py_obj` conversions.

### Motivation

Example:

```python
import time
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained('hf-internal-testing/llama-tokenizer')
token_ids = [869] * 2000

start = time.time()
for _ in range(1000):
    tok.decode(token_ids)
print(time.time() - start)
```

The trace results for the above code show that most of the time is spent on repeated `to_py_obj` calls, rather than in the actual `_decode` function:

![Image](https://github.com/user-attachments/assets/efe74538-3b3d-4565-8cf3-4081356d6ae2)

In this case, since the input is already a `List[int]`, passing it through `to_py_obj` seems redundant. By adding a conditional check to bypass this line for `List[int]` inputs:

https://github.com/huggingface/transformers/blob/6a2627918d84f25422b931507a8fb9146106ca20/src/transformers/tokenization_utils_base.py#L3868

…the example code improves by nearly 10x in my environment (from ~7s to ~0.7s).


### Your contribution

I wasn’t sure where the best place to apply this optimization would be—either within `decode()` or inside `to_py_obj()`—so I haven’t opened a PR yet. I’d be happy to contribute a fix if there’s guidance on where such a change would be most appropriate.

Comments:
This is most likely caused by this section in `to_py_obj()`:

```python
if isinstance(obj, (dict, UserDict)):
    return {k: to_py_obj(v) for k, v in obj.items()}
elif isinstance(obj, (list, tuple)):
    return [to_py_obj(o) for o in obj]
```

When a flat list is passed in, `to_py_obj()` will be called on each element of the list, which means 2000 function calls are required in your test. If you can figure out an optimization in that function that retains correct output without that speed penalty, we'd definitely welcome a PR!

Thank you for your guidance @Rocketknight1! I've opened a PR for this.
[chat templates} support loading audio from video
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

If user indicates `load_audio_from_video`, we will extract audio part of the video and use it as input audio. Most any-to-text models are used this way, according to their demo inference scripts

Needed for Qwen-Omni release

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers.

@Rocketknight1 can you take a look, it can be supported with minimal changes. Unfortunately the tests won't run, I see Phi4 merged but seems like it has not chat template. I will ask Cyril and run tests on Phi4 if it has templates

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36955). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Oke, tested that the template is applied correctly and the inputs are passed to the model with a dummy qwen2-vl processor. Working and ready for review!
Fixup for distill_any_depth conversion script
Author: qubvel
Pull request: yes

# What does this PR do?

It looks like ruff was updated before the last run of tests for https://github.com/huggingface/transformers/pull/36614

Comments:
>it would be nice to do it before merge 🙏

Sure 👍 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37043). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
omlab/omdet-turbo-swin-tiny-hf from_pretrained fails to build model
Author: Jordan-Pierce
Pull request: no
Labels: name: bug

### System Info

Windows 11
python-3.10
tokenizers-0.21.1 
transformers-4.50.1



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

>>> from transformers import AutoProcessor, OmDetTurboForObjectDetection
>>> OmDetTurboForObjectDetection.from_pretrained("omlab/omdet-turbo-swin-tiny-hf")

### Expected behavior

For it not to throw this error:

```bash
  File "C:\Users\user\Miniconda3\envs\coralnet10\lib\site-packages\timm\models\swin_transformer.py", line 51, in window_partition
    x = x.view(B, H // window_size, window_size, W // window_size, window_size, C)
RuntimeError: shape '[1, 22, 7, 22, 7, 1]' is invalid for input of size 25600
```

Comments:
Hi @Jordan-Pierce, this code works for me! Try updating `timm` with `pip install --upgrade timm`

Thanks @Rocketknight1 

I was using `timm>=0.9.2`, but I guess between that and their `timm<=1.0.11` it doesn't work.  Much appreciated.
[audio utils] fix fft_bin_width computation
Author: eustlb
Pull request: yes

# What does this PR do?

When computing triangular mel filter matrices and when we triangularize in mel space, if we have e.g. `num_frequency_bins` being 257, meaning real-valued fft as been computed on 512 points (`257 = n_fft // 2 + 1`), then the `fft_bin_width` should be `(sampling rate / 2) /  number_of_bins` with `number_of_bins =  num_frequency_bins - 1`. 

This was very likely introduced and not seen by tests for the reason that `mel_filter_bank` was called with, following the above example, `num_frequency_bins = 256` when using `triangularize_in_mel_space=True` then padded with 0s to retrieve the correct expected shape (257), **which is a bad practice and misleading for the user as we do not respect the method API!** 

I also updated the expected outputs with the one I got from running torchaudio kaldi implementation, confirming our implem was incorrect.

# TODO

- [x] update impacted tests: this will very likely break CI tests relying on this incorrect implementation.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36603). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[generate, cache] handle more complex device maps
Author: gante
Pull request: yes

# What does this PR do?

Related to #36942 

We need to allocate static caches on the correct device. In `generate`, we peek at the model's `hf_device_map` to find the correct device for each layer, so that the cache stays on the same device as the layer it is caching.

Previously, given that decoder-only LLMs were the norm, we were assuming that device mappings ending with `(...).some_int` contained the device map for the layer with index `some_int`. This is not necessarily correct: in multimodal models, we can have this pattern for other components of the model, like the vision tower. We need to know the device of the decoder, and not of other parts of the model.

This PR expands the device detection to first look for the decoder module name, and then look for the layer pattern. It relies on `model.get_decoder()` -- models without this method will use the previous device mapping strategy.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37014). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> LGTM, not sure if there's any other way layers can be represented in device-map, anyway this definitely covers most cases

@zucchini-nlp same, but let's keep an eye in case there are new patterns 👁️ 

> This means we'll need to make sure get_decoder() is available for all multimodal model (for some it is not yet available)

Yes! On the other hand, it should work for any decoder modality 🙌 It will start breaking when we allow more than a decoder per model, but that's a problem for the future.
Fix typing for None valued variables
Author: cyyever
Pull request: yes

Add Optional for more None valued variables.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.
Avoid unnecessary device operations in loss computing
Author: cyyever
Pull request: yes

Avoid unnecessary device operations for labels. Types for some parameters are added.

 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers.

@Rocketknight1 One more fix, but no more.
fixed typo
Author: finnoh
Pull request: yes


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37036). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the fix!
Add Distill Any Depth
Author: keetrap
Pull request: yes
Labels: name: New model

name: Vision

This PR add support for Distill Any Depth
Fixes #36499 

- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)?
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Did you make sure to update the documentation with your changes?
- [x] Did you write any new necessary tests?

cc: @qubvel


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

cc @qubvel!

@qubvel Thanks for the review.
So, should I close this PR or should I update the conversion script of DepthAnything.

Let's update the conversion script, but make sure we do not break the DepthAnything conversion. If it becomes too messy, we can add it as a separate file in the DepthAnything folder.

@qubvel  Added conversion script.
Please review the PR 

@qubvel Script updated.

@qubvel  Gentle reminder.


Thanks for the review. I have uploaded the models you can check [small](https://huggingface.co/keetrap/Distill-Any-Depth-Small-hf) and [large](https://huggingface.co/keetrap/Distill-Any-Depth-Large-hf) models.

Thank you! I've opened a PR to add tags to make model more discoverable (please copy them to another checkpoint as well)

https://huggingface.co/keetrap/Distill-Any-Depth-Small-hf/discussions/2

Also, can you please create more descriptive model cards? Here is an example of such a model card
https://huggingface.co/apple/DepthPro-hf


P.S. no need to make it so descriptive as for DepthPro, but some image + abstract/description + code snippet would be super nice

> Thank you! I've opened a PR to add tags to make model more discoverable (please copy them to another checkpoint as well)
> 
> https://huggingface.co/keetrap/Distill-Any-Depth-Small-hf/discussions/2

I think it should be added in original repo https://huggingface.co/xingyang1/Distill-Any-Depth-Small-hf and https://huggingface.co/xingyang1/Distill-Any-Depth-Large-hf

Oh, there are originally converted models as well! Nice. Would you be so kind as to open PRs for those repos?

PRs opened for model cards.

Thanks a lot, I've sent an email to the authors to make it merge asap, also cc @shuiyued

P.S. model cards PRs were merged 🚀
fix: Fully remove legacy cache from Llama
Author: Wheest
Pull request: yes

The legacy cache format (a list of tensors for each block) has essentially been deprecated in the LLama model: if you try and use it, the model [will fail here:](https://github.com/huggingface/transformers/blob/bc1c90a755505d576220ed2a161ca42dbf3cafaf/src/transformers/models/llama/modeling_llama.py#L650-L654)

```python
        past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
        using_static_cache = isinstance(past_key_values, StaticCache)
```

because `get_seq_length()` is a method on `Cache`.

However, the file still makes reference to the legacy cache format.

This PR fixes it, and raises an error if users attempt to use it.

The pertinent changes I made are in the first commit, the other ones are related to the `check_copies.py` models ([see CI](https://app.circleci.com/pipelines/github/huggingface/transformers/122891/workflows/08d3be73-38e8-43a5-b7d0-b21dee9b55f0/jobs/1660348/parallel-runs/0/steps/0-105)), and were generated by running:
```sh
python utils/check_copies.py --fix_and_overwrite
python3 utils/check_modular_conversion.py --fix_and_overwrite
```

Suggested reviewers: @gante @ArthurZucker

Comments:
cc @gante for cache!

Many thanks for the review @gante, the diff is now down from +884 −376 to +208 −286!

Regarding the comma at the end of the argument lists, I'm surprised that isn't a rule which `ruff` would be opinionated about, and change for me?


Thanks for the catch on the commas, didn't expect to have to revert parts of those files manually.

A scan of the diff, and it seems to be sorted.  Cheers for the patience!

And thank you for iterating with me and making the library better! 🤗
Remove deprecated batch_size parameter
Author: cyyever
Pull request: yes

It's declared to be removed in 4.49.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

cc @gante
Set weights_only in torch.load
Author: cyyever
Pull request: yes

Apply TorchFix to enhance torch.load

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

And PT 2.6.0 defaults weights_only to True.
refactor(audio_processing): replace pipe with temp files for FFmpeg processing
Author: joeyhacker
Pull request: yes

Key reasons for this change:
1. MP4 container format requires random access to metadata (moov atoms)
   which is often located at the end of the file. Pipe streaming makes
   this access pattern impossible.
2. FFmpeg's format detection works more reliably with physical files
   compared to streamed input via pipes.
3. Temporary files provide better error diagnostics since the input can
   be preserved for debugging when failures occur.
4. Some FFmpeg codecs and filters behave differently with pipe input
   versus file input due to buffering differences.

The new implementation:
- Creates properly named temporary files with correct extensions
- Uses atomic write operations with flush()
- Implements comprehensive cleanup in finally blocks
- Provides better error messages when failures occur

This fixes issues with partial file errors ("offset 0x3f9: partial file")
that occurred during demuxing of m4a files in the pipe-based approach.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

test failed
Reverse dependency map shouldn't be created when test_all is set
Author: LysandreJik
Pull request: yes


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.
Skip inexistent tokenizer in tests
Author: cyyever
Pull request: yes

Skip inexistent tokenisers.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Hi @cyyever Could you provide an CI job run page or a pytest command that shows the issue?

@ydshieh For example, https://app.circleci.com/jobs/github/huggingface/transformers/1667424

Hi again. Those failures are due to some Hub connection issues, which is not resolved by our team yet. The changes in this PR should not be necessary (by the way of how pipeline tests are written + how those tiny models are created, but that is a bit hidden logic).

At this moment, I would suggest close this PR instead. I have reported the Hub issue to the internal team but I haven't heard updates from them.



I see
Use `lru_cache` for tokenization tests
Author: ydshieh
Pull request: yes

# What does this PR do?

- use `setUpClass` instead of `setUp`
- use `lru_cache` for `get_tokenizer` and `get_rust_tokenizer` whenever possible
  - some tests will change/modify the states of a tokenizer, and for those tests, we want to avoid using cache. 
  - so far, it (almost) always uses cached tokenizer(s) but returns `copy.deepcopy` of them, to avoid being modified and affect other tests
  - (clip fast tokenizer has strange behavior and not working well with `copy.deepcopy`, so not using the copied version)

### Results:

With this PR, the `pytest` command (not counting other CircleCI job steps) run time could be reduced by around **25%**, for a fixed list of test files. (comparing different CircleCI runs is not good measure, due to the different sets of tests on runners across runs).
  - `25%` if running in a single process. When I use `-n 4`, it reduces to `20%`, and `-n 8` even lower: like `15%`.

Overall, it reduces **the total run time (so reduce the cost) a bit**, but not really the time we have to wait for the whole tokenization job to finish.

**It also helps if someone runs the tests locally.**
 
We can try to extend this for image processing / processor jobs.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36818). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Going to merge as the failing tests are irrelevant.
[blip-2] Fix dtype mismatch when keep in fp32 
Author: zucchini-nlp
Pull request: yes
Labels: name: for patch

# What does this PR do?

Seems that BLIP2 `keep_in_fp32_modules` was not picking up correctly until [36722](https://github.com/huggingface/transformers/pull/36722) (thus no errors up to today). Now that the `query_tokens` are actually kept in 32, we are getting dtype mismatch error in inference. (reported by @hmellor)

This PR fixes by casting inputs to correct dtypes, and keeping `qformer` in `fp32` as well. The fix was tested with slow BLIP-2 tests and vLLM tests


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Test is already added, slow tests were failing on `main` from the above linked PR. The reason is that we don't usually run slow tests much 🥲 

Let's merge then!