noamaanMulla-03/transformers-issues huggingface dataset:
fix: Add 'image-text-to-text' to `TASK_MAPPING`
Author: saattrupdan
Pull request: yes

# What does this PR do?

The `TASK_MAPPING` dictionary, mapping pipeline tags to the list of architectures that support that task, is missing the new [image-text-to-text pipeline tag](https://huggingface.co/models?pipeline_tag=image-text-to-text). This PR simply adds that missing one.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker @amyeroberts @qubvel 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

This is in `modelcard.py` so cc @stevhliu as well

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37107). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@stevhliu @qubvel Thanks both! It doesn't seem like the failing test has anything to do with my change though?

Yeah it seems unrelated, we can merge for you once a fix has been released on `main` :)

> Yeah it seems unrelated, we can merge for you once a fix has been released on `main` :)

It seems like the issue has been fixed on main now, but I can't re-trigger the tests - are you able to?

cc @ydshieh to merge 🤗 

Thank you for the pin. I just merge some fix, let's update the branch (I am doing it now) to see if my patch work well or not 🙏 

well, happy it works (at least for some runs so far)
No more dtype_byte_size()
Author: Rocketknight1
Pull request: yes

Our code used a `dtype_byte_size()` function, but this was unnecessary and also incorrect. In particular, it listed the byte size for `torch.bool` as `1/8`, which is not actually correct; `bool` is implemented internally in Torch/Numpy/TF/Flax as an 8-bit value.

This PR just removes the function entirely and uses methods like `element_size()` instead, which are simpler and more correct.

Fixes #37074

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37144). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
A TypeError in modeling_utils.caching_allocator_warmup function
Author: ZeroMakesAll
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.50.2
- Platform: Linux-5.15.0-1040-nvidia-x86_64-with-glibc2.35
- Python version: 3.12.9
- Huggingface_hub version: 0.29.3
- Safetensors version: 0.5.3
- Accelerate version: 1.5.2
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <yes>
- Using GPU in script?: <yes>
- GPU type: NVIDIA H800

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

1. Save bool values in model params
2. Load model use <device_map="auto">
3. An error occurred in modeling_utils.caching_allocator_warmup (line 5854), because one bool value takes 1/8 byte and then the type of byte_count is float

### Expected behavior

Before allocating video memory, do a type check on the byte_count

Comments:
Hi @ZeroMakesAll, I'm not sure I understand. `torch.bool` actually uses 8 bits per entry, so that entries are byte-aligned.

```python
>>> x = torch.ones((32768, 32768), dtype=torch.bool, device="cuda")
>>> x.untyped_storage().nbytes()
1073741824  # 32768 * 32768, 1 byte per entry
```

@Rocketknight1 Thanks, the information you provided was very helpful to me. However, I found that transformers define the size of bool here (`modeling_utils.dtype_byte_size`)
```
if dtype == torch.bool:
    return 1 / 8
```
![Image](https://github.com/user-attachments/assets/eadf5b39-f173-4b8d-af23-c7c0e4f7b939)

Seems that huggingface use this function to estimate memory allocation. It return a float and cause the TypeError in `modeling_utils` `line 5854`.
```
_ = torch.empty(byte_count // 2, dtype=torch.float16, device=device, requires_grad=False)
```
Here, `byte_count` can't be a float

Hi @ZeroMakesAll, thanks for that! This is definitely a bug in `dtype_byte_size`. I'll make a PR to fix it.

Fix open at #37144
Optionality of `attention_mask` argument in Attention classes/functions.
Author: Godofnothing
Pull request: no

In the current stable version of `transformers` `attention_mask` argument is annotated as `Optional[torch.Tensor]` (see for example [modeling_llama.py](https://github.com/huggingface/transformers/blob/d6b3c7486b441296366f788fde57109337f63bca/src/transformers/models/llama/modeling_llama.py#L269)). 

However, in fact it is a **required** argument. 

At the same time, the ancestor `LlamaDecoderLayer`  classes accepts this argument as Optional ([see](https://github.com/huggingface/transformers/blob/d6b3c7486b441296366f788fde57109337f63bca/src/transformers/models/llama/modeling_llama.py#L315)). 

Delving deeper, [flash_attention_forward](https://github.com/huggingface/transformers/blob/d6b3c7486b441296366f788fde57109337f63bca/src/transformers/integrations/flash_attention.py#L12) annotates `attention_mask`  as `Optional[torch.Tensor]` and calls inside [ _flash_attention_forward](https://github.com/huggingface/transformers/blob/d6b3c7486b441296366f788fde57109337f63bca/src/transformers/modeling_flash_attention_utils.py#L230) which takes `attention_mask` as required `torch.Tensor` argument, but there is conditional statement checking whether the `attention_mask`  is not `None` and the function can be called in fact with `attention_mask` as None.

I suggest correcting the typing by making `attention_mask` an optional argument with `None` as its default value.


Comments:
Hi! @Godofnothing, I just submiited a PR to fix this issue. Feel free to give me suggestions if there's any problem:)
[Tests] add `min_new_tokens` to prevent flaky length checks
Author: gante
Pull request: yes

# What does this PR do?

Sets `min_new_tokens` to the same value as `max_new_tokens` in a few checks to prevent failures in length-related checks. These checks expect that we generate exactly `max_new_tokens` tokens, and we might stop earlier due to stopping criteria that's often on by defaulta (e.g. due to eos token)

E.g. 
<img width="1377" alt="Screenshot 2025-04-01 at 16 11 47" src="https://github.com/user-attachments/assets/7c1877b5-668f-41f6-bcb3-93101c58094d" />

this one is no longer flaky :) (`py.test tests/models/pix2struct/test_modeling_pix2struct.py::Pix2StructModelTest::test_greedy_generate_dict_outputs_use_cache --flake-finder --flake-runs 1000`, previously had a ~3% failure rate)

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37175). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ok i see you have a screenshot :-)
Stop DOSing the Hub in the CI
Author: Rocketknight1
Pull request: yes

Several of our test suites use `setUp()` methods, but these are called **once per test**. This repeatedly loads the same files from the Hub when the test suite has lots of small tests, and this increases the risk of a single connection failure giving us a red CI, and also seems very likely to trigger DOS protection from the Hub. I'm experimenting with refactoring some tests so that Hub operations are only called once for the whole test class

Comments:
Making this change reduced the runtime for the tests on my local machine from 83s to 24s :sweat_smile: 

cc @ydshieh I ran slow tests locally and they passed - I think we can merge this now just to stop `fuyu` errors making the CI red, and I might work on a bigger PR in future to replace lots of `setUp()` methods with `setUpClass()`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37209). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Bug in Paligemma usage docs for v4.50.3
Author: EricCousineau-TRI
Pull request: no

The following doc version has a minor bug in its usage:
https://github.com/huggingface/transformers/blob/v4.50.3/docs/source/en/model_doc/paligemma.md#single-image-inference
At the time of writing, this is the default version of the docs people come across via
https://huggingface.co/docs/transformers/en/model_doc/paligemma

This has bug where it's using the tokenized input length:
```py
print(processor.decode(output[0], skip_special_tokens=True)[inputs.input_ids.shape[1]: ])
```
but it should actually be the text input length:
```py
print(processor.decode(output[0], skip_special_tokens=True)[len(prompt): ])
```

Found this out by cross-referencing the HF spaces example:
https://huggingface.co/spaces/big-vision/paligemma-hf/blob/d914d44/app.py#L38

Note sure if it's worth patching the existing docs, having a new minor release, or just closing this out as a note for others.

Comments:
@EricCousineau-TRI feel free to open a PR. We won't do patch, but it will included be in the next release

PR'd against 4.50 release branch: https://github.com/huggingface/transformers/pull/37212

The Paligemma docs changed a bit in #36469, hence targeting release branch.

If the patch isn't useful, I can close it out. Thanks!


Ah oke. We are doing release tomorrow anyway, so no big difference :)

Cool, I'll go ahead and close this issue + the PR. Thanks!
Fix small bug in PaliGemma demo docs
Author: EricCousineau-TRI
Pull request: yes

# What does this PR do?

Fixes bug in Paligemma usage example

Fixes #37181

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

\cc @zucchini-nlp

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

Closing per convo in #37181
Updated the model card for CLIP
Author: purusharthmalik
Pull request: yes

# What does this PR do?

As suggested in this issue - https://github.com/huggingface/transformers/issues/36979#issue-2947704577 - this PR updates the documentation of the CLIP model, which will now be aligned with the standardized format for all the docs.

## Who can review?

@stevhliu, please let me know if any changes are needed.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37040). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@stevhliu Are there any changes required?

All the suggested changes have been incorporated into the docs. Also, are model-specific loaders and processors being deprecated or are we just omitting them from the initial description about the model for standardization's sake?

@stevhliu Done! Let's merge this, I'll get started on another model.
Update falcon model card
Author: ricalanis
Pull request: yes

Fixes #36979 

* Updated the Falcon model card
* Did not update Falcon3 as it is not explicitly listed, but can do it also
* Did not include the attention visualizer as I was not able to implement it for Falcon models in a straightforward way.
* First contrib, thank you for your patience <3

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [] Did you write any new necessary tests? NA


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Thank you for your kind suggestions, @stevhliu. Added all of them, so ready for you to review again.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37184). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37184). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks to this build i was able to spot a typo. Fixed.
Updated model card for Qwen2 
Author: Aravind-11
Pull request: yes

# What does this PR do?

- updated model card for Qwen2. 

#36979

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? https://github.com/huggingface/transformers/issues/36979

## Who can review?
@stevhliu 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37192). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix more inefficient PT operations
Author: cyyever
Pull request: yes

Remove ``clone().detach()`` and unnecessary ``cpu()`` calls.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Hi @cyyever, is `detach()` even necessary before `tolist()`? Just calling `tolist()` on a tensor, even one with grad enabled and on the CUDA device, seems to work for me.

@Rocketknight1 I forget. If you can work on your host, they are not needed.

@gante Done

@Rocketknight1 Can you help check this one?

The changes in this fail  quite a lot of test, see [this](https://huggingface.co/datasets/hf-internal-testing/transformers_daily_ci/raw/37a90cd406e891e58865aaa4d5d192a0eef01207/2025-04-01/ci_results_run_models_gpu/new_model_failures_with_bad_commit_grouped_by_authors.json). You can search "pr_number": 37060,` to find the failing tests related to this PR.

The job run page is at https://github.com/huggingface/transformers/actions/runs/14186033478

@cyyever @Rocketknight1 I am considering to revert this PR and give @cyyever some time to address the failing tests.



That's surprising - these looked like very safe changes, and the CI was green! Feel free to revert the PR if needed.

well, daily CI is on GPU and have a more complete set of tests ...

Can you reopen it?

You mean reopen the merged PR? It seems impossible for a merged one.

BTW, let me check something first before you dive iinto it.

our bad: it's not from this PR. The report has some issue. Sorry to bother you @cyyever 

Never mind, we all hope the library becomes better.
Fix static cache export
Author: guangy10
Pull request: yes

# What does this PR do?

Fixed a test regression due to old arg deprecation.

```
RUN_SLOW=1 pytest tests/utils/test_cache_utils.py -k test_static_cache_exportability
```


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @qubvel

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.
[Phi4] add multimodal chat template
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, tested on converted Phi4.

Btw, @yonigozlan, I wanted to add processor tests but apparently the model's image processor doesn't accept `do_rescale`. Seems like a bug, given rescaling always should happen to end up with pixels in `(0, 1)`, no? 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36996). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@yonigozlan btw, for the chat template I decided to always pass-in nested list of images/videos. Since some models cannot work with flat input format (Mllama, Idefics), the safest way is to batch properly and let the processor flatten if needed

I remember you had a tracker somewhere for that. I don't think we can support both input formats for all processors, so let's make "batched" as always-supported, while "flat" is available for some models only

@zucchini-nlp yes this should be fixed with this PR https://github.com/huggingface/transformers/pull/36976
目前使用Ktransformers进行DEEPSEEK-R1满血版和4bit量化版模型进行推理，推理速度有多少tokens/s？对应的计算资源配置分别是多少？
Author: William-Cai123
Pull request: no

目前使用Ktransformers进行DEEPSEEK-R1满血版和4bit量化版模型进行推理，推理速度有多少tokens/s？对应的计算资源配置分别是多少？
目前本地部署测试能跑4bit量化版和Q2_K量化版，但推理速度只有不到0.1tokens/s，，(...o0^0o...)，使用的配置如下：
GPU：tesla A10 24G X 2
CPU：Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz X 100（--cpu_infer 100，支持AVX-512，不支持AMX）
MemTotal：256G
磁盘：12T （15000rpm gpt-1.00 partitioned partitioned:gpt，hdd）


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.