noamaanMulla-03/transformers-issues huggingface dataset:
Possible to move HybridCache from GPU to CPU?
Author: tianhaoz95
Pull request: no
Labels: name: Feature request

name: Cache

### Feature request

I'm working on PD disaggregation where I'm trying to pass KV cache produced in GPU to CPU before sending them to decode process, looks like the following implementation doesn't have a way to move once created, is that the case? If so, is it possible to add an API to do that?

https://github.com/huggingface/transformers/blob/0d6a60fe55fe051a1a68f2026d19223ed57b3c75/src/transformers/cache_utils.py#L1612

### Motivation

The motivation is that in different stages of the generation, we might want to utilize different device.

### Your contribution

I'm happen to open a PR to add if confirmed missing and we would like something to do this.

Comments:
cc @gante 

@tianhaoz95 you can move the tensors to the CPU, i.e.

```py
past_key_values = HybridCache(...)
n_layers = len(past_key_values.key_cache)
for layer_idx in range(n_layers):
    past_key_values.key_cache[layer_idx] = past_key_values.key_cache[layer_idx].to("cpu")
    past_key_values.value_cache[layer_idx] = past_key_values.value_cache[layer_idx].to("cpu")
```

(I am not inclined to add a method to do this since a) it's easy to work around; b) it's the first time I'm seeing this request; and c) more methods = more problems :D )

@gante oh got it. Yes I agree. I think this is clean enough and I don't think a API is necessary. Thanks!
Update Model Card for ModernBERT
Author: ParagEkbote
Pull request: yes

# What does this PR do?

As described in the issue, this PR updates the model card for ModernBERT. Please let me know if any modifications are required and I will make the necessary changes.

Fixes #36979

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?
@stevhliu 


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37052). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Could you please review?

cc: @stevhliu
Add Optional to types
Author: cyyever
Pull request: yes

# What does this PR do?

Add Optional to more types so that we can enable more RUFF rules on type checking.


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

It's green.

Merged!
Set non_blocking=True When moving data from the CPU to the GPU
Author: Hukongtao
Pull request: no
Labels: name: bug

### System Info

No Need

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I used the transformer's Trainer to train the model, but used my own Dataloader.
Then I used pytorch profile to check my training performance and found that the CPU execution time accounted for a high proportion

![Image](https://github.com/user-attachments/assets/8769d1c9-3224-4850-9a38-7e8d50d5b621)


After a period of investigation, it was found that the non_blocking was not set when the data was transferred from the CPU to the GPU.
https://github.com/huggingface/transformers/blob/v4.49.0/src/transformers/trainer.py#L3625-L3631
The modified code is:
```python
kwargs = {"device": self.args.device，“non_blocking”:  True}
```
Then I re-profiled my code and the results were as follows：

![Image](https://github.com/user-attachments/assets/11735e62-797f-42c6-894c-a2b63acd7617)

You can see that the performance has been greatly improved.

I'm not sure if this is a bug in the code or a problem with the way I'm using it.

But there is no doubt that setting non_blocking=True has brought a great performance improvement to my training.


Looking forward to your reply

### Expected behavior

No Need

Comments:
If this is a bug, do you have plans to fix it?

cc @muellerzr @SunMarc 

I submitted a https://github.com/huggingface/transformers/pull/36408, can you check it for me? @muellerzr @SunMarc 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Error From BitsandBytes
Author: nikithakriz
Pull request: no
Labels: name: bug

### System Info

Hi,
I tried to load model using bitsandbytesconfig, but am getting this error

```
.venv\Lib\site-packages\transformers\integrations\bitsandbytes.py:499, in _validate_bnb_multi_backend_availability(raise_exception)
    494     if find_spec("intel_extension_for_pytorch"):
    495         logger.warning(
    496             "You have Intel IPEX installed but if you're intending to use it for CPU, it might not have the right version. Be sure to double check that your PyTorch and IPEX installs are compatible."
    497         )
--> 499     available_devices.discard("cpu")  # Only Intel CPU is supported by BNB at the moment
    501 if not available_devices.intersection(bnb_supported_devices):
    502     if raise_exception:

AttributeError: 'frozenset' object has no attribute 'discard'




### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

transformers
Version: 4.49.0
bitsandbytes
Version: 0.44.1.dev0+9315692

### Expected behavior

Loading model with bitsandbytes config without any error

Comments:
Hi!
From my understanding, this could be due to an incompatibility between the installed versions of Intel IPEX and PyTorch. Would it make sense to first check the installed versions of both libraries to ensure compatibility before making any modifications?
You can find a compatibility table [here ](https://huggingface.co/docs/accelerate/en/usage_guides/ipex) 
Hope it helps!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
:Cannot infer concrete type of torch.nn.Module
Author: Jereshea
Pull request: no

### Environment Info:

- Python Version : 3.12.2
- PyTorch Version : 2.5.1+cpu
- Transformers Version : 4.49.0
- Model_name : "meta-llama/Llama-3.1-8B-Instruct"

I am trying to run inference for the model "meta-llama/Llama-3.1-8B-Instruct" using torch.jit.trace(). Below is my code snippet.

```
    input_ids = tokenizer(prompt, return_tensors="pt").input_ids
    trace_model = torch.jit.trace(model, input_ids, check_trace=False, strict=False)
    traced_model = torch.jit.freeze(trace_model)
    output = traced_model.generate(input_ids)
```
However, I am encountering errors when I try to pass the traced_model to model.generate(). Is there any possibility to overcome this error
```
RuntimeError: Tracer cannot infer type of (tensor([[[-3.1250,  1.3750,  7.2500,  ...,  1.7422,  1.7422,  1.7422],
         [ 5.8438,  7.0625,  5.8438,  ..., -4.0938, -4.0938, -4.0938],
         [ 1.1875,  0.6328,  1.5156,  ..., -8.6250, -8.6250, -8.6250],
         ...,
         [ 5.1250,  1.3047, -1.6094,  ..., -2.7500, -2.7500, -2.7500],
         [ 5.5625,  5.2188,  0.9844,  ..., -3.6094, -3.6094, -3.6094],
         [ 1.0312, -0.7852, -1.1406,  ..., -3.6562, -3.6562, -3.6562]]],
       dtype=torch.bfloat16, grad_fn=<UnsafeViewBackward0>), DynamicCache())
:Cannot infer concrete type of torch.nn.Module
```

But when I try in different version of transformers(transformers=4.43.2), I am getting different error as below

`AttributeError: 'RecursiveScriptModule' object has no attribute 'generate'`

Is there a fix for this issue in any version of Transformers, or an alternative approach to enable `generate` with a `traced model`?

Comments:
I don't think `generate()` is intended to work automatically with `torch.jit.trace()`, is it? cc @gante - do we have a recommended way to trace/export generation loops?

Is any alternate solution recommended?
 
In the meantime, I performed inference using the IPEX backend. Below is the snippet I used:
 
```
    model = ipex.llm.optimize(model, dtype=amp_dtype, inplace=True, **deployment_mode=True**)
    output = model.generate(inputs)
```
 
When deployment_mode=True is set, it internally uses torch.jit.trace ([Reference](https://github.com/intel/intel-extension-for-pytorch/blob/f20ebd03be629f56a9027538a2ee6741e4b4be18/intel_extension_for_pytorch/transformers/optimize.py#L1729)). To use the traced model for generate, some modifications have been made in IPEX ([Reference](https://github.com/intel/intel-extension-for-pytorch/blob/f20ebd03be629f56a9027538a2ee6741e4b4be18/intel_extension_for_pytorch/transformers/optimize.py#L85)). 

Do we have any similar fix or approach like this in native PyTorch?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
The arguments in `utils/modular_model_converter.py` is different from those in docs
Author: zhoubay
Pull request: no

The arguments in `utils/modular_model_converter.py` is [here](https://github.com/huggingface/transformers/blob/92c5ca9dd70de3ade2af2eb835c96215cc50e815/utils/modular_model_converter.py#L1707-L1712)

```python
parser.add_argument(
    "--files_to_parse",
    default=["all"],
    nargs="+",
    help="A list of `modular_xxxx` files that should be converted to single model file",
)
```

However, the docs is using `--files-to-parse` as following.

https://github.com/huggingface/transformers/blob/92c5ca9dd70de3ade2af2eb835c96215cc50e815/docs/source/en/modular_transformers.md?plain=1#L51-L53

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix `utils/check_bad_commit.py`
Author: ydshieh
Pull request: yes

# What does this PR do?

See the comment in the change.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37271). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix `utils/check_bad_commit.py`
Author: ydshieh
Pull request: yes

# What does this PR do?

See the comment in the change.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37272). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix deprecated PT functions
Author: cyyever
Pull request: yes

These functions are deprecated in PT 2.0

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

@gante Reverted
pin specific `natten` version in docker file 
Author: ydshieh
Pull request: yes

# What does this PR do?

We suddenly get ``cannot import name 'natten2dav' from 'natten.functional'``, so `dinat` test get `Error` at the beginning of pytest collection, not giving any output, and the slack report job fails due to missing files and we don't receive any report.

I will try to avoid such failure in a separate PR, but let's just quickly pin `natten`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37274). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
clarify error message to ensure min 28x28 image supplied
Author: rymc
Pull request: yes
Labels: name: Processing

This PR clarifies that qwen 2.5 VL requires images of at least 28x28. The current error message has an OR rather than AND which can confuse users.


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37264). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Tests] flaky `test_constrained_beam_search_generate_dict_output` 
Author: gante
Pull request: yes

# What does this PR do?

Adds `@is_flaky()` to `test_constrained_beam_search_generate_dict_output`. Flake reason in the diff :)

<img width="1049" alt="Screenshot 2025-04-04 at 12 28 29" src="https://github.com/user-attachments/assets/5d729363-bc68-4712-bafe-2b39cc585752" />


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37276). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Hugging Face Hub pin to v0.30.0 for Xet
Author: LysandreJik
Pull request: yes

Pins `huggingface_hub` to > 0.30.0 so that Xet downloads are faster

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37166). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Bug when computing positional IDs from embeddings
Author: SabrinaRichter
Pull request: no
Labels: name: bug

### System Info

https://github.com/huggingface/transformers/blob/c0f8d055ce7a218e041e20a06946bf0baa8a7d6a/src/transformers/models/esm/modeling_esm.py#L243

I think it should be `1, sequence_length + 1, dtype=torch.long, device=inputs_embeds.device`
I don't see how the index of my padding token in the alphabet has anything to do with my positions in the sequence

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

-

### Expected behavior

-

Comments:
Hi @SabrinaRichter, I actually had exactly the same thought as you when I first saw this! It seems obviously incorrect, but if you search the codebase, this line actually originates in the code for RoBERTa, and lots of people (including the ESM authors) have just been copying it ever since.

The reason, I think, was that RoBERTa (the original FAIR implementation) stored the special token embeddings and the position embeddings in the same tensor, which meant that the position embeddings started at the index after the final special token (which was the padding token). Therefore, the position ID for position x was x + self.padding_idx + 1. Of course, this is no longer the case for most models. but the offset persists because so many existing models have been trained with it.

Because of those pretrained checkpoints, we can't really "fix" this! We could consider a PR to replace `self.padding_idx` with a constant for some models, but the problem is that this would then break any models that used that code to train with a different `self.padding_idx`. 🙃 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[RoPE] abstract dynamic RoPE update under a decorator ✨ 
Author: gante
Pull request: yes

# What does this PR do?

✨ Part of the goals to streamline modeling files ✨ 

This PR abstracts the dynamic RoPE updates, "dynamic" and "longrope", under a decorator. Same capabilities for power users, simpler modeling file.

Review suggestion:
1. `modeling_rope_utils.py`
2. `llama`
3. `phi3` (longrope)
4. other models (which are either a direct copy or apply the same pattern)

Relevant tests: `py.test tests/models/ -k rope` (correct behavior of "dynamic" and "longrope" is tested with this command)

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37249). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix llava_onevision tests
Author: Rocketknight1
Pull request: yes

The `llava_onevision` tests are loading and saving the processor/tokenizer for every test, which creates a lot of slowdown and hammers the Hub very quickly. This creates the potential for intermittent test failures. This PR refactors the tests to save the processor only once, instead of once per test.

This reduces the llava_onevision processor test runtime on my local machine from 95s to 45s.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37280). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
:rotating_light::rotating_light::rotating_light: Replace SetUp with SetUpClass
Author: Rocketknight1
Pull request: yes

A lot of our test suites use `SetUp` and `TearDown` methods. These are called **before and after every test**. In a lot of cases, this is unnecessary because the contents do not change between tests. This is particularly important when the setup has expensive steps, like initializing models or processors, or loading test files/images from the Hub.

I made a PR at #37209 to change this for Fuyu, which was frequently timing out or being throttled because it kept loading the same image by URL from the Hub. This PR reduced the runtime of the Fuyu test suite on my local machine to 30% of the previous amount, so I figured I'd try it for other classes too.

This is a repo-wide search replace to turn setUp into setUpClass. It will break **everything**, but I'm hoping that with some tweaks we can make the CI pass.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37214). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

🙏 

Closing because the automated approach breaks way too many tests - will try a more limited version of this PR!
Update model card for Cohere
Author: bimal-gajera
Pull request: yes

# What does this PR do?
This PR updates the model card for Cohere and follows the template outlined in the issue. Please let me know if any changes are required.

As part of this update, I also replaced the outdated blog link  [`https://txt.cohere.com/command-r/`](https://txt.cohere.com/command-r/) with the updated official blog link  [`https://cohere.com/blog/command-r`](https://cohere.com/blog/command-r).

This PR addresses part of issue #36979 by updating the `cohere.md` model card.

## Before submitting
- [x] This PR improves the docs.

## Who can review?
@stevhliu 



Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Note: CI failures are caused by issues unrelated to this PR:

I ran the relevant checks locally:
- `make quality` fails due to a docstring mismatch in `NougatTokenizerFast`, which is not touched in this PR.
- `make repo-consistency` fails due to missing optional dependencies in `image_processing_utils`.


I've made all the suggested changes and resolved the other comments. Let me know if there's anything else you'd like me to update. Thanks again for the feedback!


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37056). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update model card for Depth Anything
Author: shubham0204
Pull request: yes

# What does this PR do?

This PR updates the model-card for the `depth_anything` model, as described in #36979, in an attempt to standardize all model-cards.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@stevhliu



Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

@stevhliu I have made all suggested changes and amended the last commit.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37065). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@stevhliu all suggested changes have been made in the latest commit 👍🏻