noamaanMulla-03/transformers-issues huggingface dataset:
Inconsistent Tensor Type in Qwen2.5-VL
Author: DeadLining
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.49.0
- Platform: Linux-5.15.0-130-generic-x86_64-with-glibc2.35
- Python version: 3.10.16
- Huggingface_hub version: 0.28.1
- Safetensors version: 0.5.2
- Accelerate version: 1.3.0
- Accelerate config: 	not found
- DeepSpeed version: 0.15.4
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: DeepSeed Zero2
- Using GPU in script?: 3 GPUS
- GPU type: NVIDIA A800-SXM4-80GB

### Who can help?

@amyeroberts 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Running Qwen2.5-VL GRPO fine-tuning for the R1-V project.

### Expected behavior

Error file: transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py 168:169
Error: `q.float().dtype != cos.dtype`, due to the mixing of bf16 and fp32.
I am temporarily using the following settings：
`q_embed = apply_rotary_emb(q.float().type_as(cos), cos, sin).type_as(q)`
`k_embed = apply_rotary_emb(k.float().type_as(cos), cos, sin).type_as(k)`

How to deal with this error?

Comments:
Hey, this was solved in https://github.com/huggingface/transformers/pull/36188. Can you try by installing transformers from `main`?

> Hey, this was solved in [#36188](https://github.com/huggingface/transformers/pull/36188). Can you try by installing transformers from `main`?

Thanks a lot! ❤️
Siglip2 support
Author: MonolithFoundation
Pull request: no
Labels: name: Feature request

### Feature request

Consider support siglip 2?

### Motivation

Consider support siglip 2?

### Your contribution

Consider support siglip 2?

Comments:
Good timing, we plan to release the model soon. PR in https://github.com/huggingface/transformers/pull/36323 :)
[CI] Check test if the `GenerationTesterMixin` inheritance is correct 🐛 🔫 
Author: gante
Pull request: yes

# What does this PR do?

If a model inherits from `GenerationMixin`, then it can call `generate`. In those models, `GenerationTesterMixin` must be added to the tester in order to test `generate`. Sometimes we forget to do it. Not anymore 😈 

This PR follows up on #33212 and removes the last piece of common human error. It adds a meta-test to ensure we have the correct inheritance, which enables us to confirm that the right tests are being run automatically. In other words, if the model inherits from `GenerationMixin`, this test confirms that the tester inherits from `GenerationTesterMixin` (and the other way around too). This test also disincentivizes complex tester inheritance hierarchies, as in `idefics`, `bark`, or `musicgen`, which hurt long-term maintenance. 

In the process, it uncovered \*many\* cases of incorrect/missing inheritance, which are dealt with in this PR. Look at all these models that should have been tested with `generate` and haven't been since they were added 😭 

👉 reviewer: please start by reviewing `tests/test_modeling_common.py` and `modeling_utils.py`, then check the other files.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36180). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zucchini-nlp PR comments addressed/replied 🤗
Default value for mean_resizing in resize_token_embeddings should be False
Author: cyr0930
Pull request: no
Labels: name: bug

### System Info

transformers>=4.46

### Who can help?

@Arthur

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When running resize_token_embeddings with additional tokens

### Expected behavior

It is much slower to run resize_token_embeddings with mean_resizing=True.
Maybe it's because nowadays token embedding size became much larger than before (like gemma2, qwen2, ...).

So I think the default value for mean_resizing in resize_token_embeddings should be False now,
or implementation has to be fix to keep resizing speed as before.

Note: Maybe it happens if I'm using deepspeed stage3, but I didn't thoroughly investigate that.

Comments:
Hi @cyr0930, we think the default value of `True` gives better downstream performance, even if it's slower. If you find that `deepspeed` speed specifically is very bad, though, it might be possible to improve that!

Okay I'll check if it depends on deepspeed setting. Thanks for replying :)

hmm running with pure deepspeed is very slow,
while running with accelerate is fairly fast...
I don't know the reason yet.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @cyr0930, we've made some changes to the function that should speed things up. Let me know if you're still having this problem on `main`!

Do you mean this one? (https://github.com/huggingface/transformers/commit/ec7afad60909dd97d998c1f14681812d69a15728)
It is much faster after I installed 4.49.0 anyway. Thanks :)
[tests] make quanto tests device-agnostic
Author: faaany
Pull request: yes

## What does this PR do?
This PR makes tests under `test_quanto.py` device-agnostic. Except the skipped cases, all passed on XPU:

```bash
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoConfigTest::test_attributes
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoTestIntegration::test_conversion_with_modules_to_not_convert
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoTestIntegration::test_weight_and_activation_quantization_conversion
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoTestIntegration::test_weight_only_quantization_conversion
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationTest::test_compare_with_quanto
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationTest::test_generate_quality_cpu
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationTest::test_generate_quality_gpu
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationTest::test_quantized_model_layers
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationTest::test_serialization_bin
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationTest::test_serialization_safetensors
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationOffloadTest::test_check_offload_quantized
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationOffloadTest::test_generate_quality_gpu
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationOffloadTest::test_quantized_model_layers
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationQBitsTensorTest::test_compare_with_quanto
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationQBitsTensorTest::test_generate_quality_cpu
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationQBitsTensorTest::test_generate_quality_gpu
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationQBitsTensorTest::test_quantized_model_layers
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationQBitsTensorTest::test_serialization_bin
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationQBitsTensorTest::test_serialization_safetensors
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationQBitsTensorOffloadTest::test_check_offload_quantized
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationQBitsTensorOffloadTest::test_generate_quality_gpu
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationQBitsTensorOffloadTest::test_quantized_model_layers
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoQuantizationActivationTest::test_quantize_activation
PASSED tests/quantization/quanto_integration/test_quanto.py::QuantoKVCacheQuantizationTest::test_quantized_cache
```
cc @ydshieh 

Comments:
Uses Collection in transformers.image_transforms.normalize
Author: CalOmnie
Pull request: yes

# What does this PR do?

Followup to [this PR](https://github.com/huggingface/transformers/pull/35773#issuecomment-2671163144)
Which introduced a bug, preventing numpy arrays to be used as `std` and `mean` parameters.
This PR uses instead the Collection type, which correctly check for the prence of both `__len__` and `__getitem__` attributes but not `__reversed__`, succesfully accepting numpy arrays

This PR also updates one of the tests to pass in numpy arrays to confirm this new type does indeed function as intended.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@Rocketknight1 
@tomasruizt


Comments:
Hi @CalOmnie, yes, this seems like a good typing update! There's one issue, though: `typing.Collection` is marked as [a deprecated alias](https://docs.python.org/3/library/typing.html#typing.Collection) - can we use [collections.abc.Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection) instead?

Hey @Rocketknight1 fair enough, made the change.

Thanks for the quick fix @CalOmnie
fix: prevent model access error during Optuna hyperparameter tuning
Author: emapco
Pull request: yes

# What does this PR do?

The `transformers.integrations.integration_utils.run_hp_search_optuna` function releases model memory and sets `trainer.model` to None after each trial. This causes an `AttributeError` when subsequent `Trainer.train` calls attempt to access the model before reinitialization.

This fix resolved the error locally.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@muellerzr @SunMarc

Comments:
Add process_retry util
Author: ivarflakstad
Pull request: yes

Calling commands sometimes fail. This small util can be used to call those processes with slightly better odds of success.

Comments:
Not sure if it's better or not, I usually do

```
            try:
                subprocess.run(command, env=env, check=True, capture_output=True)
            except subprocess.CalledProcessError as e:
                ...
```

Just a nit, nothing big here.

> Not sure if it's better or not, I usually do
> 
> ```
>             try:
>                 subprocess.run(command, env=env, check=True, capture_output=True)
>             except subprocess.CalledProcessError as e:
>                 ...
> ```
> 
> Just a nit, nothing big here.

I agree with the approach, but I know that `subprocess.run` can at least raise `ValueError` (bad input), `TimeoutExpired` (slow process), and `FileNotFoundError` (can't find command to run) as well. Probably more as I assume I don't know them all off the cuff hehe

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35908). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This could still be useful in the future, but since I've removed the functionality that this PR was supposed to improve I'm closing it for now.
T5 model stopped working correctly in v4.49
Author: danijel3
Pull request: no

I was testing a piece of code that uses the model https://huggingface.co/sdadas/byt5-text-correction, which is based on T5ForConditionalGeneration and my old tests started generating garbage outputs. I managed to recreate the error to this simple example below. Basically, if we send a batch of sequences  one-by-one everything works fine, but if we send them  all at once, things start breaking. This worked fine in `v4.48.3`, but started breaking in `v4.49`.

```python
from transformers import ByT5Tokenizer, T5ForConditionalGeneration

model_path = "sdadas/byt5-text-correction"

tokenizer = ByT5Tokenizer.from_pretrained(model_path)
model = T5ForConditionalGeneration.from_pretrained(model_path).to("cuda")

texts = [
    "<pl> halo dzień dobry",
    "<pl> nazywam się magdalena czy z panem bartoszem rozmawiam",
    "<pl> bardzo mi miło ja mam do przekazania kilka informacji jeszcze wspomnę o rejestracji połączenia już panu wyjaśniam",
]


def process(texts):
    input_ids = tokenizer(texts, return_tensors="pt", padding=True).input_ids.to("cuda")
    shape_len = input_ids.shape[1] + 16
    logits = model.generate(input_ids, max_length=shape_len)
    return tokenizer.batch_decode(logits, skip_special_tokens=True)


print([process(x)[0] for x in texts])

print("--vs--")

print(process(texts))
```

In version `v4.48.3` the output is:
```
['Halo dzień dobry.', 'Nazywam się Magdalena, czy z Panem Bartoszem rozmawiam?', 'Bardzo mi miło ja mam do przekazania kilka informacji jeszcze wspomnę o rejestracji połączenia już panu wyjaśniam.']
--vs--
['Halo dzień dobry.', 'Nazywam się Magdalena, czy z Panem Bartoszem rozmawiam?', 'Bardzo mi miło ja mam do przekazania kilka informacji jeszcze wspomnę o rejestracji połączenia już panu wyjaśniam.']
```

Whereas in `v4.49`:
```
['Halo dzień dobry.', 'Nazywam się Magdalena, czy z Panem Bartoszem rozmawiam?', 'Bardzo mi miło ja mam do przekazania kilka informacji jeszcze wspomnę o rejestracji połączenia już panu wyjaśniam.']
--vs--
['Halo dede', 'Nazywam się Magdalena czy z Panem Bartoszem rozmawiam?', 'Bardzo mi miło ja mam do przekazania kilka informacji jeszcze wspomnę o rejestracji połączenia już panu wyjaśniam.']
```

In  the example above, the differences are not huge, but when the sentences in the batch have very different sizes, things get much worse and random letters are generated everywhere.

It seems that the only commit that occurred in the T5 model in that time frame was 24405127233e1d09b69bea1321c4db920e87bd7a, so that would be my first place to look, but it could just as well be something external.

Comments:
Duplicate of https://github.com/huggingface/transformers/issues/36293

The issue was fixed recently, please try installing from `main` branch

Yep, that fixed it. I installed the main branch and it works fine. Sorry for not noticing it before.
Feature request for encoding more than one pair of texts
Author: hadaev8
Pull request: no
Labels: name: Feature request

# 🚀 Feature request

Currently, tokenizer may take only inputs like [['text_0', 'text_1']], would be beneficially to expand is as
[['text_0', 'text_1', ..., 'text_n']]

## Motivation

This would open a convenient way to deal with a new set of processing tasks.

## Your contribution

Don't have any.

Comments:
checkpoint will be saved twice at the end of training when save_strategy is epoch
Author: AaronZLT
Pull request: no
Labels: name: bug

### Reproduction

This is caused by

https://github.com/huggingface/transformers/blob/dd16acb8a3e93b643aa374c9fb80749f5235c1a6/src/transformers/trainer_callback.py#L608

and

https://github.com/huggingface/transformers/blob/dd16acb8a3e93b643aa374c9fb80749f5235c1a6/src/transformers/trainer_callback.py#L623

Saving will happen at the end of training and the end of epoch (which is also triggered when setting `save_strategy` to `epoch`). Same model will be saved twice, also, the `on_save` function will be triggered twice.


Comments:
Fixed by Nosimus AI - https://github.com/huggingface/transformers/pull/36219
[tests] enable bnb tests on xpu
Author: faaany
Pull request: yes

## What does this PR do?
As the title suggests, this PR enables all bnb-related tests on XPU.

@ydshieh 

Comments:
> Are those tests now passing on XPU?

yes, they are passing on XPU. 

Great, I will merge as @matthewdouglas also approved which gives me a lot more confidence.

Thank you for taking care XPU with 🤗 !
Add autoquant support for torchao quantizer
Author: jerryzh168
Pull request: yes

Summary:
att, also verified that autoquantized model can be saved and loaded:

save: https://gist.github.com/jerryzh168/01d367aaf44dbbbfd4068a4a10a00061
load: https://gist.github.com/jerryzh168/d5c6c401b2abdf18e0b6771341f1525c

Test Plan:
tested locally with above script
model uploaded to https://huggingface.co/jerryzh168/llama3-8b-autoquant

Reviewers:

Subscribers:

Tasks:

Tags:


Comments:
README can be updated after https://github.com/huggingface/transformers/pull/35490 is landed

cc @SunMarc can you help review?

@SunMarc please take a look again, thanks

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35503). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SunMarc thanks for the review, yeah for the doc, I plan to add autoquant after https://github.com/huggingface/transformers/pull/35490 is landed to avoid conflict

gentle ping @ArthurZucker as @jerryzh168 will soon be in PTO. 

@ArthurZucker can you help land this?

Can you update the documentation @jerryzh168 ? After that, I will merge the PR 

@ArthurZucker added the cache_implementation settings, I didn't use disable_compile right now, is it required as well?

@SunMarc added autoquant to doc
Update amd pytorch index to match base image
Author: ivarflakstad
Pull request: yes

Using `rocm/dev-ubuntu-22.04:6.2.4` implies we should use `--index-url https://download.pytorch.org/whl/rocm6.2.4` as well.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36347). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Modify save_steps when resuming from a checkpoint.
Author: dignfei
Pull request: yes


# What does this PR do?
When training is halfway through, it's common to find that the settings for save_steps, eval_steps, or logging_steps are not optimal and need to be adjusted. After stopping the training, modifying the parameters, and restarting, the new parameters cannot be overwritten by the checkpoint parameters.


Comments:
cc @sunmarc @muellerzr
notify new model merged to `main`
Author: ydshieh
Pull request: yes

# What does this PR do?

Notify new model PR being merged to `main`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36375). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update Mistral converter
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. More general.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35967). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Ok @ArthurZucker, moved the conversion logic to integrations/mistral.py and removed it from Pixtral (it was actually not used in the current script)

@Cyrilvallez does this PR allow converting mistral-nemo to HF format now?

Mistral nemo checkpoints should (and are, after a quick check) already be in HF format 😉 But yes otherwise the converter should handle them correctly

> Mistral nemo checkpoints should (and are, after a quick check) already be in HF format 😉 But yes otherwise the converter should handle them correctly

Cool, thank you! I can confirm the latest commit can convert the Mistral Nemo checkpoints (from Mistral) to HF format with no issues.
Making apply_rotary_pos_emb generic
Author: xuzifei-dmatrix
Pull request: no
Labels: name: Feature request

### Feature request

Is it possible to move the definition of apply_rotary_pos_emb function out of the model specific files (eg: modeling_llama.py) to a more generic file (eg: utils)? I noticed they are defined repeatedly for multiple models.

### Motivation

I'm trying to apply torch.fx.wrap on the apply_rotary_pos_emb function. But since they are defined in multiple model files, I need to wrap it for multiple times (eg: I would need to wrap both transformers.models.llama.modeling_llama.apply_rotary_pos_emb and transformers.models.gemma.modeling_gemma.apply_rotary_pos_emb). And I need to be aware of what new models are using apply_rotary_pos_emb and wrap their definition of the function accordingly.

### Your contribution

If you could point me to the location to place the generic apply_rotary_pos_emb. I could help to create the PR.  Thanks!

Comments:
hi @xuzifei-dmatrix, what's the reason for adding `@torch.fx.wrap` to `apply_rotary_pos_emb`?

Hi @Rocketknight1 ! For some specific project requirements, we need to fx trace models and would like to keep apply_rotary_pos_emb as leaf functions and not trace through it. That's why we are adding `@torch.fx.wrap` to `apply_rotary_pos_emb`.