noamaanMulla-03/transformers-issues huggingface dataset:
v4.55.0 Idefics3 RuntimeError Tensors on different devices
Author: noahleegithub
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.55.0
- Python version: 3.10.14
- PyTorch version (accelerator?): 2.7.1+cu126 (CUDA)
- Using GPU in script?: Yes
- GPU type: Tesla V100-PCIE-32GB

### Who can help?

@ArthurZucker @guangy10 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
import torch
from transformers.models.idefics3.modeling_idefics3 import Idefics3VisionEmbeddings
from transformers.models.idefics3.configuration_idefics3 import Idefics3VisionConfig

model = Idefics3VisionEmbeddings(Idefics3VisionConfig())
model = model.to("cuda")
pixel_values = torch.rand(1, 3, 32, 32, device="cuda")
patch_attention_mask = torch.rand(1, 1, 1, 1, device="cuda") > 0.5
output = model(pixel_values, patch_attention_mask)
```
Gives the following error:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument boundaries in method wrapper_CUDA_Tensor_bucketize)

### Expected behavior

Should output a cuda Tensor.

It looks like this is caused by #39614 , lines 150 and 151.
```python
[143] boundaries = torch.arange(1 / self.num_patches_per_side, 1.0, 1 / self.num_patches_per_side)

[150] h_indices = torch.arange(nb_patches_h, device=pixel_values.device, dtype=pixel_values.dtype)
[151] w_indices = torch.arange(nb_patches_w, device=pixel_values.device, dtype=pixel_values.dtype)

[156] bucket_coords_h = torch.bucketize(fractional_coords_h, boundaries, right=True)
[157] bucket_coords_w = torch.bucketize(fractional_coords_w, boundaries, right=True)
```
`boundaries` is created on the CPU, while `h_indices` and `w_indices` are dependent on the input device causing a mismatch.

Comments:
Yep. I wonder why the tests didn't catch it 🤔  Would you like to submit a PR?
Add a unit test for BartModel to compare eager, sdpa on one particular set of inputs
Author: xadupre
Pull request: yes

# What does this PR do?

Fixes #39365. Not a fix yet but introducing a unit test failing for the reason explained in this issue. Either the inputs are wrong, either the fix from issue #39365 is needed.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface, @SunMarc and @qgallouedec
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bart

@xadupre Still any interest in this PR/bug? Otherwise, I'd make another PR to apply this fix and propagate it :eyes:
Various test fixes for AMD
Author: remi-or
Pull request: yes

This PR introduces minor test fixes for 4 different models:
- for `internvl` we add an Expectation for ROCm 9.4 and remove a tensor being created from a tensor;
- for `llama` we add an Expectation for ROCm 9.4;
- for `llava` we add the `@require_bitsandbytes` decorator for a test requires bitsandbytes;
- for `mistral3` we add an Expectation for ROCm 9.4;

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: internvl, llama, llava, mistral3

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39978). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you both for the work and the autonomy of handling AMD PRs.

I am cooking something to ease the process of updating expected values, will share soon!
[CI] post-`GptOss` fixes for green CI
Author: gante
Pull request: yes
Labels: name: for patch

# What does this PR do?

- breaks copy on MoE loss (losses were updated in #39923, discussed on slack)
- renames `OpenAIMoE` -> `GptOss` in the model docs
- a few other minor nits to make CI happy

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: granitemoe, jamba, jetmoe, olmoe, phimoe, qwen2_moe

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39929). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix return typehint for decoder and annotate inv_freq
Author: qubvel
Pull request: yes
Labels: name: typing

# What does this PR do?

Fix return type hint for decoder layer and annotate inv_freq

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39610). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: arcee, aria, bamba, bitnet, chameleon, cohere, cohere2, csm, dbrx, deepseek_v2, deepseek_v3, dia, diffllama, doge, dots1, efficientloftr

Thanks for the review!
[RoPE] allow models to configure local RoPE
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Failing tests reported by CI bot after removing deprecations in https://github.com/huggingface/transformers/pull/38838

The main fixes are in test files and in ModernBert/Llama. The test was failing because RoPE needs to use local theta in local attention layers. I think at this point we can allow `local_rope_theta`, looks like ModernBert isn't the only arch using two different RoPE configurations

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39397). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: arcee, aria, bamba, bitnet, cohere, cohere2, csm, deepseek_v3, dia, diffllama, doge, dots1, emu3, falcon, falcon_h1, fuyu

This comment contains run-slow, running the specified jobs: 

models: ['models/arcee', 'models/aria', 'models/bamba', 'models/bitnet', 'models/cohere', 'models/cohere2', 'models/csm', 'models/deepseek_v3', 'models/dia', 'models/diffllama', 'models/doge', 'models/dots1', 'models/emu3', 'models/falcon', 'models/falcon_h1', 'models/fuyu']
quantizations: [] ...

Failing slow tests are failing on `main` as well

>  What if we make it rope_config_key: str = "rope_theta" instead?

It is not only the theta that changes, but the RoPE type might be different. For ex in gemma3 local attention is simple RoPE and the global attention applies scaling. Thus we have config keys:
`rope_scaling_global="linear"` and `rope_scaling_local=None`

Happy to simplify it if you have any ideas. Maybe we can do smth like `rope_scaling = {"global" : {global_dict_here}, "local": None}`? Though we need to make sure RoPE can change dynamically within one model, so imo `is_global` is the most straightforward way to allow it

@zucchini-nlp I see 🤔 In general, my concern is about the case where more types of RoPE are added, but the flag we add in this PR wouldn't be able to accommodate it. Secondarily, we should try to nudge users into nesting parameterization, since RoPE can take several parameters and things can get messy really fast (i.e. instead of xxx / local_xxx, have the structure you suggested by default, `rope_scaling = {"global" : {global_dict_here}, "local": None}`).

Perhaps `rope_config_key: str = "global"`, with some extra logic in the rope files to parse global/local parameters when they are not nested? Then we would simultaneously:
- support all existing formats, no BC breaking
- allow as many rope types as the user wishes
- nudge users into nested parameterization

Potentially missing: some attribute in the dict to control which layers use a certain rope type? (similar to the `layer_types` attribute)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: arcee, aria, bamba, bitnet, cohere, cohere2, csm, deepseek_v2, deepseek_v3, dia, diffllama, doge, dots1, emu3, ernie4_5, ernie4_5_moe

There is a cleaner PR
Fix consistency
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

cc @qubvel for viz after your PR!

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gpt_oss

Thanks for the fix and sorry for that 🤗 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39995). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update expected output values after #39885 (part 1)
Author: ydshieh
Pull request: yes

# What does this PR do?

The changes are expected.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: beit, conditional_detr, d_fine, dab_detr, deformable_detr, detr, dpt, fastspeech2_conformer, grounding_dino, levit, mask2former, maskformer, mobilenet_v1, mobilenet_v2, mobilevit, mobilevitv2

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39990). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Idefics] fix device mismatch
Author: zucchini-nlp
Pull request: yes
Labels: name: for patch

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/39947. We need to use consistent device when computing `position_ids`, and since we are updating `position_ids` we will use its device instead of `pixel_values`. Otherwise it raises errors because those are located in two different devices

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: idefics2, idefics3, smolvlm

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39981). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix missing video inputs for PerceptionLM.
Author: shuminghu
Pull request: yes
Labels: name: for patch


Critical: Fixes missing video input for PerceptionLM (accidentally removed in [PR](https://github.com/huggingface/transformers/pull/39583))

Minor: Add support for vanilla image that only has C,H,W dims but not tiles dim.
This is non-default image shapes used in PLM but it's useful in demos and low-resoure devices.
e.g., in just added  "PLM Simple Fine-tuning Example" under
https://huggingface.co/facebook/Perception-LM-1B#plm-usage






Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39971). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zucchini-nlp The reason shape unification is done in models rather than image_processing is that i noticed in training model sees a different input shape than in eval/inference.

- In training with `per_device_train_batch_size=8`, when image processor output 4D: `processed_images.shape:  torch.Size([1, 3, 448, 448])`, model input is 5D: `pixel_values.shape:  torch.Size([8, 1, 3, 448, 448])`. , when image processor output is 5D, model input is 6D and will error out.
- In inference, both are 5D, `processed_images.shape:  torch.Size([1, 37, 3, 448, 448])`, pixel_values.shape:  `torch.Size([1, 37, 3, 448, 448])`


@zucchini-nlp Let me split the PR and merge the more urgent fix first?

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: perception_lm

@zucchini-nlp My bad. Just realized this from collate_fn in my training script ( I added one dimension)
```
    pixel_values = torch.stack(
        [inst["pixel_values"] for inst in instances], dim=0
    )
  ```
Let me open another [PR](https://github.com/huggingface/transformers/pull/40006) for this simple fix for image_preprocessor and update corresponding training script in model card.  
  
> The reason shape unification is done in models rather than image_processing is that i noticed in training model sees a different input shape than in eval/inference.
> 
> In training with per_device_train_batch_size=8, when image processor output 4D: processed_images.shape:  torch.Size([1, 3, 448, 448]), model input is 5D: pixel_values.shape:  torch.Size([8, 1, 3, 448, 448]). , when image processor output is 5D, model input is 6D and will error out.
> In inference, both are 5D, processed_images.shape:  torch.Size([1, 37, 3, 448, 448]), pixel_values.shape: torch.Size([1, 37, 3, 448, 448])
`convert_deepseek_vl_weights_to_hf.py` not included in v4.55.0 release.
Author: rasmi
Pull request: no
Labels: name: bug


`convert_deepseek_vl_weights_to_hf.py`, introduced in #36248, is in [main](https://github.com/huggingface/transformers/blob/main/src/transformers/models/deepseek_vl/convert_deepseek_vl_weights_to_hf.py) but not in the [v4.55.0 release](https://github.com/huggingface/transformers/blob/v4.55.0/src/transformers/models/deepseek_vl/convert_deepseek_vl_weights_to_hf.py).

This causes the following two tests to fail:
* [tests/models/deepseek_vl/test_processing_deepseek_vl.py](https://github.com/huggingface/transformers/blob/ac0b4684657cbff1e8eecb0d966d10b71843dca0/tests/models/deepseek_vl/test_processing_deepseek_vl.py#L19)
* [tests/models/deepseek_vl_hybrid/test_processing_deepseek_vl_hybrid.py](https://github.com/huggingface/transformers/blob/ac0b4684657cbff1e8eecb0d966d10b71843dca0/tests/models/deepseek_vl_hybrid/test_processing_deepseek_vl_hybrid.py#L19)


```
ImportError while importing test module 'tests/models/deepseek_vl/test_processing_deepseek_vl.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/models/deepseek_vl/test_processing_deepseek_vl.py:19: in <module>
    from transformers.models.deepseek_vl.convert_deepseek_vl_weights_to_hf import CHAT_TEMPLATE
E   ModuleNotFoundError: No module named 'transformers.models.deepseek_vl.convert_deepseek_vl_weights_to_hf'
```

```
ImportError while importing test module 'tests/models/deepseek_vl_hybrid/test_processing_deepseek_vl_hybrid.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/models/deepseek_vl_hybrid/test_processing_deepseek_vl_hybrid.py:19: in <module>
    from transformers.models.deepseek_vl.convert_deepseek_vl_weights_to_hf import CHAT_TEMPLATE
E   ModuleNotFoundError: No module named 'transformers.models.deepseek_vl.convert_deepseek_vl_weights_to_hf'
```

### Who can help?

@geetu040 @zucchini-nlp @ArthurZucker 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Reproduction

Run tests on v4.55.0 release.

### Expected behavior

Tests pass.

Comments:
Tests have to be changed to not import anything from `convert_xxx.py` files, @geetu040 can you update them? We don't include convert files in PyPi releases usually

sure, got it, I'll update

Thank you both! 🙏
fix: remove CHAT_TEMPLATE import in tests for deepseek-vl
Author: geetu040
Pull request: yes

# What does this PR do?

Fixes #39966

This PR removes the `CHAT_TEMPLATE` imports from `test_processing_deepseek_vl.py` and `test_processing_deepseek_vl_hybrid.py`. These imports were referencing weight conversion scripts that are not included in the PyPI distribution, which causes the tests to fail in the v4.55.0 release.

The fix aligns with the approach used in [test_processing_emu3.py](https://github.com/huggingface/transformers/blob/main/tests/models/emu3/test_processing_emu3.py)

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you write any new necessary tests?


## Who can review?

@rasmi @zucchini-nlp @ArthurZucker

Comments:
@zucchini-nlp, this is ready for review. I've checked there is no other leaking import.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: deepseek_vl, deepseek_vl_hybrid

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40003). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Support FP8 accelerate config
Author: djsaunde
Pull request: yes

# What does this PR do?

Adds config options for mixed precision and fp8 config, which are supported in accelerate's Accelerator object.

Also parses the `config` field for the torchao ("AO") fp8 backend from dictionary values into the required `Float8LinearConfig` object.

This requires also a simple gating change to accelerate, which is actually covered by an existing PR: https://github.com/huggingface/accelerate/pull/3677/files#diff-2d7515874eaecac2687c7fc1a9c720be53f802bf14b4c3dcebe14ad443d075dcR501-R505.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

- @S1ro1
- @SunMarc
- @winglian 

## TODO

- [ ] Add docs (?)
- [ ] Add tests (?)
- [ ] Add quick benchmarks

## Note

I've tested this downstream in [`axolotl`](https://github.com/axolotl-ai-cloud/axolotl) and find that it works. Performance benefits can be had for certain models when setting `torch_compile: true`; it's not yet clear to me which models + hyperparameter settings. I'll add some quick numbers here to demonstrate this.

Comments:
> We have this old PR that cleans a bunch of things, let me know if this will clean a bunch of things for you. If so, I can try to take over this PR myself and first merge this PR

That would be great! For the moment we are patching things downstream in axolotl just to get things working. Having this eventually in an upstream release would be nice.

> That would be great! For the moment we are patching things downstream in axolotl just to get things working. Having this eventually in an upstream release would be nice.

I'll spend some time very soon on that PR to clean a bit trainer then ! I'll ping you when this is done to have it reviewed ! 

Sorry, I forgot to add the link of the PR I was talking about. Here you go  https://github.com/huggingface/transformers/pull/37259
video_inputs are not passed to perception_lm
Author: 4g
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.55.0
- Platform: Linux-6.14.0-27-generic-x86_64-with-glibc2.39
- Python version: 3.11.9
- Huggingface_hub version: 0.34.3
- Safetensors version: 0.4.3
- Accelerate version: 1.6.0
- Accelerate config: 	not found
- DeepSpeed version: 0.17.1
- PyTorch version (accelerator?): 2.7.1+cu126 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA GeForce RTX 4090


### Who can help?

@zucchini-nlp 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoProcessor, AutoModelForImageTextToText
from huggingface_hub import hf_hub_download

MODEL_PATH = "facebook/Perception-LM-1B"
processor = AutoProcessor.from_pretrained(MODEL_PATH, use_fast=True)
model = AutoModelForImageTextToText.from_pretrained(MODEL_PATH).to("cuda")

video_file = hf_hub_download(
    repo_id="shumingh/perception_lm_test_videos",
    filename="GUWR5TyiY-M_000012_000022.mp4",
    repo_type="dataset",
)
conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "video",
                "url": video_file,
            },
            {"type": "text", "text": "Can you describe the video in detail?"},
        ],
    }
]
inputs = processor.apply_chat_template(
    [conversation],
    num_frames=32,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    video_load_backend="decord",
)
inputs = inputs.to(model.device)
generate_ids = model.generate(**inputs, max_new_tokens=256)
input_length = inputs["input_ids"].shape[1]
generate_ids_without_inputs = generate_ids[:, input_length:]

for output in processor.batch_decode(
    generate_ids_without_inputs, skip_special_tokens=True
):
    print(output)

print(inputs.pixel_values_videos)
```

### Expected behavior

Error happens in the [official example](https://huggingface.co/facebook/Perception-LM-1B) on using PLM with video inputs . The output has no correlation to input, because `pixel_values_videos` key is missing from inputs. I believe this error was introduced in [this commit](https://github.com/huggingface/transformers/commit/947a37e8f5bc50bc0e9a77c0d16b038adcb056d0)
when `videos_inputs` was removed from `BatchFeature`. 

When passed video in conversaion, inputs only have these keys : 
{'input_ids', 'attention_mask'}

Comments:
Will be fixed by https://github.com/huggingface/transformers/pull/39971

Closing as resolved, because the PR was merged