noamaanMulla-03/transformers-issues huggingface dataset:
GPT2 cannot be used with device_map='auto'; Report "found at least two devices"
Author: haobozhang
Pull request: no
Labels: name: Good Difficult Issue

name: Accelerate

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-5.15.0-89-generic-x86_64-with-glibc2.35
- Python version: 3.9.18
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.1
- Accelerate version: 0.26.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

A simple reproducer here:
```python
from transformers import GPT2LMHeadModel
# create a sample input:
batch_ids = {
    'input_ids': torch.tensor([[312, 134, 56, 712, 351, 89, 63, 550, 971, 2]]),
    'attention_mask': torch.tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]),
}

gpt2_large = GPT2LMHeadModel.from_pretrained('gpt2-large', cache_dir='./cache_dir', device_map='auto')
gpt2 = GPT2LMHeadModel.from_pretrained('gpt2', cache_dir='./cache_dir', device_map='auto')

loss_gpt2_large = gpt2_large(**batch_ids, labels=batch_ids['input_ids']).loss
loss_gpt2 = gpt2(**batch_ids, labels=batch_ids['input_ids']).loss
``` 

### Expected behavior

It works well to generate `loss_gpt2_large`, but it will report error when generating `loss_gpt2`: 
`RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:7 and cpu! (when checking argument for argument index in method wrapper_CUDA__index_select)`

I am not sure why this behaves differently with the same model class. Could you please provide any comments on this? Thanks in advance!

Comments:
Finally, the checkpoint got completed after 2 hours on an Instance with 8 TPU cores, 350GB RAM, and 98 vCPUs. I was wondering, how can I make it faster. I understand that the library download the model params from TPU to CPU, and then save them. 

Am I missing some params that can make it faster to take checkpoints?

Hi @haobozhang, thanks for raising this issue and apologies for the delay. 

Pinging @muellerzr and @pacman100, as this seems to be an issue with weight offloading. 

Hi @amyeroberts, is the issue still there? I tried to reproduce it, but it didn't show me an error. 
If the issue still persists, I would like to work on this
Update modeling_llava_onevision.py
Author: yinsong1986
Pull request: yes

Fixed a potential bug in modeling_llava_onevision.py

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #36390


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
FYI: @amyeroberts @qubvel, thank you!
A potential bug in modeling_llava_onevision.py
Author: yinsong1986
Pull request: no
Labels: name: bug

### System Info

The `special_image_mask` in [this line](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llava_onevision/modeling_llava_onevision.py#L743) seems to be a typo, it should be `special_video_mask`?

Thank you!

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The `special_image_mask` in [this line](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llava_onevision/modeling_llava_onevision.py#L743) seems to be a typo, it should be `special_video_mask`

### Expected behavior

The `special_image_mask` in [this line](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llava_onevision/modeling_llava_onevision.py#L743) seems to be a typo, it should be `special_video_mask`

Comments:
WARNING: transformers 4.49.0 does not provide the extra 'emu3'
Author: Prasaderp
Pull request: no
Labels: name: Feature request

### Feature request

WARNING: transformers 4.49.0 does not provide the extra 'emu3'

### Motivation

WARNING: transformers 4.49.0 does not provide the extra 'emu3'

### Your contribution

no

Comments:
@Prasaderp hey! Can you provide a small reproducer? Are you trying to load the model, is there any regression from v4.48?
[WIP] Add support for flex attention (paged attention)
Author: blzheng
Pull request: yes

# What does this PR do?

This pr integrates PyTorch Flex attention (paged attention) into Llama and GPT-J (greedy search).

## TODOs
- [ ] Add flex attention for first token. The current implementation uses SDPA for first token and flex attention for next token
- [ ] Address re-compile issues to eliminate hard-coded logic
- [ ] Extend support to beam search

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.




Comments:
Hi @blzheng, please feel free to ping me to assign a reviewer whenever this is ready!
[modular] Do not track imports in functions
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. This was a small overlook in modular logic, as imports inside functions should not be tracked (they should stay defined in the function).
Fixes https://github.com/huggingface/transformers/issues/36208

Also updates the `examples` based on latest repo changes.

cc @ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36279). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix doc formatting in forward passes & modular
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. There were several indentation issues with the decorators for docs.
Supersedes https://github.com/huggingface/transformers/pull/36172

cc @gante 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36243). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
addressing the issue #34611 to make FlaxDinov2 compatible with any batch size
Author: MHRDYN7
Pull request: yes
Labels: name: Flax

This PR solves the batch size issue with FlaxDinov2 model #34611. The commonly used tests across the transformers library could not detect the error in the "interpolate_position_encoding" method, which only worked for single images. The issue has been fixed using jnp.tile for it's simplicity, although jnp.repeat could also be used.
The slow tests have also been modified to pass a batch of images instead of just one.

@amyeroberts could you please review the changes.

P.S. this pr does exactly what  #34620 aimed to do, however, with a few more improvements.

Comments:
@NielsRogge, since you've been working on Dinov2 registers recently, can I please request a quick review on this one as well?

@gante I'm not really sure who is currently working on flax in the hf team, could you please mention the right person? Need a quick review to merge this
Fix tied weigths isuue
Author: ydshieh
Pull request: yes

# What does this PR do?

Relate to #33689 and #33688

- This line
https://github.com/huggingface/transformers/blob/ab97a78130f96e72eec609c56cb5f719529ffb9e/src/transformers/modeling_utils.py#L4517

  may have `state_dict` containing weights with same `data_ptr` (tied). Those tensors are on `cpu` with `dtype=float32`.
  (Note this call doesn't take into account of `model.config`.)

- This line
https://github.com/huggingface/transformers/blob/ab97a78130f96e72eec609c56cb5f719529ffb9e/src/transformers/modeling_utils.py#L4537

  will arrive [line 1](https://github.com/huggingface/transformers/blob/ab97a78130f96e72eec609c56cb5f719529ffb9e/src/transformers/modeling_utils.py#L935) and **line 2**

https://github.com/huggingface/transformers/blob/ab97a78130f96e72eec609c56cb5f719529ffb9e/src/transformers/modeling_utils.py#L990

- If the specified `torch_dtype` is `torch.float32` and `device_map="cpu"`, the `line 1` and `line 2` won't create new tensors. So the model will get the weights directly from `state_dict` and the tied weights remain tied.

- In other cases, either line 1 or line 2 will create new tensors and assign to `model`, so there is no tied weights inside the model with those new values.
  - This explains why #33689 and #33688 have issue only with `cpu+float32`

This PRs provides a possible fix to avoid tied weights being assigned within `_load_state_dict_into_meta_model`. 

The tied weights will be handled after `_load_pretrained_model` is called within `from_pretrained`, see

https://github.com/huggingface/transformers/blob/ab97a78130f96e72eec609c56cb5f719529ffb9e/src/transformers/modeling_utils.py#L4057-L4058





Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33913). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

We might need similar change for `_load_state_dict_into_model`, but I would like to have a review to see if the current change is the good one.

I'll take a look next week, cc @ArthurZucker in case he has some free bandwidth before me

will try to add a test (need to find a dummy checkpoint that cause the same original issue!

@ArthurZucker test is added

https://github.com/huggingface/transformers/pull/33913/files#diff-9828696cac704ee7096167deda48195c4d51892af221f7851f6404718796c5dbR2616-R2695

kindly ping @ArthurZucker whenever you have bandwidth 🙏

Kindly ping @ArthurZucker 


> I trust you that this fixes it! I would just need a test to be added. tie weights is SO sensitive

A test is added

Kindly ping @ArthurZucker 🙏
Huggingface/transfomers v3.3.2 is throwing import errors when building on Nextjs 14
Author: cookiejul
Pull request: no
Labels: name: bug

### System Info

Operating System: MacOS Sequoia 15.2
Hardware: M1 Macbook Pro


### Who can help?

@zucchini-nlp @muellerzr

![Image](https://github.com/user-attachments/assets/11639dbb-fb13-4ea0-861a-b2a61362b63f)

I get this error when I run`pnpm build` to build my project to production but it kept throwing this error. I've tried downgrading the version of @huggingface/transformers package to v3.0.0 and v3.0.2 but I can't even get the model to run at all.  

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Here are the steps needed to reproduce the error

1. Set up a simple nextjs app
2. pnpm install @huggingface/transformers
3. run `pnpm build`

### Expected behavior

The expected behavior is to be able to successfully build the nextjs application so that I can host it and allow users to use my app

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @cookiejul and sorry for the delay! Bug reports for the JS package should probably go to https://github.com/huggingface/transformers.js/
Fix integration tests for SAM
Author: qubvel
Pull request: yes
Labels: name: Tests

name: Vision

# What does this PR do?

Fix device placement for tensors for pytorch integration tests of SAM model


Comments:
run-slow: sam

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/sam']
quantizations: [] ...

Pytorch tests are fixed; only TF fails left, won't fix 

```
FAILED tests/models/sam/test_modeling_tf_sam.py::TFSamModelIntegrationTest::test_inference_mask_generation_three_boxes_point_batch - TypeError: Cannot convert '__empty__' to EagerTensor of dtype int32
FAILED tests/models/sam/test_modeling_tf_sam.py::TFSamModelIntegrationTest::test_inference_mask_generation_two_points_point_batch - TypeError: Cannot convert '__empty__' to EagerTensor of dtype int32
FAILED tests/models/sam/test_processor_sam.py::TFSamProcessorTest::test_rle_encoding - TypeError: Only integers, slices (`:`), ellipsis (`...`), tf.newaxis (`None`) and scalar tf.int32/tf.int64 tensors are valid indices, got <tf.Tensor: shape=(0,), dtype=bool, numpy=array([], dtype=bool)>
===== 3 failed, 109 passed, 192 skipped, 26 warnings in 217.18s (0:03:37) ======
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36397). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`transformers-cli` now requires PyTorch
Author: setu4993
Pull request: no
Labels: name: bug

### System Info

N/A

### Who can help?

@gante (author of https://github.com/huggingface/transformers/pull/35714)

### Reproduction

```sh
uv run --python 3.12 --with transformers transformers-cli --help
```

### Expected behavior

Since https://github.com/huggingface/transformers/pull/35714 (released in v4.49.0), the `transformers-cli` now requires PyTorch to even run a simple: `transformers-cli --help` command.

I am pretty sure it is originating from [this line](https://github.com/huggingface/transformers/blame/626666c444208557041ef7edcda6b9e78eddfdee/src/transformers/commands/chat.py#L27).

My question is if this is intentional or unintentional. So far, PyTorch has been an optional dependency for any functionality with CLI or the package itself.

Comments:
Seems real, and I don't think we should require Torch for `transformers-cli`. I think this is probably a testing failure more than anything, though - cc @gante, maybe we should have tests for `transformers-cli` that don't `@require_torch` so we catch things like this in future?

I agree this is an issue that should be addressed &mdash; two dependencies that aren't installed without adding some `extras` elements, `pytorch` and `rich`, are now required to run `transformers-cli`. Helpfully, we get a message about PyTorch not being found and how that affects functionality, but the `chat.py` component isn't wrapped in a try / except to prevent import failure with the default requires list.

I think you're running into #35714, which made PyTorch a required dependency for `transformers-cli`.  
Even a simple `transformers-cli --help` now needs PyTorch to work.  
You can fix this by installing PyTorch (`pip install torch`) or downgrading `transformers` to a version before v4.49.0.  



gentle ping @gante! Was that dependency intentional?

@Rocketknight1 not intentional, I'll add a guard (and tag the branch for a patch release)
[CLI] add import guards
Author: gante
Pull request: yes
Labels: name: for patch

# What does this PR do?

Fixes #36244 

The introduction of `transformers-cli chat` caused ANY cli call to depend on `torch` and `rich`, both optional requirements. This PR adds the appropriate import guards -- tested against a fresh venv with `pip install -e .` (no extras).

@Rocketknight1 we don't have tests for commands 👀 I'm going to defer it for now, as we probably need to review our commands for v5.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36376). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
SamImageProcessor do_convert_rgb not working correctly
Author: MSt-10
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.49.0.dev0
- Platform: Linux-5.8.0-63-generic-x86_64-with-glibc2.31
- Python version: 3.11.10
- Huggingface_hub version: 0.26.3
- Safetensors version: 0.4.5
- Accelerate version: 1.1.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA RTX 6000 Ada Generation

### Who can help?

@qubvel 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When passing a gray scale PIL image (mode='L') to the SamProcessor, the SamImageProcessor is called to preprocess the image. This includes potentially converting the image to rgb if possible. This is done in the method [`SamImageProcessor._preprocess_image`:](https://github.com/huggingface/transformers/blob/92c5ca9dd70de3ade2af2eb835c96215cc50e815/src/transformers/models/sam/image_processing_sam.py#L293)

```
        image = to_numpy_array(image)

        # PIL RGBA images are converted to RGB
        if do_convert_rgb:
            image = convert_to_rgb(image)

        # All transformations expect numpy arrays.
        image = to_numpy_array(image)
```

Which does the following:
```
def convert_to_rgb(image: ImageInput) -> ImageInput:
    """
    Converts an image to RGB format. Only converts if the image is of type PIL.Image.Image, otherwise returns the image
    as is.
    Args:
        image (Image):
            The image to convert.
    """
    requires_backends(convert_to_rgb, ["vision"])

    if not isinstance(image, PIL.Image.Image):
        return image

    if image.mode == "RGB":
        return image

    image = image.convert("RGB")
    return image
``` 
So when passing a grayscale PIL image to the processor I would expect it to be correctly converted to RGB when setting `do_convert_rgb`.

The cause of the error is the following line:
https://github.com/huggingface/transformers/blob/92c5ca9dd70de3ade2af2eb835c96215cc50e815/src/transformers/models/sam/image_processing_sam.py#L310

The passed image gets converted to a numpy array **before** the `convert_to_rgb` method is called. 
Hence, the check if the image is a PIL image fails. 
Since there is also a conversion to numpy **after** the `convert_to_rgb` method call, I suspect this is just a simple mistake.

### Expected behavior

When passing a grayscale PIL image to the processor I would expect it to be correctly converted to RGB.

Comments:
Removed numpy conversion to correctly convert PIL image to rgb
Author: MSt-10
Pull request: yes
Labels: name: bug

name: Vision

name: Processing

# What does this PR do?
Fixes #36368 

## Who can review?
@qubvel 
@ArthurZucker 


Comments:
run-slow: sam

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/sam']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36369). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Security fix for `benchmark.yml`
Author: ydshieh
Pull request: yes

# What does this PR do?

Security is important than ever now.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36402). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix bug in prepare_inputs_for_generation function: in Qwen2-VL (#36037)
Author: JamesHujy
Pull request: yes

Fix the problem in Qwen2-VL. A similar problem is mentioned in [PR #34095](https://github.com/huggingface/transformers/pull/34095). 

When using synced_gpus, after one peer finishes generating, the cache position in the generation process continues to increase. This leads to the input IDs going out of bounds, resulting in errors. The issue specifically occurs in the following line of code:
[modeling_qwen2_vl.py#L1739](https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen2_vl/modeling_qwen2_vl.py#L1739).

The root cause seems to be the difference in the implementation of the prepare_inputs_for_generation function compared to the default implementation found here:
[utils.py#L388](https://github.com/huggingface/transformers/blob/main/src/transformers/generation/utils.py#L388).


Comments:
cc @gante (and sorry!)

@JamesHujy after sorting the conflicts, it seems like the fix is already on `main` :D Can you confirm that your intended fix is indeed already present?