SmolVLM RuntimeError Expected all tensors to be on the same device, but found at least two devices
Author: sayandipdutta
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.56.0.dev0
- Platform: Linux-6.8.0-1033-gcp-x86_64-with-glibc2.35
- Python version: 3.12.9
- Huggingface_hub version: 0.34.4
- Safetensors version: 0.6.2
- Accelerate version: 1.10.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.7.1+cu126 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA A100-SXM4-40GB

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:

1. Install transformers from `git`: `https://github.com/huggingface/transformers.git#9977cf17392fe5b821dc026ab76d3f5ed16e03f`
2. Install necessary dependencies to run the [example](https://huggingface.co/HuggingFaceTB/SmolVLM2-2.2B-Instruct#how-to-get-started) at `HuggingFaceTB/SmolVLM2-2.2B-Instruct` model card.
3. Try to run the [Simple Inference example](https://huggingface.co/HuggingFaceTB/SmolVLM2-2.2B-Instruct#simple-inference).

Produces the following output:
```
You have video processor config saved in `preprocessor.json` file which is deprecated. Video processor configs should be saved in their own `video_preprocessor.json` file. You can rename t
he file or load and save the processor back which renames it automatically. Loading from `preprocessor.json` will be removed in v5.0.
Loading checkpoint shards: 100%|ΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûê| 2/2 [00:01<00:00,  1.96it/s]
Traceback (most recent call last):
  File "/home/sayan/projects/argo/src/argo/extraction/app3.py", line 65, in <module>
    generated_ids = model.generate(**inputs, do_sample=False, max_new_tokens=64)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2542, in generate
    result = self._sample(
             ^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 3523, in _sample
    outputs = self(**model_inputs, return_dict=True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 959, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/transformers/models/smolvlm/modeling_smolvlm.py", line 950, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 959, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/transformers/models/smolvlm/modeling_smolvlm.py", line 763, in forward
    image_hidden_states = self.get_image_features(pixel_values, pixel_attention_mask).to(inputs_embeds.device)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/transformers/models/smolvlm/modeling_smolvlm.py", line 689, in get_image_features
    image_hidden_states = self.vision_model(pixel_values=pixel_values, patch_attention_mask=patch_attention_mask)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/transformers/models/smolvlm/modeling_smolvlm.py", line 445, in forward
    hidden_states = self.embeddings(pixel_values=pixel_values, patch_attention_mask=patch_attention_mask)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sayan/projects/argo/.venv/lib/python3.12/site-packages/transformers/models/smolvlm/modeling_smolvlm.py", line 148, in forward
    fractional_coords_h = h_indices / nb_patches_h * (1 - 1e-6)
                          ~~~~~~~~~~^~~~~~~~~~~~~~
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
```

### Expected behavior

The model should run without any error, and produce the expected output (description of the given image).

#### Possible Cause:

I found PR #39981 which supposedly fixes this issue. But upon further inspection, I found that it is not carrying the device information from the inputs, and the device of `h_indices` is always `cpu`. This can be fixed by creating `boundaries` and `position_ids` with the input device.

Comments:
I have a PR that addresses this. I am not sure what to do with [modeling_smolvlm.py](https://github.com/huggingface/transformers/blob/a1a4fcd03e3455772415e6400fee91f3159e7ac5/src/transformers/models/smolvlm/modeling_smolvlm.py#L148), since the error originated there and it is explicitly written not to edit it manually. However, `make repo-consistency` fails.

If this change is acceptable, I would appreciate some help on what to do about the above. Then I can remove the draft status. Thanks!

fixed #39975