[bugfix] Fix tensor device in Idefics2, Idefics3, and SmolVLM
Author: qgallouedec
Pull request: yes
Labels: name: for patch

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface, @SunMarc and @qgallouedec
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39975). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hello @zucchini-nlp @andimarafioti, Any updates on this? I’m still getting the same error while working with SmolVLM. Thanks 


```      1 # Generate outputs
----> 2 generated_ids = model.generate(**inputs, max_new_tokens=500)
      3 generated_texts = processor.batch_decode(
      4     generated_ids,
      5     skip_special_tokens=True,

/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py in decorate_context(*args, **kwargs)
    114     def decorate_context(*args, **kwargs):
    115         with ctx_factory():
--> 116             return func(*args, **kwargs)
    117 
    118     return decorate_context

/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py in generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, use_model_defaults, custom_generate, **kwargs)
   2632 
   2633             # 12. run sample (it degenerates to greedy search when `generation_config.do_sample=False`)
-> 2634             result = self._sample(
   2635                 input_ids,
   2636                 logits_processor=prepared_logits_processor,

/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py in _sample(self, input_ids, logits_processor, stopping_criteria, generation_config, synced_gpus, streamer, **model_kwargs)
   3613 
   3614             if is_prefill:
-> 3615                 outputs = self(**model_inputs, return_dict=True)
   3616                 is_prefill = False
   3617             else:

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs)
   1551             return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1552         else:
-> 1553             return self._call_impl(*args, **kwargs)
   1554 
   1555     def _call_impl(self, *args, **kwargs):

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)
   1560                 or _global_backward_pre_hooks or _global_backward_hooks
   1561                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1562             return forward_call(*args, **kwargs)
   1563 
   1564         try:

/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py in wrapper(self, *args, **kwargs)
    957         if return_dict_passed is not None:
    958             return_dict = return_dict_passed
--> 959         output = func(self, *args, **kwargs)
    960         if not return_dict and not isinstance(output, tuple):
    961             output = output.to_tuple()

/usr/local/lib/python3.10/dist-packages/transformers/models/idefics3/modeling_idefics3.py in forward(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, pixel_values, pixel_attention_mask, image_hidden_states, labels, use_cache, output_attentions, output_hidden_states, cache_position, return_dict, logits_to_keep, **kwargs)
    968 
    969         # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
--> 970         outputs = self.model(
    971             input_ids=input_ids,
    972             attention_mask=attention_mask,

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs)
   1551             return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1552         else:
-> 1553             return self._call_impl(*args, **kwargs)
   1554 
   1555     def _call_impl(self, *args, **kwargs):

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)
   1560                 or _global_backward_pre_hooks or _global_backward_hooks
   1561                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1562             return forward_call(*args, **kwargs)
   1563 
   1564         try:

/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py in wrapper(self, *args, **kwargs)
    957         if return_dict_passed is not None:
    958             return_dict = return_dict_passed
--> 959         output = func(self, *args, **kwargs)
    960         if not return_dict and not isinstance(output, tuple):
    961             output = output.to_tuple()

/usr/local/lib/python3.10/dist-packages/transformers/models/idefics3/modeling_idefics3.py in forward(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, pixel_values, pixel_attention_mask, image_hidden_states, use_cache, output_attentions, output_hidden_states, cache_position, return_dict, **kwargs)
    790             raise ValueError("You cannot specify both pixel_values and image_hidden_states at the same time")
    791         elif pixel_values is not None:
--> 792             image_hidden_states = self.get_image_features(pixel_values, pixel_attention_mask)
    793         elif image_hidden_states is not None:
    794             image_hidden_states = image_hidden_states.to(dtype=self.dtype, device=input_ids.device)

/usr/local/lib/python3.10/dist-packages/transformers/models/idefics3/modeling_idefics3.py in get_image_features(self, pixel_values, pixel_attention_mask)
    717 
    718         # Get sequence from the vision encoder
--> 719         image_hidden_states = self.vision_model(pixel_values=pixel_values, patch_attention_mask=patch_attention_mask)
    720         image_hidden_states.last_hidden_state
    721 

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs)
   1551             return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1552         else:
-> 1553             return self._call_impl(*args, **kwargs)
   1554 
   1555     def _call_impl(self, *args, **kwargs):

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)
   1560                 or _global_backward_pre_hooks or _global_backward_hooks
   1561                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1562             return forward_call(*args, **kwargs)
   1563 
   1564         try:

/usr/local/lib/python3.10/dist-packages/transformers/models/idefics3/modeling_idefics3.py in forward(self, pixel_values, patch_attention_mask, output_attentions, output_hidden_states, return_dict)
    556             patch_attention_mask = patch_attention_mask.to(dtype=torch.bool, device=pixel_values.device)
    557 
--> 558         hidden_states = self.embeddings(pixel_values=pixel_values, patch_attention_mask=patch_attention_mask)
    559 
    560         patch_attention_mask = patch_attention_mask.view(batch_size, -1)

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs)
   1551             return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1552         else:
-> 1553             return self._call_impl(*args, **kwargs)
   1554 
   1555     def _call_impl(self, *args, **kwargs):

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)
   1560                 or _global_backward_pre_hooks or _global_backward_hooks
   1561                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1562             return forward_call(*args, **kwargs)
   1563 
   1564         try:

/usr/local/lib/python3.10/dist-packages/transformers/models/idefics3/modeling_idefics3.py in forward(self, pixel_values, patch_attention_mask)
    154             fractional_coords_w = w_indices / nb_patches_w * (1 - 1e-6)
    155 
--> 156             bucket_coords_h = torch.bucketize(fractional_coords_h, boundaries, right=True)
    157             bucket_coords_w = torch.bucketize(fractional_coords_w, boundaries, right=True)
    158 

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument boundaries in method wrapper_CUDA_Tensor_bucketize)
```
transformers version -> 4.55.0

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: idefics2, idefics3, smolvlm

``` 
tests/models/smolvlm/test_modeling_smolvlm.py::SmolVLMModelTest::test_torch_export SKIPPED (test_torch_exportable=False for this model.)                                                                                                      [ 20%]
tests/models/smolvlm/test_modeling_smolvlm.py::SmolVLMForConditionalGenerationModelTest::test_torch_export SKIPPED (test_torch_exportable=False for this model.)                                                                              [ 40%]
tests/models/smolvlm/test_modeling_smolvlm.py::SmolVLMForConditionalGenerationIntegrationTest::test_export_smolvlm_connector PASSED                                                                                                           [ 60%]
tests/models/smolvlm/test_modeling_smolvlm.py::SmolVLMForConditionalGenerationIntegrationTest::test_export_smolvlm_text_decoder PASSED                                                                                                        [ 80%]
tests/models/smolvlm/test_modeling_smolvlm.py::SmolVLMForConditionalGenerationIntegrationTest::test_export_smolvlm_vision_encoder PASSED  
```