YOLO model failing on multi GPU device map auto
Author: Carotterap
Pull request: no
Labels: name: bug

### System Info

Yolo model fails: 
```python 
(Desktop) theo@arthur-brahms:~$ /home/theo/Desktop/.venv/bin/python /home/theo/Desktop/reachy/test-vision.py
Loading weights: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 212/212 [00:00<00:00, 2564.02it/s]
/home/theo/Desktop/.venv/lib/python3.12/site-packages/accelerate/utils/modeling.py:1615: UserWarning: The following device_map keys do not match any submodules in the model: ['vit.encoder.mid_position_embeddings']
  warnings.warn(
Traceback (most recent call last):
  File "/home/theo/Desktop/reachy/test-vision.py", line 4, in <module>
    print(pipe('http://images.cocodataset.org/val2017/000000039769.jpg'))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/transformers/pipelines/object_detection.py", line 115, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/transformers/pipelines/base.py", line 1276, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/transformers/pipelines/base.py", line 1283, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/transformers/pipelines/base.py", line 1175, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/transformers/pipelines/object_detection.py", line 129, in _forward
    outputs = self.model(**model_inputs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 192, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 907, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/transformers/models/yolos/modeling_yolos.py", line 622, in forward
    outputs: BaseModelOutputWithPooling = self.vit(pixel_values, **kwargs)
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 1036, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/transformers/utils/output_capturing.py", line 252, in wrapper
    outputs = func(self, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/transformers/models/yolos/modeling_yolos.py", line 496, in forward
    encoder_outputs: BaseModelOutput = self.encoder(embedding_output, height=height, width=width)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/theo/Desktop/.venv/lib/python3.12/site-packages/transformers/models/yolos/modeling_yolos.py", line 437, in forward
    hidden_states = hidden_states + interpolated_mid_position_embeddings[i]
                    ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!
``` 

on two nvidia T4. Latest version 5.12.1 , torch 2.12! 


### Who can help?

@molbap (this is @ArthurZucker's brother's first issue 😉 )

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python 
from transformers import pipeline
pipe = pipeline("object-detection", model="hustvl/yolos-base", device_map="auto")
print(pipe('http://images.cocodataset.org/val2017/000000039769.jpg'))
``` 

### Expected behavior

Works. 

Comments:
I investigated this and was able to trace the device mismatch.

The issue appears to be that `mid_position_embeddings` is stored as a parameter on `YolosEncoder`, while `device_map="auto"` can place individual `YolosLayer`s on different GPUs. During the encoder loop, `hidden_states` follows the layer device, but `interpolated_mid_position_embeddings` remains on the parameter's original device, causing the reported CUDA device mismatch.

From what I found, this looks consistent with several existing model-parallel fixes in the repository (e.g. DINOv2, AltCLIP, BigBird-Pegasus), where tensors derived from parameters are moved to `hidden_states.device` before being combined.

I also checked the regression strategy. `ModelTesterMixin.test_model_parallelism` already exercises this code path for YOLOS on multi-GPU CI, so it looks like the existing test infrastructure should cover this once the fix is applied.

I'll prepare a PR unless there's a preferred alternative approach.


Congrats on your first issue @Carotterap   Introduced a fix. I recommend switching to a more recent model though, such as RF-DETR detection models, that will perform better and be more efficient.