Qwen2.5-VL Sharding error when using Tensor Parallelism
Author: rupert404
Pull request: no
Labels: name: bug

### System Info

- transformers version: 4.53.2
- Platform: Linux-6.11.0-29-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 0.33.2
- Safetensors version: 0.5.3
- Accelerate version: 1.8.1
- PyTorch version: 2.7.1+cu128 (CUDA)
- Using distributed or parallel set-up in script?: Yes, Tensor Parallelism via PyTorch Elastic (torchrun)
- Using GPU in script?: Yes
- Tested GPU types: 
     - 1x/2x/4xA100
     - 1xRTX4090
     - 1x5090

### Who can help?

@zucchini-nlp @jla524 @VladOS95-cyber

(Tagging because of related commit-history)

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

When adapting the [example script for distributed GPU inference from the documentation](https://huggingface.co/docs/transformers/perf_infer_gpu_multi) using tensor parallelism for the Qwen-2.5-VL-family, the following errors arise.
```
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/distributed/tensor/_sharding_prop.py", line 447, in propagate_op_sharding_non_cached
[rank0]:     output_sharding = sharding_prop_func(op_schema)
[rank0]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/distributed/tensor/_ops/_conv_ops.py", line 29, in convolution_rules
[rank0]:     assert isinstance(bias_spec, DTensorSpec)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: AssertionError
```
from this exception (callstack abridged):
```
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/distributed/tensor/_sharding_prop.py", line 451, in propagate_op_sharding_non_cached
[rank0]:     raise RuntimeError(
[rank0]: RuntimeError: Sharding propagation failed on op Op(op=aten.convolution.default, args_schema=Spec(R on (1064, 3, 2, 14, 14)), Spec(R on (1280, 3, 2, 14, 14)), None, [2, 14, 14], [0, 0, 0], [1, 1, 1], False, [0, 0, 0], 1 @ mesh: (1,)).
```
This seems independent of the number of processes specified.

Minimal example:

````
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor, AutoTokenizer
from qwen_vl_utils import process_vision_info

model_id = "Qwen/Qwen2.5-VL-3B-Instruct"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(model_id, tp_plan="auto")
processor = AutoProcessor.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
msgs = [{"role": "user","content": [
    {"type": "image", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"},
    {"type": "text", "text": "What is shown in this picture?"}]}]
text = processor.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(msgs)
inputs = processor(text=[text], images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt")
inputs.to(model.device)
model.generate(**inputs)
````

Ran with `torchrun --nproc-per-node=1 test.py`
Same issue persists when actually using 2 or 4 GPUs and specifying `--nproc-per-node=2` or `--nproc-per-node=4`.

The issue seems to be with the sharding of the `conv3d-layers` and reproducible every time. I can also provide callstacks, if preferred (not sure because of the issue template). 



When removing the image input from `msgs`, the issue seems to be mixing `Tensors` and `DTensors` instead. Not sure if this is related, but might be a similar issue.

I've tried to adapt the `base_model_tp_plan` that is [specified in transformers for these models](https://github.com/huggingface/transformers/blob/903944a411c35b8f7b2b51ada77a7c2a80e7fb88/src/transformers/models/qwen2_vl/configuration_qwen2_vl.py#L173) myself (and rebuilding transformers with that). 
To my understanding it might not be wanted to shard the ViT - which probably is not the issue here, but I'm kinda lost at this point.

This issue seems to also exist for the Qwen-2-VL-family, which the `base_model_tp_plan` is derived from.

### Expected behavior

Normally, the outputs would be generated.

Comments:
Adding callstacks for completeness.
<details>
<summary>Traceback with Image</summary>

```
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/distributed/tensor/_sharding_prop.py", line 447, in propagate_op_sharding_non_cached
[rank0]:     output_sharding = sharding_prop_func(op_schema)
[rank0]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/distributed/tensor/_ops/_conv_ops.py", line 29, in convolution_rules
[rank0]:     assert isinstance(bias_spec, DTensorSpec)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: AssertionError

[rank0]: The above exception was the direct cause of the following exception:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/rupert/qwen2.5-vl-test/qwen2.5-vl-issue.py", line 15, in <module>
[rank0]:     model.generate(**inputs)
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2625, in generate
[rank0]:     result = self._sample(
[rank0]:              ^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 3606, in _sample
[rank0]:     outputs = self(**model_inputs, return_dict=True)
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 943, in wrapper
[rank0]:     output = func(self, *args, **kwargs)
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1509, in forward
[rank0]:     outputs = self.model(
[rank0]:               ^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1250, in forward
[rank0]:     image_embeds = self.get_image_features(pixel_values, image_grid_thw)
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1200, in get_image_features
[rank0]:     image_embeds = self.visual(pixel_values, grid_thw=image_grid_thw)
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 455, in forward
[rank0]:     hidden_states = self.patch_embed(hidden_states)
[rank0]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 87, in forward
[rank0]:     hidden_states = self.proj(hidden_states.to(dtype=target_dtype)).view(-1, self.embed_dim)
[rank0]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1857, in _call_impl
[rank0]:     return inner()
[rank0]:            ^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1805, in inner
[rank0]:     result = forward_call(*args, **kwargs)
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 725, in forward
[rank0]:     return self._conv_forward(input, self.weight, self.bias)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 720, in _conv_forward
[rank0]:     return F.conv3d(
[rank0]:            ^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/_compile.py", line 51, in inner
[rank0]:     return disable_fn(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
[rank0]:     return fn(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/distributed/tensor/_api.py", line 344, in __torch_dispatch__
[rank0]:     return DTensor._op_dispatcher.dispatch(
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/distributed/tensor/_dispatch.py", line 164, in dispatch
[rank0]:     return self._custom_op_handlers[op_call](op_call, args, kwargs)  # type: ignore[operator]
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/distributed/tensor/_tp_conv.py", line 239, in convolution_handler
[rank0]:     dtensor.DTensor._op_dispatcher.sharding_propagator.propagate(op_info)
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/distributed/tensor/_sharding_prop.py", line 264, in propagate
[rank0]:     OutputSharding, self.propagate_op_sharding(op_info.schema)
[rank0]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/distributed/tensor/_sharding_prop.py", line 45, in __call__
[rank0]:     return self.cache(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/distributed/tensor/_sharding_prop.py", line 451, in propagate_op_sharding_non_cached
[rank0]:     raise RuntimeError(
[rank0]: RuntimeError: Sharding propagation failed on op Op(op=aten.convolution.default, args_schema=Spec(R on (48, 3, 2, 14, 14)), Spec(R on (1280, 3, 2, 14, 14)), None, [2, 14, 14], [0, 0, 0], [1, 1, 1], False, [0, 0, 0], 1 @ mesh: (1,)).
```

</details>

<details>
<summary>Traceback without Image</summary>

```
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/rupert/qwen2.5-vl-test/qwen2.5-vl-issue.py", line 15, in <module>
[rank0]:     model.generate(**inputs)
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2625, in generate
[rank0]:     result = self._sample(
[rank0]:              ^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 3606, in _sample
[rank0]:     outputs = self(**model_inputs, return_dict=True)
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 943, in wrapper
[rank0]:     output = func(self, *args, **kwargs)
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1529, in forward
[rank0]:     logits = self.lm_head(hidden_states)
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/nn/modules/linear.py", line 125, in forward
[rank0]:     return F.linear(input, self.weight, self.bias)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/_compile.py", line 51, in inner
[rank0]:     return disable_fn(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
[rank0]:     return fn(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/distributed/tensor/_api.py", line 344, in __torch_dispatch__
[rank0]:     return DTensor._op_dispatcher.dispatch(
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/distributed/tensor/_dispatch.py", line 167, in dispatch
[rank0]:     op_info = self.unwrap_to_op_info(op_call, args, kwargs)
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/distributed/tensor/_dispatch.py", line 366, in unwrap_to_op_info
[rank0]:     self._try_replicate_spec_for_scalar_tensor(
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.12/site-packages/torch/distributed/tensor/_dispatch.py", line 468, in _try_replicate_spec_for_scalar_tensor
[rank0]:     raise RuntimeError(
[rank0]: RuntimeError: aten.mm.default: got mixed torch.Tensor and DTensor, need to convert all torch.Tensor to DTensor before calling distributed operators!
```

</details>

Again, not sure if both problems are of the same nature but seem related.

I just encountered exactly the sample problem when running exactly the same model using exactly the same demo script...
Still trying to figure a way out to fix this.

same problem

Did it work with prev versions, if anyone knows which version had no issues? I will take a look later, and since we haven't seen reports on this earlier I think this should be a regression 👀  

Edit: the last working version was v4.51.3, feel free to downgrade until it is fixed

> Did it work with prev versions, if anyone knows which version had no issues? I will take a look later, and since we haven't seen reports on this earlier I think this should be a regression 👀
> 
> Edit: the last working version was v4.51.3, feel free to downgrade until it is fixed

Thanks, downgrading helps!

I found the commit that caused issues, https://github.com/huggingface/transformers/pull/37877. Specifically, using `replicate` plan on the Conv3D layer throws errors, and commenting out "to-from-DTensor" in replicate layer works. I don't know why yet. cc @NouamaneTazi , can you take a look?

Thank you so much for looking into it!

Downgrading partially fixed the issue for me. It works fine now if I specify `--nprocs-per-node=1`, but when actually using multiple GPUs (2/4) and sharding, I still encounter issues.

Although they differ from the original ones in this issue, they are still related to the specified forward functions. Not the propagation though.
They are also different if I specify 2 or 4 GPUs, however, reproducible in both cases.

Sadly I only have very limited access to the GPU cluster of my university to debug (and my understanding of the issue is only on a conceptual basis).

If helpful, the callstacks are as follows:
<details>
<summary>Traceback @ 2xA100</summary>

```
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/rupert/qwen2.5-vl-test/qwen2_5_vl_issue.py", line 15, in <module>
[rank0]:     model.generate(**inputs)
[rank0]:                   ^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/transformers/generation/utils.py", line 2465, in generate
[rank0]:     result = self._sample(
[rank0]:              ^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/transformers/generation/utils.py", line 3431, in _sample
[rank0]:     outputs = self(**model_inputs, return_dict=True)
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1825, in forward
[rank0]:     outputs = self.model(
[rank0]:               ^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1171, in forward
[rank0]:     layer_outputs = decoder_layer(
[rank0]:                     ^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1033, in forward
[rank0]:     hidden_states, self_attn_weights, present_key_value = self.self_attn(
[rank0]:                                                           ^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 964, in forward
[rank0]:     attn_output = attn_output.view(bsz, q_len, self.hidden_size)
[rank0]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: RuntimeError: shape '[1, 40, 2048]' is invalid for input of size 40960
```

</details>

<details>
<summary>Traceback @ 4xA100</summary>

```
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/rupert/qwen2.5-vl-test/qwen2_5_vl_issue.py", line 15, in <module>
[rank0]:    model.generate(**inputs)
[rank0]:                   ^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1825, in forward
[rank0]:     outputs = self.model(
[rank0]:               ^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1171, in forward
[rank0]:     layer_outputs = decoder_layer(
[rank0]:                     ^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1033, in forward
[rank0]:     hidden_states, self_attn_weights, present_key_value = self.self_attn(
[rank0]:                                                           ^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/rupert/qwen2.5-vl-test/.venv/lib/python3.11/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 923, in forward
[rank0]:     key_states = key_states.view(bsz, q_len, -1, self.head_dim).transpose(1, 2)
[rank0]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: RuntimeError: shape '[1, 40, -1, 128]' is invalid for input of size 2560
```

</details>

Could anyone please confirm if the same is happening on their side? (Or if I'm just stupid)


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

not stale

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@rupert404 cc: @zucchini-nlp 

It looks like upgrading to `transformers==4.56.2` resolves the error you encountered and correctly generates responses.

System Info:
- `transformers` version: 4.56.2
- Platform: Linux-5.15.0-142-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.35.1
- Safetensors version: 0.6.2
- Accelerate version: 1.10.1
- Accelerate config:    not found
- DeepSpeed version: 0.17.6
- PyTorch version (accelerator?): 2.8.0+cu129 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: Yes
- Using GPU in script?: Yes
- GPU type: NVIDIA RTX 5000 Ada Generation

```py
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor, AutoTokenizer
from qwen_vl_utils import process_vision_info

model_id = "Qwen/Qwen2.5-VL-3B-Instruct"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(model_id, tp_plan="auto")
processor = AutoProcessor.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
msgs = [{"role": "user","content": [
        {"type": "image", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"},
            {"type": "text", "text": "What is shown in this picture?"}]}]
text = processor.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(msgs)
inputs = processor(text=[text], images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt")
inputs.to(model.device)
outputs = model.generate(**inputs)
print(outputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
 
```sh
torchrun --nproc-per-node=1 test.py 
```

```sh
tensor([[151644,   8948,    198,  ...,  11699,    389,    279]],
       device='cuda:0')
system
You are a helpful assistant.
user
What is shown in this picture?
assistant
The picture shows a person and a dog on a beach at sunset. The person is sitting on the
```

When I downgrade to `transformers==4.53.2`, the script raises a similar error to yours:

```sh
[rank0]: RuntimeError: Sharding propagation failed on op Op(op=aten.convolution.default, args_schema=Spec(R on (14308, 3, 2, 14, 14)), Spec(R on (1280, 3, 2, 14, 14)), None, [2, 14, 14], [0, 0, 0], [1, 1, 1], False, [0, 0, 0], 1 @ mesh: (1,)).
```

@rupert404 cc: @zucchini-nlp 

I also successfully ran it using 2 GPUs with the `torchrun --nproc-per-node=2` command. I used a Kaggle T4 x2 environment.

System Info:
- `transformers` version: 4.56.2
- Platform: Linux-6.6.56+-x86_64-with-glibc2.35
- Python version: 3.11.13
- Huggingface_hub version: 0.35.1
- Safetensors version: 0.5.3
- Accelerate version: 1.8.1
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.6.0+cu124 (CUDA)
- Tensorflow version (GPU?): 2.18.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.10.6 (gpu)
- Jax version: 0.5.2
- JaxLib version: 0.5.1
- Using distributed or parallel set-up in script?: Yes
- Using GPU in script?: Yes
- GPU type: Tesla T4

Kaggle notebook link: https://www.kaggle.com/code/shutotakahashi/t4x2-qwen-qwen2-5-vl-3b-instruct?scriptVersionId=264252752

@rupert404 cc: @zucchini-nlp 

It looks like the reason `4.56.2` works fine is that the TP plan for `Conv3d` was changed from `replicate` to nothing after PR #39501. Before this PR, the TP plan was explicitly set to `replicate` if no TP plan was specified in the model config.

https://github.com/huggingface/transformers/pull/39501/files#diff-2fa9093556eb859380534ee2d5af9bfd0710ff73f84b69e937cb8c54c337e38fL904

<img width="1210" height="324" alt="Image" src="https://github.com/user-attachments/assets/caef38b4-66e5-421c-a711-57ceb9022603" />

I inspected the applied TP plan for `Conv3d` using the code snippet below:

```py
from transformers import Qwen2_5_VLForConditionalGeneration

def inspect_module_tp_plans(model):
    """Check what TP plan is applied to each module"""
    print("Module TP Plans:")
    print("=" * 60)
    for name, module in model.named_modules():
        # Conv3d is named as `model.visual.patch_embed.proj` in Qwen2_5_VLForConditionalGeneration
        if name == "model.visual.patch_embed.proj":
            if hasattr(module, "_hf_tp_plan"):
                _hf_tp_plan = module._hf_tp_plan
            else:
                _hf_tp_plan = "No _hf_tp_plan attribute"
            print(f"{name}: {_hf_tp_plan}")

# model_id = "Qwen/Qwen2.5-VL-3B-Instruct"
model_id = "Qwen/Qwen2.5-VL-3B-Instruct-AWQ"  # Used AWQ quantized model for lower memory consumption, but TP plan should be the same as the non-quantized model
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(model_id, tp_plan="auto")
inspect_module_tp_plans(model)
```

```sh
torchrun --nproc-per-node=1 test.py 
```

**Before PR #39501** (commit: a44dcbe513e3e073271e0b8e369b75aca51affae):
```sh
Module TP Plans:
============================================================
model.visual.patch_embed.proj: replicate
```

**After PR #39501** (commit: c50f140be277642b2c1777455e71c633c253bfae. I used this commit because the PR #39501 commit raises an error for my code snippet):
```sh
Module TP Plans:
============================================================
model.visual.patch_embed.proj: No _hf_tp_plan attribute
```

<img width="796" height="196" alt="Image" src="https://github.com/user-attachments/assets/143cfe3d-d670-4ee1-8052-f39802c64bed" />

Niiice, thanks for verifying!

@zucchini-nlp 

Is it worth adding a process and test to verify that the `replicate` TP plan isn't applied to Conv3D modules to prevent issues like this from happening again? If so, I'm happy to create a PR. Though, it might be better to understand why the `replicate` TP plan cannot be applied to Conv3D modules in the first place.