DataParallel on Multi-GPU RunTimeError ('input' doesn't have the same device with 'weight'')

username: tachen-cs
cooked: <p>Hi, when I use DataParallel on multi-gpu, I got the following error message recently:</p>
<p>PS: all the codes run normally using single GPU.</p>
<blockquote>
<p>RuntimeError: Expected tensor for argument <span class="hashtag-raw">#1</span> ‘input’ to have the same device as tensor for argument <span class="hashtag-raw">#2</span> ‘weight’; but device 1 does not equal 0 (while checking arguments for cudnn_convolution)</p>
</blockquote>
<p>It seems my data are on cuda1 but network paramters are on cuda0.<br>
I google the error message and want to figure out what happened, but I cannot find a specific solution and still confused, so I post it and really appreciate for your help.</p>
<p>I want to train a custom encoder-decoder model namely <code>EncoderDecoder</code>, and it consist of multiple custom modules, e.g., <code>custom_module1</code>, <code>custom_module2</code>…, each module only have a <code>__init__</code>  and <code>forward</code> function.</p>
<p>and basically I use it like:</p>
<pre><code class="lang-auto">device = torch.device('cuda:0') if torch.cuda.is_available() else torch.device('cpu')
...
class Model:
    def __init__(self, config: Configuration, device: torch.device):
        super(Model, self).__init__()    
        ...
        self.encoder_decoder = EncoderDecoder(config, device).to(device)
        self.encoder_decoder = torch.nn.DataParallel(self.encoder_decoder) 
    
    def train_on_batch(self, ...):
        ...
        output = self.encoder_decoder(input)
</code></pre>
<p>I am using pytorch:1.3.</p>
<p>And I am not sure why this happened. so I check the devices used both outside and inside the models, in the outside (the class <code>Model</code>), the input data and the<code> self.encoder_decoder</code> are all on<code> cuda:0</code> (the device). And issue happened inside the model (<code>self.encoder_decoder</code>).</p>
<p>In the <code>custom_module1</code> of <code>self.encoder_decoder.</code></p>
<p>I checked the input data are:</p>
<blockquote>
<p>audio:  torch.Size([5, 1, 257, 376]) cuda:0 torch.float16<br>
audio:  torch.Size([5, 1, 257, 376]) cuda:1 torch.float16</p>
</blockquote>
<p>which should be correct right? data are both on cuda0 and cuda1.</p>
<p>But it seems conflict with the model weights, which obviously should be stored only on cuda0? I guess this is why this bug happened?</p>
<p>and I checked the device inside the EncoderDecoder(config, device):</p>
<blockquote>
<p>device in EncoderDecoder (an DataParallel model’s device):  cuda:0<br>
device in EncoderDecoder (an DataParallel model’s device):  cuda:0</p>
</blockquote>
<p>Is this why this bug happened? but I still don’t know how to handle it? Or, the previous information are normal and correct when using DataParallel? if then, I don’t know why this issue appears?</p>
<p>Thanks for your help!</p>

username: tachen-cs
cooked: <p>Here is the raw error message:</p>
<blockquote>
<p>Original Traceback (most recent call last):<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py”, line 60, in _worker<br>
output = module(*input, **kwargs)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 541, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/opt/conda/lib/python3.6/site-packages/apex/amp/_initialize.py”, line 197, in new_fwd<br>
**applier(kwargs, input_caster))<br>
File “/workspace/encoder-decoder/encoder_decoder.py”, line 29, in forward<br>
decodeded_carrier = self.encoder_module(carrier, msg, carrier_seqlen)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 541, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/workspace/encoder-decoder/encoder.py”, line 44, in forward<br>
encoded_audio = self.conv_layers(audio, lengths)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 541, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/workspace/encoder-decoder/mask_conv.py”, line 25, in forward<br>
x = module(x)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 541, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/workspace/encoder-decoder/glu.py”, line 22, in forward<br>
x1 = self.convx1a(padded_x)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 541, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/modules/conv.py”, line 345, in forward<br>
return self.conv2d_forward(input, self.weight)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/nn/modules/conv.py”, line 342, in conv2d_forward<br>
self.padding, self.dilation, self.groups)<br>
RuntimeError: Expected tensor for argument <span class="hashtag-raw">#1</span> ‘input’ to have the same device as tensor for argument <span class="hashtag-raw">#2</span> ‘weight’; but device 1 does not equal 0 (while checking arguments for cudnn_convolution)</p>
</blockquote>

username: ptrblck
cooked: <p>The code snippet looks alright (although you don’t need to wrap <code>encoder_decoder</code> in another class, I assume your code is more complicated).<br>
All submodules in <code>encoder_decoder</code> should be scattered to the GPUs during the forward pass.<br>
Did you push any parameter or tensor inside your custom modules to a specific GPU?<br>
This would break the automatic scattering, as you would be forcing this tensor to be e.g. on GPU0.</p>

username: tachen-cs
cooked: <p>Thanks for your kindly help!</p>
<p>So, you are meaning I shouldn’t force any tensor to a specific GPU device during the forward function? How about I just create some new tensors in the sub-modules and force them to be in the same device with the normal existing tensors (which should be in correct device by auto-scattering if I didn’t assign any device to them, right? )?</p>

username: ptrblck
cooked: <p>If you need to create tensors in the <code>forward</code> method, your suggestion of using the <code>.device</code> attribute of the input tensor or any parameter will work.</p>

username: tachen-cs
cooked: <p>Hey ptrblck,</p>
<p>I found a interesting bug in my code. I have a tensor namely <code>carrier_seqlen</code> , and the size equals with <code>batch-size</code>, e.g., when the<code> batch-size</code> equals 10, the size of <code>carrier_seqlen</code>  should be 10 too.</p>
<p>Here is the issue: when I print out the tensor <code>carrier_seqlen</code> both inside and outside my model <code>encoder-decoder</code>, the value on device 0 changes:</p>
<blockquote>
<p>carrier_seqlen outside encoder-decoder:  tensor([504, 385, 332, 324, 271, 271, 260, 245, 238, 169], device=‘cuda:0’,<br>
dtype=torch.int32) torch.Size([10]) cuda:0 torch.int32</p>
</blockquote>
<blockquote>
<p>carrier_seqlen in encoder-decoder:  tensor([257, 257, 257, 257, 257], device=‘cuda:0’, dtype=torch.int32) torch.Size([5]) cuda:0 torch.int32<br>
carrier_seqlen in encoder-decoder:  tensor([271, 260, 245, 238, 169], device=‘cuda:1’, dtype=torch.int32) torch.Size([5]) cuda:1 torch.int32</p>
</blockquote>
<p>I am not sure why this happened? I just pass this tensor inside the model and haven’t done any processing, the size is torch.Size([10]), and it should be divided into  torch.Size([5]) cause I have two gpus inside the model, right? But why the value changed?</p>
<p>Thanks for your help!</p>

username: ptrblck
cooked: <p>I cannot reproduce this issue locally with a dummy model.<br>
Could you post a code snippet to reproduce this issue?</p>