DataParallel: Arguments are located on different GPUs

username: Ian
cooked: <p>Hi, all,</p>
<p>I met following error when using parallel_model = nn.DataParallel(model). Running on one GPU is fine.</p>
<p>Traceback (most recent call last):<br>
File “main_m.py”, line 113, in <br>
train(epoch, train_batch_logger, train_loader)<br>
File “main_m.py”, line 47, in train<br>
end_point = parallel_model(sample)<br>
File “/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py”, line 143, in forward<br>
outputs = self.parallel_apply(replicas, inputs, kwargs)<br>
File “/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py”, line 153, in parallel_apply<br>
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])<br>
File “/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/parallel_apply.py”, line 83, in parallel_apply<br>
raise output<br>
RuntimeError: arguments are located on different GPUs at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:253</p>
<p>My pytorch version is v1.0. I checked all previous similar topics, none of them works for my issue. Any clue to solve this problem?</p>

username: ptrblck
cooked: <p>Do you use and <code>cuda()</code> or <code>to()</code> calls inside your <code>forward</code> method?<br>
Would it be possible to post a small executable code snippet so that we could debug this issue?</p>

username: Ian
cooked: <p>Thanks for your reply.</p>
<p>Yeah, I use data.to(device) for the input data in the forward().</p>
<p>I set the batch size as 24 and use 2 GPUs. I print label of each data in this batch: first label is on the device ‘cuda:0’, other 23 labels are in ‘cuda:1’, then I torch.cat() these labels to one tensor, it is in ‘cuda:1’ . And the output of the final layer is in ‘cuda:1’.</p>
<p>the error occurs at:  end_point, y = parallel_model(sample), end_point is the output of the final layer, y concatenated labels.</p>
<p>Traceback (most recent call last):<br>
File “main_m.py”, line 113, in <br>
train(epoch, train_batch_logger, train_loader)<br>
File “main_m.py”, line 47, in train<br>
end_point, y = parallel_model(sample)<br>
File “/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py”, line 143, in forward<br>
outputs = self.parallel_apply(replicas, inputs, kwargs)<br>
File “/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py”, line 153, in parallel_apply<br>
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])<br>
File “/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/parallel_apply.py”, line 83, in parallel_apply<br>
raise output<br>
RuntimeError: arguments are located on different GPUs at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:253</p>

username: ptrblck
cooked: <p><code>nn.DataParallel</code> will create replicas of your model and chunk the input in dim0 into smaller batches for each GPU. If you use <code>to(device)</code> in your <code>forward</code> method, you force this tensor to be on a specific device, which will throw this error.<br>
Usually you would push the input data to the default device in your training loop, not the <code>forward</code> method.<br>
Is there a specific reason, why you would use it in <code>forward</code>?</p>

username: Ian
cooked: <p>Thanks, the input data is saved as class object (Creates a data object from a python dictionary),  and they have various dimension in each key. When I generated data_loader using default collate_fn, I got error as<br>
Traceback (most recent call last):<br>
File “main_m.py”, line 118, in <br>
train(epoch, train_batch_logger, train_loader)<br>
File “main_m.py”, line 44, in train<br>
for i, sample in enumerate(train_loader):<br>
File “/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py”, line 615, in <strong>next</strong><br>
batch = self.collate_fn([self.dataset[i] for i in indices])<br>
File “/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py”, line 234, in default_collate<br>
raise TypeError((error_msg.format(type(batch[0]))))<br>
TypeError: batch must contain tensors, numbers, dicts or lists; found &lt;class ‘MyData.Data’&gt;</p>
<p>then I modified the collate_fn as following to get train_loader:<br>
train_loader = DataLoader(train_dataset, batch_size=24, shuffle=True,  collate_fn=lambda x:x)</p>
<p>I tried to push data to device in training loop using .cuda(), I got error:<br>
Traceback (most recent call last):<br>
File “main_m.py”, line 118, in <br>
train(epoch, train_batch_logger, train_loader)<br>
File “main_m.py”, line 46, in train<br>
sample = sample.cuda()<br>
AttributeError: ‘list’ object has no attribute ‘cuda’</p>
<p>So, I used data.to(device) in the forward() method of model as following to push each data to device:<br>
for i in range(len(sample)):<br>
data = sample[i].to(device)</p>
<p>Hopefully I explained problem clearly.  Any suggestions for this case ?</p>

username: ptrblck
cooked: <p>One possible approach would be to use the <code>device</code> of another parameter of your model. E.g. assuming your model contains another layer called <code>self.fc</code>:</p>
<pre><code class="lang-python">for i in range(len(sample)):
    data = sample[i].to(self.fc.weight.device)
</code></pre>
<p>Alternatively, you could also apply the for loop inside the training loop, but that might need some code changes.</p>

username: Ian
cooked: <p>Thanks, I used ‘data = sample[i].to(self.fc.weight.device)’ as you suggested, the error still occurs.</p>
<p>The following is a part of my code:</p>
<p><strong>define and parallel model:</strong><br>
device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)<br>
model = Net()<br>
parallel_model = nn.DataParallel(model).to(device)</p>
<p><strong>loop for training:</strong><br>
for i, sample in enumerate(train_loader):<br>
with autograd.detect_anomaly():<br>
optimizer.zero_grad()<br>
end_point, y = parallel_model(sample)</p>
<p><strong>model:</strong><br>
class Net(torch.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()</p>
<pre><code>    self. layer = layer(XXXXXXXX)
    self.conv2 = Conv2(XXXXXXXX)
    self.fc1 = torch.nn.Linear(16*512, 1024)

def forward(self, sample): 
    print(self.fc1.weight.device)

    for i in range(len(sample)):                  
        data = sample[i].to(self.fc1.weight.device)            
        print(data.y)

        x_1 = self. layer(data.x, data.add, data.fea)                                             
        x_2 = self.conv2(x_1)    
        x_2 = x_2.view(-1, self.fc1.weight.size(1))       
        x_2= self.fc1(x_2)

        if i == 0:       #  problem is here? how to return a y_1 in different GPUs
            y_1 = data.y
            end_point = x_2
        else:              
            y_1 = torch.cat((y_1, data.y),0)
            end_point = torch.cat((end_point, x_2),0)    

    return end_point, y_1
</code></pre>
<p>when print(self.fc1.weight.device), I can get (I am using 2 GPUs):<br>
cuda:0<br>
cuda:1</p>
<p>when print(data.y),  only one sample in cuda:0<br>
tensor([47], device=‘cuda:0’)<br>
tensor([47], device=‘cuda:1’)<br>
tensor([25], device=‘cuda:1’)<br>
tensor([1], device=‘cuda:1’)<br>
tensor([33], device=‘cuda:1’)<br>
tensor([31], device=‘cuda:1’)<br>
tensor([78], device=‘cuda:1’)<br>
tensor([64], device=‘cuda:1’)<br>
tensor([16], device=‘cuda:1’)<br>
tensor([65], device=‘cuda:1’)<br>
tensor([73], device=‘cuda:1’)<br>
tensor([29], device=‘cuda:1’)<br>
tensor([77], device=‘cuda:1’)</p>
<p>I guess the problem happens at torch.cat(), return results in different GPUs .</p>
<p>Do you have any suggestion? or whether I understand you rightl?</p>

username: hoangcuong2011
cooked: <p>I struggled a lot with this error in the last days. In the end I found using self.register_buffer fixes the problem (see here for a similar <a href="https://discuss.pytorch.org/t/index-select-with-dataparallel-arguments-are-located-on-different-gpus/10229/7">thread</a> that I learnt from).</p>
<p>In detail, let us assume we have a tensor <em>temp_tensor</em> which is <em>torch.ones(1, 2)</em>. To put <em>temp_tensor</em> on the right gpu under <em>parallel_model</em>, we need to do something like this:</p>
<p><code>self.register_buffer("temp_tensor", torch.ones(1, 2))</code></p>
<p>Hope it helps!</p>