How to solve the problem of `RuntimeError: all tensors must be on devices[0]`

kaiyuyue: <p>code:</p>
<pre><code class="lang-python">for i, (input, target) in enumerate(test_loader):
	target = target.cuda(async=True) # in test loader, pin_memory = True
	input_var = torch.autograd.Variable(input, volatile=False)

	# (Batch_Size, 10L, 3L, 32L, 224L, 224L)
	b, s, c, t, h, w = input_var.size()
	# view in (Batch_Size * 10L, 3L, 32L, 224L, 224L)
	input_var = input_var.view(-1, c, t, h, w)
	# forward
	output = model(input_var)
	# split in (Batch_Size, 10L, 400L)
	output = output.view(b, s, args.num_classes)
	# softmax
	scores = torch.sum(F.softmax(output, dim=2), dim=1, keepdim=False)
	# in-place average scores
	scores, indices = scores.div_(10).sort(dim=1, descending=True)
</code></pre>
<p>error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "inference.py", line 301, in &lt;module&gt;
    main()
  File "inference.py", line 164, in main
    inference(test_loader, model)
  File "inference.py", line 195, in inference
    output = model(input_var)
  File "/home/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 73, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 83, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py", line 67, in parallel_apply
    raise output
RuntimeError: all tensors must be on devices[0]
</code></pre>
<p>Thanks.</p>

albanD: <p>Hi,</p>
<p>When using <code>DataParallel</code>, with a list of gpu like <code>[dev0, dev1, ...]</code>, all the inputs that you give to the module have to be on <code>dev0</code>.<br>
You want to make sure that the current device when you send your data to the gpu is <code>dev0</code>.</p>

kaiyuyue: <p>how to manually send data to the gou <code>dev0</code> ?</p>

albanD: <p>Either set the current device manually <code>torch.cuda.set_device(dev0)</code> or use the context manager as follow to not change the current device globally:</p>
<pre><code class="lang-auto">with torch.cuda.device(dev0):
    t = t.cuda()
</code></pre>

kaiyuyue: <p>Is this right? I still encounter the error.</p>
<pre><code class="lang-python">model = torch.nn.DataParallel(model.cuda(), device_ids=[0,1,2,3])

for i, (input, target) in enumerate(test_loader):
        with torch.cuda.device(0):
	        target = target.cuda(async=True) # in test loader, pin_memory = True
	        input_var = torch.autograd.Variable(input, volatile=False)
</code></pre>

albanD: <p>For the <code>target</code> yes, is your <code>input</code> already on the gpu?</p>

kaiyuyue: <p>I ref <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py#L190" rel="nofollow noopener">this</a> code which directly convert input from <code>FloatTensor</code> to <code>Variable</code>, not from <code>CudaFloatTensor</code>. So should I write code like this:</p>
<pre><code class="lang-python">for i, (input, target) in enumerate(test_loader):
        with torch.cuda.device(0):
	        target = target.cuda(async=True) # in test loader, pin_memory = True
	        input = input.cuda(async=True)

        target_var = torch.autograd.Variable(target, volatile=False)
        input_var = torch.autograd.Variable(input, volatile=False)

</code></pre>

albanD: <p>Yes, it should look like this !</p>

xu_wang: <p>why do all the inputs have to be on dev0?</p>

albanD: <p>That’s a convention used by <code>DataParallel</code>. It performs the split and copy to other device by himself. The user does not know how the data will be splitted, so you cannot copy the data to the other devices in advance.</p>

xu_wang: <p>Thank you very much for your answer!</p>

xu_wang: <p>Hello,I meet a question about DataParallel.I want to use the GPU on the server,which ID is 0 and 3.</p>
<p>Traceback (most recent call last):<br>
File “Train_gpu.py”, line 67, in <br>
train()<br>
File “Train_gpu.py”, line 19, in train<br>
model=DataParallel(model,device_ids=[0,3]).to(calculator)<br>
File “/home/wangxu/miniconda3/envs/pt2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py”, line 102, in <strong>init</strong><br>
_check_balance(self.device_ids)<br>
File “/home/wangxu/miniconda3/envs/pt2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py”, line 17, in _check_balance<br>
dev_props = [torch.cuda.get_device_properties(i) for i in device_ids]<br>
File “/home/wangxu/miniconda3/envs/pt2/lib/python2.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 292, in get_device_properties<br>
raise AssertionError(“Invalid device id”)<br>
AssertionError: Invalid device id</p>
<p>And, calculator = device(‘cuda’)<br>
When I change device_ids to [0],the code work.<br>
So I am still confused about how to use DataParallel.The offical example on github doesn’t help.</p>

kaiyuyue: <p>Hi, in your bash, your command should like:</p>
<pre><code class="lang-bash">CUDA_VISIBLE_DEVICES=0,3 python train.py
</code></pre>
<p>and in your <code>train.py</code>, the gpus config should be set <code>[0,1]</code>.</p>

xu_wang: <p>Thank you for your reply.<br>
I change the code by following  your advice and get a new error:</p>
<p>/home/wangxu/miniconda3/envs/pt2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py:24: UserWarning:<br>
There is an imbalance between your GPUs. You may want to exclude GPU 0 which<br>
has less than 75% of the memory or cores of GPU 1. You can do so by setting<br>
the device_ids argument to DataParallel, or by setting the CUDA_VISIBLE_DEVICES<br>
environment variable.<br>
warnings.warn(imbalance_warn.format(device_ids[min_pos], device_ids[max_pos]))<br>
Traceback (most recent call last):<br>
File “Train_gpu.py”, line 130, in <br>
train()<br>
File “Train_gpu.py”, line 95, in train<br>
loss1_1 = loss(out1_1 * mask_var, vecmap_var)<br>
File “/home/wangxu/miniconda3/envs/pt2/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 491, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/wangxu/miniconda3/envs/pt2/lib/python2.7/site-packages/torch/nn/modules/loss.py”, line 372, in forward<br>
return F.mse_loss(input, target, size_average=self.size_average, reduce=self.reduce)<br>
File “/home/wangxu/miniconda3/envs/pt2/lib/python2.7/site-packages/torch/nn/functional.py”, line 1569, in mse_loss<br>
input, target, size_average, reduce)<br>
File “/home/wangxu/miniconda3/envs/pt2/lib/python2.7/site-packages/torch/nn/functional.py”, line 1537, in _pointwise_loss<br>
return lambd_optimized(input, target, size_average, reduce)<br>
RuntimeError: after cudaLaunch in triple_chevron_launcher::launch(): out of memory</p>

ptrblck: <p>Is someone else using the GPU or does the server have different cards installed?<br>
Could you explain a bit about the setup.</p>
<p>Might be unrelated here, but sometimes the <code>device_ids</code> don’t match with what <code>nvidia_smi</code> claims.</p>

xu_wang: <p>I was watching the nvidia-smi per 3 seconds.It seems no one was using the GPU ID 0 &amp; 3 .<br>
On the server there are 8 GPU installed and they are all the same type 1080Ti.<br>
What else do you want to know?</p>

ptrblck: <p>Could you add <code>CUDA_DEVICE_ORDER=PCI_BUS_ID</code> before in front of your Python call?<br>
Just curious, if the device ids are differently assigned.</p>

ptrblck: <p>Something doesn’t seem right.<br>
The <code>_check_balance</code> method only checks for the GPU specs (total memory and multi processor count).<br>
So if all cards are 1080Tis the warning shouldn’t come.</p>
<p>In your first attempt to use <code>DataParallel</code> you used <code>device_ids=[0, 3]</code> and got “Invalid device id” back.</p>
<p>Are you sure you are working on the right server?</p>

xu_wang: <p>What do you mean by the right server?I mean that if I don’t use the right server,I should not sing in successfully.<br>
And I don’t konw what do the error and warning mean.<br>
By the way, I set the input like this:<br>
calculator = device(‘cuda’)<br>
input_var = data.to(calculator)</p>

xu_wang: <p>I have a test,using the offcial example/mnist.<br>
Change the code this way:</p>
<p>From:<br>
model = Net().to(device)<br>
To:<br>
model = Net()<br>
model=nn.DataParallel(model, device_ids=[0, 1]).to(device)</p>
<p>Then run :CUDA_VISIBLE_DEVICES=0,1 python main.py<br>
The same warning came out appear.</p>