multi-GPU model parallelism: device error

cs123951: <p>Hi,<br>
My model is too large to fit into one GPU, so I split it into two GPUs. Part of the models are as follows:</p>
<pre><code class="lang-auto">cur_layer_input = []
for t in range(seq_len):
    d1 = self.down1.cuda(self.device_ids[0])(img_seq_ring[:, t]).cuda(self.device_ids[0])
   d2 = self.down2.cuda(self.device_ids[0])(d1)
   cur_layer_input.append(d2.cuda(self.device_ids[1]))
</code></pre>
<p>I used DDP to wrap this model:</p>
<pre><code class="lang-auto">from torch.nn.parallel import DistributedDataParallel as DDP
model = DDP(model, device_ids=[device_ids[0]], find_unused_parameters=True).
</code></pre>
<p>However, errors occured:</p>
<pre><code class="lang-auto">  File "xxx.py", line 141, in train_model
    loss.backward()
  File "/home/anaconda3/envs/VIBE/lib/python3.7/site-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/anaconda3/envs/VIBE/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: grad.device() == bucket_view.device() INTERNAL ASSERT FAILED at /pytorch/torch/csrc/distributed/c10d/reducer.cpp:206, please report a bug to PyTorch. 
</code></pre>
<p>I do not know what bucket_view.device() is. So I do not know how to solve this problem.<br>
Could you help me?</p>
<p>Thank you very much.</p>

agolynski: <p>Hi,</p>
<p>I am not sure how exactly you are trying to use DDP to split your model between devices: DDP usecase is to parallelize training. Could you take a look at model parallel tutorial instead? (you can also combine it with DDP once you set it up), e.g.<br>
<a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html</a><br>
it requires you to move inputs between GPUs, not just the models e.g.</p>
<pre><code class="lang-auto">x = self.relu(self.net1(x.to('cuda:0'))) 
return self.net2(x.to('cuda:1'))
</code></pre>

cs123951: <p>Yes, I have read the tutorial several times and I have moved inputs to the same device.</p>
<p>I have made some modifications so that the model can be fitted into one GPU for testing.<br>
The question is, when I used a single GPU for data distributed parallel, it worked well.<br>
But when I moved part of the model to another GPU, the error occurred.</p>
<p>I used a BiRNN model. The code is as follows:</p>
<pre><code class="lang-auto">def forward(self, img):
	self.reverse_net = self.reverse_net.cuda(self.device_ids[1])
	img_ring = torch.cat([img, img[:,0].unsqueeze(1)], dim=1).cuda(self.device_ids[1])
	seq_len = img_ring.shape[1]  

	cur_layer_input = []
	cur_layer_input_rev = []

	for t in range(seq_len):
		d2 = F.interpolate(img_ring[:, t], scale_factor=1 / 4, mode="trilinear", align_corners=True)
		cur_layer_input.append(d2.cuda(self.device_ids[0]))
		cur_layer_input_rev.append(d2.cuda(self.device_ids[1]))

	cur_layer_input_rev = cur_layer_input_rev[::-1] 
	y_out_fwd = self.forward_net(cur_layer_input)  # forward RNN
	y_out_rev = self.reverse_net(cur_layer_input_rev)  # backward RNN

	y_out_fwd = torch.stack(y_out_fwd, dim=0)
	y_out_rev = torch.stack(y_out_rev, dim=0)  
	y_out_rev = torch.flip(y_out_rev, dims=[1])

	ycat = torch.cat((y_out_fwd.cuda(self.device_ids[1]), y_out_rev), dim=2)

	disp_list = []
	for t in range(1, seq_len-1):
		disp_list.append(self.outconv3.cuda(self.device_ids[1])(ycat[t])) 

	for t in range(len(disp_list)):
		disp_list[t] = F.interpolate(disp_list[t], scale_factor=4, mode="trilinear", align_corners=True)
	return disp_list
</code></pre>
<p>The error information is:</p>
<pre><code class="lang-auto">  File "xxx.py", line 141, in train_model
    loss.backward()
  File "/home/anaconda3/envs/VIBE/lib/python3.7/site-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/anaconda3/envs/VIBE/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: grad.device() == bucket_view.device() INTERNAL ASSERT FAILED at /pytorch/torch/csrc/distributed/c10d/reducer.cpp:206, please report a bug to PyTorch. 
</code></pre>
<p>So I want to know what is bucket_view and what may cause the above errors.</p>
<p>It is weird. I can train using one GPU. But when I moved part of the model to another GPU, errors occurred. The only changed thing is the device.</p>

agolynski: <p>cc <a class="mention" href="/u/yanli_zhao">@Yanli_Zhao</a> who worked on bucket code. Yanli, could you help here?</p>

Yanli_Zhao: <p><a class="mention" href="/u/cs123951">@cs123951</a> for multi-device module, you should not pass device_ids while wrapping your module using DDP.</p>
<p>Try this:</p>
<p>DDP(model,  find_unused_parameters=True)</p>

Yanli_Zhao: <p>or if it is single-device module, just call torch.cuda.set_device(rank) before wrapping DDP, pass device_ids=[rank]</p>

cs123951: <p>Thank you for your kind advice. I tried your advice, but problems still happened.</p>
<p>I created two files in my github(<a href="https://github.com/cs123951/temp_public_files" rel="noopener nofollow ugc">https://github.com/cs123951/temp_public_files</a>)  to explain my problem.</p>
<p>I constructed a bidirectional RNN model using model parallelism like this.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/b/ab9307c28274fea454cf0b130a2457c5dc70d8bf.png" data-download-href="https://discuss.pytorch.org/uploads/default/ab9307c28274fea454cf0b130a2457c5dc70d8bf" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/ab9307c28274fea454cf0b130a2457c5dc70d8bf_2_690x294.png" alt="image" data-base62-sha1="otOEP1WdZe0wG1MHlczC6e5OKTR" width="690" height="294" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/ab9307c28274fea454cf0b130a2457c5dc70d8bf_2_690x294.png, https://discuss.pytorch.org/uploads/default/original/3X/a/b/ab9307c28274fea454cf0b130a2457c5dc70d8bf.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/b/ab9307c28274fea454cf0b130a2457c5dc70d8bf.png 2x" data-dominant-color="E3ECDE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1034×442 28.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The difference is that the RNN is bidirectional so I made some modifications.</p>
<p>I distributed the GPU-0 to the first several layers, and the GPU-1 to the last layer.</p>
<p>The forward process is normal. But when it comes to the loss.backward(), error will happen:</p>
<pre><code class="lang-auto">RuntimeError: grad.device() == bucket_view.device() INTERNAL ASSERT FAILED at /pytorch/torch/csrc/distributed/c10d/reducer.cpp:206, please report a bug to PyTorch. 
</code></pre>
<p>You can reproduce my results using the simple_bug_file2.py in <a href="https://github.com/cs123951/temp_public_files/blob/main/simple_bug_file2.py" rel="noopener nofollow ugc">my github</a> by the following command:<br>
<code>python -m torch.distributed.launch --nproc_per_node=2 --nnodes=1 --node_rank=0 --master_addr=xxx.xxx.xx.xx --master_port=31234 simple_bug_file2.py</code></p>
<p>Don’t forget to change the master address in the command and the visible GPUs in the file.</p>
<p>My environment is</p>
<pre><code class="lang-auto">PyTorch version: 1.4.0
Is debug build: False
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A
GPU: GeForce GTX 1080 Ti

OS: Ubuntu 18.04.3 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.3.2

Python version: 3.7 (64-bit runtime)
Nvidia driver version: 455.28
HIP runtime version: N/A
MIOpen runtime version: N/A

pip 21.0.1
</code></pre>
<p>By the way, when I changed <code>down_device = self.device_ids[1]</code> into <code>down_device = self.device_ids[0]</code>. It performs well.</p>
<p>I guess maybe bucket_view is related to the unroll operation of the RNN. But I am not familiar with it. Could you give me some more advice?</p>
<p>Thank you very much.</p>

Yanli_Zhao: <p>bucket_view is a copy of parameter’s grad. ‘bucket_view.device=param.device’ was set before training loop starts.</p>
<p>‘bucket_view.device != param.grad.device’ means grad of this param changed device during training loop.</p>
<p>Would you please confirm whether param/grad device changed during training loop?</p>

cs123951: <p>hi, thank you for your explanation.</p>
<p>I have checked the logic of the model and I am sure that it is fine. Now that the forward pass works well, I can not figure out why there is something wrong with the backward pass.</p>
<p>The error occurs in the file reducer.cpp. I have no idea how to debug the C++ file when I was running a Python program.</p>
<p>Could you give me some advice on how to debug and checking the param/grad device for loss.backward() during the training loop?</p>
<p>Thank you very much!</p>

cs123951: <p>Hello, I have figured out what mistakes I take!<br>
I make mistakes in the <strong>init</strong>() of the model like this:</p>
<pre><code class="lang-auto">mp_model = ToyMpModel(devices_list).cuda(devices_list[0])
ddp_mp_model = DDP(mp_model, device_ids=[0], find_unused_parameters=True)
</code></pre>
<p>I used multi-GPU model, but when I created model, I gave the model only one GPU.<br>
Thus <code>self.is_multi_device_module = len({p.device for p in module.parameters()}) &gt; 1</code> is False in the distributed.py file.<br>
I changed the GPU in the forward function. Thus triggering the error.</p>
<p>The solution is:<br>
I defined the GPU in the <strong>init</strong>() function of the model. And then I changed the call function to</p>
<pre><code class="lang-auto">mp_model = ToyMpModel(devices_list)
ddp_mp_model = DDP(mp_model, device_ids=[], output_device=[], find_unused_parameters=True)
</code></pre>
<p>I hope my mistakes could bring some warnings to the latercommers.<br>
I feel sorry for the confusion caused to you.</p>
<p>This problem can be closed.</p>