distrubutedDataParallel and dataParallel hangs in specified model

username: damaoooo
cooked: <p>I was trying to train my NLP model in multGPU with 2 <code>K80s</code>, each <code>K80</code> has 2 cores, my model works fine in CPU or single GPU with <code>DataParallel</code> or <code>distributedDataParallel</code>, but when I use 2 or more cores, embarrassing things happened, it hangs always,this is the symptom</p>
<blockquote>
<p>DataParallel</p>
</blockquote>
<pre data-code-wrap="python"><code class="lang-python">clothModel = myModel.cuda()
clothModel = nn.DataParallel(clothModel) # &lt;-- it works fine
······
out, loss = clothModel(input) #  &lt;-- program always hang in this line, even I can't use ctrl+C to shut it down and I get this infomation by using VSCode debug 
</code></pre>
<p>when I turn to <code>nvidia-smi</code>, I found this</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.39       Driver Version: 418.39       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           On   | 00000000:08:00.0 Off |                  Off |
| N/A   45C    P0    70W / 149W |   2113MiB / 12206MiB |    100%   E. Process |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           On   | 00000000:09:00.0 Off |                  Off |
| N/A   35C    P0    70W / 149W |    322MiB / 12206MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           On   | 00000000:86:00.0 Off |                  Off |
| N/A   39C    P0    57W / 175W |    311MiB / 12206MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           On   | 00000000:87:00.0 Off |                  Off |
| N/A   31C    P0    71W / 175W |    320MiB / 12206MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
</code></pre>
<p>after a night, it remains this still</p>
<blockquote>
<p>distributedDataParallel</p>
</blockquote>
<p>after failed to use DataParallel, I turned to distributedDataParallel which is recommanded, and it hangs in <code>clothModel = nn.parallel.DistributedDataParallel(clothModel)</code><br>
I turn to <code>nvidia-smi</code>, it seems almost the same as when I use <code>nn.DataParallel</code><br>
and this time I can use <code>ctrl + C</code> while the processes are still remain in cuda so that the only way to exit is <code>kill -9 PID</code><br>
when I use <code>ctrl+c</code>, it displayed this</p>
<pre><code class="lang-auto">  File "/home/damaoooo/.conda/envs/test/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/damaoooo/.conda/envs/test/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/damaoooo/.conda/envs/test/lib/python3.6/site-packages/torch/distributed/launch.py", line 263, in &lt;module&gt;
    main()
  File "/home/damaoooo/.conda/envs/test/lib/python3.6/site-packages/torch/distributed/launch.py", line 256, in main
    process.wait()
  File "/home/damaoooo/.conda/envs/test/lib/python3.6/subprocess.py", line 1477, in wait
    (pid, sts) = self._try_wait(0)
  File "/home/damaoooo/.conda/envs/test/lib/python3.6/subprocess.py", line 1424, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)

</code></pre>
<p>and more interesting, I made a simply CNN for MNIST and turn to DataParallel or distributedDataParallel, it works perfect… I wonder is there something wrong with my <code>clothModel</code>？if there is, why I turn to single GPU, it works fine?<br>
and how can I solve the confusing hang?</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="damaoooo" data-post="1" data-topic="74322">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/damaoooo/48/22003_2.png" class="avatar"> damaoooo:</div>
<blockquote>
<p>and it hangs in <code>clothModel = nn.parallel.DistributedDataParallel(clothModel)</code></p>
</blockquote>
</aside>
<p>You mean DDP hangs at constructor? Can you attach the process to gdb and check the trace to see line is causing the hang?</p>
<p>Have you set <code>CUDA_VISIBLE_DEVICES</code> or pass in <code>device_ids</code> arg properly for DDP? Each DDP process should exclusively work on one GPU.</p>
<blockquote>
<p>I wonder is there something wrong with my  <code>clothModel</code>?</p>
</blockquote>
<p>Given the trace, I assume you are using the launch script. With that, DDP should be constructed in the following way:</p>
<pre data-code-wrap="python"><code class="lang-python">clothModel = DistributedDataParallel(clothModel, device_ids=[arg.local_rank], output_device=arg.local_rank)
</code></pre>

username: damaoooo
cooked: <p>thanks a lot! that’s the key to the question, after tried that, I successed to run, but How about the DataParallel hang in the first question?</p>

username: mrshenli
cooked: <p>Not sure why DataParallel stuck. The source code is <a href="https://github.com/pytorch/pytorch/blob/df8d6eeb19423848b20cd727bc4a728337b73829/torch/nn/parallel/data_parallel.py#L141-L156" rel="nofollow noopener">here</a>. Can you attach the process/threads to GDB and backtrace the stack?</p>