Distributed training hangs

amsword: <p>The training hangs without printing any logs. Observations/configurations:</p>
<ul>
<li>4 nodes. 4 GPU/node.</li>
<li>distributed training with each process taking 1 GPU.</li>
<li>pytorch version = 1.1; cuda version = 9.0; gpu driver version: 410.78</li>
<li>use the code base of facebook/maskrcnn-benchmark, but i thought it is just normal pytorch code.</li>
<li>GPU utility is close to 100%, but there is no more log.</li>
<li>it has finished 278K iterations, and then hangs there without any progress (no more snapshot, no more logs)</li>
<li>gdb attached to one of the process (sudo gdb -p process_id) and it seems like it hangs at cuMemcpyHtoDAsync_v2.</li>
</ul>
<pre><code class="lang-auto">(gdb) where
#0  0x00007ffe309e1b6d in clock_gettime ()
#1  0x00007f8cc536f876 in __GI___clock_gettime (clock_id=4, tp=0x7ffe30898660) at ../sysdeps/unix/clock_gettime.c:115
#2  0x00007f8c6c7ecc4e in ?? () from /usr/local/nvidia/lib64/libcuda.so.1
#3  0x00007f8c6c87b8d3 in ?? () from /usr/local/nvidia/lib64/libcuda.so.1
#4  0x00007f8c6c89b81f in ?? () from /usr/local/nvidia/lib64/libcuda.so.1
#5  0x00007f8c6c7c8737 in ?? () from /usr/local/nvidia/lib64/libcuda.so.1
#6  0x00007f8c6c6d9e4e in ?? () from /usr/local/nvidia/lib64/libcuda.so.1
#7  0x00007f8c6c6dbfc3 in ?? () from /usr/local/nvidia/lib64/libcuda.so.1
#8  0x00007f8c6c829c82 in cuMemcpyHtoDAsync_v2 () from /usr/local/nvidia/lib64/libcuda.so.1
#9  0x00007f8cbe7ad49c in ?? () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcudart-f7fdd8d7.so.9.0
#10 0x00007f8cbe78a573 in ?? () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcudart-f7fdd8d7.so.9.0
#11 0x00007f8cbe7c3d86 in cudaMemcpyAsync () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcudart-f7fdd8d7.so.9.0
#12 0x00007f8c836a9f4b in (anonymous namespace)::copy_from_cpu(at::Tensor&amp;, at::Tensor const&amp;) () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so
#13 0x00007f8c8374a875 in void (anonymous namespace)::_copy__cuda&lt;float&gt;(at::Tensor&amp;, at::Tensor const&amp;, bool) () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so
#14 0x00007f8c836aafb8 in at::native::_s_copy__cuda(at::Tensor&amp;, at::Tensor const&amp;, bool) () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so
#15 0x00007f8c826d47ef in at::CUDAType::s_copy_(at::Tensor&amp;, at::Tensor const&amp;, bool) const () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so
#16 0x00007f8c7764033d in at::native::copy_(at::Tensor&amp;, at::Tensor const&amp;, bool) () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2.so
#17 0x00007f8cbf546dc9 in torch::autograd::VariableType::copy_(at::Tensor&amp;, at::Tensor const&amp;, bool) const () from /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so.1
#18 0x00007f8c777829cc in at::native::to(at::Tensor const&amp;, c10::TensorOptions const&amp;, bool, bool) () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2.so
#19 0x00007f8c77a01857 in at::TypeDefault::to(at::Tensor const&amp;, c10::TensorOptions const&amp;, bool, bool) const () from /opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2.so
#20 0x00007f8cbf31cb52 in torch::autograd::VariableType::to(at::Tensor const&amp;, c10::TensorOptions const&amp;, bool, bool) const ()
   from /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so.1
#21 0x00007f8cc0bb8eb3 in torch::autograd::dispatch_to(at::Tensor const&amp;, c10::Device, bool, bool) () from /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so
#22 0x00007f8cc0bb9598 in torch::autograd::THPVariable_to(_object*, _object*, _object*) () from /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so
#23 0x0000556d518096a6 in PyCFunction_Call () at /tmp/build/80754af9/python_1546130271559/work/Objects/methodobject.c:98
#24 0x0000556d518b74ad in do_call_core (kwdict=0x7f8b9b42b168, callargs=0x7f8bb1deec18, func=0x7f8b9b42b510) at /tmp/build/80754af9/python_1546130271559/work/Python/ceval.c:5116
#25 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1546130271559/work/Python/ceval.c:3404

</code></pre>

amsword: <p>found the issue.</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/20630" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/prafullasd" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/5523595?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/20630" target="_blank" rel="nofollow noopener">Issue: distributed all_reduce deadlocks in v1.1</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/prafullasd" target="_blank" rel="nofollow noopener">prafullasd</a>
	on <a href="https://github.com/pytorch/pytorch/issues/20630" target="_blank" rel="nofollow noopener">2019-05-17</a>
	</div>
	<div class="user">
	closed by <a href="https://github.com/facebook-github-bot" target="_blank" rel="nofollow noopener">facebook-github-bot</a>
	on <a href="https://github.com/pytorch/pytorch/issues/20630" target="_blank" rel="nofollow noopener">2019-05-26</a>
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">🐛 Bug
I'm doing multi-node training (8 nodes, 8 gpu's each, NCCL backend) and am using DistributedDataParallel for syncing grads and distributed.all_reduce()...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: cuda</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: distributed</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


gemfield: <aside class="quote quote-modified" data-post="9" data-topic="8009">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gemfield/48/27799_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/distributed-data-parallel-freezes-without-error-message/8009/9">Distributed data parallel freezes without error message</a> 
  </div>
  <blockquote>
    Env: 

Ubuntu 18.04
Pytorch 1.6.0
CUDA 10.1

Actually, I am using Docker image gemfield/pytorch:1.6.0-devel which stated in <a href="https://github.com/DeepVAC/deepvac" rel="noopener nofollow ugc">https://github.com/DeepVAC/deepvac</a> (same with above env), and use PyTorch DDP (by use the class DeepvacDDP in <a href="https://github.com/DeepVAC/deepvac/blob/master/deepvac/syszux_deepvac.py" rel="noopener nofollow ugc">https://github.com/DeepVAC/deepvac/blob/master/deepvac/syszux_deepvac.py</a>) to train my model, which the code worked perfect yesterday. But today when I launch the train program again, the DDP is stucked in loss.backward()， with cpu 100% and GPU 100%。 
There has no co…
  </blockquote>
</aside>