Job hangs at backward

username: amsword
cooked: <p>pytorch 1.9 on A100. This is a large project and it is not easy to extract a reproducible script. The job runs well for several thousands of iterations, but it hangs at the backward.</p>
<p>here is the result of where under gdb</p>
<pre data-code-wrap="shell"><code class="lang-nohighlight">#0  0x00007f9219231376 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/x86_64-linux-gnu/libpthread.so.0                                                                                 [323/1863]#1  0x00007f921509e4d1 in __gthread_cond_wait (__mutex=&lt;optimized out&gt;, __cond=&lt;optimized out&gt;)
    at /home/conda/feedstock_root/build_artifacts/gcc_compilers_1634095553113/work/build/x86_64-conda-linux-gnu/libstdc++-v3/include/x86_64-conda-linux-gnu/bits/gthr-default.h:865
#2  std::__condvar::wait (__m=..., this=&lt;optimized out&gt;)                                                                                                                                                 at /home/conda/feedstock_root/build_artifacts/gcc_compilers_1634095553113/work/build/x86_64-conda-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155                                            #3  std::condition_variable::wait (this=&lt;optimized out&gt;, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41                                                               #4  0x00007f91cf9ad66b in torch::autograd::ReadyQueue::pop() () from /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so                                                                #5  0x00007f91cf9b1cf1 in torch::autograd::Engine::thread_main(std::shared_ptr&lt;torch::autograd::GraphTask&gt; const&amp;) () from /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so          #6  0x00007f91cf9ad3fb in torch::autograd::Engine::execute_with_graph_task(std::shared_ptr&lt;torch::autograd::GraphTask&gt; const&amp;, std::shared_ptr&lt;torch::autograd::Node&gt;, torch::autograd::InputBuffer&amp;&amp;
) () from /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so                                                                                                                           #7  0x00007f91d61940fd in torch::autograd::python::PythonEngine::execute_with_graph_task(std::shared_ptr&lt;torch::autograd::GraphTask&gt; const&amp;, std::shared_ptr&lt;torch::autograd::Node&gt;, torch::autograd::InputBuffer&amp;&amp;) () from /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so                                                                                                          #8  0x00007f91cf9af184 in torch::autograd::Engine::execute(std::vector&lt;torch::autograd::Edge, std::allocator&lt;torch::autograd::Edge&gt; &gt; const&amp;, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt; const&amp;, bool, bool, bool, std::vector&lt;torch::autograd::Edge, std::allocator&lt;torch::autograd::Edge&gt; &gt; const&amp;) () from /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so                  #9  0x00007f91d619404e in torch::autograd::python::PythonEngine::execute(std::vector&lt;torch::autograd::Edge, std::allocator&lt;torch::autograd::Edge&gt; &gt; const&amp;, std::vector&lt;at::Tensor, std::allocator&lt;at
::Tensor&gt; &gt; const&amp;, bool, bool, bool, std::vector&lt;torch::autograd::Edge, std::allocator&lt;torch::autograd::Edge&gt; &gt; const&amp;) () from /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so
#10 0x00007f91d6194b76 in THPEngine_run_backward(_object*, _object*, _object*) () from /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so
#11 0x00005655023b2914 in cfunction_call_varargs (kwargs=&lt;optimized out&gt;, args=&lt;optimized out&gt;, func=&lt;built-in method run_backward of torch._C._EngineBase object at remote 0x7f9209b2ff80&gt;)             at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:743                                                                                                 #12 PyCFunction_Call (func=&lt;built-in method run_backward of torch._C._EngineBase object at remote 0x7f9209b2ff80&gt;, args=&lt;optimized out&gt;, kwargs=&lt;optimized out&gt;)                                         at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:773
#13 0x00005655023b5ebe in _PyObject_MakeTpCall (callable=&lt;built-in method run_backward of torch._C._EngineBase object at remote 0x7f9209b2ff80&gt;, args=&lt;optimized out&gt;, nargs=&lt;optimized out&gt;,            keywords=&lt;optimized out&gt;) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:159                                                                       #14 0x0000565502457c85 in _PyObject_Vectorcall (kwnames=('allow_unreachable', 'accumulate_grad'), nargsf=&lt;optimized out&gt;, args=&lt;optimized out&gt;,
    callable=&lt;built-in method run_backward of torch._C._EngineBase object at remote 0x7f9209b2ff80&gt;)                                                                                                     at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Include/cpython/abstract.h:125                                                                                     #15 _PyObject_Vectorcall (kwnames=('allow_unreachable', 'accumulate_grad'), nargsf=&lt;optimized out&gt;, args=&lt;optimized out&gt;,                                                                                callable=&lt;built-in method run_backward of torch._C._EngineBase object at remote 0x7f9209b2ff80&gt;)                                                                                                     at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Include/cpython/abstract.h:115                                                                                     #16 call_function (kwnames=('allow_unreachable', 'accumulate_grad'), oparg=&lt;optimized out&gt;, pp_stack=&lt;synthetic pointer&gt;, tstate=&lt;optimized out&gt;)
    at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:4963
#17 _PyEval_EvalFrameDefault (f=&lt;optimized out&gt;, throwflag=&lt;optimized out&gt;) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:3515
#18 0x0000565502439433 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f91042d0440, for file /opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py, line 659, in backward (tensors=(&lt;Tensor at remote 0x7f9104743bc0&gt;,), grad_tensors=None, retai
n_graph=False, create_graph=False, grad_variables=None, inputs=(), grad_tensors_=(&lt;Tensor at remote 0x7f9104b21340&gt;,)))
    at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:741
#19 _PyEval_EvalCodeWithName (_co=&lt;optimized out&gt;, globals=&lt;optimized out&gt;, locals=&lt;optimized out&gt;, args=&lt;optimized out&gt;, argcount=&lt;optimized out&gt;, kwnames=&lt;optimized out&gt;,
    kwargs=&lt;optimized out&gt;, kwcount=&lt;optimized out&gt;, kwstep=&lt;optimized out&gt;, defs=&lt;optimized out&gt;, defcount=&lt;optimized out&gt;, kwdefs=&lt;optimized out&gt;, closure=&lt;optimized out&gt;, name=&lt;optimized out&gt;,
    qualname=&lt;optimized out&gt;) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:4298
#20 0x000056550243a818 in _PyFunction_Vectorcall (func=&lt;optimized out&gt;, stack=0x7f91043247a8, nargsf=&lt;optimized out&gt;, kwnames=&lt;optimized out&gt;)
    at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:436
#21 0x0000565502453eb2 in _PyObject_Vectorcall (kwnames=('inputs',), nargsf=&lt;optimized out&gt;, args=&lt;optimized out&gt;, callable=&lt;function at remote 0x7f915ab6ca60&gt;)
    at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Include/cpython/abstract.h:127
#22 call_function (kwnames=('inputs',), oparg=&lt;optimized out&gt;, pp_stack=&lt;synthetic pointer&gt;, tstate=&lt;optimized out&gt;)
    at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:4963
#23 _PyEval_EvalFrameDefault (f=&lt;optimized out&gt;, throwflag=&lt;optimized out&gt;) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:3515
#24 0x0000565502439433 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f9104324610, for file /opt/conda/lib/python3.8/site-packages/torch/_tensor.py, line 511, in backward (self=&lt;Tensor at remote 0x7f9104743bc0&gt;, gradient=None, retain_graph=False, creat
--Type &lt;RET&gt; for more, q to quit, c to continue without paging--

</code></pre>
<p>here is the result of py-bt under gdb</p>
<pre><code class="lang-auto">(gdb) py-bt
Traceback (most recent call first):
  &lt;built-in method run_backward of torch._C._EngineBase object at remote 0x7f9209b2ff80&gt;
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 659, in backward
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 511, in backward
    return super(Tensor, self).split(split_size, dim)
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 53, in backward
    scaled_loss.backward(retain_graph=retain_graph)
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 2409, in backward
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2739, in backward

  File "/tmp/code/quickdetection/src/qd/opt/trainer.py", line 1621, in inner_loop_step
    self.model_engine.backward(loss)
  File "/tmp/code/quickdetection/src/qd/opt/trainer.py", line 1716, in inner_loop
    # not well tested
  File "/tmp/code/quickdetection/src/qd/opt/trainer.py", line 1827, in do
    if self.gradient_acc_end:
  File "/tmp/code/quickdetection/src/qd/pipelines/uni_pipeline.py", line 1900, in do_train

  File "/tmp/code/quickdetection/src/qd/pipelines/uni_pipeline.py", line 4597, in train
  File "/tmp/code/quickdetection/src/qd/pipelines/uni_pipeline.py", line 1664, in ensure_train
    ).do()
  File "/tmp/code/quickdetection/src/qd/pipeline.py", line 680, in pipeline_train_eval_multi
    pip.ensure_train()
  File "src/qd/common.py", line 3377, in execute_func
  File "src/qd/common.py", line 4416, in &lt;module&gt;

</code></pre>