Backward() method cause hang deadlock and GPU 100% using DDP

BruceXu: <p>env: use docker with ubuntu 20.04 system<br>
Driver Version: 450.51.05<br>
CUDA Version: 11.0<br>
glibc 2.17</p>
<p>I tried to use DistributedDataParallel on single node with 2 GPUs, when start main process, the rank 0 process returned and finished, but rank1 still hang with GPU 1 100% and CPU 100%.</p>
<p>the model is causalnex and I tried to improve this model to GPU version.<br>
model ref: <a href="https://github.com/quantumblacklabs/causalnex" class="inline-onebox" rel="noopener nofollow ugc">GitHub - mckinsey/causalnex: A Python library that helps data scientists to infer causation rather than observing correlation.</a></p>
<p>my running code:</p>
<pre><code class="lang-auto">import numpy as np
import pandas as pd
import os

import causalnex
from causalnex.inference import InferenceEngine
from causalnex.network import BayesianNetwork
from causalnex.structure import StructureModel
from causalnex.structure.notears import from_pandas

from sklearn.datasets import load_diabetes
import torch
from sklearn.preprocessing import StandardScaler
from causalnex.structure import DAGRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.datasets import load_iris
from causalnex.structure import DAGClassifier

def main():

    data = load_iris()
    X, y = data.data, data.target
    names = data["feature_names"]
    os.environ['CUDA_LAUNCH_BLOCKING'] = '1'
    sys.path.insert(0, "/home/casual_infer/causal_server/causalnex_dist_version")

    clf = DAGClassifier(
        alpha=0.1,
        beta=0.9,
        hidden_layer_units=[5],
        fit_intercept=True,
        standardize=True
    )

    X = pd.DataFrame(X, columns=names)
    y = pd.Series(y, name="Flower_Class")
    print('print clf.fit(X, y)')
    clf.fit(X, y)
    for i in range(clf.coef_.shape[0]):
        print(f"MEAN EFFECT DIRECTIONAL CLASS {i}:")
        print(pd.Series(clf.coef_[i, :], index=names).sort_values(ascending=False))
    #clf.plot_dag(True)


if __name__ == '__main__':
    main()

</code></pre>
<p>below is the source snippet code where it hang<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fd75d85c84ec870abe0de483bd391bd374125c75.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fd75d85c84ec870abe0de483bd391bd374125c75" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fd75d85c84ec870abe0de483bd391bd374125c75_2_690x469.jpeg" alt="image" data-base62-sha1="AadhusF5UwYZNL3hqn5vlSZCCtT" width="690" height="469" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fd75d85c84ec870abe0de483bd391bd374125c75_2_690x469.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fd75d85c84ec870abe0de483bd391bd374125c75_2_1035x703.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fd75d85c84ec870abe0de483bd391bd374125c75_2_1380x938.jpeg 2x" data-dominant-color="313232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1770×1204 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">running log:
Epoch [1/100], Step [1/5], h: 0.0495,  gpu:1
Epoch [1/100], Step [1/5], h: 0.0421,  gpu:0
Epoch [1/100], Step [2/5], h: 0.0078,  gpu:0
Epoch [1/100], Step [2/5], h: 0.0052,  gpu:1
Epoch [1/100], Step [3/5], h: 0.0008,  gpu:0
return and finish gpu: 0


gdb py-bt result:
(gdb) py-bt
Traceback (most recent call first):
  &lt;built-in method run_backward of torch._C._EngineBase object at remote 0x7fb927d31c80&gt;
  File "/usr/local/lib/python3.8/dist-packages/torch/autograd/__init__.py", line 386, in backward
  File "/usr/local/lib/python3.8/dist-packages/torch/tensor.py", line 477, in backward

  File "/home/casual_infer/causal_server/causalnex_dist_version/causalnex/structure/pytorch/core.py", line 453, in _func
    primal_obj.backward()
  File "/usr/local/lib/python3.8/dist-packages/scipy/optimize/optimize.py", line 68, in _compute_if_needed
    fg = self.fun(x, *args)
  File "/usr/local/lib/python3.8/dist-packages/scipy/optimize/optimize.py", line 74, in __call__
    self._compute_if_needed(x, *args)
  File "/usr/local/lib/python3.8/dist-packages/scipy/optimize/_differentiable_functions.py", line 130, in fun_wrapped
    return fun(x, *args)
  File "/usr/local/lib/python3.8/dist-packages/scipy/optimize/_differentiable_functions.py", line 133, in update_fun
    self.f = fun_wrapped(self.x)
  File "/usr/local/lib/python3.8/dist-packages/scipy/optimize/_differentiable_functions.py", line 226, in _update_fun
    self._update_fun_impl()
  File "/usr/local/lib/python3.8/dist-packages/scipy/optimize/_differentiable_functions.py", line 260, in fun_and_grad
    self._update_fun()
  File "/usr/local/lib/python3.8/dist-packages/scipy/optimize/lbfgsb.py", line 1128, in _minimize_lbfgsb
  File "/usr/local/lib/python3.8/dist-packages/scipy/optimize/_minimize.py", line 5483, in minimize
  File "/home/casual_infer/causal_server/causalnex_dist_version/causalnex/structure/pytorch/core.py", line 1236, in _dual_ascent_step
  File "/home/casual_infer/causal_server/causalnex_dist_version/causalnex/structure/pytorch/core.py", line 801, in fit
  File "/home/casual_infer/causal_server/causalnex_dist_version/causalnex/structure/pytorch/notears.py", line 1045, in dist_train_worker
  File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.8/multiprocessing/process.py", line 571, in _bootstrap
  File "/usr/lib/python3.8/multiprocessing/spawn.py", line 129, in _main
    return self._bootstrap(parent_sentinel)
  File "/usr/lib/python3.8/multiprocessing/spawn.py", line 628, in spawn_main
  File "&lt;string&gt;", line 1, in &lt;module&gt;


**gdb bt result:**
gdb) bt
#0  futex_wait_cancelable (private=&lt;optimized out&gt;, expected=0, futex_word=0x60012388) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x60012390, cond=0x60012360) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x60012360, mutex=0x60012390) at pthread_cond_wait.c:638
#3  0x00007fb92cbf2e30 in std::condition_variable::wait(std::unique_lock&lt;std::mutex&gt;&amp;) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007fb88b88d17b in torch::autograd::ReadyQueue::pop() () from /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_cpu.so
#5  0x00007fb88b8922c9 in torch::autograd::Engine::thread_main(std::shared_ptr&lt;torch::autograd::GraphTask&gt; const&amp;) () from /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_cpu.so
#6  0x00007fb88b890310 in torch::autograd::Engine::execute_with_graph_task(std::shared_ptr&lt;torch::autograd::GraphTask&gt; const&amp;, std::shared_ptr&lt;torch::autograd::Node&gt;) ()
   from /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_cpu.so
#7  0x00007fb926795dcc in torch::autograd::python::PythonEngine::execute_with_graph_task(std::shared_ptr&lt;torch::autograd::GraphTask&gt; const&amp;, std::shared_ptr&lt;torch::autograd::Node&gt;) ()
   from /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_python.so
#8  0x00007fb88b88f36d in torch::autograd::Engine::execute(std::vector&lt;torch::autograd::Edge, std::allocator&lt;torch::autograd::Edge&gt; &gt; const&amp;, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt; const&amp;, bool, bool, std::vector&lt;torch::autograd::Edge, std::allocator&lt;torch::autograd::Edge&gt; &gt; const&amp;) () from /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_cpu.so
#9  0x00007fb926795bce in torch::autograd::python::PythonEngine::execute(std::vector&lt;torch::autograd::Edge, std::allocator&lt;torch::autograd::Edge&gt; &gt; const&amp;, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt; const&amp;,bool, bool, std::vector&lt;torch::autograd::Edge, std::allocator&lt;torch::autograd::Edge&gt; &gt; const&amp;) () from /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_python.so
#10 0x00007fb926796c9f in THPEngine_run_backward(THPEngine*, _object*, _object*) () from /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_python.so
#11 0x00000000005f5e79 in cfunction_call_varargs (kwargs=&lt;optimized out&gt;, args=&lt;optimized out&gt;, func=&lt;built-in method run_backward of torch._C._EngineBase object at remote 0x7fb927d31c80&gt;) at ../Objects/call.c:773
#12 PyCFunction_Call (func=&lt;built-in method run_backward of torch._C._EngineBase object at remote 0x7fb927d31c80&gt;, args=&lt;optimized out&gt;, kwargs=&lt;optimized out&gt;) at ../Objects/call.c:773
#13 0x00000000005f6a46 in _PyObject_MakeTpCall (callable=&lt;built-in method run_backward of torch._C._EngineBase object at remote 0x7fb927d31c80&gt;, args=&lt;optimized out&gt;, nargs=&lt;optimized out&gt;, keywords=&lt;optimized out&gt;)
    at ../Include/internal/pycore_pyerrors.h:13
#14 0x0000000000570f1d in _PyObject_Vectorcall (kwnames=('allow_unreachable',), nargsf=&lt;optimized out&gt;, args=&lt;optimized out&gt;,
    callable=&lt;built-in method run_backward of torch._C._EngineBase object at remote 0x7fb927d31c80&gt;) at ../Include/cpython/abstract.h:125
#15 _PyObject_Vectorcall (kwnames=('allow_unreachable',), nargsf=&lt;optimized out&gt;, args=&lt;optimized out&gt;, callable=&lt;built-in method run_backward of torch._C._EngineBase object at remote 0x7fb927d31c80&gt;)
    at ../Include/cpython/abstract.h:115
#16 call_function (kwnames=('allow_unreachable',), oparg=&lt;optimized out&gt;, pp_stack=&lt;synthetic pointer&gt;, tstate=&lt;optimized out&gt;) at ../Python/ceval.c:4963
#17 _PyEval_EvalFrameDefault (f=&lt;optimized out&gt;, throwflag=&lt;optimized out&gt;) at ../Python/ceval.c:3515
#18 0x00000000005696da in PyEval_EvalFrameEx (throwflag=0, f=
    Frame 0x7fb74933d610, for file /usr/local/lib/python3.8/dist-packages/torch/autograd/__init__.py, line 386, in backward (tensors=(&lt;Tensor at remote 0x7fb749346040&gt;,), grad_tensors=None, retain_graph=False, create_graph=False, grad_variables=None, grad_tensors_=(&lt;Tensor at remote 0x7fb749346500&gt;,))) at ../Python/ceval.c:741
#19 _PyEval_EvalCodeWithName (_co=&lt;optimized out&gt;, globals=&lt;optimized out&gt;, locals=&lt;optimized out&gt;, args=&lt;optimized out&gt;, argcount=&lt;optimized out&gt;, kwnames=&lt;optimized out&gt;, kwargs=0x7fb74ddeb210,
    kwcount=&lt;optimized out&gt;, kwstep=1, defs=0x7fb9253c1788, defcount=4, kwdefs=0x0, closure=0x0, name='backward', qualname='backward') at ../Python/ceval.c:4298
#20 0x00000000005f6403 in _PyFunction_Vectorcall (func=&lt;optimized out&gt;, stack=0x7fb74ddeb1f0, nargsf=&lt;optimized out&gt;, kwnames=&lt;optimized out&gt;) at ../Objects/call.c:436
#21 0x00000000005703e6 in _PyObject_Vectorcall (kwnames=0x0, nargsf=&lt;optimized out&gt;, args=0x7fb74ddeb1f0, callable=&lt;function at remote 0x7fb925343310&gt;) at ../Include/cpython/abstract.h:127
#22 call_function (kwnames=0x0, oparg=&lt;optimized out&gt;, pp_stack=&lt;synthetic pointer&gt;, tstate=0x1782e90) at ../Python/ceval.c:4963
#23 _PyEval_EvalFrameDefault (f=&lt;optimized out&gt;, throwflag=&lt;optimized out&gt;) at ../Python/ceval.c:3469
#24 0x00000000005696da in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fb74ddeb040, for file /usr/local/lib/python3.8/dist-packages/torch/tensor.py, line 477, in backward (self=&lt;Tensor at remote 0x7fb749346040&gt;, gradient=None, retain_graph=None, create_graph=False, relevant_args=(&lt;...&gt;,), has_torch_function=&lt;function at remote 0x7fb92531b160&gt;, handle_torch_function=&lt;function at remote 0x7fb92531b0d0&gt;)) at ../Python/ceval.c:741
#25 _PyEval_EvalCodeWithName (_co=&lt;optimized out&gt;, globals=&lt;optimized out&gt;, locals=&lt;optimized out&gt;, args=&lt;optimized out&gt;, argcount=&lt;optimized out&gt;, kwnames=&lt;optimized out&gt;, kwargs=0x4bb58948,
    kwcount=&lt;optimized out&gt;, kwstep=1, defs=0x7fb92559a758, defcount=3, kwdefs=0x0, closure=0x0, name='backward', qualname='Tensor.backward') at ../Python/ceval.c:4298
#26 0x00000000005f6403 in _PyFunction_Vectorcall (func=&lt;optimized out&gt;, stack=0x4bb58940, nargsf=&lt;optimized out&gt;, kwnames=&lt;optimized out&gt;) at ../Objects/call.c:436
#27 0x000000000056b5e0 in _PyObject_Vectorcall (kwnames=0x0, nargsf=&lt;optimized out&gt;, args=0x4bb58940, callable=&lt;function at remote 0x7fb9255a95e0&gt;) at ../Include/cpython/abstract.h:127
#28 call_function (kwnames=0x0, oparg=&lt;optimized out&gt;, pp_stack=&lt;synthetic pointer&gt;, tstate=0x1782e90) at ../Python/ceval.c:4963
#29 _PyEval_EvalFrameDefault (f=&lt;optimized out&gt;, throwflag=&lt;optimized out&gt;) at ../Python/ceval.c:3486
#30 0x00000000005696da in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x4bb58730, for file /home/casual_infer/causal_server/causalnex_dist_version/causalnex/structure/pytorch/core.py, line 453, in _func (flat_params=&lt;numpy.ndarray at remote 0x7fb74a5b8750&gt;, n_features=7, X_--Type &lt;RET&gt; for more, q to quit, c to continue without paging--

</code></pre>
<p>I have tried a lot and can’t find out the main reason.</p>

albanD: <p>Hi,</p>
<p>This is surprising indeed.<br>
If you can come up with a simple repro for this, it would be nice to open a github issue to get more visibility on this.</p>
<p>Thanks!</p>

BruceXu: <p>Hi,</p>
<p>Thanks for your reply,  I have found the reason is that different worker has different iterate times, which cause the hang between the process.</p>
<p>The method sopt.minimize() will run random time which cause the hang. I found that the algorithm is not originally designed for distributed running.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3a2d7976c841a022747d706e9e34d8bffc18125.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d3a2d7976c841a022747d706e9e34d8bffc18125" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3a2d7976c841a022747d706e9e34d8bffc18125.jpeg" alt="image" data-base62-sha1="ucdDiZ6zoaHGnehiUeHwsJtwnHL" width="690" height="304" data-dominant-color="313030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">707×312 32.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>