GLOO/NCCL connection issues [build from source]

username: ccoulombe
cooked: <p>Hello!!</p>
<h1>
<a name="summary-1" class="anchor" href="#summary-1"></a>Summary</h1>
<p>My build of Pytorch v1.10.0 from source seem to have issues with the <code>gloo</code> and <code>nccl</code> backends, but works fine with <code>mpi</code>. The error alternate between:</p>
<ul>
<li>Connection refused</li>
<li>Connection reset by peer</li>
<li>Socket Timeout</li>
</ul>
<p>Even when the port is free and available.</p>
<p>Using the pre-built wheel from upstream (<code>torch-1.10.0+cu113-cp39-cp39-linux_x86_64.whl</code>), the issue cannot be reproduced. In other words, it works as expected.</p>
<p>Building GLOO with the same configuration (cuda and nccl) and running the tests suite : all tests passes.<br>
Running NVIDIA NCCL examples works fine.</p>
<h1>
<a name="test-snippet-2" class="anchor" href="#test-snippet-2"></a>Test snippet</h1>
<pre><code class="lang-python">import torch.nn.parallel
import torch.distributed as dist
import os

os.environ['MASTER_ADDR'] = str(os.environ.get('HOST', '127.0.0.1'))
os.environ['MASTER_PORT'] = str(os.environ.get('PORT', 29500))
os.environ['RANK'] = str(os.environ.get('SLURM_LOCALID', 0))
os.environ['WORLD_SIZE'] = str(os.environ.get('SLURM_NTASKS', 2))

backend = os.environ.get('BACKEND', 'mpi')
print('Using backend:', backend)

dist.init_process_group(backend)
# dist.init_process_group(backend, init_method=f"tcp://{master_add}:{master", rank=rank, world_size=size)
my_rank = dist.get_rank()
my_size = dist.get_world_size()

print("my rank = %d  my size = %d" % (my_rank, my_size))

dist.destroy_process_group()
</code></pre>
<p>Ran on an <strong>exclusive</strong> single node (SLURM) with two tasks:</p>
<pre><code class="lang-auto">srun python ddp_torch.py
</code></pre>
<h1>
<a name="details-3" class="anchor" href="#details-3"></a>Details</h1>
<h2>
<a name="configuration-4" class="anchor" href="#configuration-4"></a>Configuration</h2>
<pre><code class="lang-auto">&gt;&gt;&gt; print(torch.__config__.show())
PyTorch built with:
  - GCC 9.3
  - C++ Version: 201402
  - Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.4
  - NVCC architecture flags: -gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80
  - CuDNN 8.2  (built against CUDA 11.3)
  - Magma 2.6.1
  - Build settings:
      BLAS_INFO=flexi
      BUILD_TYPE=Release
      CUDA_VERSION=11.4
      CUDNN_VERSION=8.2.0
      CXX_COMPILER=/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/bin/c++
      CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow
      LAPACK_INFO=flexi
      PERF_WITH_AVX=1
      PERF_WITH_AVX2=1
      PERF_WITH_AVX512=1
      TORCH_VERSION=1.10.0
      USE_CUDA=ON
      USE_CUDNN=ON
      USE_EXCEPTION_PTR=1
      USE_GFLAGS=OFF
      USE_GLOG=OFF
      USE_MKLDNN=ON
      USE_MPI=ON,
      USE_NCCL=ON,
      USE_NNPACK=ON,
      USE_OPENMP=ON,
</code></pre>
<p>Missing info from the above: NCCL v2.11.4</p>
<h2>
<a name="build-summary-5" class="anchor" href="#build-summary-5"></a>Build summary</h2>
<pre><code class="lang-bash">--   USE_DISTRIBUTED       : ON
--     USE_MPI               : ON
--     USE_GLOO              : ON
--     USE_GLOO_WITH_OPENSSL : OFF
--     USE_TENSORPIPE        : ON
</code></pre>
<h2>
<a name="issues-6" class="anchor" href="#issues-6"></a>Issues</h2>
<h3>
<a name="tests-suite-7" class="anchor" href="#tests-suite-7"></a>Tests suite</h3>
<p>Pytorch test suite with my build from source:</p>
<ul>
<li>distributed/test_nccl.py passes</li>
<li>distributed/test_c10d_gloo.py fails with connection errors</li>
<li>distributed/test_c10d_nccl.py fails with connection errors</li>
</ul>
<p>Pytorch test suite with upstream wheel (<code>torch-1.10.0+cu113-cp39-cp39-linux_x86_64.whl</code>):</p>
<ul>
<li>distributed/test_nccl.py passes</li>
<li>distributed/test_c10d_gloo.py passes</li>
<li>distributed/test_c10d_nccl.py passes</li>
</ul>
<h3>
<a name="manually-running-the-test-code-8" class="anchor" href="#manually-running-the-test-code-8"></a>Manually running the test code</h3>
<h4>
<a name="gloo-9" class="anchor" href="#gloo-9"></a>GLOO</h4>
<pre><code class="lang-bash">(2149) ~ $ export BACKEND=gloo
(2149) ~ $ export PORT=$(python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')
(2149) ~ $ srun python ddp_torch.py
terminate called after throwing an instance of 'std::system_error'
  what():  Connection refused
</code></pre>
<h4>
<a name="nccl-10" class="anchor" href="#nccl-10"></a>NCCL</h4>
<pre><code class="lang-auto">(2149) ~ $ export NCCL_DEBUG=INFO
(2149) ~ $ export BACKEND=gloo
(2149) ~ $ export PORT=$(python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')
(2149) ~ $ srun python ddp_torch.py
Using backend: nccl
my rank = 0  my size = 2
Using backend: nccl
my rank = 1  my size = 2
terminate called after throwing an instance of 'std::system_error'
  what():  Connection reset by peer
</code></pre>
<p>Or sometimes:</p>
<pre><code class="lang-auto">  what():  Connection refused
</code></pre>
<p>Or sometimes:</p>
<pre><code class="lang-auto">  what():  Socket Timeout
</code></pre>
<p>Or in rare cases, it works as expected.</p>
<h3>
<a name="expected-output-11" class="anchor" href="#expected-output-11"></a>Expected output</h3>
<h4>
<a name="nccl-12" class="anchor" href="#nccl-12"></a>NCCL</h4>
<pre><code class="lang-bash">(32450) ~ $ export NCCL_DEBUG=INFO
(32450) ~ $ export BACKEND=nccl
(32450) ~ $ export PORT=$(python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')
(32450) ~ $ srun python ddp_torch.py
Using backend: nccl
my rank = 0  my size = 2
Using backend: nccl
my rank = 1  my size = 2
</code></pre>
<h4>
<a name="gloo-13" class="anchor" href="#gloo-13"></a>GLOO</h4>
<pre><code class="lang-bash">(32450) ~ $ export BACKEND=gloo
(32450) ~ $ export PORT=$(python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')
(32450) ~ $ srun python ddp_torch.py
Using backend: gloo
my rank = 0  my size = 2
Using backend: gloo
my rank = 1  my size = 2
</code></pre>
<hr>
<h1>
<a name="epilog-14" class="anchor" href="#epilog-14"></a>Epilog</h1>
<p>Any clues or hint on what might be the issue with the build from source?</p>
<p>Next is to build with debug and see if <code>TORCH_DISTRIBUTED_DETAIL=DEBUG</code> can help.</p>
<p>Related questions:</p>
<ol>
<li>When using NCCL backend, with environment variable <code>NCCL_DEBUG=INFO</code>, no NCCL output is produced. How come?</li>
<li>Where can I find the build configuration from the CI build? The equivalent of CMake summary. I looked in the Github actions log, but did not find it.</li>
</ol>
<p>Thank you very much!</p>

username: rvarm1
cooked: <p>Hi, can you try building against the latest master branch and see if the issues persist/paste the error logs?   A useful PR <a href="https://github.com/pytorch/pytorch/pull/68226" class="inline-onebox" rel="noopener nofollow ugc">Revise the socket implementation of c10d by cbalioglu · Pull Request #68226 · pytorch/pytorch · GitHub</a> has just landed significantly improving the implementation and error logging of the c10d store so the logs should provide a lot more details in the case of errors.</p>
<p>In addition, since this seems like a reproducible bug, could you file an issue over at <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/pytorch · GitHub</a> with reproduction instructions so we can take a deeper look?</p>

username: ccoulombe
cooked: <p>Thanks! Building from HEAD to include the PR, I got:</p>
<pre><code class="lang-bash">(31286) ~ $ export TORCH_DISTRIBUTED_DETAIL=DEBUG
(31286) ~ $ export PORT=$(python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')
(31286) ~ $ echo $PORT
32967
(31286) ~ $ export BACKEND=gloo
(31286) ~ $ srun python ddp_torch.py
[W socket.cpp:634] The server socket on [localhost]:32967 is not yet listening (generic error: 111 - Connection refused).
terminate called after throwing an instance of 'std::system_error'
  what():  Connection reset by peer
Using backend: gloo
my rank = 1  my size = 2
</code></pre>
<p>It worked partially, as it is missing rank 0.<br>
With NCCL backend, no information is printed, only the error.</p>
<p>I’ll create an issue then.<br>
Issue : <a href="https://github.com/pytorch/pytorch/issues/69003" class="inline-onebox" rel="noopener nofollow ugc">GLOO/NCCL connection issues [build from source] · Issue #69003 · pytorch/pytorch · GitHub</a></p>