Need Help Solving DDP Connection Failures

TheVictor_777: <p>Hello, I have been trying for weeks now to use DDP as a PyTorch training accelerator. Unfortunately, I have been met with a wall of failures. I am trying to run <a href="https://github.com/pytorch/examples/blob/main/distributed/ddp-tutorial-series/multinode.py" rel="noopener nofollow ugc">PyTorch’s DDP multi-node example</a>, but with no success. I execute the following on my nodes:</p>
<pre><code class="lang-auto">TP_SOCKET_IFNAME=&lt;Socket&gt; NCCL_SOCKET_IFNAME=&lt;Socket&gt; GLOO_SOCKET_IFNAME=&lt;Socket&gt; time torchrun --nnodes=2 --nproc-per-node=1 --master-addr=&lt;Master Address&gt; --node_rank=&lt;0 to 1&gt; --master-port=7777 --start-method=spawn multinode.py 50 10
</code></pre>
<p>So far, I have experienced a barrage of failures. First, I discovered that I forgot to open port 7777 to TCP on both nodes. With that done, the client now spits this out while the server waits:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/&lt;User&gt;/.local/bin/torchrun", line 8, in &lt;module&gt;
    sys.exit(main())
             ^^^^^^
  File "/home/&lt;User&gt;/.local/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/&lt;User&gt;/.local/lib/python3.12/site-packages/torch/distributed/run.py", line 812, in main
    run(args)
  File "/home/&lt;User&gt;/.local/lib/python3.12/site-packages/torch/distributed/run.py", line 803, in run
    elastic_launch(
  File "/home/&lt;User&gt;/.local/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/&lt;User&gt;/.local/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 259, in launch_agent
    result = agent.run()
             ^^^^^^^^^^^
  File "/home/&lt;User&gt;/.local/lib/python3.12/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
  File "/home/&lt;User&gt;/.local/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 727, in run
    result = self._invoke_run(role)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/&lt;User&gt;/.local/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 862, in _invoke_run
    self._initialize_workers(self._worker_group)
  File "/home/&lt;User&gt;/.local/lib/python3.12/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
  File "/home/&lt;User&gt;/.local/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 699, in _initialize_workers
    self._rendezvous(worker_group)
  File "/home/&lt;User&gt;/.local/lib/python3.12/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
  File "/home/&lt;User&gt;/.local/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 542, in _rendezvous
    store, group_rank, group_world_size = spec.rdzv_handler.next_rendezvous()
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/&lt;User&gt;/.local/lib/python3.12/site-packages/torch/distributed/elastic/rendezvous/static_tcp_rendezvous.py", line 55, in next_rendezvous
    self._store = TCPStore(  # type: ignore[call-arg]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.distributed.DistNetworkError: Connection reset by peer
Command exited with non-zero status 1
1.92user 0.49system 0:01.96elapsed 123%CPU (0avgtext+0avgdata 378640maxresident)k
0inputs+8outputs (0major+51822minor)pagefaults 0swaps
</code></pre>
<p>By using ‘nc’ to test the port functionality on both nodes, I have confirmed that both PCs can connect to each other through port 7777. What could possibly be wrong? <img src="https://discuss.pytorch.org/images/emoji/apple/person_shrugging.png?v=12" title=":person_shrugging:" class="emoji" alt=":person_shrugging:" loading="lazy" width="20" height="20"> Any help is greatly appreciated!</p>