Trouble launching on multiple nodes

username: bobby3605
cooked: <p>I’m trying to set up pytorch with slurm and nccl.<br>
I have 2 nodes, each with one GPU.<br>
There is an ethernet and infiniband connection between the two nodes.<br>
There is also a separate ethernet connection on the master node with its public address.<br>
This is the file I’m using to launch a job.</p>
<details>
<summary>
run.sh</summary>
<p>#!/bin/bash</p>
<p><span class="hashtag">#SBATCH</span> --partition=gpu<br>
<span class="hashtag">#SBATCH</span> --time=1-00:00:00<br>
<span class="hashtag">#SBATCH</span> --nodes=2<br>
<span class="hashtag">#SBATCH</span> --ntasks-per-node=1<br>
<span class="hashtag">#SBATCH</span> --job-name=elastic_ddp_test</p>
<p>head_node_ip=192.168.0.1<br>
head_node_port=29500</p>
<p>echo Node IP: $head_node_ip<br>
export LOGLEVEL=DEBUG</p>
<p>elastic_ddp_test=" <br>
–nnodes=2 <br>
–nproc_per_node=1 <br>
–rdzv_id=100 <br>
–rdzv_backend=c10d <br>
–rdzv_endpoint=$head_node_ip:$head_node_port <br>
–master_addr $head_node_ip <br>
–master_port $head_node_port <br>
elastic_ddp_test.py"</p>
<p>torchrun $elastic_ddp_test</p>
</details>
<p>I’m launching it with ‘sbatch run.sh’<br>
The address of the head node that the second node can access is 192.168.0.1. The second node does not have public internet access. It’s only network interfaces are an ethernet and infiniband connection to the head node.<br>
Running this fails to create the c10d store.<br>
If I change head_node_ip to localhost, it creates the store, but then gets stuck on ‘Rendezvous’ing worker group’.<br>
If I change head_node_ip to localhost and only run it on the head node, then it successfully runs the job.<br>
Using localhost also uses the public interface, which the secondary node cannot connect to.<br>
Part of this issue seems to have something to do with torchrun only creating a store on ipv6.<br>
If I run ‘netstat -ntlp’ while the job is running, python is only listening on ipv6.<br>
The secondary node does have an ipv6 address though, but if the master node ip is ipv4, then the secondary node will try to connect over ipv4, which will fail.<br>
If I try to use the ipv6 address of the head node, then it fails with ‘the ipv4 network addresses cannot be retrieved’.<br>
Is there a way to force torchrun to open a c10d store on a specific interface or address? I believe that would solve my issue.<br>
I have confirmed that I can open tcp port 29500 on the head node and connect to it from the second node using netcat.</p>

username: H-Huang
cooked: <aside class="quote no-group" data-username="bobby3605" data-post="1" data-topic="172582">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bobby3605/48/57338_2.png" class="avatar"> bobby3605:</div>
<blockquote>
<p>Is there a way to force torchrun to open a c10d store on a specific interface or address? I believe that would solve my issue.</p>
</blockquote>
</aside>
<p>You can configure this with <a href="https://pytorch.org/docs/stable/elastic/run.html#note-on-rendezvous-backend" rel="noopener nofollow ugc"><code>--rdvz_enpoint</code></a>. I would also first suggest that you try to write a simple script that creates a store and have both nodes run that to make sure they can connect without <code>torchrun</code>.</p>

username: Loubna_ben_allal
cooked: <p>Hi, If I launch a job without specifying the --rdv_backend, which backend is used by default?<br>
I am running a training with this lanucher on multiple nodes</p>
<pre><code class="lang-auto">export LAUNCHER="torchrun \
     --nproc_per_node $GPUS_PER_NODE \
     --nnodes $NNODES \
     --rdzv_endpoint $MASTER_ADDR:$MASTER_PORT \
    --rdzv_backend c10d \
     --max_restarts 0 \
     --tee 3 \
     "
</code></pre>
<p>But it fails with error:</p>
<pre><code class="lang-auto">torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
[E socket.cpp:860] [c10d] The client socket has timed out after 60s while trying to connect to (MASTER ADDR, Port)
</code></pre>
<p>If I remove <code>--rdzv_backend c10d</code>  the training runs successfully (also note that the nodes don"t have access to internet) is there a reason this causes failure and will removing this flag impact my training in any way?</p>