Correct usage of torch.distributed.run (multi-node multi-gpu)

amirhf: <p>Hello,</p>
<p>I used to launch a multi node multi gpu code using <code>torch.distributed.launch</code> on two cloud servers using two different .sh script in each machine:</p>
<pre><code class="lang-auto">#machine 1 script
export NUM_NODES=2
export NUM_GPUS_PER_NODE=4
export HOST_NODE_ADDR=10.70.202.133

python -m torch.distributed.launch \
    --nproc_per_node=$NUM_GPUS_PER_NODE \
    --nnodes=$NUM_NODES \
    --node_rank=0 \
    --master_addr=$HOST_NODE_ADDR \
    --master_port=1234 \
    train.py --debug
</code></pre>
<pre><code class="lang-auto">#machine 2 script
export NUM_NODES=2
export NUM_GPUS_PER_NODE=4
export HOST_NODE_ADDR=10.70.202.133

python -m torch.distributed.launch \
    --nproc_per_node=$NUM_GPUS_PER_NODE \
    --nnodes=$NUM_NODES \
    --node_rank=1 \
    --master_addr=$HOST_NODE_ADDR \
    --master_port=1234 \
    train.py --debug
</code></pre>
<p>So when I started to work with PyTOrch 1.9, it says that <code>torch.distributed.launch</code> is deprecated and I have to migrate to <code>torch.distributed.run</code>. Unfortunately, there is not enough information about this module in the documentation on how it replaces <code>torch.distributed.launch</code>. When I try to work with the new method and I use the two new scripts in each machine:</p>
<pre><code class="lang-auto">#machine 1
export NUM_NODES=2
export NUM_GPUS_PER_NODE=4
export HOST_NODE_ADDR=10.70.202.133:1234
export JOB_ID=22641

python -m torch.distributed.run \
    --nnodes=$NUM_NODES \
    --nproc_per_node=$NUM_GPUS_PER_NODE \
    --node_rank=0 \
    --rdzv_id=$JOB_ID \
    --rdzv_backend=c10d \
    --rdzv_endpoint=$HOST_NODE_ADDR \
    train.py --debug
</code></pre>
<pre><code class="lang-auto">#machine 2
export NUM_NODES=2
export NUM_GPUS_PER_NODE=4
export HOST_NODE_ADDR=10.70.202.133:1234
export JOB_ID=22641

python -m torch.distributed.run \
    --nnodes=$NUM_NODES \
    --nproc_per_node=$NUM_GPUS_PER_NODE \
    --node_rank=1 \
    --rdzv_id=$JOB_ID \
    --rdzv_backend=c10d \
    --rdzv_endpoint=$HOST_NODE_ADDR \
    train.py --debug
</code></pre>
<p>I get this error:</p>
<pre><code class="lang-auto">[ERROR] 2021-07-09 19:37:35,417 error_handler: {
  "message": {
    "message": "RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.",
    "extraInfo": {
</code></pre>
<p>Here’s how I setup my training script:</p>
<pre><code class="lang-auto">if "RANK" in os.environ and "WORLD_SIZE" in os.environ:
        config["RANK"] = int(os.environ["RANK"])
        config["WORLD_SIZE"] = int(os.environ["WORLD_SIZE"])
        config["GPU"] = int(os.environ["LOCAL_RANK"])
    else:
        config["DISTRIBUTED"] = False
        config["GPU"] = 1
        config["WORLD_SIZE"] = 1
        return

    config["DISTRIBUTED"] = True
    torch.cuda.set_device(config["GPU"])
    config["DIST_BACKEND"] = "nccl"
    torch.distributed.init_process_group(
        backend=config["DIST_BACKEND"],
        init_method=config["DIST_URL"],
        world_size=config["WORLD_SIZE"],
        rank=config["RANK"],
    )
</code></pre>
<p>and I also parse the following argument:</p>
<pre><code class="lang-auto">parser.add_argument("--local_rank", type=int)
</code></pre>
<p>Am I missing something?</p>

rvarm1: <p>Thanks for raising this issue! Since this seems like it could be a possible bug, or at the very least, a migration issue, can you file an issue (essentially this post) over at <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/pytorch · GitHub</a> so that we can take a deeper look?</p>
<p>cc <a class="mention" href="/u/cbalioglu">@cbalioglu</a> <a class="mention" href="/u/h-huang">@H-Huang</a> <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a> <a class="mention" href="/u/aivanou">@aivanou</a></p>

rvarm1: <p>Also, IIUC, <code>torch.distributed.run</code> should be fully backward-compatible with <code>torch.distributed.launch</code>. Have you tried simply dropping in torch.distributed.run with the same launch arguments, and if so what sort of issues did you hit there?</p>

Kiuk_Chung: <p>The docs for <code>torch.distributed.launch|run</code> needs some improvements to match the warning message. This issue is being tracked here: <a href="https://github.com/pytorch/pytorch/issues/60754" class="inline-onebox" rel="noopener nofollow ugc">dist docs need an urgent serious update · Issue #60754 · pytorch/pytorch · GitHub</a>. And most of it has been addressed in the nightly docs: <a href="https://pytorch.org/docs/master/elastic/run.html" class="inline-onebox" rel="noopener nofollow ugc">torch.distributed.run (Elastic Launch) — PyTorch master documentation</a>.</p>
<p>For the time being here are the things to note:</p>
<ol>
<li>
<p><code>torch.distributed.run</code> does not support parsing <code>--local_rank</code> as cmd arguments. If your script does this, then change it to getting local rank from <code>int(os.environ["LOCAL_RANK"])</code>. If you can’t change the script, then stick to <code>torch.distributed.launch</code> for now.</p>
</li>
<li>
<p>As <a class="mention" href="/u/rvarm1">@rvarm1</a> mentioned, <code>torch.distributed.run</code>'s arguments are mostly backwards compatible with <code>torch.distributed.launch</code> (the exception is <code>--use_env</code> which is now set as True by default since we are planning to deprecate reading local_rank from cmd args in favor of env).</p>
</li>
</ol>

amirhf: <p>I tried <code>torch.distributed.run</code> with the same legacy arguments, and it works. Seems like it has a problem with the new <code>rndvz</code> arguments (or maybe I am not setting them up correctly).</p>

Mattiasmar: <p>Hi,</p>
<p>Did you understand whether one has to initialize the c10d backend manually (and in that case how)?</p>
<p>From the latest documentation I see that the ‘torchrun’ should be used. However, when using it on a setup of two nodes, I get rendezvous connection error on the worker node (the master node remains in a waiting state).</p>

aivanou: <p>Hello! This usually means that nodes cannot reach each other due to hostname or ports being unavailable. What configuration are you using to launch jobs? Are you launching it on ec2?</p>

lqrhy3: <p>I think the problem is that <code>--rdzv_endpoint</code> argument expects <strong>hostname</strong> not IP of a master node. That is why <code>--rdzv_endpoint=$HOST_NODE_ADDR</code> causes connection error.</p>