DDP launch.py: how different processes can receive different local rank?

Olivier-CR: <p>Hi, I’m using this doc to launch a DDP script <a href="https://github.com/pytorch/examples/blob/master/distributed/ddp/README.md" class="inline-onebox" rel="noopener nofollow ugc">examples/README.md at master · pytorch/examples · GitHub</a></p>
<p>my launch code is</p>
<pre><code class="lang-auto"> python /home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/distributed/launch.py \
    --nnodes 1 \
    --node_rank=0 \
    --nproc_per_node 4 \
    train.py \
    --gpu-count 4 \
    --dataset . \
    --cache tmp \
    --height 604 \
    --width 960 \
    --checkpoint-dir . \
    --batch 10 \
    --workers 24 \
    --log-freq 20 \
    --prefetch 2 \
    --bucket $bucket \
    --eval-size 10 \
    --iterations 20 \
    --class-list a2d2_images/camera_lidar_semantic/class_list.json
</code></pre>
<p>However, when I print the content of each process I see that on each process local_rank is set to -1<br>
How to get different and unique values in the local_rank argument? I thought launch.py was handling that?</p>

cbalioglu: <p>cc <a class="mention" href="/u/aivanou">@aivanou</a>, <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a></p>

Kiuk_Chung: <p>Hi, I wasn’t able to repro your issue with torch-1.10.</p>
<p>Here’s the test script I tested with</p>
<pre><code class="lang-auto"># save this as test.py

import argparse
import os
import sys

from torch.distributed.elastic.multiprocessing.errors import record


def parse_args(argv):
    parser = argparse.ArgumentParser(description="test script")
    parser.add_argument("--local_rank", type=int)
    return parser.parse_args()

@record
def main():
    args = parse_args(sys.argv[1:])
    print(f"local_rank={args.local_rank}")

if __name__ == "__main__":
    main()
</code></pre>
<p>Then run</p>
<pre><code class="lang-auto">$ python -m torch.distributed.launch --nnode=1 --node_rank=0 --nproc_per_node=4 test.py
Note that --use_env is set by default in torchrun.
If your script expects `--local_rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See
https://pytorch.org/docs/stable/distributed.html#launch-utility for
further instructions

  warnings.warn(
WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
local_rank=3
local_rank=2
local_rank=0
local_rank=1
</code></pre>
<p>The example seems out of date. Please follow the instructions here: <a href="https://pytorch.org/docs/stable/elastic/run.html" class="inline-onebox" rel="noopener nofollow ugc">torchrun (Elastic Launch) — PyTorch 1.10.0 documentation</a></p>

Olivier-CR: <p>thanks a lot I’ll try that!<br>
so you’re using</p>
<pre><code class="lang-auto">python -m torch.distributed.launch
</code></pre>
<p>?</p>
<p>isn’t the fresh guidance to use <code>torchrun</code>?</p>

Olivier-CR: <aside class="quote no-group" data-username="Kiuk_Chung" data-post="3" data-topic="135146">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kiuk_chung/48/22504_2.png" class="avatar"> Kiuk_Chung:</div>
<blockquote>
<p><code>from torch.distributed.elastic.multiprocessing.errors import record</code></p>
</blockquote>
</aside>
<p>What is that line doing?</p>

Kiuk_Chung: <p>That line is there to make sure the error trace info on the trainer (a different process from the agent process) can be propagated to the agent for error summary and reporting purposes. See: <a href="https://pytorch.org/docs/stable/elastic/errors.html" class="inline-onebox" rel="noopener nofollow ugc">Error Propagation — PyTorch 1.10.0 documentation</a></p>
<p>If the <span class="mention">@record</span> is not there then no trace information will be logged in the error summary table. So you’ll have to dig through the logs for the exception stack trace.</p>