What is the difference among these elastic launcher?

zizhao.mo: <p>Hi,</p>
<p>I found that there are many launcher can be used to start elastic distributed training, e.g., torchelastic, torchrun, torch.distributed.run. But which one is recommended now? Is the training script applicable for A is also applicable for B?</p>
<p>Thanks</p>

pbelevich: <p><a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a> can you please help here?</p>

Kiuk_Chung: <p>They all use <code>torch.distributed.run</code> under the hood, which is using torchelastic.</p>
<p><code>torchrun</code> is effectively equal to <code>torch.distributed.run</code> but it is a “console script” (<a href="https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html" class="inline-onebox" rel="noopener nofollow ugc">Command Line Scripts — Python Packaging Tutorial</a>) that we include for convenience so that you don’t have to run <code>python -m torch.distributed.run</code> every time and can simply invoke <code>torchrun &lt;same arguments&gt;</code>.</p>
<p><code>torch.distributed.launch</code> is now on the path of deprecation, and internally calls <code>torch.distributed.run</code>. So it has a more restrictive set of options and a few option remappings when compared to <code>torch.distributed.run</code> for backwards compatibility.</p>
<p>We’ve updated our nightly documentation to explain this situation <a href="https://pytorch.org/docs/master/elastic/run.html#launcher-api" class="inline-onebox" rel="noopener nofollow ugc">torchrun (Elastic Launch) — PyTorch 1.12 documentation</a></p>

zizhao.mo: <p>Hi, thanks for your response. I found that I cannot scale out the worker number of jobs (from 2 to 3) dynamically. Is this the inherit constraint from pytorch?</p>

Kiuk_Chung: <p>Torchelastic should handle admission and departure of nodes. How are you running the min=2 max=3 job?</p>

zizhao.mo: <p>Hi Kiuk,</p>
<p>I am sorry that I didn’t explain my dilemma well. Now I can use torchrun script to start a training in one host, to be specific, I start the script by using: torchrun --nnodes=1:2 --nproc_per_node=1 --max_restarts=3 --rdzv_id=1 --rdzv_backend=c10d --rdzv_endpoint=slave6:1234 main.py . In standalone mode, this runs well. However, when I start this script in another host, i.e. slave7, this script fails. The log of new node I added is as below:</p>
<p>05/02/2022 23:18:20 PM - INFO - Added key: store_based_barrier_key:1 to store for rank: 1<br>
05/02/2022 23:18:20 PM - INFO - Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.<br>
05/02/2022 23:18:24 PM - ERROR - NCCL error in: …/torch/csrc/distributed/c10d/NCCLUtils.hpp:125, internal error, NCCL version 21.0.3<br>
ncclInternalError: Internal check failed. This is either a bug in NCCL or due to memory corruption<br>
Traceback (most recent call last):<br>
File “main.py”, line 74, in <br>
model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[local_rank])<br>
File “/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py”, line 641, in <strong>init</strong><br>
dist._verify_params_across_processes(self.process_group, parameters)<br>
RuntimeError: NCCL error in: …/torch/csrc/distributed/c10d/NCCLUtils.hpp:125, internal error, NCCL version 21.0.3<br>
ncclInternalError: Internal check failed. This is either a bug in NCCL or due to memory corruption</p>
<p>So I fail to dynamically add workers. How can I check what’s wrong in it?</p>
<p>Thanks for your patient response.</p>

Kiuk_Chung: <p>The usage docs (<a href="https://pytorch.org/docs/stable/elastic/run.html#usage" class="inline-onebox" rel="noopener nofollow ugc">torchrun (Elastic Launch) — PyTorch 1.11.0 documentation</a>) has examples for different use-cases.</p>
<p><code>etcd</code> is only required if:</p>
<ol>
<li>you need a high degree of fault tolerance (aka node 0 fault-tolerance). By default <code>rdzv_backend=c10d</code> will create a data-plane on node 0, so if node 0 dies, then your job cannot recover and the job has to be retried. Using an external ectd store prevents this but the probability of node 0 failure is also pretty low. Ultimately it is your decision on whether you want to deal with setting up an external store for higher degree of fault tolerance, or prefer simplicity.</li>
<li>you don’t have the IP/hostname of “one” of the nodes up front. In this case you use the etcd url (which you will know up front since you setup an etcd cluster somewhere).</li>
</ol>
<p>For the particular issue you are having with the two node setup, could you copy paste the exact CMD you ran on each node? Also could you set the env var <code>LOGLEVEL=INFO</code> when running <code>torchrun</code> (e.g. <code>$ LOGLEVEL=INFO torchrun ...</code>) and copy paste the outputs on each node?</p>

zizhao.mo: <p>Hi Kiuk,</p>
<p>I use command  <em>LOGLEVEL=INFO torchrun --nnodes=1:2 --nproc_per_node=1 --max_restarts=3 --rdzv_id=1 --rdzv_backend=c10d --rdzv_endpoint=slave6:1234 main.py --if_name enp4s0 --batch_size 512</em>  to start my script, where the last two parameters are customized for my application. By this command, my training script can run well in standalone mode. In this host, the stdout print the text below, including the events generated by new host admission (after the <em>SIGTERM</em>):</p>
<p>INFO:torch.distributed.launcher.api:Starting elastic_operator with launch configs:<br>
entrypoint       : main.py<br>
min_nodes        : 1<br>
max_nodes        : 2<br>
nproc_per_node   : 1<br>
run_id           : 1<br>
rdzv_backend     : c10d<br>
rdzv_endpoint    : slave6:1234<br>
rdzv_configs     : {‘timeout’: 900}<br>
max_restarts     : 3<br>
monitor_interval : 5<br>
log_dir          : None<br>
metrics_cfg      : {}</p>
<p>INFO:torch.distributed.elastic.agent.server.local_elastic_agent:log directory set to: /tmp/torchelastic_2bl7vzba/1_8s8bqrfc<br>
INFO:torch.distributed.elastic.agent.server.api:[default] starting workers for entrypoint: python3<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous’ing worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:<br>
restart_count=0<br>
master_addr=slave6<br>
master_port=57905<br>
group_rank=0<br>
group_world_size=1<br>
local_ranks=[0]<br>
role_ranks=[0]<br>
global_ranks=[0]<br>
role_world_sizes=[1]<br>
global_world_sizes=[1]</p>
<p>INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_2bl7vzba/1_8s8bqrfc/attempt_0/0/error.json<br>
Files already downloaded and verified<br>
Files already downloaded and verified<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Detected 1 new nodes from group_rank=0; will restart worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1114254 closing signal SIGTERM<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous’ing worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:<br>
restart_count=0<br>
master_addr=slave6<br>
master_port=55107<br>
group_rank=0<br>
group_world_size=2<br>
local_ranks=[0]<br>
role_ranks=[0]<br>
global_ranks=[0]<br>
role_world_sizes=[2]<br>
global_world_sizes=[2]</p>
<p>INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_2bl7vzba/1_8s8bqrfc/attempt_0/0/error.json<br>
Files already downloaded and verified<br>
Files already downloaded and verified</p>
<p>In another host, I enter the same command:   <em>LOGLEVEL=INFO torchrun --nnodes=1:2 --nproc_per_node=1 --max_restarts=3 --rdzv_id=1 --rdzv_backend=c10d --rdzv_endpoint=slave6:1234 main.py --if_name enp4s0 --batch_size 512</em>  and the stdout are listed below:</p>
<p>INFO:torch.distributed.launcher.api:Starting elastic_operator with launch configs:<br>
entrypoint       : main.py<br>
min_nodes        : 1<br>
max_nodes        : 2<br>
nproc_per_node   : 1<br>
run_id           : 1<br>
rdzv_backend     : c10d<br>
rdzv_endpoint    : slave6:1234<br>
rdzv_configs     : {‘timeout’: 900}<br>
max_restarts     : 3<br>
monitor_interval : 5<br>
log_dir          : None<br>
metrics_cfg      : {}</p>
<p>INFO:torch.distributed.elastic.agent.server.local_elastic_agent:log directory set to: /tmp/torchelastic_x_83j4wn/1_wimfrwjn<br>
INFO:torch.distributed.elastic.agent.server.api:[default] starting workers for entrypoint: python3<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous’ing worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:<br>
restart_count=0<br>
master_addr=slave6<br>
master_port=55107<br>
group_rank=1<br>
group_world_size=2<br>
local_ranks=[0]<br>
role_ranks=[1]<br>
global_ranks=[1]<br>
role_world_sizes=[2]<br>
global_world_sizes=[2]</p>
<p>INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_x_83j4wn/1_wimfrwjn/attempt_0/0/error.json<br>
Files already downloaded and verified<br>
Files already downloaded and verified</p>
<p>In this host, the exception thrown by the application is logged as below:</p>
<p>05/03/2022 10:11:51 AM - INFO - Added key: store_based_barrier_key:1 to store for rank: 1<br>
05/03/2022 10:11:51 AM - INFO - Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.<br>
05/03/2022 10:11:55 AM - ERROR - NCCL error in: …/torch/csrc/distributed/c10d/NCCLUtils.hpp:125, internal error, NCCL version 21.0.3<br>
ncclInternalError: Internal check failed. This is either a bug in NCCL or due to memory corruption<br>
Traceback (most recent call last):<br>
File “main.py”, line 74, in <br>
model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[local_rank])<br>
File “/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py”, line 641, in <strong>init</strong><br>
dist._verify_params_across_processes(self.process_group, parameters)<br>
RuntimeError: NCCL error in: …/torch/csrc/distributed/c10d/NCCLUtils.hpp:125, internal error, NCCL version 21.0.3<br>
ncclInternalError: Internal check failed. This is either a bug in NCCL or due to memory corruption</p>
<p>It looks strange that my log in the first node doesn’t record any exceptions.</p>
<p>Thanks for your patient help.</p>

Kiuk_Chung: <p>could you run this as a single node job on the host that throws the error? e.g.</p>
<pre><code class="lang-auto"># assuming the host has two gpus if not make --nproc_per_node=1
LOGLEVEL=INFO torchrun --nnodes=1 --nproc_per_node=2  --rdzv_id=1 --rdzv_backend=c10d --rdzv_endpoint=localhost:29500 main.py --if_name enp4s0 --batch_size 512
</code></pre>

zizhao.mo: <p>Hi Kiuk,</p>
<p>I run this command in one host, and no error occurs. 2-worker training starts successfully.</p>
<p>Thanks</p>