Elastic agents cannot properly shutdown/restart after failure

Yoanis_Gil: <p>I am trying to run a distributed training using the Elastic Job Controller (from here: <a href="https://github.com/pytorch/elastic/tree/master/kubernetes" rel="noopener nofollow ugc">https://github.com/pytorch/elastic/tree/master/kubernetes</a>) and while testing the distributed agent does not seem to be able to shutdown themselves properly. Here is a stacktrace of what happens, eventually, when one of the pods die:</p>
<blockquote>
<p>[E ProcessGroupGloo.cpp:136] Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.<br>
Error executing job with overrides: [‘learning_rate=single’, ‘model.architecture=efflitenet40’, ‘dataset.val.dataset_dpath=/data/wild_v4_tiny_fr_val’, ‘optimization.batch_size=256’, ‘backend.num_nodes=4’, ‘experiment_id=yoanis-autoresume-01’, ‘logging.ckpt_nsteps=500’, ‘logging.run_id=c61a45bf-6a05-4435-944b-ecc895b75a19’, ‘backend.num_nodes=4’, ‘resume_training.auto_resume=True’, ‘backend.wandb_group=multinode-c61a45bf-6a05-4435-944b-ecc895b75a19’, ‘backend.num_cores=1’]<br>
Traceback (most recent call last):<br>
File “pl_trainer.py”, line 422, in main<br>
trainer.fit(pl_model, pl_data)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py”, line 768, in fit<br>
self._call_and_handle_interrupt(<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py”, line 721, in _call_and_handle_interrupt<br>
return trainer_fn(*args, **kwargs)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py”, line 809, in _fit_impl<br>
results = self._run(model, ckpt_path=self.ckpt_path)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py”, line 1234, in _run<br>
results = self._run_stage()<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py”, line 1321, in _run_stage<br>
return self._run_train()<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py”, line 1351, in _run_train<br>
self.fit_loop.run()<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/base.py”, line 204, in run<br>
self.advance(*args, **kwargs)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/fit_loop.py”, line 269, in advance<br>
self._outputs = self.epoch_loop.run(self._data_fetcher)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/base.py”, line 204, in run<br>
self.advance(*args, **kwargs)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py”, line 208, in advance<br>
batch_output = self.batch_loop.run(batch, batch_idx)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/base.py”, line 204, in run<br>
self.advance(*args, **kwargs)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/batch/training_batch_loop.py”, line 88, in advance<br>
outputs = self.optimizer_loop.run(split_batch, optimizers, batch_idx)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/base.py”, line 204, in run<br>
self.advance(*args, **kwargs)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py”, line 203, in advance<br>
result = self._run_optimization(<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py”, line 256, in _run_optimization<br>
self._optimizer_step(optimizer, opt_idx, batch_idx, closure)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py”, line 369, in _optimizer_step<br>
self.trainer._call_lightning_module_hook(<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py”, line 1593, in _call_lightning_module_hook<br>
output = fn(*args, **kwargs)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/core/lightning.py”, line 1625, in optimizer_step<br>
optimizer.step(closure=optimizer_closure)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/core/optimizer.py”, line 168, in step<br>
step_output = self._strategy.optimizer_step(self._optimizer, self._optimizer_idx, closure, **kwargs)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/strategies/ddp.py”, line 278, in optimizer_step<br>
optimizer_output = super().optimizer_step(optimizer, opt_idx, closure, model, **kwargs)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/strategies/strategy.py”, line 193, in optimizer_step<br>
return self.precision_plugin.optimizer_step(model, optimizer, opt_idx, closure, **kwargs)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/plugins/precision/native_amp.py”, line 85, in optimizer_step<br>
closure_result = closure()<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py”, line 148, in <strong>call</strong><br>
self._result = self.closure(*args, **kwargs)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py”, line 134, in closure<br>
step_output = self._step_fn()<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py”, line 427, in _training_step<br>
training_step_output = self.trainer._call_strategy_hook(“training_step”, *step_kwargs.values())<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py”, line 1763, in _call_strategy_hook<br>
output = fn(*args, **kwargs)<br>
File “/usr/local/lib/python3.8/dist-packages/pytorch_lightning/strategies/ddp.py”, line 341, in training_step<br>
return self.model(*args, **kwargs)<br>
File “/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py”, line 1110, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py”, line 955, in forward<br>
self._sync_buffers()<br>
File “/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py”, line 1602, in _sync_buffers<br>
self._sync_module_buffers(authoritative_rank)<br>
File “/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py”, line 1606, in _sync_module_buffers<br>
self._default_broadcast_coalesced(authoritative_rank=authoritative_rank)<br>
File “/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py”, line 1627, in _default_broadcast_coalesced<br>
self._distributed_broadcast_coalesced(<br>
File “/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py”, line 1543, in _distributed_broadcast_coalesced<br>
dist._broadcast_coalesced(<br>
RuntimeError: Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.<br>
Original exception:<br>
[…/third_party/gloo/gloo/transport/tcp/pair.cc:598] Connection closed by peer [10.116.5.70]:40678</p>
<p>Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.<br>
wandb: Waiting for W&amp;B process to finish… (failed 1). Press Control-C to abort syncing.<br>
wandb:<br>
wandb: Synced playful-jazz-804: _____<br>
wandb: Synced 5 W&amp;B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)<br>
wandb: Find logs at: ./wandb/run-20220421_193319-ny9ee8dv/logs</p>
<h1>ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 421) of binary: /usr/bin/python<br>
INFO:torch.distributed.elastic.agent.server.api:Local worker group finished (FAILED). Waiting 300 seconds for other agents to finish<br>
ERROR:torch.distributed.elastic.agent.server.api:Error waiting on exit barrier. Elapsed: 300.1109097003937 seconds<br>
Traceback (most recent call last):<br>
File “/usr/local/lib/python3.8/dist-packages/torch/distributed/elastic/agent/server/api.py”, line 906, in _exit_barrier<br>
store_util.barrier(<br>
File “/usr/local/lib/python3.8/dist-packages/torch/distributed/elastic/utils/store.py”, line 67, in barrier<br>
synchronize(store, data, rank, world_size, key_prefix, barrier_timeout)<br>
File “/usr/local/lib/python3.8/dist-packages/torch/distributed/elastic/utils/store.py”, line 53, in synchronize<br>
agent_data = get_all(store, key_prefix, world_size)<br>
File “/usr/local/lib/python3.8/dist-packages/torch/distributed/elastic/utils/store.py”, line 31, in get_all<br>
data = store.get(f"{prefix}{idx}“)<br>
File “/usr/local/lib/python3.8/dist-packages/torch/distributed/elastic/rendezvous/etcd_store.py”, line 75, in get<br>
raise LookupError(f"Key {key} not found in EtcdStore”)<br>
LookupError: Key torchelastic/agent/terminal_state3 not found in EtcdStore<br>
INFO:torch.distributed.elastic.multiprocessing.errors:local_rank 1 FAILED with no error file. Decorate your entrypoint fn with <span class="mention">@record</span> for traceback info. See: _____<br>
Traceback (most recent call last):<br>
File “/usr/lib/python3.8/runpy.py”, line 194, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “/usr/lib/python3.8/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py”, line 728, in <br>
main()<br>
File “/usr/local/lib/python3.8/dist-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 345, in wrapper<br>
return f(*args, **kwargs)<br>
File “/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py”, line 724, in main<br>
run(args)<br>
File “/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py”, line 715, in run<br>
elastic_launch(<br>
File “/usr/local/lib/python3.8/dist-packages/torch/distributed/launcher/api.py”, line 131, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/usr/local/lib/python3.8/dist-packages/torch/distributed/launcher/api.py”, line 245, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</h1>
<h2>pl_trainer.py FAILED</h2>
<h2>Failures:<br>
&lt;NO_OTHER_FAILURES&gt;</h2>
<p>Root Cause (first observed failure):<br>
[0]:<br>
time      : 2022-04-21_19:48:31<br>
host      : job-c61a45bf-6a05-4435-944b-ecc895b75a19-worker-0<br>
rank      : 1 (local_rank: 0)<br>
exitcode  : 1 (pid: 421)<br>
error_file: &lt;N/A&gt;</p>
</blockquote>
<p>I know this is a very complex issue and more information could be required, but wantet to reach out to the broader community to see if anyone has ran into a similar issue.</p>

cbalioglu: <p><a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a> might help you further.</p>

Kiuk_Chung: <p>From the stack trace it looks like there was a legitimate error on one of the nodes which doesn’t make the workers on other nodes fail (but rather they succeed early). What you are observing is that on the node where the workers who (falsely) finish successfully, the elastic agent will wait for 5min by default for other workers on other nodes to finish and eventually times out b/c the other workers don’t.</p>
<p>Its hard to say what the exact root cause of the error is without looking at the error summary (the last few lines of the log file, those lines that start with <code>Root Cause (first observed failure)...</code>) of all the nodes.</p>
<p>I suggest a few things:</p>
<ol>
<li>torchelastic has been upstreamed to pytorch and the k8s CRD on the elastic repo is now no longer being actively maintained. You can try launching DDP jobs  with TorchX. Here’s the instructions for that
<ol>
<li>Setting up on Kubernetes: <a href="https://pytorch.org/torchx/latest/schedulers/kubernetes.html" class="inline-onebox" rel="noopener nofollow ugc">Kubernetes — PyTorch/TorchX main documentation</a>.</li>
<li>Launching distributed jobs: <a href="https://pytorch.org/torchx/latest/components/distributed.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed — PyTorch/TorchX main documentation</a>
</li>
</ol>
</li>
<li>Turn on error reporting by annotating your trainer main method with <code>torch.distributed.elastic.multiprocessing.errors.record</code> (follow instructions here: <a href="https://pytorch.org/docs/stable/elastic/errors.html#torch.distributed.elastic.multiprocessing.errors.record" class="inline-onebox" rel="noopener nofollow ugc">Error Propagation — PyTorch 1.11.0 documentation</a>). The error summary table should’ve prompted you to do this as well.</li>
<li>Reply to this with the exact commands and parameters you ran with (num nodes, num procs per node, your training script etc)</li>
</ol>

Yoanis_Gil: <p><a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a> thanks for the reply!</p>
<p>To give you a bit more of background, I was running a distributed run with the Elastic Job Controller and ensuring that a training job would be able to resume after one of the pods died, so I was essentially just doing <code>kubectl delete pod POD_ID</code>. The behavior I was observing is that after killing one of the pods, the training job would not resume. But I then realized that I forgot to set <code>--max-restarts</code> to a value higher than 1. Once  I set <code>--max-restarts</code> to 2 or more, everything started working as I expected (training job would resume in the case of a pod or VM dying).</p>
<p>On a side topic, and if I get it right, it seems the official replacement to the TorchElastic Controller is torchx? If that’s the case then it would be really nice to include that reference <a href="https://github.com/pytorch/elastic/blob/master/kubernetes/README.md" rel="noopener nofollow ugc">here</a> and make it more explicit <a href="https://github.com/pytorch/elastic" rel="noopener nofollow ugc">here</a>. We’re using the TorchElastic Controller at the moment and I was just worried that we were using a no longer maintained solution but at the same time, up until now, I was not able to find an official replacement. Thanks for letting me know :)!</p>

Kiuk_Chung: <p>Glad you figured this one out! and thanks for the suggestion about updating the docs in TorchElastic to point to TorchX, just submitted a PR to update the README. <a href="https://github.com/pytorch/elastic/pull/167" class="inline-onebox" rel="noopener nofollow ugc">(torchelastic) update README to point elastic CRD users to TorchX by kiukchung · Pull Request #167 · pytorch/elastic · GitHub</a></p>