What closes Rendevezvous in torch elastic?

username: aguirguis
cooked: <p>Hello,</p>
<p>In some weird cases (with scaling up and down), I get the following error:<br>
<code>{"name": "torchelastic.worker.status.FAILED", "source": "AGENT", "timestamp": 0, "metadata": {"run_id": "elastic-imagenet-wxmlb", "global_rank": null, "group_rank": null, "worker_id": null, "role": "default", "hostname": "elastic-imagenet-wxmlb-worker-7", "state": "FAILED", "total_run_time": 0, "rdzv_backend": "etcd", "raw_error": "Traceback (most recent call last):\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/launcher/api.py\", line 238, in launch_agent\n    result = agent.run()\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/metrics/api.py\", line 125, in wrapper\n    result = f(*args, **kwargs)\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py\", line 700, in run\n    result = self._invoke_run(role)\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py\", line 822, in _invoke_run\n    self._initialize_workers(self._worker_group)\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/metrics/api.py\", line 125, in wrapper\n    result = f(*args, **kwargs)\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py\", line 670, in _initialize_workers\n    self._rendezvous(worker_group)\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/metrics/api.py\", line 125, in wrapper\n    result = f(*args, **kwargs)\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py\", line 530, in _rendezvous\n    store, group_rank, group_world_size = spec.rdzv_handler.next_rendezvous()\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/rendezvous/etcd_rendezvous.py\", line 152, in next_rendezvous\n    rdzv_version, rank, world_size = self._rdzv_impl.rendezvous_barrier()\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/rendezvous/etcd_rendezvous.py\", line 284, in rendezvous_barrier\n    return self.init_phase()\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/rendezvous/etcd_rendezvous.py\", line 346, in init_phase\n    raise RendezvousClosedError()\ntorch.distributed.elastic.rendezvous.api.RendezvousClosedError\n", "metadata": "{\"group_world_size\": null, \"entry_point\": \"python\"}", "agent_restarts": 0}}</code><br>
It is not clear from the logs nor the error message what closed the Rendezvous backend. This error forces the whole task to fail and without enough understanding to what’s going on I cannot fix this issue.<br>
Any help?</p>
<p>Thank you very much</p>

username: H-Huang
cooked: <p>How are you scaling up and scaling down? The RendezvousClosedError is raised when the whole gang is not accepting anymore rendezvous (for example when a job if finished). Perhaps one of your nodes left the group and now is trying to rejoin when the job finished completing? It would be useful to have an example to reproduce what you are seeing.</p>
<p>cc <a class="mention" href="/u/cbalioglu">@cbalioglu</a></p>

username: aguirguis
cooked: <p>Actually I realized now that this happens even without scaling. The output of the other node is:<br>
<code> INFO 2021-08-03 11:20:51,599 Rendezvous timeout occured in EtcdRendezvousHandler {"name": "torchelastic.worker.status.FAILED", "source": "AGENT", "timestamp": 0, "metadata": {"run_id": "elastic-imagenet-dzq2v", "global_rank": null, "group_rank": null, "worker_id": null, "role": "default", "hostname": "elastic-imagenet-dzq2v-worker-0", "state": "FAILED", "total_run_time": 901, "rdzv_backend": "etcd", "raw_error": "Traceback (most recent call last):\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/launcher/api.py\", line 238, in launch_agent\n    result = agent.run()\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/metrics/api.py\", line 125, in wrapper\n    result = f(*args, **kwargs)\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py\", line 700, in run\n    result = self._invoke_run(role)\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py\", line 822, in _invoke_run\n    self._initialize_workers(self._worker_group)\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/metrics/api.py\", line 125, in wrapper\n    result = f(*args, **kwargs)\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py\", line 670, in _initialize_workers\n    self._rendezvous(worker_group)\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/metrics/api.py\", line 125, in wrapper\n    result = f(*args, **kwargs)\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py\", line 530, in _rendezvous\n    store, group_rank, group_world_size = spec.rdzv_handler.next_rendezvous()\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/rendezvous/etcd_rendezvous.py\", line 152, in next_rendezvous\n    rdzv_version, rank, world_size = self._rdzv_impl.rendezvous_barrier()\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/rendezvous/etcd_rendezvous.py\", line 284, in rendezvous_barrier\n    return self.init_phase()\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/rendezvous/etcd_rendezvous.py\", line 349, in init_phase\n    return self.join_phase(state[\"version\"])\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/rendezvous/etcd_rendezvous.py\", line 390, in join_phase\n    active_version = self.wait_for_peers(expected_version)\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/rendezvous/etcd_rendezvous.py\", line 563, in wait_for_peers\n    active_version, state = self.try_wait_for_state_change(\n  File \"/job/.local/lib/python3.8/site-packages/torch/distributed/elastic/rendezvous/etcd_rendezvous.py\", line 859, in try_wait_for_state_change\n    raise RendezvousTimeoutError()\ntorch.distributed.elastic.rendezvous.api.RendezvousTimeoutError\n", "metadata": "{\"group_world_size\": null, \"entry_point\": \"python\"}", "agent_restarts": 0}}</code></p>
<p>Probably this node closed the Rendezvous?</p>

username: H-Huang
cooked: <p>Can you try using the new <a href="https://pytorch.org/docs/master/elastic/rendezvous.html#torch.distributed.elastic.rendezvous.c10d_rendezvous_backend.C10dRendezvousBackend" class="inline-onebox" rel="noopener nofollow ugc">Rendezvous — PyTorch master documentation</a> over the etcd rendezzvous and see if you still run into the same error?</p>

username: aguirguis
cooked: <p>Would you please explain how I can do this?<br>
It seems I need to replace <code>--rdzv_backend=etcd</code> in the <code>python -m torchelastic.distributed.launch</code> command with something, right? What this thing should be?</p>

username: H-Huang
cooked: <p>Correct, you replace replace <code>--rdzv_backend=etcd</code> with <code>--rdzv_backend=c10d</code>. Command will look something like</p>
<pre><code class="lang-auto">python -m torch.distributed.run
    --nnodes=1:4
    --nproc_per_node=$NUM_TRAINERS
    --rdzv_id=$JOB_ID
    --rdzv_backend=c10d
    --rdzv_endpoint=$HOST_NODE_ADDR
    YOUR_TRAINING_SCRIPT.py (--arg1 ... train script args...)
</code></pre>
<p>docs: <a href="https://pytorch.org/docs/master/elastic/run.html#module-torch.distributed.run" class="inline-onebox" rel="noopener nofollow ugc">torch.distributed.run (Elastic Launch) — PyTorch master documentation</a></p>

username: aguirguis
cooked: <p>Thanks for the answer.<br>
What should I run on <code>$HOST_NODE_ADDR</code> (as replacement of <code>etcd</code>)?</p>

username: H-Huang
cooked: <p>If you are doing single node training you can use <code>localhost</code> otherwise you can select one of your machines and find it’s <code>hostname</code> and use that.</p>

username: aguirguis
cooked: <p>I am using the <a href="https://github.com/pytorch/elastic/tree/5c5accdb19636f543c04e26cea9cc773bb0f781f/kubernetes" rel="noopener nofollow ugc">Kubernetes controller</a> for my experiments. Is this new backend also supported by this controller? If yes, is there any tutorial I could follow regarding this setup?</p>

username: sachin_chandra
cooked: <p>You can add the --rdzv_backend=c10d flag in the args when you start your job using the operator</p>

username: sachin_chandra
cooked: <p>Hey <a class="mention" href="/u/aguirguis">@aguirguis</a>  I just wrote a tutorial for setting up YoloV5 using Pytorch and the C10d backend. You can follow this guide.</p>
<p><a href="https://medium.com/p/8a4f07a77cf" class="onebox" target="_blank" rel="noopener nofollow ugc">https://medium.com/p/8a4f07a77cf</a></p>
<p>You can refer the direct YAML I used here: <a href="https://gist.github.com/sachin-netbook/072e384f8c1e4ae3074ed4a82658ff89#file-trainer-yaml" class="inline-onebox" rel="noopener nofollow ugc">Trainer.Yaml · GitHub</a></p>

username: f_g
cooked: <p>hi <a class="mention" href="/u/h-huang">@H-Huang</a>, regarding your comment on “The RendezvousClosedError is raised when the whole gang is not accepting anymore rendezvous (for example when a job if finished). Perhaps one of your nodes left the group and now is trying to rejoin when the job finished completing?”</p>
<p>if a node left and tries to rejoin, but the job hasn’t finished, should it still be accepted? I just saw the <code>RendezvousClosedError</code> when one node tried to rejoin, but the job is still running. This then caused the job to fail.</p>

username: Leandro_Libutti
cooked: <p>I have the same problem. When I scaled down my app from 2 to 1 nodes, the rendezvous broke with “RendezvousClosedError” message. I’m sending a stop signal to the process of the node that I want to down, is that a problem? How can I test that the elasticity really works without that error?</p>