[Elastic Distributed Training] Will the master node be reselected and restarted if the master node fails?

username: logicShu
cooked: <p>When elastic training uses <code>c10d</code> as the backend store and the master node fails, will the program restart?</p>
<p>Will <code>elastic</code> choose a new master node if I use <code>etcd</code> ?</p>

username: pritamdamania87
cooked: <p><a class="mention" href="/u/cbalioglu">@cbalioglu</a> Was wondering if you knew what is the expected behavior here?</p>

username: cbalioglu
cooked: <p>As of today we do no have a builtin failover mechanism, so if the master node fails, it will cause the training to terminate.</p>

username: cbalioglu
cooked: <blockquote>
<p>Will <code>elastic</code> choose a new master node if I use <code>etcd</code> ?</p>
</blockquote>
<p>Sorry, I missed the second question. Yes, if you use <code>etcd</code>, then if the worker on the master node fails, the agents will try to establish another round of rendezvous (up to <code>--max-restarts</code> option you specified).</p>
<p>In summary <code>c10d</code> is ideal if you don’t want to deal with installing and running a 3rd party dependency, <code>etcd</code> is ideal if you care more about fault tolerance and fail over.</p>

username: logicShu
cooked: <p>Thank you very much for your reply!</p>
<p>After reading the source code, I understood some execution mechanisms. Your reply makes me confirm that <code>etcd</code> is a better choice for me.</p>
<p>I will deploy etcd server on a stable cpu machine, so that I can dynamically increase or decrease nodes without worrying about whether or not the master node fails, as long as the etcd server does not fail.</p>

username: logicShu
cooked: <p>I am curious about one more thing.</p>
<p>If I use <code>c10d</code> to run elastic training and set <code>--nproc_per_node</code> to zero on a cpu machine, does the similar function of <code>etcd</code> backend be achieved ?</p>

username: cbalioglu
cooked: <p>Unfortunately not. You will still have a single point of failure even if the c10d store runs on a separate host. If that host fails, you would end up with a failure of the whole job. By the way you would have the same problem if you had a single etcd instance. The advantage you have with etcd is that you can set up a small cluster (2 or more machines) of etcd servers for failover handling.</p>

username: logicShu
cooked: <p>Oh, I see. The most important thing about etcd is its distributed reliable key-value store. When using etcd, it is meaningless to have only one etcd instance.</p>

username: cbalioglu
cooked: <p>Yep, that is correct.</p>

username: logicShu
cooked: <p>Thank you for your patient reply! It’s a pleasure to communicate with you. I hope to discuss with you more in the future. Thank you again!</p>

username: cbalioglu
cooked: <p>Thanks for the kind words! You are welcome!</p>

username: zzibc
cooked: <p>Hi, <a class="mention" href="/u/cbalioglu">@cbalioglu</a> !<br>
If I can ensure that my CPU machine will not fail, can I use <code>c10d</code> as backend and set <code>--nproc_per_node=0</code> on the CPU machine to achieve the similar function of <code>etcd</code> backend? Will agents try to establish another round of rendezvous when the master node fails?</p>