Using Kubernetes API server as Rendezvous backend

Sathyanarayanan: <p>Hi team, is it a good idea to use kubernetes API server as rendezvous backend for running multi-node distributed jobs within a kuberenetes cluster? I have created the following issue on github. Please provide your thoughts on what kind of effects it will have on the storage and performance of API server.</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/121486">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/121486" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/121486" target="_blank" rel="noopener nofollow ugc">Kubernetes API server as Rendezvous backend</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2024-03-08" data-time="11:13:15" data-timezone="UTC">11:13AM - 08 Mar 24 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/sathyanarays" target="_blank" rel="noopener nofollow ugc">
          <img alt="sathyanarays" src="https://avatars.githubusercontent.com/u/107846526?v=4" class="onebox-avatar-inline" width="20" height="20">
          sathyanarays
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          oncall: distributed
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">### 🚀 The feature, motivation and pitch

The `torchrun` supports `etcd` and `c10<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">d` as rendezvous backend. If torch run jobs need to be run in kubernetes clusters, we need to rely on `etcd` (either self hosted or hosted by the cloud provider). The Kubernetes API server is also based on `etcd`. If we can implement a `RendezvousHandler` that uses Kubernetes API server as rendezvous backend, we can run `torchrun` without any external dependencies on Kubernetes clusters.

Please provide your thoughts on this!

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @chauhang</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>