Research Distributed learning platform

sakh251: <p>Hello,<br>
My research is about distributed deep learning, and I am looking for a research developing platform in which new ideas could be implemented. Regarding complex architecture, we should have more access to communications between nodes and GPUs.<br>
I found TF-Replicator from Tensorflow, I saw Pytorch RPC and DDP APIs, and also Ray that is a new designed intesteing platform for emerged AI.</p>
<p>I could found any example and tutorial (Good ones) for TF-Repliicato or Ray. However, Pytorch has some good documents and I feel it gives more control over how we implement more complicated architecture and communication schema.</p>
<p>Therefore It would be great if others can also share their thoughts.<br>
Thank you</p>

mrshenli: <p>Hey <a class="mention" href="/u/sakh251">@sakh251</a></p>
<p>I just wanna mention one relevant WIP project that we are working on. We are running internal reviews on the code-level design on it, and should be able to share on Github in the next few weeks. This should help make customizing DDP a lot easier.</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/37002" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/37002" target="_blank" rel="noopener nofollow ugc">[RFC] Modularize DistributedDataParallel</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-04-21" data-time="16:09:12" data-timezone="UTC">04:09PM - 21 Apr 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/mrshenli" target="_blank" rel="noopener nofollow ugc">
          <img alt="mrshenli" src="https://avatars2.githubusercontent.com/u/16999635?v=4" class="onebox-avatar-inline" width="20" height="20">
          mrshenli
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Summary
This project aims at decomposing existing DistributedDataParallel (DDP) implementation into multiple smaller pluggable and customizable building blocks. So that applications can...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">feature</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">oncall: distributed</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


pritamdamania87: <aside class="quote no-group" data-username="sakh251" data-post="1" data-topic="100998">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/bbce88/48.png" class="avatar"> sakh251:</div>
<blockquote>
<p>I could found any example and tutorial (Good ones) for TF-Repliicato or Ray</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/sakh251">@sakh251</a> In terms of TF-Replicator (which is now part of tf.distribute.Strategy) are you referring to dedicated support for Parameter Server based training?</p>

sakh251: <p><a class="mention" href="/u/pritamdamania87">@pritamdamania87</a><br>
I am looking for some high level api which can control the behaviour of learning. For example sending more data along with model or gradient. Or output of one network should feed to other network like GANs or Autoencoder when networks are on different machine.  In this case we should have something like parameter server with more control. Now the question is which of these platforms provide more flexible high level api that is suitable for researchers? Not just use the developed strategies.</p>

sakh251: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a> it sounds interesting, I hope it would be available soon.</p>