Multi-node (multiple GPUs per node) training with model parallelism

username: matrix
cooked: <p>I want to train a large network using model parallelism on multiple machines (multiple GPUs per machine),<br>
for that I am following this article</p>
<p><a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#combine-ddp-with-model-parallelism" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#combine-ddp-with-model-parallelism</a></p>
<p>This article doesn’t set up any multi machine cluster, so how will it train on multiple machines? Also I am not able to understand following terms in my scenario,</p>
<p>world size<br>
rank<br>
spawn<br>
processes<br>
process group</p>
<p>I have already installed NCCL in all nodes. How can I make it work?</p>

username: conrad
cooked: <p>This is good place to start:</p>
<aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/pytorch/examples/tree/master/imagenet" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars.githubusercontent.com/u/21003710?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="60" height="60">

<h3><a href="https://github.com/pytorch/examples/tree/master/imagenet" target="_blank" rel="noopener nofollow ugc">pytorch/examples</a></h3>

<p><a href="https://github.com/pytorch/examples/tree/master/imagenet" target="_blank" rel="noopener nofollow ugc">master/imagenet</a></p>

  <p><span class="label1">A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc.</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>The example script and README show how to setup multi-node training for ImageNet. You may also want to try out PyTorch Lightning which has a simple API for multi-node training:</p>
<p><a href="https://pytorch-lightning.readthedocs.io/en/stable/multi_gpu.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch-lightning.readthedocs.io/en/stable/multi_gpu.html</a></p>

username: wayi
cooked: <p>If you want to explore model parallelism in a distributed environment, you need to use <a href="https://pytorch.org/docs/master/rpc.html" rel="noopener nofollow ugc">Distributed RPC framework</a>.</p>
<p>The tutorial page of DDP + RPC can be found here:<br>
<a href="https://pytorch.org/tutorials/advanced/rpc_ddp_tutorial.html#" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/rpc_ddp_tutorial.html#</a></p>

username: matrix
cooked: <p>This example is perfect. Though will it work with model parallelism?</p>

username: wayi
cooked: <p>That example shows how to use DDP on multiple nodes, but model parallelism requires RPC in PyTorch.</p>

username: matrix
cooked: <p>In conclusion, single machine model parallelism can be done as shown in the article I listed in my question, multi node training without model parallelism (with DDP) is shown in the example listed by <a class="mention" href="/u/conrad">@conrad</a> &amp; multi node training with model parallelism can only be implemented using PyTorch RPC. Is it right <a class="mention" href="/u/wayi">@wayi</a> ?</p>

username: wayi
cooked: <p>You are totally right!</p>
<p>RPC is the only way to support model parallelism in PyTorch distributed training. There may be some higher level APIs in the future, but they are all RPCs under the hood.</p>