Distributed Machine Learning on multiple cores

username: abodh_ltd
cooked: <p>I was looking into training machine learning models in multiple cores. To be more clear, suppose I have “N” machine learning units (for eg. three layered neural network [in-hid-out] ). Each of the units are identical to each other. However, I want to train each network with different input of same nature (for eg. If I have 10 machine learning units with MNIST data as input, each of the 10 units will be trained on different sets of data). You can think of it as training the networks with MNIST in 10 geographically dispersed location where we are not sure which network will have what set of inputs. However, at somepoint I want to communicate between the machine learning models while updating the weights. I want to distribute same weights to all the models. For people who know federated learning, its like applying federated learning in multiple CPU/GPU cores.</p>
<p>Is it possible to do something like this in multiple cores of a CPU or GPU? Or is there any documentation that you can provide me?</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/rasbt">@rasbt</a></p>

username: mrshenli
cooked: <p>IIUC, this is typical <a href="https://pytorch.org/docs/master/notes/ddp.html" rel="nofollow noopener"><code>DistributedDataParallel</code></a> training? If so, yes, PyTorch natively support that. Here is <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="nofollow noopener">another tutorial</a>.</p>

username: abodh_ltd
cooked: <p>I am not sure what to call this. But its kind of distributed training where each of the neural network in different processes communicate while updating the weight. Would the approaches you provided do the same?</p>

username: abodh_ltd
cooked: <p>P.S. I don’t want to distribute a single set of input to multiple nodes/processes. Suppose I have 2 nodes N1 and N2, then I need to send a set of input for N1 and another set of input for N2, which is different than the set for N1 (and not collected in batches from a common data set). I am not sure if I explained it correctly. Sorry about that.</p>

username: mrshenli
cooked: <blockquote>
<p>But its kind of distributed training where each of the neural network in different processes communicate while updating the weight</p>
</blockquote>
<p>Does it have to communicate parameters instead of gradients? If all you need is to keep the parameters on all processes in sync, communicating gradients should be sufficient I think.</p>
<blockquote>
<p>I don’t want to distribute a single set of input to multiple nodes/processes.</p>
</blockquote>
<p>Yep, DDP does not split inputs, instead each process need to prepare its own inputs.</p>
<p>One question is, does the parameters/gradients communication occur in a synchronized or asynchronized fashion? “Synchronized” means all processes communicate at exactly the same time, while asynchronized can be something like gossip. DDP only works for synchronized use cases. If you need asynchronized communication, you can directly use c10d (allreduce, allgather, broadcast, etc.) and create multiple sub-groups to perform communicatioin.</p>

username: abodh_ltd
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f4682282a262d04e764a930f227012c92ddbee7.png" data-download-href="https://discuss.pytorch.org/uploads/default/8f4682282a262d04e764a930f227012c92ddbee7" title="distributed"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f4682282a262d04e764a930f227012c92ddbee7_2_514x500.png" alt="distributed" data-base62-sha1="krtlcbiT38iEHdqysulENbPxkDt" width="514" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f4682282a262d04e764a930f227012c92ddbee7_2_514x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f4682282a262d04e764a930f227012c92ddbee7_2_771x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f4682282a262d04e764a930f227012c92ddbee7.png 2x" data-dominant-color="232121"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">distributed</span><span class="informations">980×953 97.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am not a Machine Learning savvy, so please mind the errors while I write this:</p>
<p>I think the figure will explain this right. Since I will be gathering the weights of all the networks residing at different processes, I need to pass the weight parameters right? And the weight gathering needs to happen asynchronously, at different point of time. But to get a start, at this point, we can assume that the weight gathering happens synchronously, at the same point of time.</p>
<p>Furthermore, the gathered weights need to be averaged at the root process and the aggregated weight should be broadcasted again to the networks for next epoch. I am not sure if you get this due to my weird way of explaining things but thanks for being modest and replying promptly.</p>
<p>If you could tell me a specific way to handle this, I could narrow down my scope of researching the documents and tutorials. Thank you again.</p>

username: abodh_ltd
cooked: <p>P.S. each of the networks will have local training epochs (hence update the weights in different period of time)</p>

username: osalpekar
cooked: <p>Based on the diagram and explanations, it seems like you are trying to train the network, with each node training on its own data, and ensuring that the parameters stay in sync across all the nodes. If this is the case, you will want to communicate the gradients, and not the weights.</p>
<p>DDP works perfectly for synchronous distributed training. Each node will independently perform the forward and backward pass on its own batch of data. Then, each node will send its computed gradients to every other node. Once each node has the gradients for all other nodes, they will independently average all the gradients and run the optimizer to perform the gradient update. One note is that there is no “root” process responsible for aggregating the gradients (what you’re describing is similar to a parameter server). In DDP, the nodes communicate with each to exchange gradients.</p>
<p>For asynchronous distributed training, you can use the c10d communication primitives as <a class="mention" href="/u/mrshenli">@mrshenli</a> described above.</p>