Problem in computing loss in multiple cpu distribution training

Sanjayvarma11: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
<span class="mention">@soumith</span> chintala</p>
<p>Hello everyone.My friend and I are trying to implement multiple cpu distributed training in pytorch using python sockets.we looked at pytorch inbuilt functions but we didn’t get any starter code for detecting whether cpu’s are connected or not.so we decided to implement it in sockets</p>
<p>Main problem:</p>
<p>We distributed data perfectly but I have a theoretical doubt.So my plan is I will compute loss for distributed cpu’s for their batches and send all those losses to main computer and compute average.</p>
<p>Now I will send those average loss to all the distributed cpu’s and update model according to that loss so that model wts will be synchronized.but the avg loss which I am getting from server computer will not no relationship graph wrt to the model stored in the computer.so how can I tackle this??</p>
<p>Main doubt is:</p>
<p>Can we update model wrt to loss that has no relation wrt to model??</p>

pritamdamania87: <p>Typically you want to run the forward and backward pass on each process separately and then average the gradients across all processes and then run the optimizer independently on each process.</p>
<p>I’m wondering what is the reason you’re trying to build this yourself. PyTorch has a <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">Distributed<br>
Data Parallel</a> module, which does all of this for you.</p>

Sanjayvarma11: <p>Hi sir.Thank you for posting.Yesterday we tried it and we forgot to update here.</p>
<p><strong>First problem</strong><br>
Coming to your question we looked at various articles but we are so confused on using this,Like we have  to find initially whether pytorch identifes our connected computers.So can you please provide a starter code to check that??</p>
<p><strong>Second problem</strong><br>
So if our main computer runs training code then what our client computers should run in order to accept and process data.We dont have any idea.We connected our computers using LAN cable.Thank you sir for answering…)</p>

pritamdamania87: <aside class="quote no-group" data-username="Sanjayvarma11" data-post="3" data-topic="104859">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sanjayvarma11/48/20339_2.png" class="avatar"> Sanjayvarma11:</div>
<blockquote>
<p>Coming to your question we looked at various articles but we are so confused on using this,Like we have to find initially whether pytorch identifes our connected computers.So can you please provide a starter code to check that??</p>
</blockquote>
</aside>
<p>You can find a tutorial for this here: <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.1.1+cu121 documentation</a>. In particular the <code>init_process_group</code> part that is called on every node is how PyTorch identifies all the connected nodes.</p>
<aside class="quote no-group" data-username="Sanjayvarma11" data-post="3" data-topic="104859">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sanjayvarma11/48/20339_2.png" class="avatar"> Sanjayvarma11:</div>
<blockquote>
<p>So if our main computer runs training code then what our client computers should run in order to accept and process data.We dont have any idea.We connected our computers using LAN cable.Thank you sir for answering…)</p>
</blockquote>
</aside>
<p>To use DDP, all your nodes would actually run the same code and train the same model on all nodes. DDP would take care of gradient synchronization across all nodes to ensure all the models are in sync.</p>