Synchronization steps in distributed data parallel

username: pooya_khandel
cooked: <p>As far as I understood, the <code>DistributedDataParallel</code> module performs gradient synchronization between different nodes automatically, one thing I don’t understand clearly is when this synchronization is done exactly?</p>
<p>For example, the below snippet is from <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="nofollow noopener">GETTING STARTED WITH DISTRIBUTED DATA PARALLEL</a> PyTorch documentation with small change:</p>
<pre><code class="lang-auto">def demo_basic(rank, world_size):
    setup(rank, world_size)

    # setup devices for this process, rank 1 uses GPUs [0, 1, 2, 3] and
    # rank 2 uses GPUs [4, 5, 6, 7].
    n = torch.cuda.device_count() // world_size
    device_ids = list(range(rank * n, (rank + 1) * n))

    # create model and move it to device_ids[0]
    model = ToyModel().to(device_ids[0])
    # output_device defaults to device_ids[0]
    ddp_model = DDP(model, device_ids=device_ids)

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    optimizer.zero_grad()
    outputs = ddp_model(torch.randn(20, 10))
    labels = torch.randn(20, 5).to(device_ids[0])

    loss = loss_fn(outputs, labels)
    loss.backward()
#    loss_fn(outputs, labels).backward()

    optimizer.step()

    cleanup()
</code></pre>
<p>In the above example, Is computed loss synchronized among all nodes? i.e., Does loss value represents only each node loss or it is averaged among all nodes?</p>

username: albanD
cooked: <p>The <code>DDP()</code> wrapper takes care of all the synchronizations and offer a <code>nn.Module</code> like api so that you can use it transparently.</p>

username: meilu_zhu
cooked: <p>Hi, Do you know where the script of gradients synchronization during backward is in pytorch source code?</p>

username: pietern
cooked: <p><a class="mention" href="/u/meilu_zhu">@meilu_zhu</a> The DDP wrapper creates a <code>c10d.Reducer</code> which is responsible for concatenating multiple gradients into larger buckets and reducing them. You can find the source code at <code>torch/csrc/distributed/c10d/reducer.cpp</code>.</p>

username: meilu_zhu
cooked: <p>Hi, <a class="mention" href="/u/pietern">@pietern</a>. Thanks for your answer.  “DistributedDataParallel” automatically averages the gradient when calling <code>loss.backward()</code>, But I didn’t find the corresponding script about how calling <code>loss.backward()</code> triggers <code>torch/csrc/distributed/c10d/reducer.cpp</code> to concatenate multiple gradients in pytorch source code? Could you tell me where it is, please?</p>

username: hhxx
cooked: <p>In <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html</a>, the code <strong>demo_checkpoint</strong> really confused me. There are two main quetions:<br>
(1) In <strong>demo_checkpoint</strong>, all processes need to be synchronized by loading the same checkponts saved by process-0. If it is necessary when we train model with  DistributedDataParallel.?<br>
(2) If it is necessary to synchronize model across multi nodes and gpus by loading the same checkpoint, how can node-B load checkpoints saved in node-A?<br>
I don’t know if I understand <strong>demo_checkpoint</strong> in a right way. Could you please help answer this question? Thanks !</p>

username: mrshenli
cooked: <p><a class="mention" href="/u/hhxx">@hhxx</a></p>
<blockquote>
<p>(1) In  <strong>demo_checkpoint</strong> , all processes need to be synchronized by loading the same checkponts saved by process-0. If it is necessary when we train model with DistributedDataParallel.?</p>
</blockquote>
<p>No, this is not necessary. This is only useful when your training job take very long and can crash in the middle. You can then use the checkpoint to recover instead of starting over from scratch.</p>
<blockquote>
<p>(2) If it is necessary to synchronize model across multi nodes and gpus by loading the same checkpoint, how can node-B load checkpoints saved in node-A?<br>
I don’t know if I understand  <strong>demo_checkpoint</strong>  in a right way. Could you please help answer this question? Thanks !</p>
</blockquote>
<p>The recovery scheme should be application-dependent. That tutorial demonstrates single-machine multi-GPU DDP with checkpointing. So, all DDP processes can read from the same file. If you need checkpoint, and if your training spans multiple machines, you can load it from rank0 and then broadcast it to other ranks using <code>torch.distributed.braodcast</code>.</p>
<p>BTW, we probably should call that tutorial “intermediate”, and use <a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="noopener nofollow ugc">this one</a> as a starting example.</p>

username: hhxx
cooked: <p>Thanks very much! It is very clear.<br>
Another question is that  when we train a model with DistributedDataparallel, DistributedSampler is suggested to use together.  If we need add <strong>sampler.set_epoch(epoch)</strong> before each epoch start according to <a href="https://github.com/PyTorchLightning/pytorch-lightning/issues/224" rel="nofollow noopener">set_epoch for DistributedSampler</a>.</p>

username: mrshenli
cooked: <p>cc <a class="mention" href="/u/vincentqb">@vincentqb</a> on <code>DistributedSampler</code> question <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>