Why the second barrier is used in the DDP tutorial?

username: tengerye
cooked: <p>I am reading the <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="nofollow noopener">DistributedDataparallel tutorial</a>. The last line from the following snippet confuses me:</p>
<pre><code class="lang-auto">   if rank == 0:
        # All processes should see same parameters as they all start from same
        # random parameters and gradients are synchronized in backward passes.
        # Therefore, saving it in one process is sufficient.
        torch.save(ddp_model.state_dict(), CHECKPOINT_PATH)

    # Use a barrier() to make sure that process 1 loads the model after process
    # 0 saves it.
    dist.barrier()
    # configure map_location properly
    map_location = {'cuda:%d' % 0: 'cuda:%d' % rank}
    ddp_model.load_state_dict(
        torch.load(CHECKPOINT_PATH, map_location=map_location))

    optimizer.zero_grad()
    outputs = ddp_model(torch.randn(20, 10))
    labels = torch.randn(20, 5).to(rank)
    loss_fn = nn.MSELoss()
    loss_fn(outputs, labels).backward()
    optimizer.step()

    # Use a barrier() to make sure that all processes have finished reading the
    # checkpoint
    **dist.barrier()**
</code></pre>
<ol>
<li>If the last line is used to ensure all processes finish reading, why does not it directly follow <code>ddp_model.load_state_dict</code>?</li>
<li>For each iteration, do we need to call <code>dist.barrier()</code>?</li>
</ol>

username: mrshenli
cooked: <blockquote>
<p>If the last line is used to ensure all processes finish reading, why does not it directly follow  <code>ddp_model.load_state_dict</code> ?</p>
</blockquote>
<p>Good catch! The original reason for adding that barrier is to guard the file deletion below:</p>
<pre data-code-wrap="python"><code class="lang-python">    if rank == 0:
        os.remove(CHECKPOINT_PATH)
</code></pre>
<p>But looking at it again, this barrier is not necessary. Because the <code>backward()</code> on the DDP model is also a synchronization point as it calls <code>AllReduce</code> internally. Let me remove that.</p>
<blockquote>
<p>For each iteration, do we need to call  <code>dist.barrier()</code> ?</p>
</blockquote>
<p>No. Two common reasons of using a <code>barrier</code> are</p>
<ol>
<li>to avoid <code>AllReduce</code> timeout caused by skewed workloads across DDP processes</li>
<li>code after <code>barrier()</code> on rank A depends on the completion of code before <code>barrier()</code> on node B.</li>
</ol>
<p>If none of the bot is a concern in your use case, then <code>barrier</code> shouldn’t be necessary.</p>

username: tengerye
cooked: <p>If I just want to save a model, I don’t need <code>dist.barrier()</code>, right?</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="tengerye" data-post="3" data-topic="87849" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/a6a055/48.png" class="avatar"> tengerye:</div>
<blockquote>
<p>If I just want to save a model, I don’t need <code>dist.barrier()</code> , right?</p>
</blockquote>
</aside>
<p>Yep, that should be fine. If only rank 0 saves the model and that might take very long, you can set the <code>timeout</code> argument in <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group" rel="noopener nofollow ugc"><code>init_process_group</code></a> to a larger value. The default is 30min.</p>