Multiprocessing - Barrier Blocks all Processes?

soulslicer: <p>I am trying to use dist.barrier() to sync my all my processes so that they can finish one epoch together. But as soon as the first process hits the barrier, it stops all the other processes! Why is this?</p>

mrshenli: <p>Hey <a class="mention" href="/u/soulslicer">@soulslicer</a>,</p>
<blockquote>
<p>But as soon as the first process hits the barrier, it stops all the other processes! Why is this?</p>
</blockquote>
<p>This is not the expected behavior. Could you please share the code to repro? Do other processes call c10d APIs (e.g., allrecue/allgather) but the first process is not participating?</p>

soulslicer: <p>This is only happening if i use it with multi GPU’s, but in CPU mode it works fine. Repo is currently private, but yes i could share it with you</p>

soulslicer: <p>Actually, even when i simplified my code down to the basics. just launching two processes for two GPU’s, as soon as the first process, hits the barrier, it stops the other process.</p>
<p>But, if i remove the loss.backward() and the optimizer stuff, it works</p>

mrshenli: <p>You don’t need to share the private code, a min repro would help.</p>
<blockquote>
<p>But, if i remove the loss.backward() and the optimizer stuff, it works</p>
</blockquote>
<p>In this case I assume you are using <code>DistributedDataParallel</code> (DDP)? DDP calls allreduce internally. If the first process is also in the same process group but is not running backward on DDP model, other processes would hang on backward, because they need 1st process to join the allreduce.</p>
<p>One requirement of collective communications is that all members need to call the same collective API in the same order.</p>
<p>If you need some side channel to do the barrier, you can create a new process group using the <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.new_group" rel="noopener nofollow ugc"><code>new_group</code></a> API, and then call barrier on that.</p>

soulslicer: <p>wait…let me get this straight:</p>
<p>so lets say one process calls dist.barrier(), then the other process will be stuck on loss.backward()??</p>

mrshenli: <blockquote>
<p>so lets say one process calls dist.barrier(), then the other process will be stuck on loss.backward()??</p>
</blockquote>
<p>Yep, that’s possible if they are using the same process group and if you are using DistributedDataParallel. That’s the contract of collective communications. If process x is calling allreduce-&gt;allreduce-&gt;barrier, and process Y is calling barrier, the allrecue would block on process x, as process Y never joined the collective comm.</p>

soulslicer: <p>Yes they are in the same process group and yes I am using DistributedDataParallel</p>
<p>Here is a quick code snippet and output</p>
<pre><code>    if self.id == 0:
        ep = 10
    elif self.id == 1:
        ep = 20

    for i in range(0, ep):

        self._log.info(self.id, "image.. " + str(i))
        image = torch.zeros((1,2,3,256,384)).cuda(self.id)
        input = {"rgb": image}
        self._log.info(self.id, "model.. " + str(i))
        output = self.model(input)
        loss = torch.sum(output["output"][-1])
        time.sleep(0.1)

        self._log.info(self.id, "zerograd.. " + str(i))
        self.optimizer.zero_grad()
        self._log.info(self.id, "backward.. " + str(i))
        loss.backward()
        self._log.info(self.id, "step.. " + str(i))
        self.optimizer.step()
        #print("step: " + str(self.id) + " " + str(i))
        self._log.info(self.id, "done " + str(i))

    self._log.info(self.id, "barrier.. ")
    dist.barrier()
</code></pre>
<p>Here, I am having two processes run. And one finishes faster than the other and needs to wait on the first one. But instead as soon as 0 calls barrier, it blocks 1 completely.</p>
<p>Here is the output</p>
<pre><code class="lang-auto">[INFO]  [0] zerograd.. 9
[INFO]  [0] backward.. 9
[INFO]  [0] step.. 9
[INFO]  [1] model.. 10
[INFO]  [0] done 9
[INFO]  [0] barrier.. 
[INFO]  [1] zerograd.. 10
[INFO]  [1] backward.. 10
[INFO]  [1] step.. 10
[INFO]  [1] done 10
[INFO]  [1] image.. 11
</code></pre>
<p>As you can see process 1 is stuck on image…</p>

soulslicer: <p>If i change the code so both have the same number of epochs in the for loop, then it works</p>

mrshenli: <p>I see, that’s the uneven DDP input batches issue. See the previous tracking issue <a href="https://github.com/pytorch/pytorch/issues/33148" rel="nofollow noopener">here</a>. We have just finished design discussion on this, and plan to create a <code>ddp.join()</code> context manager to solve this problem. <a class="mention" href="/u/rvarm1">@rvarm1</a> will post a design RFC later.</p>
<p>For short-term unblock, you can either make sure that all DDP instances are running through the same number of batches or detect that proactively in the application, sth like:</p>
<pre><code class="lang-python">for batch in get_batch():
    signal = torch.tensor([1])
    work = all_reduce(signal, async_op=True)
    loss = model(batch).sum()
    work.wait()
    if signal.item() &lt; world_size:
        break
    loss.backward()


all_reduce(torch.tensor([0]))
</code></pre>

soulslicer: <p>oh wow…it took a lot of effort to finally find the right github issue. thanks.</p>
<p>can you explain more what the all_reduce operation is doing? Why are you doing it to a torch.tensor([1])?</p>

mrshenli: <p><a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_reduce" rel="nofollow noopener">here</a> is the allreduce API doc. By default, it is summing the provided tensor across all processes. The above code snippet is using allreduce to detect if any process has finished processing all inputs. If they are still in the loop, they use all_reduce(1) to add one to the sum, otherwise, they do all_reduce(0). So as long as the all_reduce sum is smaller than the world_size, it means some process has exited the loop. And in this case, no other process should launch the backward, otherwise the allreduce in the backward would hang.</p>
<p>Above is basically detecting and then skipping remaining batches.</p>

soulslicer: <p>Can i check. Instead of using this approach:</p>
<p>Can I create a shared memory torch tensor. And as soon as one process finishes its batch, i can write to the tensor, and have the other processes read it so it can break it’s loop</p>

mrshenli: <p>Sure, that will also work. You can use a <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#reuse-buffers-passed-through-a-queue" rel="nofollow noopener"><code>torch.mutliprocessing.Queue</code></a> to do that, but this solution cannot scale across multiple machines.</p>

soulslicer: <aside class="quote no-group" data-username="mrshenli" data-post="11" data-topic="80345">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>all_reduce(torch.tensor([0]))</p>
</blockquote>
</aside>
<p>Final question. Not clear why you have another all reduce at the end</p>

soulslicer: <p>Hmm…i tried this method and it works. But now the second process hangs at dist.barrier().</p>

mrshenli: <blockquote>
<p>Final question. Not clear why you have another all reduce at the end</p>
</blockquote>
<p>That’s to indicate one process has exited the loop, otherwise the sum would never be smaller than the world_size. It needs to be guarded by a flag, because only the processes that exit early need that. Sth like:</p>
<pre data-code-wrap="python"><code class="lang-python">for batch in get_batch():
    ....

if signal.item() &gt;= world_size:
    all_reduce(torch.tensor([0]))
</code></pre>

soulslicer: <aside class="quote no-group" data-username="mrshenli" data-post="11" data-topic="80345">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>all_reduce(torch.tensor([0]))</p>
</blockquote>
</aside>
<p>I tried implement it as you suggested, but my second process gets stuck on the barrier</p>
<pre><code>    if self.id == 0:
        ep = 10
        sl = 0.1
    elif self.id == 1:
        ep = 15
        sl = 0.2

    for i in range(0, ep):

        # if i == 10:
        #     break

        signal = torch.tensor([1]).to(self.device)
        work = dist.all_reduce(signal, async_op=True)

        self._log.info(self.id, "image.. " + str(i))
        image = torch.zeros((1,2,3,256,384)).cuda(self.id)
        input = {"rgb": image}
        output = self.model(input)
        loss = torch.sum(output["output"][-1])
        
        work.wait()
        if signal.item() &lt; self.cfg.mp.workers:
            break

        self.optimizer.zero_grad()
        self._log.info(self.id, "backward.. " + str(i))
        loss.backward()
        self.optimizer.step()
        print("step: " + str(self.id) + " " + str(i))
        self._log.info(self.id, "done " + str(i))

        time.sleep(sl)

    self._log.info(self.id, "reduce.. ")
    signal = torch.tensor([0]).to(self.device)
    dist.all_reduce(signal)

    self._log.info(self.id, "barrier.. ")
    dist.barrier()

    return
</code></pre>

soulslicer: <aside class="quote no-group" data-username="soulslicer" data-post="20" data-topic="80345">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/soulslicer/48/9454_2.png" class="avatar"> soulslicer:</div>
<blockquote>
<p>ist.all_reduce(signal)</p>
</blockquote>
</aside>
<p>the final all reduce should only be done by the process that first exits. How do i ensure that?</p>

mrshenli: <p>You can guard that with a flag. For the first process that exits (call it X), its <code>signal</code> tensor should always equal to <code>world_size</code>. But for other processes, they would see <code>signal.item() &lt; world_size</code>, because their <code>all_reduce</code> in the for loop will join with the after-loop <code>all_reduce</code> on X.</p>