Question about pipeline parallelism

username: ConnollyLeon
cooked: <p>I notice that in  <a>https://github.com/pytorch/examples/tree/master/distributed/rpc/pipeline</a>, every input batch is divided into micro-batches. This sorta likes GPipe method. But the backward pass is not parallelized, which contradicts GPipe. I wonder if there is any papers or references that can explain what pipeline algorithm are being used.</p>
<p>I wonder if  it works like this gantt chart.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/c/8c34fc239831ad84eab4a5d7261daee35d8dcde2.png" data-download-href="https://discuss.pytorch.org/uploads/default/8c34fc239831ad84eab4a5d7261daee35d8dcde2" title="PastedGraphic-4"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c34fc239831ad84eab4a5d7261daee35d8dcde2_2_690x142.png" alt="PastedGraphic-4" data-base62-sha1="k0kmF5b4UoYRe7XHmSCuiPBCYzU" width="690" height="142" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c34fc239831ad84eab4a5d7261daee35d8dcde2_2_690x142.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c34fc239831ad84eab4a5d7261daee35d8dcde2_2_1035x213.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/c/8c34fc239831ad84eab4a5d7261daee35d8dcde2.png 2x" data-dominant-color="DBE4EA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">PastedGraphic-4</span><span class="informations">1370×282 28.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ConnollyLeon
cooked: <p>If so, is there any chances to enhance it? Makes it more like this chart below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1b740c0a1f7073d7c930b6932125c7528346625.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e1b740c0a1f7073d7c930b6932125c7528346625" title="WeChat03357ce5fb12f7ef959d4a752be604f1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1b740c0a1f7073d7c930b6932125c7528346625_2_690x142.jpeg" alt="WeChat03357ce5fb12f7ef959d4a752be604f1" data-base62-sha1="wcM30i3fov6DuHIGZVOdWaOSfYN" width="690" height="142" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1b740c0a1f7073d7c930b6932125c7528346625_2_690x142.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1b740c0a1f7073d7c930b6932125c7528346625_2_1035x213.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1b740c0a1f7073d7c930b6932125c7528346625_2_1380x284.jpeg 2x" data-dominant-color="E0E1E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WeChat03357ce5fb12f7ef959d4a752be604f1</span><span class="informations">1504×310 104 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: mrshenli
cooked: <p>Yep, the first chart looks correct to me.</p>
<blockquote>
<p>If so, is there any chances to enhance it? Makes it more like this chart below.</p>
</blockquote>
<p>Do you need single-machine multi-GPU pipeline parallel or multi-machine pipeline parallel?</p>
<ul>
<li>
<p>If it is within a single machine, it is possible to parallelize backward as well. Check this project <a href="https://github.com/kakaobrain/torchgpipe" rel="noopener nofollow ugc">torchgpipe</a>. It inserts phony dependencies between stages of different micro batches.</p>
</li>
<li>
<p>If it’s multi-machine pipeline parallel, then you will need RPC. As of today distributed autograd cannot parallelize backward, because the <a href="https://github.com/pytorch/pytorch/issues/23110" rel="noopener nofollow ugc">smart mode</a> has not been implemented yet. To get around this, you can still use RPC and RRef, but cannot use distributed autograd and will need to manually stitch together local autograd.</p>
<p>Another possibility (not 100% sure if this would work) is to create one distributed autograd context per micro batch, and manually call <code>__enter__</code> and <a href="https://github.com/pytorch/pytorch/blob/356aa54694cc4f1c9599fdf8320a4759aa63a12f/torch/distributed/autograd/__init__.py#L36" rel="noopener nofollow ugc"><code>__exit__</code></a> on distributed autograd context. As a result, the gradients for different microbatches will be stored in different contexts, and hence you will need to call <code>dist_optimizer.step(ctx_id)</code> multiple times to apply the gradients.</p>
<p>Things will become a lot easier when we add smart mode distributed autograd.</p>
</li>
</ul>

username: ConnollyLeon
cooked: <p>Thanks for your replying! This help me a lot.<br>
I will keep trying to enhance the training speed by utilizing pipeline.</p>