Why torch.distributed.all_reduce with nccl backend issues so many D2H and H2D Memcpy and runs slow?

puddingfjz: <p>I run torch.distributed.all_reduce in my code with the nccl backend.<br>
I find that it takes very long to finish (about 2s). Then I profile the code and find that during the all_reduce, there are a lot of memory copy operations between the device and the host.<br>
The profiling result is shown below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac4a936f6511d13608aff31ab4835972a2b39303.png" data-download-href="https://discuss.pytorch.org/uploads/default/ac4a936f6511d13608aff31ab4835972a2b39303" title="96d2e4eca7abe3df40f8d8dfa362227"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ac4a936f6511d13608aff31ab4835972a2b39303_2_690x362.png" alt="96d2e4eca7abe3df40f8d8dfa362227" data-base62-sha1="oA9TZsCeWY9RtcrSGEE0KxvMtMv" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ac4a936f6511d13608aff31ab4835972a2b39303_2_690x362.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ac4a936f6511d13608aff31ab4835972a2b39303_2_1035x543.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ac4a936f6511d13608aff31ab4835972a2b39303_2_1380x724.png 2x" data-dominant-color="E2E1E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">96d2e4eca7abe3df40f8d8dfa362227</span><span class="informations">1877×985 91.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I also find that when I call all_reduce twice, the second one will be much faster.<br>
Can anyone help me with this problem?</p>

puddingfjz: <p>These are the flow events during all_reduce (may contain some other events before and after all_reduce):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f2b83b41ee807fe7ad2952eeffe607a3cc71b087.png" data-download-href="https://discuss.pytorch.org/uploads/default/f2b83b41ee807fe7ad2952eeffe607a3cc71b087" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2b83b41ee807fe7ad2952eeffe607a3cc71b087_2_690x436.png" alt="image" data-base62-sha1="yDcgeZ7jgXftORUt3ClhWQFu7u7" width="690" height="436" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2b83b41ee807fe7ad2952eeffe607a3cc71b087_2_690x436.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2b83b41ee807fe7ad2952eeffe607a3cc71b087_2_1035x654.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2b83b41ee807fe7ad2952eeffe607a3cc71b087_2_1380x872.png 2x" data-dominant-color="FDFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1859×1175 172 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

H-Huang: <p>Hi <a class="mention" href="/u/puddingfjz">@puddingfjz</a>. In PyTorch, nccl is initialized lazily, so on the first collective (all_reduce in your case) the nccl communicators are created (<a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/communicators.html" class="inline-onebox" rel="noopener nofollow ugc">Creating a Communicator — NCCL 2.20.3 documentation</a>) which could explain those operations.</p>
<p>When profiling its usually common practice to take a profile after a few steps into training so setup and cache warmup do not play a factor in the traces</p>

puddingfjz: <p>Thanks for your answer! I will check it.</p>