Nccl allreduce performace

username: JF-D
cooked: <h1><a name="question-1" class="anchor" href="#question-1"></a>Question</h1>
<p>I want to analysis the speed of distributed data parallel training of alexnet. But I find that the speed of allreduce invoked during alexnet training is slower than that directly invoked by nccl-tests or pytorch allreduce test.</p>
<h1><a name="environment-2" class="anchor" href="#environment-2"></a>Environment</h1>
<ol>
<li>pytorch1.5 + cuda9.0 + nccl2.7.8</li>
<li>4x1080ti</li>
</ol>
<h1><a name="details-of-performance-3" class="anchor" href="#details-of-performance-3"></a>Details of Performance</h1>
<p>Alexnet has about 244MB parameters, so I test the allreduce performance of this communication volume with 4 devices.</p>
<h2><a name="speed-of-nccl-allreduce-by-directly-testing-4" class="anchor" href="#speed-of-nccl-allreduce-by-directly-testing-4"></a>Speed of nccl allreduce by directly testing</h2>
<p>I use nccl-tests to test allreduce. The time cost is about 50.5ms.<br>
I also test it with the following pytorch code. The time cost is about 50.4ms.</p>
<pre><code class="lang-auto">def all_reduce_latency(nbytes):
    buf = torch.randn(nbytes // 4).cuda()
    torch.cuda.synchronize()
    # warmup
    for _ in range(5):
        dist.all_reduce(buf)
    torch.cuda.synchronize()

    torch.cuda.synchronize()
    begin = time.perf_counter()
    for _ in range(25):
        dist.all_reduce(buf)
    torch.cuda.synchronize()
    end = time.perf_counter()
    avg_speed = (end - begin) * 1e6 / 25
    return avg_speed
</code></pre>
<h2><a name="speed-of-nccl-allreduce-in-alexnet-training-5" class="anchor" href="#speed-of-nccl-allreduce-in-alexnet-training-5"></a>Speed of nccl allreduce in alexnet training</h2>
<p>I use ddp with a very large bucket size to force that all gradient are fused to a single buffer, and the gradient communication is not overlapped with computation. I found that the speed of allreduce is about 67.2ms, which is very slow.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9165f0ac8469579354ee11c7228b64b69f40308.png" data-download-href="https://discuss.pytorch.org/uploads/default/d9165f0ac8469579354ee11c7228b64b69f40308" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9165f0ac8469579354ee11c7228b64b69f40308_2_690x241.png" alt="image" data-base62-sha1="uYrxyHUeYboxkp1v5eIVlUuSSy4" width="690" height="241" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9165f0ac8469579354ee11c7228b64b69f40308_2_690x241.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9165f0ac8469579354ee11c7228b64b69f40308_2_1035x361.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9165f0ac8469579354ee11c7228b64b69f40308_2_1380x482.png 2x" data-dominant-color="EDEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2523×884 274 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I also tried manually fuse gradients to a buffer to call allreduce instead of using ddp. There is no difference. NCCL allreduce is much slower when called during alexnet training. I don’t know what cause this. Can you give some suggestions to find this problem?</p>
<p>Thanks a lot!</p>

username: H-Huang
cooked: <p>cc <a class="mention" href="/u/mrshenli">@mrshenli</a> for NCCL allreduce delay</p>