Why each GPU measures a difference allreduce duration?

username: Olivier-CR
cooked: <p>Hi, I’m running an allreduce with torch.distributed on a cluster of 2 AWS P4d instances (2*8 A100 GPUs).</p>
<p>I’m launching the 16 processes with MPI, and the allreduce with that function:</p>
<pre><code class="lang-auto">def run(rank, local_rank):
        """ Simple allreduce. """
        tensor = torch.rand(int(1e6)*args.tensor_size_mm, dtype=torch.float32).to(cuda)
        t1 = time.time()
        dist.all_reduce(tensor, op=dist.ReduceOp.SUM)
        dist.barrier()        
        t2 = time.time()

        return tensor.element_size()*tensor.numel()/1e6, t2-t1

results = []  # average over N allreduces to remove jitter
for i in range(args.n_attempts):
    size, duration = run(WORLD_RANK, LOCAL_RANK)
    results.append(duration)
    print("Run {} - allreduce of {}MB tensor done in {}s".format(
        i, size, duration))
</code></pre>
<p>I run 5 allreduces one after the other to remove the impact of network jitter. Suprisingly, the measured allreduce duration is <strong>vastly different</strong> on every GPU! with differences as big as 5x ratio. Below a measurement I did with a 8GB tensor</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/1/71b4f91ab58d3db2f476199d2dfe2eae6d99f3e5.png" data-download-href="https://discuss.pytorch.org/uploads/default/71b4f91ab58d3db2f476199d2dfe2eae6d99f3e5" title="Capture d’écran 2022-10-24 à 17.50.46"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/71b4f91ab58d3db2f476199d2dfe2eae6d99f3e5_2_690x286.png" alt="Capture d’écran 2022-10-24 à 17.50.46" data-base62-sha1="gdTGBrSAzOUUAfsHVmrpSPo64bX" width="690" height="286" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/71b4f91ab58d3db2f476199d2dfe2eae6d99f3e5_2_690x286.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/71b4f91ab58d3db2f476199d2dfe2eae6d99f3e5_2_1035x429.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/71b4f91ab58d3db2f476199d2dfe2eae6d99f3e5_2_1380x572.png 2x" data-dominant-color="F3F5F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture d’écran 2022-10-24 à 17.50.46</span><span class="informations">1960×814 56.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Why does each GPU measures a different allreduce time? What is the proper way to measure the duration of an allreduce operation, if all processes report something different</p>

username: cbalioglu
cooked: <p><a class="mention" href="/u/kwen2501">@kwen2501</a> any thoughts on this?</p>

username: Olivier-CR
cooked: <p>It was presumably because each rank was starting at slightly different times. When I add an extra barrier, the allreduce durations are identical to the millisecond</p>
<pre><code>def run(rank, local_rank):
    """ Simple allreduce. """
    coeffs = int(1e6*args.tensor_size_mb*8/32)
    tensor = torch.rand(coeffs, dtype=torch.float32).to(cuda)
    dist.barrier()  # HERE extra barrier before the allreduce
    t1 = time.time()
    print("rank {} started allreduce at time {}".format(WORLD_RANK, t1))
    dist.all_reduce(tensor, op=dist.ReduceOp.SUM)
    dist.barrier()        
    t2 = time.time()
    print("rank {} finished allreduce at time {}".format(WORLD_RANK, t2))
</code></pre>

username: kwen2501
cooked: <p>Thanks for finding the solution. Yes, it would be better to synchronize the processes before entering the timing region.</p>