NCCL backend not safe for ProcessGroups

Ankur_Mahesh: <p>On this <a href="https://pytorch.org/docs/stable/distributed.html" rel="noopener nofollow ugc">page</a>, there is a warning that “Using multiple process groups with the NCCL backend concurrently is not safe”</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/3/9366f52b1e5fa1eb56839fe807e094e858bf9578.png" data-download-href="https://discuss.pytorch.org/uploads/default/9366f52b1e5fa1eb56839fe807e094e858bf9578" title="Screen Shot 2023-06-26 at 5.55.50 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/9366f52b1e5fa1eb56839fe807e094e858bf9578_2_690x467.png" alt="Screen Shot 2023-06-26 at 5.55.50 PM" data-base62-sha1="l1YLZcXhQRzQCejyrT5ZA5ziebm" width="690" height="467" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/9366f52b1e5fa1eb56839fe807e094e858bf9578_2_690x467.png, https://discuss.pytorch.org/uploads/default/original/3X/9/3/9366f52b1e5fa1eb56839fe807e094e858bf9578.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/3/9366f52b1e5fa1eb56839fe807e094e858bf9578.png 2x" data-dominant-color="ECE4E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2023-06-26 at 5.55.50 PM</span><span class="informations">875×593 74.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Does this mean that assigning one specific task to multiple process groups is unsafe?  If so, then would switching from NCCL to Gloo be a good solution?</p>
<p>Or does it mean that having multiple ProcessGroups is unsafe, even if all the process groups are disjoint sets with no overlapping elements?</p>

ptrblck: <p>You can use multiple <code>ProcessGroups</code> in NCCL (and it’s also done in some LLM training scripts), but would need to make sure to properly synchronize these allowing them to finish their execution to avoid deadlocks.<br>
I don’t know enough about the Gloo backend and its limitations.</p>

Ankur_Mahesh: <p>Thanks so much for your response and your help!</p>
<p>To be clear, the synchronization is necessary even if the process groups are completely disjoint sets of ranks, right?</p>
<p>Does that mean that the code in <a href="https://discuss.pytorch.org/t/simultaneous-communications-in-different-process-groups/75838/2">this answer on the pytorch forum</a> should be modified to add explicit synchronization?</p>
<p>In that answer, there are multiple process groups that are simultaneously communicating, but there isn’t explicit synchronization to ensure that one process group finishes before the next one starts.</p>
<p>If I understand correctly, the code should be modified to</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp
import os

def run(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '29500'

    torch.cuda.set_device(rank)
    # global group
    torch.distributed.init_process_group(backend='nccl', rank=rank, world_size=world_size)
    g0 = torch.distributed.new_group(ranks=[0,1,2,3])
    g1 = torch.distributed.new_group(ranks=[4,5,6,7])
    # tensor to bcast over group
    t = torch.tensor([1]).float().cuda().fill_(rank)
    if rank &lt; 4:
        torch.distributed.all_reduce(t, group=g0)
    torch.distributed.barrier()
    if rank &gt; 4:
        torch.distributed.all_reduce(t, group=g1)
    print('rank: {} - val: {}'.format(rank, t.item()))


def main():
    world_size = 8
    mp.spawn(run,
        args=(world_size,),
        nprocs=world_size,
        join=True)

if __name__=="__main__":
    main()
</code></pre>
<p>The only change I made is adding the <code>torch.distributed.barrier()</code> after the first process group is done.  Then I changed the <code>else</code> to another if statement indicating the opposite of the first.</p>

ptrblck: <p>Your use case looks alright as something like this could deadlock:</p>
<pre><code class="lang-python"># rank 0
dist.all_reduce(x, group1)
dist.all_reduce(y, group2)

#rank 1 
dist.all_reduce(x, group2)
dist.all_reduce(y, group1)
</code></pre>
<p>More information can be found in the <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/communicators.html#using-multiple-nccl-communicators-concurrently">NCCL docs</a>.</p>