Simultaneous communications in different process groups

bapi: <p>Referring to this old issue <a href="https://github.com/pytorch/pytorch/issues/14528" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/14528</a>, which was closed, I need to do communications (all_reduce/reduce/broadcast) in two or more different groups simultaneously. For example, if a process’ rank belongs to say the group g0, it could participate in that communication only.</p>
<p>Essentially, referring to <a href="https://github.com/pytorch/pytorch/issues/14528" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/14528</a>, I would need to apply if conditional –</p>
<pre><code class="lang-auto"> if local_rank in g0: 
torch.distributed.all_reduce(t, group=g0) 
elif local_rank in g1: 
torch.distributed.all_reduce(t, group=g1)
</code></pre>
<p>– for participating in all_reduce so that simultaneous communications could happen in non-intersecting groups.</p>
<p>However, that results in  <code>NCCL INFO Call to connect returned Connection refused, retrying</code>. Whereas, doing these all_reduce operations sequentially, which essentially would imply that the collective communications in different groups are sequentiallized, works fine.</p>
<p>Is there something that I am missing?</p>

mrshenli: <p>Hey <a class="mention" href="/u/bapi">@bapi</a></p>
<p>I tried the following, and it worked for me:</p>
<pre><code class="lang-python">import torch
import torch.multiprocessing as mp
import os

def run(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '29500'

    torch.cuda.set_device(rank)
    # global group
    torch.distributed.init_process_group(backend='nccl', rank=rank, world_size=world_size)
    # torch.distributed.init_process_group(backend='gloo', init_method='env://')
    g0 = torch.distributed.new_group(ranks=[0,1,2,3])
    g1 = torch.distributed.new_group(ranks=[4,5,6,7])
    # tensor to bcast over group
    t = torch.tensor([1]).float().cuda().fill_(rank)
    if rank &lt; 4:
        torch.distributed.all_reduce(t, group=g0)
    else:
        torch.distributed.all_reduce(t, group=g1)
    print('rank: {} - val: {}'.format(rank, t.item()))


def main():
    world_size = 8
    mp.spawn(run,
        args=(world_size,),
        nprocs=world_size,
        join=True)

if __name__=="__main__":
    main()
</code></pre>
<p>outputs:</p>
<pre><code class="lang-auto">$ python test.py 
rank: 0 - val: 6.0
rank: 1 - val: 6.0
rank: 3 - val: 6.0
rank: 2 - val: 6.0
rank: 7 - val: 22.0
rank: 5 - val: 22.0
rank: 6 - val: 22.0
rank: 4 - val: 22.0
</code></pre>

mrshenli: <aside class="quote no-group" data-username="bapi" data-post="1" data-topic="75838">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bapi/48/27077_2.png" class="avatar"> bapi:</div>
<blockquote>
<p>if local_rank in g0:</p>
</blockquote>
</aside>
<p>I am not sure if you acquired <code>g0</code> and <code>g1</code> from <code>new_group</code> API. If so, they are process group objects. So the above check would result in the following error:</p>
<pre><code class="lang-auto">TypeError: argument of type 'object' is not iterable
</code></pre>

bapi: <p>Dear <a class="mention" href="/u/mrshenli">@mrshenli</a>, thanks very much for your response. Indeed that works.</p>
<p>Let me post my code adapted to your nice illustration that I actually need to work with as a part of a much longer implementation:</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp
import os


def run(rank, groups, world_size):
    print("Rank ", rank, " Group ", groups[str(rank)])
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '29500'
    torch.cuda.set_device(groups[str(rank)]['gpu'])
    # global group
    torch.distributed.init_process_group(backend='nccl',
                                         rank=rank,
                                         world_size=world_size)
    my_group = torch.distributed.new_group(ranks=groups[str(rank)]['grp'])
    # tensor to bcast over group
    t = torch.tensor([1]).float().cuda().fill_(rank)
    torch.distributed.all_reduce(t, group=my_group)
    print('rank: {} - val: {}'.format(rank, t.item()))


def assign_groups(num_masters, workers_per_master, available_gpus):
    groups = {}
    distranks = num_masters
    gpu_allocated = 1
    for i in range(num_masters):
        my_group = [i]
        gpus = [available_gpus[0]]
        for j in range(workers_per_master):
            my_group.append(distranks)
            gpus.append(available_gpus[gpu_allocated])
            distranks += 1
            gpu_allocated += 1
        for r, g in zip(my_group, gpus):
            groups.update({
                str(r): {
                    'grp': my_group,
                    'gpu': g
                }
            })
    return groups


def main():
    num_masters = 3
    workers_per_master = 1
    available_gpus = [0, 1, 2, 3]
    groups = assign_groups(num_masters, workers_per_master, available_gpus)
    world_size = 6
    mp.spawn(run, args=(groups, world_size), nprocs=world_size, join=True)


if __name__ == "__main__":
    main()

</code></pre>
<p>Essentially, I have 3 master processes here and each of the masters has a worker. The masters with their respective workers make a group. Thus there are three groups altogether. Although in the above code the masters are assigned to cuda:0 and each of the workers to the next cuda devices, it may change depending on the setting. Thus it is immaterial in this illustration whether I comment out the line <code>torch.cuda.set_device(groups[str(rank)]['gpu'])</code>.</p>
<p>Now, for sure for each of the masters and workers, I have its assigned group as <code>groups[str(rank)]['grp']</code>. Thus, when I run this code it should behave like those distributed communications being called concurrently as also illustrated by your example. However, it results in <code>NCCL INFO Call to connect returned Connection refused</code>.</p>
<p>Definitely I am doing something wrong here, not sure what.<br>
Thanks again.</p>

bapi: <p>I think I figured out what exactly I was missing. So, basically each of the new process groups should be defined at each of the processes before using them in any manner whatsoever.</p>
<p>A bit modified version of your example that most likely solves my purpose is the following:</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp
import os


def run(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '29500'
    # global group
    torch.distributed.init_process_group(backend='nccl',
                                         rank=rank,
                                         world_size=world_size)
    # torch.distributed.init_process_group(backend='gloo', init_method='env://')
    g0 = torch.distributed.new_group(ranks=[0, 2])
    g1 = torch.distributed.new_group(ranks=[1, 3])
    # tensor to bcast over group
    t = torch.tensor([1]).float().cuda().fill_(rank)
    if rank in [0, 2]:
        my_group = g0
    else:
        my_group = g1
    torch.distributed.all_reduce(t, group=my_group)
    print('rank: {} - val: {}'.format(rank, t.item()))


def main():
    world_size = 4
    mp.spawn(run, args=(world_size,), nprocs=world_size, join=True)


if __name__ == "__main__":
    main()

</code></pre>
<p>It works. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
<p>Thanks again.</p>

mrshenli: <p>Happy to see that worked <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Yes, the <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.new_group" rel="noopener nofollow ugc"><code>new_group</code></a> requires the following:</p>
<blockquote>
<p>This function requires that all processes in the main group (i.e. all processes that are part of the distributed job) enter this function, even if they are not going to be members of the group. Additionally, groups should be created in the same order in all processes.</p>
</blockquote>

bapi: <p>Exactly. Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>