Ddp: diff between dist.all_gather and dist.all_gather_multigpu?

sbelharbi: <p>hi,<br>
what is the difference between <code>torch.distributed.all_gather</code> and <code>torch.distributed.all_gather_multigpu</code>?<br>
they both have the same definition: <code>Gathers tensors from the whole group in a list</code>.<br>
but, <code>torch.dist.all_gather_multigpu</code> has a different case usage…<br>
<code>*_multigpu</code> is supposed to work for multi-nodes. <code>all_gather</code> should also work in multi-nodes…<br>
this is the example provided in the doc for <code>*_multigpu</code>  with 2 nodes for <code>all_reduce_multigpu</code>, but thety all work the same.</p>
<p>node 0</p>
<pre><code class="lang-python">import torch
import torch.distributed as dist

dist.init_process_group(backend="nccl",
                        init_method="file:///distributed_test",
                        world_size=2,
                        rank=0)
tensor_list = []
for dev_idx in range(torch.cuda.device_count()):
    tensor_list.append(torch.FloatTensor([1]).cuda(dev_idx))

dist.all_reduce_multigpu(tensor_list)
</code></pre>
<p>node 1:</p>
<pre><code class="lang-python">import torch
import torch.distributed as dist

dist.init_process_group(backend="nccl",
                        init_method="file:///distributed_test",
                        world_size=2,
                        rank=1)
tensor_list = []
for dev_idx in range(torch.cuda.device_count()):
    tensor_list.append(torch.FloatTensor([1]).cuda(dev_idx))

dist.all_reduce_multigpu(tensor_list)
</code></pre>
<p>what usage case is this?<br>
and most importantly, why a process in a node with multigpus want to access to all devices. in this case, it is creating tensors on each device? this is not a real case…</p>
<p>i assume here they want to <code>duplicate</code> the tensor we want to synch on each device of the local node.<br>
in <code>all_gather_multigpu</code>, the output is a list of size world_size * nbr_gpus_in_node.<br>
so, when calling the synch function, each process will have the same output list that contains the tensor on each gpu in the world.</p>
<p>why we need this <code>all_gather_multigpu</code> while <code>all_gather</code> can already do this easily without duplicating tensor on gpus??</p>
<p>also, the downside of <code>all_gather_multigpu</code> is that it requires that EACH NODE NEEDS TO HAVE THE SAME NUMBER OF GPUS. in practice, this is less likely to happen on clusters. in slurm, you can request 8 gpus, you can have  in the same node, but the rest are dispatched over 4 nodes with 1 gpu per node…</p>
<p>an <a href="https://discuss.pytorch.org/t/ddp-evaluation-gather-output-loss-and-stuff-how-to/130593/2">example</a> on how to synch a tensor across all gpus.</p>
<p>thanks</p>

gcramer23: <aside class="quote no-group" data-username="sbelharbi" data-post="1" data-topic="130653">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/f475e1/48.png" class="avatar"> sbelharbi:</div>
<blockquote>
<p>they both have the same definition: <code>Gathers tensors from the whole group in a list</code>.</p>
</blockquote>
</aside>
<p>A single tensor is broadcast from a process when using all_gather. A list of tensors is broadcast from a process when using all_gather_multigpu.</p>
<blockquote>
<p>what usage case is this?<br>
and most importantly, why a process in a node with multigpus want to access to all devices. in this case, it is creating tensors on each device? this is not a real case…</p>
</blockquote>
<p>Its an example of using the PyTorch API.</p>
<blockquote>
<p>also, the downside of <code>all_gather_multigpu</code> is that it requires that EACH NODE NEEDS TO HAVE THE SAME NUMBER OF GPUS. in practice, this is less likely to happen on clusters. in slurm, you can request 8 gpus, you can have in the same node, but the rest are dispatched over 4 nodes with 1 gpu per node…</p>
</blockquote>
<p>You would simply need to configure your resources correctly.</p>

sbelharbi: <aside class="quote no-group" data-username="gcramer23" data-post="2" data-topic="130653">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gcramer23/48/37599_2.png" class="avatar"> gcramer23:</div>
<blockquote>
<p>A single tensor is broadcast from a process when using all_gather. A list of tensors is broadcast from a process when using all_gather_multigpu.</p>
</blockquote>
</aside>
<p>not sure about that.<br>
they are both used to sync one tensor.<br>
the output of both methods is a list of tensors where each tensor comes from a process.<br>
then, you can merge this list as you like to get the final synched tensor as in <a href="https://discuss.pytorch.org/t/ddp-evaluation-gather-output-loss-and-stuff-how-to/130593/2">here</a>.<br>
broadcasting is done using <code>torch.dist.broadcast</code>  i think, and it copies a tensor from source and diffuses it to the rest. <code>all_gather</code> copies all tensors across all process into a list and makes sure that all processes have the same exact list.</p>

gcramer23: <blockquote>
<p>they are both used to sync one tensor.</p>
</blockquote>
<p>all_gather <a href="https://github.com/pytorch/pytorch/blob/101a6263309ae2f9e52947c7d02d630e1190b6c3/torch/distributed/distributed_c10d.py#L1940" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/101a6263309ae2f9e52947c7d02d630e1190b6c3/torch/distributed/distributed_c10d.py#L1940</a></p>
<p>all_gather_multigpu <a href="https://github.com/pytorch/pytorch/blob/101a6263309ae2f9e52947c7d02d630e1190b6c3/torch/distributed/distributed_c10d.py#L1450" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/101a6263309ae2f9e52947c7d02d630e1190b6c3/torch/distributed/distributed_c10d.py#L1450</a></p>

omsrisagar: <p>I think <code>multigpu</code> versions are needed when you run a single process for all the GPUs in that node. Most commonly we run one process per GPU, in which case <code>dist.all_gather</code> will be sufficient to gather tensors from different GPUs in different nodes. In case we run only one process for all the GPUs in a given node (as in the example code at <a href="https://pytorch.org/docs/stable/distributed.html#multi-gpu-collective-functions" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.11.0 documentation</a>), we need to make use of <code>dist.all_gather_multigpu</code> to aggregate data from all the GPUs—because in this case each rank has 8 GPUs under it.<br>
You can see in this example code that the world_size is 2, which means that there are only two processes, one on each node, covering the 16 GPUs. In a general implementation, we create 16 processes in total (8 on one node and 8 on another node), using <code>torch.multiprocessing.spawn()</code>.</p>