How to synchronize lists across gpus using torch.distributed.launch

huangdi: <p>Hi.</p>
<p>I want to concat lists <strong>with different lengths</strong> across different gpus using <code>torch.distributed.launch</code>. Is there any api like <code>torch.distributed.all_reduce()</code> can help me?</p>
<p>Example Code (test.py):</p>
<pre><code class="lang-auto">import random
import torch
l = []
length = random.randint(5, 8)
for i in range(length):
    l.append(i)
print(l)
</code></pre>
<p>Run:</p>
<pre><code class="lang-auto">python -m torch.distributed.launch \
    --nproc_per_node=4  \
    --use_env \
    --master_port=$RANDOM \
    test.py
</code></pre>
<p>Result:</p>
<pre><code class="lang-auto">[1, 2, ..., length in GPU 0]
[1, 2, ..., length in GPU 1]
[1, 2, ..., length in GPU 2]
[1, 2, ..., length in GPU 3]
</code></pre>
<p>What I want (concat/synchronize the list in 4 different gpus together):</p>
<pre><code class="lang-auto">[1, 2, ..., length in GPU 0, ..., length in GPU 1, ..., length in GPU 2, ..., length in GPU 3]
[1, 2, ..., length in GPU 0, ..., length in GPU 1, ..., length in GPU 2, ..., length in GPU 3]
[1, 2, ..., length in GPU 0, ..., length in GPU 1, ..., length in GPU 2, ..., length in GPU 3]
[1, 2, ..., length in GPU 0, ..., length in GPU 1, ..., length in GPU 2, ..., length in GPU 3]
</code></pre>
<p>Thanks!</p>

pritamdamania87: <p>You can use <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_gather" rel="noopener nofollow ugc">allgather</a> for this purpose.</p>

huangdi: <p>Hi, Thanks for your nice suggestion!</p>
<p>Another harder problem for me is that, when there are too many 1D tensors with <strong>different lengths</strong> on each gpu, is there any method to gather them easier without a loop?<br>
Situation:</p>
<pre><code class="lang-auto">GPU 0: [torch.Tensor(101), torch.Tensor(102), torch.Tensor(103), ..., torch.Tensor(200)]
GPU 1: [torch.Tensor(201), torch.Tensor(202), torch.Tensor(203), ..., torch.Tensor(300)]
GPU 2: [torch.Tensor(301), torch.Tensor(302), torch.Tensor(303), ..., torch.Tensor(400)]
GPU 3: [torch.Tensor(401), torch.Tensor(402), torch.Tensor(403), ..., torch.Tensor(500)]
</code></pre>
<p>Result:</p>
<pre><code class="lang-auto">GPU 0: [torch.Tensor(101), torch.Tensor(102), ..., torch.Tensor(200), ...  torch.Tensor(201), ..., torch.Tensor(500)]
GPU 1: [torch.Tensor(101), torch.Tensor(102), ..., torch.Tensor(200), ...  torch.Tensor(201), ..., torch.Tensor(500)]
GPU 2: [torch.Tensor(101), torch.Tensor(102), ..., torch.Tensor(200), ...  torch.Tensor(201), ..., torch.Tensor(500)]
GPU 3: [torch.Tensor(101), torch.Tensor(102), ..., torch.Tensor(200), ...  torch.Tensor(201), ..., torch.Tensor(500)]
</code></pre>
<p>The order of tensors in the output list doesn’t matter.</p>

huangdi: <p>Hi, this is indeed what I need!</p>
<p>However, in my situation, <strong>data is generated dynamically during training on each GPU</strong>. What I need to do is to gather the data, and then use distributedSampler to sample them. I’m stucked at the step of gathering.</p>
<p>Do you have any good ideas?</p>

pritamdamania87: <aside class="quote no-group" data-username="huangdi" data-post="3" data-topic="119203">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/8dc957/48.png" class="avatar"> huangdi:</div>
<blockquote>
<p>Another harder problem for me is that, when there are too many 1D tensors with <strong>different lengths</strong> on each gpu, is there any method to gather them easier without a loop?</p>
</blockquote>
</aside>
<p>You can pad each tensor to the maximum size and then use allgather. If you don’t know the maximum size before hand, you can first perform an allgather to collect the size of each tensor and then calculate the max.</p>