How do I reduce the tensors in 4 diffrent GPUs using 2 processes?

ronda: <pre><code class="lang-auto">import os
import torch
import torch.distributed as dist
from torch.multiprocessing import Process

def run(rank, size):
    tensor1 = torch.ones(1, device=2*rank)
    tensor2 = torch.ones(1, device=2*rank+1)

    dist.all_reduce(tensor1, op=dist.ReduceOp.SUM)
    dist.all_reduce(tensor2, op=dist.ReduceOp.SUM)
    print('Rank ', rank, ' has data ', tensor1[0], tensor2[0])

def init_process(rank, size, fn, backend='nccl'):
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29501'
    dist.init_process_group(backend, rank=rank, world_size=size)
    fn(rank, size)

if __name__ == "__main__":
    size = 2
    processes = []
    for rank in range(size):
        p = Process(target=init_process, args=(rank, size, run))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()
</code></pre>
<p>I have a computer with four GPUs,<br>
GPU0: tensor1,  GPU1: tensor2<br>
GPU2: tensor1,  GPU3: tensor2<br>
I expect  to reduce tensor1 in GPU0 and GPU2,  reduce tensor2 in GPU1 and GPU3. But when I execute the above code,  the program keeps blocking.<br>
I don’t know why. Can someone help me, thanks!</p>

cbalioglu: <p>Unfortunately I couldn’t reproduce your problem. This is the output of your script when I run it on a 4 GPU machine:</p>
<pre><code class="lang-auto">WARNING: Logging before InitGoogleLogging() is written to STDERR
I1025 17:28:00.405782 39617 ProcessGroupNCCL.cpp:520] [Rank 0] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
NCCL_DEBUG: UNSET
I1025 17:28:00.405783 39637 ProcessGroupNCCL.cpp:621] [Rank 0] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1025 17:28:00.422144 39618 ProcessGroupNCCL.cpp:520] [Rank 1] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
NCCL_DEBUG: UNSET
I1025 17:28:00.422158 39638 ProcessGroupNCCL.cpp:621] [Rank 1] NCCL watchdog thread started!
Rank  1  has data  tensor(2., device='cuda:2') tensor(2., device='cuda:3')
Rank  0  has data  tensor(2., device='cuda:0') tensor(2., device='cuda:1')
</code></pre>
<p>Are you sure that your script is able to see all your CUDA devices (e.g. <code>CUDA_VISIBLE_DEVICES=0,1,2,3</code>)?</p>

ronda: <p>But my results are as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be4abc9d937f994dac88db177817caa18c72fcd5.png" data-download-href="https://discuss.pytorch.org/uploads/default/be4abc9d937f994dac88db177817caa18c72fcd5" title="Snipaste_2021-10-27_23-27-33"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be4abc9d937f994dac88db177817caa18c72fcd5.png" alt="Snipaste_2021-10-27_23-27-33" data-base62-sha1="r9oQdp4bzJEJ7VmhrMXnzH8ZDIp" width="690" height="51" data-dominant-color="090A09"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Snipaste_2021-10-27_23-27-33</span><span class="informations">741×55 1.51 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
My result  just keeps blocking.</p>
<p>GPU Usage Details:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/1/1124f3bdafaa913141788b586152acbe38b184ae.png" data-download-href="https://discuss.pytorch.org/uploads/default/1124f3bdafaa913141788b586152acbe38b184ae" title="Snipaste_2021-10-27_23-45-40"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/1/1124f3bdafaa913141788b586152acbe38b184ae.png" alt="Snipaste_2021-10-27_23-45-40" data-base62-sha1="2rFhLrhQwuBaXBEAOYcdpCYQ7XE" width="632" height="499" data-dominant-color="0E0D0D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Snipaste_2021-10-27_23-45-40</span><span class="informations">726×574 14.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Feels like a weird GPU usage</p>
<p>My configuration:<br>
pytorch1.7.1+cu101<br>
cuda10.0.130<br>
GPU:GeForce RTX 2080Ti</p>