SendRecv exchange messages with wrong tag

username: sebk26
cooked: <p>Hello,<br>
i encounter a strange behavior with messages that get exchanged even though their tag mismatch.</p>
<h2>
<a name="question-1" class="anchor" href="#question-1"></a>Question</h2>
<p>Why is the first message used in <code>dist.recv()</code> even though the tag obviously mismatch?</p>
<h2>
<a name="minimal-example-2" class="anchor" href="#minimal-example-2"></a>Minimal Example</h2>
<pre><code class="lang-python">"""SEND RECV TEST WITH MULTIPLE TAGS """

import os
import torch
import horovod.torch as hvd
import torch.distributed as dist


if __name__ == '__main__':
    
    print('CUDA AVAILABLE:',torch.cuda.is_available())
    
    print('CUDA device count:',torch.cuda.device_count())
    
    neighbors_left = [0,1]
    neighbors_right = [1,2]
    
    DIR = os.getcwd()
    init_file = f'file://{DIR}{os.sep}sharedfile'
    
    hvd.init()
    rank = hvd.local_rank()
    torch.cuda.set_device(rank)
    print('Current Device:',torch.cuda.current_device())
    
    device = torch.device('cuda')
    dist.init_process_group(backend="nccl", init_method=init_file,rank=rank, world_size=hvd.size())
    
    left = rank - 1
    right = rank + 1
    
    a = torch.ones((3,4)) 
    
    a[:,0] = a[:,0]*((rank+1) * 100)
    a[:,-1] = a[:,-1]*((rank+1) * 10)
    
    a = a.to(device, non_blocking=True)
    send_buff = a[:,0].contiguous()
    recv_buff = a[:,-1].contiguous()
    for i in range(hvd.size()):
        if rank == i:
             print(f'{rank=}: {a=}')  
        dist.barrier()
   
    
    dist.barrier()
    sends = []
    print(f'{hvd.local_rank()=} : {dist.get_rank()=}')
    if left &gt;= 0:
        print(f'{hvd.local_rank()=} sends to {left=}')
        sends.append(dist.isend(send_buff+8, left, tag=8))
        sends.append(dist.isend(send_buff, left, tag=1))
        sends.append(dist.isend(send_buff+3, left, tag=3))
    
		for s in sends:
		  s.wait()
		
    if right &lt; hvd.size():
        print(f'{hvd.local_rank()=} receives from {right=}')
        dist.recv(recv_buff, right, tag=1)
        a[:,-1] = recv_buff
    
    dist.barrier()
    for i in range(hvd.size()):
        if rank == i:
             print(f'{rank=}: {a=}')  
        
        dist.barrier() 
</code></pre>
<p>the example is called with <code>horovodrun -np 4 python ./torch_dist_sendleft.py</code></p>
<h2>
<a name="expected-output-3" class="anchor" href="#expected-output-3"></a>Expected Output</h2>
<pre data-code-wrap="cmd"><code class="lang-nohighlight">[1,0]&lt;stdout&gt;:RANK: 0 A' = 
[1,0]&lt;stdout&gt;: tensor([[100.,   1.,   1., 200.],
[1,0]&lt;stdout&gt;:        [100.,   1.,   1., 200.],
[1,0]&lt;stdout&gt;:        [100.,   1.,   1., 200.]], device='cuda:0')
[1,1]&lt;stdout&gt;:RANK: 1 A' = 
[1,1]&lt;stdout&gt;: tensor([[200.,   1.,   1., 300.],
[1,1]&lt;stdout&gt;:        [200.,   1.,   1., 300.],
[1,1]&lt;stdout&gt;:        [200.,   1.,   1., 300.]], device='cuda:1')
[1,2]&lt;stdout&gt;:RANK: 2 A' = 
[1,2]&lt;stdout&gt;: tensor([[300.,   1.,   1., 400.],
[1,2]&lt;stdout&gt;:        [300.,   1.,   1., 400.],
[1,2]&lt;stdout&gt;:        [300.,   1.,   1., 400.]], device='cuda:2')
[1,3]&lt;stdout&gt;:RANK: 3 A' = 
[1,3]&lt;stdout&gt;: tensor([[400.,   1.,   1.,  40.],
[1,3]&lt;stdout&gt;:        [400.,   1.,   1.,  40.],
[1,3]&lt;stdout&gt;:        [400.,   1.,   1.,  40.]], device='cuda:3')
</code></pre>
<h2>
<a name="current-output-with-logs-4" class="anchor" href="#current-output-with-logs-4"></a>Current Output (with logs)</h2>
<pre data-code-wrap="cmd"><code class="lang-nohighlight">Module GCC/10.3.0 and 3 dependencies loaded.
Module CUDA/11.3.1 loaded.
Module NCCL/2.9.9-CUDA-11.3.1 loaded.
Module OpenMPI/4.1.1 and 10 dependencies loaded.
Module Python/3.9.5-bare and 6 dependencies loaded.
 Running on multiple GPU devices on single node

 Run started at:- 
Sa 14. Mai 14:18:52 CEST 2022
[1,0]&lt;stdout&gt;:CUDA AVAILABLE: True
[1,0]&lt;stdout&gt;:CUDA device count: 4
[1,2]&lt;stdout&gt;:CUDA AVAILABLE: True
[1,2]&lt;stdout&gt;:CUDA device count: 4
[1,3]&lt;stdout&gt;:CUDA AVAILABLE: True
[1,3]&lt;stdout&gt;:CUDA device count: 4
[1,1]&lt;stdout&gt;:CUDA AVAILABLE: True
[1,1]&lt;stdout&gt;:CUDA device count: 4
[1,3]&lt;stdout&gt;:Current Device: 3
[1,1]&lt;stdout&gt;:Current Device: 1
[1,0]&lt;stdout&gt;:Current Device: 0
[1,2]&lt;stdout&gt;:Current Device: 2
[1,3]&lt;stdout&gt;:rank=3: a=tensor([[400.,   1.,   1.,  40.],
[1,3]&lt;stdout&gt;:        [400.,   1.,   1.,  40.],
[1,3]&lt;stdout&gt;:        [400.,   1.,   1.,  40.]], device='cuda:3')
[1,0]&lt;stdout&gt;:rank=0: a=tensor([[100.,   1.,   1.,  10.],
[1,0]&lt;stdout&gt;:        [100.,   1.,   1.,  10.],
[1,0]&lt;stdout&gt;:        [100.,   1.,   1.,  10.]], device='cuda:0')
[1,2]&lt;stdout&gt;:rank=2: a=tensor([[300.,   1.,   1.,  30.],
[1,2]&lt;stdout&gt;:        [300.,   1.,   1.,  30.],
[1,2]&lt;stdout&gt;:        [300.,   1.,   1.,  30.]], device='cuda:2')
[1,1]&lt;stdout&gt;:rank=1: a=tensor([[200.,   1.,   1.,  20.],
[1,1]&lt;stdout&gt;:        [200.,   1.,   1.,  20.],
[1,1]&lt;stdout&gt;:        [200.,   1.,   1.,  20.]], device='cuda:1')
[1,2]&lt;stdout&gt;:hvd.local_rank()=2 : dist.get_rank()=2
[1,2]&lt;stdout&gt;:hvd.local_rank()=2 sends to left=1
[1,3]&lt;stdout&gt;:hvd.local_rank()=3 : dist.get_rank()=3
[1,3]&lt;stdout&gt;:hvd.local_rank()=3 sends to left=2
[1,1]&lt;stdout&gt;:hvd.local_rank()=1 : dist.get_rank()=1
[1,1]&lt;stdout&gt;:hvd.local_rank()=1 sends to left=0
[1,0]&lt;stdout&gt;:hvd.local_rank()=0 : dist.get_rank()=0
[1,0]&lt;stdout&gt;:hvd.local_rank()=0 receives from right=1
[1,1]&lt;stdout&gt;:hvd.local_rank()=1 receives from right=2
[1,2]&lt;stdout&gt;:hvd.local_rank()=2 receives from right=3
[1,0]&lt;stdout&gt;:RANK: 0 A' = 
[1,0]&lt;stdout&gt;: tensor([[100.,   1.,   1., 208.],
[1,0]&lt;stdout&gt;:        [100.,   1.,   1., 208.],
[1,0]&lt;stdout&gt;:        [100.,   1.,   1., 208.]], device='cuda:0')
[1,1]&lt;stdout&gt;:RANK: 1 A' = 
[1,1]&lt;stdout&gt;: tensor([[200.,   1.,   1., 308.],
[1,1]&lt;stdout&gt;:        [200.,   1.,   1., 308.],
[1,1]&lt;stdout&gt;:        [200.,   1.,   1., 308.]], device='cuda:1')
[1,2]&lt;stdout&gt;:RANK: 2 A' = 
[1,2]&lt;stdout&gt;: tensor([[300.,   1.,   1., 408.],
[1,2]&lt;stdout&gt;:        [300.,   1.,   1., 408.],
[1,2]&lt;stdout&gt;:        [300.,   1.,   1., 408.]], device='cuda:2')
[1,3]&lt;stdout&gt;:RANK: 3 A' = 
[1,3]&lt;stdout&gt;: tensor([[400.,   1.,   1.,  40.],
[1,3]&lt;stdout&gt;:        [400.,   1.,   1.,  40.],
[1,3]&lt;stdout&gt;:        [400.,   1.,   1.,  40.]], device='cuda:3')
Run completed at:- 
Sa 14. Mai 14:19:12 CEST 2022
</code></pre>