How to do simultanous isend/irecv

Zitong_Li: <p>Is there a way to have rank 0 and 1 send a message to each other at the same time and then recv the message sent at the same time without resulting in deadlock?</p>
<p>I’ve tried the following code</p>
<pre><code class="lang-auto">reqs = []
if rank == 0:
  neighbour = 1
if rank ==1:
  neighbour = 0

reqs.append(dist.isend(tensor=send_tensor, dst=neighbour))
reqs.append(dist.irecv(tensor=recv_tensor, src=neighbour))
for req in reqs:
    req.wait()
</code></pre>
<p>But as you can imagine, the first “req” in the “reqs” array is the send request for both processors. And both end up waiting on the other one to receive, resulting in deadlock.</p>
<p>Of course you can do send-&gt;recv in rank 1 and recv-&gt;send in rank 2. But this would take twice the amount of time.</p>
<p>In MPI there is a wait_all function such that the requests don’t have to be fulfilled in a predetermined order to prevent the deadlock. But there isn’t one in pyTorch.</p>
<p>This seems a very simple thing to do but I couldn’t figure out how to do it. Any help would be appreciated.</p>
<p>Thank you thank you!</p>

eqy: <p>Have you taken a look at e.g., <code>batch_isend_irecv</code>?</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/d3e1f165b36dc8b6be8c7bdffd1a7b8cc1be221d/torch/distributed/distributed_c10d.py#L1417">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/d3e1f165b36dc8b6be8c7bdffd1a7b8cc1be221d/torch/distributed/distributed_c10d.py#L1417" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/d3e1f165b36dc8b6be8c7bdffd1a7b8cc1be221d/torch/distributed/distributed_c10d.py#L1417" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/d3e1f165b36dc8b6be8c7bdffd1a7b8cc1be221d/torch/distributed/distributed_c10d.py#L1417</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="1407" style="counter-reset: li-counter 1406 ;">
          <li>def _coalescing_manager(group, device, reqs):</li>
          <li>    if group is None:</li>
          <li>        group = _get_default_group()</li>
          <li>    group._start_coalescing(device)</li>
          <li>    try:</li>
          <li>        yield</li>
          <li>    finally:</li>
          <li>        group._end_coalescing(device, reqs)</li>
          <li>
          </li>
<li>
          </li>
<li class="selected">def batch_isend_irecv(p2p_op_list):</li>
          <li>    """</li>
          <li>    Send or Receive a batch of tensors asynchronously and return a list of requests.</li>
          <li>
          </li>
<li>    Process each of the operations in ``p2p_op_list`` and return the corresponding</li>
          <li>    requests. NCCL, Gloo, and UCC backend are currently supported.</li>
          <li>
          </li>
<li>    Args:</li>
          <li>        p2p_op_list: A list of point-to-point operations(type of each operator is</li>
          <li>            ``torch.distributed.P2POp``). The order of the isend/irecv in the list</li>
          <li>            matters and it needs to match with corresponding isend/irecv on the</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Zitong_Li: <p>Thanks for the reply!</p>
<p>I did some benchmarking on my machine. It seems like doing the following:</p>
<pre><code class="lang-auto">send_tensor = torch.zeros(1).cuda()
recv_tensor = torch.zeros(1).cuda()

if rank == 0:
  sendOp = dist.P2POp(dist.isend, send_tensor, 1)
  recvOp = dist.P2POp(dist.irecv, recv_tensor, 1)
  reqs = dist.batch_isend_irecv([sendOp, recvOp])
  for req in reqs:
    req.wait()
elif rank == 1:
   sendOp = dist.P2POp(dist.isend, send_tensor, 0)
   recvOp = dist.P2POp(dist.irecv, recv_tensor, 0)
   reqs = dist.batch_isend_irecv([sendOp, recvOp])
   for req in reqs:
     req.wait()
</code></pre>
<p>takes twice amount of time as doing 1 isend as the following:</p>
<pre><code class="lang-auto">if rank == 0:
  reqs.append(dist.isend(tensor=send_tensor, dst=1))
else:
  reqs.append(dist.irecv(tensor=recv_tensor, src=0))
for req in reqs:
  req.wait()
</code></pre>
<p>Looking at this github <a href="https://github.com/pytorch/pytorch/issues/43995" rel="noopener nofollow ugc">issue</a> It does seem like the batch_isend_irecv() is supposed to support concurrent send/recv. So I don’t know what I’m missing here.</p>