What's the performance difference between isend/irecv and batch_isend_irecv

username: AndroidSheepy
cooked: <p>Hi dear developers,</p>
<p>Recently, I read the context parallelism code of attention in TransformerEngine and found that it uses a ring P2P communication architecture to transfer data, the code below is the core part of the communication that sends and receives a tensor at a time:</p>
<pre><code class="lang-auto">    send_recv_ops = []

    if batch_p2p_comm:
        if rank % 2 == 0:
            send_op = torch.distributed.P2POp(torch.distributed.isend,
                                              send_tensor,
                                              send_dst,
                                              cp_group)
            recv_op = torch.distributed.P2POp(torch.distributed.irecv,
                                              recv_tensor,
                                              recv_src,
                                              cp_group)
            send_recv_ops.append(send_op)
            send_recv_ops.append(recv_op)
        else:
            recv_op = torch.distributed.P2POp(torch.distributed.irecv,
                                              recv_tensor,
                                              recv_src,
                                              cp_group)
            send_op = torch.distributed.P2POp(torch.distributed.isend,
                                              send_tensor,
                                              send_dst,
                                              cp_group)
            send_recv_ops.append(recv_op)
            send_recv_ops.append(send_op)
        send_recv_reqs = torch.distributed.batch_isend_irecv(send_recv_ops)
    else:
        if rank % 2 == 0:
            send_op = torch.distributed.isend(send_tensor, send_dst, cp_group)
            recv_op = torch.distributed.irecv(recv_tensor, recv_src, cp_group)
            send_recv_ops.append(send_op)
            send_recv_ops.append(recv_op)
        else:
            recv_op = torch.distributed.irecv(recv_tensor, recv_src, cp_group)
            send_op = torch.distributed.isend(send_tensor, send_dst, cp_group)
            send_recv_ops.append(recv_op)
            send_recv_ops.append(send_op)
        send_recv_reqs = send_recv_ops

    return send_recv_reqs
</code></pre>
<p>I noticed a flag called <em>batch_p2p_comm</em> (default is False) which enables batch_isend_irecv() instead of the individual isend() and irecv() calls. I enabled this flag to see if it could accelerate the communication, but after profiling, I found that the communication time using single isend() and irecv() was nearly the same as using batch_isend_irecv().</p>
<p>Given these results, I am curious about the differences between batch_isend_irecv() and isend()/irecv(), since both support asynchronous communication. Can batch_isend_irecv() improve communication speed?</p>
<p>Any help would be appreciated, thank you very much.</p>

username: wconstab
cooked: <p>I did a relevant experiment a while back, probably on H100 hardware (8 gpus via nvlink):</p>
<aside class="onebox githubgist" data-onebox-src="https://gist.github.com/wconstab/88ce5d34e355fa8ebbb3a76e67209270">
  <header class="source">

      <a href="https://gist.github.com/wconstab/88ce5d34e355fa8ebbb3a76e67209270" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/wconstab/88ce5d34e355fa8ebbb3a76e67209270" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/wconstab/88ce5d34e355fa8ebbb3a76e67209270</a></h4>

  <h5>ring.py</h5>
  <pre><code class="Python">import argparse
import torch
import logging
from tabulate import tabulate
import timeit
import torch.distributed as dist

from torch._logging._internal import TorchLogsFormatter
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()</code></pre>
   This file has been truncated. <a href="https://gist.github.com/wconstab/88ce5d34e355fa8ebbb3a76e67209270" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>This isn’t fully conclusive, i didn’t test various versions of NCCL, different GPU types, and importantly, larger clusters (over ethernet or infiniband).</p>

username: Chen_Hongyu
cooked: <p>I guess this could have relationship with ncclgroup…?</p>