Does the NCCL operation use the default stream as other computations?

lhb8125: <aside class="onebox githubissue">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/60511" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/60511" target="_blank" rel="noopener nofollow ugc">Does the NCCL operation use the default stream as other computations?</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2021-06-23" data-time="04:46:10" data-timezone="UTC">04:46AM - 23 Jun 21 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/lhb8125" target="_blank" rel="noopener nofollow ugc">
          <img alt="lhb8125" src="https://avatars.githubusercontent.com/u/16619280?v=4" class="onebox-avatar-inline" width="20" height="20">
          lhb8125
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: nccl</span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## ❓ Questions and Help

### Does the overlap occur between communication and <span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">computation?

Let's take the NCCL backend as an example, if I launch a collective operation, and then another related computation:
```
dist.allreduce(tensor, op=dist.ReduceOp.SUM, group=group)
tensor = tensor * 2
```
Is a CUDA synchronization essential?

### Related documentation

&gt; Synchronous operation - the default mode, when async_op is set to False. When the function returns, it is guaranteed that the collective operation is performed. In the case of CUDA operations, it is not guaranteed that the CUDA operation is completed, since CUDA operations are asynchronous. For CPU collectives, any further function calls utilizing the output of the collective call will behave as expected. For CUDA collectives, function calls utilizing the output on the same CUDA stream will behave as expected. Users must take care of synchronization under the scenario of running under different streams. For details on CUDA semantics such as stream synchronization, see CUDA Semantics. See the below script to see examples of differences in these semantics for CPU and CUDA operations.
 
Base on the above description, I guess synchronization is unnecessary.

However, my previous investigation shows that the NCCL operations are launched on a separated stream, "ncclStream":
[code](https://github.com/pytorch/pytorch/blob/ed1da5be210c31cc07b033ac0f19f3dd6366feac/torch/lib/c10d/ProcessGroupNCCL.cpp#L1073)
So if I don't specify any stream, will the communication and computation be launched on the same stream?</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


pritamdamania87: <p>Responded in <a href="https://github.com/pytorch/pytorch/issues/60511" class="inline-onebox" rel="noopener nofollow ugc">Does the NCCL operation use the default stream as other computations? · Issue #60511 · pytorch/pytorch · GitHub</a>, please don’t duplicate questions across forums and github.</p>