MPI cuda stream

seliad: <p>When we do</p>
<pre><code class="lang-auto">with torch.cuda.stream(stream):
    torch.distributed.isend(...)
</code></pre>
<p>Will if effect the stream (cuda-aware) MPI uses for communication, or rather its some inside MPI implementation detail?</p>

mrshenli: <p>Given the implementation below, it does not seem <code>ProcessGroupMPI</code> uses any dedicated CUDA streams. So I would assume it’s fully delegated to MPI’s implementation?</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/f314d9a0774062a20015ae522d33eadd45293328/torch/lib/c10d/ProcessGroupMPI.cpp#L791-L814" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/f314d9a0774062a20015ae522d33eadd45293328/torch/lib/c10d/ProcessGroupMPI.cpp#L791-L814" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/f314d9a0774062a20015ae522d33eadd45293328/torch/lib/c10d/ProcessGroupMPI.cpp#L791-L814</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="791" style="counter-reset: li-counter 790 ;">
<li>std::shared_ptr&lt;ProcessGroup::Work&gt; ProcessGroupMPI::send(</li>
<li>    std::vector&lt;at::Tensor&gt;&amp; tensors,</li>
<li>    int dstRank,</li>
<li>    int tag) {</li>
<li>  checkSingleTensor(tensors);</li>
<li>
</li><li>  auto&amp; tensor = tensors[0];</li>
<li>  MPI_Request request = MPI_REQUEST_NULL;</li>
<li>
</li><li>  {</li>
<li>    c10::DeviceGuard guard(tensor.device());</li>
<li>    std::unique_lock&lt;std::mutex&gt; globalLock(pgGlobalMutex_);</li>
<li>    MPI_CHECK(MPI_Isend(</li>
<li>        tensor.data_ptr(),</li>
<li>        tensor.numel(),</li>
<li>        mpiDatatype.at(tensor.scalar_type()),</li>
<li>        dstRank,</li>
<li>        tag,</li>
<li>        pgComm_,</li>
<li>        &amp;request));</li>
</ol></code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/f314d9a0774062a20015ae522d33eadd45293328/torch/lib/c10d/ProcessGroupMPI.cpp#L791-L814" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


seliad: <p>You made me open the black box  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I verified its an inside MPI implementation detail. I found it in their code.<br>
<a href="https://github.com/open-mpi/ompi/blob/e9e4d2a4bc4ca34d8f426a5f175e3a9eabe50a66/opal/mca/common/cuda/common_cuda.c" rel="nofollow noopener">for example openmpi</a><br>
they use their own streams.</p>
<p><strong>This is critical,</strong><br>
because unless the streams they create can be accessed somehow  (so far I did not find a way to do it in the cuda manual. but ill look deeper),<br>
it means that the only way to change the behavior is editing the MPI C code and compiling.</p>
<p><strong>Why do normal pytorch users should care?</strong><br>
Because for normal cuda-aware usage this is super-duper risky, as their streams don’t wait for the our streams, meaning cuda-aware MPI is prune to failure unless we fully synchronize our streams before each MPI call.<br>
This would result in slower (or incorrect) program.<br>
(I personally spent tons of time debugging this…)</p>
<p>I’d like to know what you think, maybe we should open an issue.</p>
<p>By the way, I wonder why in the file you mention irecv uses <code>MPI_ANY_SOURCE</code>:<br>
is that intentional?<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/f314d9a0774062a20015ae522d33eadd45293328/torch/lib/c10d/ProcessGroupMPI.cpp#L856" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/f314d9a0774062a20015ae522d33eadd45293328/torch/lib/c10d/ProcessGroupMPI.cpp#L856" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/f314d9a0774062a20015ae522d33eadd45293328/torch/lib/c10d/ProcessGroupMPI.cpp#L856</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="846" style="counter-reset: li-counter 845 ;">
<li>  auto&amp; tensor = tensors[0];</li>
<li>  MPI_Request request = MPI_REQUEST_NULL;</li>
<li>
</li><li>  {</li>
<li>    c10::DeviceGuard guard(tensor.device());</li>
<li>    std::unique_lock&lt;std::mutex&gt; globalLock(pgGlobalMutex_);</li>
<li>    MPI_CHECK(MPI_Irecv(</li>
<li>        tensor.data_ptr(),</li>
<li>        tensor.numel(),</li>
<li>        mpiDatatype.at(tensor.scalar_type()),</li>
<li class="selected">        MPI_ANY_SOURCE,</li>
<li>        tag,</li>
<li>        pgComm_,</li>
<li>        &amp;request));</li>
<li>  }</li>
<li>
</li><li>  return std::make_shared&lt;AsyncWork&gt;(tensor, request);</li>
<li>}</li>
<li>
</li><li>std::shared_ptr&lt;ProcessGroup::Work&gt; ProcessGroupMPI::barrier(</li>
<li>    const BarrierOptions&amp; opts) {</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


mrshenli: <aside class="quote no-group" data-username="seliad" data-post="3" data-topic="80702">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/seliad/48/16357_2.png" class="avatar"> seliad:</div>
<blockquote>
<p>By the way, I wonder why in the file you mention irecv uses <code>MPI_ANY_SOURCE</code> :<br>
is that intentional?</p>
</blockquote>
</aside>
<p>I am not aware of the history here. <a class="mention" href="/u/pietern">@pietern</a> and <a class="mention" href="/u/teng-li">@teng-li</a> would know more.</p>
<blockquote>
<p><strong>Why do normal pytorch users should care?</strong><br>
Because for normal cuda-aware usage this is super-duper risky, as their streams don’t wait for the our streams, meaning cuda-aware MPI is prune to failure unless we fully synchronize our streams before each MPI call.<br>
This would result in slower (or incorrect) program.</p>
</blockquote>
<p>I agree, full synchronization is not acceptable here. Can MPI take a CUDA stream as an argument and then work on that stream like NCCL does? If this is possible we can let <code>ProcessGroupMPI</code> manage the streams and use CUDA event to synchronize.</p>

seliad: <p>Created an issue in openmpi repo.<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/open-mpi/ompi/issues/7733" target="_blank" rel="nofollow noopener">github.com/open-mpi/ompi</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/open-mpi/ompi/issues/7733" target="_blank" rel="nofollow noopener">synchronize cuda-aware mpi streams</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-05-13" data-time="14:02:54" data-timezone="UTC">02:02PM - 13 May 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/saareliad" target="_blank" rel="nofollow noopener">
          <img alt="saareliad" src="https://avatars1.githubusercontent.com/u/22762845?v=4" class="onebox-avatar-inline" width="20" height="20">
          saareliad
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Background information

v4.0.3

installed from source (tar)

cuda aware mpi

cuda 10.2

This is not a system problem, but suspected behavior/implementation issue in cuda-aware MPI. it...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p><a class="mention" href="/u/mrshenli">@mrshenli</a> As far as I know,  MPI doesn’t support what you suggest so better ask them directly?</p>

seliad: <p><a class="mention" href="/u/mrshenli">@mrshenli</a><br>
I think what  <a href="https://github.com/open-mpi/ompi/issues/7733#issuecomment-629806195" rel="nofollow noopener">https://github.com/open-mpi/ompi/issues/7733#issuecomment-629806195</a><br>
Suggests is what should be implemented inside Pytorch in cpp, if we want to use MPI process group correctly.</p>
<p>Maybe add optional event argument to torch.dist calls (<code>cuda_event_to_sync_with</code> or something).</p>
<p>As far as I know, callbacks on cuda events are not exposed to python API.<br>
(Too bad they aren’t, actually)</p>

mrshenli: <p>I see. Thanks for sharing!</p>
<blockquote>
<p>Maybe add optional event argument to torch.dist calls ( <code>cuda_event_to_sync_with</code>  or something).</p>
</blockquote>
<p>I am not sure whether we should add this to c10d Python API if this is only required by the MPI backend. Could you please add an issue on GH to kick off the discussion on the pitch? Let’s discuss there to see what are the options.</p>
<blockquote>
<p>As far as I know, callbacks on cuda events are not exposed to python API.<br>
(Too bad they aren’t, actually)</p>
</blockquote>
<p>We are actually exploring CUDA event callback for RPC, and also considering using it to handle CUDA errors. Let me create an issue to track this.</p>

mrshenli: <aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/38671" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/38671" target="_blank" rel="nofollow noopener">Add CUDA callback to Python API</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-05-18" data-time="18:38:43" data-timezone="UTC">06:38PM - 18 May 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/mrshenli" target="_blank" rel="nofollow noopener">
          <img alt="mrshenli" src="https://avatars2.githubusercontent.com/u/16999635?v=4" class="onebox-avatar-inline" width="20" height="20">
          mrshenli
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">As discussed in this forum thread. CUDA stream/event callback can be a useful feature. It might be helpful to add it...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">feature</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: cuda</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: rpc</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>