How does pytorch transfer data between GPUs

Yanan: <p><a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html</a><br>
This page explains how we can split a big nn model on multiple GPUs. But I don’t understand when the intermediate training result needs to be transferred from one GPU to another (by using “.to(‘cuda:1’)”), does pytorch runtime moves the data to CPU mem first and then to another gpu’s mem, or pytorch actually utilizes some direct data transfer technique between GPUs like SLI?</p>
<p>Any help would be appreciated.</p>

mrshenli: <p>Hey <a class="mention" href="/u/yanan">@Yanan</a></p>
<p>In this case, the tensor will be directly copied from device to device using <code>cudaMemcpyAsync</code>. The C++ implementation is linked below:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/89c0efb30b5bad49e717c1b1a797bac3c62e8b7e/aten/src/ATen/native/cuda/Copy.cu#L56-L67" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/89c0efb30b5bad49e717c1b1a797bac3c62e8b7e/aten/src/ATen/native/cuda/Copy.cu#L56-L67" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/89c0efb30b5bad49e717c1b1a797bac3c62e8b7e/aten/src/ATen/native/cuda/Copy.cu#L56-L67</a></h4>
<pre class="onebox"><code class="lang-cu"><ol class="start lines" start="56" style="counter-reset: li-counter 55 ;">
<li>if (memcpy_eligible) {</li>
<li>  void *dst = iter.data_ptr(0);</li>
<li>  void *src = iter.data_ptr(1);</li>
<li>  size_t size = numel * iter.element_size(0);</li>
<li>  if (src != dst || src_device != dst_device) {</li>
<li>    // Perform the copy</li>
<li>    AT_CUDA_CHECK(cudaMemcpyAsync(</li>
<li>        dst, src, size,</li>
<li>        cudaMemcpyDeviceToDevice,</li>
<li>        copy_stream));</li>
<li>  }</li>
<li>} else {</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Yanan: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> Thanks! This really helps.<br>
I have another question that does data copy  between GPUs in DataParallel module utilize the same kernel?<br>
Sorry I forgot to ask this before.</p>

mrshenli: <blockquote>
<p>I have another question that does data copy between GPUs in DataParallel module utilize the same kernel?</p>
</blockquote>
<p>Yes. <code>DataParallel</code> calls into scatter. Its C++ implementation is linked below. It’s basically calling <code>Tensor.to</code> in a loop</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/89c0efb30b5bad49e717c1b1a797bac3c62e8b7e/torch/csrc/cuda/comm.cpp#L201-L206">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/89c0efb30b5bad49e717c1b1a797bac3c62e8b7e/torch/csrc/cuda/comm.cpp#L201-L206" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/89c0efb30b5bad49e717c1b1a797bac3c62e8b7e/torch/csrc/cuda/comm.cpp#L201-L206" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/89c0efb30b5bad49e717c1b1a797bac3c62e8b7e/torch/csrc/cuda/comm.cpp#L201-L206</a></h4>



    <pre class="onebox"><code class="lang-cpp">
      <ol class="start lines" start="201" style="counter-reset: li-counter 200 ;">
          <li>chunks[chunk] =</li>
          <li>    chunks[chunk].to(</li>
          <li>        {DeviceType::CUDA, device_index},</li>
          <li>        /*non_blocking=*/true,</li>
          <li>        /*copy=*/false,</li>
          <li>        /*memory_format=*/at::MemoryFormat::Preserve);</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>