Non blocking memory transfer to GPU

yiftach: <p>Hi,</p>
<p>I have a few questions about non blocking memory transfer from CPU to GPU.</p>
<ol>
<li>CUDA opeations are asynchronous, so as far as I understand when I run <code>x.square()</code> (for an <code>x</code> that is on GPU) I actually just enqueue the square kernel to be run at a later time. Why then can’t all memory transfers from CPU to GPU be async? Can’t a code like the following first enqueue a “move to GPU memory” event followed by an “execute the square kernel” event?</li>
</ol>
<pre><code class="lang-auto">x = x.to('cuda', non_blocking=True)
x = x.square()
</code></pre>
<ol start="2">
<li>Why can non-blocking transfer only happen for memory-pinned (page-locked) tensors?</li>
<li>What happens if I run the following?</li>
</ol>
<pre><code class="lang-auto">x = x.to('cuda', non_blocking=True)
...
x = x.to('cuda')
</code></pre>
<p>+does the answer differ if the second line is called when the data of <code>x</code> has already been transferred? Specifically, if I use a non-blocking transfer and somewhere later in the code a blocking transfer occurs, do I risk losing the benefits?</p>
<p>Thanks,<br>
Yiftach</p>

ptrblck: <p>You can make data transfers asynchronous if you want to pin the corresponding host memory. Since allocating a lot of pinned memory might cause issues it’s not the default but can be enabled in the <code>DataLoader</code>.<br>
<a href="https://docs.nvidia.com/cuda/cuda-runtime-api/api-sync-behavior.html#api-sync-behavior__memcpy-async">These docs</a> explain the sync/async behavior in more detail.</p>

yiftach: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, I’m aware of pinning memory on DataLoader. The text in the link has me confused:</p>
<blockquote>
<ol>
<li>For transfers from pageable host memory to device memory, a stream sync is performed before the copy is initiated. The function will return once the pageable buffer has been copied to the staging memory for DMA transfer to device memory, but the DMA to final destination may not have completed.</li>
<li>For transfers from pinned host memory to device memory, the function is synchronous with respect to the host.</li>
</ol>
</blockquote>
<p>Sounds to me opposite of what PyTorch docs say (pinned can be async, default non-pinned is sync).<br>
I’m aware the misunderstanding is most likely on my side, but would still love a more in-depth explanation (or a reference to a more detailed resource). I’d also still love an answer to each of the questions, if possible.</p>

bloos: <p>can you elaborate on the issues that can be caused by allocating a lot of pinned memory or give a link to a resource? In my project where I’m hunting my OOM Issues ( <a href="https://discuss.pytorch.org/t/out-of-memory-after-2-hours/188904/3">Out of Memory after 2 hours - audio - PyTorch Forums</a>) I’ve actually set pin_memory to True and I’m interested in whether this might have caused the problems.</p>

yiftach: <p>Pinned memory is page-locked - the OS cannot swap it out, see e.g.:</p><aside class="quote" data-post="13" data-topic="1702">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-is-the-disadvantage-of-using-pin-memory/1702/13">What is the disadvantage of using pin_memory?</a> 
  </div>
  <blockquote>
    Pinning memory disallows the system to use this memory for its allocations. A very simple example of an undesired behavior would be to pin too much memory, which would then force your OS into memory thrashing and moving pages into the swap thus tanking the performance of your entire system.
  </blockquote>
</aside>