Can we overlap compute operation with memory operation without pinned memory on CPU?

username: rhwang
cooked: <p>Hi, I`m trying to overlap the computation and memory operation with HuggingFace SwitchTransformer.</p>
<p>Here’s a detailed explanation.</p>
<ul>
<li>The memory operation is for data movement from CPU to GPU, and its size is 4MB per block.</li>
<li>The number of blocks is variable (typically from 2 to 6 in total).</li>
<li>The computation operation comprises several very small computation operations like GEMM, which takes 10s to 100s microseconds per each.</li>
<li>I’m trying to use CudaStream, so I created two different Cuda streams and pushed memory operation and computation operation to each of them.</li>
<li>But it had not been overlapped.</li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c81014c2b4596e02379fe63c3c782d8ba943fe8f.png" data-download-href="https://discuss.pytorch.org/uploads/default/c81014c2b4596e02379fe63c3c782d8ba943fe8f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c81014c2b4596e02379fe63c3c782d8ba943fe8f_2_690x295.png" alt="image" data-base62-sha1="sxPWKSrfPFEp9w66SFTicyWbIIL" width="690" height="295" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c81014c2b4596e02379fe63c3c782d8ba943fe8f_2_690x295.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c81014c2b4596e02379fe63c3c782d8ba943fe8f_2_1035x442.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/8/c81014c2b4596e02379fe63c3c782d8ba943fe8f.png 2x" data-dominant-color="E0D8D2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1057×453 114 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And here’s my question.</p>
<ol>
<li>
<p>Firstly, I’d learn that to overlap the memory operation (CPU-&gt;GPU) and computation operation, the memory in the CPU should be pinned. But in my case, as can be seen in the figure, it is pageable memory, not pinned. Is it a reason that this cannot be overlapped?</p>
</li>
<li>
<p>Second, I conducted an experiment to prove it with a simple example (overlapping GEMM with CPU-&gt;GPU memory operation), and here`s the output.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4abf69916ab6bcd0cd45ed4cbd76b0d4503eeffb.png" data-download-href="https://discuss.pytorch.org/uploads/default/4abf69916ab6bcd0cd45ed4cbd76b0d4503eeffb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4abf69916ab6bcd0cd45ed4cbd76b0d4503eeffb_2_690x192.png" alt="image" data-base62-sha1="aFfqMmlynVnXGuXzp1iOJilkyAb" width="690" height="192" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4abf69916ab6bcd0cd45ed4cbd76b0d4503eeffb_2_690x192.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4abf69916ab6bcd0cd45ed4cbd76b0d4503eeffb_2_1035x288.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4abf69916ab6bcd0cd45ed4cbd76b0d4503eeffb_2_1380x384.png 2x" data-dominant-color="D3DFCB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1473×410 16.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This is pageable memory.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/8261178566d0f8a63b7271c28860109f109a94fc.png" data-download-href="https://discuss.pytorch.org/uploads/default/8261178566d0f8a63b7271c28860109f109a94fc" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/8261178566d0f8a63b7271c28860109f109a94fc_2_690x179.png" alt="image" data-base62-sha1="iBo5yemOfpQvPKIydDnoq1Vgi16" width="690" height="179" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/8261178566d0f8a63b7271c28860109f109a94fc_2_690x179.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/8261178566d0f8a63b7271c28860109f109a94fc_2_1035x268.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/8261178566d0f8a63b7271c28860109f109a94fc_2_1380x358.png 2x" data-dominant-color="BCDAD6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1446×376 18.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This is pinned memory.</p>
</li>
</ol>
<p>It seems like pageable memory also can be overlapped.<br>
Then, what is the reason that my application is not overlapping?</p>

username: DeanHHH
cooked: <p>I am also very concerned about this question but I have not found the answer. Do you have any progress?</p>