Tensor.to, 'non-blocking' & cuda streams optimisation

username: wmcelderry
cooked: <p>Hi All,</p>
<h2>
<a name="thank-you-1" class="anchor" href="#thank-you-1"></a>Thank you!</h2>
<p>Thanks for all the hard work: making some incredibly powerful software, and helping newer members of the community (like me) to get to grips with it!</p>
<h2>
<a name="h-2-low-level-questions-2" class="anchor" href="#h-2-low-level-questions-2"></a>2 low-level questions</h2>
<p>I’m hoping someone can help me get to the bottom of how</p>
<pre><code class="lang-auto">Tensor.to(device=gpu,non_blocking=True)
</code></pre>
<p>actually works.  (I’ve tried searching the GitHub Repo, but it looks like a complex hierarchy of classes and implementation that has left me a bit <em>bewildered</em> - probably due to being very efficient &amp; implemented in C/C++?)</p>
<ol>
<li>Where is the implementation for ‘Tensor.to’?</li>
<li>and/or how does ‘non_blocking’ relate to cuda streams?</li>
</ol>
<h2>
<a name="my-motivation-3" class="anchor" href="#my-motivation-3"></a>My motivation:</h2>
<p>I’ve been reading some of the posts around cuda streams on the nvidia blog and want to implement my own strategy to optimise transfers from system RAM to GPU RAM, computation and then returning the results to the system RAM.  I’ve got some code running and it is more efficient than without the streams, but there seems to be a lot more data transfer than I’m expecting, and I’m wondering if this is somehow linked with ‘non_blocking’.</p>
<h2>
<a name="wider-context-again-4" class="anchor" href="#wider-context-again-4"></a>Wider context again:</h2>
<p>I want the transfers to be non-blocking to allow the CPU to queue up multiple transfers while the GPU is performing calculations, to maximise throughput.</p>
<p>To that end I envisage two strategies right now:</p>
<ul>
<li>A) use one cuda stream for data-transfer, subsequent computation and return to the host, repeat many times in parallel  (possibly using an event to wait for a computation to finish before starting the next to ensure that there are never too many kernels running in parallel that cause an OOM error)</li>
<li>B) use one stream for data-transfers and use events to synchronise prior to computation in another cuda stream (likely slower as computation would be forced to be more sequential, but as long as the queue is big enough, this would also ensure models do not start running in parallel causing OOM errors), then use another event to synchronise a return to the system RAM.</li>
</ul>
<p>I would not be surprised to discover this has been implemented - I cannot seem to find ‘torch::deploy’ in the docs, so please forgive me if I’m missing something obvious!</p>
<h2>
<a name="h-2-high-level-questions-5" class="anchor" href="#h-2-high-level-questions-5"></a>2 high-level questions</h2>
<ol start="3">
<li>
<p>Would both strategies work with streams and ‘non_blocking’ transfers, or do ‘non_blocking’ transfers use a dedicated stream meaning there has to be more consideration of synchronisation?</p>
</li>
<li>
<p>Any thoughts on other strategies I should consider?</p>
</li>
</ol>
<p>Thanks!</p>

username: googlebot
cooked: <p>I’ll quote cuda c programming guide:</p>
<h3>
<a name="h-3264-concurrent-data-transfershttpsdocsnvidiacomcudacuda-c-programming-guideindexhtmlconcurrent-data-transfers-1" class="anchor" href="#h-3264-concurrent-data-transfershttpsdocsnvidiacomcudacuda-c-programming-guideindexhtmlconcurrent-data-transfers-1"></a><a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#concurrent-data-transfers" rel="noopener nofollow ugc">3.2.6.4. Concurrent Data Transfers</a>
</h3>
<p>Some devices of compute capability 2.x and higher can overlap copies to and from the device. Applications may query this capability by checking the asyncEngineCount device property (see <a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#device-enumeration" rel="noopener nofollow ugc">Device Enumeration</a>), which is equal to 2 for devices that support it. In order to be overlapped, any host memory involved in the transfers must be page-locked.</p>
<p>(3.2.5.3) Mapped memory<br>
…<br>
There is no need to use streams (see <a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#concurrent-data-transfers" rel="noopener nofollow ugc">Concurrent Data Transfers</a>) to overlap data transfers with kernel execution; the kernel-originated data transfers automatically overlap with kernel execution.</p>
<p>see also: <a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#overlapping-behavior" rel="noopener nofollow ugc">3.2.6.5.5. Overlapping Behavior</a></p>
<p>so, basically, x.pin_memory().to(…,non_blocking=True) corresponds to cudaMemcpyAsync, however pin_memory() has to copy data first, unless it is page locked in advance…</p>

username: wmcelderry
cooked: <p>Hi Alex,</p>
<p>Thanks for your reply!</p>
<p>I haven’t found those guides before, so I’ll spend a while reading through in the next few days. I think they will provide a wealth of clues, but may lead to me struggling to relate the cuda concepts to pytorch code even more.</p>
<p>If anyone can provide an insight in to the relationships between cuda and pytorch (I guess there could be internal development docs?) that may short-cut a number of follow up questions.</p>

username: googlebot
cooked: <p>relation? well, python frontend connects to c++ backend, the backend dispatches to device-specific modules, including cuda (nvcc compiled .cu). naturally, there is a lot of adaptation code in all layers.</p>

username: wmcelderry
cooked: <p>Hi Alex,</p>
<p>Thanks again for that.</p>
<p>I guess there may be some kind of demarcation of “python frontend”, “c++ backend”  and “device-specific modules” inside the source repo?</p>
<p>I have only just found the <a href="https://pytorch.org/cppdocs/" rel="noopener nofollow ugc">documentation on the C++ API</a>, so I can begin reading that as well, and that will give me a better understanding of the underlying structure too by the look of it.</p>

username: googlebot
cooked: <p>well, it is a big c++ codebase, they’re seldom easily approachable… apart from website docs, there are also some scattered readme.md files and some non-published memos in docs/source/, but it is easier to to search for something and then navigate from there.</p>
<p>e.g. Tensor::to is in aten/src/ATen/native/TensorConversions.cpp, I think execution will lead to aten/src/ATen/native/cuda/Copy.cu. in general, most of interesting stuff is under aten/src/ATen.</p>

username: marksaroufim
cooked: <p>I realize this not a direct answer to your question but this tutorial by Ed Yang really helped me understand a bit more what’s going on <a href="http://blog.ezyang.com/2020/09/lets-talk-about-the-pytorch-dispatcher/" class="inline-onebox" rel="noopener nofollow ugc">Let’s talk about the PyTorch dispatcher : ezyang’s blog</a></p>