Libtorch memory options for tensors - pinned memory, zero copy memory

SM19: <p>Hello Pytorch Team,</p>
<p>I have an application running using Libtorch + TorchTensorrt.<br>
For this, I create input by first allocating a tensor of shape BCHW on GPU and then writing values into the pixels. However, I see this is super slow - slower than Python when copying data from numpy array.</p>
<p>So, I was wondering if using a pinned memory/ zero copy memory would help???</p>
<p>In any case, I would like to know how can I create tensors with pinned and zero-copy memory models?<br>
Also, if I pre-allocate such an array on GPU using normal CUDA APIs, is it possible to then later copy it to libtorch tensor?</p>
<p>Please help me understand this scenario.</p>
<p>Summary - What is the fastest way to create/ copy data to GPU in Libtorch?</p>
<p>Apart from this - a remark: The documentation about Torch-Tensorrt seems to be heavily outdated.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96405ff6c5bc31d18cc98e99361ff5dd6397e57a.png" data-download-href="https://discuss.pytorch.org/uploads/default/96405ff6c5bc31d18cc98e99361ff5dd6397e57a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96405ff6c5bc31d18cc98e99361ff5dd6397e57a.png" alt="image" data-base62-sha1="lrbxFy5UrP9SSwVGV0gHyl4vHK2" width="690" height="110" data-dominant-color="E9E9E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">691×111 6.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is on the main documentation page and it would probably not work as the input should be of IValue type - this can be found in other pages of documentation.</p>
<p>Please update the information if this is indeed the case.</p>
<p>Best Regards<br>
Sambit</p>

ptrblck: <aside class="quote no-group" data-username="SM19" data-post="1" data-topic="157420">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sm19/48/48889_2.png" class="avatar"> SM19:</div>
<blockquote>
<p>how can I create tensors with pinned and zero-copy memory models?</p>
</blockquote>
</aside>
<p>You should be able to specify it in the options you would pass the the tensor initialization:</p>
<pre><code class="lang-auto">TensorOptions options = TensorOptions().dtype(dtype).layout(layout).device(device).pinned_memory(pin_memory);
</code></pre>
<aside class="quote no-group" data-username="SM19" data-post="1" data-topic="157420">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sm19/48/48889_2.png" class="avatar"> SM19:</div>
<blockquote>
<p>Also, if I pre-allocate such an array on GPU using normal CUDA APIs, is it possible to then later copy it to libtorch tensor?</p>
</blockquote>
</aside>
<p>Yes, <code>from_blob</code> should work but you would have to make sure the actual memory doesn’t go out of scope or is released, as PyTorch will not trigger a copy (you can of course manually trigger a copy if needed).</p>

SM19: <p>Two small follow up questions:</p>
<ol>
<li>
<p>Does creating tensor on GPU take significant time? Or is it only data transfer that takes time? I mean, if my function creates a static zero tensor on GPU, should I expect to see any improvement in speed, given the fact that I will still create input data in a normal array and copy it to the zero GPU tensor using from_blob, in each call to the function.</p>
</li>
<li>
<p>What is the C++ equivalent for - torch.cuda.empty_cache()</p>
</li>
</ol>