How to free GPU memory of at::Tensor in ATen, C++?

temtemp3211345_tempt: <p>I would like to use network in C++ by building tensors and operations of ATen using GPU, but it seems to be impossible to free GPU memory of tensors automatically. Is there any way to use garbage collector or some thing like it supported by ATen? Used platform are Windows 10, CUDA 8.0, CUDNN 7, Pytorch 0.4.0.</p>
<p>I found that ATen library provides automatically releasing memory of a tensor when its reference count becomes 0 if the tensor resides in CPU memory. In below code, I could find that CPU memory of the tensor is freed at (3*).</p>
<pre><code class="lang-auto">/* example 1 */
// cpu memory check (1*)
{
  auto tensor = at::CPU(at::kFloat).ones({1000*1000*400});
  // cpu memory chec (2*)
}
// cpu memory check (3*)
</code></pre>
<p>However, I found that CUDA tensor is not released (3**) in example 2 and that it is released only after the end of the program. I checked GPU memory at every (**) point with MSI Afterburner monitoring program.</p>
<pre><code class="lang-auto">/* example 2 */
// cpu memory check (1**)
{
  auto tensor = at::CUDA(at::kFloat).ones({1000*1000*400});
  // cuda memory chec (2**)
}
// cuda memory check (3**)
</code></pre>
<p>It found that CUDA tensor of ATen can be freed as following although I do not know that it is intended way and recommanded method to release memory. However, it seems that we can hardly use cudaFree to every single tensor output of ATen function such as at::conv2d</p>
<pre><code class="lang-auto">auto tensor = at::CUDA(at::kFloat).ones({1000*1000*400});
cudaFree(tensor.storage()-&gt;data()); // ok. gpu memory is freed
...
out = at::conv2d(out, ...); // how to delete this intermediate tensor without (auto out2 = at::conv2d(out);)?
out = at::conv2d(out, ...);
out = at::conv2d(out, ...);
</code></pre>

SimonW: <p>This is because of the caching allocator. <a href="https://pytorch.org/docs/master/notes/cuda.html#memory-management">https://pytorch.org/docs/master/notes/cuda.html#memory-management</a>. Not sure what is the equivalent in c++, but the python way is to <code>torch.cuda.empty_cache()</code>.</p>

tom: <p>Given that people link this thread and appear to be looking for it:</p>
<p>As Simon says, when a Tensor (or all Tensors referring to a memory block (a Storage)) goes out of scope, the memory goes back to the cache PyTorch keeps. You can free the memory from the cache using</p>
<p><code>#include &lt;c10/cuda/CUDACachingAllocator.h&gt;</code></p>
<p>and then calling</p>
<p><code>c10::cuda::CUDACachingAllocator::emptyCache();</code></p>
<p>(of course, you could try using <code>torch::</code> instead of <code>c10::</code> and see if it is automatically imported somewhere).</p>
<p>Best regards</p>
<p>Thomas</p>

dshin: <aside class="quote no-group" data-username="tom" data-post="3" data-topic="18408">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<pre><code class="lang-auto">#include &lt;c10/cuda/CUDACachingAllocator.h&gt;
</code></pre>
</blockquote>
</aside>
<p>I find that merely including this file leads to a compile error:</p>
<pre><code class="lang-auto">/home/dshin/libtorch/include/c10/cuda/CUDAStream.h: In constructor ‘c10::cuda::CUDAStream::CUDAStream(c10::Stream)’:
/home/dshin/libtorch/include/c10/cuda/CUDAStream.h:69:25: error: expected unqualified-id before ‘(’ token
   69 |     TORCH_CHECK(stream_.device_type() == DeviceType::CUDA);
      |                         ^~~~~~~~~~~
</code></pre>
<p>My build-version is <code>1.13.0+cu116</code>.</p>