Copy_() memory leak?

dshin: <p>I have the following simple function:</p>
<pre><code class="lang-auto">void experiment(const char* filename, bool use_copy) {
  torch::jit::script::Module module(torch::jit::load(filename));
  module.to(torch::kCUDA);
  torch::Tensor input = torch::zeros({8, 2, 7, 6});
  torch::Tensor output = torch::zeros({8, 7});

  std::vector&lt;torch::jit::IValue&gt; input_vec;

  for (int i = 0; i &lt; 10000; ++i) {
    torch::Tensor gpu_input = input.clone().to(torch::kCUDA);
    input_vec.push_back(gpu_input);
    auto gpu_output = module.forward(input_vec).toTuple()-&gt;elements()[0].toTensor();
    if (use_copy) {
      output.copy_(gpu_output);
    } else {
      output = gpu_output;
      output.to(torch::kCPU);
    }
    input_vec.clear();
    if (i % 100 == 0) {
      dump_cuda_memory_info();
    }
  }
}
</code></pre>
<p>When I use <code>use_copy=true</code>, I observe the free memory of the GPU rapidly decrease, indicating a memory leak:</p>
<pre><code class="lang-auto">GPU 0 memory: free=21759655936, total=25438715904
GPU 0 memory: free=21078081536, total=25438715904
GPU 0 memory: free=20396507136, total=25438715904
GPU 0 memory: free=19714932736, total=25438715904
GPU 0 memory: free=19033358336, total=25438715904
GPU 0 memory: free=18351783936, total=25438715904
GPU 0 memory: free=17670209536, total=25438715904
GPU 0 memory: free=16988635136, total=25438715904
GPU 0 memory: free=16307060736, total=25438715904
GPU 0 memory: free=15625486336, total=25438715904
...
</code></pre>
<p>When I pass <code>use_copy=false</code>, the free memory stays fixed, indicating no memory leak.</p>
<p>In my application, however, the <code>use_copy=false</code> approach is not viable, as I require the CPU output tensor’s data ptr to be unchanging.</p>
<p><strong>What is the right way to copy tensors from GPU to a fixed CPU memory address without leaking memory?</strong></p>
<p>Elsewhere in these forums, I found the <a href="https://discuss.pytorch.org/t/how-to-free-gpu-memory-of-at-tensor-in-aten-c/18408/3">suggestion</a> to free memory from the cache by calling <code>emptyCache()</code>. This does not seem viable anymore, as simply adding the line:</p>
<pre><code class="lang-auto">#include &lt;c10/cuda/CUDACachingAllocator.h&gt;
</code></pre>
<p>leads to a <a href="https://discuss.pytorch.org/t/how-to-free-gpu-memory-of-at-tensor-in-aten-c/18408/4">compile error</a> using build-version <code>1.13.0+cu116</code>.</p>

eqy: <p>Could you check if detaching the tensor if you are not using the computation graph or calling backward before calling <code>copy_</code> (similar to as described here <a href="https://discuss.pytorch.org/t/copy-operations-get-repeated-in-autograd-computation-graph/122547" class="inline-onebox">`copy_` operations get repeated in autograd computation graph</a>) helps?<br>
Alternatively, if you know that gradients are not needed anywhere, you could try using the <code>no_grad</code> guard as well:<br>
<a href="https://pytorch.org/cppdocs/api/typedef_namespacetorch_1abf2c764801b507b6a105664a2406a410.html#typedef-torch-nogradguard" class="inline-onebox" rel="noopener nofollow ugc">Typedef torch::NoGradGuard — PyTorch master documentation</a></p>

dshin: <p>Thanks, calling <code>.detach()</code> did the trick!</p>