Who get memory rights on tensors created in Cuda code?

username: el_samou_samou
cooked: <p>Hi,</p>
<p>I was wondering who have “memory rights” on tensors created in a pytorch Cuda extension.</p>
<p>I have created several custom cuda extension following this tutorial: <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_extension.html</a>.</p>
<p>I am now using those extensions in a training loop (i.e. a network is using some of my custom operations) and I noticed that the memory usage strictly increase at each training iteration. I think it might come from those custom operations so I would like to know:</p>
<ol>
<li>
<p>If a tensor is created in Cuda code binded to python and then returned by the custom function, if then the python script delete this tensor, does it effectively clear the memory?</p>
</li>
<li>
<p>What about if a tensor is created in Cuda code but just contains intermediate results and so is not return by the Cuda function, is it deleted once we exit the function scope like in Python or do we have to delete it manually? If so and if the answer to the first question is yes, is the recommended practice to return all created tensors to then delete it in python or to handle those in the Cuda code itself?</p>
</li>
</ol>
<p>I hope my question is clear, thank you for your help in advance.</p>
<p>Samuel</p>

username: albanD
cooked: <blockquote>
<p>if then the python script delete this tensor, does it effectively clear the memory?</p>
</blockquote>
<p>Yes</p>
<blockquote>
<p>is it deleted once we exit the function scope like in Python or do we have to delete it manually?</p>
</blockquote>
<p>It is deleted when you exit the scope, unless you return it.</p>
<p>How do you monitor the cuda memory? Note that we have a caching allocator that does not return the memory to the driver. So <code>nvidia-smi</code> memory report will always grow.<br>
You can use <code>torch.cuda.memory_allocated()</code> to see the memory actually used to store Tensors.</p>

username: el_samou_samou
cooked: <p>Thank you for your answer. I used <code>torch.cuda.empty_cache()</code> followed by <code>torch.cuda.memory_summary()</code> to monitor the memory. I noticed an increasing gap between the total allocated memory and the total freed memory. This gap gets larger with the number of iteration. This is why I assume an issue like some memory created at every iteration not being freed.</p>

username: albanD
cooked: <p>But do you actually run out of memory?<br>
It is expected that pytorch will allocate more and more memory for the first iterations then it will stabilize after a while.</p>
<p>If you do run out of memory, you should try and pinpoint where it happens. Try replacing your cpp code by a python version the just returns an output of the right size. And see if it still happens.</p>

username: el_samou_samou
cooked: <p>Yes I do run out of memory very fast actually as the difference between allocated memory and freed memory is approximately 2GB at every iteration. I am almost certain it is not a normal behavior and there is something I do wrong.</p>
<p>I will try your suggestion and see how it goes. Thank you for your help.</p>

username: el_samou_samou
cooked: <p>Hi,</p>
<p>So for those of you who have similar issue in the future. I solved my problem by seeing this post:</p><aside class="quote quote-modified" data-post="9" data-topic="6522">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aviolov/48/1589_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/when-should-you-save-for-backward-vs-storing-in-ctx/6522/9">When should you save_for_backward vs. storing in ctx</a> 
  </div>
  <blockquote>
    We experienced a memory leak on a custom module using custom-make GPU kernels. Storing intermediate results in ctx.x, ctx.z etc resulted in them not being released over multiple mini-batches, quickly exhausting GPU memory. We did not experience this in pytorch 0.3.1, it sounds like it is due to some changes in pytorch 0.4. Either explicitly del ctx.x etc or adding x,z to .save_for_backward both fixed the memory leak. I would still like to get a good explanation for why you would want to use save…
  </blockquote>
</aside>
<p>
And so indeed I manually deleted context attributes corresponding to tensors I obtained with my custom Cuda operations and it solved the memory leak.</p>

username: albanD
cooked: <p>You should use <code>save_for_backward()</code> instead of deleting them <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
Nice find on the root cause.</p>