Element-wise operations between two convolution cause memory leak

username: Qizhou
cooked: <p>Hi! I found that torch.softmax cause GPU memory leak.<br>
My pytorch version is 1.8.1+cu111.<br>
When I run the code below:</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.nn import functional as F
from torch import cuda 
 
def test(inp): 
  w = torch.rand([32, 1, 1, 1],device='cuda') 
  y = torch.softmax(F.conv2d(inp, w), 1)  
  y = F.conv_transpose2d(y, w) 
  return y

imgs = torch.zeros([128,1,512,512],device='cuda')
outp = test(imgs)  
 
# del outp
cuda.empty_cache()
print(cuda.memory_summary()) 
</code></pre>
<p>The output is:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c875dd778808ac55145e395445d3c4874957bb6e.png" data-download-href="https://discuss.pytorch.org/uploads/default/c875dd778808ac55145e395445d3c4874957bb6e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c875dd778808ac55145e395445d3c4874957bb6e.png" alt="image" data-base62-sha1="sBm1aHuq0mbgv611ryp5c7VM2YK" width="448" height="499" data-dominant-color="323132"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">624×696 15.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
After the function outputs the result, the variables inside the function should be released, but it did not. In addition, if you delete the variable<code>outp</code>, the redundant occupied memory will be released.<br>
For conparison, I wrote a softmax myself and it did not has the condition of memory leak. The code is below:</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.nn import functional as F
from torch import cuda 

def softmax(x,dim):
  ex = torch.exp(x)
  return ex/torch.sum(ex, dim,keepdim=True)

def test(inp): 
  w = torch.rand([32, 1, 1, 1],device='cuda') 
  y = softmax(F.conv2d(inp, w), 1)  
  y = F.conv_transpose2d(y, w) 
  return y

imgs = torch.zeros([128,1,512,512],device='cuda')
outp = test(imgs)  
 
cuda.empty_cache()
print(cuda.memory_summary()) 
</code></pre>
<p>The output:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7d469e0a55e46c76f2a8b7b19b44374c4e58521.png" data-download-href="https://discuss.pytorch.org/uploads/default/d7d469e0a55e46c76f2a8b7b19b44374c4e58521" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7d469e0a55e46c76f2a8b7b19b44374c4e58521.png" alt="image" data-base62-sha1="uNjKvA7smMSmtHUGZRdM8A08Ux3" width="451" height="500" data-dominant-color="323132"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">627×695 16.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I don’t know what mechanism the non-releasable memory is. If it’s a bug, please fix it as soon as possible. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/pleading_face.png?v=12" title=":pleading_face:" class="emoji" alt=":pleading_face:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/pleading_face.png?v=12" title=":pleading_face:" class="emoji" alt=":pleading_face:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/pleading_face.png?v=12" title=":pleading_face:" class="emoji" alt=":pleading_face:" loading="lazy" width="20" height="20"></p>
<p>Another weird phenomenon！ <img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=12" title=":expressionless:" class="emoji" alt=":expressionless:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=12" title=":expressionless:" class="emoji" alt=":expressionless:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=12" title=":expressionless:" class="emoji" alt=":expressionless:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=12" title=":expressionless:" class="emoji" alt=":expressionless:" loading="lazy" width="20" height="20"><br>
I found that as long as I multiply a constant immediately after the first convolution, this kind of memory leak will occur. If you multiply the constant before feed into the ‘conv_transpose2d’ or not multiply the constant, the memory leak will disappear. The codes and results for the three cases are as follows:</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.nn import functional as F
from torch import cuda  

def test(inp): 
  w = torch.rand([32, 1, 1, 1],device='cuda') 
  a = F.conv2d(inp, w)*5
  y = F.conv_transpose2d(a, w) 
  return y

imgs = torch.zeros([128,1,512,512],device='cuda')
outp = test(imgs)  
 
cuda.empty_cache()
print(cuda.memory_summary()) 
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/0/000ca4bc26ed4e03f53459a86e344d88f3a3883a.png" data-download-href="https://discuss.pytorch.org/uploads/default/000ca4bc26ed4e03f53459a86e344d88f3a3883a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/0/000ca4bc26ed4e03f53459a86e344d88f3a3883a.png" alt="image" data-base62-sha1="r5urz0r06QdVAuX5uyPc4QIYy" width="453" height="500" data-dominant-color="323132"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">620×684 16.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">
import torch
from torch import nn
from torch.nn import functional as F
from torch import cuda  

def test(inp): 
  w = torch.rand([32, 1, 1, 1],device='cuda') 
  a = F.conv2d(inp, w)
  y = F.conv_transpose2d(a*5, w) 
  return y

imgs = torch.zeros([128,1,512,512],device='cuda')
outp = test(imgs)  
 
cuda.empty_cache()
print(cuda.memory_summary()) 
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/9/c97ac307ebef5034db352f296f8dbe2d4c1d3210.png" data-download-href="https://discuss.pytorch.org/uploads/default/c97ac307ebef5034db352f296f8dbe2d4c1d3210" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/9/c97ac307ebef5034db352f296f8dbe2d4c1d3210.png" alt="image" data-base62-sha1="sKmZei1xEjW6goNew5C32eMeTGU" width="449" height="500" data-dominant-color="323132"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">618×688 17 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.nn import functional as F
from torch import cuda  

def test(inp): 
  w = torch.rand([32, 1, 1, 1],device='cuda') 
  a = F.conv2d(inp, w)
  y = F.conv_transpose2d(a, w) 
  return y

imgs = torch.zeros([128,1,512,512],device='cuda')
outp = test(imgs)  
 
cuda.empty_cache()
print(cuda.memory_summary()) 
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38adee8cfc75175d2ec9c30418e78243a2da000c.png" data-download-href="https://discuss.pytorch.org/uploads/default/38adee8cfc75175d2ec9c30418e78243a2da000c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38adee8cfc75175d2ec9c30418e78243a2da000c.png" alt="image" data-base62-sha1="85pnUd9CXz8GngetZPp22pSWSpS" width="454" height="500" data-dominant-color="323132"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">628×691 16.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Please help me!! <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=12" title=":cry:" class="emoji" alt=":cry:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=12" title=":cry:" class="emoji" alt=":cry:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=12" title=":cry:" class="emoji" alt=":cry:" loading="lazy" width="20" height="20"></p>

username: eqy
cooked: <p>I don’t know if this is a bug because in your example <code>outp</code> is still in scope and should not be deleted.<br>
However, it is interesting that the native implementation of softmax causes the computation to tie up more memory.</p>

username: Qizhou
cooked: <p><code>del outp</code> was commented out in the code. I just did not post the result when <code>outp</code> was deleted.</p>

username: Qizhou
cooked: <p>I don’t know why <code>outp</code> takes up so much non-releasable memory when softmax is used, or even just multiply a constant, as I illustrated above.</p>

username: tom
cooked: <p>This isn’t a memory leak as the memory is still available within PyTorch.<br>
Roughly speaking, PyTorch allocates CUDA memory in chunks and then puts tensors in it. It can only “return” (with empty caches) chunks that are completely unused (i.e. don’t have any tensors in them).<br>
What happens here is that with the additional intermediate the data of <code>outp</code> sits inside the 4GB allocation that is mostly free and available to PyTorch but cannot be returned.<br>
You can get a glimpse of this by calling: <a href="https://pytorch.org/docs/master/generated/torch.cuda.memory_snapshot.html#torch.cuda.memory_snapshot" rel="noopener nofollow ugc">torch.cuda.memory_snapshot</a> and it also links to a brief note on memory management (e.g. how to avoid caching allocations for debugging):<br>
<code>print(torch.cuda.memory_snapshot())</code></p>
<p>Best regards</p>
<p>Thomas</p>

username: Qizhou
cooked: <p>Thank you very much for your answer! But I still don’t quite understand. After the function returns, why are the intermediate variables inside the function not released completely? After all, I only need the tensor returned by the function, and I don’t need anything else. And this doesn’t always happen. For example, the penultimate and the antepenultimate examples I gave, just because the timing of multiplying the constant is different, the function takes 4GB more memory after returning.</p>

username: eqy
cooked: <p>I think the memory summary is confusing here. Are you seeing 4GB more used, or 4GB that is used and not available for allocations? A litmus test here would be to keep allocating tensors after the function returns and see how many succeed before an OOM error is thrown.</p>

username: Qizhou
cooked: <p>It is true that the non-releasable memory can be reallocated to the new tensor, but when the new tensor is large, the non-releasable memory can not be combined with the released memory to obtain a whole block of memory to hold the new tensor. When the available video memory is 14GB (colab), the following two examples show this situation:</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.nn import functional as F
from torch import cuda  

def test(inp): 
  w = torch.rand([32, 1, 1, 1],device='cuda') 
  a = F.conv2d(inp, w)*5 # 
  y = F.conv_transpose2d(a, w) # 
  return y
 
outp = test(torch.zeros([128,1,512,512],device='cuda'))  
cuda.empty_cache()
print(cuda.memory_summary())  

a = torch.zeros(11,256,1024,1024,device='cuda')  
print(cuda.memory_summary())  
</code></pre>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.nn import functional as F
from torch import cuda  

def test(inp): 
  w = torch.rand([32, 1, 1, 1],device='cuda') 
  a = F.conv2d(inp, w) #Only modified here
  y = F.conv_transpose2d(a*5, w) #Only modified here
  return y
 
outp = test(torch.zeros([128,1,512,512],device='cuda'))  
cuda.empty_cache()
print(cuda.memory_summary())  

a = torch.zeros(11,256,1024,1024,device='cuda')  
print(cuda.memory_summary())  
</code></pre>
<p>How to solve this problem?</p>

username: Qizhou
cooked: <p>You are right, the non-releasable memory indeed can be reused by pytorch, but it is not shared with the released memory. When the new defined tensor is large, the non-releasable memory can not be applied with released memory simultaneously.</p>

username: tom
cooked: <p>So memory fragmentation (which is likely the more precise than memory leak to describe the situation) is a thing with the caching allocator. If you wanted, you could set <code>PYTORCH_NO_CUDA_MEMORY_CACHING=1</code> to get around this at the expense of doing all allocations/deallocations through cuda.</p>