Pytorch 1.6.0 seem to leak memory in conv2d

username: Wang-zipeng
cooked: <p>I’m using pytorch 1.6.0,mmdetection 2.2.0, maskrcnn for inference. In Resnet the first conv2d seem to consume about 300M and not release after that. I’m using the “nvidia-smi -lms 100” and windows Task Manager to monitor the usage of GPU, so There is any wrong in monitor the usage of GPU or just pytorch leak GPU memory in conv2d?</p>

username: tom
cooked: <p>There are a few things to keep in mind</p>
<ul>
<li>PyTorch will keep GPU memory around after the tensors have been deallocated (the CUDA memory cache). nvidia-smi would see that as still in use. You can check allocated / cached memory and release cached memory using the functions in torch.cuda.</li>
<li>You would want to wrap things in <code>with torch.no_grad()</code> to instruct PyTorch to not keep the information around for backward.</li>
</ul>
<p>While bugs in PyTorch, including memory leaks, can occur, it would seem that it is unlikely that benign use of Conv2D has it because it would affect very many people.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Wang-zipeng
cooked: <p>Thank you for your answer, I think your answer about nvidia-smi is right.<br>
But I have another question, could you please explain why for me?<br>
We want to deploy more than one pytorch model on single GPU, So we care about peak usage more.I have monitored the usage of GPU by nvidia-smi or torch.cuda.memory_summary(), For resnet_maskrcnn nvdia-smi result is 3095MB, and torch.cuda.max_memory_reserved() result is 1778384896byte(1696MB). For mobilenetV3_maskrcnn nvidia-smi result is 5964MB, and torch.cuda.max_memory_reserved() result 4640997376byte(4426MB), both are bigger than resnet_maskrcnn, but we can deploy three mobilenetV3_maskrcnn on 1050ti, and only two resnet_maskrcnn on 1050ti. Could you explain why or there is somthing wrong in my profile method.<br>
I have summary the result for a table:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/7/27213845d42541b673719b7bb54bcfbc87f51f63.png" alt="image" data-base62-sha1="5A9NfXor70lzaGFWlnICJZzLmzV" width="547" height="56"></p>

username: tom
cooked: <p>You might have allocations outside the PyTorch memory allocator. I think the cuda context <a href="https://github.com/pytorch/pytorch/issues/37250#issuecomment-619786918" rel="nofollow noopener">itself isn’t showing as a PyTorch allocation</a>, if I interpret <a class="mention" href="/u/ptrblck">@ptrblck</a>’s comment correctly.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Wang-zipeng
cooked: <p>I agree with your opinion about cuda context will consume some GPU memory. I think that’s why nvidia-smi result different with torch.cuda.max_memory_reserved() result. But that answer can’t explain why mobilenetV3_maskrcnn use more GPU memory on peak but they can be deploy three model currently into single GPU, but resnet50_maskrcnn could do this two. Or there is something wrong in my understanding?</p>

username: Wang-zipeng
cooked: <p>I did a very simple experiment today, codes are as follows:<br>
with torch.no_grad():<br>
print(torch.cuda.memory_reserved())<br>
print(torch.cuda.memory_allocated())<br>
print(os.system(‘nvidia-smi’))<br>
inputs = torch.randn(1, 3, 512, 512).cuda()<br>
print(os.system(‘nvidia-smi’))<br>
conv = torch.nn.Conv2d(3, 64, (7, 7), stride=2, padding=3, bias=False).cuda()<br>
inputs = conv(inputs)<br>
print(torch.cuda.memory_reserved())<br>
print(torch.cuda.memory_allocated())<br>
torch.cuda.empty_cache()<br>
print(torch.cuda.memory_reserved())<br>
print(torch.cuda.memory_allocated())<br>
outputs are as follows(remove some unused line):<br>
0 ---- print(torch.cuda.memory_reserved())<br>
0 ---- torch.cuda.memory_allocated()</p>
<p>487MiB /  8192MiB ---- print(os.system(‘nvidia-smi’))<br>
before “torch.randn(1, 3, 512, 512).cuda()”, at this point maybe torch is not init and same as cuda, so this 487MiB is all windows used i think.</p>
<p>938MiB /  8192MiB ---- print(os.system(‘nvidia-smi’))<br>
after “torch.randn(1, 3, 512, 512).cuda()”, at this point torch is init and “inputs” only consume little GPU memory, so about 500MB(944-487) is cuda context and so far everything can be explained</p>
<p>23068672 ---- print(torch.cuda.memory_reserved())<br>
17863680 ---- print(torch.cuda.memory_allocated())<br>
after inputs = conv(inputs), only one conv consume about 2G GPU memory, is this normal? and after the conv I think it should return to 938MiB, so I have thought there is memory leak in Pytorch.</p>
<p>23068672 ---- print(torch.cuda.memory_reserved())<br>
17863680 ---- print(torch.cuda.memory_allocated())<br>
after torch.cuda.empty_cache(), There is nothing happen, so I don’t understand why pytorch hold so many GPU memory</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="Wang-zipeng" data-post="6" data-topic="96461">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wang-zipeng/48/28845_2.png" class="avatar"> Wang-zipeng:</div>
<blockquote>
<p>23068672 ---- print(torch.cuda.memory_reserved())<br>
17863680 ---- print(torch.cuda.memory_allocated())<br>
after inputs = conv(inputs), only one conv consume about 2G GPU memory</p>
</blockquote>
</aside>
<p>That’s ~23MB.</p>
<p>Note that there are additional lazy initializations for things like cublas, curand, cudnn.</p>

username: Wang-zipeng
cooked: <p>you are right, my fault <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

username: Wang-zipeng
cooked: <p>I did a experiment in resnet50_maskrcnn, here is the code:<br>
print(torch.cuda.memory_reserved())<br>
print(torch.cuda.memory_allocated())<br>
x = self.extract_feat(img)<br>
print(torch.cuda.memory_reserved())<br>
print(torch.cuda.memory_allocated())</p>
<p>model had call eval(), so the is_training is False<br>
201326592(about 192MB) ---- print(torch.cuda.memory_reserved())<br>
193334272(about 184.377MB) ---- print(torch.cuda.memory_allocated())<br>
before the inference pytorch reserved about 192MB, I can understand that.</p>
<p>x = self.extract_feat(img)<br>
this is resnet backbone, x consume about 87306240(83.262MB)</p>
<p>1304428544(1244MB) ---- print(torch.cuda.memory_reserved())<br>
1267727360(1208.999MB) ---- print(torch.cuda.memory_allocated())<br>
My question is why after the backbone inference pytorch hold so many GPU memory? There is only x in the GPU and it only consume about 83.262MB, what about 1GB(1244 - 83.262) GPU memory used for?</p>

username: googlebot
cooked: <p>probably due to pending garbage collection (python side)</p>

username: Wang-zipeng
cooked: <p>Maybe you are right, but how could you explain why 1050ti can load three mobilenetV3_maskrcnn but only load two resnet50_maskrcnn ?</p>

username: googlebot
cooked: <p>No idea, sorry. Check topics like <a href="https://discuss.pytorch.org/t/how-to-debug-causes-of-gpu-memory-leaks/6741">this</a>. Fragmentation may be an issue, if you’re stress testing.</p>