GPU memory usage is growing after training

username: MikeHsu
cooked: <p>He guys, I am using U-net and RNN.<br>
I found my GPU memory is increasing after each step instead of remaining stable.<br>
After about 40 steps, run out of memory error occurs.<br>
Seems there is something stock in GPU memory and cause leak.<br>
I am using RTX 2080 ti.<br>
My code is here: <a href="https://github.com/vagr8/R_Unet/blob/master/R_Unet.py" rel="nofollow noopener">https://github.com/vagr8/R_Unet/blob/master/R_Unet.py</a><br>
Thanks!</p>

username: albanD
cooked: <p>This most likely happen because you either store things in a list that ever grows at each iteration or if you hold onto the computational graph of the whole history.</p>
<p>You should be able to check the first one by printing list size in your code (in particular, buffer)</p>
<p>For the second one, you want to make sure that if you store things for which you only want the value, and not gradients to be backpropagated, use <code>.detach()</code>.<br>
You can you use <a href="https://github.com/szagoruyko/pytorchviz">torchviz</a> to print the computational graph associated with your loss to make sure it does not grow at every iteration. Otherwise you need to identify where it links to the previous operations, and use <code>.detach()</code> to break it at this point.</p>

username: MikeHsu
cooked: <p>Thank you!<br>
To test lists/buffer problem, I disable my RNN unit. After that, memory usage looks right.<br>
Now I know where the problem is.</p>

username: MikeHsu
cooked: <p>I wrote a  (1) simple LSTM class and the memory usage is unusual.<br>
However using (2) LSTM directly without writing a LSTM class to use will not cause problem.</p>
<p>What makes the different from using sequential class for RNN and directly use RNN?</p>
<p>Here is the code for (1) and (2):<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/vagr8/R_Unet/blob/master/rnn_test.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/vagr8/R_Unet/blob/master/rnn_test.py" target="_blank" rel="nofollow noopener">vagr8/R_Unet/blob/master/rnn_test.py</a></h4>
<pre><code class="lang-py">import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
import os
from torch.autograd import Variable
import torch.autograd as autograd
import psutil
import gc

cuda_gpu = torch.cuda.is_available()

class LSTMM(nn.Sequential):
    def __init__(self):
        super(LSTMM, self).__init__()
        self.lstm = nn.LSTM(10, 30, batch_first=True)
        self.h0 = Variable(torch.randn(1, 50, 30))
        self.c0 = Variable(torch.randn(1, 50 ,30))
    
</code></pre>

  This file has been truncated. <a href="https://github.com/vagr8/R_Unet/blob/master/rnn_test.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Edit: I found the way I provide code is hard to read, so I put a Github link instead.</p>

username: MikeHsu
cooked: <p>After periodically detach hidden layers in rnn, problem is solved.<br>
Thank you!</p>