Loss.backward() cause CPU memory leak

username: wizkid
cooked: <p>hello, thank you for pytorch</p>
<p>I am studying beginner tutorials<br>
when I run cifar10_tutorial.py in Deep Learning with PyTorch: A 60 minutes Blitz, I find memory leak in loss.backward()</p>
<p>To run using gpu and train a larger network, I revised the tutorial code like below</p>
<p>origianal code--------------------------------------------------------------------------------------</p>
<blockquote>
<p>from torch.autograd import Variable<br>
import torch.nn as nn<br>
import torch.nn.functional as F</p>
</blockquote>
<blockquote>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(3, 6, 5)<br>
self.pool  = nn.MaxPool2d(2, 2)<br>
self.conv2 = nn.Conv2d(6, 16, 5)<br>
self.fc1   = nn.Linear(16 * 5 * 5, 120)<br>
self.fc2   = nn.Linear(120, 84)<br>
self.fc3   = nn.Linear(84, 10)</p>
</blockquote>
<blockquote>
<pre><code>def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.view(-1, 16 * 5 * 5)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x
</code></pre>
</blockquote>
<blockquote>
<p>net = Net()<br>
import torch.optim as optim</p>
</blockquote>
<blockquote>
<p>criterion = nn.CrossEntropyLoss()<br>
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)</p>
</blockquote>
<blockquote>
<p>for epoch in range(2):  # loop over the dataset multiple times</p>
</blockquote>
<blockquote>
<pre><code>running_loss = 0.0
for i, data in enumerate(trainloader, 0):
    # get the inputs
    inputs, labels = data
</code></pre>
</blockquote>
<blockquote>
<pre><code>    # wrap them in Variable
    inputs, labels = Variable(inputs), Variable(labels)
</code></pre>
</blockquote>
<blockquote>
<pre><code>    # zero the parameter gradients
    optimizer.zero_grad()
</code></pre>
</blockquote>
<blockquote>
<pre><code>    # forward + backward + optimize
    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()
</code></pre>
</blockquote>
<blockquote>
<pre><code>    # print statistics
    running_loss += loss.data[0]
    if i % 2000 == 1999:    # print every 2000 mini-batches
        print('[%d, %5d] loss: %.3f' % (epoch+1, i+1, running_loss / 2000))
        running_loss = 0.0
</code></pre>
</blockquote>
<p>revised code---------------------------------------------------------------------------------------------</p>
<blockquote>
<p>from torch.autograd import Variable<br>
import torch.nn as nn<br>
import torch.nn.functional as F</p>
</blockquote>
<blockquote>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(3,128, 5)<br>
self.pool  = nn.MaxPool2d(2, 2)<br>
self.conv2 = nn.Conv2d(128, 128, 5)<br>
self.fc1   = nn.Linear(128 * 5 * 5, 120)<br>
self.fc2   = nn.Linear(120, 84)<br>
self.fc3   = nn.Linear(84, 10)</p>
</blockquote>
<blockquote>
<pre><code>def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.view(-1, 128 * 5 * 5)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x
</code></pre>
</blockquote>
<blockquote>
<p>net = Net().cuda()</p>
</blockquote>
<blockquote>
<p>import torch.optim as optim</p>
</blockquote>
<blockquote>
<p>criterion = nn.CrossEntropyLoss().cuda()<br>
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)</p>
</blockquote>
<blockquote>
<p>for epoch in range(2):  # loop over the dataset multiple times</p>
</blockquote>
<blockquote>
<pre><code>running_loss = 0.0
for i, data in enumerate(trainloader, 0):
    # get the inputs
    inputs, labels = data
</code></pre>
</blockquote>
<blockquote>
<pre><code>    # wrap them in Variable
    inputs, labels = Variable(inputs.cuda()), Variable(labels.cuda())
</code></pre>
</blockquote>
<blockquote>
<pre><code>    # zero the parameter gradients
    optimizer.zero_grad()
</code></pre>
</blockquote>
<blockquote>
<pre><code>    # forward + backward + optimize
    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()
</code></pre>
</blockquote>
<blockquote>
<pre><code>    # print statistics
    running_loss += loss.data[0]
    if i % 2000 == 1999:    # print every 2000 mini-batches
        print('[%d, %5d] loss: %.3f' % (epoch+1, i+1, running_loss / 2000))
        running_loss = 0.0
</code></pre>
</blockquote>
<hr>
<p>when I run the above code, I found memory leak(abount 70MB) from one epoch and the next.<br>
If I delete loss.backward, memory leak doesn’t occur<br>
Also, If I add torch.backend.cudnn.enalbled = False and don’t delete loss.backward, memory leak doesn’t occur, but speed is slow</p>
<p>Install information</p>
<ul>
<li>install method : pip install <a href="http://download.pytorch.org/whl/cu80/torch-0.1.11.post5-cp27-none-linux_x86_64.whl" rel="noopener nofollow ugc">http://download.pytorch.org/whl/cu80/torch-0.1.11.post5-cp27-none-linux_x86_64.whl</a>.</li>
<li>cudnn : 5.1</li>
<li>python version : 2.7</li>
<li>os : linux min 18.1</li>
<li>cuda : 8.0</li>
</ul>

username: bearbear
cooked: <p>안녕하세요. 3년전 게시글이지만 혹시 문제 해결하셨는지 궁금합니다…!</p>

username: ptrblck
cooked: <p>Translated:</p>
<blockquote>
<p>Good morning. It was posted three years ago, but I wonder if you have solved the problem… !</p>
</blockquote>
<p><a class="mention" href="/u/bearbear">@bearbear</a><br>
Could you try to translate your posts before posting, please, as this would make it easier to continue the discussion? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>