Increase the CUDA memory twice then stop increasing

username: klory
cooked: <p>I have the code below and I don’t understand why the memory increase twice then stops</p>
<p>I searched the forum and can not find answer</p>
<p>env: PyTorch 0.4.1, Ubuntu16.04, Python 2.7, CUDA 8.0/9.0</p>
<pre><code class="lang-auto">from torchvision.models import vgg16
import torch
import pdb

net = vgg16().cuda()
data1 = torch.rand(16,3,224,224).cuda()

for i in range(10):
    pdb.set_trace()
    out1 = net(data1)
</code></pre>
<ol>
<li>first stop, this is what <code>data1</code> and <code>vgg16</code>  take<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/1/108affe140760d91999223b1d3a10d11f99c62b5.png" alt="1" data-base62-sha1="2mlrzebGRMZHoKAqIxe9Hkm5pKR" width="672" height="21"></li>
<li>second stop, this is what the intermediate status of <code>vgg16</code> take<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/9/9e86fc9c54d8d520225269d1743fa63c4bd4d8aa.png" alt="2" data-base62-sha1="mCoEbpQGKeTLv0rorZDGfowzbjY" width="673" height="19"></li>
<li>third stop, <strong>WHY it increase again?</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/7/7f683c6b8141f0beb5224b4990cb4e290892fa3e.png" alt="3" data-base62-sha1="ib5XJerkiRcHRXZ5tN3qOQYClhY" width="668" height="17"></li>
<li>forth stop, <strong>WHY it stops increasing?</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/7/7f683c6b8141f0beb5224b4990cb4e290892fa3e.png" alt="3" data-base62-sha1="ib5XJerkiRcHRXZ5tN3qOQYClhY" width="668" height="17"></li>
</ol>

username: colesbury
cooked: <p>The memory is from the output <code>out1</code> and intermediate activations needed to compute the gradient. The first increase is from computing <code>out1</code>. The second increase is from computing <code>net(data1)</code> while <code>out1</code> is still alive. The reason is that in:</p>
<pre><code class="lang-auto">out1 = net(data1)
</code></pre>
<p>The right-hand side <code>net(data1)</code> is evaluated before the assignment. Memory usage, as reported by the system, doesn’t generally decrease. If it had, then it would decrease back to <code>2872Mi</code> after the assignment operation.</p>
<p>You can rewrite your program to avoid keeping two versions of <code>out1</code> alive at once:</p>
<pre><code class="lang-auto">def eval(network, input):
  out1 = network(input)
  # maybe use out1 here

for i in range(10):
  eval(net, data1)
</code></pre>
<p>As long as you don’t return <code>out1</code> from <code>eval</code>, <code>out1</code> will be freed before the next call, so you’ll only use <code>2872Mi</code>.</p>

username: klory
cooked: <aside class="quote no-group" data-username="colesbury" data-post="2" data-topic="28462">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/colesbury/48/26_2.png" class="avatar"> colesbury:</div>
<blockquote>
<p><code>ta1)</code> is evaluated before the assignment</p>
</blockquote>
</aside>
<p>But why it did not increase for the third time?</p>
<p>Is there some optimization in the compiler of PyTorch, e.g., doing the first Backprop and the second Forward at the same time? this is the only reason I could think of why there are the two copies</p>

username: colesbury
cooked: <p>The old value in <code>out1</code> gets deleted since there are no longer any references to it. It holds onto all the internal state needed to compute the gradient, so when it gets deleted that internal state gets deleted too.</p>

username: farazk86
cooked: <p>Hi,</p>
<p>I changed my code as suggested but my GPU memory still doubles during evaluation. Wrapping my evaluation code in <code>with torch.no_grad()</code> helped the code run as I am no longer getting CUDA out of memory errors but my GPU memory still doubles</p>
<p>Before reading your post my code was structured like:</p>
<pre><code class="lang-auto">for i in range(50):
    for idx, (inputs, targets) in enumerate(loaders):
        inputs, targets = inputs.to(device), targets.to(device)
        optimizer.zero_grad()

        outputs = model(inputs)

        loss = loss_criterion(outputs, targets)

        predicted_output = torch.argmax(outputs.detach(), 0).squeeze(0)


        error_ = error(predicted_output.data.cpu(), targets.data.cpu().long().squeeze())

        running_loss.update(loss.item(), inputs.size(0))
        running_error.update(error_.item(), inputs.size(0))
</code></pre>
<p>But after your suggestion I changed it to:</p>
<pre><code class="lang-auto">def forward_pass(model_, input_, target_, loss_):
    outputs = model_(input_)

    loss = loss_(outputs, target_)

    predicted_output = torch.argmax(outputs.detach(), 0).squeeze(0)

    return loss, predicted_output

for i in range(50):
    for idx, (inputs, targets) in enumerate(loaders):
        inputs, targets = inputs.to(device), targets.to(device)
        optimizer.zero_grad()

        loss, predicted_output = forward_pass(model, inputs, targets, loss_criterion)

        error_ = error(predicted_output.data.cpu(), targets.data.cpu().long().squeeze())

        running_loss.update(loss.item(), inputs.size(0))
        running_error.update(error_.item(), inputs.size(0))
</code></pre>
<p>this made no difference on memory usage.</p>
<p>Thanks</p>

username: colesbury
cooked: <aside class="quote no-group" data-username="farazk86" data-post="5" data-topic="28462">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/farazk86/48/7071_2.png" class="avatar"> farazk86:</div>
<blockquote>
<p>I changed my code as suggested but my GPU memory still doubles during evaluation.</p>
</blockquote>
</aside>
<p>That’s because you still assign <code>inputs</code>, <code>targets</code>, <code>loss</code>, etc. in the for loop and not in the function.</p>

username: farazk86
cooked: <p>Sorry if I misunderstood. Based on what you mentioned above:</p>
<aside class="quote no-group" data-username="colesbury" data-post="2" data-topic="28462">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/colesbury/48/26_2.png" class="avatar"> colesbury:</div>
<blockquote>
<p>You can rewrite your program to avoid keeping two versions of <code>out1</code> alive at once:</p>
<pre><code class="lang-auto">def eval(network, input):
  out1 = network(input)
  # maybe use out1 here

for i in range(10):
  eval(net, data1)
</code></pre>
<p>As long as you don’t return <code>out1</code> from <code>eval</code> , <code>out1</code> will be freed before the next call, so you’ll only use <code>2872Mi</code> .</p>
</blockquote>
</aside>
<p>I took the prediction of output out of the for loop and as suggested used the output in that function and calculated loss and accuracy there and only returned the loss and predicted tensor. <code>output</code> was not returned from the function.</p>
<p>I would eventually have to assign the input in a for loop as that is how I would get data from mt DataLoader in order to pass to the evaluation function. I dont see how I would get this to work without returning loss for <code>loss.backward()</code> and without using a for loop fro the data loading. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

username: colesbury
cooked: <p>That’s because you’re still returning “loss” which unless you have <code>torch.no_grad()</code> holds onto a lot of data so that you can call <code>loss.backward()</code> in the future.</p>
<p>Put everything in a function (including the GPU copies of input and targets) or use <code>del</code>.</p>
<pre><code class="lang-auto">...

def step(input, targets):
    inputs, targets = inputs.to(device), targets.to(device)
    optimizer.zero_grad()

    loss, predicted_output = forward_pass(model, inputs, targets, loss_criterion)

    error_ = error(predicted_output.data.cpu(), targets.data.cpu().long().squeeze())

    running_loss.update(loss.item(), inputs.size(0))
    running_error.update(error_.item(), inputs.size(0))

for i in range(50):
    for idx, (inputs, targets) in enumerate(loaders):
        step(input, targets)
</code></pre>

username: klory
cooked: <p>Thank you colesbury, that helps.</p>