Out of memory error during evaluation but training works fine!

wasiahmad: <p>I have recently upgraded pytorch from 0.2 to 0.3. Surprisingly my old programs are throwing an out of memory error during evaluation (in eval() mode) but training works just fine. I am using the same batch size for training and evaluation. I am totally clueless what is happening? Did anyone face similar issue? Is there any possible solution?</p>

ptrblck: <p>Sounds strange.<br>
Did you use the <code>volatile=True</code> param on your Variables?<br>
Is the batch size larger during eval than train?<br>
Do you use cuDNN in both cases?</p>
<p>Could you post a code snippet reproducing the issue?</p>

wasiahmad: <p>I tried using volatile=True param on the variables and it didn’t help. I am using the same batch size. I am not doing anything special to use cuDNN. I am using the default setting.</p>
<pre><code>def validate(self, dev_corpus):
    # Turn on evaluation mode which disables dropout.
    self.model.eval()

    dev_batches = helper.batchify(dev_corpus.data, self.config.batch_size)
    print('number of dev batches = ', len(dev_batches))

    dev_loss = 0
    num_batches = len(dev_batches)
    for batch_no in range(1, num_batches + 1):
        session_queries, session_query_length, rel_docs, rel_docs_length, doc_labels = helper.session_to_tensor(
            dev_batches[batch_no - 1], self.dictionary)
        if self.config.cuda:
            session_queries = session_queries.cuda()
            session_query_length = session_query_length.cuda()
            rel_docs = rel_docs.cuda()
            rel_docs_length = rel_docs_length.cuda()
            doc_labels = doc_labels.cuda()

        loss = self.model(session_queries, session_query_length, rel_docs, rel_docs_length, doc_labels)
        if loss.size(0) &gt; 1:
            loss = loss.mean()
        dev_loss += loss.data[0]

    return dev_loss / num_batches
</code></pre>
<p>I am using the above function for evaluation. Here, session_queries, session_query_length, … rest variables are created by enabling volatile=True.</p>
<p>I am not sure what is hapenning!!</p>

qoqo: <p>Hi. Did you do well your problem? Now, I meet the same problem as well. How do you do in this situation?</p>

ptrblck: <p>The <code>volatile</code> flag is deprecated. In the latest stable release (<code>0.4.0</code>) you should use a context manager:</p>
<pre><code>with torch.no_grad():
    # Your eval code
</code></pre>
<p>Have a look at the <a href="https://pytorch.org/">website</a> for install instructions.<br>
You can find the migration guide <a href="https://pytorch.org/2018/04/22/0_4_0-migration-guide.html">here</a>.</p>

qoqo: <p>hi. ptrblck. thank you for your help. it works now.</p>

DuaneNielsen: <p>Hit the same problem, same solution worked, pytorch 4.0.1.  Seems like there might be something weird going on with the eval mode memory management.</p>

InnovArul: <p>A relevant clear-cut answer on ‘model.eval()’ vs ‘with torch.no_grad()’ from <a class="mention" href="/u/alband">@albanD</a>:</p>
<aside class="quote" data-post="2" data-topic="19615">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/model-eval-vs-with-torch-no-grad/19615/2">'model.eval()' vs 'with torch.no_grad()'</a> 
  </div>
  <blockquote>
    Hi, 
These two have different goals: 


model.eval() will notify all your layers that you are in eval mode, that way, batchnorm or dropout layers will work in eval mode instead of training mode.

torch.no_grad() impacts the autograd engine and deactivate it. It will reduce memory usage and speed up computations but you won’t be able to backprop (which you don’t want in an eval script).
  </blockquote>
</aside>


DuaneNielsen: <p>Thanks Arul, That’s helpful…</p>
<p>Still doesn’t explain why eval mode appears to use more memory than training mode though.  Theoretically it would just use the same.</p>

two_four: <p>I am a novice. Have the same problem but during the inference, I have never met ‘out of memory’ error without using the <code>torch.no_grad()</code> or <code>volatile=True</code> before. But at this time it seems not to work without using <code>torch.no_grad()</code>. pytorch 3.0.0.</p>

jia_lee: <p>+1, why use more memory than training?</p>

ptrblck: <p>You might run out of memory if you still hold references to some tensors from your training iteration.<br>
Since Python uses function scoping, these variables are still kept alive, which might result in your OOM issue. To avoid this, you could wrap your training and validation code in separate functions. Have a look at <a href="https://discuss.pytorch.org/t/spliting-my-model-on-4-gpu-and-cuda-out-of-memory-problem/37311/3?u=ptrblck">this post</a> for more information.</p>

DuaneNielsen: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , did you mean this post?</p>
<aside class="quote quote-modified" data-post="1" data-topic="28462">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/klory/48/5814_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/increase-the-cuda-memory-twice-then-stop-increasing/28462">Increase the CUDA memory twice then stop increasing</a> 
  </div>
  <blockquote>
    I have the code below and I don’t understand why the memory increase twice then stops 
I searched the forum and can not find answer 
env: PyTorch 0.4.1, Ubuntu16.04, Python 2.7, CUDA 8.0/9.0 
from torchvision.models import vgg16
import torch
import pdb

net = vgg16().cuda()
data1 = torch.rand(16,3,224,224).cuda()

for i in range(10):
    pdb.set_trace()
    out1 = net(data1)


first stop, this is what data1 and vgg16  take 
[1]

second stop, this is what the intermediate status of vgg16 take 
[2…
  </blockquote>
</aside>


ptrblck: <p>Yes, <a class="mention" href="/u/colesbury">@colesbury</a> explains, why the memory usage might grow if some tensors weren’t deleted using function scoping.</p>

Flo: <p>What is the effect if you forget torch.no_grad() besides the increased memory. Will you accumulate gradients in the validation block?</p>

ptrblck: <p>The computation graph will be created and intermediate tensors are stored.<br>
If you don’t call <code>backward</code> (which wouldn’t even be possible in a <code>torch.no_grad()</code> block), nothing else will change.</p>

Flo: <p>Well you would call backward in the training portion. So would you then update the net with the grads tracked in the validation portion as well as those in the training portion? Assuming that torch.no_grad() was forgotten in validation.</p>

ptrblck: <p>During training a new computation graph would usually be created, as long as you don’t pass e.g. the output of your validation phase as the new input to the model during training.</p>
<pre><code class="lang-python">model = models.resnet18(pretrained=True)

# Pseudo validation phase
x1 = torch.randn(1, 3, 224, 224)
out = model(x1)

# Pseudo training phase
x1 = torch.ones(1, 3, 224, 224)
out = model(x1)
out.mean().backward()
</code></pre>
<p>In this code snippet you have “forgotten” to use <code>torch.no_grad()</code> during the validation phase.<br>
However, since <code>out</code> is not used, it won’t have any effect on the gradients, but will just use unnecessary memory.</p>

Flo: <p>Ok cool, what about if it’s set up this way.</p>
<pre><code class="lang-auto">crit = nn.SomeLoss()
optim = optim.SGD()
net = models.resnet18()

for e in range(num_epochs):
    
    # training
    pred = net(some_data)
    optim.zero_grad()
    loss = crit(pred, target)
    loss.backward() 
    optim.step()

    # validation
    valid_pred = net(some_validation_data)
    loss = crit(valid_pred, valid_target)
</code></pre>
<p>Would zero_grad take care of that?</p>

ptrblck: <p>As long as you don’t calculate gradients via a <code>backward</code> call, no gradients will be accumulated.</p>