CUDA error: out of memory

Sunil_Tapashetti: <p>Hi there,</p>
<p>I am encountering CUDA error: out of memory while implementing a simple CNN.<br>
The error apparently is pointing to loss,backward(). I also used .detach() wherever I am saving metrics of accuracy and loss to a list. Am I missing something?</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-14-e531653ce2a2&gt; in &lt;module&gt;
     20         outputs = net(items)
     21         loss = criterion(outputs,labels)
---&gt; 22         loss.backward()
     23         optimizer.step()
     24 

~\Anaconda3\envs\pytorch\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)
     91                 products. Defaults to ``False``.
     92         """
---&gt; 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)
     94 
     95     def register_hook(self, hook):

~\Anaconda3\envs\pytorch\lib\site-packages\torch\autograd\__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     88     Variable._execution_engine.run_backward(
     89         tensors, grad_tensors, retain_graph, create_graph,
---&gt; 90         allow_unreachable=True)  # allow_unreachable flag
     91 
     92 

RuntimeError: CUDA error: out of memory

</code></pre>
<p><strong>My code is as follows:</strong></p>
<pre><code class="lang-auto">num_epochs = 50
train_loss = []
valid_loss = []
train_accuracy = []
valid_accuracy = []


for epoch in range(num_epochs):
    iter_loss = 0
    correct = 0
    # Train the network
    net.train()
    for i,(items,labels) in enumerate(train_loader):
        items = Variable(items)
        labels = Variable(labels)
        if cuda.is_available():
            items = items.cuda()
            labels = labels.cuda()
        optimizer.zero_grad()    
        outputs = net(items)
        loss = criterion(outputs,labels)
        loss.backward()
        optimizer.step()
        
        # Book keeping for iterations
        
        iter_loss += loss
        _,predictions = torch.max(outputs,1)
        correct_pred = (predictions==labels).sum()
        correct += correct_pred
        
    # Book keeping for Epochs
    
    train_loss.append(iter_loss.detach())
    train_accuracy.append((100*correct / len(MNIST_train)).detach())
        
    # Validate 
    net.eval()
    iter_loss = 0
    correct = 0
    for i,(items,labels) in enumerate(valid_loader): 
        items = Variable(items)
        labels = Variable(labels)
        if cuda.is_available():
            items = items.cuda()
            labels = labels.cuda()
        outputs = net(items)
        loss = criterion(outputs,labels)
        
        # Book keeping for iterations
        
        iter_loss += loss
        _,predictions = torch.max(outputs,1)
        correct_pred = (predictions==labels).sum()
        correct += correct_pred
        
    # Book keeping for Epochs
    
    valid_loss.append(iter_loss.detach())
    valid_accuracy.append((100*correct / len(MNIST_valid)).detach())
    
    # Print perfomance metrics
    
    print('epoch: {}/{} ,train_accuracy={},valid_accuracy={}'.format(epoch+1,num_epochs,train_accuracy[-1],valid_accuracy[-1]))
          

</code></pre>

ptrblck: <p>It seems you are storing the computation graph in this line: <code>iter_loss += loss</code>. Use <code>iter_loss += loss.item()</code>.<br>
Also add <code>with torch.no_grad():</code> before the validation loop, as this will save some memory by avoiding storing variables necessary to calculate gradients.</p>

Sunil_Tapashetti: <p>Does’t net.eval() do the same?</p>

ptrblck: <p>Not really. <code>model.eval()</code> changes the behavior of some layers. For example <code>nn.BatchNorm</code> layers will use their running stats (in the default mode) and <code>nn.Dropout</code> will be deactivated.<br>
If you don’t want to calculate gradients, which is the common case during evaluation, you should wrap the evaluation code into <code>with torch.no_grad()</code>.</p>

Sunil_Tapashetti: <p>Thanks. That was very helpful.</p>

Sunil_Tapashetti: <p>I find that I get cuda error inconsistently. Some times there is no error while the same code throws up error in another session. Any way to flush gpu cache or any other possible solution??</p>

ptrblck: <p>What kind of error do you get? Is your GPU running out of memory?</p>
<p>Did you check, if other applications or maybe dead kernels are using the GPU?<br>
If so, you could try to kill them before running your script.<br>
If you can’t detect any other applications, did you notice a certain pattern, when the OOM occurs?</p>

Sunil_Tapashetti: <p>I will check that. Thanks for the prompt response.</p>

saba: <p>Hello Ptrblck</p>
<p>My CNN network is shallow and batch size is 64 , I run my code one time but for the next time it give me CUDA out of memory ?<br>
Do you have any suggestion to check?</p>

ptrblck: <p>Do you see an increase in memory usage during training?<br>
If so, you might accidentally store the computation graph, e.g. by storing <code>loss</code> in a Python list.<br>
If you see the OOM error in the second epoch / iteration, you could try to wrap your training procedure into a function, since Python uses function scoping as described <a href="https://discuss.pytorch.org/t/spliting-my-model-on-4-gpu-and-cuda-out-of-memory-problem/37311/3">here</a>.</p>
<p>If neither of these two suggestions helps, could you post your code so that we could have a look?</p>

saba: <p>Hi Ptrblck</p>
<p>I really appreciate your answer. My problem is now that I am out of quote and I just install anaconda. Dose bracewell has anaconda? Using this command can give me more space?</p>
<p><code>torch.cuda.empty_cache()? </code></p>

ptrblck: <p>By “out of quote”, do you mean out of memory?<br>
I’m not sure, what bracewell is.</p>
<p>Emptying the cache should give you just the cached memory back, which should not avoid OOM issues and might make your code run slower.</p>

saba: <p>Thanks a lot. I request more storage from our service help.</p>
<p>I appreciate your help</p>
<p>Cheers</p>
<p>Saba</p>

saba: <p>Hi</p>
<p>Would you please introduce me some practical books regarding pytorch and Deep learning. I saw some tutorials but need to know more.</p>
<p>Cheers</p>
<p>Saba</p>