CUDA out of memory after 12 steps

username: bruce
cooked: <p>±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 440.36       Driver Version: 440.36       CUDA Version: 10.2     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|===============================+======================+======================|<br>
|   0  GeForce GTX 108…  On   | 00000000:83:00.0 Off |                  N/A |<br>
| 21%   34C    P8    10W / 250W |      1MiB / 11178MiB |      0%      Default |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                       GPU Memory |<br>
|  GPU       PID   Type   Process name                             Usage      |<br>
|=============================================================================|<br>
|  No running processes found                                                 |<br>
±----------------------------------------------------------------------------+<br>
0<br>
gpu03<br>
/home/cwzhou/.conda/envs/pytorch/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:118: UserWarning: Detected call of <code>lr_scheduler.step()</code> before <code>optimizer.step()</code>. In PyTorch 1.1.0 and later, you should call them in the opposite order: <code>optimizer.step()</code> before <code>lr_scheduler.step()</code>.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at <a href="https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate" rel="nofollow noopener">https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate</a><br>
warnings.warn("Detected call of <code>lr_scheduler.step()</code> before <code>optimizer.step()</code>. "<br>
/home/cwzhou/.conda/envs/pytorch/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:143: UserWarning: The epoch parameter in <code>scheduler.step()</code> was not necessary and is being deprecated where possible. Please use <code>scheduler.step()</code> to step the scheduler. During the deprecation, if epoch is different from None, the closed form is used instead of the new chainable form, where available. Please open an issue if you are unable to replicate your use case: <a href="https://github.com/pytorch/pytorch/issues/new/choose" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/new/choose</a>.<br>
warnings.warn(EPOCH_DEPRECATION_WARNING, UserWarning)<br>
/home/cwzhou/.conda/envs/pytorch/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:395: UserWarning: To get the last learning rate computed by the scheduler, please use <code>get_last_lr()</code>.<br>
warnings.warn("To get the last learning rate computed by the scheduler, "<br>
/home/cwzhou/.conda/envs/pytorch/lib/python3.8/site-packages/torch/nn/functional.py:1558: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.<br>
warnings.warn(“nn.functional.tanh is deprecated. Use torch.tanh instead.”)<br>
[epoch 0 step 0 loss 2588.6765]<br>
[epoch 0 step 1 loss 2246.1255]<br>
[epoch 0 step 2 loss 1857.5001]<br>
[epoch 0 step 3 loss 1608.0941]<br>
[epoch 0 step 4 loss 1188.3115]<br>
[epoch 0 step 5 loss 1005.5057]<br>
[epoch 0 step 6 loss 950.6890]<br>
[epoch 0 step 7 loss 861.3840]<br>
[epoch 0 step 8 loss 711.6000]<br>
[epoch 0 step 9 loss 639.1685]<br>
[epoch 0 step 10 loss 689.5997]<br>
[epoch 0 step 11 loss 668.2221]<br>
[epoch 0 step 12 loss 672.7594]<br>
Traceback (most recent call last):<br>
File “main.py”, line 146, in <br>
y_val = model(x_test)<br>
File “/home/cwzhou/.conda/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/cwzhou/20200729_H2B_tocaiwei/PReNet.py”, line 96, in forward<br>
x = F.relu(self.res_conv4(x)+resx)<br>
RuntimeError: CUDA out of memory. Tried to allocate 114.00 MiB (GPU 0; 10.92 GiB total capacity; 10.33 GiB already allocated; 59.06 MiB free; 10.34 GiB reserved in total by PyTorch)</p>
<p>what should I do?I was confused</p>

username: ptrblck
cooked: <p>A common issue is storing the whole computation graph in each iteration.<br>
Make sure you are not storing any tensors in e.g. a <code>list</code> or any other container, which might be still attached to the computation graph, as this will increase the memory usage in each iteration.<br>
E.g. if you want to store the loss, use <code>losses.append(loss.item())</code> instead of directly appending the <code>loss</code>.</p>

username: bruce
cooked: <p>you are right.it seems that I didn’t  Release memory. how should I do to release them?<br>
here is my training code<br>
‘’</p>
<pre><code>for i in range(args.epochs):
    scheduler.step(i)

    for j,(data,label) in enumerate(train_dl,0):
        model.train()
        model.zero_grad()
        optimizer.zero_grad()
        if args.use_gpu:
            data = data.cuda()
            label = label.cuda()
            
        pred,_ = model(data)
        loss =1e3*criterion(pred,label)
        
        loss.backward()
        optimizer.step()
#        optimizer.zero_grad()
        
        # training curve plot
        print("[epoch %d step %d loss %.4f]"%(i,j,loss.item()))
        if j%10==0:
            writer.add_scalar('loss', loss.item(),j)
    model.eval()
    x_val,_ = model(x_test)
    val_loss = criterion(x_val ,y_test)
    writer.add_scalar('val_loss', val_loss,i)
    if i%50==0:
        # if y_test.dim() == 4 and y_test.size(1)==1:
        #     y_test = torch.cat((y_test,y_test,y_test),dim=1)
        #     x_val = torch.cat((x_val,x_val,x_val),dim=1)
        #     x_test = torch.cat((x_test,x_test,x_test),dim=1)
    
        clean_grid = torchvision.utils.make_grid(y_test,nrow=16)
        writer.add_image('clean image',clean_grid,dataformats='CHW')
        dirty_grid = torchvision.utils.make_grid(x_test,nrow=16)
        writer.add_image('dirty image',dirty_grid,dataformats='CHW')
        debackground_grid = torchvision.utils.make_grid(x_val,nrow=16)
        writer.add_image('debackground image',debackground_grid,dataformats='CHW')
    print("[epoch %d val_loss %.4f]"%(i,val_loss))
        
    if i%10==0:
        path = os.path.join(os.getcwd(),'model','deback_epoch%d.pth'%(i+1))
        torch.save(model.state_dict(),path)
</code></pre>
<p>‘’’</p>

username: bruce
cooked: <p>after I reduce the depth of the dl model,here is the report:<br>
[epoch 0 step 0 loss 2560.6475]<br>
[epoch 0 step 1 loss 2552.1030]<br>
[epoch 0 step 2 loss 2326.0481]<br>
[epoch 0 step 3 loss 2074.4417]<br>
[epoch 0 step 4 loss 1908.8021]<br>
[epoch 0 step 5 loss 1693.8888]<br>
[epoch 0 step 6 loss 1524.7013]<br>
[epoch 0 step 7 loss 1367.1302]<br>
[epoch 0 step 8 loss 1221.5389]<br>
[epoch 0 step 9 loss 1098.3346]<br>
[epoch 0 step 10 loss 927.1423]<br>
[epoch 0 step 11 loss 826.2545]<br>
[epoch 0 step 12 loss 796.7938]<br>
[epoch 0 val_loss 650.3557]<br>
[epoch 1 step 0 loss 632.1362]<br>
[epoch 1 step 1 loss 594.0154]<br>
[epoch 1 step 2 loss 552.5127]<br>
[epoch 1 step 3 loss 544.2441]<br>
[epoch 1 step 4 loss 562.6210]<br>
[epoch 1 step 5 loss 585.8839]<br>
[epoch 1 step 6 loss 566.8837]<br>
[epoch 1 step 9 loss 519.0481]<br>
[epoch 1 step 10 loss 503.7430]<br>
[epoch 1 step 11 loss 480.8072]<br>
[epoch 1 step 12 loss 470.9222]<br>
[epoch 1 val_loss 945.2809]<br>
[epoch 2 step 0 loss 450.9913]<br>
[epoch 2 step 1 loss 405.8258]<br>
[epoch 2 step 2 loss 399.1376]<br>
[epoch 2 step 3 loss 404.3622]<br>
[epoch 2 step 4 loss 408.4886]<br>
[epoch 2 step 5 loss 387.0332]<br>
[epoch 2 step 6 loss 388.2880]<br>
[epoch 2 step 7 loss 349.8983]<br>
[epoch 2 step 8 loss 364.2084]<br>
[epoch 2 step 9 loss 388.2591]<br>
[epoch 2 step 10 loss 343.5857]<br>
[epoch 2 step 11 loss 356.0178]<br>
[epoch 2 step 12 loss 359.7800]<br>
Traceback (most recent call last):<br>
File “main.py”, line 158, in <br>
y_val = model(x_test)<br>
File “/home/cwzhou/.conda/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/cwzhou/20200729_H2B_tocaiwei/networks.py”, line 210, in forward<br>
x = F.relu(self.res_conv3(x) + resx)<br>
File “/home/cwzhou/.conda/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/cwzhou/.conda/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/container.py”, line 100, in forward<br>
input = module(input)<br>
File “/home/cwzhou/.conda/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/cwzhou/.conda/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/activation.py”, line 94, in forward<br>
return F.relu(input, inplace=self.inplace)<br>
File “/home/cwzhou/.conda/envs/pytorch/lib/python3.8/site-packages/torch/nn/functional.py”, line 1063, in relu<br>
result = torch.relu(input)<br>
RuntimeError: CUDA out of memory. Tried to allocate 114.00 MiB (GPU 0; 10.92 GiB total capacity; 10.37 GiB already allocated; 3.06 MiB free; 10.39 GiB reserved in total by PyTorch)</p>

username: ptrblck
cooked: <p>Your code snippet doesn’t show any part which might store the computation graph.<br>
Maybe this line of code could be problematic (I haven’t tested it):</p>
<pre><code class="lang-python">writer.add_scalar('val_loss', val_loss,i)
</code></pre>
<p>Could you use <code>writer.add_scalar('val_loss', val_loss.item(), i)</code> and rerun the code?<br>
Also, did you verify that the GPU memory is increasing in each iteration via <code>nvidia-smi</code>?</p>

username: bruce
cooked: <p>thanks for your patience.<br>
I add with torch.no_grad() before model eval and all the problems are solved.<br>
I am very thank you and your colleague for your contributions to the pytorch community</p>

username: ptrblck
cooked: <p>Good to hear it’s solved now and for the sake of completeness: your approach of wrapping the validation loop into the <code>no_grad()</code> block is the better approach anyway <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: minato
cooked: <aside class="quote no-group" data-username="bruce" data-post="3" data-topic="96457">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/ccd318/48.png" class="avatar"> bruce:</div>
<blockquote>
<pre><code class="lang-auto"> clean_grid = torchvision.utils.make_grid(y_test,nrow=16)
        writer.add_image('clean image',clean_grid,dataformats='CHW')
</code></pre>
</blockquote>
</aside>
<p>Today, I used <code>torchvision.utils.make_grid</code> to generate images and <code>writer.add_image</code> to incorporate them into TensorBoard. I faced a similar issue after 27 steps.</p>
<p>The problem was resolved by including <code>with torch.no_grad</code>. Thank you.</p>
<p>Adding <code>with torch.no_grad</code> helped fix it. Thanks.</p>