Understanding CUDA out of memory error

username: alamin
cooked: <p>Hello, I have the following code snippet which gives me ‘cuda out of memory error’ after several passes of the for loop using batch size (b) 50 or above. However if I use batch size less than 40, it seems run fine. My question is in case of larger batch size (&gt;=50), why it is giving error after several passes? Is there any memory overhead accumulating inside the loop?</p>
<pre><code class="lang-auto">model = Video3DCNN('resnet34',10, sample_size=224)
model.to('cuda:0')
optimizer1 = optim.Adam(model.parameters(), lr=0.0005)
loss = nn.CrossEntropyLoss()
b = 50
ep_loss = 0.
for i in range(100):
  torch.cuda.empty_cache()
  logits = model(torch.randn(b, 3, 10, 224, 224).cuda())
  label = torch.randint(high=1000, size=(b,)).cuda()
  loss_ = loss(logits, label)
  optimizer1.zero_grad()
  print (i, loss_.item())
  print (logits.size())
  loss_.backward()
  optimizer1.step()
  ep_loss += loss_.detach().item()
</code></pre>
<p>Output for batch size b==50,<br>
/home/ahosain/workspace/cslr/resnet_3d.py:152: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal_.<br>
m.weight = nn.init.kaiming_normal(m.weight, mode=‘fan_out’)<br>
0 6.941838264465332<br>
torch.Size([50, 1000])<br>
1 7.274449348449707<br>
torch.Size([50, 1000])<br>
2 7.0289177894592285<br>
torch.Size([50, 1000])<br>
3 7.262955188751221<br>
torch.Size([50, 1000])<br>
Traceback (most recent call last):<br>
File “test.py”, line 38, in <br>
loss_.backward()<br>
File “/home/ahosain/workspace/dl_env/lib/python3.6/site-packages/torch/tensor.py”, line 166, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/ahosain/workspace/dl_env/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 99, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: CUDA out of memory. Tried to allocate 1.50 GiB (GPU 0; 10.76 GiB total capacity; 5.23 GiB already allocated; 1.48 GiB free; 3.19 GiB cached)</p>

username: alex.veuthey
cooked: <p>Could you plot the GPU memory usage along your script running time? There’s a torch.cuda function for that <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.max_memory_allocated" rel="nofollow noopener">here</a>. (In particular, it would be good to also reset the value at each iteration, as stated in the doc).</p>
<p>It’s weird though that it crashes at step 3-4 in your batch-size 50 run and not at all for your batch-size 40 run…</p>

username: alamin
cooked: <p>Hi, Thanks for the reply and the function reference. I used it and found something confusing. I have two RTX 2080 Ti gpus. The every third number shows the cache size at that point using the function <code>torch.cuda.max_memory_cached ( )</code></p>
<p>When I use gpu 0, get the following output,</p>
<pre><code class="lang-auto">0 6.981476306915283
torch.Size([50, 1000])
10034.0
1 6.936072826385498
torch.Size([50, 1000])
10034.0
2 7.297829627990723
torch.Size([50, 1000])
10034.0
3 7.23666524887085
torch.Size([50, 1000])
Traceback (most recent call last):
  File "test.py", line 44, in &lt;module&gt;
    loss_.backward()
  File "/home/ahosain/workspace/dl_env/lib/python3.6/site-packages/torch/tensor.py", line 166, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/ahosain/workspace/dl_env/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA out of memory. Tried to allocate 1.50 GiB (GPU 0; 10.76 GiB total capacity; 5.23 GiB already allocated; 1.48 GiB free; 3.19 GiB cached)
</code></pre>
<p>In case of gpu 1,  it seems running fine with following output,</p>
<pre><code class="lang-auto">0 7.1220622062683105
torch.Size([50, 1000])
8964.0
1 7.066858291625977
torch.Size([50, 1000])
9512.0
2 7.112499237060547
torch.Size([50, 1000])
9512.0
3 7.206641674041748
torch.Size([50, 1000])
9536.0
4 7.3534345626831055
torch.Size([50, 1000])
9536.0
5 7.341899871826172
torch.Size([50, 1000])
9536.0
6 7.346314907073975
torch.Size([50, 1000])
9536.0
7 7.240469932556152
torch.Size([50, 1000])
9536.0
.
.
.
.
</code></pre>

username: alex.veuthey
cooked: <p><img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"> that’s odd, you don’t even use DataParallel in your code sample, and you empty the cache at each iteration…</p>
<p>I’ve seen something in your original code, maybe instead of calling <code>.cuda()</code> on your logits and label tensors, try calling <code>.to('cuda:0')</code>, the same as your model. I have no idea if that would change anything, since it would probably not work if the tensors were on a different GPU than the model, but it’s worth trying.</p>