Spliting my model on 4 GPU and cuda out of memory problem

joeyIsWrong: <p>In order to deal with high resolution 2K images and progressive training of GAN and limited GPU memory(12GB), I am forced to split my model on 4 GPU. However, there seems to be the problem of CUDA OUT OF MEMORY error each time.</p>
<p>And the strange thing is that it happens after one iteration(which means forward and backward pass are both performed). I have checked my code and there definitely will not be any case that my code behaves in the following way:</p>
<p>total loss = 0</p>
<p>for (iteration ):<br>
loss = …<br>
total loss = loss + total loss</p>
<p>Except the above case, I really cannot think of a possibility how there will be an CUDA out of memory bug after one iteration (Also, I do not accumulate gradient as well)</p>
<p>Can anyone help?</p>
<p>My memory usage after the first iteration</p>
<p>memory.used [MiB], memory.free [MiB]<br>
4229 MiB, 7978 MiB<br>
3686 MiB, 8521 MiB<br>
1104 MiB, 11103 MiB<br>
5754 MiB, 6453 MiB</p>
<p>Best Regards</p>

joeyIsWrong: <p>My error message:<br>
Traceback (most recent call last):<br>
File “train.py”, line 215, in <br>
loss_G1.backward()<br>
File “/scratch_net/biwidl207/ligua/anaconda/lib/python2.7/site-packages/torch/tensor.py”, line 93, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/scratch_net/biwidl207/ligua/anaconda/lib/python2.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 90, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: CUDA error: out of memory</p>

colesbury: <p>If the OOM happens on the second iteration then it may be because the variables inside the loop are still alive during the second run. Python has function scoping (not block or loop scoping) so any variables declared during the first iteration remain alive in subsequent iterations until they are redefined.</p>
<p>Here is another post about this:</p><aside class="quote quote-modified" data-post="1" data-topic="28462">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/klory/48/5814_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/increase-the-cuda-memory-twice-then-stop-increasing/28462">Increase the CUDA memory twice then stop increasing</a> 
  </div>
  <blockquote>
    I have the code below and I don’t understand why the memory increase twice then stops 
I searched the forum and can not find answer 
env: PyTorch 0.4.1, Ubuntu16.04, Python 2.7, CUDA 8.0/9.0 
from torchvision.models import vgg16
import torch
import pdb

net = vgg16().cuda()
data1 = torch.rand(16,3,224,224).cuda()

for i in range(10):
    pdb.set_trace()
    out1 = net(data1)


first stop, this is what data1 and vgg16  take 
[1]

second stop, this is what the intermediate status of vgg16 take 
[2…
  </blockquote>
</aside>

<p>See my comment about how you can rewrite your program to avoid two versions alive at once.</p>

joeyIsWrong: <p>Ok, now I have a more detailed analysis of what happened.</p>
<p>First iteration</p>
<p>At the start of iteration:<br>
memory.used [MiB], memory.free [MiB]<br>
533 MiB, 11674 MiB<br>
1147 MiB, 11060 MiB<br>
533 MiB, 11674 MiB<br>
1147 MiB, 11060 MiB</p>
<p>After forward pass:<br>
memory.used [MiB], memory.free [MiB]<br>
2903 MiB, 9304 MiB<br>
2232 MiB, 9975 MiB<br>
2899 MiB, 9308 MiB<br>
2233 MiB, 9974 MiB</p>
<p>At the end of iteration, after forward and backward pass:<br>
memory.used [MiB], memory.free [MiB]<br>
1104 MiB, 11103 MiB<br>
4231 MiB, 7976 MiB<br>
5754 MiB, 6453 MiB<br>
3718 MiB, 8489 MiB<br>
memeda!</p>
<p>Second iteration:</p>
<p>memory.used [MiB], memory.free [MiB]<br>
1104 MiB, 11103 MiB<br>
4231 MiB, 7976 MiB<br>
5754 MiB, 6453 MiB<br>
3718 MiB, 8489 MiB<br>
memory.used [MiB], memory.free [MiB]<br>
3289 MiB, 8918 MiB<br>
4440 MiB, 7767 MiB<br>
5754 MiB, 6453 MiB<br>
4344 MiB, 7863 MiB</p>
<p>memory.used [MiB], memory.free [MiB]<br>
4720 MiB, 7487 MiB<br>
5591 MiB, 6616 MiB<br>
5888 MiB, 6319 MiB<br>
5054 MiB, 7153 MiB<br>
memeda!</p>
<p>Third iteration:</p>
<p>memory.used [MiB], memory.free [MiB]<br>
4720 MiB, 7487 MiB<br>
5591 MiB, 6616 MiB<br>
5888 MiB, 6319 MiB<br>
5054 MiB, 7153 MiB<br>
memory.used [MiB], memory.free [MiB]<br>
4720 MiB, 7487 MiB<br>
5591 MiB, 6616 MiB<br>
5888 MiB, 6319 MiB<br>
5054 MiB, 7153 MiB</p>
<p>Traceback (most recent call last):<br>
File “train.py”, line 218, in <br>
loss_G1.backward()<br>
File “/scratch_net/biwidl207/ligua/anaconda/lib/python2.7/site-packages/torch/tensor.py”, line 93, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/scratch_net/biwidl207/ligua/anaconda/lib/python2.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 90, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: CUDA error: out of memory</p>
<p>This is weird since this time it run through 3 iterations instead of 2.</p>
<p>The number of iterations the program can run through is very different each time. Sometimes 2, sometimes 3, sometimes it run through without CUDA OOM error.</p>
<p>I am wondering what the cause may be.</p>
<p>Theory one: By default, the cuda() should send the result to the first GPU(default). Is it possible it behaves otherwise?</p>
<p>I am just confused where should I look since the program is huge and there are way too many codes to look it</p>
<p>Best Regards</p>

polmonroig: <p>I am having the same issue, some times it runs more iterations and some times less if I modify the number of sub-processes (when loading the dataset) it seams it also affects the number of iterations it can perform even though it does not make sense since they are loaded using my CPU and the training is still sequential. Well in my case when I say iterations I mean to say forward passes or batch passes.</p>

Kevinkevin189: <p>Hi!<br>
I countered the same problem.And I can’t find a proper way to show my memory usage after every train epoch iteration.<br>
Could u give me your code sample about how to print the mem usage after every iter?</p>