Memory Error : FPN+FasterRCNN in parallel

username: Sentient07
cooked: <p>Hi,</p>
<p>I am running into memory error while trying to re-train an FPN+FasterRCNN network in a similar fashion as shown here: <a href="https://github.com/pytorch/vision/blob/master/references/detection/train.py" rel="noopener nofollow ugc">https://github.com/pytorch/vision/blob/master/references/detection/train.py</a>. My GPU cards are 3 x Nvidia 2080 Tis, Torchvision version: 0.4.0.</p>
<p>To get the above code working for my dataset and machine configuration(2GPU), I referred to these comments : [1] <a href="https://github.com/pytorch/pytorch/issues/25627#issuecomment-527992106" class="inline-onebox" rel="noopener nofollow ugc">FasterRCNN and MaskRCNN doesn't work with DataParallel or DistributedDataParallel · Issue #25627 · pytorch/pytorch · GitHub</a></p>
<p>My training code successfully completes the first epoch. But fails with memory error in the second epoch. Complete Traceback:</p>
<blockquote>
<p>(base) rsundara@igrida-abacus6:~/Code/fpnfrcnn_det$ CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 --use_env train.py --world-size 2 --batch_size 2 --lr 1e-4</p>
<hr>
<p>Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.</p>
<hr>
<p>| distributed init (rank 1): env://<br>
| distributed init (rank 0): env://<br>
ARGS GPU is 0<br>
Multi GPU training<br>
Epoch: [1]  [   0/5944]  eta: 23:42:17  lr: 0.000100  loss: 1.9594 (1.9594)  loss_box_reg: 0.0522 (0.0522)  loss_classifier: 0.7208 (0.7208)  loss_objectness: 0.6930 (0.6930)  loss_rpn_box_reg: 0.4934 (0.4934)  time: 14.3570  data: 0.6149  max mem: 9659<br>
Traceback (most recent call last):<br>
File “train.py”, line 111, in <br>
train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=200)<br>
File “/udd/rsundara/Code/fpnfrcnn_det/vision/engine.py”, line 46, in train_one_epoch<br>
losses.backward()<br>
File “/udd/rsundara/.local/lib/python3.6/site-packages/torch/tensor.py”, line 118, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/udd/rsundara/.local/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 93, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: CUDA out of memory. Tried to allocate 8.51 GiB (GPU 0; 10.73 GiB total capacity; 1.19 GiB already allocated; 8.51 GiB free; 109.16 MiB cached)<br>
Traceback (most recent call last):<br>
File “/soft/igrida/spack/opt/spack/linux-debian8-x86_64/gcc-9.1.0/python-3.6.5-xu2dmz5rdvjfmiizbt65hyy2hsqsn3ri/lib/python3.6/runpy.py”, line 193, in _run_module_as_main<br>
“<strong>main</strong>”, mod_spec)<br>
File “/soft/igrida/spack/opt/spack/linux-debian8-x86_64/gcc-9.1.0/python-3.6.5-xu2dmz5rdvjfmiizbt65hyy2hsqsn3ri/lib/python3.6/runpy.py”, line 85, in _run_code<br>
exec(code, run_globals)<br>
File “/udd/rsundara/.local/lib/python3.6/site-packages/torch/distributed/launch.py”, line 246, in <br>
main()<br>
File “/udd/rsundara/.local/lib/python3.6/site-packages/torch/distributed/launch.py”, line 242, in main<br>
cmd=cmd)<br>
subprocess.CalledProcessError: Command ‘[’/soft/igrida/spack/opt/spack/linux-debian8-x86_64/gcc-9.1.0/python-3.6.5-xu2dmz5rdvjfmiizbt65hyy2hsqsn3ri/bin/python’, ‘-u’, ‘train.py’, ‘–world-size’, ‘2’, ‘–batch_size’, ‘2’, ‘–lr’, ‘1e-4’]’ returned non-zero exit status 1.</p>
</blockquote>
<p>The training however, successfully progresses without using the <code>torch.utils.data.distributed.DistributedSampler</code>, i.e on a single GPU.</p>
<p>What could be going wrong with the MultiGPU training?</p>
<p>Regards,</p>

username: Sentient07
cooked: <p>Here’s the second link I followed : [2] : <a href="https://discuss.pytorch.org/t/size-mismatch-when-running-fasterrcnn-in-parallel/54979/7" class="inline-onebox">Size mismatch when running FasterRCNN in parallel</a>.</p>
<p>Due to the restriction on the link a new user could post, I couldn’t share it in my original post.</p>

username: ptrblck
cooked: <p>Based on the error message it seems you might be close to the memory limit in the first epoch, and creating some additional tensors might push you over the edge.<br>
Did you already wrap the training and validation loops in separate methods?<br>
This could save some memory, as Python uses function scoping and would free all unused tensors after the function scope is left.</p>
<p>Also, I assume you are wrapping the validation loop in a <code>with torch.no_grad()</code> block?</p>
<p>If that’s the case, then you could</p>
<ul>
<li>reduce the batch size</li>
<li>decrease the number of parameters or activations in your model</li>
<li>use <a href="https://pytorch.org/docs/master/amp.html">native mixed-precision</a> with the nightly binaries or master build</li>
<li>trade compute for memory using <code>torch.utils.checkpoint</code>
</li>
</ul>

username: Sentient07
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>. Thank you for your response. I am replying to your questions as inline below,</p>
<aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="82797">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Based on the error message it seems you might be close to the memory limit in the first epoch</p>
</blockquote>
</aside>
<p>Apparently not. I have ~8Gb of memory left in the first iteration on both the GPUs.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="82797">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Did you already wrap the training and validation loops in separate methods?</p>
</blockquote>
</aside>
<p>Yes, I don’t validate the model until the end of training as I’m Finetuning with pre-trained weights.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="82797">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Also, I assume you are wrapping the validation loop in a <code>with torch.no_grad()</code> block?</p>
</blockquote>
</aside>
<p>Yes, it’s same as <code>torchvision.references.detection.engine.evaluate</code> where it’s decorated with <code>@torch.no_grad()</code>.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="82797">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<ul>
<li>reduce the batch size</li>
</ul>
</blockquote>
</aside>
<p>My batchsize is 2. It works quite well on single GPU case but just not for the multi-GPU case.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="82797">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<ul>
<li>trade compute for memory using <code>torch.utils.checkpoint</code></li>
</ul>
</blockquote>
</aside>
<p>Could you please elaborate this?</p>
<p>I am curious why training on 2 GPUs fail while it progresses well on single GPU?</p>
<p>Thanks again for your time and help <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Sentient07" data-post="4" data-topic="82797">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/c67d28/48.png" class="avatar"> Sentient07:</div>
<blockquote>
<p>I am curious why training on 2 GPUs fail while it progresses well on single GPU?</p>
</blockquote>
</aside>
<p>It might be “bad luck” due to a change in memory fragmentation, as I still think you might be close to the limit:</p>
<blockquote>
<p>Tried to allocate <strong>8.51 GiB</strong> (GPU 0; 10.73 GiB total capacity; 1.19 GiB already allocated; <strong>8.51 GiB free</strong>; 109.16 MiB cached)</p>
</blockquote>
<p>so it seems that vert little memory might be missing.</p>
<aside class="quote no-group" data-username="Sentient07" data-post="4" data-topic="82797">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/c67d28/48.png" class="avatar"> Sentient07:</div>
<blockquote>
<p>Could you please elaborate this?</p>
</blockquote>
</aside>
<p>The <a href="https://pytorch.org/docs/stable/checkpoint.html">docs</a> explain the general use case and <a href="https://github.com/prigoyal/pytorch_memonger/blob/master/tutorial/Checkpointing_for_PyTorch_models.ipynb">this notebook</a> gives you an example (the notebook is quite old by now, but should still give you an idea of the usage).</p>

username: Sentient07
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> thank you again for your reply. Is there a way by which I can check if the GPU memory doesn’t get cleared between subsequent iterations?(on the second GPU especially) I suspect this could be happening as the single GPU training works perfectly fine for me, even at batchsize=current_batch*8. It’s just the multi-GPU case which crashes, despite two same cards.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="82797">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>The <a href="https://pytorch.org/docs/stable/checkpoint.html" rel="noopener nofollow ugc">docs</a> explain the general use case and <a href="https://github.com/prigoyal/pytorch_memonger/blob/master/tutorial/Checkpointing_for_PyTorch_models.ipynb" rel="noopener nofollow ugc">this notebook</a> gives you an example (the notebook is quite old by now, but should still give you an idea of the usage).</p>
</blockquote>
</aside>
<p>Sure, I will have a look, thanks!</p>
<p>Regards,</p>

username: ptrblck
cooked: <p>If you are running out of memory on a device, PyTorch will clear the cache and try to reallocate the memory. There is unfortunately not much you can check besides <code>torch.cuda.memory_summary()</code>.</p>
<p>You could try to <code>del</code> unnecessary tensors early, so that you might get potentially more memory once you hit the OOM issue.</p>

username: Sentient07
cooked: <p>Okay, I think I have narrowed down the problem and realise your answer is pertinent <a class="mention" href="/u/ptrblck">@ptrblck</a>. The network takes in images of varying sizes and not a fixed input, causing the memory to fluctuate.<br>
I have managed to train the network on V100 GPU and here are the output from first two(200*2) iterations,</p>
<blockquote>
<p>Epoch: [1]  [   0/7925]  eta: 6:09:17  lr: 0.000500  loss: 2.0259 (2.0259)  loss_classifier: 0.6635 (0.6635)  loss_box_reg: 0.0236 (0.0236)  loss_objectness: 0.6907 (0.6907)  loss_rpn_box_reg: 0.6483 (0.6483)  time: 2.7958  data: 1.1561  max mem: 7464<br>
Epoch: [1]  [ 200/7925]  eta: 1:25:06  lr: 0.000500  loss: 1.0027 (1.3102)  loss_classifier: 0.1189 (0.2118)  loss_box_reg: 0.0081 (0.0198)  loss_objectness: 0.5959 (0.6495)  loss_rpn_box_reg: 0.3003 (0.4291)  time: 0.6534  data: 0.0127  max mem: 10348</p>
</blockquote>
<p>I have opened an <a href="https://github.com/pytorch/vision/issues/2263" rel="noopener nofollow ugc">issue</a> this morning, a proposal to remove <code>GeneralisedTransformation</code> as a compulsory transformation to all FasterRCNN models and I believe fixed size inputs would keep the memory from fluctuating.</p>
<p>Thanks again!</p>

username: ptrblck
cooked: <p>That’s interesting. Would it be possible to somehow sort the images so that you would start with a large one and using smaller inputs in the following iterations?<br>
This could probably reuse the memory without reallocating new memory.<br>
Maybe you’ll be able to lower the memory footprint using it.</p>