OOM issue : how to manage GPU memory?

hw-kim-ivan: <p>Hello:) I’m qurious about how Pytorch handles GPU allocation with reserved, free, allocated memory.</p>
<p>As trying to train Seq2Seq image generation model with single rtx 3070(8gb), there is OOM issue when the mini batch is over 2.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/4/049c185dfc0361de87447eb922a2ff4a1cc3bec8.png" data-download-href="https://discuss.pytorch.org/uploads/default/049c185dfc0361de87447eb922a2ff4a1cc3bec8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/049c185dfc0361de87447eb922a2ff4a1cc3bec8_2_690x230.png" alt="image" data-base62-sha1="EMldGQ1XFJ55bdAFYBJdw8i8sM" width="690" height="230" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/049c185dfc0361de87447eb922a2ff4a1cc3bec8_2_690x230.png, https://discuss.pytorch.org/uploads/default/original/3X/0/4/049c185dfc0361de87447eb922a2ff4a1cc3bec8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/4/049c185dfc0361de87447eb922a2ff4a1cc3bec8.png 2x" data-dominant-color="A0A0A0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">781×261 45.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The picture shows allocated, free, reserved memory are not linearly associated with the batch size, and even(not in pic) the size of model(num of hidden layer) and the size of input image(225x225 or 160x160).</p>
<p>Is there any clear interpretation between (reserved,free,allocated) GPU memory and the model(size of the model, batch size…)?</p>
<p>I think just reducing mini batch size is not the most smart answer for OOM issue.</p>

ptrblck: <p>The error messages might point to different allocation failures, since increasing the batch size might also change the point of failure.<br>
E.g. <code>batch_size=4</code> might fail in the backward pass when the gradient of layer17 is calculated, which would need 38MiB, while <code>batch_size=32</code> might fail in the forward pass when layer3 is trying to allocate memory of 296MiB for the forward activation.</p>
<p>Reducing the batch size or model is one possible way to avoid the OOM, using <code>torch.utils.checkpoint</code> could be used to trade compute for memory.</p>

hw-kim-ivan: <p>Thanks for your advice for using torch.utils.checkpoint.<br>
Let me see what i can do with it and share the progress soon:)</p>

hw-kim-ivan: <p>It’s a been while:) I handled the OOM issue by using gradient accumulation with small batch! Thanks again!</p>

david_littlefield: <p>Thank you, <a class="mention" href="/u/hw-kim-ivan">@hw-kim-ivan</a> and <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p>This <a href="https://stackoverflow.com/questions/66472201/gradient-accumulation-with-custom-model-fit-in-tf-keras" rel="noopener nofollow ugc">post</a> has a nice explanation and code example. The code is written in Keras/TF though.</p>