OOM error while caching

khaquan_sh: <p>Hi, my team and I were working with a ViT for a project but ran into a OOM issue. We have a couple of variants of ViT:</p>
<ol>
<li>Normal ViT</li>
<li>ViT with some form of caching</li>
</ol>
<p>The OOM error only occurs for ViT with caching but only on model sizes larger than 200M parameters (works fine for ViT of upto 76M parameters, i.e. it works fine for base and small, but not for large).</p>
<p>Right now we have tried the following to prevent memory leakages:</p>
<ol>
<li>Tried to avert it with detach (as can be seen)</li>
<li>Checking if there was a broadcasting making the tensors bigger than they actually are (we printed out the cache in the forward pass and it had the same shape throughout, so it’s not looking like that’s the issue)</li>
<li>Also tried to instantiate the cache using self.register_buffer(…) in the hopes that it would automatically fix any accumulated gradient issues but that didn’t work in our last run either - we’re not crystal clear on how these work in the context of the computational graph and tracking of gradients so this was a hail mary for us</li>
</ol>
<p>I have attached part of the code where we work with the cache. The update_splits_cache_training is called after a window_size number of iterations have passed. Assume that the self.splits_func is set to get_ratios_diff.</p>
<p>We want to know whether our handling of cache has implications on back-prop which we have not figured out (for eg. using while loops, converting tensors to lists and tuples etc) which might be contributing to the OOM.</p>
<p>An interesting note is that after a window_size number of iterations, we also cleared the cache (torch.cuda.empty_cache) which actually allowed the training to go on much longer but eventually another OOM error showed up.</p>
<p>Any help would be really really appreciated! Thanks.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74852c055917f33729c1a88c588ac1ad971e0dd1.png" data-download-href="https://discuss.pytorch.org/uploads/default/74852c055917f33729c1a88c588ac1ad971e0dd1" title="Screenshot 2024-07-19 at 3.03.26 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74852c055917f33729c1a88c588ac1ad971e0dd1_2_690x468.png" alt="Screenshot 2024-07-19 at 3.03.26 PM" data-base62-sha1="gCMHMwQZHqHW0kEIjeCngS3wI93" width="690" height="468" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74852c055917f33729c1a88c588ac1ad971e0dd1_2_690x468.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74852c055917f33729c1a88c588ac1ad971e0dd1_2_1035x702.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74852c055917f33729c1a88c588ac1ad971e0dd1_2_1380x936.png 2x" data-dominant-color="F5F6F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-07-19 at 3.03.26 PM</span><span class="informations">1608×1092 133 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Detaching the <code>curr_cache</code> is a good idea, but since this does not seem to fix the issue, you could check if other tensors (which are accumulated or stored in any container) have a valid <code>.grad_fn</code>. If so, you could also detach these (assuming they are not needed in the <code>backward</code> pass).<br>
If this does not help, you might need to add debug <code>print</code> statements into your training code trying to isolate which part of the code increases the memory usage unexpectedly.</p>