Questions about GPU memory usage

username: rkd1137
cooked: <p>I’m trying to profile a model’s memory usage right now using this tutorial: <a href="https://pytorch.org/blog/understanding-gpu-memory-1/" class="inline-onebox" rel="noopener nofollow ugc">Understanding GPU Memory 1: Visualizing All Allocations over Time | PyTorch</a>. Looking at the output, almost all of the memory usage is listed as Unknown (screenshot attached). When I step through the code watching <code>nvidia-smi</code>, it looks like the biggest increase in memory comes during the forward pass of the model. Does anyone have suggestions on how to debug this further? I can post my code, but my model/dataset are spread over several files. Aside from the model/dataset import, I exactly follow the code in Appendix B of the link above. Is there a more methodical way to find out which parts of my model are leading to the largest costs in memory?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8522efa8a2216c23307e7533068cef78cd5332e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b8522efa8a2216c23307e7533068cef78cd5332e" title="Screenshot 2025-02-04 at 4.40.52 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8522efa8a2216c23307e7533068cef78cd5332e_2_690x409.jpeg" alt="Screenshot 2025-02-04 at 4.40.52 PM" data-base62-sha1="qizVMj1CC7NCwICbZbK1NaUfgPc" width="690" height="409" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8522efa8a2216c23307e7533068cef78cd5332e_2_690x409.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8522efa8a2216c23307e7533068cef78cd5332e_2_1035x613.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8522efa8a2216c23307e7533068cef78cd5332e_2_1380x818.jpeg 2x" data-dominant-color="DDDDDD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2025-02-04 at 4.40.52 PM</span><span class="informations">2508×1490 147 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Similarly, here’s the screenshot showing memory by tensor. But as I don’t see an obvious way to associate that to specific tensors in the model, I’m not sure how to use this to debug further (I’m using four random batches with different sequence lengths).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc8b3abf60ecd8814b81c87210d4f89d95c582d0.png" data-download-href="https://discuss.pytorch.org/uploads/default/bc8b3abf60ecd8814b81c87210d4f89d95c582d0" title="Screenshot 2025-02-04 at 5.06.44 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc8b3abf60ecd8814b81c87210d4f89d95c582d0_2_690x260.png" alt="Screenshot 2025-02-04 at 5.06.44 PM" data-base62-sha1="qTW3ReqXdJA3z2DkalcOddIJL8I" width="690" height="260" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc8b3abf60ecd8814b81c87210d4f89d95c582d0_2_690x260.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc8b3abf60ecd8814b81c87210d4f89d95c582d0_2_1035x390.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc8b3abf60ecd8814b81c87210d4f89d95c582d0_2_1380x520.png 2x" data-dominant-color="E8E2DD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2025-02-04 at 5.06.44 PM</span><span class="informations">2826×1068 178 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/rkd1137">@rkd1137</a>  there is no easy way to do this. Can share a few general pointers</p>
<p>There is a good article <a href="https://pytorch.org/docs/main/torch_cuda_memory.html" class="inline-onebox" rel="noopener nofollow ugc">Understanding CUDA Memory Usage — PyTorch main documentation</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aa50e015fadfdd174af55dcc9cf33aed572ba636.png" data-download-href="https://discuss.pytorch.org/uploads/default/aa50e015fadfdd174af55dcc9cf33aed572ba636" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aa50e015fadfdd174af55dcc9cf33aed572ba636_2_690x168.png" alt="image" data-base62-sha1="oiGryYDPssn3ylJU6wXoTorZZ78" width="690" height="168" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aa50e015fadfdd174af55dcc9cf33aed572ba636_2_690x168.png, https://discuss.pytorch.org/uploads/default/original/3X/a/a/aa50e015fadfdd174af55dcc9cf33aed572ba636.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/a/aa50e015fadfdd174af55dcc9cf33aed572ba636.png 2x" data-dominant-color="C8BBA9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">874×213 55.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ol>
<li>
<p>Check if there are certain allocations that are continuous across the iterations. These are probably hogging memory without being part of the training cycle</p>
</li>
<li>
<p>If the stack shows *_backward they are part of backward propagation<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/3/e3b99e11328167e1ad4d342fdb9054e978d85446.png" data-download-href="https://discuss.pytorch.org/uploads/default/e3b99e11328167e1ad4d342fdb9054e978d85446" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e3b99e11328167e1ad4d342fdb9054e978d85446_2_690x69.png" alt="image" data-base62-sha1="wuy4iyGeiXXPhdWVuyYvtX6ipb8" width="690" height="69" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e3b99e11328167e1ad4d342fdb9054e978d85446_2_690x69.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e3b99e11328167e1ad4d342fdb9054e978d85446_2_1035x103.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/3/e3b99e11328167e1ad4d342fdb9054e978d85446.png 2x" data-dominant-color="DDDDDD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1156×117 59.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
<li>
<p>If the stack shows just a plain operation like convolution and increase upto a point they are part of the forward flow for each epoch<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e236822c22dc4f9ba45c1d89eb19f1497ee55a1.png" data-download-href="https://discuss.pytorch.org/uploads/default/5e236822c22dc4f9ba45c1d89eb19f1497ee55a1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e236822c22dc4f9ba45c1d89eb19f1497ee55a1_2_690x57.png" alt="image" data-base62-sha1="dqMJW132RFQ2Qy4XatVult8alqN" width="690" height="57" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e236822c22dc4f9ba45c1d89eb19f1497ee55a1_2_690x57.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e236822c22dc4f9ba45c1d89eb19f1497ee55a1_2_1035x85.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e236822c22dc4f9ba45c1d89eb19f1497ee55a1_2_1380x114.png 2x" data-dominant-color="E1E1E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1841×154 97.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
</ol>

username: python
cooked: <p>Yeah, the “Unknown” memory category can make profiling frustrating. One thing that might help is breaking down the forward pass into smaller chunks and using <code>torch.cuda.memory_allocated()</code> or <code>torch.cuda.memory_snapshot()</code> at different points to narrow down where the spikes are happening. Also, wrapping parts of your forward method with <code>torch.autograd.profiler.profile(use_cuda=True)</code> might give you better granularity. If your model is spread over multiple files, maybe try isolating smaller parts and running them with dummy inputs to see how much memory each one uses. That might help tie the large tensors back to specific layers or ops.</p>