Memory Footprint of Eager Mode vs Torch Compiled (Default, with CUDA Graph, and CUDA Graph Dynamic)

Abhishek_Ghosh: <p>I was looking into the performance numbers in the <a href="https://hud.pytorch.org/benchmark/compilers" rel="noopener nofollow ugc">PyTorch Dashboard</a> - the peak memory footprint stats caught my attention.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5113c0a93f7d516213929ea4796ba0fae8269ec.png" data-download-href="https://discuss.pytorch.org/uploads/default/f5113c0a93f7d516213929ea4796ba0fae8269ec" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5113c0a93f7d516213929ea4796ba0fae8269ec_2_690x173.png" alt="image" data-base62-sha1="yXXU78QVGaXmq5aWHasJGKhaerG" width="690" height="173" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5113c0a93f7d516213929ea4796ba0fae8269ec_2_690x173.png, https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5113c0a93f7d516213929ea4796ba0fae8269ec.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5113c0a93f7d516213929ea4796ba0fae8269ec.png 2x" data-dominant-color="E2F4E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">941×237 27.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The plots:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c7cb557fa1272d3b53c279cd9f63d2a2e9ac6f6.png" data-download-href="https://discuss.pytorch.org/uploads/default/2c7cb557fa1272d3b53c279cd9f63d2a2e9ac6f6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c7cb557fa1272d3b53c279cd9f63d2a2e9ac6f6_2_690x178.png" alt="image" data-base62-sha1="6lyc6bDAZNUm3BC1quHssyWf1NI" width="690" height="178" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c7cb557fa1272d3b53c279cd9f63d2a2e9ac6f6_2_690x178.png, https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c7cb557fa1272d3b53c279cd9f63d2a2e9ac6f6.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c7cb557fa1272d3b53c279cd9f63d2a2e9ac6f6.png 2x" data-dominant-color="F7F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">953×247 23.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3b940aaf131746d4c494ec69ff014b8ffdeccd07.png" data-download-href="https://discuss.pytorch.org/uploads/default/3b940aaf131746d4c494ec69ff014b8ffdeccd07" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3b940aaf131746d4c494ec69ff014b8ffdeccd07_2_690x185.png" alt="image" data-base62-sha1="8v3lCr1KZQBkCZ2rNgt1eaijcPR" width="690" height="185" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3b940aaf131746d4c494ec69ff014b8ffdeccd07_2_690x185.png, https://discuss.pytorch.org/uploads/default/original/3X/3/b/3b940aaf131746d4c494ec69ff014b8ffdeccd07.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/b/3b940aaf131746d4c494ec69ff014b8ffdeccd07.png 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">948×255 21.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I assume the following:</p>
<ul>
<li><code>default</code> in the above plots, refers to <code>torch.compile(mode="default")</code></li>
<li><code>cudagraphs</code> refers to <code>torch.compile(mode="reduce-overhead")</code></li>
<li><code>cudagraphs_dynamic</code> refers to <code>torch.compile(mode="reduce-overhead", dynamic=True)</code></li>
<li><code>inductor_max_autotune</code> refers to <code>torch.compile(mode="max-autotune")</code></li>
</ul>
<p>In the <a href="https://pytorch-dev-podcast.simplecast.com/episodes/cuda-graph-trees" rel="noopener nofollow ugc">PyTorch CUDA Graph Trees podcast</a>, it is mentioned that CUDA Graphs tend to bloat memory. (It is the general trend I guess, CUDA Graphs are known to bloat memory usage - statically allocated memory blocks, memory pools and others, and the CUDA Graph data structure itself occupies memory)</p>
<p>I don’t know whether I am interpreting the above plots correctly, but it seems that using CUDA Graphs reduces memory usage:</p>
<ul>
<li>From the “Peak dynamo memory usage” plot, it seems that, <code>default</code> takes up the most memory (the yellow curve), followed by <code>cudagraphs</code> (the dark blue curve) and <code>cudagraphs_dynamic</code> taking the least (the light green curve) in case of TorchBench.</li>
</ul>
<p>Am I reading the plots correctly? If so, what is causing this reduction in memory usage? (It seems a bit counterintuitive to me.)</p>
<hr>
<p>Also, I tried to run the <a href="https://github.com/pytorch/pytorch/blob/6b1f13ea2f3b1bcd575620eecd7d84a4d2e3eb76/benchmarks/dynamo/huggingface.py" rel="noopener nofollow ugc">huggingface.py</a> benchmark locally:</p>
<pre><code class="lang-auto">./benchmarks/dynamo/huggingface.py --performance --training --amp --backend=inductor --output=huggingface_training.csv
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b1888073a90fba0f7c1db4ef0015e83efa7c9b48.png" data-download-href="https://discuss.pytorch.org/uploads/default/b1888073a90fba0f7c1db4ef0015e83efa7c9b48" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b1888073a90fba0f7c1db4ef0015e83efa7c9b48_2_690x101.png" alt="image" data-base62-sha1="pkwXT7njDMqgkd0elyhMShP6jzy" width="690" height="101" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b1888073a90fba0f7c1db4ef0015e83efa7c9b48_2_690x101.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b1888073a90fba0f7c1db4ef0015e83efa7c9b48_2_1035x151.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b1888073a90fba0f7c1db4ef0015e83efa7c9b48_2_1380x202.png 2x" data-dominant-color="2B2C2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1858×274 54.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Eager Mode in the above table I guess refers to the execution of the PyTorch Program without using <code>torch.compile</code>.</p>
<p>The above table shows some good compression ratios of the peak memory usage - between Eager Mode and Dynamo. What is the reason for this compression?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6fbb54709a6dbfb92e4c123b576d0ee371bd4997.png" data-download-href="https://discuss.pytorch.org/uploads/default/6fbb54709a6dbfb92e4c123b576d0ee371bd4997" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6fbb54709a6dbfb92e4c123b576d0ee371bd4997_2_690x57.png" alt="image" data-base62-sha1="fWqlNo548Zf5wOli3jf9VtbudmL" width="690" height="57" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6fbb54709a6dbfb92e4c123b576d0ee371bd4997_2_690x57.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6fbb54709a6dbfb92e4c123b576d0ee371bd4997_2_1035x85.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6fbb54709a6dbfb92e4c123b576d0ee371bd4997_2_1380x114.png 2x" data-dominant-color="2E2E2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1866×156 32.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Yet again, there are certain models in the local execution, where the compression ratio is <code>&lt;1</code>. What is the reason for that?</p>

ptrblck: <p>CC <a class="mention" href="/u/marksaroufim">@marksaroufim</a> for the dashboard questions as you might know the corresponding <code>compile</code> modes.</p>

marksaroufim: <p>Yeah I’m also confused by this now cc <a class="mention" href="/u/eellison">@eellison</a></p>

Sarang_Zambare: <p>any update on this? trying to understand if cudagraphs is a whole new backend or just an alias for reduce_overhead</p>

soulitzer: <blockquote>
<p>The above table shows some good compression ratios of the peak memory usage - between Eager Mode and Dynamo. What is the reason for this compression?</p>
</blockquote>
<p>I don’t know about compile w/ cuda graph vs compile w/o cuda graph, but if we’re talking about eager vs compile, there is the min-cut partitioner <a href="https://dev-discuss.pytorch.org/t/min-cut-optimal-recomputation-i-e-activation-checkpointing-with-aotautograd/467" class="inline-onebox" rel="noopener nofollow ugc">Min-cut optimal(*) recomputation (i.e. activation checkpointing) with AOTAutograd - compiler - PyTorch Developer Mailing List</a>.</p>