Best way to detect and fix graphs breaks

Naveen_Marri: <p>Hi, how do we check for graph breaks when using <code>torch.compile()</code> ? I used <code>tlparse</code> to get the logs but the logs ended up with <code>Metrics were missing</code> . The training goes fine when using <code>torch.compile(model, fullgraph=True)</code> , does this mean there are no graph breaks?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b13b009039286b7e6cbf35cea804a2fe6ddfe1f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1b13b009039286b7e6cbf35cea804a2fe6ddfe1f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b13b009039286b7e6cbf35cea804a2fe6ddfe1f_2_690x231.jpeg" alt="image" data-base62-sha1="3Rx4tOcVsAaxnVTGbyQLJyOgLpR" width="690" height="231" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b13b009039286b7e6cbf35cea804a2fe6ddfe1f_2_690x231.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b13b009039286b7e6cbf35cea804a2fe6ddfe1f_2_1035x346.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b13b009039286b7e6cbf35cea804a2fe6ddfe1f_2_1380x462.jpeg 2x" data-dominant-color="EAEAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2328×782 294 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

bdhirsh: <p>Hmm, this usually means that compile crashed midway through, before it finished dumping logs to tlparse.</p>
<p>A few questions:</p>
<p>(1) Does your program crash (with a stacktrace) when you run it?</p>
<p>(2) I think tlparse is generally great for debugging. Another lighter weight option for graph breaks specifically is running with TORCH_LOGS=graph_breaks. Does that tell you anything?</p>
<p>(3) finally, if you think your error is easily reproable (e.g by cloning a GitHub repro and installing dependencies), you can file a issue on PyTorch GitHub and I or someone can take a look</p>

Naveen_Marri: <ol>
<li>Program doesn’t crash with any errors. I do get decent performance gains from <code>torch.compile(model, fullgraph=True)</code></li>
</ol>
<pre><code class="lang-auto">tlparse /tmp/tracedir/dedicated_log_torch_trace_rank_0_jk_xflq1.log  -o ~/tracedir-out/
Detected rank: Some(0)
  [00:00:00] [##################################################################################################################################################################################################################################################################################################################################################] 6.94 MiB/6.94 MiB [192.78 MiB/s] (0s)
  Stats { ok: 759, other_rank: 0, fail_glog: 0, fail_json: 0, fail_payload_md5: 0, fail_dynamo_guards_json: 0, fail_parser: 0, unknown: 0 }                                                                                                                                                                                                                                                            Stats { ok: 760, other_rank: 0, fail_glog: 0, fail_json: 0, fail_payload_md5: 0, fail_dynamo_guards_json: 0, fail_parser: 0, unknown: 0 }
</code></pre>
<ol start="2">
<li><code>TORCH_LOGS=graph_breaks</code> doesn’t give any informative things, only log I get is the following</li>
</ol>
<pre><code class="lang-auto">[rank1]:W0321 23:10:13.467000 457758 site-packages/torch/_dynamo/backends/distributed.py:89] [0/0] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance.
[rank0]:W0321 23:10:13.493000 457757 site-packages/torch/_dynamo/backends/distributed.py:89] [0/0] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance.
[rank1]:W0321 23:10:13.558000 457758 site-packages/torch/_dynamo/backends/distributed.py:106] [0/0] DDPOptimizer extended these buckets to ensure per-subgraph output nodes:
[rank1]:W0321 23:10:13.558000 457758 site-packages/torch/_dynamo/backends/distributed.py:106] [0/0] ┌─────────┬─────────────┬────────────────────────┐
[rank1]:W0321 23:10:13.558000 457758 site-packages/torch/_dynamo/backends/distributed.py:106] [0/0] │   Index │   Extra Ops │   Extra Param Size (b) │
[rank1]:W0321 23:10:13.558000 457758 site-packages/torch/_dynamo/backends/distributed.py:106] [0/0] ├─────────┼─────────────┼────────────────────────┤
[rank1]:W0321 23:10:13.558000 457758 site-packages/torch/_dynamo/backends/distributed.py:106] [0/0] │       1 │           1 │                      0 │
[rank1]:W0321 23:10:13.558000 457758 site-packages/torch/_dynamo/backends/distributed.py:106] [0/0] └─────────┴─────────────┴────────────────────────┘
[rank0]:W0321 23:10:13.574000 457757 site-packages/torch/_dynamo/backends/distributed.py:106] [0/0] DDPOptimizer extended these buckets to ensure per-subgraph output nodes:
[rank0]:W0321 23:10:13.574000 457757 site-packages/torch/_dynamo/backends/distributed.py:106] [0/0] ┌─────────┬─────────────┬────────────────────────┐
[rank0]:W0321 23:10:13.574000 457757 site-packages/torch/_dynamo/backends/distributed.py:106] [0/0] │   Index │   Extra Ops │   Extra Param Size (b) │
[rank0]:W0321 23:10:13.574000 457757 site-packages/torch/_dynamo/backends/distributed.py:106] [0/0] ├─────────┼─────────────┼────────────────────────┤
[rank0]:W0321 23:10:13.574000 457757 site-packages/torch/_dynamo/backends/distributed.py:106] [0/0] │       1 │           1 │                      0 │
[rank0]:W0321 23:10:13.574000 457757 site-packages/torch/_dynamo/backends/distributed.py:106] [0/0] └─────────┴─────────────┴────────────────────────┘
/opt/.pyenv/versions/3.10.14/lib/python3.10/site-packages/torch/_inductor/lowering.py:1713: UserWarning: Torchinductor does not support code generation for complex operators. Performance may be worse than eager.
  warnings.warn(
/opt/.pyenv/versions/3.10.14/lib/python3.10/site-packages/torch/_inductor/lowering.py:1713: UserWarning: Torchinductor does not support code generation for complex operators. Performance may be worse than eager.
  warnings.warn(
</code></pre>
<ol start="3">
<li>I do have a complex codebase, I can try minifying tool but from past PyTorch issues it seems like it doesn’t work well.</li>
</ol>

jjjj: <aside class="quote no-group" data-username="Naveen_Marri" data-post="1" data-topic="218127">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/naveen_marri/48/76283_2.png" class="avatar"> Naveen_Marri:</div>
<blockquote>
<p>tlparse</p>
</blockquote>
</aside>
<p>Is it is compiling/running successfully with fullgraph=True then there isn’t any graph breaks?</p>

Naveen_Marri: <aside class="quote no-group" data-username="jjjj" data-post="4" data-topic="218127">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jjjj/48/76279_2.png" class="avatar"> jjjj:</div>
<blockquote>
<p>Is it is compiling/running successfully with fullgraph=True then there isn’t any graph breaks?</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/jjjj">@jjjj</a> It does compile successfully with <code>fullgraph=True</code></p>

bdhirsh: <p>jjjj is right- this means that your code is compiling without any graph breaks. Is there another question here?</p>