Autograd process isn't available when I profile via torch.profiler

joshkim: <p>Hi,</p>
<p>I’m using torch.profiler to profile both forward and backward in my code. I also use the Tensorboard plugin (I know its deprecated) and Chrome trace to visualize the result.</p>
<p>Here is the result of profiling two steps of my training function:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7adb1cfe80dc25bff8bd335c9fa43f090bfd0e5a.png" data-download-href="https://discuss.pytorch.org/uploads/default/7adb1cfe80dc25bff8bd335c9fa43f090bfd0e5a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7adb1cfe80dc25bff8bd335c9fa43f090bfd0e5a_2_517x240.png" alt="image" data-base62-sha1="hwPHmrevQgYvSwMzKgn8KPFjlNg" width="517" height="240" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7adb1cfe80dc25bff8bd335c9fa43f090bfd0e5a_2_517x240.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7adb1cfe80dc25bff8bd335c9fa43f090bfd0e5a_2_775x360.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7adb1cfe80dc25bff8bd335c9fa43f090bfd0e5a_2_1034x480.png 2x" data-dominant-color="D6D9E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1216×564 139 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As you can clearly see, there are three traces: main process, the autograd process, and the GPU process.</p>
<p>However, the trace of autograd process disappears when I use CUDA graph.</p>
<p>My code uses <code>torch.cuda.CUDAGraph()</code> to manually build a CUDA graph of my training code (both forward and backward including optimizer). However, that somehow eliminates the<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/2/329e7516d3cfc5ef50f0e9911226058fe0e5e501.png" data-download-href="https://discuss.pytorch.org/uploads/default/329e7516d3cfc5ef50f0e9911226058fe0e5e501" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/329e7516d3cfc5ef50f0e9911226058fe0e5e501_2_517x372.png" alt="image" data-base62-sha1="7dNmKkK8iHEAvt03fRjIhtLHz57" width="517" height="372" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/329e7516d3cfc5ef50f0e9911226058fe0e5e501_2_517x372.png, https://discuss.pytorch.org/uploads/default/original/3X/3/2/329e7516d3cfc5ef50f0e9911226058fe0e5e501.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/2/329e7516d3cfc5ef50f0e9911226058fe0e5e501.png 2x" data-dominant-color="CCD6E0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">695×501 28.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Question: Is this expected or is it a bug? I understand CUDA graphs launches a series of CUDA kernels at once, but why would this eliminate the entire autograd process?</strong></p>

ptrblck: <p>I would expect this behavior since you are capturing the kernel launches and let the device execute it via <code>cudaGraphLaunch</code>. In PyTorch Eager the host launches the kernel from the main and backward thread.</p>

joshkim: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for your reply!</p>
<p>I see. So let me get this straight. Are you saying there is NO backward thread running at all when I use CUDA graph or the profiler simply doesn’t show the thread?</p>
<p>Also, the profiler trace doesn’t show GPU Est. SM Efficiency and Utilization when I use CUDA graph like the eager mode (both Chrome trace and Tensorboard). Is this expected?</p>
<p>Thanks for your help!</p>

ptrblck: <aside class="quote no-group" data-username="joshkim" data-post="3" data-topic="219049">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/a8b319/48.png" class="avatar"> joshkim:</div>
<blockquote>
<p>Are you saying there is NO backward thread running at all when I use CUDA graph or the profiler simply doesn’t show the thread?</p>
</blockquote>
</aside>
<p>Since you are launching a single CUDAGraph, no host thread is involved in any other launches which is one of the main advantages of using CUDAGraphs.</p>
<aside class="quote no-group" data-username="joshkim" data-post="3" data-topic="219049">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/a8b319/48.png" class="avatar"> joshkim:</div>
<blockquote>
<p>Also, the profiler trace doesn’t show GPU Est. SM Efficiency and Utilization when I use CUDA graph like the eager mode (both Chrome trace and Tensorboard). Is this expected?</p>
</blockquote>
</aside>
<p>I’m not deeply familiar with the native profiler visualization in chrome and use Nsight Systems most of the time, so don’t know if these attributes are expected to be missing.</p>