How to trace kernels in pytorch

oracal: <p>I wanna trace all related kernels when making DL inference in pytorch and store kernel information locally for other purposes. Unfortunately, I cannot find where I could get this in source code. In other words， where pytorch launch kernels when DL  inference. Could you  give me some suggestions?<br>
Thanks so much !<br>
God bless you</p>

ptrblck: <p>I’m not sure which “kernel information” you would like to store, but would profiling the workload work as described <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html">here</a>?</p>
<aside class="quote no-group" data-username="oracal" data-post="1" data-topic="160340">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/919ad9/48.png" class="avatar"> oracal:</div>
<blockquote>
<p>In other words， where pytorch launch kernels when DL inference.</p>
</blockquote>
</aside>
<p>There is not a single place where the kernels are launched and you would need to check which operations are used. E.g. a lot of kernels are launched in files stored in <a href="https://github.com/pytorch/pytorch/tree/master/aten/src/ATen"><code>aten/src/ATen/</code></a>, but I also don’t fully understand your use case and why you are looking for the locations of all kernel launches.</p>

oracal: <p>When I profiled vgg11 model inference with batchsize 1 , using Nsight System, I found that there are 45 kernels executed in total . I wanna trace all these kernels and re-organize them into a graph or stream besides some academic purpose. So how can I get that</p>

ptrblck: <p>I still don’t know what exactly you want to read in the end, but since you are already using Nsight you might want to add the <code>--trace</code> option and try to process the profile created by Nsight Systems, as it would show the callstack for each kernel.</p>

oracal: <p>actually, I wanna implement cuda-graph in C++ level manually rather than depending on Pytorch</p>

ptrblck: <p>Unsure why you are not using the built-in CUDA Graphs util. as it sounds quite challenging, but I’m sure you have valid reasons for it. Good luck and let me know how it goes!</p>

oracal: <p>I do not know how to collect kernels using pytorch API…</p>

oracal: <p>I am reading pytorch source code related  to CUDA graph. I am confused why torch devs designs cuda generator in <strong>struct <em>CUDAGraph</em></strong> in file <em><strong>CUDAGraph.h</strong></em> How can I use it</p>

ptrblck: <p>You can use CUDA Graphs in PyTorch as described <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-graphs">here</a>. I don’t know what exactly is confusing about the file naming.</p>

Vinayaka_Hegde: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> sorry for reviving an old topic.<br>
But, I also needed some help in getting the exact number of kernel launches that CUDA does for a given DNN training . I needed this for academic research, where we are trying to model the interference of 2 models as the number of kernel launches happening per layer on Nvidia Jetsons.</p>
<p>I have the following specific questions :</p>
<ol>
<li>
<p>Is there a flag or env variable that I can supply to pytorch at compile time in order to get info on how many kernels are launched for a DNN training ?</p>
</li>
<li>
<p>I used pytorch profiler and was able to get the <span class="hashtag-raw">#of</span> calls done for each operation. AFAIK, operators like <code>aten::convolution</code> or <code>aten::addmm</code> go as the input to <code>torch_dispatch</code>. Just wanted to confirm if these are also the names of kernels being called? If not, how can I get the actual kernel names and the # of times that kernel was invoked using Pytorch’s profiler ?</p>
</li>
</ol>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85b668f9c9df866a4218a72ca0bf1911aaabfd21.png" data-download-href="https://discuss.pytorch.org/uploads/default/85b668f9c9df866a4218a72ca0bf1911aaabfd21" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85b668f9c9df866a4218a72ca0bf1911aaabfd21.png" alt="image" data-base62-sha1="j4Sjy6xGCQJqA3rs1c5BQzY3Fnz" width="690" height="316" data-dominant-color="0F0F0F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1905×875 28.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks for your guidance!</p>

ptrblck: <p>You could use a CUDA profiler such as Nsight Systems, which will show all CUDA kernel launches.</p>