Interpreting profiler results

username: vermont_waters
cooked: <p>When profiling a network, I see that the non-zero operation aparently takes a lot of time around 20ms:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/a/da0a86d807577e8b8639624d86d77aa51b2c6d5e.png" data-download-href="https://discuss.pytorch.org/uploads/default/da0a86d807577e8b8639624d86d77aa51b2c6d5e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da0a86d807577e8b8639624d86d77aa51b2c6d5e_2_690x362.png" alt="image" data-base62-sha1="v6SDNM4Lmp10tXJEkhb4NLcDdOC" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da0a86d807577e8b8639624d86d77aa51b2c6d5e_2_690x362.png, https://discuss.pytorch.org/uploads/default/original/3X/d/a/da0a86d807577e8b8639624d86d77aa51b2c6d5e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/a/da0a86d807577e8b8639624d86d77aa51b2c6d5e.png 2x" data-dominant-color="E1D2D1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">984×517 40.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Digging deeper into this I used<br>
<code>torch.cuda.synchronize()</code> before profiling that operator to see that the non-zero operator takes only around 1ms<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c74d6ad215911da1d94e10729c50728c4d17cecc.png" data-download-href="https://discuss.pytorch.org/uploads/default/c74d6ad215911da1d94e10729c50728c4d17cecc" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c74d6ad215911da1d94e10729c50728c4d17cecc_2_690x201.png" alt="image" data-base62-sha1="sr6SHL2hCHMtyZJ3cjnqeJWj5Ew" width="690" height="201" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c74d6ad215911da1d94e10729c50728c4d17cecc_2_690x201.png, https://discuss.pytorch.org/uploads/default/original/3X/c/7/c74d6ad215911da1d94e10729c50728c4d17cecc.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/7/c74d6ad215911da1d94e10729c50728c4d17cecc.png 2x" data-dominant-color="E7DEE5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">969×283 21.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am thinking, that some other kernel might take some time before and therefore the long wait/synchronize is added. However, I am not sure how to procede from there. I have no information about which operators are running. Is there any detailed tutorial on the profiler, or can someone help out here?</p>

username: ptrblck
cooked: <p>The <code>nonzero</code> operation will synchronize the code and will thus accumulate the execution time of running (async) CUDA operations. If you manually synchronize the code the actual time will be displayed.</p>

username: vermont_waters
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> that is my understanding. What I am trying to grasp is which operators are running in the background that cause the syncing to take this much time, independent of which operator triggers it. I am guessing there are some operators from the layers before, mainly convolutions, which still need more time to finish, e.g. volta_sgemm_128x32_tn?! It’s not clear to me which these are, to find out if i can optimize sth in the layers before.</p>
<p>E.g. I ran the network removing the intermediate layers with the non-zero function, which lead to a profiled speedup of roughly the syncing time as well, so I assume there is some potential there.</p>

username: ptrblck
cooked: <p>Yes, the previous operations will be synchronized and will be shown in the profile. I don’t fully understand what exactly you are trying to speed up though. In the optimal case, don’t call into <code>nonzero</code> as there is currently no way around the sync.</p>

username: GrantElliot
cooked: <p>It seems like there might be a kernel execution delay before the non-zero operation. Profiling with more granularity is essential. You can check PyTorch’s official documentation or seek guidance on forums like Stack Overflow for detailed profiling and troubleshooting tips regarding your specific setup<a href="https://raiderlinkttu.us/" rel="noopener nofollow ugc">.</a></p>

username: ptrblck
cooked: <p>You could explain what “kernel execution delay” means in this context and how it’s related to the issue?</p>