Weird Pytorch profiler output

username: benoriol
cooked: <p>Hello,</p>
<p>I am profiling my training code and I’m struggling to understand the output. The different steps seem to be taking different amount of time and I am not sure why.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7baeb1e08f8c7ba35939433d1187ca88845a97e0.png" data-download-href="https://discuss.pytorch.org/uploads/default/7baeb1e08f8c7ba35939433d1187ca88845a97e0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7baeb1e08f8c7ba35939433d1187ca88845a97e0_2_690x233.png" alt="image" data-base62-sha1="hE90FAiB5VAebbOBsrKDefJ6rMk" width="690" height="233" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7baeb1e08f8c7ba35939433d1187ca88845a97e0_2_690x233.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7baeb1e08f8c7ba35939433d1187ca88845a97e0_2_1035x349.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7baeb1e08f8c7ba35939433d1187ca88845a97e0_2_1380x466.png 2x" data-dominant-color="DACFDC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">3082×1044 378 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>A simplified version of my code would be:</p>
<pre><code class="lang-auto">with profiler.profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], ...) as p:
    for x, y in training_dataloader:
        p.step()
        out = model(x)
        loss = loss_fn(out,y)
        loss.backward()
        optim.step()
        optim.zero_grad()
</code></pre>
<p>Scheduler is set to <code>skip_first=10, wait=5, warmup=1, active=5, repeat=1</code></p>
<p>If I then add some operation that downloads something to cpu memory such as <code>l = loss.item()</code> after the optimizer step, I get something like this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b38181857a25f0a25d36e09230101b85264ee966.png" data-download-href="https://discuss.pytorch.org/uploads/default/b38181857a25f0a25d36e09230101b85264ee966" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b38181857a25f0a25d36e09230101b85264ee966_2_690x237.png" alt="image" data-base62-sha1="pBYVOcf4VpX4cJm1WKGWdascabQ" width="690" height="237" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b38181857a25f0a25d36e09230101b85264ee966_2_690x237.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b38181857a25f0a25d36e09230101b85264ee966_2_1035x355.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b38181857a25f0a25d36e09230101b85264ee966_2_1380x474.png 2x" data-dominant-color="DFD4E0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">3048×1050 292 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
In both scenarios the average train step is the same even though each profiler step seems to vary in time in the first case.</p>
<p>I am a bit confused by this behaviour. My hypothesis is that my profiler might be measuring the async python function calls and not the actual cuda kernels. But isn’t the point of the profiler to measure the cuda kernel time? I am interested in analysing how long does my CPU and GPU take in each part of the code, and I guess here I am just seeing the CPU?</p>
<p>If I go down below in the GPU thread part of the profiler, I get this, which I don’t understand or is giving me too little information about how long is spent in each kernel by the gpu.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/9/6926e6259784645fd5c11a6fe5d79efabbe9b025.png" data-download-href="https://discuss.pytorch.org/uploads/default/6926e6259784645fd5c11a6fe5d79efabbe9b025" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/6926e6259784645fd5c11a6fe5d79efabbe9b025_2_690x134.png" alt="image" data-base62-sha1="f0dtss8BRbvXLuj56t7O8nptcSp" width="690" height="134" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/6926e6259784645fd5c11a6fe5d79efabbe9b025_2_690x134.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/6926e6259784645fd5c11a6fe5d79efabbe9b025_2_1035x201.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/6926e6259784645fd5c11a6fe5d79efabbe9b025_2_1380x268.png 2x" data-dominant-color="BDCCCC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">4106×798 183 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Note that this screenshot is taken for the first example where i do NOT add l = loss.item() and all the different steps are taking different time in cpu time. Yet in the GPU thread they seem to be taking the same amount of time.</p>
<p>I would appreciate any insights on why this is happening or if my intuitions are correct and what is the best way to measure how long is my GPU spending in each kernel.</p>
<p>I am using a 4090 GPU and the model is a simple UNet with 128x128 image input.</p>
<p>Thanks,<br>
Benet</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="benoriol" data-post="1" data-topic="208078">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/benoriol/48/15004_2.png" class="avatar"> benoriol:</div>
<blockquote>
<p>If I then add some operation that downloads something to cpu memory such as <code>l = loss.item()</code> after the optimizer step, I get something like this:</p>
</blockquote>
</aside>
<p>This is expected since the <code>item()</code> call is synchronizing the GPU.</p>
<aside class="quote no-group" data-username="benoriol" data-post="1" data-topic="208078">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/benoriol/48/15004_2.png" class="avatar"> benoriol:</div>
<blockquote>
<p>I am a bit confused by this behaviour. My hypothesis is that my profiler might be measuring the async python function calls and not the actual cuda kernels.</p>
</blockquote>
</aside>
<p>Yes, this seems to be the case and I would expect to see different time lines: one for the host and another one for the device.</p>
<p>I’m not familiar with the native PyTorch profiler visualization as I am using Nsight Systems for profiling. You could check <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59">this post</a> for instructions on how to use <code>nsys</code>.</p>

username: benoriol
cooked: <p>That makes sense, thanks.<br>
One more question, is there a limit to the number of async python calls that pytorch can do without waiting for the actual computation to be done in the GPU? It seems that it is doing multiple faster non-blocking calls to training steps (<span class="hashtag-raw">#16</span> and <span class="hashtag-raw">#17</span>) before it starts to take longer in step <span class="hashtag-raw">#18</span>, I assume waiting for GPU to finish the previous ones. In general, what is a good resource where I can learn more about how Pytorch works under the hood?</p>
<p>In any case I will switch to nsys profiler, seems like a much better option. Thanks a lot.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="benoriol" data-post="3" data-topic="208078">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/benoriol/48/15004_2.png" class="avatar"> benoriol:</div>
<blockquote>
<p>One more question, is there a limit to the number of async python calls that pytorch can do without waiting for the actual computation to be done in the GPU?</p>
</blockquote>
</aside>
<p>Yes, the queue handling the CUDA kernel launches is limited and the host will be blocked if the queue is saturated.</p>