Why doesn't the instrumented execution time match the time captured by nsys?

username: lilill
cooked: <p>Forgive me for asking what may seem like a silly question.</p>
<p>When I measure time using instrumentation, I only get a few milliseconds. Here’s my measurement method:</p>
<pre><code class="lang-auto">start_event = torch.cuda.Event(enable_timing=True)
end_event = torch.cuda.Event(enable_timing=True)

start_event.record()
// module
end_event.record()

torch.cuda.synchronize()

elapsed_time = start_event.elapsed_time(end_event)
print(f"total time: {elapsed_time} ms")
</code></pre>
<p>However, when I look at the timeline captured by nsys, I can see that there’s a 1-second gap just between two batch norm kernel calls.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1adff5de2c74aab04a96c04485fb26b043843dcb.png" data-download-href="https://discuss.pytorch.org/uploads/default/1adff5de2c74aab04a96c04485fb26b043843dcb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1adff5de2c74aab04a96c04485fb26b043843dcb_2_690x429.png" alt="image" data-base62-sha1="3PKfkjNgzakLV9icJKQrdXuq4OL" width="690" height="429" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1adff5de2c74aab04a96c04485fb26b043843dcb_2_690x429.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1adff5de2c74aab04a96c04485fb26b043843dcb_2_1035x643.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1adff5de2c74aab04a96c04485fb26b043843dcb_2_1380x858.png 2x" data-dominant-color="E3E8E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1380×858 408 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Since there’s a 1-second gap even between two batch norm kernels, the total execution time certainly shouldn’t be just a few milliseconds. Why is this happening? Am I misunderstanding the data captured by nsys? Or is there an issue with my measurement method? Or does this measurement method only count the sum of kernel execution times without including idle periods?</p>
<p>Additionally, could I use the time elapsed between <code>CUDA profiling initialization</code> and <code>CUDA profiling data flush</code> as my total time (including time for H to D or D to H data transfers)?</p>

username: ptrblck
cooked: <p>Your events measure the kernel execution time on the GPU, not host overhead, IDLE times, CPU bottlenecks etc. The Nsight System view shows you the actual timeline with kernel lauches, kernel execution time, etc.</p>

username: lilill
cooked: <p>Thank you very much for your response!! I think I understand what you mean.</p>
<p>I’d like to ask some more detailed questions:</p>
<p>If I want to get the actual execution time (including kernel launch and idle waiting times in between), can I get that directly through instrumentation in PyTorch?</p>
<p>Or can I get it directly from nsys, but I’m not sure which point in the nsys timeline I should use as the start of timing and which point as the end?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="lilill" data-post="3" data-topic="218025">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lilill/48/76247_2.png" class="avatar"> lilill:</div>
<blockquote>
<p>Or can I get it directly from nsys, but I’m not sure which point in the nsys timeline I should use as the start of timing and which point as the end?</p>
</blockquote>
</aside>
<p>I would recommend sticking to a visual profiler to get a full overview of the execution timeline. Nsight systems or the native PyTorch profiler would be some options.<br>
You can use <code>nvtx</code> markers inside your code to mark regions which would show up in <code>nsys</code>.</p>

username: lilill
cooked: <p>I’d like to ask one more question:</p>
<p>Why is the idle time between kernels so long? The kernel execution time is only a few milliseconds, but the waiting time is over a second, which is a completely different order of magnitude.</p>
<p>Is this normal behavior without applying other optimizations? What causes this situation?<br>
Or is it because the model I’m executing or the input data is relatively small, resulting in this phenomenon?</p>

username: ptrblck
cooked: <p>Your workload might be CPU-limited meaning your CPU is not fast enough in scheduling the kernels. You could try to apply CUDA Graphs to reduce the CPU overhead and profile it again.</p>