How will torch.cuda.Event `elapsed_time` method behave without explicit torch.cuda.synchronize?

username: justinliu
cooked: <p>Say I have some code that times some CUDA calls like this</p>
<pre><code class="lang-auto">import torch

start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)

start.record()
#
# some GPU work
#
end.record()
torch.cuda.synchronize()  # &lt;--- what if i remove this?
print(start.elapsed_time(end))
</code></pre>
<p>What will happen if I remove the explicit synchronization? Will <code>elapsed_time</code> actually block until both event finishes and still report the correct time, or will it report something unexpected?</p>

username: ptrblck
cooked: <p>From the <a href="https://developer.nvidia.com/blog/how-implement-performance-metrics-cuda-cc/#timing_using_cuda_events">NVIDIA DevBlog</a>:</p>
<blockquote>
<p>CUDA events are of type <a href="http://docs.nvidia.com/cuda/cuda-runtime-api/index.html#group__CUDART__TYPES_1gea2f543a9fc0e52fe4ae712920fd1247"><code>cudaEvent_t</code></a> and are created and destroyed with <code>cudaEventCreate()</code> and <code>cudaEventDestroy()</code>. In the above code <code>cudaEventRecord()</code> places the start and stop events into the default stream, stream 0. The device will record a time stamp for the event when it reaches that event in the stream. The function <code>cudaEventSynchronize()</code> blocks CPU execution until the specified event is recorded. The <code>cudaEventElapsedTime()</code> function returns in the first argument the number of milliseconds time elapsed between the recording of <code>start</code> and <code>stop</code>. This value has a resolution of approximately one half microsecond.</p>
</blockquote>
<p>You wouldn’t strictly need to synchronize the entire device and can also call <code>end.synchronize()</code> (or sync the stream). However, in minimal code snippets, you might not want to care if you sync the device.<br>
If you don’t call any sync, the code will properly fail with:</p>
<pre data-code-wrap="python"><code class="lang-python">RuntimeError: CUDA error: device not ready
</code></pre>

username: justinliu
cooked: <p>Thanks for sharing the reference and the explanation! In hindsight I guess I should rephrase my question as follows instead:</p>
<p>When I have multiple things to time in my code (let’s say I want to time different parts of my model’s forward), and I don’t really care about immediately logging the elapsed time as they finish (i.e. I log everything at the end), do I have to call synchronize in between start/end event pairs or can I just do one synchronize at the end before I log everything?</p>
<p>In code</p>
<pre><code class="lang-auto">import torch

start1 = torch.cuda.Event(enable_timing=True)
end1 = torch.cuda.Event(enable_timing=True)

start2 = torch.cuda.Event(enable_timing=True)
end2 = torch.cuda.Event(enable_timing=True)

start1.record()
#
# 1st part of GPU work
#
end1.record()
torch.cuda.synchronize()  # &lt;--- is this needed at all for the 2 prints to correctly report time in the end?

start2.record()
#
# 2nd part of GPU work
#
end2.record()
torch.cuda.synchronize()

print(start1.elapsed_time(end1))
print(start2.elapsed_time(end2))
</code></pre>

username: justinliu
cooked: <p>I guess the answer is we don’t need the intermediate syncs? I experimented with this myself and it seems to be that way but wasn’t sure.</p>