How does torch.cuda.synchronize() behave?

username: Alexander_Soare
cooked: <p>According to the PyTorch <a href="https://pytorch.org/docs/stable/generated/torch.cuda.synchronize.html" rel="noopener nofollow ugc">documentation</a> <code>torch.cuda.synchronize</code> “Waits for all kernels in all streams on a CUDA device to complete.”. Questions:</p>
<ol>
<li>
<p>Should this say “Waits for all kernels in all streams <strong>initiated by this Python session</strong> on a CUDA device to complete”? In other words, if Python session A is running CUDA operations, and I call <code>torch.cuda.synchronize()</code> in Python session B, that <strong>won’t</strong> care about what’s happening in Python session A right?</p>
</li>
<li>
<p>Surely if we don’t call <code>torch.cuda.synchronize()</code>, but try to work with any python code referencing the tensors in the computation graph, then it’s like implicitly calling it right?</p>
</li>
</ol>
<p>Q2 in code:</p>
<pre><code class="lang-auto">output = model(inputs)  # cuda starts working here
a = 1 + 1  # cuda might still be running the previous line. This line can run at the same time
other_model(output) # This implicitly does the same thing as torch.cuda.synchronize() then does a forward pass of other_model
b = a + a  # This line can't happen until cuda is done and the previous line has been executed
</code></pre>

username: ptrblck
cooked: <ol>
<li>
<p>Work of independent processes should be serialized (CUDA MPS might be the exception). Process A doesn’t know anything about process B, so a <code>synchronize()</code> (or <code>cudaDeviceSynchronize</code>) call would synchronize the work of the current process. However, if process B uses the GPU for a display output etc. you might see a latency increase depending when the context switch occurs.</p>
</li>
<li>
<p>It depends what you are trying to do in your Python code. E.g. if you are only scheduling work, no synchronizations are needed and won’t be added. On the other hand, if you are e.g. printing a value of a <code>CUDATensor</code>, then an implicit sync is added since the value has to be computed and pushed to the CPU first.</p>
</li>
</ol>

username: Alexander_Soare
cooked: <p>Thanks so much for picking this up <a class="mention" href="/u/ptrblck">@ptrblck</a> ! That clarifies some things for me. So a related follow up: in various blogs/snippets showing how to time a model forward pass correctly I’ve <a href="https://deci.ai/resources/blog/measure-inference-time-deep-neural-networks/" rel="noopener nofollow ugc">seen a pattern like</a>:</p>
<pre><code class="lang-python">starter, ender = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
with torch.inference_mode():
    starter.record()
    model.inference(inp)
    ender.record()
    torch.cuda.synchronize()
total_time += starter.elapsed_time(ender)
</code></pre>
<p>Although, <a href="https://gist.github.com/aurotripathy/2ee988e9bbaecc78a509fd6a21d3076d#file-time_all_reduce_vgg_model-py-L35-L38" rel="noopener nofollow ugc">at least once</a> I’ve also seen it with  the <code>ender.record()</code> and <code>torch.cuda.synchronize()</code> swapped. I can see why someone might think you should swap them. Maybe they consider that <code>ender.record()</code> might run before the inference is done if synchronization is not done first, following the logic in <a href="https://discuss.pytorch.org/t/how-to-measure-execution-time-in-pytorch/111458">this snippet</a>.  Although I’m wondering if <code>ender.record()</code> behaves differently from <code>time.time()</code>. <strong>Long question short: which way is correct and why?</strong></p>

username: ptrblck
cooked: <p>Generally, <code>torch.utils.benchmark</code> is a great tool to profile code, as it’s adding warmup iterations, synchronizes, etc.<br>
For information about using <code>cudaEvent</code>s for profiling, take a look at <a href="https://developer.nvidia.com/blog/how-implement-performance-metrics-cuda-cc/">this post</a>, which shows an example as:</p>
<pre data-code-wrap="python"><code class="lang-python">cudaEvent_t start, stop;
cudaEventCreate(&amp;start);
cudaEventCreate(&amp;stop);

cudaMemcpy(d_x, x, N*sizeof(float), cudaMemcpyHostToDevice);
cudaMemcpy(d_y, y, N*sizeof(float), cudaMemcpyHostToDevice);

cudaEventRecord(start);
saxpy&lt;&lt;&lt;(N+255)/256, 256&gt;&gt;&gt;(N, 2.0f, d_x, d_y);
cudaEventRecord(stop);

cudaMemcpy(y, d_y, N*sizeof(float), cudaMemcpyDeviceToHost);

cudaEventSynchronize(stop);
float milliseconds = 0;
cudaEventElapsedTime(&amp;milliseconds, start, stop);
</code></pre>
<blockquote>
<p>CUDA events are of type <a href="http://docs.nvidia.com/cuda/cuda-runtime-api/index.html#group__CUDART__TYPES_1gea2f543a9fc0e52fe4ae712920fd1247"><code>cudaEvent_t</code></a> and are created and destroyed with <code>cudaEventCreate()</code> and <code>cudaEventDestroy()</code>. In the above code <code>cudaEventRecord()</code> places the start and stop events into the default stream, stream 0. The device will record a time stamp for the event when it reaches that event in the stream. The function <code>cudaEventSynchronize()</code> blocks CPU execution until the specified event is recorded. The <code>cudaEventElapsedTime()</code> function returns in the first argument the number of milliseconds time elapsed between the recording of <code>start</code> and <code>stop</code>. This value has a resolution of approximately one half microsecond.</p>
</blockquote>
<p>The TL;DR: you have to synchronize the event either directly via the event object or globally via <code>torch.cuda.synchronize()</code>.</p>

username: Alexander_Soare
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> this answers my follow up question.</p>

username: Sayak_Paul
cooked: <p>For the interested readers.</p>
<p>Simple time-based benchmarks also benefit from <code>torch.cuda.synchronize()</code>: <a href="https://colab.research.google.com/gist/sayakpaul/9ca8fe1fb620d936241fd6f51a5a2814/scratchpad.ipynb" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>