Clarification on torch.cuda.synchronize() and CPU-GPU Synchronization

anonymous_anonymous1: <p>Hello everyone,</p>
<p>I have a question regarding <code>torch.cuda.synchronize()</code> and how the CPU interacts with GPU tasks in PyTorch.</p>
<p>I’ve already read this thread: <a href="https://discuss.pytorch.org/t/how-to-measure-execution-time-in-pytorch/111458" class="inline-onebox">How to measure execution time in PyTorch?</a>, but I still have some doubts about how synchronization works.</p>
<p>It is commonly stated that if you issue a command for the GPU, the CPU will continue executing without waiting for the GPU to finish its process. I wanted to explore this behavior using two similar code snippets:</p>
<p><strong>Code 1</strong></p>
<pre><code class="lang-auto">import torch

device = "cuda:0"
size = 2**22
x = torch.arange(0, 2**20) / size + 1j * torch.arange(0, 2**20) / size
x = x.to(device)
X = 0  # Initial value that will be overwritten
X = torch.fft.fft(x)  # This operation runs asynchronously on the GPU
y = X + 1  # CPU continues execution and should use the initial X value if it runs too soon
print(y)  # The result is printed. Should be 1 because fft is heavy and takes time
</code></pre>
<p><strong>Code 2</strong></p>
<pre><code class="lang-auto">import torch

device = "cuda:0"
size = 2**22
x = torch.arange(0, 2**20) / size + 1j * torch.arange(0, 2**20) / size
x = x.to(device)
X = torch.fft.fft(x)  # This operation runs asynchronously on the GPU
torch.cuda.synchronize()  # Explicit synchronization with the GPU
y = X + 1  # Now the CPU waits for the GPU to finish and will use the X from fft computation
print(y)  # The result is printed

</code></pre>
<p>Based on the explanation in the discussion thread, <strong>code 1</strong> should print <code>y = 1</code> since the CPU proceeds without waiting for the GPU result, and <strong>code 2</strong> should print <code>y = X + 1</code>, given that synchronization is forced. However, when I run both codes, they produce the <strong>same correct result</strong> (i.e., <code>y = X + 1</code> in both cases).</p>
<h3><a name="p-456545-my-question-is-1" class="anchor" href="#p-456545-my-question-is-1"></a>My question is:</h3>
<p>What mechanism ensures that the CPU in <strong>code 1</strong> does not print an incorrect result (like <code>y = 1</code>), even though <code>torch.cuda.synchronize()</code> is not explicitly called? How does the CPU “know” to wait for the GPU result before proceeding to the next line?</p>

ptrblck: <aside class="quote no-group" data-username="anonymous_anonymous1" data-post="1" data-topic="211147">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anonymous_anonymous1/48/73154_2.png" class="avatar"> anonymous_anonymous1:</div>
<blockquote>
<p>Based on the explanation in the discussion thread, <strong>code 1</strong> should print <code>y = 1</code> since the CPU proceeds without waiting for the GPU result,</p>
</blockquote>
</aside>
<p>No, it shouldn’t since <code>y = X + 1</code> will schedule another CUDA kernel on the same default CUDAStream and will thus not create a race condition as you expect.</p>

anonymous_anonymous1: <p>CPU has reached the y = X + 1 line before X = torch.fft.fft(x) finished.</p>
<p>How does the CPU knows to launch a CUDA kernel on y = X + 1 since for the CPU X = 0 which is an integer not a CUDA variable ?</p>

ptrblck: <aside class="quote no-group" data-username="anonymous_anonymous1" data-post="3" data-topic="211147">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anonymous_anonymous1/48/73154_2.png" class="avatar"> anonymous_anonymous1:</div>
<blockquote>
<p>since for the CPU X = 0 which is an integer not a CUDA variable</p>
</blockquote>
</aside>
<p>That’s not the case since <code>X</code> is the return value of a CUDA kernel and PyTorch will thus schedule the launch of the next CUDA operation using this tensor as its input.</p>