Why we use torch.cuda.synchronize()?

Mahdi_Amrollahi: <p>Why we use torch.cuda.synchronize()? When we do an operation on <code>cuda</code> device, does not it mean that it has done one the same line of code? Should we always wait for the ongoing operations on <code>cuda</code>?</p>
<pre><code class="lang-auto">import torch

# Check if GPU is available
if torch.cuda.is_available():
    # Move tensor to GPU
    device = torch.device("cuda")
    x = torch.randn(1000, 1000, device=device)

    y = x.matmul(x)

    # Wait for GPU operations to finish
    torch.cuda.synchronize()
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Mahdi_Amrollahi" data-post="1" data-topic="193111">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahdi_amrollahi/48/62481_2.png" class="avatar"> Mahdi_Amrollahi:</div>
<blockquote>
<p>Why we use torch.cuda.synchronize()?</p>
</blockquote>
</aside>
<p>To explicitly synchronize the code, e.g. to profile the kernel execution time.</p>
<aside class="quote no-group" data-username="Mahdi_Amrollahi" data-post="1" data-topic="193111">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahdi_amrollahi/48/62481_2.png" class="avatar"> Mahdi_Amrollahi:</div>
<blockquote>
<p>When we do an operation on <code>cuda</code> device, does not it mean that it has done one the same line of code?</p>
</blockquote>
</aside>
<p>No, since CUDA operations are executed asynchronously w.r.t. the CPU.</p>
<aside class="quote no-group" data-username="Mahdi_Amrollahi" data-post="1" data-topic="193111">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahdi_amrollahi/48/62481_2.png" class="avatar"> Mahdi_Amrollahi:</div>
<blockquote>
<p>Should we always wait for the ongoing operations on <code>cuda</code>?</p>
</blockquote>
</aside>
<p>No, as it will slow down your code since your CPU is constantly blocked.<br>
PyTorch adds needed synchronizations for you, e.g. if you want to print a <code>CUDATensor</code>.</p>

Mahdi_Amrollahi: <p>Sorry, but I am a bit confused. Suppose that we are doing an operation on cuda to predict some items and next we are going to calculate the loss e.g:</p>
<pre><code class="lang-auto">y_pred = model(x)

loss = criterion(y_pred, y_true)
</code></pre>
<p>So, should we explicitly add the synchronization before calculating the loss? Because we need to make sure that <code>y_pred</code> is ready, does not it?</p>
<p>Another question: how can <code>cuda</code> understand that for which operation should it wait for? For example, there are lots of processes running on GPU like any game, Blender, Adobe, or whatever. Which one should it wait for?</p>
<p>Thanks</p>

ptrblck: <p>No, you don’t need to manually synchronize the code as the next operation will be queued to the same (default) CUDAStream. Explicit synchronizations would be needed if you use custom streams.<br>
PyTorch creates an own CUDAContext for its execution and does not interfere with other applications. The driver schedules the execution.</p>