How does model(data) work?

ihchoi12: <p>Hello!</p>
<p>I’m trying to understand how PyTorch launches CUDA kernels during the forward propagation.<br>
For example, this is the forward() function of resnet50:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d77bba944abaa50e015957298756a160989ed281.png" data-download-href="https://discuss.pytorch.org/uploads/default/d77bba944abaa50e015957298756a160989ed281" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d77bba944abaa50e015957298756a160989ed281_2_417x500.png" alt="image" data-base62-sha1="uKfK7FQqoy0avahPRLKqQD2v2lr" width="417" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d77bba944abaa50e015957298756a160989ed281_2_417x500.png, https://discuss.pytorch.org/uploads/default/original/3X/d/7/d77bba944abaa50e015957298756a160989ed281.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/7/d77bba944abaa50e015957298756a160989ed281.png 2x" data-dominant-color="292B2A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">498×596 37.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Looks like it launches the kernels layer by layer sequentially, but I heard that launching CUDA kernel is asynchronous, so CPU thread does not wait for the kernel to be completed.</p>
<p>So, could someone explain to me how these kernels can be launched asynchronously? In other words, how can the kernel for layer N be launched without waiting for the result from the layer N-1? Thanks!</p>

ptrblck: <p>The kernels are launches on the same CUDAStream and will thus be executed in order. The execution is still asynchronous and the CPU can thus run ahead and enqueue the next kernel without blocking on the return value.<br>
If you are using e.g. data-dependent control flow (e.g. by checking a value of an output tensor in a condition) PyTorch will synchronize for you, since the actual return value is needed to select the next code execution path.</p>

ihchoi12: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for the clarification!</p>
<p>I have a follow-up question on that:<br>
The overhead for the host thread to launch (enqueue) kernels to the CUDAStream is bigger when GPU is busy? I observe that the time for each step varies quite a lot depending on whether I run <code>cuda.synchronize()</code> in each iteration. Here is the details.</p>
<p>This is the iteration code I’m trying (I run it without enabling the nsys profiler):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7254fabbf2a6ce9857f84216d6dc6e985e0e380.png" data-download-href="https://discuss.pytorch.org/uploads/default/a7254fabbf2a6ce9857f84216d6dc6e985e0e380" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7254fabbf2a6ce9857f84216d6dc6e985e0e380_2_472x500.png" alt="image" data-base62-sha1="nQDFWmuLgTHjAXhiN1W179WHrI4" width="472" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7254fabbf2a6ce9857f84216d6dc6e985e0e380_2_472x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7254fabbf2a6ce9857f84216d6dc6e985e0e380_2_708x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7254fabbf2a6ce9857f84216d6dc6e985e0e380_2_944x1000.png 2x" data-dominant-color="262626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1214×1284 168 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I run it <strong>without</strong> the <code>torch.cuda.synchronize()</code> after FP, the result is like this.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cacbfc9febb928de3f15ef8c53ac692a358f313.png" data-download-href="https://discuss.pytorch.org/uploads/default/6cacbfc9febb928de3f15ef8c53ac692a358f313" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cacbfc9febb928de3f15ef8c53ac692a358f313.png" alt="image" data-base62-sha1="fvnGaJkFdBXdlk7IexCThBj9xLB" width="690" height="213" data-dominant-color="3A3A3A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">948×294 17.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I observe that launching kernels for forward propagation takes ~40ms, and for gradient computation takes ~54ms.</p>
<p>When I run it <strong>with</strong> the <code>torch.cuda.synchronize()</code> after FP, the result is like this.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/4/94bb3835e8e862f3cbacddf2df74fc90d4160bf6.png" data-download-href="https://discuss.pytorch.org/uploads/default/94bb3835e8e862f3cbacddf2df74fc90d4160bf6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/4/94bb3835e8e862f3cbacddf2df74fc90d4160bf6.png" alt="image" data-base62-sha1="ldJMsTzMWQcxBZwD9jmqZYeIRoi" width="690" height="273" data-dominant-color="3A3A3A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">894×354 20.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Here, I observe that launching kernels for forward propagation takes ~17ms, and for gradient computation takes ~16ms.</p>
<p>It looks quite weird to me that asynchronous kernel launches (i.e., without waiting for the kernel completion) take very different times depending on the GPU’s state. Could you kindly give me any insight on this? Thanks a lot!</p>
<p>Inho</p>

ptrblck: <aside class="quote no-group" data-username="ihchoi12" data-post="3" data-topic="156008">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ihchoi12/48/49424_2.png" class="avatar"> ihchoi12:</div>
<blockquote>
<p>The overhead for the host thread to launch (enqueue) kernels to the CUDAStream is bigger when GPU is busy?</p>
</blockquote>
</aside>
<p>No, I don’t think that should be the case.<br>
I would not dive into your profiling results as you interpreting profiling results without proper synchronizations is invalid and might be too confusing since you would need to “know” where the code would naturally synchronize.<br>
If you are concerned about the kernel launch times, profile the code with Nsight Systems and check where the kernel launches are vs. their execution and how long each step takes.</p>

ihchoi12: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="156008">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>you would need to “know” where the code would naturally synchronize.</p>
</blockquote>
</aside>
<p>Does it means that the PyTorch does naturally synchronize sometimes even without explicit cuda.synchronize() call?<br>
Could you give me some typical example cases where the natural synchronizations happen?<br>
Thanks!</p>

ptrblck: <p>Yes, some operations are performing computations on the CPU e.g. in <code>tensor.nonzero()</code>, each <code>print(cuda_tensor)</code> statement (since the CPU needs to read the actual value), memcopies if <code>non_blocking=False</code> and thus pageable memory is used, every control flow which needs to read tensor values (e.g. <code>if (cuda_tensor &gt; 0).all()</code>).</p>

ptrblck: <p>Btw. if you are interested in reducing the kernel launch overheads (which would be especially beneficial if your actual GPU workload is small compared to the launches) you could check the <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-graphs">CUDA Graphs</a> util.</p>

ihchoi12: <p>I see, thanks for the great suggestions! I will check them out.</p>