Large time consuming cudaMemcpyAsync between forward and backward pass in collected PyTorch profiler trace

username: Spam_Me
cooked: <p>There is a large amount of time spend in <code>cudaMemcpyAsync</code>, related to <code>Memcpy DtoH (Device -&gt; Pageable)</code> operation ,  between forward and backward pass, I do not know where it comes from.</p>
<p>I noticed this when I used PyTorch profiler.</p>
<p>I am attaching the observation when viewed in chrome tracing.</p>
<p>I am using the PyTorch data parallel example <a href="https://github.com/pytorch/examples/blob/main/imagenet/main.py" rel="noopener nofollow ugc">code</a> available in the PyTorch documentation.</p>
<p>Model: ResNet 18 from torchvision standard models.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/e/0e035eca73702c2109905e3f58d9133f6063fb4f.png" data-download-href="https://discuss.pytorch.org/uploads/default/0e035eca73702c2109905e3f58d9133f6063fb4f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0e035eca73702c2109905e3f58d9133f6063fb4f_2_690x203.png" alt="image" data-base62-sha1="1ZXU8fbY675p8INa1QxJhY8YoRV" width="690" height="203" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0e035eca73702c2109905e3f58d9133f6063fb4f_2_690x203.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0e035eca73702c2109905e3f58d9133f6063fb4f_2_1035x304.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/e/0e035eca73702c2109905e3f58d9133f6063fb4f.png 2x" data-dominant-color="F0E4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1218×359 23 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>As the profile shows it comes from <code>aten::item</code>, which is triggered by <code>tensor.item()</code> in Python and synchronizes the host with the GPU.<br>
Remove unneeded <code>item()</code> calls to avoid synchronizing the code as the <code>memcpy</code> kernel doesn’t take such a long time but needs to wait for the GPU to finish its execution before it can copy the output tensor to the CPU.</p>

username: edwardmilsom
cooked: <p>Sorry to piggyback on this topic, but I’m also experience a similar issue. However, in my case, I’m not the one calling item(), but instead it seems (according to the trace) that linalg_check_errors is calling it after computing the cholesky decomposition (all my tensors should be on the CUDA device):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/3027c08d4189f1568bea4a9511b566be9bc2fb76.png" data-download-href="https://discuss.pytorch.org/uploads/default/3027c08d4189f1568bea4a9511b566be9bc2fb76" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/3027c08d4189f1568bea4a9511b566be9bc2fb76_2_690x51.png" alt="image" data-base62-sha1="6S05p7Q2qwcHImDGi84RJ5ka0Pc" width="690" height="51" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/3027c08d4189f1568bea4a9511b566be9bc2fb76_2_690x51.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/3027c08d4189f1568bea4a9511b566be9bc2fb76_2_1035x76.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/3027c08d4189f1568bea4a9511b566be9bc2fb76_2_1380x102.png 2x" data-dominant-color="BB9F81"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1869×140 11.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is for some custom code which is computing the cholesky at every layer, so (at least according to the flamegraph) it is forming the major bottleneck in my code. Any insight would be appreciated!</p>

username: ptrblck
cooked: <p>This is also expected as described in the <a href="https://pytorch.org/docs/stable/generated/torch.linalg.cholesky.html#torch.linalg.cholesky">docs</a>:</p>
<blockquote>
<p>When inputs are on a CUDA device, this function synchronizes that device with the CPU.<br>
<a href="https://pytorch.org/docs/stable/generated/torch.linalg.cholesky_ex.html#torch.linalg.cholesky_ex"><code>torch.linalg.cholesky_ex()</code></a> for a version of this operation that skips the (slow) error checking by default and instead returns the debug information. This makes it a faster way to check if a matrix is positive-definite.</p>
</blockquote>

username: yjguo
cooked: <p>I understand the synchronizing consumes the time, but why the consumed time is shown in the figure by “cudaMemcpyAsync” which is an async function?  My expectation is that another function like cudaSync… consumes the time in the figure. thanks.</p>

username: ptrblck
cooked: <p><code>cudaMemcpyAsync</code> will be async is possible and will fall back to a synchronizing operation if needed, e.g. if pageable host memory is used.</p>

username: yjguo
cooked: <p>thanks.  And other possible case that <code>cudaMemcpyAsync</code> will fall back to sync operation?  In this example, I think it is D2D copy, not the case of host memory, why still fall back to sync operation.</p>