A question about time consumption

Ran: <p>A<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f80074305405976430244f18e907c8d746c3da8e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f80074305405976430244f18e907c8d746c3da8e" title="2019-08-05_161214"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f80074305405976430244f18e907c8d746c3da8e_2_517x201.jpeg" alt="2019-08-05_161214" data-base62-sha1="znVnSchoqEkGzUy04VTQB8rqDsa" width="517" height="201" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f80074305405976430244f18e907c8d746c3da8e_2_517x201.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f80074305405976430244f18e907c8d746c3da8e_2_775x301.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/f80074305405976430244f18e907c8d746c3da8e.jpeg 2x" data-dominant-color="EFEDEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2019-08-05_161214</span><span class="informations">912×356 144 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
B<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/8e6c9516b515135dd579f5e2e1f331aaff8f56ae.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8e6c9516b515135dd579f5e2e1f331aaff8f56ae" title="2019-08-05_160344"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/8/8e6c9516b515135dd579f5e2e1f331aaff8f56ae_2_517x208.jpeg" alt="2019-08-05_160344" data-base62-sha1="kjWr7EzV35goLPrNcNF4GdEIfqC" width="517" height="208" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/8/8e6c9516b515135dd579f5e2e1f331aaff8f56ae_2_517x208.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/8/8e6c9516b515135dd579f5e2e1f331aaff8f56ae_2_775x312.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/8/8e6c9516b515135dd579f5e2e1f331aaff8f56ae_2_1034x416.jpeg 2x" data-dominant-color="F4F3F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2019-08-05_160344</span><span class="informations">1107×447 150 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
C<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/aeaa67ead6fecf73a84d1a07fbfd9e6121ea1318.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/aeaa67ead6fecf73a84d1a07fbfd9e6121ea1318" title="2019-08-05_161725"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/aeaa67ead6fecf73a84d1a07fbfd9e6121ea1318_2_517x211.jpeg" alt="2019-08-05_161725" data-base62-sha1="oVaaIwwvWrWByZtLV0bZ7MK60FW" width="517" height="211" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/aeaa67ead6fecf73a84d1a07fbfd9e6121ea1318_2_517x211.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/a/aeaa67ead6fecf73a84d1a07fbfd9e6121ea1318_2_775x316.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/a/aeaa67ead6fecf73a84d1a07fbfd9e6121ea1318.jpeg 2x" data-dominant-color="EFEDEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2019-08-05_161725</span><span class="informations">911×373 156 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>(The third column means the time consumption(unit: μs))<br>
The three images are from three slightly different code snippets. Anyway, there will always be a line that takes up 60ms.<br>
I would like to know why this happens and how to shorten the time?</p>

ptrblck: <p>All “slow” lines contain a <code>cpu()</code> call, which will create a synchronization if your script runs on the GPU.<br>
To properly time CUDA code, you should synchronize before starting and stopping the timer (if you are manually profiling).</p>
<pre><code class="lang-python">torch.cuda.synchronize()
t0 = time.time()
...
torch.cuda.synchronize()
t1 = time.time()
</code></pre>
<p>You could also use the <a href="https://pytorch.org/docs/stable/autograd.html#profiler" rel="nofollow noopener">profiler</a> to measure the execution of your code.</p>

Ran: <p>The second picture doesn’t contain cpu().data, it is max_ids=ids.max(), ids is torch.cuda.Tensor.</p>

Ran: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3350edfddf6da604a2fa8a897cf3165e1d630449.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3350edfddf6da604a2fa8a897cf3165e1d630449" title="2019-08-05_215805"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3350edfddf6da604a2fa8a897cf3165e1d630449_2_690x304.jpeg" alt="2019-08-05_215805" data-base62-sha1="7jXK0apjPeAxt6jgbsFsLlz2p2F" width="690" height="304" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3350edfddf6da604a2fa8a897cf3165e1d630449_2_690x304.jpeg, https://discuss.pytorch.org/uploads/default/original/2X/3/3350edfddf6da604a2fa8a897cf3165e1d630449.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/3/3350edfddf6da604a2fa8a897cf3165e1d630449.jpeg 2x" data-dominant-color="F1F1F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2019-08-05_215805</span><span class="informations">965×426 175 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I added ‘torch.cuda.synchronize()’, and this line takes up 60ms. Is there a way to remove synchronization time?</p>