Time profile the code and pytorch behave oddly

JisongXie: <p>Originally, I want to optimize my code, figure out which operation is time consuming, to make my code run more fast.</p>
<p>This is my original code. It’s just some feature extracted to be inferred by model, and then compute some distances of the embeddings. It works with GPU.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/6/360f71da316a8f9bab2e3a2ffeccba9426292815.png" data-download-href="https://discuss.pytorch.org/uploads/default/360f71da316a8f9bab2e3a2ffeccba9426292815" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/360f71da316a8f9bab2e3a2ffeccba9426292815_2_690x421.png" alt="image" data-base62-sha1="7IeSa6MW9QuEdWJnyoNzupuud8x" width="690" height="421" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/360f71da316a8f9bab2e3a2ffeccba9426292815_2_690x421.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/360f71da316a8f9bab2e3a2ffeccba9426292815_2_1035x631.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/6/360f71da316a8f9bab2e3a2ffeccba9426292815.png 2x" data-dominant-color="272727"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1277×780 77.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I add time called almost each line to show the time consuming. kind of crazy…<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/9/59d3f17f467eb4bae57b6789ddd4701c8262015e.png" data-download-href="https://discuss.pytorch.org/uploads/default/59d3f17f467eb4bae57b6789ddd4701c8262015e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/59d3f17f467eb4bae57b6789ddd4701c8262015e_2_561x499.png" alt="image" data-base62-sha1="cOEA2TMxUowQGJTfepbCFccKzMy" width="561" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/59d3f17f467eb4bae57b6789ddd4701c8262015e_2_561x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/59d3f17f467eb4bae57b6789ddd4701c8262015e_2_841x748.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/59d3f17f467eb4bae57b6789ddd4701c8262015e_2_1122x998.png 2x" data-dominant-color="2C2625"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1302×1160 107 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>There is a 9 times loop in my code. It shows that this line（blue box）of the 1st loop ost most of the time.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2cfeaa8137f9e3a66cd9a2782c8bb55b7603201b.png" data-download-href="https://discuss.pytorch.org/uploads/default/2cfeaa8137f9e3a66cd9a2782c8bb55b7603201b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2cfeaa8137f9e3a66cd9a2782c8bb55b7603201b_2_563x500.png" alt="image" data-base62-sha1="6q2CWEwpWDlxsTX8MeEhNqTuZd1" width="563" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2cfeaa8137f9e3a66cd9a2782c8bb55b7603201b_2_563x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2cfeaa8137f9e3a66cd9a2782c8bb55b7603201b_2_844x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2cfeaa8137f9e3a66cd9a2782c8bb55b7603201b_2_1126x1000.png 2x" data-dominant-color="262727"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1287×1142 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The time is show as follow（red box).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/befca27ab392f9f514fcda1d57f96fb4bda31f78.png" data-download-href="https://discuss.pytorch.org/uploads/default/befca27ab392f9f514fcda1d57f96fb4bda31f78" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/befca27ab392f9f514fcda1d57f96fb4bda31f78.png" alt="image" data-base62-sha1="rfxZaSvMve0EejtXPqNySyWSdte" width="690" height="119" data-dominant-color="F0EFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1356×235 5.61 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I change above blue box line as the blue box line as follow. That is</p>
<pre><code class="lang-auto">positive_embedding = anchor_positive_embedding[enroll_idx].mean(dim=0)
</code></pre>
<p>to</p>
<pre><code class="lang-auto">positive_embedding = anchor_positive_embedding[2]
</code></pre>
<p>Now the code like this, just the blue box code changed.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1ce38b31b7d6e11990a6cdcb7c4b5353932ecdf8.png" data-download-href="https://discuss.pytorch.org/uploads/default/1ce38b31b7d6e11990a6cdcb7c4b5353932ecdf8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1ce38b31b7d6e11990a6cdcb7c4b5353932ecdf8_2_571x500.png" alt="image" data-base62-sha1="47ySwg8KnPf1bVNJ6KFnmcTzK9W" width="571" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1ce38b31b7d6e11990a6cdcb7c4b5353932ecdf8_2_571x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1ce38b31b7d6e11990a6cdcb7c4b5353932ecdf8_2_856x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1ce38b31b7d6e11990a6cdcb7c4b5353932ecdf8_2_1142x1000.png 2x" data-dominant-color="262727"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1303×1140 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
However, the most time consuming operation changes.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/3/33b187fd86bf4e2cdc5e4b62c7f123a247e03ad9.png" data-download-href="https://discuss.pytorch.org/uploads/default/33b187fd86bf4e2cdc5e4b62c7f123a247e03ad9" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/3/33b187fd86bf4e2cdc5e4b62c7f123a247e03ad9.png" alt="image" data-base62-sha1="7niI0HK5BNfVA9xhNK41FLkByCB" width="690" height="116" data-dominant-color="F0EFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1380×233 5.56 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
That is：</p>
<pre><code class="lang-auto">y_pred.extend(dists.cpu().tolist())
</code></pre>
<p>While this line doesn’t consume too much time before the blue box line changing.</p>
<p>It’s obviously odd. I use line_profiler to obtain time statistic. Almost the same.</p>
<p>Moreover, when I uncomment the</p>
<pre><code class="lang-auto">print(dists)
</code></pre>
<p>in my code, this line becomes the most time consuming, only in the 1st loop. So strange.</p>
<p>More infos about my environment:<br>
python3.7.9<br>
torch 1.4.0<br>
cuda 10.1<br>
docker environment</p>
<p>If need more infos, I will provide.</p>
<p>Is it a bug? Or which operation is the most time consuming in my code, in fact?</p>

ptrblck: <p>CUDA operations are executed asynchronously, so you would need to synchronize the code before starting and stopping the code manually via <code>torch.cuda.synchronize()</code> or use a built-in utility such as <code>torch.utils.benchmark</code> from the nightly/master, which will add it for you (as well as warmup iterations).</p>