The results of torch.profiler() and time.time() do not match

dujiangsu: <p><strong>When training a 3D CNN model on a 4-tesla-v100 node with mixed precision, I got a wired result:</strong></p>
<p>The mixed-precision (O1 &amp;&amp; O3) result is slower than the FP32 result when batch size is 1 using time.time() for recording the execution time.</p>
<p>Using the torch.profiler(), it displays that the mixed-precision indeed speeds up the training in terms of the CPU time and the CUDA time.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/40a4db2e2ec2e62afd8f78374c15f66b637dab22.png" data-download-href="https://discuss.pytorch.org/uploads/default/40a4db2e2ec2e62afd8f78374c15f66b637dab22" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/0/40a4db2e2ec2e62afd8f78374c15f66b637dab22.png" alt="image" data-base62-sha1="9dRLaDqaLPeCicY3xehi6QPIVa2" width="690" height="58" data-dominant-color="1B0B0A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1652×140 6.83 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Notably, the problem only exists when batch size is equal to 1 (batch size = 4 is accelerated as predicted) and I tried two scales of the 3D CNN models. (The large model can only be trained with batch size =1 for it is too large.)</p>
<p><strong>Question:</strong><br>
It seems that there exists a large portion of the execution time that is not related to computing.<br>
Do you have any idea about it?<br>
Why the total execution of 3D CNN in mixed precision is slower than the FP32 when batch size =1？</p>
<p><strong>Env:</strong><br>
apex:0.1<br>
pytorch:1.5 &amp;&amp; 1.3<br>
hardware: DGX</p>

ptrblck: <p>If you are manually timing CUDA operations, you would need to synchronize the code before starting and stopping the timer via <code>torch.cuda.synchronize()</code>.<br>
Also, the first CUDA operation will create the CUDA context etc. and will be slower than the following calls, so you should add some warmup iterations.</p>

dujiangsu: <p>Thanks.<br>
I have used torch.cuda.synchronize(), and there is about hundreds of iterations in one epoch with the same problem.<br>
I will provide you the detailed results soon.</p>

dujiangsu: <p><strong>FP32 training without apex:</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fc469724c578d76593b024001f912aff9204c9f7.png" data-download-href="https://discuss.pytorch.org/uploads/default/fc469724c578d76593b024001f912aff9204c9f7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc469724c578d76593b024001f912aff9204c9f7_2_690x488.png" alt="image" data-base62-sha1="zZJyPycfmn3cDeiLljr6xpTrKiH" width="690" height="488" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc469724c578d76593b024001f912aff9204c9f7_2_690x488.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc469724c578d76593b024001f912aff9204c9f7_2_1035x732.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc469724c578d76593b024001f912aff9204c9f7_2_1380x976.png 2x" data-dominant-color="353630"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2358×1671 619 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>FP32 training with apex O0:</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/e/feb24d5e87f48d28dfac05a286ade69408310c4f.png" data-download-href="https://discuss.pytorch.org/uploads/default/feb24d5e87f48d28dfac05a286ade69408310c4f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/feb24d5e87f48d28dfac05a286ade69408310c4f_2_690x378.png" alt="image" data-base62-sha1="Al9hS1cOJJwRkn3UjDGeYrqE795" width="690" height="378" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/feb24d5e87f48d28dfac05a286ade69408310c4f_2_690x378.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/feb24d5e87f48d28dfac05a286ade69408310c4f_2_1035x567.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/feb24d5e87f48d28dfac05a286ade69408310c4f_2_1380x756.png 2x" data-dominant-color="353630"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1684×924 149 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/0/809603cdb00a85fa9382a45864d87e3997ad1644.png" data-download-href="https://discuss.pytorch.org/uploads/default/809603cdb00a85fa9382a45864d87e3997ad1644" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/809603cdb00a85fa9382a45864d87e3997ad1644_2_690x348.png" alt="image" data-base62-sha1="ilwwkU09QRbJ3v9lNeLu7wu4aj2" width="690" height="348" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/809603cdb00a85fa9382a45864d87e3997ad1644_2_690x348.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/809603cdb00a85fa9382a45864d87e3997ad1644_2_1035x522.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/809603cdb00a85fa9382a45864d87e3997ad1644_2_1380x696.png 2x" data-dominant-color="33342E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2383×1204 377 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>mixed precision training with apex O1:</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7d64eca98541903d429bb32419b516dc43f737f.png" data-download-href="https://discuss.pytorch.org/uploads/default/b7d64eca98541903d429bb32419b516dc43f737f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7d64eca98541903d429bb32419b516dc43f737f_2_690x444.png" alt="image" data-base62-sha1="qeiwOYE7CufbHOKSDVQMS0om9oX" width="690" height="444" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7d64eca98541903d429bb32419b516dc43f737f_2_690x444.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7d64eca98541903d429bb32419b516dc43f737f_2_1035x666.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7d64eca98541903d429bb32419b516dc43f737f_2_1380x888.png 2x" data-dominant-color="363732"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1695×1093 190 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96e7417b0fc3f302ca9a1f763dadd06cdf6106da.png" data-download-href="https://discuss.pytorch.org/uploads/default/96e7417b0fc3f302ca9a1f763dadd06cdf6106da" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96e7417b0fc3f302ca9a1f763dadd06cdf6106da_2_690x395.png" alt="image" data-base62-sha1="lwX5c7MBORV7Ns01Mfytsq9KViW" width="690" height="395" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96e7417b0fc3f302ca9a1f763dadd06cdf6106da_2_690x395.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96e7417b0fc3f302ca9a1f763dadd06cdf6106da_2_1035x592.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96e7417b0fc3f302ca9a1f763dadd06cdf6106da_2_1380x790.png 2x" data-dominant-color="34352F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2385×1368 483 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Both the forward and backward passes seem to see a speedup between <code>O0</code> and <code>O1</code>.<br>
Are you seeing that the complete training time in <code>O1</code> is still higher than in <code>O0</code>?</p>

jerryzh168: <p>this is not related to quantization, can we have a mixed recision/amp tag?</p>

ptrblck: <p>That sounds like a good idea!</p>

dujiangsu: <p>The main concern is that these two kinds of profiling methods produce very different results.</p>

AnushaHegde: <p>Hello,</p>
<p>I am new to pyTorch profiling and facing a similar issue where torch.profile() is giving different total cpu time compared to time.perf_counter(). Could you please share if you had found the cause for this?</p>
<p>Thanks you!!</p>