Different runtime for the same FLOPs

Vinayaka_Hegde: <p>Hi, I am measuring the total number of Floating point operations and also the layer-wise and operation-wise runtime of Resnet18 using a tool.</p>
<p>The format of the output is as follows :<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba4d4ead91d8f07e18edbbaff5ffcf85cc3780e6.png" alt="image" data-base62-sha1="qA6rgetBhA1S185yPio20Dr2NL0" width="258" height="199"></p>
<p>The core operation of the tool is as follows :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/0/90a51e515938bad25b878071e491432f0d7c5c28.png" data-download-href="https://discuss.pytorch.org/uploads/default/90a51e515938bad25b878071e491432f0d7c5c28" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/90a51e515938bad25b878071e491432f0d7c5c28_2_690x463.png" alt="image" data-base62-sha1="kDAwgdcQZnOlJgsrcF9qD73BP60" width="690" height="463" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/90a51e515938bad25b878071e491432f0d7c5c28_2_690x463.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/90a51e515938bad25b878071e491432f0d7c5c28_2_1035x694.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/0/90a51e515938bad25b878071e491432f0d7c5c28.png 2x" data-dominant-color="252727"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1120×752 73.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Torch dispatcher intercepts the operation calls - like <code>aten.conv</code> and <code>aten.conv_backward</code> and I have used a custom function that calculates FLOPs based on the type of the operator and dimensions of the operator. I also profile the per-operator run time for all the layers using <code>torch.cuda.Event</code> and append both the FLOPs and runtime of every <code>Conv</code> and <code>FCL</code> into a dictionary.</p>
<p>I have done a warmup for 30 iterations and then have profiled for the following variants (<code>all with batch_size as 1</code>)</p>
<ol>
<li>single randn input</li>
<li>single image input (from imagenet dataset)</li>
<li>100 images input</li>
</ol>
<p>Output when Resnet18 is fed with a single input image from Imagenet is as follows :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a3b55758895f7b9a2c601376db6537baefee0f3.png" data-download-href="https://discuss.pytorch.org/uploads/default/3a3b55758895f7b9a2c601376db6537baefee0f3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a3b55758895f7b9a2c601376db6537baefee0f3_2_394x499.png" alt="image" data-base62-sha1="8j8OytcSdOdVTWQqJMCqMh2tN0n" width="394" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a3b55758895f7b9a2c601376db6537baefee0f3_2_394x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a3b55758895f7b9a2c601376db6537baefee0f3_2_591x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a3b55758895f7b9a2c601376db6537baefee0f3.png 2x" data-dominant-color="242525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">688×871 40.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c35ba405ddff7230e526008c203166ce3df0c2e.png" data-download-href="https://discuss.pytorch.org/uploads/default/9c35ba405ddff7230e526008c203166ce3df0c2e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c35ba405ddff7230e526008c203166ce3df0c2e_2_420x500.png" alt="image" data-base62-sha1="mhTAXuWyUeJq7dW7GPYEy0tUx8G" width="420" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c35ba405ddff7230e526008c203166ce3df0c2e_2_420x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c35ba405ddff7230e526008c203166ce3df0c2e_2_630x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c35ba405ddff7230e526008c203166ce3df0c2e.png 2x" data-dominant-color="232524"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">724×861 38.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And the main driver code is as follows :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ecd8b9b9f89d1ad3db9c7f7832557551b5877ff6.png" data-download-href="https://discuss.pytorch.org/uploads/default/ecd8b9b9f89d1ad3db9c7f7832557551b5877ff6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ecd8b9b9f89d1ad3db9c7f7832557551b5877ff6.png" alt="image" data-base62-sha1="xNf0WDL4AcbzDJPmxWr02WoTUp0" width="690" height="274" data-dominant-color="272929"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">785×312 25.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For all the input variants, I’m facing the below issues :</p>
<ol>
<li>
<p>Unusual increase in runtime for operations in different layers when the FLOPs are same (Ex for layer4 the runtime is greater than other layers)</p>
</li>
<li>
<p>In some cases, runtime for an operation for Forward pass takes more time than that of Backward pass (Ex for conv1 layer in the above output)</p>
</li>
</ol>
<p>It would be great if someone can reason why there is variation in runtime even when FLOPs are the same</p>