Difference in inference time betweeen resnet50 from github and torchvision code

11179: <p>Hello, I’m trying to compare how inference time getting faster by reducing FLOPs through changing input sizes.<br>
However, torchvision resnet 50 does not seems to get faster by size reduction.<br>
So I test on github (address below) resnet 50 that gets faster by smaller input.</p>
<p>What makes the difference?<br>
(Besides time takes almost 4 times bigger in github code on same 300x300 size)</p>
<p>x-axis showing the length of the input, test by Tensor shape (1, 3, x , x)<br>
y-axis showing inference time<br>
The line with orange means a theoretical possible reduction by FLOPs change.<br>
The blue line is experimental num.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9e21a20f0104672507e074a1abc17dd0d0b5d649.png" alt="image" data-base62-sha1="myTuVlBjxHN13QYoS9VxWZZQvlD" width="274" height="177"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/9/99ab21a15b033511443f5148b5f90461161bab87.png" alt="image" data-base62-sha1="lVpHpJQKaoHqJMUDvaNXYRPxXzp" width="277" height="190"></p>
<ol>
<li>github code (with no Fully connected layer and avgpool layer)</li>
<li>torchvsion code (with changing Fully connected layer and avgpool layer to Idenetity layer)</li>
</ol>
<aside class="onebox githubblob" data-onebox-src="https://github.com/kuangliu/pytorch-cifar/blob/master/models/resnet.py">
  <header class="source">

      <a href="https://github.com/kuangliu/pytorch-cifar/blob/master/models/resnet.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/kuangliu/pytorch-cifar/blob/master/models/resnet.py" target="_blank" rel="noopener nofollow ugc">kuangliu/pytorch-cifar/blob/master/models/resnet.py</a></h4>


      <pre><code class="lang-py">'''ResNet in PyTorch.

For Pre-activation ResNet, see 'preact_resnet.py'.

Reference:
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
    Deep Residual Learning for Image Recognition. arXiv:1512.03385
'''
import torch
import torch.nn as nn
import torch.nn.functional as F


class BasicBlock(nn.Module):
    expansion = 1

    def __init__(self, in_planes, planes, stride=1):
        super(BasicBlock, self).__init__()
        self.conv1 = nn.Conv2d(
            in_planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
</code></pre>



  This file has been truncated. <a href="https://github.com/kuangliu/pytorch-cifar/blob/master/models/resnet.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>Could you disable cudnn via <code>torch.backends.cudnn.enabled = False</code> and check, if the experimental speed would better fit the theoretical one?</p>

11179: <p>Whole code, directly can run on colab notebook.</p>
<pre><code class="lang-auto">import torchvision
import numpy as np
import torch
import torch.nn as nn
import matplotlib.pyplot as plt

# torch.backends.cudnn.benchmark = False


model = torchvision.models.resnet50(pretrained= False)
model.avgpool = nn.Identity()
model.fc = nn.Identity()
# model = ResNet50()

x_axis = range(50,301, 10)
all_t_resnet =[]
model = model.cuda().eval()
with torch.no_grad():
    for i in x_axis:
        t = []
        for _ in range(100):
            start = torch.cuda.Event(enable_timing=True)
            end = torch.cuda.Event(enable_timing=True) 
            size = [i, i]
            x= torch.randn(1,3,size[0],size[1]).cuda()
            torch.cuda.synchronize()
            start.record()  
            _ = model(x)
            end.record()
            torch.cuda.synchronize()
            measure_t = start.elapsed_time(end)
            t.append(measure_t)
        all_t_resnet.append(np.mean(t))


theoritic_resnet = (np.array(x_axis)/x_axis[-1])**2*all_t_resnet[-1]


# plt.gca().set_color_cycle(['red', 'green'])

plt.plot(x_axis, all_t_resnet)
plt.plot(x_axis, theoritic_resnet)

plt.legend(['experiment_resnet50', 'theoritical_resnet50'], loc='upper left')

plt.show()
</code></pre>

11179: <p>Thank you for replying <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>
<p>Yes, I did it, but it does not change the result. Below is the result run on colab.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/9/099364eda613cbe1cef8219e2700d54ca1ca26f6.png" alt="image" data-base62-sha1="1mI5s7xJFgd9jxKuWdubsvP8wVo" width="320" height="210"></p>
<p>However, as the input size gets bigger, it seems to converge to the experimental line.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e8f10af7ddd67a87460adfa901043f19633066a.png" alt="image" data-base62-sha1="6DStIHoDEC5b5NAWzdX6rIkY7YC" width="330" height="212"></p>
<p>(Post images aer run on RTX 2080ti, while this reply images are on colab Tesla P100)</p>

ptrblck: <p>Thanks for the update!<br>
The plateau might be hit due to a constant overhead in the kernel launches, dispatching mechanism etc.</p>

11179: <p>Oh, I see. Thanks.<br>
I will try to find overhead and reason. Thank you. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>