Unexpected PyTorch Profiler results

naokishibuya: <p>Hello,</p>
<p>I ran the PyTorch profiler to measure the total CPU times for the following models:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Model name</th>
<th>CPU total time</th>
</tr>
</thead>
<tbody>
<tr>
<td>ResNet18</td>
<td>37.236ms</td>
</tr>
<tr>
<td>ProxylessNAS (CPU)</td>
<td>69.561ms</td>
</tr>
<tr>
<td>MobileNetV2</td>
<td>64.838ms</td>
</tr>
</tbody>
</table>
</div><p>The machine has 14-cores Intel Core-i9 10940X CPU @ 3.30GHz.</p>
<p>Both ProxylessNAS (CPU) and MobileNetV2 are slower than ResNet18.</p>
<p>But I was expecting the opposite.</p>
<p>The script is as follows.  Do I miss anything?</p>
<pre><code class="lang-auto">import numpy as np
import torch
from torch.autograd import profiler


#model = torch.hub.load('pytorch/vision:v0.6.0', 'resnet18', pretrained=True)
#model = torch.hub.load('mit-han-lab/ProxylessNAS', 'proxyless_cpu', pretrained=True)
model = torch.hub.load('pytorch/vision:v0.6.0', 'mobilenet_v2', pretrained=True) #models.mobilenetv2()

inputs = torch.randn((1, 3, 224, 224))

with torch.no_grad():
    with profiler.profile(record_shapes=True) as prof:
        with profiler.record_function('model_inference'):                                                                                                                                                                                                     
            model(inputs)

print(prof.key_averages().table(sort_by='cpu_time_total', row_limit=10))                                                                                                                                                                                      
</code></pre>
<p>The detailed outputs are as follows:</p>
<p><strong>ResNet18</strong></p>
<pre><code class="lang-auto">---------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------                                                                                                                                               
Name                         Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     Number of Calls                                                                                                                                              
---------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------                                                                                                                                               
model_inference              6.02%            2.241ms          100.00%          37.236ms         37.236ms         1                                                                                                                                                             
conv2d                       0.17%            63.016us         46.30%           17.240ms         861.987us        20                                                                                                                                                            
convolution                  0.20%            75.310us         46.13%           17.177ms         858.836us        20                                                                                                                                                            
_convolution                 0.69%            256.361us        45.93%           17.101ms         855.071us        20                                                                                                                                                            
mkldnn_convolution           44.84%           16.696ms         45.14%           16.810ms         840.478us        20                                                                                                                                                            
batch_norm                   0.21%            78.296us         31.22%           11.626ms         581.318us        20                                                                                                                                                            
_batch_norm_impl_index       0.22%            80.086us         31.01%           11.548ms         577.404us        20                                                                                                                                                            
native_batch_norm            18.85%           7.018ms          30.71%           11.436ms         571.781us        20                                                                                                                                                            
max_pool2d                   0.05%            17.052us         12.65%           4.709ms          4.709ms          1                                                                                                                                                             
max_pool2d_with_indices      12.58%           4.684ms          12.60%           4.692ms          4.692ms          1                                                                                                                                                             
---------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------                                                                                                                                               
Self CPU time total: 37.236ms                                                          
</code></pre>
<p><strong>ProxylessNAS (CPU)</strong></p>
<pre><code class="lang-auto">--------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
Name                        Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     Number of Calls  
--------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
model_inference             8.69%            6.046ms          100.00%          69.561ms         69.561ms         1                
conv2d                      0.20%            140.364us        44.30%           30.818ms         505.213us        61               
convolution                 0.20%            137.516us        44.10%           30.678ms         502.912us        61               
_convolution                1.09%            754.904us        43.90%           30.540ms         500.657us        61               
batch_norm                  0.29%            203.942us        43.69%           30.394ms         498.257us        61               
_batch_norm_impl_index      0.31%            214.853us        43.40%           30.190ms         494.914us        61               
native_batch_norm           23.48%           16.333ms         42.96%           29.885ms         489.923us        61               
mkldnn_convolution          42.20%           29.355ms         42.66%           29.678ms         486.520us        61               
select                      13.17%           9.160ms          17.69%           12.303ms         3.191us          3855             
as_strided                  3.23%            2.247ms          3.23%            2.247ms          0.582us          3858             
--------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
Self CPU time total: 69.561ms
</code></pre>
<p><strong>MobileNetV2</strong></p>
<pre><code class="lang-auto">--------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
Name                        Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     Number of Calls  
--------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
model_inference             7.19%            4.662ms          100.00%          64.838ms         64.838ms         1                
batch_norm                  0.27%            174.627us        50.49%           32.736ms         629.534us        52               
_batch_norm_impl_index      0.29%            186.420us        50.22%           32.561ms         626.176us        52               
native_batch_norm           27.76%           18.001ms         49.82%           32.301ms         621.170us        52               
conv2d                      0.18%            118.212us        38.70%           25.091ms         482.519us        52               
convolution                 0.22%            144.360us        38.52%           24.973ms         480.245us        52               
_convolution                1.23%            800.032us        38.29%           24.828ms         477.469us        52               
mkldnn_convolution          36.46%           23.642ms         36.91%           23.934ms         460.270us        52               
select                      15.08%           9.777ms          20.58%           13.342ms         3.572us          3735             
as_strided                  3.94%            2.553ms          3.94%            2.553ms          0.683us          3738             
--------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
Self CPU time total: 64.838ms
</code></pre>