Why is AvgPool2D taking longer than Conv2D for the same input?
Labels: module: performance, module: cpu, triaged

### 🐛 Describe the bug

I could be missing something obvious, but when I feed the same input tensor (with shape (20, 1, 28, 28)) to both AvgPool2D and Conv2D, I notice that AvgPool2D takes 56% of the running time and Conv2D takes 39% of the running time. i.e. AvgPool2D is taking 143% of the time that Conv2D is taking for the same input tensor.

Please find the reproduction here: https://colab.research.google.com/drive/1qu7Y9qe6yts7Dp4W4eL8j0Fe-5sgXQvm

Here's the code that can reproduce it (mentioned inline for readability - is the same as in the colab, which also has the output/results).

```
import torch
from torch import nn
from torch.profiler import profile, record_function, ProfilerActivity

with torch.inference_mode():

  avgpool2d = nn.AvgPool2d(3, stride=1)
  conv2d = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=3, padding=0)

  cv2d_input = torch.randn((20, 1, 28, 28))
  ap2d_input = torch.randn((20, 1, 28, 28)) # conv2d(cv2d_input)
  print("Shape of Conv2d Input: {}".format(cv2d_input.shape))
  print("Shape of AvgPool2D Input: {}".format(ap2d_input.shape))

  with profile(activities=[ProfilerActivity.CPU], profile_memory=True, record_shapes=True) as prof:
    for _ in range(1000):
      avgpool2d(ap2d_input)
      conv2d(cv2d_input)

  print(prof.key_averages().table(sort_by="cpu_time_total", row_limit=20))
```

Output:

```
Shape of Conv2d Input: torch.Size([20, 1, 28, 28])
Shape of AvgPool2D Input: torch.Size([20, 1, 28, 28])
----------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                        Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg       CPU Mem  Self CPU Mem    # of Calls  
----------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
            aten::avg_pool2d        56.80%     540.816ms        56.80%     540.816ms     540.816us      51.57 Mb      51.57 Mb          1000  
                aten::conv2d         1.11%      10.533ms        42.40%     403.724ms     403.724us      51.57 Mb     211.25 Kb          1000  
           aten::convolution         1.03%       9.773ms        41.54%     395.539ms     395.539us      51.57 Mb       1.03 Mb          1000  
          aten::_convolution         1.09%      10.391ms        41.00%     390.432ms     390.432us      51.57 Mb     211.25 Kb          1000  
    aten::mkldnn_convolution        39.20%     373.237ms        39.98%     380.677ms     380.677us      51.57 Mb       5.93 Mb          1000  
                 aten::empty         0.60%       5.713ms         0.60%       5.713ms       2.857us      51.42 Mb      51.42 Mb          2000  
           aten::as_strided_         0.18%       1.727ms         0.18%       1.727ms       1.727us      -5.78 Mb      -5.78 Mb          1000  
                    [memory]         0.00%       0.000us         0.00%       0.000us       0.000us    -103.15 Mb    -103.15 Mb          2000  
----------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 952.190ms
```


### Versions

1.13.1+cu116


cc @ngimel @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10