Inconsistent performance degradation of 3x3 convolution (torch 2.0.1+cu118)
Labels: module: performance, module: cudnn, module: cuda, triaged, module: regression

### 🐛 Describe the bug

When using `torch 2.0.1+cu118`, nn.Conv2d` slows down significantly compare to previous versions when,

- `in_channels` is around 5 ~ 6,
- `out_channels` is less than 5, and
- `kernel_size` is 3.

Here is an example snippet. I also posted a full benchmark script to my gist ([torch_conv_bench.py](https://gist.github.com/daitakahashi/3796d2f9ef8d9976b66fe28a2f73aec4)).
(note: following the reply, I updated both gist and this snipped to use `torch.cuda.synchronize()` instead of implicit synchronization. All benchmark values are also updated)
```python
import torch
import time
def bench(t, net):
    s = 0
    _tm = 0
    for _ in range(1000):
        tcuda = t.to('cuda')
        torch.cuda.synchronize(device='cuda')
        t0 = time.time()
        output = net(tcuda)
        torch.cuda.synchronize(device='cuda')
        _tm += time.time() - t0
        del tcuda, output
    return _tm

t = torch.rand((32, 5, 256, 256), dtype=torch.float32)
net = torch.nn.Conv2d(in_channels=5, out_channels=3, kernel_size=3)
net.to('cuda',  dtype=torch.float32)
t33 = bench(t, net)

net = torch.nn.Conv2d(in_channels=5, out_channels=3, kernel_size=5)
net.to('cuda',  dtype=torch.float32)
t55 = bench(t, net)
print(f'3x3 conv: {t33} sec')
print(f'5x5 conv: {t55} sec')
```

Here, I got following output from the above snippet. I expected that 3x3 convolution is slightly faster than 5x5 convolution, but it wasn't.
```
3x3 conv: 2.52791428565979 sec
5x5 conv: 0.48103857040405273 sec
```

Further investigation shows large performance drop of particular combinations of `input_channels`, `output_channels`, `kernel_size`, and sizes of input images (using the benchmark script mentioned above).
```
ksize = 3               32x32     64x64    128x128    256x256    512x512                                                 
-------------------  --------  --------  ---------  ---------  ---------
input channels = 1   33182.78  44201.75   24545.32   10968.94    3123.48
input channels = 2   20744.37  31105.78   26677.93   17649.08    5557.87
input channels = 3   33778.72  29220.45   22613.24   11869.78    3426.66
input channels = 4    5601.89  36399.41    1635.79     424.04     107.14
input channels = 5   13535.69  33803.22    1605.76     407.32     101.99
input channels = 6   13487.81  32671.01    1604.28     405.84     101.73
input channels = 7   12630.02  31555.10    1603.55     404.17     101.31
input channels = 8   24448.03  25338.63   15652.72    5564.58    1603.36
input channels = 9   23494.87  24206.75   14651.05    4750.06    1449.85
input channels = 10  23030.44  23641.87   13704.64    4347.19    1322.83

ksize = 5               32x32     64x64    128x128    256x256    512x512                                                 
-------------------  --------  --------  ---------  ---------  ---------
input channels = 1   41266.27  43873.47   27887.66   17051.40    5940.69
input channels = 2   38550.59  32192.06   22522.17   11380.56    3265.98
input channels = 3   35037.21  29987.16   19737.90    8577.48    2358.35
input channels = 4   32110.73  27921.08   17161.64    6840.58    1846.33
input channels = 5   30548.46  26225.87   15369.94    5661.70    1509.87
input channels = 6   28692.73  25061.57   13826.16    4838.33    1285.15
input channels = 7   23937.36  23960.61   12576.25    4184.26    1118.09
input channels = 8   22090.40  22103.20   11404.07    3707.90     987.81
input channels = 9   21284.40  21384.24   10553.04    3186.89     885.77
input channels = 10  20585.54  20573.42    9701.63    2919.36     802.35

Varying the number of input/output channels (iter/sec): ksize = 3, image size = 128                                      
  #in \ #out         1         2         3         4         5         6         7         8         9        10
------------  --------  --------  --------  --------  --------  --------  --------  --------  --------  --------
           1  28311.20  27228.67  23465.95  25141.19  19142.46   9680.35  17146.20  17596.51  17885.40  17655.02
           2  40591.35  22569.44  16899.57  17445.74  12961.38  16267.71  16226.80  16005.13  16102.21  16021.64
           3  31669.47  17316.09  14334.11  11955.72  10367.31  14547.39  14746.87  14420.35  14772.32  14643.89
           4   1649.70   1634.17   1630.71   1662.88   1623.31  13391.78  13476.54  13190.05  13540.06  13503.01
           5   1619.71   1601.55   1600.76   1636.16   1588.84  12037.38  12226.87  12189.20  12336.55  12262.25
           6   1618.65   1601.30   1601.04   1636.97   1584.05  11234.84  11405.93  11334.43  11519.65  11478.35
           7   1617.09   1599.51   1598.28   1633.56   1581.09  10471.88  10646.52  10653.28  10765.11  10696.75
           8  19439.67  10382.97   7811.06   6283.41   5261.95   9858.05  10022.47   9893.39  10066.49   9971.72
           9  18115.60   9504.00   7176.25   5720.23   4805.52   7349.27   7364.63   7387.59   7478.74   7464.10
          10  16871.02   8906.05   6638.66   5267.31   4401.30   7418.03   7424.47   7338.60   7425.12   7443.31
```

I observed the performance degradation on RTX 4090 and 4080. So far I only see the issue when I use `torch 2.0.1+cu118`. Both `torch 2.0.1` from pypi.org (using cuda 11.7) and `torch 1.13.1` (also from pypi.org) do not show this issue.

Thank you very much in advance.

### Versions

Collecting environment information...
PyTorch version: 2.0.1+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.2 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: version 3.25.0
Libc version: glibc-2.35

Python version: 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-5.19.0-41-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 4090
Nvidia driver version: 530.30.02
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   48 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          32
On-line CPU(s) list:             0-31
Vendor ID:                       AuthenticAMD
Model name:                      AMD Ryzen 9 7950X 16-Core Processor
CPU family:                      25
Model:                           97
Thread(s) per core:              2
Core(s) per socket:              16
Socket(s):                       1
Stepping:                        2
Frequency boost:                 enabled
CPU max MHz:                     5879.8818
CPU min MHz:                     3000.0000
BogoMIPS:                        8983.28
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d
Virtualization:                  AMD-V
L1d cache:                       512 KiB (16 instances)
L1i cache:                       512 KiB (16 instances)
L2 cache:                        16 MiB (16 instances)
L3 cache:                        64 MiB (2 instances)
NUMA node(s):                    1
NUMA node0 CPU(s):               0-31
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==2.0.1+cu118
[pip3] torchaudio==2.0.2+cu118
[pip3] torchinfo==1.7.2
[pip3] torchvision==0.15.2+cu118
[pip3] triton==2.0.0
[conda] Could not collect

cc @ngimel @csarofeen @ptrblck @xwang233