ncu python conv2d.py runs indefinitely after activating cudnn.benchmark
Labels: module: cudnn, module: cuda, triaged

### 🐛 Describe the bug

I was trying to see how cudnn lib selects the most performant algorithm for depthwise convolution, so I used ncu to display the candidates perfomance, but unfortunately the kernels kept running indefinitely. Here is the script I used

```python
import torch

torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False

[N, C, H, W] = [2, 512, 64, 128]
kernel_size = (3, 3)
stride = 1
perf_repeat = 1
bias = False
dtype = torch.float

memory_format = torch.channels_last

input_1 = torch.ones(N, C, H, W, dtype = dtype).to("cuda")
if memory_format == torch.channels_last:
    input_1 = input_1.to(memory_format = memory_format)

dwconv_layer = torch.nn.Conv2d(in_channels = C, out_channels = C, kernel_size = kernel_size,
                            stride = stride, dilation = 1, padding = 1, groups = C, bias = bias, dtype = dtype).to("cuda")

if memory_format == torch.channels_last:
    dwconv_layer = dwconv_layer.to(memory_format = memory_format)

for i in range(perf_repeat):
    output = dwconv_layer(input_1)
```
the command I used is `ncu python the_above_script.py`, and the profiling output is

> ==PROF== Connected to process 2373 (/usr/bin/python3.10)
==PROF== Profiling "elementwise_kernel" - 0: 0%....50%....100% - 10 passes
==PROF== Profiling "conv2d_c1_k1_nhwc_kernel" - 1: 0%....50%....100% - 10 passes
==PROF== Profiling "conv2d_c1_k1_nhwc_kernel" - 2: 0%....50%....100% - 10 passes
==PROF== Profiling "conv2d_grouped_direct_kernel" - 3: 0%....50%....100% - 10 passes
==PROF== Profiling "conv2d_grouped_direct_kernel" - 4: 0%....50%....100% - 10 passes
==PROF== Profiling "nhwcToNchwKernel" - 5: 0%....50%....100% - 10 passes
==PROF== Profiling "kern_precompute_indices" - 6: 0%....50%....100% - 10 passes
==PROF== Profiling "precomputed_convolve_sgemm" - 7: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 8: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 9: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 10: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 11: 0%....50%....100% - 10 passes
==PROF== Profiling "nhwcToNchwKernel" - 12: 0%....50%....100% - 10 passes
==PROF== Profiling "kern_precompute_indices" - 13: 0%....50%....100% - 10 passes
==PROF== Profiling "precomputed_convolve_sgemm" - 14: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 15: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 16: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 17: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 18: 0%....50%....100% - 10 passes
==PROF== Profiling "nhwcToNchwKernel" - 19: 0%....50%....100% - 10 passes
==PROF== Profiling "kern_precompute_indices" - 20: 0%....50%....100% - 10 passes
==PROF== Profiling "precomputed_convolve_sgemm" - 21: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 22: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 23: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 24: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 25: 0%....50%....100% - 10 passes
==PROF== Profiling "nhwcToNchwKernel" - 26: 0%....50%....100% - 10 passes
==PROF== Profiling "kern_precompute_indices" - 27: 0%....50%....100% - 10 passes
==PROF== Profiling "precomputed_convolve_sgemm" - 28: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 29: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 30: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 31: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 32: 0%....50%....100% - 10 passes
==PROF== Profiling "nhwcToNchwKernel" - 33: 0%....50%....100% - 10 passes
==PROF== Profiling "kern_precompute_indices" - 34: 0%....50%....100% - 10 passes
==PROF== Profiling "precomputed_convolve_sgemm" - 35: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 36: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 37: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 38: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 39: 0%....50%....100% - 10 passes
==PROF== Profiling "nhwcToNchwKernel" - 40: 0%....50%....100% - 10 passes
==PROF== Profiling "kern_precompute_indices" - 41: 0%....50%....100% - 10 passes
==PROF== Profiling "precomputed_convolve_sgemm" - 42: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 43: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 44: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 45: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 46: 0%....50%....100% - 10 passes
==PROF== Profiling "nhwcToNchwKernel" - 47: 0%....50%....100% - 10 passes
==PROF== Profiling "kern_precompute_indices" - 48: 0%....50%....100% - 10 passes
==PROF== Profiling "precomputed_convolve_sgemm" - 49: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 50: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 51: 0%....50%....100% - 10 passes
==PROF== Profiling "implicit_convolve_sgemm" - 52: ^C0%....50%...==PROF== Received signal
==PROF== Trying to shutdown target application

It didn't seem to gonna stop, so I had to kill it in the end. 

But without ncu，i.e. just `python the_above_script.py`, it finishes in a flash.

The environment I ran the test is nvcr.io/nvidia/pytorch:23.08-py3.

### Versions

Collecting environment information...
PyTorch version: 2.1.0a0+29c30b1
Is debug build: False
CUDA used to build PyTorch: 12.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.2 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.27.1
Libc version: glibc-2.35

Python version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-150-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.2.128
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A100-PCIE-40GB
Nvidia driver version: 535.104.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   43 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          48
On-line CPU(s) list:             0-47
Vendor ID:                       AuthenticAMD
Model name:                      AMD EPYC 7402P 24-Core Processor
CPU family:                      23
Model:                           49
Thread(s) per core:              2
Core(s) per socket:              24
Socket(s):                       1
Stepping:                        0
Frequency boost:                 enabled
CPU max MHz:                     2800.0000
CPU min MHz:                     1500.0000
BogoMIPS:                        5589.71
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca
Virtualization:                  AMD-V
L1d cache:                       768 KiB (24 instances)
L1i cache:                       768 KiB (24 instances)
L2 cache:                        12 MiB (24 instances)
L3 cache:                        128 MiB (8 instances)
NUMA node(s):                    1
NUMA node0 CPU(s):               0-47
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Vulnerable
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] numpy==1.22.2
[pip3] onnx==1.14.0
[pip3] pytorch-quantization==2.1.2
[pip3] torch==2.1.0a0+29c30b1
[pip3] torch-tensorrt==2.0.0.dev0
[pip3] torchdata==0.7.0a0
[pip3] torchtext==0.16.0a0
[pip3] torchvision==0.16.0a0
[pip3] triton==2.1.0+440fd1b
[conda] Could not collect

cc @csarofeen @ptrblck @xwang233