PyTorch built with CuDNN-8.8.1 crashes if CuDNN-8.9.2 is installed on the system
Labels: module: cudnn, module: ci, triaged

### 🐛 Describe the bug

When PyTorch manywheels docker container was finally updated to 8.9.2 as part of https://github.com/pytorch/builder/pull/1436 but before PyPI dependence were updated https://github.com/pytorch/pytorch/pull/104757 basic attempts to use CuDNN (for example by running https://github.com/pytorch/builder/blob/main/test_example_code/cnn_smoke.py ) segfaulted:
```
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:
/opt/python/cp38-cp38/bin/python  /builder/test_example_code/cnn_smoke.py
Segmentation fault (core dumped)
```

And backtrace was exceedingly unhelpful:
```
* thread #1: tid = 789, 0x00007f3fe257c140 libcuda.so.525.105.17`??? + 272, name = 'python', stop reason = invalid address (fault address: 0x20000002ee0)
    frame #0: 0x00007f3fe257c140 libcuda.so.525.105.17`??? + 272
libcuda.so.525.105.17`??? + 272:
-> 0x7f3fe257c140:  movq   (%rsi), %rax
   0x7f3fe257c143:  testq  %rax, %rax
   0x7f3fe257c146:  je     0x7f3fe257c1d0            ; ??? + 416
   0x7f3fe257c14c:  movq   %rax, 0x18(%rsp)
```

But list of loaded shared libraries provided an insight, that it happens because some CuDNN libraries were loaded from 8.8(fetch from pypi) and some from 8.9(installed in /usr/local/cuda) cudnn (see lines 90 and 91):
````
(lldb) image list 
[  0] 6C2263D9-55FE-580A-8929-71B53C242367-3B19A31F                    /opt/python/cp38-cp38/bin/python 
[  1] BFBE63A1-682F-2156-75D7-35A28ADB0399-1C056202                    /opt/_internal/cpython-3.8.1/bin/../lib/libpython3.8.so.1.0 
[  2] 97BE6F91-99FE-D449-1B00-AA91F7E6EACC-4D5328F7                    /lib64/libcrypt.so.1 
[  3] E10CC8F2-B932-FC3D-AEDA-22F8DAC5EBB9-69524E5B                    /lib64/libpthread.so.0 
[  4] 7F2E9CB0-769D-7E57-BD66-9B485A74B537-B63A57C4                    /lib64/libdl.so.2 
[  5] FF2196BD-22A8-4430-54C8-3031E0E76EB0-1BA1219C                    /lib64/libutil.so.1 
[  6] 7615604E-AF4A-068D-FAE5-085444D15C0D-EE93DFBD                    /lib64/libm.so.6 
[  7] 9470E279-388F-7F9C-B2ED-3B2872D0C209-5B191FF4                    /lib64/libc.so.6 
[  8] 020C788B-41DC-C71A-EE66-B822D7670BC4-347DA006                    /lib64/libfreebl3.so 
[  9] 62C44997-4331-341B-B08D-CCE3859560A2-2AF1E172                    /lib64/ld-linux-x86-64.so.2 
[ 10] 0D1845E2-DCE9-324E-DA46-3088C89DE7E8-59E28958                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/math.cpython-38-x86_64-linux-gnu.so 
[ 11] 73EAA68F-88B6-DE75-8625-0A356F401B8B-89782B06                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_heapq.cpython-38-x86_64-linux-gnu.so 
[ 12] 36C78741-A4D8-6CE0-8FB1-1C276CCA289D-A58C7A57                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so 
[ 13] 7BE2DC09-660D-2D01-F922-AA5F2E9BBD01-D01E8906                    /usr/lib64/libffi.so.6.0.1 
[ 14] AB0E08CA-A10B-F4C6-DA16-B3995B4CBF39-48429C5E                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_struct.cpython-38-x86_64-linux-gnu.so 
[ 15] 8C967D92-A6FC-0E72-412C-BAC5E0722D8D-5C30865B                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_opcode.cpython-38-x86_64-linux-gnu.so 
[ 16] EF853282-CC54-DDDE-E64B-59AAA353E15D-B67B8274                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/zlib.cpython-38-x86_64-linux-gnu.so 
[ 17] B9D5F734-28BD-6AD6-8C96-986B57BEA3B7-CEDB9745                    /usr/lib64/libz.so.1.2.7 
[ 18] A9645FE2-9750-BD31-6283-2DF059E69B86-13DA5C78                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_bz2.cpython-38-x86_64-linux-gnu.so 
[ 19] 0C85C038-6F0C-F41E-A399-69CF7F58A558-D1AD3235                    /usr/lib64/libbz2.so.1.0.6 
[ 20] 66A8EE3A-92BF-464D-2C71-56A708236DB5-3428B8E1                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_lzma.cpython-38-x86_64-linux-gnu.so 
[ 21] F744796E-A6FA-4D80-1941-4F5487EE85DE-74DC5ADC                    /usr/lib64/liblzma.so.5.2.2 
[ 22] D8F0C268-9D0F-2CF2-E591-17522E780D04-1358A073                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/grp.cpython-38-x86_64-linux-gnu.so 
[ 23] B66EE92B-0B94-A948-10A0-1477AEEE7BD4-CED92595                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_bisect.cpython-38-x86_64-linux-gnu.so 
[ 24] 29B99260-D20D-F204-D75F-032FAB3F784C-89D2CD4E                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_sha512.cpython-38-x86_64-linux-gnu.so 
[ 25] CF95AB0E-35CC-43A7-C148-CA3B6E9D0A91-CFA55D55                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_random.cpython-38-x86_64-linux-gnu.so 
[ 26] 3585F61A-C47F-B239-8DC7-A16CA843AE20-A5FC2992                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_posixsubprocess.cpython-38-x86_64-linux-gnu.so 
[ 27] EFB58F7F-8A1C-D376-980F-6248D9A1A77F-734464AA                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/select.cpython-38-x86_64-linux-gnu.so 
[ 28] 0658FA78-ADAF-06A7-E581-AED8E4067BAB-0AAB9E41                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/torch/lib/libtorch_global_deps.so 
[ 29] 5272492A-E71E-0A00-0A8F-2BF5B6727F82-D695650D                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/nvidia/cufft/lib/libcufft.so.11 
[ 30] 861D8F1C-AD8C-0333-7FAA-099C54118A3A-A4BB1B1B                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/nvidia/curand/lib/libcurand.so.10 
[ 31] 76DF3592-B88C-3034-3C5A-520ACB5219B8-19E26DB0                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/nvidia/cublas/lib/libcublas.so.12 
[ 32] 7B302283-5A98-F516-72BC-A6F6AFCBC04B-2B2EC0A3                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/nvidia/cublas/lib/libcublasLt.so.12 
[ 33] 3130AC46-E7FB-8FE6-09E7-5651BC752B0C-A5BDE565                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/nvidia/cuda_runtime/lib/libcudart.so.12 
[ 34] 8AF27612-DA52-D3A7-806C-B5C936DCCF53-6CAEDF7C                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/nvidia/nvtx/lib/libnvToolsExt.so.1 
[ 35] 5F4FB88A-F97B-E3EC-ACC7-1363136BB015-B2A07119                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/torch/lib/libgomp-a34b3233.so.1 
[ 36] 3E44DF70-5594-2478-D052-E40FDD1F5B78-62B152B0                    /usr/lib64/librt-2.17.so 
[ 37] EDF51350-C7F7-1496-149D-064AA8B1441F-786DF88A                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1 
[ 38] 09CFB171-3101-10BC-7EA9-F4476C9FA044-D85BAFF4                    /usr/lib64/libstdc++.so.6.0.19 
[ 39] 51435A43-4D86-772E-08CF-3DC21F4C06EB-DD3ED21D                    /usr/lib64/libcuda.so.525.105.17 
[ 40] 26C34EAE-C0EE-9260-E96A-B862426D0C98-E3CD59BE                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/torch/_C.cpython-38-x86_64-linux-gnu.so 
[ 41] CB0C8FA4-6BD3-9CBA-6EF5-9F214A2F4280-70926875                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/torch/lib/libtorch_python.so 
[ 42] 899D0D99-E8FB-2D1F-4664-C75C38C58973-B6F7564C                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/torch/lib/libshm.so 
[ 43] E51BEADA-63C5-41AE-2C21-91BAC506413C-D3A69A09                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/torch/lib/libtorch.so 
[ 44] B69D8FC0-0EDA-9E06-976F-F5DDF07F3976-0135764C                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so 
[ 45] 37A63A40-59E3-0C41-7142-06C304712A9E-991F5719                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so 
[ 46] B987F520-A143-3C26-F8E7-21C520AA0141-5206B7E4                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/torch/lib/libc10_cuda.so 
[ 47] 8C597938-23B1-B92F-F463-C85D3B6B3F47-777647C7                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/torch/lib/libc10.so 
[ 48] 114B774D-3354-90C2-1AA9-55D7E18032C6-1D0A73FF                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/nvidia/cudnn/lib/libcudnn.so.8 
[ 49] 398392FB-0000-0000-0000-000000000000                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/nvidia/cuda_cupti/lib/libcupti.so.12 
[ 50] FB88E924-2FD2-A229-005F-3C1520C10393-28FDB9B6                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/nvidia/cusparse/lib/libcusparse.so.12 
[ 51] 1DAD5F31-1750-CD20-62C4-F0915AEC7610-635E0A7D                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/nvidia/nccl/lib/libnccl.so.2 
[ 52] 83B6DD72-558E-E08A-0675-E7BC58C04170-F02E42CB                    /usr/local/cuda-12.1/targets/x86_64-linux/lib/libnvJitLink.so.12.1.55 
[ 53] 4757FADA-DDCC-A281-D3B0-2988FBB8A437-24FCBEFF                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/torch/lib/libnvfuser_codegen.so 
[ 54] 5535086D-3380-568F-8EAE-CFA2E73F456F-1EDD94EC                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/nvidia/cuda_nvrtc/lib/libnvrtc.so.12 
[ 55] B4662EB0-AFDF-C43A-60A2-5B1C5AC6C64B-F0CABC9A                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_json.cpython-38-x86_64-linux-gnu.so 
[ 56] 9AF3FB5E-D069-3360-93BB-BA1892AC868E-59D621BB                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so 
[ 57] 8C5774C0-E3F6-3EBC-5CD3-4216721D90FD-4AC8050C                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/numpy.libs/libopenblas64_p-r0-15028c96.3.21.so 
[ 58] 5BBE74EB-6855-E0A2-C043-C0BEC2F484BF-3E9F14C0                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/numpy.libs/libgfortran-040039e1.so.5.0.0 
[ 59] 549B4C82-3477-8545-9571-C79239872AD3-1509DCF4                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/numpy.libs/libquadmath-96973f99.so.0.0.0 
[ 60] 42B95253-F816-764F-54D2-51562B70103E-0D1784A0                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_datetime.cpython-38-x86_64-linux-gnu.so 
[ 61] A96F33E1-AD90-BC7A-56FB-75417FA015F6-2C86AFFB                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_pickle.cpython-38-x86_64-linux-gnu.so 
[ 62] 6F9D5615-1A0A-60B5-0E75-B75D5B31CCB8-3647E64F                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_contextvars.cpython-38-x86_64-linux-gnu.so 
[ 63] 95E12C42-F6DC-52C4-0771-4BC5368B9194-D159EC7F                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/numpy/core/_multiarray_tests.cpython-38-x86_64-linux-gnu.so 
[ 64] 4A6F85D5-9ADF-BB83-0408-89A6CE872F2A-86400F82                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/numpy/linalg/_umath_linalg.cpython-38-x86_64-linux-gnu.so 
[ 65] 7178A59D-0F32-4EE7-7027-7F2FD4134453-743F845E                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/numpy/fft/_pocketfft_internal.cpython-38-x86_64-linux-gnu.so 
[ 66] 6E091182-1E1A-C873-9EBB-353B3754F257-EA7A5198                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/numpy/random/mtrand.cpython-38-x86_64-linux-gnu.so 
[ 67] E61710B6-BA4C-131A-47EF-45980C3BC833-74003CCB                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/numpy/random/bit_generator.cpython-38-x86_64-linux-gnu.so 
[ 68] 78AD06AD-F174-A044-8054-922C701AB973-F2E762F2                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/numpy/random/_common.cpython-38-x86_64-linux-gnu.so 
[ 69] 8129B613-27DE-6C7C-9BC0-8FA8A258BA82-4DCD11B8                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/binascii.cpython-38-x86_64-linux-gnu.so 
[ 70] 9F92A5F5-D4A1-2173-519C-2D9C50C7BFCB-D03AF5AC                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_hashlib.cpython-38-x86_64-linux-gnu.so 
[ 71] 9A3DB2C4-F4E4-E261-9ECB-6A322E88E3AE-65F66327                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_blake2.cpython-38-x86_64-linux-gnu.so 
[ 72] 6E2F9EEB-3FA2-EEF6-C332-CC5980E9E0B9-A887579B                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_sha3.cpython-38-x86_64-linux-gnu.so 
[ 73] B5015B93-D909-5C06-9885-88CBB3B8E0D5-8226CD6C                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/numpy/random/_bounded_integers.cpython-38-x86_64-linux-gnu.so 
[ 74] D5C1B0AB-8061-76E6-EA15-F365C47134D3-C3C7E59D                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/numpy/random/_mt19937.cpython-38-x86_64-linux-gnu.so 
[ 75] 7985FDCE-EF43-F84E-C1D9-B01D696F712B-C8D3B0C9                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/numpy/random/_philox.cpython-38-x86_64-linux-gnu.so 
[ 76] 64078B93-4F31-DAC6-2149-836EA303D489-C6550A12                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/numpy/random/_pcg64.cpython-38-x86_64-linux-gnu.so 
[ 77] 470E2A1D-8BB2-53EE-BEC1-582F43F4871C-098E049F                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/numpy/random/_sfc64.cpython-38-x86_64-linux-gnu.so 
[ 78] 4BE2A108-0EF6-2E2C-DFCF-06A498EA34D3-97E63274                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/numpy/random/_generator.cpython-38-x86_64-linux-gnu.so 
[ 79] 0A973651-2284-007D-E38E-08BACF8645D2-5EE14141                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/cmath.cpython-38-x86_64-linux-gnu.so 
[ 80] 44B37B80-7FDD-5D29-3751-6D93A148DA4B-5C738939                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_socket.cpython-38-x86_64-linux-gnu.so 
[ 81] 60FF3E5D-49DD-339D-4C36-C844707A6DCD-3D9B2DCE                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/array.cpython-38-x86_64-linux-gnu.so 
[ 82] D5B0A305-F4ED-8584-D4F4-ED26D80156E9-61B7B32F                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_multiprocessing.cpython-38-x86_64-linux-gnu.so 
[ 83] D476780F-37C6-DE60-EF7F-6BC76C18BDE9-EA5B1D8A                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so 
[ 84] 3E0CE9DE-5FC0-FBD3-C6E1-BD515885008A-5A589D5E                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_queue.cpython-38-x86_64-linux-gnu.so 
[ 85] F59B22DB-1AD8-17BC-2300-396FD62CA269-25B91238                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_asyncio.cpython-38-x86_64-linux-gnu.so 
[ 86] 20069E7C-6172-9980-1E27-00893D29E322-5822355B                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/_decimal.cpython-38-x86_64-linux-gnu.so 
[ 87] 62807570-4697-0A8B-9A08-965750BE07BA-0098AE75                    /opt/_internal/cpython-3.8.1/lib/python3.8/lib-dynload/unicodedata.cpython-38-x86_64-linux-gnu.so 
[ 88] 9E6CEDED-CD8B-82BA-57C1-98B7F514C711-CCC9199F                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/nvfuser/_C.cpython-38-x86_64-linux-gnu.so 
[ 89] F70D737E-0000-0000-0000-000000000000                    /usr/lib64/libnvidia-ml.so.525.105.17 
[ 90] AE971699-772B-BDA3-51A3-28B8F318DAFD-4C28A1B4                    /usr/local/cuda-12.1/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.9.2 
[ 91] EFA0AE70-BC7A-9C92-0E45-060D810FAF61-07BD23BE                    /opt/_internal/cpython-3.8.1/lib/python3.8/site-packages/nvidia/cudnn/lib/libcudnn_cnn_infer.so.8 
[ 92] 1AD32154-4717-01BD-AB25-B9F04FAFA0AA-56210145                    /usr/local/cuda-12.1/targets/x86_64-linux/lib/libnvrtc.so.12.1.55 
````

I think it would be great to have 2 followups from this issue:
 - Make sure that there is only one definition of CuDNN version (probably in builder repo)
 - Make sure that cudnn dynamic loading prefers full-versioned libraries to a partially versioned ones (cc: @ptrblck)

### Versions

Nightly/CI

cc @csarofeen @ptrblck @xwang233 @seemethere @pytorch/pytorch-dev-infra