Can't import torch --> OSError related to libcublasLt.so.11
Labels: oncall: binaries, module: cuda, module: regression

### 🐛 Describe the bug

When I try to import torch in my docker container, I get an OSError:
`/usr/local/lib/python3.8/dist-packages/torch/lib/../../nvidia/cublas/lib/libcublas.so.11: undefined symbol: cublasLtGetStatusString, version libcublasLt.so.11`

I've followed the step from [Issue 51080](https://github.com/pytorch/pytorch/issues/51080), which seems like it might be similar, but to no effect. Note, I don't have conda, so I didn't follow those specific steps, just the pip ones.

However, I have found that if I import tensorflow first, list my devices and then import torch it works fine...

```
try:
    import torch
except OSError as e:
    print(e)
    import tensorflow as tf
    tf.config.list_physical_devices()
    import torch
    print([torch.device(i) for i in range(torch.cuda.device_count())])
```
```
>>> /usr/local/lib/python3.8/dist-packages/torch/lib/../../nvidia/cublas/lib/libcublas.so.11: undefined symbol: cublasLtGetStatusString, version libcublasLt.so.11
>>> [device(type='cuda', index=0), device(type='cuda', index=1)]
```
Which makes me suspect that this is a different issue to [Issue 51080](https://github.com/pytorch/pytorch/issues/51080).

The full traceback of `import torch` is:
```
OSError                                   Traceback (most recent call last)
/root/GI_Data/KPVESQC5_AI4Q_P/Exp_workflow_a.ipynb Cell 2' in <cell line: 1>()
----> [1](vscode-notebook-cell://attached-container%2B7b22636f6e7461696e65724e616d65223a222f74625f656173657163222c2273657474696e6773223a7b22686f7374223a227373683a2f2f4750555f75736572227d7d/root/GI_Data/KPVESQC5_AI4Q_P/Exp_workflow_a.ipynb#ch0000009vscode-remote?line=0) import torch

File /usr/local/lib/python3.8/dist-packages/torch/__init__.py:191, in <module>
    [180](file:///usr/local/lib/python3.8/dist-packages/torch/__init__.py?line=179) else:
    [181](file:///usr/local/lib/python3.8/dist-packages/torch/__init__.py?line=180)     # Easy way.  You want this most of the time, because it will prevent
    [182](file:///usr/local/lib/python3.8/dist-packages/torch/__init__.py?line=181)     # C++ symbols from libtorch clobbering C++ symbols from other
   (...)
    [188](file:///usr/local/lib/python3.8/dist-packages/torch/__init__.py?line=187)     #
    [189](file:///usr/local/lib/python3.8/dist-packages/torch/__init__.py?line=188)     # See Note [Global dependencies]
    [190](file:///usr/local/lib/python3.8/dist-packages/torch/__init__.py?line=189)     if USE_GLOBAL_DEPS:
--> [191](file:///usr/local/lib/python3.8/dist-packages/torch/__init__.py?line=190)         _load_global_deps()
    [192](file:///usr/local/lib/python3.8/dist-packages/torch/__init__.py?line=191)     from torch._C import *  # noqa: F403
    [194](file:///usr/local/lib/python3.8/dist-packages/torch/__init__.py?line=193) # Appease the type checker; ordinarily this binding is inserted by the
    [195](file:///usr/local/lib/python3.8/dist-packages/torch/__init__.py?line=194) # torch._C module initialization code in C

File /usr/local/lib/python3.8/dist-packages/torch/__init__.py:153, in _load_global_deps()
    [150](file:///usr/local/lib/python3.8/dist-packages/torch/__init__.py?line=149) here = os.path.abspath(__file__)
    [151](file:///usr/local/lib/python3.8/dist-packages/torch/__init__.py?line=150) lib_path = os.path.join(os.path.dirname(here), 'lib', lib_name)
--> [153](file:///usr/local/lib/python3.8/dist-packages/torch/__init__.py?line=152) ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)

File /usr/lib/python3.8/ctypes/__init__.py:373, in CDLL.__init__(self, name, mode, handle, use_errno, use_last_error, winmode)
    [370](file:///usr/lib/python3.8/ctypes/__init__.py?line=369) self._FuncPtr = _FuncPtr
    [372](file:///usr/lib/python3.8/ctypes/__init__.py?line=371) if handle is None:
--> [373](file:///usr/lib/python3.8/ctypes/__init__.py?line=372)     self._handle = _dlopen(self._name, mode)
    [374](file:///usr/lib/python3.8/ctypes/__init__.py?line=373) else:
    [375](file:///usr/lib/python3.8/ctypes/__init__.py?line=374)     self._handle = handle

OSError: /usr/local/lib/python3.8/dist-packages/torch/lib/../../nvidia/cublas/lib/libcublas.so.11: undefined symbol: cublasLtGetStatusString, version libcublasLt.so.11
```

### Versions

Python version: 3.8.10
Docker version: 20.10.21
BASE_IMAGE=tensorflow/tensorflow
IMAGE_VERSION=2.9.1-gpu-jupyter

Output of `nvcc -V`:
```
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Feb_14_21:12:58_PST_2021
Cuda compilation tools, release 11.2, V11.2.152
Build cuda_11.2.r11.2/compiler.29618528_0
```

Output of `torch.__version__` in python:
```
'1.13.0+cu117'
```

Output of collect_env.py:
```
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.8.10 (default, Mar 15 2022, 12:22:08)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.29
Is CUDA available: N/A
CUDA runtime version: 11.2.152
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: 
GPU 0: NVIDIA A10
GPU 1: NVIDIA A10

Nvidia driver version: 520.61.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

Versions of relevant libraries:
[pip3] numpy==1.22.4
[pip3] torch==1.13.0
[pip3] torchaudio==0.13.0
[pip3] torchvision==0.14.0
[conda] Could not collect
```

cc @ezyang @seemethere @malfet @ngimel