Nvidia-smi is OK. torch.cuda.is_available() FAILS in Docker

Poke: <p>Setup: Windows 11 with Docker, WSL2, Ubuntu 20.04 docker image<br>
So nvidia-smi works:<br>
±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 515.76.02    Driver Version: 517.48       CUDA Version: 11.7     |<br>
|-------------------------------±---------------------±---------------------+<br>
and all the usual info here.</p>
<p>But torch fails with error.<br>
/usr/local/lib/python3.8/dist-packages/torch/cuda/<strong>init</strong>.py:83: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 2: out of memory (Triggered internally at  …/c10/cuda/CUDAFunctions.cpp:109.)</p>
<p>I tried with latest nightly with 11.7 cuda:<br>
python collect_env.py<br>
Collecting environment information…<br>
/usr/local/lib/python3.8/dist-packages/torch/cuda/<strong>init</strong>.py:88: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 2: out of memory (Triggered internally at …/c10/cuda/CUDAFunctions.cpp:109.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0<br>
PyTorch version: 1.14.0.dev20221008+cu117<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.7<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 20.04.4 LTS (x86_64)<br>
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0<br>
Clang version: Could not collect<br>
CMake version: Could not collect<br>
Libc version: glibc-2.31</p>
<p>Python version: 3.8.10 (default, Jun 22 2022, 20:18:18)  [GCC 9.4.0] (64-bit runtime)<br>
Python platform: Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.29<br>
Is CUDA available: False<br>
CUDA runtime version: 11.7.99<br>
GPU models and configuration:<br>
All cards listed here</p>
<p>Nvidia driver version: 517.48<br>
cuDNN version: Probably one of the following:<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.5.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.5.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.5.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.5.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.5.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.5.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.5.0<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.23.3<br>
[pip3] torch==1.14.0.dev20221008+cu117<br>
[pip3] torchaudio==0.13.0.dev20221006+cu117<br>
[pip3] torchvision==0.15.0.dev20221008+cu117<br>
[conda] Could not collect</p>
<p>Also tried with the latest pytorch with 11.6 cuda:<br>
python collect_env.py<br>
Collecting environment information…<br>
/usr/local/lib/python3.8/dist-packages/torch/cuda/<strong>init</strong>.py:83: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 2: out of memory (Triggered internally at  …/c10/cuda/CUDAFunctions.cpp:109.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0<br>
PyTorch version: 1.12.1+cu116<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.6<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 20.04.4 LTS (x86_64)<br>
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0<br>
Clang version: Could not collect<br>
CMake version: Could not collect<br>
Libc version: glibc-2.31</p>
<p>Python version: 3.8.10 (default, Jun 22 2022, 20:18:18)  [GCC 9.4.0] (64-bit runtime)<br>
Python platform: Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.29<br>
Is CUDA available: False<br>
CUDA runtime version: 11.7.99<br>
GPU models and configuration:<br>
All cards listed here</p>
<p>Nvidia driver version: 517.48<br>
cuDNN version: Probably one of the following:<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.5.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.5.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.5.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.5.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.5.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.5.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.5.0<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.23.3<br>
[pip3] torch==1.12.1+cu116<br>
[pip3] torchaudio==0.12.1+cu116<br>
[pip3] torchvision==0.13.1+cu116<br>
[conda] Could not collect</p>
<p>Reinstalled drivers on the host.</p>
<p>and I’m out of ideas.</p>

ptrblck: <p>Could you check if the right drivers are installed for WSL2 and docker as described <a href="https://docs.nvidia.com/cuda/wsl-user-guide/index.html#support-matrix-for-wsl2">here</a>, please?</p>

Poke: <p>I reinstalled everything a few times.<br>
Latest CUDA + latest Drivers<br>
Older CUDA + latest Drivers<br>
Older CUDA + older drivers<br>
I am sure I am doing something wrong but don’t know what.</p>