Maybe a BUG: `torch.cuda.is_available()` returns `False` in certain torch, CUDA and driver version

DicardoX: <p>Hi, I’m trying to create a Docker container with the following CUDA <code>12.4.1</code> Dockerfile (host info: Driver Version: <code>550.107.02</code>     CUDA Version: <code>12.4</code>, detailed versions can be found in: <a href="https://github.com/pytorch/pytorch/issues/135508" class="inline-onebox" rel="noopener nofollow ugc">BUG: `torch.cuda.is_available()` returns `False` in certain torch, CUDA and driver version · Issue #135508 · pytorch/pytorch · GitHub</a>):</p>
<pre><code class="lang-auto">FROM nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04

ARG DEBIAN_FRONTEND=noninteractive

# Install common tool &amp; conda
RUN apt-get update &amp;&amp; apt-get install -y \
    software-properties-common \
    &amp;&amp; add-apt-repository ppa:deadsnakes/ppa \
    &amp;&amp; apt install -y python3.10 \
    &amp;&amp; rm -rf /var/lib/apt/lists/*

RUN apt update &amp;&amp; \
    apt install wget -y &amp;&amp; \
    apt install git -y &amp;&amp; \
    apt install curl -y &amp;&amp; \
    apt install vim -y &amp;&amp; \
    apt install bc &amp;&amp; \
    apt-get install net-tools -y &amp;&amp; \
    apt install ssh -y &amp;&amp; \
    wget --quiet https://repo.anaconda.com/archive/Anaconda3-2022.05-Linux-x86_64.sh -O ~/anaconda.sh &amp;&amp; \
    /bin/bash ~/anaconda.sh -b -p /opt/conda &amp;&amp; \
    rm ~/anaconda.sh &amp;&amp; \
    mkdir -p /opt/conda/envs/finetune &amp;&amp; \
    ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh &amp;&amp; \
    echo ". /opt/conda/etc/profile.d/conda.sh" &gt;&gt; ~/.bashrc &amp;&amp; \
    echo "conda activate base" &gt;&gt; ~/.bashrc


# Workspace
WORKDIR /app

# Install conda finetune env
# COPY requirements.txt requirements.txt
RUN . /opt/conda/etc/profile.d/conda.sh &amp;&amp; \
    conda create --name finetune python=3.10 -y &amp;&amp; \
    conda activate finetune &amp;&amp; \
    curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10

# Cuda path
ENV CUDA_PATH=/usr/local/cuda
ENV LD_LIBRARY_PATH=$CUDA_PATH/lib64:$CUDA_PATH/compat:/usr/lib/x86_64-linux-gnu:$CUDA_PATH/targets/x86_64-linux/lib/stubs/:$LD_LIBRARY_PATH
ENV CUDNN_PATH=/usr/include
# Transformer engine path
ENV NVTE_FRAMEWORK=pytorch

# Copy workspace
COPY . .

# Enterpoint for bash shell
ENTRYPOINT ["/bin/bash"]
</code></pre>
<p>This just create a basic <code>nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04</code> image and install <code>conda</code> and <code>pip</code>. Then, I run the container with the following command:</p>
<pre><code class="lang-auto">docker run --runtime=nvidia -it --rm --gpus all --shm-size 64g --network=host --privileged --volume [USER_PATH]/.cache:/root/.cache --env NVIDIA_DISABLE_REQUIRE=1 username/imagename:tag
</code></pre>
<p>Then, inside the container, I install the latest stable torch (<code>2.4.1</code>) by:</p>
<pre><code class="lang-auto">pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
</code></pre>
<p>After that, I run the simplest torch cuda test by:</p>
<pre><code class="lang-auto">python -c "import torch; print(torch.cuda.is_available())"
</code></pre>
<p>What I got is:</p>
<pre><code class="lang-auto">/opt/conda/envs/finetune/lib/python3.10/site-packages/torch/cuda/__init__.py:128: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() &gt; 0
False
</code></pre>
<p>This is quite strange, since if I simlply turn to use the base image <code>nvidia/cuda:12.5.1-cudnn-devel-ubuntu22.04</code>, I can got the correct result that <code>torch.cuda.is_available()</code> returns <code>True</code>.</p>
<p>Any advice will be sincerely appreciated, thx!</p>

ptrblck: <p>The corresponding issue on GitHub has a valid suggestion of compiling a small standalone CUDA example to verify the container itself is working in your environment.</p>