Docker: Error 804: forward compatibility (...)

username: nicoco
cooked: <p>Hi all,</p>
<p>I fail to get a minimal, bleeding edge, CUDA-enabled pytorch container working. I followed the build instructions on pytorch’s README. On the same machine, <a href="https://ngc.nvidia.com/catalog/containers/nvidia:pytorch" rel="noopener nofollow ugc">nvidia’s pytorch image</a> works just fine, launched with the same <code>docker run</code> arguments. The only difference I could spot was CUDA 11.2 in nvidia’s image vs CUDA 11.3 in mine. Is this known not to work? I am rebuilding with CUDA 11.2 right now but it takes a while, so I figured I’d ask here in the meantime in case I am missing something obvious and someone sees this.</p>
<p>This is what I get (inside the container):</p>
<pre><code class="lang-auto">root@42afc3dc16b0:/# nvidia-smi
Fri May 28 15:00:14 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.73.01    Driver Version: 460.73.01    CUDA Version: 11.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GT 710      On   | 00000000:09:00.0 N/A |                  N/A |
| 50%   43C    P8    N/A /  N/A |    222MiB /   973MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 1080    On   | 00000000:0A:00.0 Off |                  N/A |
|  0%   29C    P8     8W / 210W |      6MiB /  8119MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
root@42afc3dc16b0:/# python3 -c "import torch;print(torch.cuda.is_available())"
/opt/miniconda/lib/python3.9/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at  ../c10/cuda/CUDAFunctions.cpp:115.)
  return torch._C._cuda_getDeviceCount() &gt; 0
False
</code></pre>
<p>This is my Dockerfile:</p>
<pre><code class="lang-auto">FROM nvidia/cuda:11.3.0-cudnn8-devel-ubuntu20.04 AS condainstall

COPY Miniconda3-py39_4.9.2-Linux-x86_64.sh /root/
RUN sh /root/Miniconda3-py39_4.9.2-Linux-x86_64.sh -b -p /opt/miniconda &amp;&amp; \
    /opt/miniconda/bin/conda install -y\
        astunparse \
        numpy\
        ninja\
        pyyaml\
        mkl\
        mkl-include\
        setuptools\
        cmake\
        cffi\
        typing_extensions\
        future\
        six\
        requests\
        dataclasses &amp;&amp; \
    /opt/miniconda/bin/conda clean --all

FROM nvidia/cuda:11.3.0-cudnn8-devel-ubuntu20.04 AS torchbuild

COPY --from=condainstall /opt/miniconda /opt/miniconda

# cloned from github
COPY pytorch /root/pytorch

WORKDIR /root/pytorch

RUN apt-get update &amp;&amp; apt install gcc g++ -y

RUN eval "$(/opt/miniconda/bin/conda shell.bash hook)" &amp;&amp; \
    export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"} &amp;&amp; \
    python3 setup.py install

FROM nvidia/cuda:11.3.0-cudnn8-devel-ubuntu20.04

COPY --from=torchbuild /opt/miniconda /opt/miniconda

ENV PATH /opt/miniconda/bin:$PATH

ENTRYPOINT ["python3", "-c", "import torch;print(torch.cuda.is_available())"]
</code></pre>

username: ptrblck
cooked: <p>Based on the error message I guess you might need to update your drivers on the bare metal node.</p>

username: nicoco
cooked: <p>It turns out using CUDA 11.2 instead of 11.3 solved my problem.</p>
<p>Thanks for your reply <a class="mention" href="/u/ptrblck">@ptrblck</a>. Updating the drivers on the bare metal might work too, but I am happy to just let debian’s APT handle the drivers on the bare metal node for now.</p>