Pytorch Cuda returns Error 500: named symbol not found

torch_fan: <p>Dear community,<br>
I try to train a simple model in a docker container deployed on my local machine.  I use the following docker image for my development container: pytorch/pytorch:2.6.0-cuda12.6-cudnn9-runtime</p>
<p>Whenever I try to use my gpu (nvidia rtx 4090) (by typing"<br>
python -c “import torch; print(torch.cuda.is_available())”), I get the symptom as described below:</p>
<pre><code class="lang-auto">RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 500: named symbol not found
</code></pre>
<p>What steps did I take:</p>
<ul>
<li>nvidia-smi returns correct driver and cuda version (VM):</li>
</ul>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.07             Driver Version: 572.83         CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        On  |   00000000:01:00.0 Off |                  Off |
|  0%   36C    P8             10W /  450W |    1248MiB /  24564MiB |      2%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
</code></pre>
<ul>
<li>nvidia-smi Host:</li>
</ul>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 572.83                 Driver Version: 572.83         CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090      WDDM  |   00000000:01:00.0 Off |                  Off |
|  0%   37C    P8             11W /  450W |    1226MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
</code></pre>
<ul>
<li>nvcc --version (VM) returns:<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2023 NVIDIA Corporation<br>
Built on Mon_Apr__3_17:16:06_PDT_2023<br>
Cuda compilation tools, release 12.1, V12.1.105<br>
Build cuda_12.1.r12.1/compiler.32688072_0</li>
</ul>
<p>Therefore the gpu seems to be correctly recognized by the container.</p>
<p>Furthermore I tried the following:</p>
<ul>
<li>Updating nvidia drivers on host with reboot after installation</li>
<li>different pytorch docker images (2.3.1; 2.5.1; 2.6) devel and runtime</li>
<li>Manual removal and reinstall of pytorch in VM</li>
</ul>
<p>Further Setup Details:<br>
Windows 11 24H2<br>
Nvidia RTX4090<br>
Docker Desktop Engine v26.0.0</p>
<p>I do not know what else to do in order to make pytorch work with my gpu. Any help is appreciated. I’ll gladly give more details if needed.</p>
<p>Thank you in advance!</p>

ptrblck: <p>This error could be raised by a missing or old NVIDIA Container Toolkit. Which version are you using?</p>

dbrenes: <p>Hi <a class="mention" href="/u/torch_fan">@torch_fan</a>.<br>
Your <code>nvidia-smi</code> command results shows <code>CUDA Version: 12.8</code>, however, this just means that the highest CUDA version that your driver supports is 12.8. Meanwhile, <code>nvcc --version</code> shows that you actually have CUDA 12.1 installed on your host device.</p>
<p>Since you are using the <code>pytorch/pytorch:2.6.0-cuda12.6-cudnn9-runtime</code> Docker image, the error you’re seeing could be due to a mismatch in the supported CUDA versions between the container and your host device. You could try out a Docker image that matches your installed CUDA version, such as <code>2.5.1-cuda12.1-cudnn9-runtime</code>.</p>
<p>Machine Learning Engineer at RidgeRun.ai<br>
Contact us: <a href="mailto:support@ridgerun.ai">support@ridgerun.ai</a><br>
<a href="https://www.ridgerun.ai/" rel="noopener nofollow ugc">https://www.ridgerun.ai</a></p>

torch_fan: <p>Isn’t the return of the command “nvcc --version” indicative that the nvidia toolkit was properly installed? - In this case it tells me the following:</p>
<p>Cuda compilation tools, release 12.1, V12.1.105<br>
Build cuda_12.1.r12.1/compiler.32688072_0</p>

torch_fan: <p>I don’t have Cuda installed on my host machine (the excerpt you see above is from the guest). Installation of Cuda was not necessary yet. Windows has updated my graphics driver unwillingly which broke cuda in my VM. Therefore I know that it has once worked with an older version of pytorch (2.3.1) and an older nvidia driver (version unknown).</p>

ptrblck: <p>I don’t mean the CUDA compiler (<code>nvcc</code>) but the NVIDIA <em>Container</em> Toolkit allowing you to use your GPU inside containers.</p>

torch_fan: <aside class="quote no-group" data-username="torch_fan" data-post="1" data-topic="218501">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/a698b9/48.png" class="avatar"> torch_fan:</div>
<blockquote>
<p>python -c “import torch; print(torch.cuda.is_available())”</p>
</blockquote>
</aside>
<p>I’ve installed the following version in the guest VM:</p>
<p>nvidia-ctk --version<br>
NVIDIA Container Toolkit CLI version 1.17.5<br>
commit: f785e908a7f72149f8912617058644fd84e38cde</p>
<p>No change regarding the error-message</p>
<p>EDIT: According to <a href="https://github.com/tensorflow/tensorflow/issues/68711" class="inline-onebox" rel="noopener nofollow ugc">"CUDA_ERROR_NOT_FOUND: named symbol not found" in Docker container · Issue #68711 · tensorflow/tensorflow · GitHub</a> people encountered a very similar issue with tensorflow + Cuda; Maybe someone is willing to share their working setup configuration w.r.t. pytorch docker image version (incl. Cuda) + driver version + docker version</p>

torch_fan: <p>After many attempts, I have found the solution:<br>
<strong>I needed to upgrade the Docker Desktop version</strong></p>
<p>For future reference - the following configuration works for me:</p>
<ul>
<li>Docker Engine v28.0.4 with Docker Desktop 4.40.0 (187762)</li>
<li>Docker image: pytorch/pytorch:2.6.0-cuda12.6-cudnn9-runtime</li>
<li>nvidia driver version: 572.83; Cuda-version: 12.8</li>
</ul>
<p>There is no need to install any additional cuda framework on the host nor on the client. I didn’t need to install any nvidia-ctk (container-toolkit) as well.</p>
<p>Thank you for your great support nevertheless!</p>

Epiphany_Fall: <p>The same, Error:500. Solved my problem, tks.<br>
I use 4.23.X before.<br>
Now:<br>
Win11: CUDA Version: 12.9 Driver Version: 576.02  NVIDIA-SMI 576.02 nothing else;<br>
Docker Desktop: 4.40<br>
Image: [pytorch/pytorch:2.6.0-cuda12.6-cudnn9-runtime]<br>
Container: All CUDA info inherit windows host, <strong>regardless of the “written” version matching which has no relationship</strong>, did nothing, dont do anything.</p>
<p>Only update docker engine (docker desktop)</p>
<p>Thankyou all</p>

Liviere: <p>I had the same situation.</p>
<p>Docker Engine before: 26.0.0</p>
<p>Docker Engine now: 28.5.1</p>
<p>After the update, everything started working properly.</p>