RuntimeError: CUDA error: device kernel image is invalid

sicroci: <p>I get the following error in Windows: <code>RuntimeError: CUDA error: device kernel image is invalid</code></p>
<ul>
<li><code>print(torch.__version__)</code></li>
</ul>
<p><code>2.6.0+cu126</code></p>
<ul>
<li>nvidia-smi:</li>
</ul>
<pre><code class="lang-auto">Wed Apr  2 23:40:03 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 572.61                 Driver Version: 572.61         CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4080 ...  WDDM  |   00000000:E1:00.0 Off |                  N/A |
|  0%   42C    P8              4W /  320W |     360MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            7696    C+G   ...ll Peripheral Manager\DPM.exe      N/A      |
|    0   N/A  N/A           11288    C+G   C:\Windows\explorer.exe               N/A      |
|    0   N/A  N/A           11352    C+G   ...indows\System32\ShellHost.exe      N/A      |
|    0   N/A  N/A           13408    C+G   ..._cw5n1h2txyewy\SearchHost.exe      N/A      |
|    0   N/A  N/A           13424    C+G   ...y\StartMenuExperienceHost.exe      N/A      |
|    0   N/A  N/A           15524    C+G   ...ouryDevice\asus_framework.exe      N/A      |
|    0   N/A  N/A           17764    C+G   ...gato\CameraHub\Camera Hub.exe      N/A      |
|    0   N/A  N/A           22980    C+G   ...t\Edge\Application\msedge.exe      N/A      |
|    0   N/A  N/A           24416    C+G   ...t\Edge\Application\msedge.exe      N/A      |
+-----------------------------------------------------------------------------------------+
</code></pre>
<ul>
<li><code>nvcc --version</code></li>
</ul>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Feb_21_20:42:46_Pacific_Standard_Time_2025
Cuda compilation tools, release 12.8, V12.8.93
Build cuda_12.8.r12.8/compiler.35583870_0
</code></pre>

ptrblck: <p>Could you either post a minimal and executable code snippet reproducing the issue or let us know which kernel raises the error?</p>

sicroci: <p>This is another code that generates the same error:</p>
<pre><code class="lang-auto">import torch
device = torch.device("cuda")
torch.rand(10).to(device)
</code></pre>
<p>This is the complete error message:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "C:\Users\Thanos\miniconda3\lib\site-packages\torch\_tensor.py", line 568, in __repr__
    return torch._tensor_str._str(self, tensor_contents=tensor_contents)
  File "C:\Users\Thanos\miniconda3\lib\site-packages\torch\_tensor_str.py", line 704, in _str
    return _str_intern(self, tensor_contents=tensor_contents)
  File "C:\Users\Thanos\miniconda3\lib\site-packages\torch\_tensor_str.py", line 621, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "C:\Users\Thanos\miniconda3\lib\site-packages\torch\_tensor_str.py", line 353, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "C:\Users\Thanos\miniconda3\lib\site-packages\torch\_tensor_str.py", line 146, in __init__
    tensor_view, torch.isfinite(tensor_view) &amp; tensor_view.ne(0)
RuntimeError: CUDA error: device kernel image is invalid
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

</code></pre>
<p>I have uninstalled and reinstalled all NVIDIA software including the drivers in Windows 11, and I get the same problem.</p>
<p>Here more information:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print(torch.version.cuda)
12.6
&gt;&gt;&gt; print(torch.cuda.is_available())
True
&gt;&gt;&gt; print(torch.cuda.get_device_name(0))  # Get GPU name
NVIDIA GeForce RTX 4080 SUPER
</code></pre>
<p>Output of nvcc --version:</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Fri_Jun_14_16:44:19_Pacific_Daylight_Time_2024
Cuda compilation tools, release 12.6, V12.6.20
Build cuda_12.6.r12.6/compiler.34431801_0

</code></pre>
<p>Output of nvidia-smi</p>
<pre><code class="lang-auto">Thu Apr  3 19:53:37 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 572.83                 Driver Version: 572.83         CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4080 ...  WDDM  |   00000000:E1:00.0 Off |                  N/A |
|  0%   42C    P5             20W /  320W |     252MiB /  16376MiB |      6%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            8460    C+G   ...ouryDevice\asus_framework.exe      N/A      |
|    0   N/A  N/A           12136    C+G   ...ll Peripheral Manager\DPM.exe      N/A      |
|    0   N/A  N/A           14012    C+G   C:\Windows\explorer.exe               N/A      |
|    0   N/A  N/A           14020    C+G   ...indows\System32\ShellHost.exe      N/A      |
|    0   N/A  N/A           14964    C+G   ..._cw5n1h2txyewy\SearchHost.exe      N/A      |
|    0   N/A  N/A           14988    C+G   ...y\StartMenuExperienceHost.exe      N/A      |
|    0   N/A  N/A           20576    C+G   ...gato\CameraHub\Camera Hub.exe      N/A      |
+-----------------------------------------------------------------------------------------+

</code></pre>

sicroci: <p>I have solved the problem by uninstalling and then installing all NVIDIA software (drivers, cuda, …) and this time also PyTorch.</p>