NVIDIA L40S-48Q and "RuntimeError: CUDA error: operation not supported"

Chris_Palmer: <p>We have been allocated a brand new NVIDIA L40S-48Q in our research environment.</p>
<p>Unfortunately, when we try and use the GPU we get “RuntimeError: CUDA error: operation not supported”. We have tried switching the background CUDA version between 12.1 and 11.8 with an appropriately matched Pytorch installation. We have tried Pytorch 2.5, 2.6 nightly, and 2.3. We have the correct values in CUDA_HOME, PATH, and LD_LIBRARY_PATH, as others have suggested. We have tried Python versions ranging from 3.9 to 3.11.</p>
<p>The research environment has other GPUs available with 24GB of RAM, and they work with the same settings, so it seems to be a problem with the GPU itself.</p>
<p>Any suggestions?</p>

ptrblck: <p>Could you post a minimal and executable code snippet to reproduce the issue, please?</p>

Chris_Palmer: <p>Thanks for replying <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>First of all I am displaying the environment - we have Cuda 2.1 installed (though nvidia-smi says 12.2 - I think because it was originally installed under 12.2).</p>
<p>(env39) xxx:~$ nvidia-smi<br>
Sat Nov 9 05:42:03 2024<br>
±--------------------------------------------------------------------------------------+<br>
| NVIDIA-SMI 535.183.06             Driver Version: 535.183.06   CUDA Version: 12.2     |<br>
|-----------------------------------------±---------------------±---------------------+<br>
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |<br>
|                                         |                      |               MIG M. |<br>
|=========================================+======================|<br>
|   0  NVIDIA L40S-48Q                On  | 00000000:00:05.0 Off |                    0 |<br>
| N/A   N/A    P8              N/A /  N/A |     51MiB / 49152MiB |      0%      Default |<br>
|                                         |                      |             Disabled |<br>
±----------------------------------------±---------------------±---------------------+<br>
| Processes:                                                                            |<br>
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |<br>
|        ID   ID                                                             Usage      |<br>
|================================================================|<br>
|    0   N/A  N/A      1580      G   /usr/lib/xorg/Xorg                           51MiB |<br>
±--------------------------------------------------------------------------------------+</p>
<p>(env39) xxx:~$ ls -lh /etc/alternatives/cuda<br>
lrwxrwxrwx 1 root root 20 Nov  8 16:07 /etc/alternatives/cuda → /usr/local/cuda-12.1</p>
<p>(env39) xxx:~$ echo $CUDA_HOME<br>
/usr/local/cuda</p>
<p>(env39) xxx:~$ echo $PATH<br>
/usr/local/cuda:/usr/local/cuda/lib64:/home/unimelb.edu.au/xxx/anaconda3/envs/env39/bin:/home/unimelb.edu.au/xxx/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/fastx/3/apps</p>
<p>(env39) xxx:~$ echo $LD_LIBRARY_PATH<br>
/usr/local/cuda/lib64:</p>
<p>(env39) xxx:~$ python<br>
Python 3.9.20 (main, Oct  3 2024, 07:27:41)</p>
<pre><code class="lang-auto">import torch
torch.cuda.is_available()
True

torch.zeros(1).cuda()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: CUDA error: operation not supported
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.</code></pre>

ptrblck: <aside class="quote no-group" data-username="Chris_Palmer" data-post="3" data-topic="212716">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chris_palmer/48/12322_2.png" class="avatar"> Chris_Palmer:</div>
<blockquote>
<p>we have Cuda 2.1 installed (though nvidia-smi says 12.2 - I think because it was originally installed under 12.2).</p>
</blockquote>
</aside>
<p>CUDA 2.1 was released in Jan 2009 and does support Tesla compute capabilities only, so I doubt you have it installed.<br>
In any case, the locally installed CUDA toolkit won’t matter unless you build PyTorch from source.</p>
<p>Since the very first call already fails, I would start by either running any other CUDA-enabled application to verify it can run or by reinstalling the NVIDIA driver.</p>

Chris_Palmer: <p>Sorry, I mistyped - we have CUDA 12.1, not 2.1. Can you suggest another CUDA enabled app we can try?</p>
<p>I will have to ask the administrators of the system to re-install the NVIDIA driver, but it would be helpful to text it with some other app that ought to run…</p>

ptrblck: <p>Try to compile a few CUDA samples and execute them.</p>

Chris_Palmer: <p>Thanks, <a class="mention" href="/u/ptrblck">@ptrblck</a> - this is not something I am familiar with doing, I just use GPU for NLP. Can you point me to the best resource for getting the compilation tools and samples? Or, can you suggest a Linux application I can download to see if it runs on CUDA?</p>

ptrblck: <p><a href="https://github.com/NVIDIA/cuda-samples"><code>NVIDIA/cuda-samples</code></a> might be a good start to check if your setup is able to compile and execute a few example applications.</p>

Chris_Palmer: <p>Thanks, <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>See a message from our network admin below. After he rebooted the server the problem went away. He thinks it was due to a CUDA context not being able to be created because the NVIDIA driver was unlicensed. Does this make sense to you?</p>
<blockquote>
<p>I installed a GPU test software container and it was failing with an error that it was unable to create a CUDA context.</p>
<p>I noticed that the output of ‘nvidia-smi -q’ stated that the driver was unlicensed, so I rebooted and the error went away. Now the GPU diagnostic tool works without error.</p>
<p>Can you please try your software again and see if the license was the problem for you also?</p>
<p>The GPU diagnostic tool I used is here:</p>
<p><a href="https://url.au.m.mimecastprotect.com/s/ZVhmC6XQXPFZZJOqHmi2H5lLiZ?domain=together.ai" rel="noopener nofollow ugc">https://www.together.ai/blog/a-practitioners-guide-to-testing-and-running-large-gpu-clusters-for-training-generative-ai-models</a></p>
</blockquote>

ptrblck: <p>Good to hear it’s working now!<br>
I’m not familiar with vGPUs and how the driver expects to be licensed, but the debugging steps sound valid and I’m glad your network admin was able to narrow down the issue by trying to create a simple CUDA workflow.</p>

briskajanis1: <p>We also had similar error in the VM with vGPU and <code>nvidia-smi -q</code> showed that licence is not active. It appears that it will throw error CUDA: Operation not supported if the vGPU licence is not active.</p>
<p>We resolved this by VM maintainers just setting up full GPU passthrough instead of vGPU. The CUDA started working.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://docs.nvidia.com/vgpu/13.0/grid-vgpu-user-guide/index.html#cuda-open-cl-support-vgpu">
  <header class="source">
      <img src="https://docs.nvidia.com/favicon-32x32.png" class="site-icon" width="32" height="32">

      <a href="https://docs.nvidia.com/vgpu/13.0/grid-vgpu-user-guide/index.html#cuda-open-cl-support-vgpu" target="_blank" rel="noopener nofollow ugc">NVIDIA Docs</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://docs.nvidia.com/vgpu/13.0/grid-vgpu-user-guide/index.html#cuda-open-cl-support-vgpu" target="_blank" rel="noopener nofollow ugc">Virtual GPU Software User Guide - NVIDIA Docs</a></h3>

  <p>Documentation for administrators that explains how to install and configure NVIDIA Virtual GPU manager, configure virtual GPU software in pass-through mode, and install drivers on guest operating systems.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Hope this helps someone.</p>