Error RuntimeError: CUDA error: operation not supported when tried to locate something into cuda:0

ferscp: <p>hi everyone</p>
<p>here is my code</p>
<pre><code class="lang-auto">from transformers import AutoModelForCausalLM, AutoTokenizer, QuantoConfig
import torch
device = "cuda:0"
model_id = "bigscience/bloom-560m"
quantization_config = QuantoConfig(weights="int8")

model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float32,  device_map=device)

tokenizer = AutoTokenizer.from_pretrained(model_id)

text = "Hello my name is"
inputs = tokenizer(text, return_tensors="pt").to(device)
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
</code></pre>
<p>when i run i obtain the next error:</p>
<blockquote>
<p>RuntimeError: CUDA error: operation not supported<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
</blockquote>
<p>however when i check if cuda is available i obtain:</p>
<pre><code class="lang-auto">print('-------------------------------')
print(torch.cuda.is_available())
print(torch.cuda.device_count())
print(torch.cuda.current_device())
print(torch.cuda.device(0))
print(torch.cuda.get_device_name(0))
print('Memory Usage:')
print('Allocated:', round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
print('Cached:   ', round(torch.cuda.memory_reserved(0)/1024**3,1), 'GB')
</code></pre>
<blockquote>
<p>True<br>
1<br>
0<br>
&lt;torch.cuda.device object at 0x7f8bf6d4a9b0&gt;<br>
GRID T4-16Q<br>
Memory Usage:<br>
Allocated: 0.0 GB<br>
Cached:    0.0 GB</p>
</blockquote>
<p>i run this code on colab and i do not have any issues, also i run the code on another machine with anoter gpu and in runs as expected,</p>
<p>the configuration of the machine in where i need to run and fails is</p>
<blockquote>
<p>(test310) admin@appdev-llm-lnx1:~/llm/ModelsService$ nvidia-smi<br>
Tue Jun 18 16:02:22 2024<br>
±----------------------------------------------------------------------------------------+<br>
| NVIDIA-SMI 550.90.07              Driver Version: 550.90.07      CUDA Version: 12.4     |<br>
|-----------------------------------------±-----------------------±---------------------+<br>
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |<br>
|                                         |                        |               MIG M. |<br>
|=========================================+========================+======================|<br>
|   0  GRID T4-16Q                    On  |   00000000:02:01.0 Off |                    0 |<br>
| N/A   N/A    P8             N/A /  N/A  |       1MiB /  16384MiB |      0%      Default |<br>
|                                         |                        |                  N/A |<br>
±----------------------------------------±-----------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------------------+<br>
| Processes:                                                                              |<br>
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |<br>
|        ID   ID                                                               Usage      |<br>
|=========================================================================================|<br>
|  No running processes found                                                             |<br>
±----------------------------------------------------------------------------------------+</p>
</blockquote>
<p>and the libraries</p>
<blockquote>
<p>accelerate               0.31.0<br>
aiohttp                  3.9.5<br>
aiosignal                1.3.1<br>
async-timeout            4.0.3<br>
attrs                    23.2.0<br>
certifi                  2024.6.2<br>
charset-normalizer       3.3.2<br>
datasets                 2.20.0<br>
dill                     0.3.8<br>
filelock                 3.15.1<br>
frozenlist               1.4.1<br>
fsspec                   2024.5.0<br>
huggingface-hub          0.23.4<br>
idna                     3.7<br>
Jinja2                   3.1.4<br>
MarkupSafe               2.1.5<br>
mpmath                   1.3.0<br>
multidict                6.0.5<br>
multiprocess             0.70.16<br>
networkx                 3.3<br>
ninja                    1.11.1.1<br>
numpy                    2.0.0<br>
nvidia-cublas-cu12       12.1.3.1<br>
nvidia-cuda-cupti-cu12   12.1.105<br>
nvidia-cuda-nvrtc-cu12   12.1.105<br>
nvidia-cuda-runtime-cu12 12.1.105<br>
nvidia-cudnn-cu12        8.9.2.26<br>
nvidia-cufft-cu12        11.0.2.54<br>
nvidia-curand-cu12       10.3.2.106<br>
nvidia-cusolver-cu12     11.4.5.107<br>
nvidia-cusparse-cu12     12.1.0.106<br>
nvidia-nccl-cu12         2.20.5<br>
nvidia-nvjitlink-cu12    12.5.40<br>
nvidia-nvtx-cu12         12.1.105<br>
packaging                24.1<br>
pandas                   2.2.2<br>
pip                      24.0<br>
psutil                   5.9.8<br>
pyarrow                  16.1.0<br>
pyarrow-hotfix           0.6<br>
python-dateutil          2.9.0.post0<br>
pytz                     2024.1<br>
PyYAML                   6.0.1<br>
quanto                   0.2.0<br>
regex                    2024.5.15<br>
requests                 2.32.3<br>
safetensors              0.4.3<br>
setuptools               65.5.0<br>
six                      1.16.0<br>
sympy                    1.12.1<br>
tokenizers               0.19.1<br>
torch                    2.3.1<br>
tqdm                     4.66.4<br>
transformers             4.42.0.dev0<br>
triton                   2.3.1<br>
typing_extensions        4.12.2<br>
tzdata                   2024.1<br>
urllib3                  2.2.2<br>
xxhash                   3.4.1<br>
yarl                     1.9.4</p>
</blockquote>
<p>i do not if this affect but the machine is a virtual machine with wmware under a vgpu, also i tried to run a simple nn just for checking if the problem id with the transformers libray but i obtained the same error when i tried to locate info to the gpu</p>
<pre><code class="lang-auto">    import torch
    import torch.nn as nn
    dev = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
    t1 = torch.randn(1,2)
    t2 = torch.randn(1,2).to(dev)
    print(t1)  # tensor([[-0.2678,  1.9252]])
    print(t2)  # tensor([[ 0.5117, -3.6247]], device='cuda:0')
    t1.to(dev)
    print(t1)  # tensor([[-0.2678,  1.9252]])
    print(t1.is_cuda) # False
    t1 = t1.to(dev)
    print(t1)  # tensor([[-0.2678,  1.9252]], device='cuda:0')
    print(t1.is_cuda) # True

    class M(nn.Module):
        def __init__(self):        
            super().__init__()        
            self.l1 = nn.Linear(1,2)

        def forward(self, x):                      
            x = self.l1(x)
            return x
    model = M()   # not on cuda
    model.to(dev) # is on cuda (all parameters)
    print(next(model.parameters()).is_cuda) # True
</code></pre>
<blockquote>
<p>Traceback (most recent call last):<br>
File “/home/admin/llm/ModelsService/test.py”, line 14, in <br>
t2 = torch.randn(1,2).to(dev)<br>
RuntimeError: CUDA error: operation not supported<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
</blockquote>
<p>by the way here info about my cuda</p>
<blockquote>
<p>(test310) admin@appdev-llm-lnx1:~/llm/ModelsService$ nvcc --version<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2019 NVIDIA Corporation<br>
Built on Sun_Jul_28_19:07:16_PDT_2019<br>
Cuda compilation tools, release 10.1, V10.1.243</p>
</blockquote>
<p>regards</p>

ptrblck: <aside class="quote no-group" data-username="ferscp" data-post="1" data-topic="204861">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/d78d45/48.png" class="avatar"> ferscp:</div>
<blockquote>
<p>i do not if this affect but the machine is a virtual machine with wmware under a vgpu, also i tried to run a simple nn just for checking if the problem id with the transformers libray but i obtained the same error when i tried to locate info to the gpu</p>
</blockquote>
</aside>
<p>It seems your VM setup has issues and you could try to run any other CUDA sample to make sure your setup is able to use the GPU.</p>

ferscp: <p>i run the comman:</p>
<pre><code class="lang-auto">(test310) scpadmin@appdev-llm-lnx1:~/llm/ModelsService$ lspci | grep -i nvidi
02:01.0 VGA compatible controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
</code></pre>
<p>also i tried the next code</p>
<pre><code class="lang-auto">cuda = torch.device('cuda')     # Default CUDA device
cuda0 = torch.device('cuda:0')
x = torch.tensor([1., 2.], device=cuda0)
y = torch.tensor([1., 2.]).cuda()
</code></pre>
<p>and i have the same result:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “/home/admin/llm/ModelsService/test.py”, line 13, in <br>
x = torch.tensor([1., 2.], device=cuda0)<br>
RuntimeError: CUDA error: operation not supported<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
</blockquote>
<p>however torch said that cuda is available</p>

ferscp: <p>ok, i will responds myself<br>
if someone has this problem just need to create the next variables</p>
<p>export CUDA_HOME=/usr/local/cuda<br>
export PATH=${CUDA_HOME}/bin:${PATH}<br>
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH</p>
<p>after that pytorch works</p>