RuntimeError: CUDA error: CUDA-capable device(s) is/are busy or unavailable

Abdellah_Bousouf: <p>Hello,</p>
<p>i’m woring with <span class="hashtag-raw">#kubernetes</span> so i install nvidia driver 470, then i install cuda toolkit 11.8 in my worker in kubernities</p>
<p>i’m trying to run a python code in my pod to test my GPUs, so first i install the following libraries</p>
<p>• pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url <a href="https://download.pytorch.org/whl/cu118" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu118</a></p>
<h5><a name="p-462217-h-1" class="anchor" href="#p-462217-h-1"></a></h5>
<h1><a name="p-462217-this-is-the-code-that-im-trying-to-compile-2" class="anchor" href="#p-462217-this-is-the-code-that-im-trying-to-compile-2"></a>this is the code that i’m trying to compile:</h1>
<h5><a name="p-462217-h-3" class="anchor" href="#p-462217-h-3"></a></h5>
<p>import torch<br>
import time</p>
<p>def test_gpu():</p>
<h1><a name="p-462217-check-if-gpu-is-available-4" class="anchor" href="#p-462217-check-if-gpu-is-available-4"></a>Check if GPU is available</h1>
<p>if not torch.cuda.is_available():<br>
print(“No GPU detected. Please ensure PyTorch is installed with GPU support and a compatible GPU is available.”)<br>
return</p>
<p>print(“GPU detected! Details:”)<br>
print(f"GPU Name: {torch.cuda.get_device_name(0)}“)<br>
print(f"CUDA Capability: {torch.cuda.get_device_capability(0)}”)<br>
print(f"Total Memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.2f} GB")</p>
<h1><a name="p-462217-perform-a-simple-computation-to-stress-test-the-gpu-5" class="anchor" href="#p-462217-perform-a-simple-computation-to-stress-test-the-gpu-5"></a>Perform a simple computation to stress test the GPU</h1>
<p>print(“\nStarting GPU stress test…”)<br>
matrix_size = 50<br>
print(f"Matrix size: {matrix_size} x {matrix_size}")</p>
<h1><a name="p-462217-create-random-matrices-on-gpu-6" class="anchor" href="#p-462217-create-random-matrices-on-gpu-6"></a>Create random matrices on GPU</h1>
<p>device = torch.device(“cuda”)<br>
a = torch.rand((matrix_size, matrix_size), device=device)<br>
b = torch.rand((matrix_size, matrix_size), device=device)</p>
<h1><a name="p-462217-perform-matrix-multiplication-7" class="anchor" href="#p-462217-perform-matrix-multiplication-7"></a>Perform matrix multiplication</h1>
<p>start_time = time.time()<br>
result = torch.mm(a, b)<br>
torch.cuda.synchronize() # Ensure computation is finished<br>
end_time = time.time()</p>
<p>print(“GPU stress test completed.”)<br>
print(f"Time taken for matrix multiplication: {end_time - start_time:.2f} seconds")</p>
<p>if <strong>name</strong> == “<strong>main</strong>”:<br>
test_gpu()</p>
<p>###################<br>
this is the result:<br>
###################</p>
<p>(base) jovyan@testgpu2222222-0:~$ python /home/jovyan/test2.py<br>
GPU detected! Details:<br>
GPU Name: NVIDIA A10-6C<br>
CUDA Capability: (8, 6)<br>
Total Memory: 6.27 GB<br>
Starting GPU stress test…<br>
Matrix size: 50 x 50</p>
<p>File “/home/jovyan/test2.py”, line 22, in test_gpu<br>
a = torch.rand((matrix_size, matrix_size), device=device)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
RuntimeError: CUDA error: CUDA-capable device(s) is/are busy or unavailable<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.</p>
<p>as u can see the pod can detect gpu, but while running the python code inside the pod we get this error:</p>
<p>RuntimeError: CUDA error: CUDA-capable device(s) is/are busy or unavailable</p>
<p>Even no process is running and the GPU is in the default mode.</p>

johann-petrak: <p>Getting this intermittently again and again too and there is no indication what could be going on as the GPU is not actually used that much:  only 1 of 4G are used according to nvidia-smi<br>
This is extremely annoying and I have found quite a few reports on the internet where people have the same problem but no real reproducable solution or even an explanation of what is going on.</p>