How can I be sure that PyTorch 2.1.2 is working properly with cuda 12.1?

Keiku: <p>I am using an NVIDIA GeForce RTX 4060 Ti in Ubuntu 22.04 and <code>torch.cuda.is_available()</code> is <code>True</code>, but when I watch the GPU usage, it never gets close to 100%. It just hovers around 0 to 40%. How can I make sure that PyTorch 2.1.2 is working properly in the following environment?</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0
</code></pre>
<pre><code class="lang-auto">Wed Jun 26 22:03:29 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.42.02              Driver Version: 555.42.02      CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4060 Ti     Off |   00000000:01:00.0  On |                  N/A |
|  0%   53C    P3             21W /  165W |    1493MiB /  16380MiB |      7%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      2439      G   /usr/lib/xorg/Xorg                            755MiB |
|    0   N/A  N/A      2568      G   /usr/bin/gnome-shell                          133MiB |
|    0   N/A  N/A      4560      G   ...36,262144 --variations-seed-version        515MiB |
+-----------------------------------------------------------------------------------------+
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Keiku" data-post="1" data-topic="205341">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/keiku/48/36340_2.png" class="avatar"> Keiku:</div>
<blockquote>
<p>How can I make sure that PyTorch 2.1.2 is working properly in the following environment?</p>
</blockquote>
</aside>
<p>Your GPU is already working given you can allocate tensors on the device and are seeing a GPU utilization.<br>
If you are concerned about the 0-40% util. you could profile your code to check if you are CPU-limited or what exactly is slowing down your code leaving the GPU idle.</p>

Keiku: <p>Thank you for your comment. I would like to see if it is a code problem.</p>
<p>If you have any other recommended stable cuda or driver versions for the 4060 Ti, I would appreciate it if you could let me know.</p>

MyCenturaHealth: <aside class="quote no-group" data-username="Keiku" data-post="1" data-topic="205341" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/keiku/48/36340_2.png" class="avatar"> Keiku:</div>
<blockquote>
<p>I am using an NVIDIA GeForce RTX 4060 Ti in Ubuntu 22.04 and <code>torch.cuda.is_available()</code> is <code>True</code>, but when I watch the GPU usage, it never gets close to 100%. It just hovers around 0 to 40%. How can I make sure that PyTorch 2.1.2 is working properly in the following environment?</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0
</code></pre>
<pre><code class="lang-auto">Wed Jun 26 22:03:29 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.42.02              Driver Version: 555.42.02      CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. | 
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4060 Ti     Off |   00000000:01:00.0  On |                  N/A |
|  0%   53C    P3             21W /  165W |    1493MiB /  16380MiB |      7%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      2439      G   /usr/lib/xorg/Xorg                            755MiB |
|    0   N/A  N/A      2568      G   /usr/bin/gnome-shell                          133MiB |
|    0   N/A  N/A      4560      G   ...36,262144 --variations-seed-version        515MiB |
+-----------------------------------------------------------------------------------------+
</code></pre>
</blockquote>
</aside>
<p>To ensure PyTorch 2.1.2 works with CUDA 12.1, run intensive CUDA operations and check for errors or performance issues. Use torch.cuda.is_available() to confirm GPU usage, and monitor GPU utilization with tools like NVIDIA-SMI for detailed insights.</p>