It seems Pytorch doesn't use GPU

minsung: <p>First, i apologize for my poor English.</p>
<p>Recently, I bought RTX2060 for deep learning. I installed pytorch-gpu with <code>conda</code> by <code>conda install pytorch torchvision cudatoolkit=10.1 -c pytorch</code>. Of course, I setup NVIDIA Driver too.</p>
<p>But when i ran my pytorch code, it was so slow to train. So i checked task manger and it seems torch doesn’t using GPU at all!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a930a672c4e33d2b7193aaad7f01f2a1e2af931.png" data-download-href="https://discuss.pytorch.org/uploads/default/0a930a672c4e33d2b7193aaad7f01f2a1e2af931" title="캡처"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a930a672c4e33d2b7193aaad7f01f2a1e2af931.png" alt="캡처" data-base62-sha1="1vxO5f9it6UFstLjSxXzgyExhpD" width="557" height="500" data-dominant-color="F3F6F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">캡처</span><span class="informations">847×760 30.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Rather, as shown in picture, CPU was used highly more than GPU. It’s replying <code>true</code> for <code>torch.cuda.is_available()</code>, but overall training speed and task manager’s graph seems torch can’t utilize GPU well. Is there something I missed? Here are my  <code>nvidia-smi</code>.</p>
<p>Sun Mar 29 15:25:26 2020<br>
±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 431.91       Driver Version: 431.91       CUDA Version: 10.1     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|===============================+======================+======================|<br>
|   0  GeForce RTX 2060   WDDM  | 00000000:01:00.0 Off |                  N/A |<br>
| N/A   73C    P2    61W /  N/A |   2779MiB /  6144MiB |     46%      Default |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                       GPU Memory |<br>
|  GPU       PID   Type   Process name                             Usage      |<br>
|=============================================================================|<br>
|    0      2984      C   …01\Anaconda3\envs\PyTorchGPU\python.exe N/A      |<br>
|    0     10636      C   …01\Anaconda3\envs\PyTorchGPU\python.exe N/A      |<br>
±----------------------------------------------------------------------------+</p>

charan_Vjy: <p>Did you push your model to the GPU as shown below?</p>
<pre><code class="lang-auto">device = 'cuda'
model = model.to(device)
</code></pre>

minsung: <p>Yes. And I’m using <code>nn.Sequential</code> with <code>.to(device)</code></p>
<pre><code class="lang-auto">self.middle_fc = nn.Sequential(
            nn.Linear(35,1024), 
            nn.ReLU(),
            nn.Linear(1024,248),
            nn.ReLU(),
            nn.Linear(248,128),
            nn.ReLU()
        ).to(device)
</code></pre>

charan_Vjy: <p>I had actually missed the nvidia-smi output:<br>
It shows that 2779MiB / 6144MiB (46%) is being utilzed. This implies that the model was successfully loaded into the GPU. One empirical way to verify this is to time it using <code>device = 'cpu'</code> and then time it using <code>device = 'cuda'</code> and verify the different runtimes for a batch size greater than 1(Preferabbly, keep as high a batch size as possible). If the runtimes are the same, there is indeed some issue. Really do not have an answer for the 2% part, though.</p>

minsung: <p>I’ll be back with MNIST training😁</p>

ptrblck: <aside class="quote no-group" data-username="charan_Vjy" data-post="4" data-topic="74673">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/charan_vjy/48/13238_2.png" class="avatar"> charan_Vjy:</div>
<blockquote>
<p>Really do not have an answer for the 2% part, though.</p>
</blockquote>
</aside>
<p>I assume this points towards the shown GPU activity in the task manager.<br>
If so, note that the Windows task manager doesn’t show the <code>compute</code> tab by default and you would have to select it via the drop-down menu.<br>
I would recommend to stick to <code>nvidia-smi</code> to see the device utilization.</p>

minsung: <p>I found that when I run the code with <code>cuda</code>, <code>nvidia-smi</code> prints something different from <code>cpu</code>!</p>
<p>±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 431.91       Driver Version: 431.91       CUDA Version: 10.1     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|===============================+======================+======================|<br>
|   0  GeForce RTX 2060   WDDM  | 00000000:01:00.0 Off |                  N/A |<br>
| N/A   58C    P2    29W /  N/A |   5148MiB /  6144MiB |     11%      Default |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                       GPU Memory |<br>
|  GPU       PID   Type   Process name                             Usage      |<br>
|=============================================================================|<br>
|    0      2984      C   …01\Anaconda3\envs\PyTorchGPU\python.exe N/A      |<br>
|    0      5480    C+G   …01\Anaconda3\envs\PyTorchGPU\python.exe N/A      |<br>
|    0     10256      C   …01\Anaconda3\envs\PyTorchGPU\python.exe N/A      |<br>
|    0     10636      C   …01\Anaconda3\envs\PyTorchGPU\python.exe N/A      |<br>
±----------------------------------------------------------------------------+</p>
<p>Could you let me know what the ‘11%’ means?</p>

ptrblck: <p><code>11%</code> refer to the relative time spend in executing a kernel on the device given a reference timeframe.<br>
E.g. in the last 100ms, 11ms were spent in executing a kernel on the device.</p>
<p>You can push the GPU utilization by trying to push big workloads to the device, as e.g. a lot of small computations will add overhead of the kernel launches etc.</p>

minsung: <p>I understand! So thankful</p>

minsung: <p>I found that GPU is generally faster than CPU in my MNIST example.<br>
I think that the problem was the batch-size. It was so small so that difference between GPU and CPU was low. Thanks a lot!!</p>

Deepesh7: <p>Selecting the Cuda tab(instead of 3D,video encode or whatever is default) in task manager worked for me.</p>