Very low GPU utilization when training YOLOV8

mahdiHash: <p>I’m training a model on a small dataset (139 images, total size of 14MB, stored on HDD) for an object detection project. I have a GTX 1650 with 4GB memory on my laptop and I’m using YOLOV8 pre-trained models. I start the training using <code>yolo detect train [arguments]</code> with these arguments:</p>
<ul>
<li><code>data=config.yaml</code></li>
<li><code>model=yolov8m.pt</code></li>
<li><code>epochs=3</code></li>
<li><code>imgsz=640</code></li>
<li><code>batch=4</code> (can’t set to more than 4, memory issue)</li>
<li><code>workers=8</code></li>
<li><code>device=0</code> (no difference if it’s omitted or not)</li>
</ul>
<p>I’ve checked whether PyTorch recognizes my GPU using the code below, and it does:</p>
<pre data-code-wrap="py"><code class="lang-py">import torch
torch.cuda.is_available() // True
</code></pre>
<p>But during the training process, the GPU utilization is 0-2%, GPU memory 3/4GB, CPU usage 10-16%, and RAM usage 14/16GB. Why is GPU utilization so low? Am I doing somehting wrong?</p>
<ul>
<li>Windows 10</li>
<li>CUDA 12.1</li>
<li>PyTorch 2.4.0</li>
<li>Ultralytics 8.2.77</li>
<li>torchvision 0.19.0</li>
</ul>
<p>A screenshot of task manager during training:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b35db8224d07fed6909325b361fe3e786206d7bf.png" data-download-href="https://discuss.pytorch.org/uploads/default/b35db8224d07fed6909325b361fe3e786206d7bf" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b35db8224d07fed6909325b361fe3e786206d7bf.png" alt="Capture" data-base62-sha1="pAKg6MbwHr6dgdGTp5fHYZSyjKv" width="561" height="500" data-dominant-color="F4F6F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">834×742 30.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Soumya_Kundu: <p>Not sure if I can pinpoint but older GPUs may not be able to maximally utilise the newer versions I guess.</p>
<p>Plus your data is quite small as well.</p>
<p>Another thing could be your torch environment itself?</p>
<p>Look into the issues mentioned on the YOLOv8 repo. Quite a few people have faced this and there isn’t any one solution unfortunately.</p>

ptrblck: <aside class="quote no-group" data-username="mahdiHash" data-post="1" data-topic="208216">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahdihash/48/72189_2.png" class="avatar"> mahdiHash:</div>
<blockquote>
<p>But during the training process, the GPU utilization is 0-2%</p>
</blockquote>
</aside>
<p>You need to either enable the <code>Compute</code> view in your Task Manager or use <code>nvidia-smi</code> to see the GPU utilization on Windows.</p>

Soumya_Kundu: <p>Not a windows user — but why does one have explicitly open the compute view? Is it not automatically tracked?</p>

ptrblck: <p>As you can see in the posted picture, it’s not as only <code>3D</code>, <code>Copy</code>, <code>Video Encode/Decode</code> are shown. I’m also not using Windows, but know that the default view does not show any CUDA compute kernels.<br>
Based on <a href="https://discuss.pytorch.org/t/no-gpu-utilization-although-cuda-seems-to-be-activated/40964/26">this post</a> it seems a <code>CUDA</code> view might also be available, but unsure.</p>
<p>I would recommend sticking to <code>nvidia-smi</code> as it will show the GPU utilization properly.</p>

Soumya_Kundu: <p>Ah okay. Thanks! <code>nvidia-smi</code> (and nvitop) is my goto anyway.</p>

s.sni: <p>It seems a few months after my post, a new feature was added to windows 10 (also 11) called " <strong>Hardware-Accelerated GPU Scheduling</strong> ."<br>
When this feature is on, CUDA is not visible as one of the dropdown menu options in GPU tab in Task Manager (<a href="https://answers.microsoft.com/en-us/windows/forum/all/cuda-not-in-task-manager/a65eed92-828f-4d92-b9c9-cb2666bdd87f" rel="noopener nofollow ugc">Source</a>). Thus, turning off this feature by following these steps (in windows 11) will probably return CUDA back to Task Manager:</p>
<ol>
<li>Open Windows Setting &gt; System &gt; Display&gt; Graphics</li>
<li>Select  <strong>Change default graphics settings</strong></li>
<li>Turn off " <strong>Hardware-Accelerated GPU Scheduling</strong> ."</li>
<li>Reboot the system.</li>
</ol>
<p><strong>Note:</strong>  These steps might be different in windows 10. Googling will easily show you how to turn it off.</p>