Low GPU Utilization while training, should i buy better CPU?

Samuel_Bachorik: <p>Hi I bought new GPU - RTX 3060 12GB. While training on 480x640 RGB images (1200 images) with batch 32. I have very low GPU utilization.<br>
As you can see on this image,<br>
CPU Intel I5 10 th gen 6 core 3.8 ghz - Ulitization  98-100%<br>
I have 16 GB RAM 3200 Mhz - 14GB of 16 filled<br>
GPU - 11 of 12 GB memory filled and Ulitization only about 5%</p>
<p>My question -, is CPU or PC RAM limiting my GPU ? Is it possible to Utilize this GPU more and speed up training ? Should I buy new CPU or more memory ? ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c7e84e1736b3455c9bc82782bf5ca3d182b475f.png" data-download-href="https://discuss.pytorch.org/uploads/default/9c7e84e1736b3455c9bc82782bf5ca3d182b475f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c7e84e1736b3455c9bc82782bf5ca3d182b475f.png" alt="image" data-base62-sha1="mkpyaXbFWD8omtSQzrRauAj6Qr5" width="690" height="467" data-dominant-color="F2F3F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">923×625 35.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Nvidia SMI<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5eeb277c8492c6f6a194de503e9acac073785877.png" data-download-href="https://discuss.pytorch.org/uploads/default/5eeb277c8492c6f6a194de503e9acac073785877" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5eeb277c8492c6f6a194de503e9acac073785877.png" alt="image" data-base62-sha1="dxGHgAYm7HPHc3wV80eM19a5931" width="530" height="500" data-dominant-color="1A1A1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">647×610 16.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

tom: <p>Probably first look at why your data loading is slow by benchmarking which part causes it. Moving preprocessing to the GPU often is crucial, as is using a fast storage medium (ie SSD, not HDD). But it’s not optimization until you do your own perf measurements…</p>
<p>Best regards</p>
<p>Thomas</p>

Samuel_Bachorik: <p>Thank you for your help but please can you give me some specific TIP what to do ? My images loader is from internet so it should be good. Only PIL, numpy and os used there.</p>
<blockquote>
<p>Look at why your data loading is slow</p>
</blockquote>
<p>How do you know my data loading is slow ? Or what do you mean by that ?<br>
I am using M2 SSD.</p>

tom: <p>So dataloading to me here is everything inside the dataset and dataloader. Reading from disk, dataset augmentation etc.<br>
You can time it by running an epoch with just the loop and no model, loss, or backward. Then you can comment out bits and compare or use <code>time.perfcounter</code> or so for timing (or a profiler would work, too, of course).</p>

Samuel_Bachorik: <p>Hi <a class="mention" href="/u/tom">@tom</a>, I made test and this is result.</p>
<p>just for info-<br>
When I am loading photos every loop and pushing them to Model training the training lasts 5 hours(test on 700 images).</p>
<p><strong>Test 1: - Still slow</strong><br>
I deleted from training loop model, backward, loss, and left there only images loader, (no GPU work) - “training” lasts 4 hours.</p>
<p><strong>Test 2: – Extremely fast</strong><br>
I deleted from training loop images loader and I loaded batch only once before training loop. So GPU was working with only one batch whole training. <strong>And training lasts only 1 hour and GPU Utilization was 100% all time.</strong></p>
<p>From this we know, loading of images is slowing down whole training. When CPU is loading images for so long GPU does nothing.</p>
<p><strong>Can you now confirm or refute if this is weak CPU or bad images loader optimization  ?</strong></p>
<p>If you want to see this is images loader -<a href="https://github.com/Samuel-Bachorik/PyTorch-AI-road-detection-classification/blob/main/Model_loader.py" class="inline-onebox" rel="noopener nofollow ugc">PyTorch-AI-road-detection-classification/Model_loader.py at main · Samuel-Bachorik/PyTorch-AI-road-detection-classification · GitHub</a></p>

tom: <p>I would try to replace the PIL augmentation by one on GPU using torchvision or eg kornia.<br>
Also, you would probably do yourself a favour by using Dataset and Dataloader from PyTorch.</p>