How to increase GPU usage during training

luminoussin: <p>Hello, I just learned using pytorch recently and I have stumbled upon some problem. Somehow when I do training for some image producing model I realized that my GPU utilization is very low (1144/ 11000) around 10% of my GPU capabilities, my PC has GTX 1080Ti. When I asked my friend they said that I need to increase the batch size during data loading, so I increased it from 8 -&gt; 70 and it does make my GPU usage become (10000/11000) but it seems that the training time it took isn’t that different and to my surprise the output result is way worse compared to when I use batch size of 8. So is there any way to speed up my training process without affecting the output result? Thank you</p>

Eta_C: <ul>
<li>
<p>In <code>nvidia-smi</code>, <code>Memory-Usage</code> is how much GPU memory does this process use. <code>GPU-Util</code> reports what percentage of time one or more GPU kernel(s) was active for a given time perio. You say it seems that the training time isn’t different. Check <code>GPU-Util</code>.</p>
</li>
<li>
<p>In general, if you use <code>BatchNorm</code>, increasing the batchsize will lead to better results. Since the batchsize is increased by 9 times, if you still use the same hyperparameter configuration, such as <code>num_epochs</code>, you may need to check for overfitting (just a possible conjecture).</p>
</li>
</ul>

luminoussin: <p>Hi, Thank you for your reply. I forgot to mention that I checked using watch -n 1 nvidia-smi and the GPU-util part usually changes between 0-&gt;98-&gt;30-&gt;99 and around that. I also use a lot of BatchNorm on my model. I did finish another training session from scratch and this time I changed the batch size from 70 to 5 and the result is even better than when I used batch size of 8. So I just assume that bigger batch size would result  in worse result in my model but is there a way to speed things up just like what I asked on my post?</p>

Eta_C: <aside class="quote no-group" data-username="luminoussin" data-post="3" data-topic="66416">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luminoussin/48/19567_2.png" class="avatar"> luminoussin:</div>
<blockquote>
<p>the GPU-util part usually changes between 0-&gt;98-&gt;30-&gt;99 and around that</p>
</blockquote>
</aside>
<p>It seems that your GPUs are waiting for your CPUs. Maybe most of the time is spent loading and preprocessing data.</p>

luminoussin: <p>So what would you suggest to make it faster?</p>

Isaac_Kargar: <p>Related to this:</p><aside class="quote quote-modified" data-post="1" data-topic="33208">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raphaeltrench/48/11473_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-increase-gpu-usage/33208">How to increase GPU usage?</a> 
  </div>
  <blockquote>
    Hello, I recently started learning pytorch, and started to play around a bit with regression. 
For that, I made simple Neural Network, which I am training with a dataset consisting of roughly 780 000 samples, with 34 features. 
Since its alot of data, I wanted to use my GPU(NVIDIA 1050Ti) instead of CPU for training. 
However the training process still seems a bit slow for me, and when I checked GPU usage at the windows task manager, it was basically locked at 5%. 
I would like to know if its po…
  </blockquote>
</aside>


kaiseryet: <p>Once I read somewhere said using cudnn can accelerate training by increase GPU usage.</p>

chen_shao: <p>I am not an expert, but theoretically, larger batch size provides more accurate gradient estimation.</p>