Low GPU Usage during Training

Vishal_R: <p>Hi! I am training a Convnet to classify CIFAR10 images on RTX 3080 GPU. For some reason, when I look at the GPU usage in task manager, it shows 3% GPU usage as shown in the image.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/7/37ca8508d13b9d63f7f5c2521666cad11f24a523.png" data-download-href="https://discuss.pytorch.org/uploads/default/37ca8508d13b9d63f7f5c2521666cad11f24a523" title="GPU"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/7/37ca8508d13b9d63f7f5c2521666cad11f24a523.png" alt="GPU" data-base62-sha1="7Xy9KuSmw8lJ9aEcXSemBqdYOH1" width="690" height="222" data-dominant-color="F7F9FB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">GPU</span><span class="informations">1143×368 8.59 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The model is as follows</p>
<pre><code class="lang-auto">class ConvNet(nn.Module):
    
    def __init__(self):
        super(ConvNet,self).__init__()
        
        self.conv1 = nn.Conv2d(in_channels=3,out_channels=8,stride=1,kernel_size=(3,3),padding=1)
        self.conv2 = nn.Conv2d(in_channels=8,out_channels=32,kernel_size=(3,3),padding=1,stride=1)
        self.conv3 = nn.Conv2d(in_channels=32,out_channels=64,kernel_size=(3,3),padding=1,stride=1)
        self.conv4 = nn.Conv2d(in_channels=64,out_channels=128,kernel_size=(3,3),padding=1,stride=1)
        self.conv5 = nn.Conv2d(in_channels=128,out_channels=256,kernel_size=(3,3),stride=1)

        self.fc1 = nn.Linear(in_features=6*6*256,out_features=256)
        self.fc2 = nn.Linear(in_features=256,out_features=128)
        self.fc3 = nn.Linear(in_features=128,out_features=64)
        self.fc4 = nn.Linear(in_features=64,out_features=10)
        
        self.max_pool = nn.MaxPool2d(kernel_size=(2,2),stride=2)
        self.dropout = nn.Dropout2d(p=0.5)
        
    def forward(self,x,targets):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = self.max_pool(x)
        x = self.conv3(x)
        x = F.relu(x)
        x = self.dropout(x)
        x = self.conv4(x)
        x = F.relu(x)
        x = self.max_pool(x)
        x = self.conv5(x)
        x = F.relu(x)
        x = self.dropout(x)
        x = x.view(-1,6*6*256)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = F.relu(x)
        x = self.fc3(x)
        x = F.relu(x)
        logits = self.fc4(x)
        
        loss = None
        if targets is not None:
            loss = F.cross_entropy(logits,targets)
        return logits,loss
    
    def configure_optimizers(self,config):
        optimizer = optim.Adam(self.parameters(),lr=config.lr,betas=config.betas,weight_decay=config.weight_decay)
        return optimizer
</code></pre>
<p>Training Configurations are as follows:<br>
Epochs : 300<br>
Batch Size : 64<br>
Weight Decay : 7.34e-4<br>
Learning Rate : 3e-4<br>
Optimizer : Adam</p>
<p>Also I am running several transforms such as Normalization, RandomRotation, RandomHorizontalFlips.</p>
<p>Also I have another bug. When I change the number of workers in DataLoader, the training just begin at all. In jupyter notebook, it shows that cell is being executed but no output is shown. So I am forced to run with <code>num_workers=0</code>. Anything above 0 breaks for some reason.</p>

jmandivarapu1: <aside class="quote no-group" data-username="Vishal_R" data-post="1" data-topic="121588">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vishal_r/48/36271_2.png" class="avatar"> Vishal_R:</div>
<blockquote>
<p>all. In jupyter notebook, it shows that cell is being executed but no output is shown. So I am forced to run with <code>num_workers=0</code>. Anything above 0 breaks for some reason.</p>
</blockquote>
</aside>
<p>try typing</p>
<pre><code class="lang-auto">watch nvidia-smi
</code></pre>
<p>in your shell at the same time while training is happening it will show the real memory usage of your model and utilization.</p>

Vishal_R: <p>I am on Windows 10. I tried using <code>watch nvidia-smi</code> in notebook. It gives syntax error.</p>
<p>Update:</p>
<p>I managed to find a command to get GPU stats. It shows that it is using 14% GPU. Isn’t that low? I am training a big model right?</p>

jmandivarapu1: <p>In jupyternotebook when you press new then click terminal, tyoe the comman there<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/e/0e1789bbc9a3225f48231b178e8f7804e912416a.png" data-download-href="https://discuss.pytorch.org/uploads/default/0e1789bbc9a3225f48231b178e8f7804e912416a" title="Screen Shot 2021-05-18 at 12.56.13 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0e1789bbc9a3225f48231b178e8f7804e912416a_2_690x149.png" alt="Screen Shot 2021-05-18 at 12.56.13 AM" data-base62-sha1="20F76o4FeqXxMWvZwSi6g3GLMwy" width="690" height="149" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0e1789bbc9a3225f48231b178e8f7804e912416a_2_690x149.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0e1789bbc9a3225f48231b178e8f7804e912416a_2_1035x223.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0e1789bbc9a3225f48231b178e8f7804e912416a_2_1380x298.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-05-18 at 12.56.13 AM</span><span class="informations">2832×614 54 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

jmandivarapu1: <aside class="quote no-group" data-username="Vishal_R" data-post="1" data-topic="121588">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vishal_r/48/36271_2.png" class="avatar"> Vishal_R:</div>
<blockquote>
<p>Batch Size : 64</p>
</blockquote>
</aside>
<p>Yeah even though you use bigger model it depends on the batch size and total computation done by the GPU. Try increasing the batch size and do watch nvidia-smi to see continuously on the memory and utilization.</p>

Vishal_R: <p>I increased batch_size to 256. The GPU usage is now 10%. It was 14% when batch_size was 64.</p>
<p>Also I have seen in many YouTube videos that if we use a very large batch size the overall generalization of the model decreases and hence the validation accuracy goes down. Is that true?</p>

jmandivarapu1: <p><a href="https://discuss.pytorch.org/t/low-gpu-utilization-problem/11527/2">Low GPU utilization problem - PyTorch Forums</a></p>
<p>As you see the link you need to increase the num_workers. As that might be one of the cause</p>

Vishal_R: <p>Yeah.</p>
<p>But if I increase the num_workers to say like 2 or something, for some reason, it breaks the training process. It doesn’t start training at all. It only trains when <code>num_workers=0</code>. I don’t know why it is happening.</p>

Vishal_R: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4e73ee9bc596f8f1ed61393295547989593172c4.png" data-download-href="https://discuss.pytorch.org/uploads/default/4e73ee9bc596f8f1ed61393295547989593172c4" title="Workers"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4e73ee9bc596f8f1ed61393295547989593172c4_2_690x391.png" alt="Workers" data-base62-sha1="bc1COkIE29umtCZ5hGKC0WyaDn6" width="690" height="391" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4e73ee9bc596f8f1ed61393295547989593172c4_2_690x391.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4e73ee9bc596f8f1ed61393295547989593172c4_2_1035x586.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4e73ee9bc596f8f1ed61393295547989593172c4_2_1380x782.png 2x" data-dominant-color="F1F2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Workers</span><span class="informations">1918×1088 142 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is the problem I am getting if I change the <code>num_workers</code> to anything above <code>0</code>. I don’t know why it doesn’t work for me.</p>
<p>Also I have seen some YouTube videos suggesting to keep the <code>batch_size</code> to 32,64 like that. They tell not to use very large batch_sizes as it reduces the generalization of the model. Is it true?</p>

jmandivarapu1: <p>Yeah generalization of model dependent and according to me depend on the no of classes you have in your dataset. So it is mostly dependent on the type of the dataset you have in hand. But ideally 32,64,128,256 works depends on datasets. If someone have very big images they will use batch sizes like 4 4,8,16 also because of memory constraints.</p>

jmandivarapu1: <p>I am not sure about which trainer you are using.</p>

Vishal_R: <p>I am using a custom training loop that I found on Andrej Karpathy’s MinGPT repo. I thought it was a nice way of doing it. Even if I did not use that trainer and used a simple training loop, the DataLoader with <code>num_workers&gt;0</code> doesn’t work.</p>
<p>Link : <a href="https://github.com/karpathy/minGPT/blob/master/mingpt/trainer.py" rel="noopener nofollow ugc">trainer.py</a></p>

jmandivarapu1: <p>can you print loss after line 83</p>
<pre><code class="lang-auto">print("---",loss.item())
</code></pre>

Vishal_R: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/6/e63f9bf33da01ee87cb6427f046195d8bde88b8d.png" data-download-href="https://discuss.pytorch.org/uploads/default/e63f9bf33da01ee87cb6427f046195d8bde88b8d" title="loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e63f9bf33da01ee87cb6427f046195d8bde88b8d_2_690x220.png" alt="loss" data-base62-sha1="wQS6dkQDOdA73Qs0ijp38PmyUKp" width="690" height="220" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e63f9bf33da01ee87cb6427f046195d8bde88b8d_2_690x220.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e63f9bf33da01ee87cb6427f046195d8bde88b8d_2_1035x330.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e63f9bf33da01ee87cb6427f046195d8bde88b8d_2_1380x440.png 2x" data-dominant-color="F0E9E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss</span><span class="informations">1904×609 85.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As you mentioned, I have printed loss after the loss.mean() line in the trainer.</p>

jmandivarapu1: <p>increase the batch size and workers and see if the loss is still printing</p>

Vishal_R: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54c7dadb0e39f44bfb32f32c50e13df5e5390994.png" data-download-href="https://discuss.pytorch.org/uploads/default/54c7dadb0e39f44bfb32f32c50e13df5e5390994" title="workers-1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54c7dadb0e39f44bfb32f32c50e13df5e5390994_2_690x62.png" alt="workers-1" data-base62-sha1="c60igJLG8KOs5N6QYxZIYZAx6IY" width="690" height="62" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54c7dadb0e39f44bfb32f32c50e13df5e5390994_2_690x62.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54c7dadb0e39f44bfb32f32c50e13df5e5390994_2_1035x93.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54c7dadb0e39f44bfb32f32c50e13df5e5390994_2_1380x124.png 2x" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">workers-1</span><span class="informations">1859×169 25.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>No it does not print anything. It gets stuck like this and I cannot interrupt the kernel as well.</p>

jmandivarapu1: <p>can you change batchsize to like 4 or 8 or 16</p>

Vishal_R: <p>I am still getting the same result.</p>
<p>I tried running an earlier project that I did for training MNIST digits. There I changed <code>num_workers=2</code> and ran it in terminal instead of running on notebook.</p>
<p>This is what I have got.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f2f7ec1d99bd7053b2f1e50f735b4bcfd06c06e.png" data-download-href="https://discuss.pytorch.org/uploads/default/8f2f7ec1d99bd7053b2f1e50f735b4bcfd06c06e" title="error"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f2f7ec1d99bd7053b2f1e50f735b4bcfd06c06e.png" alt="error" data-base62-sha1="kqG2eYvaP6BrTRIWbBk44m67yai" width="568" height="500" data-dominant-color="191919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error</span><span class="informations">1218×1071 55.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have used PyTorch’s MNIST dataset itself and trained using the trainer class.</p>

jmandivarapu1: <p>can you share your complete mnist code here?</p>

Vishal_R: <p>I have uploaded it to my github.</p>
<p>Link : <a href="https://github.com/iVishalr/MNIST-PyTorch" rel="noopener nofollow ugc">MNIST-PyTorch</a></p>