Why is my val and training accuracy so low?

username: xsquared
cooked: <p>Hello,</p>
<p>I’d like to know why my validation and training accuracy are so low; in fact, they’re the same. I am working on <code>torchvisions</code> MNIST dataset.</p>
<p><code>Accuracy train: 0.10; Accuracy val: 0.10</code></p>
<p>Here is the model:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    """A representation of a convolutional neural network comprised of VGG blocks."""
    def __init__(self, n_channels):
        super(Net, self).__init__()
        # VGG block 1
        self.conv1 = nn.Conv2d(n_channels, 64, (3,3))
        self.act1 = nn.ReLU()
        self.pool1 = nn.MaxPool2d((2,2), stride=(2,2))
        self.dropout = nn.Dropout(0.2)
        # VGG block 2
        self.conv2 = nn.Conv2d(64, 64, (3,3))
        self.act2 = nn.ReLU()
        self.pool2 = nn.MaxPool2d((2,2), stride=(2,2))
        self.dropout2 = nn.Dropout(0.2)
        # VGG block 3
        self.conv3 = nn.Conv2d(64, 128, (3,3))
        self.act3 = nn.ReLU()
        self.pool3 = nn.MaxPool2d((2,2), stride=(2,2))
        self.dropout3 = nn.Dropout(0.2)
        # Fully connected layer
        self.f1 = nn.Linear(128 * 1 * 1, 1000)
        self.dropout4 = nn.Dropout(0.5)
        self.act4 = nn.ReLU()
        # Output layer
        self.f2 = nn.Linear(1000, 10)
        self.act5 = nn.Softmax(dim=1)

    def forward(self, X):
        """This function forward propagates the input."""
        # VGG block 1
        X = self.conv1(X)
        X = self.act1(X)
        X = self.pool1(X)
        X = self.dropout(X)
        # VGG block 2
        X = self.conv2(X)
        X = self.act2(X)
        X = self.pool2(X)
        X = self.dropout2(X)
        # VGG block 3
        X = self.conv3(X)
        X = self.act3(X)
        X = self.pool3(X)
        X = self.dropout3(X)
        # Flatten
        X = X.view(-1, 128)
        # Fully connected layer
        X = self.f1(X)
        X = self.act4(X)
        X = self.dropout4(X)
        # Output layer
        X = self.f2(X)
        X = self.act5(X)

        return X
</code></pre>
<p>and here is how I am computing accuracy:</p>
<pre><code class="lang-auto">def validate(model, train_loader, val_loader):
    for name, loader in [("train", train_loader), ("val", val_loader)]:
        correct = 0
        total = 0

        with torch.no_grad():
            for imgs, labels in loader:
                outputs = model(imgs)
                _, predicted = torch.max(outputs, dim=1)
                total += labels.shape[0]
                correct += int((predicted == labels).sum())
                
        print("Accuracy {}: {:.2f}".format(name, correct / total))
</code></pre>

username: mmg
cooked: <aside class="quote no-group" data-username="xsquared" data-post="1" data-topic="113423">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/a5b964/48.png" class="avatar"> xsquared:</div>
<blockquote>
<p><code>with torch.no_grad():</code></p>
</blockquote>
</aside>
<p>You are not using any gradients for both the train and val loaders. So no gradient descent =&gt; no updates!. Try making separate loops for both loaders: train_loader must not have <code>no_grad()</code>.</p>
<p>Hope that helps!</p>

username: xsquared
cooked: <p>I am following the book “Deep Learning with Pytorch” and it provides <code>validates</code>. If I run the training loop first will it help? Either way, could you provide an code example to illustrate your point? thanks</p>

username: mmg
cooked: <aside class="quote no-group" data-username="xsquared" data-post="1" data-topic="113423">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/a5b964/48.png" class="avatar"> xsquared:</div>
<blockquote>
<p><code>for imgs, labels in loader:</code></p>
</blockquote>
</aside>
<p>Are you running the code on page 214 Section 8.4.1? That block has been annotated as “We do not want gradients…”. That code is just for measuring accuracy, not training.</p>
<p>Use the <code>training_loop</code> function (pg 215) - that is the one for training. The github repo @ <a href="https://github.com/deep-learning-with-pytorch/dlwpt-code" rel="noopener nofollow ugc">https://github.com/deep-learning-with-pytorch/dlwpt-code</a> will help you out.</p>

username: xsquared
cooked: <p>Yes, I am following the code in pg 214.</p>
<p>So, If I run the <code>training_loop</code> function before I run <code>validate</code> I will get a better accuracy?</p>
<p>the <code>training_loop</code> function only outputs the loss not the accuracy.</p>
<p>What I am trying to say is if  the<code>training_loop</code> function changes the state of <code>model</code>? <code>model</code> will then be fed to <code>validate</code>. Is this whats happening?</p>

username: mmg
cooked: <p>Run the <code>validate</code> function to get a baseline accuracy. This is what the model would predict without training. Then train the model using the <code>training_loop</code> . Then measure the accuracy again using the <code>validate</code> function. It will be higher than 10%.</p>
<p>The code in github does exactly that. I hope you will take a look at that very soon  <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: mmg
cooked: <aside class="quote no-group" data-username="xsquared" data-post="5" data-topic="113423">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/a5b964/48.png" class="avatar"> xsquared:</div>
<blockquote>
<p>What I am trying to say is if the<code>training_loop</code> function changes the state of <code>model</code>? <code>model</code> will then be fed to <code>validate</code>. Is this whats happening?</p>
</blockquote>
</aside>
<p>Yes. The model’s weight get update after each epoch of the <code>training_loop</code> function. Its values get updated. So the <code>validate</code> function will automatically see those updated values when you pass model as an argument</p>