Poor Performance for the MNIST Digits problem (using MSELoos and SGD)

username: akib62
cooked: <p>I want to consider MNIST digits as a regression problem (as we can do for the <code>house price prediction</code>).</p>
<p>I used <code>MSELoos</code> and <code>SGD</code> optimizer. The last layer of the CNN model is linear with one neuron. The structure of the model is given below</p>
<pre><code class="lang-auto">class CNN(nn.Module):
        def __init__(self):
            super(CNN, self).__init__()
            self.conv1 = nn.Sequential(        
                nn.Conv2d(in_channels=1, out_channels=16, kernel_size=5,stride=1,padding=2,),                              
                nn.ReLU(),                      
                nn.MaxPool2d(kernel_size=2),    
            )
            self.conv2 = nn.Sequential(        
                nn.Conv2d(16, 32, 5, 1, 2),    
                nn.ReLU(),                      
                nn.MaxPool2d(2),                
            )
            self.out = nn.Linear(32 * 7 * 7, 1)
        def forward(self, x):
            x = self.conv1(x)
            x = self.conv2(x)
            x = x.view(x.size(0), -1)      
            output = self.out(x)
            return output, x
</code></pre>
<p>Train code snippets</p>
<pre><code class="lang-auto">optimizer = optim.SGD(params=model.parameters(), lr=LR)
criterion = nn.MSELoss()

def train(NB_EPOCS, model, loaders):
    model.train()
    total_step = len(loaders['train'])
    for epoch in range(NB_EPOCS):
        for i, (images, labels) in enumerate(loaders['train']):
            b_x = Variable(images)   # batch x
            b_y = Variable(labels)   # batch y
            output, _ = model(b_x)  
            loss = criterion(output, b_y.float())
            optimizer.zero_grad()          
            loss.backward()    
            optimizer.step()            
            if (i+1) % 100 == 0:
                print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                       .format(epoch + 1, NB_EPOCS, i + 1, total_step, loss.item()))
</code></pre>
<p>I am getting a warning</p>
<pre><code class="lang-auto">UserWarning: Using a target size (torch.Size([100])) that is different to the input size (torch.Size([100, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
</code></pre>
<p>The loss is very high</p>
<pre><code class="lang-auto">Epoch [1/10], Step [100/600], Loss: 7.7500
Epoch [1/10], Step [200/600], Loss: 8.3003
Epoch [1/10], Step [300/600], Loss: 8.7280
Epoch [1/10], Step [400/600], Loss: 8.4920
Epoch [1/10], Step [500/600], Loss: 8.5399
Epoch [1/10], Step [600/600], Loss: 8.8300
Epoch [2/10], Step [100/600], Loss: 10.8930
Epoch [2/10], Step [200/600], Loss: 10.0020
Epoch [2/10], Step [300/600], Loss: 7.9896
Epoch [2/10], Step [400/600], Loss: 7.2748
Epoch [2/10], Step [500/600], Loss: 9.4017
</code></pre>
<p>However, at the time of testing, the model is predicting all images as 0.</p>
<pre><code class="lang-auto">Test Accuracy of the model on the 10000 test images: %.2f 0.16
Prediction Number: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
Actual Number [3 8 1 8 3 0 3 9 0 9 3 3 3 1 2 9 7 4 9 1 4 7 4 7 4 3 7 3 1 4 1 7 4 1 6 2 5
 0 0 6 8 8 3 2 5 1 6 3 9 3 8 1 4 1 7 7 5 8 3 2 0 4 3 5 9 3 9 4 7 6 0 7 2 3
 9 2 6 7 3 5 6 8 2 3 7 2 6 5 6 3 6 4 5 0 0 7 0 7 6 6]
</code></pre>
<p>I think the warning is one of the reasons for the poor performance. Could you tell me how can I resolve the warning and improve the performance?</p>

username: Unity05
cooked: <p>Hi,<br>
you need to squeeze your input tensor, so that input and target are of same size.<br>
Just like the warning says: broadcasting ruins your loss computation.<br>
In your specific case, PyTorch would broadcast both tensors to tensors of size ([100, 100]).</p>

username: akib62
cooked: <p><a class="mention" href="/u/unity05">@Unity05</a> could you tell me, how I have to do squeeze?</p>

username: Unity05
cooked: <p><code>loss = criterion(output.squeeze(-1), b_y.float())</code><br>
Btw, Variables are deprecated. Nowadays, normal tensors can have gradients as well.</p>

username: akib62
cooked: <p><a class="mention" href="/u/unity05">@Unity05</a>  Thanks a lot. It works. Now the loss is not terrible! And the training phase is not bad,</p>
<p>Training phase prediction</p>
<pre><code class="lang-auto">[ 7.6779],
        [ 4.2894],
        [ 4.9621],
        [ 3.0036],
        [ 1.2166],
        [ 4.3545],
        [ 6.1130],
        [ 8.3136],
        [ 6.8330],
        [ 2.3176],
        [ 4.7833],
        [-0.7525],
        [ 3.9115],
        [ 3.6507]]
</code></pre>
<p>Training phase actual</p>
<pre><code class="lang-auto">[9, 6, 6, 2, 1, 2, 8, 9, 7, 3, 6, 0, 5, 5]
</code></pre>
<p>But at the time of testing I am getting 0 not any numbers!!</p>
<pre><code class="lang-auto">Test Accuracy of the model on the 10000 test images: %.2f 0.05
Prediction Number: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
Actual Number [9 0 5 4 9 4 9 9 6 4 8 6 3 3 7 8 8 9 6 6 7 2 8 1 6 6 7 9 2 0 1 8 9 4 3 2 6
 6 0 1 5 2 7 2 8 2 0 5 7 4 6 2 4 2 1 2 8 2 2 9 6 8 1 7 3 4 1 4 2 3 1 6 4 8
 7 0 2 5 5 1 4 4 5 1 1 3 7 1 9 0 7 4 2 0 7 0 2 8 5 3]
</code></pre>
<p>Any idea, why this is happening? Or I am doing something wrong at the time of testing?</p>
<pre><code class="lang-auto">#Loading Model
model = my_model.get_model()
model.load_state_dict(torch.load(os.path.join(CHECKPOINT, "baseline.h5")))

#loading test data
loaders = preprocess.data_loaders()

def test():
    # Test the model
    with torch.no_grad():
        correct = 0
        total = 0
        for images, labels in loaders['test']:
            test_output, last_layer = model(images)
            pred_y = torch.max(test_output, 1)[1].data.squeeze()
            accuracy = (pred_y == labels).sum().item() / float(labels.size(0))
            
    print('Test Accuracy of the model on the 10000 test images: %.2f', accuracy)
    
    
test()
</code></pre>
<p>This is the actual prediction from test images</p>
<pre><code class="lang-auto">        [ 7.0612],
        [ 4.3061],
        [ 6.4712],
        [ 0.7163],
        [ 3.1995],
        [ 2.7388],
        [ 3.8326],
        [ 3.6017],
        [ 7.4603],
        [ 0.7398],
        [ 8.2723],
        [ 1.7648]], grad_fn=&lt;AddmmBackward&gt;)
</code></pre>
<p>I think, I am doing something wrong in this line</p>
<pre><code class="lang-auto">pred_y = torch.max(test_output, 1)[1].data.squeeze()
</code></pre>

username: Unity05
cooked: <p>That happens because the second output of <code>torch.max()</code> contains the respective indices.<br>
Furthermore, you set <code>dim=1</code>, but your outputs only have ine element for dim 1, so the argmax always is 0. I guess, you want to change your network to have 10 output nodes to regress the probabilities for each digit?</p>

username: akib62
cooked: <p>No, I need only one value and I used the bellow line and working fine.</p>
<pre><code class="lang-auto">pred_y = test_output.squeeze()
</code></pre>

username: Unity05
cooked: <p>Okay then. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
May I ask why you don’t train a classifier for MNIST? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=10" title=":thinking:" class="emoji" alt=":thinking:"><br>
And btw, it’s more expressive to use an AverageMeter for the loss logs instead of only one batche’s loss. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>

username: akib62
cooked: <p><a class="mention" href="/u/unity05">@Unity05</a>  Out of curiosity. I didn’t get you “AverageMeter for the loss logs”, could you tell me a little more?</p>

username: Unity05
cooked: <p>That’s just taking the average over all batches in that intervall.</p>

username: Samuel_Bachorik
cooked: <p>Hi <a class="mention" href="/u/akib62">@akib62</a> , for good performance on MNIST use this model<br>
with nn.NLLLoss()<br>
You will get over 99% accuracy with learning rate=0.0001,optimizer Adam and 30 epochs.<br>
I got 99,56% with this model<br>
Model</p>
<pre><code class="lang-auto">from torch import nn
import torch
class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

        self.conv1 = nn.Conv2d(1, 32, kernel_size=3, stride=2, padding=1)
        self.activation1 = nn.ReLU()

        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, stride=2, padding=1)
        self.activation2 = nn.ReLU()

        self.conv3 = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1)
        self.activation3 = nn.ReLU()
        self.conv4 = nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=0)
        self.activation4 = nn.ReLU()

        self.linear1 = nn.Linear(128 * 5 * 5, 10)
        self.soft = nn.LogSoftmax(dim=1)

    def forward(self, xb):
        xb = self.conv1(xb)
        xb = self.activation1(xb)

        xb = self.conv2(xb)
        xb = self.activation2(xb)

        xb = self.conv3(xb)
        xb = self.activation3(xb)
        xb = self.conv4(xb)
        xb = self.activation4(xb)

        xb = xb.reshape(-1, 128 * 5 * 5)

        xb = self.linear1(xb)
        xb = self.soft(xb)

        return xb
</code></pre>
<p>If you are interested here is full training loop and test of accuracy on my github, you can try</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/Samuel-Bachorik/MNIST_PyTorch_concurrent-loader">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/Samuel-Bachorik/MNIST_PyTorch_concurrent-loader" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/61a44ddb1ba343588b981ccef3d61006b5ffc3b6b43b2f5cf8ba55733d1f47bf/Samuel-Bachorik/MNIST_PyTorch_concurrent-loader" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/Samuel-Bachorik/MNIST_PyTorch_concurrent-loader" target="_blank" rel="noopener nofollow ugc">GitHub - Samuel-Bachorik/MNIST_PyTorch_concurrent-loader: Concurrent loader...</a></h3>

  <p>Concurrent loader for MNIST images, AI model, Training loop, Test - GitHub - Samuel-Bachorik/MNIST_PyTorch_concurrent-loader: Concurrent loader for MNIST images, AI model, Training loop, Test</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>