The size of tensor a (10) must match the size of tensor b (100) at non-singleton dimension 1

igkrk: <p>Hi there, im following tutorial of simple neural network with 2 hidden layers and I wanted to add MSE and CE charts at the end of file, however i have no idea what should i do, any ideas?</p>
<pre><code class="lang-auto">import torch
import torchvision
import torch.nn as nn
import torch.nn.functional as f
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
import sklearn 
import numpy as np
import pandas as pd
import sklearn.metrics
import matplotlib.pyplot as plt

torch.manual_seed(101)

Transform = transforms.ToTensor()

train = torchvision.datasets.MNIST('', train=True, download=True, transform=Transform)
test = torchvision.datasets.MNIST('', train=False, download=True, transform=Transform)

trainset = DataLoader(train, batch_size=100, shuffle=True)
testset = DataLoader(test, batch_size=500, shuffle=False)

class Model(nn.Module):
    def __init__(self, input_size=784, output_size=10, layers=[120,84]):
        super().__init__()
        self.d1 = nn.Linear(input_size, layers[0])
        self.d2 = nn.Linear(layers[0], layers[1])
        self.d3 = nn.Linear(layers[1], output_size)
    
    def forward(self, X):
        X = f.relu(self.d1(X))
        X = f.relu(self.d2(X))
        X = self.d3(X)
        return f.log_softmax(X, dim=1)

model = Model()

ce = nn.CrossEntropyLoss()
mse = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

train_losses = []
train_mse_losses = []
test_losses = []
test_mse_losses = []

train_correct = []
test_correct = []

for i in range(10):
    trn_cor = 0
    tst_cor = 0
    for b, (X_train, y_train) in enumerate(trainset):
        b+=1
        y_pred = model(X_train.view(100, -1))
        loss = ce(y_pred, y_train)
        loss_mse = mse(y_pred, y_train)

        predicted = torch.max(y_pred.data, 1)[1]
        batch_cor = (predicted==y_train).sum()
        trn_cor += batch_cor

        optimizer.zero_grad()
        loss.backward()
        loss_mse.backward()
        optimizer.step()

        if b%600 == 0:
            print(f'epoch: {i:2} train loss: {loss.item():10.6f}')

    train_losses.append(loss)
    train_mse_losses.append(loss_mse)
    train_correct.append(trn_cor)

    with torch.no_grad():
        for b, (X_test, y_test) in enumerate(testset):
            y_val = model(X_test.view(500, -1))

            predicted = torch.max(y_val.data, 1)[1]
            tst_cor += (predicted == y_test).sum()
    
    loss = ce(y_val, y_test)
    loss_mse = mse(y_val, y_test)
    test_losses.append(loss)
    test_mse_losses.append(loss_mse)
    test_correct.append(tst_cor)

print(f'test acc: {test_correct[-1].item()*100/10000:.3f}%')

plt.subplot(3,1,1)
plt.plot(train_losses, label='training loss')
plt.plot(test_losses, label='validation loss')
plt.title('Loss at the end of each epoch')

plt.subplot(3,1,3)
plt.plot([t/600 for t in train_correct], label='training acc')
plt.plot([t/100 for t in train_correct], label='validation acc')
plt.title('Accuracy at the end of each epoch')

plt.legend()
</code></pre>

igkrk: <p>Everything was working until I’ve added</p>
<pre><code class="lang-auto">mse = nn.MSELoss()
</code></pre>

KFrank: <p>Hi Igor!</p>
<aside class="quote no-group" data-username="igkrk" data-post="1" data-topic="125717" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/igkrk/48/30564_2.png" class="avatar"> igkrk:</div>
<blockquote>
<pre><code class="lang-auto">        loss = ce(y_pred, y_train)
        loss_mse = mse(y_pred, y_train)
</code></pre>
</blockquote>
</aside>
<p>Your problem is that <code>CrossEntropyLoss</code> and <code>MSELoss</code> work rather<br>
differently (both conceptually and mechanically).</p>
<p><code>CrossEntropyLoss</code> expects its <code>input</code> (your <code>y_pred</code>) to be a set of<br>
<em>logits</em> (see below) for the classes your model outputs, typically of<br>
shape <code>[nBatch, nClass]</code> (in your case <code>[100, 10]</code>), and its <code>target</code><br>
(your <code>y_train</code>) to be integer class labels, typically of shape <code>[nBatch]</code><br>
(no <code>nClass</code> dimension) with values that range from <code>0</code> to <code>nClass - 1</code>.</p>
<p>In contrast, <code>MSELoss</code> expects its <code>input</code> and <code>target</code> to have the same<br>
shape as one another and uses mean-squared-error to measure, on<br>
average,  how close the individual elements of <code>input</code> and <code>target</code> are<br>
to one another.  You are passing an <code>input</code> of shape <code>[100, 10]</code> and a<br>
<code>target</code> of shape <code>[100]</code> to <code>MSELoss</code>, hence the error.</p>
<p>But furthermore, conceptually, a network that is trained to perform<br>
classification outputs logits (or other probability-like values) for each<br>
of the classes.  In the typical use case, these values are not naturally<br>
comparable to other numbers, which is to say, using them with<br>
<code>MSELoss</code> doesn’t really make sense.</p>
<p>You need to think about the meaning of your <code>y_train</code> values (and<br>
similarly <code>y_pred</code>) and whether it even makes sense to use them<br>
with <code>MSELoss</code>.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        loss.backward()
        loss_mse.backward()
</code></pre>
</blockquote>
</aside>
<p>As an aside, it’s a little more efficient to go <code>backward()</code> through your<br>
network just once (per optimizer step), so you could rewrite this as</p>
<pre data-code-wrap="python"><code class="lang-python">        loss_total = loss + loss_mse
        loss_total.backward()
        # or just  (loss + loss_mse).backward()
</code></pre>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        return f.log_softmax(X, dim=1)
</code></pre>
</blockquote>
</aside>
<p>Note that <code>CrossEntropyLoss</code> has <code>log_softmax()</code> built into it, so you<br>
don’t want this.  <code>CrossEntropyLoss</code> takes raw-score logits that run<br>
from <code>-inf</code> to <code>inf</code> and are typically just the output of your final <code>Linear</code><br>
layer.</p>
<p>If you want that final <code>log_softmax()</code> you should use <code>NLLLoss</code> instead<br>
pf <code>CrossEntropyLoss</code>.  (<code>CrossEntropyLoss</code> is just <code>log_softmax()</code><br>
and <code>NLLLoss</code> put together for you.)</p>
<p>Best.</p>
<p>K. Frank</p>