Warning appearing depending on loss function?

Arthur_Conmy: <p>I’ve been working through a PyTorch tutorial here: <a href="https://jovian.ml/aakashns/fdaae0bf32cf4917a931ac415a5c31b0" rel="nofollow noopener">https://jovian.ml/aakashns/fdaae0bf32cf4917a931ac415a5c31b0</a> and have been trying to implement the neural network from the first chapter of <a href="http://neuralnetworksanddeeplearning.com/" rel="nofollow noopener">http://neuralnetworksanddeeplearning.com/</a>. One of the changes to make to the tutorial’s neural network is changing the loss function from cross entropy to MSE. After initialising the network, the tutorial runs the following</p>
<pre><code class="lang-auto">for images, labels in train_loader:
    outputs = model(images)

    print(outputs.shape,labels.shape)
    loss = F.cross_entropy(outputs, labels)
    
    print('Loss:', loss.item())
    break

print('outputs.shape : ', outputs.shape)
print('Sample outputs :\n', outputs[:2].data)
</code></pre>
<p>which prints</p>
<pre><code class="lang-auto">torch.Size([128, 10]) torch.Size([128])
Loss: 2.3188605308532715
outputs.shape :  torch.Size([128, 10])
Sample outputs: ...
</code></pre>
<p>However when running essentially the same code after changing the loss function to <code>MSELoss</code>:</p>
<pre><code class="lang-auto">for images, labels in train_loader:
    outputs = model(images)

    print(outputs.shape, labels.shape)

    loss_fn = nn.MSELoss()
    loss = loss_fn(outputs, labels) # Calculate loss
    
    print('Loss:', loss.item())
    break

print('outputs.shape : ', outputs.shape)
print('Sample outputs :\n', outputs[:2].data)
</code></pre>
<p>I get the following output (note a different batch size is used) and then warning</p>
<pre><code class="lang-auto">torch.Size([10, 10]) torch.Size([10])
Loss: 42.0984992980957
...
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py:432: UserWarning: Using a target size (torch.Size([10])) that is different to the input size (torch.Size([10, 10])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
</code></pre>
<p><strong>Why is this warning appearing for one loss function and not the other?</strong></p>
<p>Exra: in addition, why doesn’t writing <code>loss = nn.MSELoss(outputs, labels)</code> rather than the <code>loss_fn</code> workaround I’ve been using work?</p>

KFrank: <p>Hello Arthur -</p>
<aside class="quote no-group quote-modified" data-username="Arthur_Conmy" data-post="1" data-topic="89503" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arthur_conmy/48/26759_2.png" class="avatar"> Arthur_Conmy:</div>
<blockquote>
<pre><code class="lang-auto">    print(outputs.shape,labels.shape)
    loss = F.cross_entropy(outputs, labels)
</code></pre>
<p>which prints</p>
<pre><code class="lang-auto">torch.Size([128, 10]) torch.Size([128])
</code></pre>
<p>after changing the loss function to <code>MSELoss</code>:</p>
<pre><code class="lang-auto">    print(outputs.shape, labels.shape)
    loss_fn = nn.MSELoss()
</code></pre>
<p>I get the following output</p>
<pre><code class="lang-auto">torch.Size([10, 10]) torch.Size([10])
Loss: 42.0984992980957
...
UserWarning: Using a target size (torch.Size([10])) that is different to the input size (torch.Size([10, 10])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
</code></pre>
</blockquote>
</aside>
<p><code>cross_entropy()</code> expects <code>outputs</code> of shape <code>[nBatch, nClass]</code> and<br>
<code>labels</code> of shape <code>[nBatch]</code> (no <code>nClass</code> – each single <code>label</code> value is<br>
an integer class label that runs from <code>0</code> to <code>nClass - 1</code>).</p>
<p><code>MSELoss</code> expects <code>outputs</code> of shape <code>[nBatch, nValues]</code> and <code>labels</code><br>
(which really shouldn’t be called “labels” for <code>MSELoss</code>) also of shape<br>
<code>[nBatch, nValues]</code>.</p>
<p>I’m not really sure what the tutorial is looking for here, because if<br>
<code>cross_entropy()</code> makes sense for a model, <code>MSELoss</code> is unlikely<br>
to also make sense.</p>
<p>Having said that, you could either one-hot your integer class labels<br>
to get <code>labels</code> of shape <code>[nBatch, nClass]</code> to match the output of<br>
your model, or you could change your model to output a single<br>
value (instead of <code>nClass</code> values) so that its output for a batch will<br>
be of shape <code>[nBatch]</code> and thus match the shape of your <code>labels</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>Exra: in addition, why doesn’t writing <code>loss = nn.MSELoss(outputs, labels)</code> rather than the <code>loss_fn</code> workaround I’ve been using work?</p>
</blockquote>
</aside>
<p><code>torch.nn.MSELoss</code> is a class, while <code>torch.nn.functional.mse_loss()</code><br>
is a function.  Thus <code>loss = nn.MSELoss(outputs, labels)</code> is<br>
incorrectly calling the <em>constructor</em> of the <code>MSELoss</code> class with the<br>
wrong arguments.</p>
<p>Instead of:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = nn.MSELoss (outputs, labels)   # wrong
</code></pre>
<p>you should either write:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = nn.MSELoss() (outputs, labels)
</code></pre>
<p>(note the extra pair of parentheses), or:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = F.mse_loss (outputs, labels)
</code></pre>
<p>Good luck.</p>
<p>K. Frank</p>