Only CrossEntropyLoss seems to work? MSE and other fail?

alx: <p>I run into a similar problem for every loss type I’m trying:</p>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (5) must match the size of tensor b (8) at non-singleton dimension 1
</code></pre>
<p>Not sure how to fix it.</p>

ptrblck: <p>Based on the error message dim1 has a different shape for your model output and target.<br>
Note that <code>nn.CrossEntropyLoss</code> assumes the output is passed as <code>[batch_size, nb_classes, *]</code>, while the target should have the shape <code>[batch_size, *]</code> (missing <code>nb_class</code> dimension).<br>
Other loss functions, such as <code>nn.MSELoss</code> expect the same shapes or are broadcasting internally, if possible.</p>

alx: <p>Alright so I checked and the predicted label shape is torch.Size([32, 10])<br>
while the ground_truth is torch.Size([32])</p>
<pre><code class="lang-auto">        pred_label = model(image)
        loss = criterion(pred_label, label)
</code></pre>
<p>32 is my batch size but I’m not sure where the 10 comes from.</p>

ptrblck: <p>The <code>10</code> should correspond to the number of classes, i.e. your model would output 10 logits for each sample in the batch of 32 samples.<br>
These shapes should work (if you are really working with 10 classes).<br>
However, the error message points to different shapes in dim1 (5 vs. 8), so there still seem to be a discrepancy between the error and the shapes you’ve posted.</p>

alx: <p>So I guess I need to do an argmax on my batch because right now every image in the batch has a 1000 class predictions.</p>
<p>If I understand correctly a batch of 8 should have 8 class predictions or 1 per image.</p>
<p>Do you know how I can achieve this with tensors?</p>
<pre><code class="lang-auto"> for image_batch, label_batch, path in train_dl:

    pred_batch = model(image_batch.to(device))
    #pred_batch.shape = torch.Size([8, 1000])
    #something here, to reshape to [8, 8]
    batch_loss = criterion(pred_batch, label_batch.to(device))
</code></pre>
<p>Sorry if the numbers keep changing, Its different runs but ultimately same issue</p>

ptrblck: <p>It seems you might use a pretrained model, since your output has a shape of <code>[batch_size, 1000]</code> (the 1000 classes would then correspond to the ImageNet classes).<br>
Based on your description, you are dealing with 8 classes, so I would recommend to change the last linear layer so that only 8 logits will be used.<br>
Have a look at the <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html">transfer learning tutorial</a> for more information.</p>

alx: <p>Yes exactly!</p>
<p>However, I have 5 classes and my batch size is 8.<br>
I previously tried what you suggested and here’s what I got:</p>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (5) must match the size of tensor b (8) at non-singleton dimension 1
</code></pre>
<p>torch.Size([8, 5])</p>

ptrblck: <p>For a batch size of 8 and 5 classes your outputs should have the shape <code>[8, 5]</code> and the targets <code>[8]</code> containing values in <code>[0, 4]</code>.</p>

alx: <p>Yes, I agree. Here’s what I have;</p>
<pre><code class="lang-auto">  for image_batch, label_batch in train_dl:
    optimizer.zero_grad()

    pred_batch = model(image_batch.to(device))
    print(pred_batch.shape)
    print(label_batch.shape)
</code></pre>
<pre><code class="lang-auto">torch.Size([8, 5])
torch.Size([8])
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py:431: UserWarning: Using a target size (torch.Size([8])) that is different to the input size (torch.Size([8, 5])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
</code></pre>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (5) must match the size of tensor b (8) at non-singleton dimension 1
</code></pre>

ptrblck: <p>The provided shapes are for <code>nn.CrossEntropyLoss</code> and <code>nn.MSELoss</code> expects the tensors to have the same shape or broadcastable as explained in the first post.<br>
If you want to use <code>nn.MSELoss</code> for a classification use case, you could probably create a one-hot encoded tensor via:</p>
<pre><code class="lang-python">label_batch = F.one_hot(label_batch, num_classes=5)
</code></pre>