I do not understand loos function

szandala: <p>I am learning to re-train Resnet, with one more layer of 9 classes on top of final 1000.<br>
I think I do not understand criterion function.</p>
<pre><code class="lang-auto">criterian = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr = 0.0001, momentum = 0.9)
for epoch in range(20):
    running_loss  = 0.0
    for X, Y in training:
        optimizer.zero_grad()
        output = net(X)

        print(output)
        print(output.shape)
        print(Y)
        print(Y.shape)
        print(Y.argmax())
        loss = criterian(output, Y.argmax())
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    print(epoch, ':', running_loss)
</code></pre>
<p>It seems legit for me (since I don’t know yet how to create batches I am providing image one-by-one).<br>
I have output of tensor of shape 1x9 and target is a one element tensor, which specifies into which class it should be targeting.<br>
However the output is:</p>
<pre><code class="lang-auto">tensor([[0.1639, 0.2125, 0.1067, 0.1212, 0.1434, 0.0613, 0.0512, 0.0225, 0.1172]],
       device='cuda:0', grad_fn=&lt;SoftmaxBackward&gt;)
torch.Size([1, 9])
tensor([0., 0., 1., 0., 0., 0., 0., 0., 0.], device='cuda:0')
torch.Size([9])
tensor(2, device='cuda:0')
Traceback (most recent call last):
  File "im9_train.py", line 173, in &lt;module&gt;
    loss = criterian(output, Y.argmax())
  File "/home/szandala/imagenet/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/szandala/imagenet/venv/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 961, in forward
    return F.cross_entropy(input, target, weight=self.weight,
  File "/home/szandala/imagenet/venv/lib/python3.8/site-packages/torch/nn/functional.py", line 2468, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/szandala/imagenet/venv/lib/python3.8/site-packages/torch/nn/functional.py", line 2260, in nll_loss
    if input.size(0) != target.size(0):
</code></pre>
<p>What am I doing wrong?</p>

KFrank: <p>Hi Szandala!</p>
<p>The short story is that – for separate reasons – you are passing an<br>
improper <code>input</code> and <code>target</code> to your <code>CrossEntropyLoss</code> criterion<br>
function.</p>
<aside class="quote no-group quote-modified" data-username="szandala" data-post="1" data-topic="107194" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/szandala/48/32079_2.png" class="avatar"> szandala:</div>
<blockquote>
<pre><code class="lang-auto">criterian = nn.CrossEntropyLoss()
...
        loss = criterian(output, Y.argmax())
</code></pre>
<p>I have output of tensor of shape 1x9</p>
</blockquote>
</aside>
<p>This is appropriate.  You have a batch size of 1, with 9 predictions<br>
for a nine-class classification problem.</p>
<aside class="quote no-group">
<blockquote>
<p>and target is a one element tensor,</p>
</blockquote>
</aside>
<p>This isn’t actually true.</p>
<p>Your <code>Y</code> is a one-dimensional tensor of length nine.  <code>Y.argmax()</code> is a<br>
<em>zero</em>-dimensional tensor.  It is a “scalar” that holds a single value, but<br>
it is not a one-dimensional tensor of length one.</p>
<p>You should feed <code>Y.argmax().unsqueeze (dim = 0)</code> into your loss<br>
criterion.  This will have <code>shape = [1]</code>, and so will have the necessary<br>
batch size of one.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">tensor([[0.1639, 0.2125, 0.1067, 0.1212, 0.1434, 0.0613, 0.0512, 0.0225, 0.1172]],
       device='cuda:0', grad_fn=&lt;SoftmaxBackward&gt;)
</code></pre>
</blockquote>
</aside>
<p>This tells me that your model has a final <code>Softmax</code> layer.  This is incorrect<br>
for <code>CrossEntropyLoss</code>, which has, in effect, <code>Softmax</code> already built in.</p>
<p>Most likely, you should have the final output of your model be the output<br>
of your last <code>Linear</code> layer (which should have <code>out_features = 9</code>).</p>
<p>Good luck.</p>
<p>K. Frank</p>

szandala: <p>Thank You.<br>
Tho I am surprised I cannot use Softmax.<br>
SO I should remove now Softmax and append it when I will be using the network?</p>

KFrank: <p>Hi Szandala!</p>
<aside class="quote no-group" data-username="szandala" data-post="3" data-topic="107194" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/szandala/48/32079_2.png" class="avatar"> szandala:</div>
<blockquote>
<p>Tho I am surprised I cannot use Softmax.</p>
</blockquote>
</aside>
<p>As per the documentation, <code>CrossEntropyLoss</code> has <code>Softmax</code> built in.<br>
This is for reasons of numerical stability and convenience.  If you had<br>
a <code>Softmax</code> at the end of your network, you would be running your<br>
predictions through <code>Softmax</code> twice, which is not what you want.</p>
<aside class="quote no-group">
<blockquote>
<p>SO I should remove now Softmax and append it when I will be using the network?</p>
</blockquote>
</aside>
<p>You should remove the final <code>Softmax</code> from your model.</p>
<p>But, in general, you do not need <code>Softmax</code> for anything, so you normally<br>
won’t want to append it when you are “using the network.”</p>
<p>Your network will emit raw-score <em>logits</em> as its predictions.  If you pass<br>
them through <code>Softmax</code> you will get probabilities.  But the logits and<br>
probabilities contain the same information, so, in general, you can work<br>
directly with the logits and not need to convert them to probabilities.</p>
<p>Note that the <code>argmax()</code> of a set of logits will be the same as that of<br>
the corresponding set of probabilities.  So if you make a firm “class-X”<br>
prediction by taking the <code>argmax()</code> of your probabilities, your can<br>
equivalently take the <code>argmax()</code> of the logits.</p>
<p>Best.</p>
<p>K. Frank</p>

szandala: <p>Now got it, thank You very much!</p>