Hello, loss is marked as zero

111394: <p>epoch: 25  Loss: 0.0 %<br>
output : tensor([[-21.6479, -24.8251,  46.5167],<br>
[ -7.8592, -16.6111,  24.4777]], device=‘cuda:0’)<br>
target :   tensor([2, 2], device=‘cuda:0’)</p>
<p>In terms of value, the answer is correct.<br>
Because it is correct, loss is marked as zero.<br>
So learning is not going on.</p>
<p>thank you, everything</p>

RoySadaka: <p>Can you explain what the question is?</p>
<p>Roy</p>

faizan_shaikh: <aside class="quote no-group" data-username="111394" data-post="1" data-topic="100623">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/1/e36b37/48.png" class="avatar"> 111394:</div>
<blockquote>
<p>output : tensor([[-21.6479, -24.8251, 46.5167],<br>
[ -7.8592, -16.6111, 24.4777]], device=‘cuda:0’)</p>
<p>target : tensor([2, 2], device=‘cuda:0’)</p>
</blockquote>
</aside>
<p>The shape of output and target isn’t the same. This shouldn’t be the expected result, right?</p>
<p>If not, how do you define your comparison of output and target to calculate loss? Are you defining a custom loss function for this?</p>

111394: <p>My source is to classify three classes. I use this.</p>
<h2>criterion = nn.CrossEntropyLoss().to(device)</h2>
<p>optimizer_ft= optim.SGD(model_ft.parameters(), lr=0.01, momentum=0.9)<br>
’<br>
’<br>
loss = criterion(outputs, target)<br>
’<br>
’</p>
<p>output: tensor ([[-21.6479, -24.8251, 46.5167],<br>
[-7.8592, -16.6111, 24.4777]], device = ‘cuda : 0’)</p>
<p>target: tensor ([2, 2], device = ‘cuda : 0’)</p>
<p>I don’t know how to compare output and target.</p>

111394: <p>My source is to classify three classes. I use this.</p>
<h2>criterion = nn.CrossEntropyLoss().to(device)</h2>
<p>optimizer_ft= optim.SGD(model_ft.parameters(), lr=0.01, momentum=0.9)<br>
’<br>
’<br>
loss = criterion(outputs, target)<br>
’<br>
’</p>
<p>output: tensor ([[-21.6479, -24.8251, 46.5167],<br>
[-7.8592, -16.6111, 24.4777]], device = ‘cuda : 0’)</p>
<p>target: tensor ([2, 2], device = ‘cuda : 0’)</p>
<p>I don’t know how to compare output and target.</p>

RoySadaka: <p>Your loss is fine, so im still not sure what is the question here, are you asking how you check the accuracy?</p>
<p><code>loss = criterion(outputs, target)</code><br>
Is correct,  CrossEntropyLoss applies LogSoftmax on your logits and then uses NLLoss.<br>
your output tensor looks like Logits, and you have 3 classes, and both samples in outputs marked <strong>class 2</strong> as the chosen class, by a large margin from the other classes.</p>
<p>Your target says <strong>class 2</strong> for the 1st sample and <strong>class 2</strong> for the second sample,<br>
so in that case, loss=0.0 (which is perfectly fine in this example)</p>
<p>so when you say:</p>
<blockquote>
<p>So learning is not going on.</p>
</blockquote>
<p>It seems strange, cause it looks like learning was done, and it was very good (maybe even overfitting <img src="https://discuss.pytorch.org/images/emoji/apple/man_shrugging/2.png?v=12" title=":man_shrugging:t2:" class="emoji" alt=":man_shrugging:t2:" loading="lazy" width="20" height="20">).</p>
<p>Just in case your question was how to check the <strong>accuracy</strong> of the model, then you probably want to use<br>
<code>CategoricalAccuracyWithLogits</code>, <a href="https://github.com/RoySadaka/lpd" rel="noopener nofollow ugc">lpd</a> package has this metric and some more, so you may choose to use that if you want structured train process and metrics,</p>
<p>For the sake of simplicity, you can use this snippet:</p>
<pre data-code-wrap="python"><code class="lang-python">def categorical_accuracy_with_logits(y_pred, y_true):
    indices = torch.max(y_pred, 1)[1]
    correct = torch.eq(indices, y_true).view(-1)
    accuracy = correct.float().sum() / correct.shape[0]
    return accuracy
</code></pre>
<p>Roy</p>

111394: <p>thanks RoySadaKa</p>
<p>I checked ACC but always 1.<br>
Loss is repetition. Like this</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b19439b4e9d25e88dc80771b98d3548f4297313.png" data-download-href="https://discuss.pytorch.org/uploads/default/0b19439b4e9d25e88dc80771b98d3548f4297313" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b19439b4e9d25e88dc80771b98d3548f4297313.png" alt="image" data-base62-sha1="1AbnzCEnJznfmmFsMHclND6zjYn" width="259" height="500" data-dominant-color="F0F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">493×950 4.29 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

RoySadaka: <p>Can you show the same image but for epochs 0,1,2 ?</p>
<p>Roy</p>

111394: <p>Thanks</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d80cdd9c08de053761f58d905cbe34173db042c1.png" data-download-href="https://discuss.pytorch.org/uploads/default/d80cdd9c08de053761f58d905cbe34173db042c1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d80cdd9c08de053761f58d905cbe34173db042c1.png" alt="image" data-base62-sha1="uPgHhLHE0OEP4a59FemGZkrC9dT" width="246" height="500" data-dominant-color="F0F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">460×933 5.18 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/2/226d19beb6c151b8ef953824a5bddabc4a238d56.png" data-download-href="https://discuss.pytorch.org/uploads/default/226d19beb6c151b8ef953824a5bddabc4a238d56" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/2/226d19beb6c151b8ef953824a5bddabc4a238d56.png" alt="image" data-base62-sha1="4UxYOviNdeRCZLLOtPlCKGegwWG" width="259" height="500" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">489×943 7.24 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

RoySadaka: <p>thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">  im not sure i fully understand the log, but it seems like the loss &gt; 0 in the first few epochs, and reaching 0 at epoch 1408, so can you explain what are your concerns?</p>
<p>Also, can you explain the challenge you try to solve? i see the input shape is<br>
(8, 3, 40, 128, 128), please provide some context about the input, I’m guessing the image sizes are 128x128, there are 3 channels, your batch size = 8, not sure what is the 40?</p>
<p>Knowing what you try to do can help a lot, even share some code.</p>
<p>Roy</p>

111394: <p>Thanks, for Answer<br>
I try to make 3dCNN for Action recognition<br>
(8, 3, 40, 128, 128) =&gt; (batch, channel, frame_len, height, width)<br>
put in all at once Image * frame_len , and get one label (ex… fight,walk,stand…)</p>
<p><strong>This is my code (train)</strong></p>
<p>ave_loss =0<br>
epoch_loss = 0<br>
ave_loss =0<br>
for epoch in range(nb_epoch):<br>
print(“ㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡ”,epoch)<br>
running_loss = 0.0</p>
<pre><code>epoch_num = 1
for i, data in enumerate(train_loader, 0):
    model_ft.train()
  
    inputs, labels = data
    inputs = inputs.to(device)
    
    
    labels = torch.argmax(labels, dim=1)
    
    labels = labels.to(device)

  
    optimizer_ft.zero_grad()

    print("input shape : ",inputs.permute(0,4,1,2,3).shape)
    outputs = model_ft(inputs.permute(0,4,1,2,3))
    
    
    _, predicted = torch.max(outputs.data, 1)
   
 
    loss = criterion(outputs, labels)
    print(i)
    print('epoch: {}  Loss: {} %'.format(epoch, loss.data))
  
    loss.backward()
    optimizer_ft.step()

    
    running_loss += loss.item()
    epoch_loss = running_loss/(i+1)
 
 
    print("epoch_loss : ",epoch_loss)
    print("ACC : ",categorical_accuracy_with_logits(outputs,labels))


    value_tracker(loss_plt, torch.Tensor([epoch_loss]), torch.Tensor([i + epoch*len(train_loader) ]))
    epoch_num+=1


running_loss = 0.0
if epoch%5==0:
    acc = acc_check(model_ft, test_loader, epoch, save=1)
    value_tracker(acc_plt, torch.Tensor([acc]), torch.Tensor([epoch]))
    
if epoch%20==0:
    torch.save(model_ft, "D:/data/3dcnn_weights/basic128n_epoch_{}_acc_{}.pt".format(epoch, int(acc)))
</code></pre>
<p>print(‘Finished Training’)</p>
<p>I used the accuracy measurement method that I told you, but it always comes out as 1.<br>
But the val accuracy is low.</p>
<p>thank you so much</p>

RoySadaka: <aside class="quote no-group" data-username="111394" data-post="12" data-topic="100623">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/1/e36b37/48.png" class="avatar"> 111394:</div>
<blockquote>
<p>I used the accuracy measurement method that I told you, but it always comes out as 1.<br>
But the val accuracy is low.</p>
</blockquote>
</aside>
<p>using the data you provided in the initial post, the accuracy behaves as expected:</p>
<pre><code class="lang-auto">In  [1]: import torch
In  [2]: a = torch.tensor([[-21.6479, -24.8251, 46.5167],[-7.8592, -16.6111, 24.4777]])
In  [3]: a
Out [3]: tensor([[-21.6479, -24.8251,  46.5167],
                 [ -7.8592, -16.6111,  24.4777]])

In  [4]: b = torch.tensor([2, 2])
In  [5]: categorical_accuracy_with_logits(a,b)
Out [5]: tensor(1.)

In  [6]: b = torch.tensor([2, 1])
In  [7]: categorical_accuracy_with_logits(a,b)
Out [7]: tensor(0.5000)

In  [8]: b = torch.tensor([0, 0])
In  [9]: categorical_accuracy_with_logits(a,b)
Out [9]: tensor(0.)
</code></pre>
<p>Maybe there is overfitting happening on the trainset (you showed 1408 epochs maybe it’s too much), have you considered using EarlyStopping? (to stop the train if some metric is not improving for N amount of epoch, e.g. loss not decreasing anymore, or accuracy not increasing)</p>
<p>Roy</p>