CrossEntropyLoss getting value > 1

Srinjoy_Mukherjee: <p>Hey all,<br>
I am training my highly imbalanced sentiment classification dataset using transformers’ library’s ELECTRA(similar to BERT) model by appending a classification head on top of it. The dataset has 5 classes. And I logging the loss every 10 steps.</p>
<p>The imbalance dataset stats are as follows:<br>
The number of 1 labels: 135<br>
The number of 2 labels: 43<br>
The number of 3 labels: 74<br>
The number of 4 labels: 303<br>
The number of 5 labels: 2242</p>
<p>The batch_size I am using is 16.<br>
The</p>
<p>The problem I am facing is the training loss I am encountering is starts from 1.5, which reduces to about 0.7.</p>
<p>Following are the doubts I have:</p>
<ol>
<li>Is it theoretically possible to have loss output from nn.CrossEntropyLoss &gt;1?</li>
<li>Also I thought loss output comes in terms of %, so how am I supposed to interpret this loss which starts with greater than 1 value?</li>
</ol>

archit-spec: <p>its is , it just gives you -(summation (log(p_i))) where p_i is the confidence in the prediction of i’th class , so in theory since natural log function is bound from 0 to - inf between 0 and 1 (since we are taking a negative of that ) the cross entory function is bound between 0 to inf<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d49a23c573bcb2997233aca196fc6e463da4621.png" data-download-href="https://discuss.pytorch.org/uploads/default/6d49a23c573bcb2997233aca196fc6e463da4621" title="cal"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d49a23c573bcb2997233aca196fc6e463da4621_2_690x236.png" alt="cal" data-base62-sha1="fANNPTXq3sAnLVXScU7uVRViaKR" width="690" height="236" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d49a23c573bcb2997233aca196fc6e463da4621_2_690x236.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d49a23c573bcb2997233aca196fc6e463da4621_2_1035x354.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d49a23c573bcb2997233aca196fc6e463da4621.png 2x" data-dominant-color="F3F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cal</span><span class="informations">1132×388 48.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Mah_Neh: <p>adding up to prev answer, use some metric to measure model improvement like accuracy, losses aren’t that easy to introspect, unless it’s MSE imho.</p>

Mah_Neh: <p>logs are bound to -inf to +inf;</p>
<ul>
<li>ln(e^-100000) = -100000</li>
<li>ln(e^1000000) = 1000000</li>
</ul>
<p>Yet as <a class="mention" href="/u/archit-spec">@archit-spec</a> indicates, if <code>\hat{y}</code> is in the range 0-1, then the log will be within <code>-inf</code> to <code>0</code>, and <code>-log</code> within <code>0</code> to <code>+inf</code></p>

Srinjoy_Mukherjee: <p>First of all, Thanks for the replies.<br>
Also then, what do you all suggest I should Log in the logger to plot, the Cross entropy loss coming from the network , or use calculate precision, recall by hand and plot it? <a class="mention" href="/u/archit-spec">@archit-spec</a>  <a class="mention" href="/u/mah_neh">@Mah_Neh</a></p>
<p>Specifically , I want to report that the model is learning, what loss/metric should I report?</p>

Mah_Neh: <p>I tend to log train loss, test loss, and train test accuracies as well. There is some overlap but also they provide different information.</p>
<p>However I do not know what is a standard metric for NLP.</p>

Srinjoy_Mukherjee: <p>Oh no, just from categorical/classification point of view u can answer. I am doing simple sentiment analysis which is just classification.<br>
So, I should calculate the accuracy of the predictions using the target(output from model) and predicted labels, is this what you are saying? Please clarify .Thanks again.</p>

Mah_Neh: <p>Yes, exactly, either manually or using some torcheval metric (torcheval is a package, utils for evaluating your model.(</p>
<p>You can also calculate the multi / binary class confusion matrix, at least I find it useful to understand what is going on.</p>
<p>I do not think you can include the confusion matrix in Tensorboard though.</p>

Srinjoy_Mukherjee: <p>Thank you for the reply, I will try this.</p>