Calculate loss between a target in one-hot encoded with class probabilities

username: DRISS_ELALAOUI
cooked: <p>please I have two tensors:</p>
<pre><code class="lang-auto">print(pred)
&gt;&gt; tensor([[0.05369152 0.09913312 0.5724052  0.13255161 0.041731   0.10048745]
 [0.13759595 0.13779168 0.11650976 0.27918294 0.1462693  0.18265037]
 [0.07497178 0.5721384  0.02145172 0.10340305 0.17965348 0.04838147]])
print(target)
&gt;&gt;tensor([[0 0 0 1 0 0]
 [0 1 0 0 0 0]
 [0 1 0 0 0 0]])
</code></pre>
<p>namely I have 6 classes: 0 to 5<br>
I want to calculate loss between them, what type of loss function should I choose in this case (ex: <code>nn.CrossEntropyLoss</code>) ? and how ?<br>
and thanks in advance</p>

username: KFrank
cooked: <p>Hi Driss!</p>
<aside class="quote no-group" data-username="DRISS_ELALAOUI" data-post="1" data-topic="180148" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/driss_elalaoui/48/35900_2.png" class="avatar"> DRISS_ELALAOUI:</div>
<blockquote>
<p>I want to calculate loss between them, what type of loss function should I choose in this case (ex: <code>nn.CrossEntropyLoss</code>) ?</p>
</blockquote>
</aside>
<p>First let me suggest what you ought to do and then answer your specific<br>
question:</p>
<p>You want <code>pred</code> to consist of (unnormalized) log-probabilities.  To achieve<br>
this you should have the final layer of your network be a <code>Linear</code> with<br>
<code>out_features = 6</code>, <em>not</em> followed by <code>Softmax</code> (nor any other “activation”<br>
layer).  This is for improved numerical stability (compared with having <code>pred</code> be<br>
probabilities, as you appear to have), but also because <code>CrossEntropyLoss</code><br>
<em>requires</em> <code>pred</code> to be log-probabilities.</p>
<p>You also want <code>target</code> to be integer class labels (with shape <code>[nBatch]</code> and<br>
no class dimension).  (You can use a one-hot encoded <code>target</code> but there is no<br>
real point to doing so, and it is somewhat less efficient.)  You can convert<br>
<code>target</code> to integer class labels with <code>target.argmax (dim = 1)</code>.</p>
<p>Now to answer your question in the specific context you give (but doing things<br>
this way will be sub-optimal):</p>
<p>You must convert <code>pred</code> to log-probabilities by passing it through <code>log()</code>.</p>
<p>You must also convert one-hot-encoded <code>target</code> from integers to floating-point<br>
numbers, e.g., with <code>float()</code>.  This is because <code>CrossEntropyLoss</code> works in two<br>
modes: One is with integer class labels (of shape <code>[nBatch]</code>), and the  other is<br>
with (floating-point) probabilistic “soft” labels (of shape <code>[nBatch, nClass]</code>).<br>
(One-hot-encoded labels can be understood to be a special case of probabilistic<br>
labels where the probabilities happen to be <code>0.0</code> or <code>1.0</code>.)  <code>CrossEntropyLoss</code><br>
decides which mode to use based, in part, on whether <code>target</code> is integer or<br>
floating-point.  If you pass in an integer <code>target</code>, <code>CrossEntropyLoss</code> will try<br>
to interpret it as integer class labels rather that as probabilistic “soft” labels<br>
(of which one-hot-encoded labels are a special case).</p>
<p>Best.</p>
<p>K. Frank</p>

username: DRISS_ELALAOUI
cooked: <p>please can you explain to me with an example by applying what you said about these two variables “target” and “pred”? because my real source code is very large… I tried to apply what you said all day, but without success.</p>

username: KFrank
cooked: <p>Hi Driss!</p>
<aside class="quote no-group" data-username="DRISS_ELALAOUI" data-post="3" data-topic="180148" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/driss_elalaoui/48/35900_2.png" class="avatar"> DRISS_ELALAOUI:</div>
<blockquote>
<p>I tried to apply what you said all day, but without success.</p>
</blockquote>
</aside>
<p>I would suggest that you set up a toy example, using as example data the <code>pred</code><br>
and <code>target</code> you posted above, and attempt to pass them to <code>CrossEntropyLoss</code><br>
following the suggestions I made.</p>
<p>If you still have issues, post a small, fully-self-contained, runnable script that<br>
shows what you’ve tried, together with its output.  Let us know what you think<br>
isn’t working and ask any specific questions you might have.</p>
<p>Best.</p>
<p>K. Frank</p>

username: MicahZoltu
cooked: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="180148">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>This is for improved numerical stability</p>
</blockquote>
</aside>
<p>Can you provide some additional details on this, ideally in a way that a software engineer/non-mathematician can understand?  Why does one want “numerical stability”?  What effect does that have on training/inference?  Since we are working with floating point numbers, there doesn’t seem like there should be any obvious advantage to using numbers inside a particular range vs some other range.</p>

username: KFrank
cooked: <p>Hi Micah!</p>
<aside class="quote no-group" data-username="MicahZoltu" data-post="5" data-topic="180148" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/958977/48.png" class="avatar"> MicahZoltu:</div>
<blockquote>
<p>Since we are working with floating point numbers, there doesn’t seem like there should be any obvious advantage to using numbers inside a particular range vs some other range.</p>
</blockquote>
</aside>
<p>Even though floating-point numbers can represent a large range, that range<br>
is finite, so floating-point numbers can “underflow” to zero and “overflow” to <code>inf</code>.</p>
<p>The problem is that when you have log-probabilities in a very reasonable range,<br>
the process of using <code>exp()</code> to convert them to probabilities greatly expands that<br>
range, making underflow and overflow much more likely.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.1.0
&gt;&gt;&gt;
&gt;&gt;&gt; # unnormalized log-probabilities in a very reasonable range
&gt;&gt;&gt; log_prob = torch.tensor ([-150.0, -120.0, -100.0, -50.0, 0.0, 50.0, 100.0, 120.0, 150.0])
&gt;&gt;&gt;
&gt;&gt;&gt; # but they "saturate" to 0.0 and 1.0 when converted to probabilities
&gt;&gt;&gt; log_prob.softmax (0)
tensor([0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 3.7835e-44,
        1.9287e-22, 9.3576e-14, 1.0000e+00])
&gt;&gt;&gt;
&gt;&gt;&gt; # this is because softmax() uses exp() internally which underflows to 0.0 and overflows to inf
&gt;&gt;&gt; log_prob.exp()
tensor([0.0000e+00, 0.0000e+00, 3.7835e-44, 1.9287e-22, 1.0000e+00, 5.1847e+21,
               inf,        inf,        inf])
&gt;&gt;&gt;
&gt;&gt;&gt; # we can use pytorch's log_softmax() to convert unnormalized log-probabilities to normalized log-probabilities
&gt;&gt;&gt; log_prob.log_softmax (0)
tensor([-300., -270., -250., -200., -150., -100.,  -50.,  -30.,    0.])
</code></pre>
<p>If you’re training and you try to compute <code>softmax()</code> yourself, if your not careful<br>
you will get <code>inf</code>s and <code>nan</code>s that will pollute your parameters with <code>inf</code>s and<br>
<code>nan</code>s.  Even if you use pytorch’s <code>softmax()</code>, when a probability saturates at<br>
<code>1.0</code> its gradient will become zero, and training will not progress.  When a<br>
probability saturates at <code>0.0</code>, the <code>log()</code> inside of the cross-entropy function<br>
will give you <code>inf</code> for your loss and your training will break down.</p>
<p>Avoiding this is what I meant by “numerical stability.”</p>
<p>Because cross-entropy uses the logs of the probabilities in its formula, it never<br>
needs to compute that actual probabilities.  Instead, it takes unnormalized<br>
log-probabilities as its input and converts them to normalized log-probabilities<br>
with <code>log_softmax()</code>.  By leaving the predicted probabilities in “log-space,” so to<br>
speak, <code>CrossEntropyLoss</code> essentially eliminates the possibility of zero gradients<br>
and <code>inf</code>s in this part of the computation.</p>
<p>Best.</p>
<p>K. Frank</p>

username: MicahZoltu
cooked: <p>Very helpful, thank you K!</p>
<p>Given that softmax will lead to <code>0</code>/<code>inf</code>, what is the best (safest) way to convert my model’s output into normal probabilities that I can use as output during model execution (not training)?  Unitless numbers like <code>-300</code> and <code>72</code> aren’t particularly useful in turning the model output into human understandable probabilities, and if I run it through softmax it sounds like I will run into the problem you described, and log_softmax will just give me more unitless numbers that don’t appear to have a meaningful range on them.</p>
<p>My expectation was that I would be able to have my model spit out numbers in the range [0,1] where the sum of the probabilities are 1, which is very easy for my feeble human brain to interpret meaningfully without any additional context.</p>

username: KFrank
cooked: <p>Hi Micah!</p>
<aside class="quote no-group" data-username="MicahZoltu" data-post="7" data-topic="180148" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/958977/48.png" class="avatar"> MicahZoltu:</div>
<blockquote>
<p>Given that softmax will lead to <code>0</code>/<code>inf</code></p>
</blockquote>
</aside>
<p><code>softmax()</code> does not lead to <code>inf</code> – it produces probabilities that range from<br>
<code>0.0</code> to <code>1.0</code>.  (<code>exp()</code> can lead to <code>inf</code>).</p>
<aside class="quote no-group">
<blockquote>
<p>what is the best (safest) way to convert my model’s output into normal probabilities</p>
</blockquote>
</aside>
<p>Pass your model’s output through <code>softmax()</code> to get probabilities (assuming<br>
that your model has been trained to predict (unnormalized) log-probabilities,<br>
typically by passing the output of your model into <code>CrossEntropyLoss</code> as the<br>
loss criterion you use when training).</p>
<p>You will get perfectly sound probabilities.  Just be aware that you can lose a<br>
little bit of information when converting log-probabilities into probabilities in<br>
that two probabilities that are different and both very close to zero (if you were<br>
working with infinite precision) could both underflow to the same value of zero<br>
even though they can be seen to be non-zero and different in log-space.<br>
Similarly, a probability that is close to, but not equal to one could (when using<br>
finite precision) saturate to one when converted from log-space to regular<br>
probability space.</p>
<p>For many purposes this modest loss of information won’t matter.</p>
<aside class="quote no-group">
<blockquote>
<p>I run it through softmax it sounds like I will run into the problem you described</p>
</blockquote>
</aside>
<p>Using <code>softmax()</code> to convert your model’s predictions to probabilities won’t<br>
break anything – you’ll just potentially lose a little bit of information, as described<br>
above.</p>
<aside class="quote no-group">
<blockquote>
<p>My expectation was that I would be able to have my model spit out numbers in the range [0,1] where the sum of the probabilities are 1</p>
</blockquote>
</aside>
<p><em>Train</em> with (unnormalized) log-probabilities as your predictions, but if you then<br>
want to view those predictions as probabilities, pass them through <code>softmax()</code>.</p>
<p>Best.</p>
<p>K. Frank</p>