Exponential Loss Function

Maria_Fernanda_De_La: <p>I was looking through the documentation and I was not able to find the standard exponential loss function. Is there a simple way to implement my own exponential function.</p>
<p>Thank you in advance!</p>

KFrank: <p>Hello Maria!</p>
<aside class="quote no-group" data-username="Maria_Fernanda_De_La" data-post="1" data-topic="49146" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maria_fernanda_de_la/48/11812_2.png" class="avatar"> Maria_Fernanda_De_La:</div>
<blockquote>
<p>I was looking through the documentation and I was not able to find the standard exponential loss function. Is there a simple way to implement my own exponential function.</p>
</blockquote>
</aside>
<p>Sure.  And if you use normal tensor operations, autograd<br>
will work for you.</p>
<p>Let’s say you have a binary classification problem and<br>
<code>logits</code> are your predictions for a batch (running from<br>
-infinity to +infinity) and <code>labels</code> are your known class<br>
labels for that batch (equal to 0 or 1).</p>
<p>Then:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

def myExpLoss (logits, labels):
    return  (((2.0 * labels.float() - 1.0) * logits).exp()).mean()

logits = torch.tensor ([-0.2, 1.3, .3], requires_grad = True)
labels = torch.tensor ([0, 0, 1])

myExpLoss (logits, labels)
</code></pre>
<p>[Edit:  The  above expression for <code>myExpLoss</code> is missing a minus<br>
sign, and this causes the loss to incorrectly increase for more<br>
accurate predictions.  I haven’t edited in the correction above,<br>
but the correct expression appears in my next post, below.]</p>
<p>This yields:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; myExpLoss (logits, labels)
tensor(0.9479, grad_fn=&lt;MeanBackward1&gt;)
</code></pre>
<p>so you can see that autograd is working.</p>
<p>[Edit:  This loss value, <code>0.9479</code>, is less than 1, incorrectly indicating<br>
the the predictions are more right than wrong.  Although the first<br>
and last predictions are right, they are only mildly right, while<br>
the second prediction is strongly wrong, making the predictions<br>
overall more wrong than right.  The correct value of the loss (given<br>
by the corrected expression in my next post) is <code>1.7429</code>.]</p>
<p>This version returns the per-sample exponential loss averaged<br>
over the batch.</p>
<p>As an aside, you might not want to use an exponential loss.<br>
Depending on your problem, some other loss function might<br>
work better.</p>
<p>Good luck.</p>
<p>K. Frank</p>

Maria_Fernanda_De_La: <p>Thank you so much! This is so helpful!</p>

Maria_Fernanda_De_La: <p>Hi I’m sorry is this has an obvious answer, but I’m struggling a bit understanding the exponential loss function.</p>
<p>Why do you multiply the labels times 2.0 and subtract 1.0</p>
<p>Thank you so much in advance!</p>

KFrank: <p>Hi Maria!</p>
<aside class="quote no-group" data-username="Maria_Fernanda_De_La" data-post="4" data-topic="49146" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maria_fernanda_de_la/48/11812_2.png" class="avatar"> Maria_Fernanda_De_La:</div>
<blockquote>
<p>Hi I’m sorry is this has an obvious answer, but I’m struggling a bit understanding the exponential loss function.</p>
<p>Why do you multiply the labels times 2.0 and subtract 1.0</p>
</blockquote>
</aside>
<p>In short,  <code>2 * label - 1</code> maps a label equal to the values {0, 1}<br>
to the values {-1, 1}, to better match your output (the “logit”)<br>
running from -infinity to +infinity.</p>
<p>Some more details:</p>
<p>First, I don’t know if there is one single standard definition of<br>
exponential loss function.  (There might be; I just don’t know.)</p>
<p>Anyway, I think of it as being:</p>
<pre><code class="lang-plaintext">   exp (mismatch between prediction and actual),
</code></pre>
<p>where <code>exp  (mismatch)</code> is used to strongly amplify being wrong.</p>
<p>In the case of a binary classification problem, it is often convenient<br>
to have (and you often do have) numerical class labels where<br>
<code>label = 0</code> indicates the “negative” class and <code>label = 1</code><br>
indicates the “positive” class.</p>
<p>To be concrete, let’s say that the input to your neural-network<br>
model is 65,536 floating-point grayscale values that represent<br>
the pixels in a 256x256 grayscale image.  Your model has a<br>
single floating-point output.  Your known class labels are 0 and<br>
1, where 1 indicates that the image is a picture of a bird, and<br>
0 indicates that it is not a bird.  (“Not a bird” means that it might<br>
be a picture of nothing,  or it might be a picture of something<br>
other than a bird, but it’s not a bird.)</p>
<p>The number-one rule is that your output means whatever you<br>
train it to mean, but we will understand your output as follows:<br>
A large positive number means that your model really thinks<br>
the image is a bird; a moderate positive number means probably<br>
a bird; 0 means your model has no opinion – it might be a bird,<br>
but, equally likely, might not; a negative number means it probably<br>
isn’t a bird.  Algebraically larger (more positive or less negative)<br>
means it’s more likely to be a bird, and algebraically smaller (more<br>
negative or less positive) means less likely.</p>
<p>Now to your specific question:</p>
<p>Your label values are 0 and 1.  <code>2 * label - 1</code> maps these two<br>
values to -1 and 1.  When <code>label = 0</code>, <code>exp (label * output)</code><br>
would always be <code>1</code>, regardless of the value of <code>output</code>.  So, for a<br>
non-bird image (<code>label = 0</code>), your model would not be penalized<br>
for returning a very large, positive output – “Yes, it’s really a bird” –<br>
nor rewarded for returning a negative output – “No, it’s not a bird.”<br>
Mapping {0, 1} to {-1, 1} fixes this.</p>
<p>If you have a sharp eye, you should now notice that I missed a<br>
minus sign in the loss function I posted above.  It should actually<br>
be:</p>
<pre data-code-wrap="python"><code class="lang-python">def myExpLoss (logits, labels):
    return  (( -( (2.0 * labels.float() - 1.0) * logits ) ).exp()).mean()
</code></pre>
<p>With this minus sign, this loss penalizes (loss &gt; 1) for being<br>
wrong, and rewards (loss &lt; 1) for being right.  (A bigger – more<br>
positive – loss means your model is getting things wrong; a<br>
smaller loss means your model is getting things right.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

Maria_Fernanda_De_La: <p>This makes perfect sense now, thank you so much!</p>

ZN_W: <p>Hi Frank!</p>
<p>“A large positive number means that your model really thinks<br>
the image is a bird; a moderate positive number means probably<br>
a bird; 0 means your model has no opinion – it might be a bird,<br>
but, equally likely, might not; a negative number means it probably<br>
isn’t a bird. Algebraically larger (more positive or less negative)<br>
means it’s more likely to be a bird, and algebraically smaller (more<br>
negative or less positive) means less likely.”</p>
<p>Does this acturally also the explanation of a model with last node activated by Tanh()?</p>
<p>Thanks!</p>

KFrank: <p>Hi ZN!</p>
<aside class="quote no-group" data-username="ZN_W" data-post="7" data-topic="49146" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zn_w/48/23729_2.png" class="avatar"> ZN_W:</div>
<blockquote>
<p>Does this acturally also the explanation of a model with last node activated by Tanh()?</p>
</blockquote>
</aside>
<p>Let me say again what I said at the beginning of the paragraph<br>
you quoted from:</p>
<p>“The number-one rule is that your output means whatever you<br>
train it to mean …”</p>
<p>So, yes, you can have the output of your model pass through<br>
a final <code>tanh()</code>.  And if you then <em>train</em> your model so that<br>
algebraically larger values mean that your sample is more<br>
likely to be a bird, then, yes, that is what the output of your<br>
model will mean.</p>
<p>Best.</p>
<p>K. Frank</p>

ZN_W: <p>Many thanks for your reply!</p>
<p>Cheers!</p>