The effects of Double Logarithms (Log Cross Entropy Loss) + Overfitting

username: tsuijenk
cooked: <p>Hello everyone,</p>
<p>My neural network involves pixel classifications. It involves two losses: one is a binary cross entropy, and the other is a multi-label cross entropy.</p>
<p>The yellow graphs are the ones with double logarithm, meaning that we log(sum(ce_loss)). The red pink graphs are the ones with just sum(ce_loss).</p>
<p>The dash lines represent validation step. The solid lines represent training step.</p>
<p>The top yellow and top red-pink figures both represent the count of 1s. Both are supposed to converge to 30. It is clear that the top yellow figure demonstrate that both training and validation converge to 30. While the bottom red-pink figure demonstrate that only training converges to 30, while validation converges to 0… !</p>
<p>My questions are:</p>
<ol>
<li>I have not been able to find any literature whatsoever that anyone ever used a double logarithm (referencing yellow graphs). But the results are clearly much better than just a typical cross entropy loss (referencing red graphs).</li>
</ol>
<p>Does anyone know why adding a logarithm on a cross entropy would improve the results? My original purpose of adding the outer logarithm on a CE loss was to increase computational stability. And it does seems to serve my purpose.</p>
<ol start="2">
<li>
<p>The yellow graphs have high training and validation accuracy (due to expected counts of thirty 1s), though the validation loss (middle graph) is increasing. Is this a case of overfitting?</p>
</li>
<li>
<p>The red graphs have high training accuracy yet poor validation accuracy. The validation loss (middle and right graph) for both losses are increasing. Is this also a case of overfitting?</p>
</li>
</ol>
<p>My colleague questioning the usage of double logarithm, but clearly it seems that double logarithm is performing better than without the second(outer) logarithm.</p>
<p>Any advice and suggestions would be great! Thank you.</p>
<p><a href="https://i.stack.imgur.com/ptW5i.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/ptW5i.png" alt="enter image description here" width="616" height="478"></a><br>
<a href="https://i.stack.imgur.com/K2w1W.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/K2w1W.png" alt="enter image description here" width="690" height="167"></a><br>
<a href="https://i.stack.imgur.com/1GPcr.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/1GPcr.png" alt="enter image description here" width="690" height="462"></a></p>

username: my3bikaht
cooked: <p>I’m not sure what are you trying to achieve by log over ce loss</p>
<p>With regular training you are minimizing ce loss, mean or sum doesn’t matter that much, bcs in the end increased scale will be compensated by smaller model weights.<br>
But by applying log, your new minimum will be achieved if ce loss = 1/N, bcs log(1)=0</p>

username: KFrank
cooked: <p>Hi Sergey!</p>
<aside class="quote no-group" data-username="my3bikaht" data-post="2" data-topic="142867" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/50afbb/48.png" class="avatar"> my3bikaht:</div>
<blockquote>
<p>But by applying log, your new minimum will be achieved if ce loss = 1/N, bcs log(1)=0</p>
</blockquote>
</aside>
<p>To avoid any confusion, let’s be clear that this is not correct.</p>
<p>It is true that many loss functions have zero as their minimum, but<br>
gradient descent and the pytorch optimizers don’t care whether this<br>
is true or not; they simply push the loss function to smaller – and<br>
potentially negative – values.</p>
<p><code>log()</code> is a so-called <em>monotone</em> function so <code>log (ce_loss)</code> achieves<br>
its minimum exactly where <code>ce_loss</code> achieves its minimum.  <code>ce_loss</code><br>
ranges from <code>0.0</code> to <code>inf</code> and, correspondingly, <code>log (ce_loss)</code> ranges<br>
from <code>-inf</code> to <code>inf</code>.  When your predictions are “perfect,” <code>ce_loss</code> is <code>0.0</code>,<br>
its minimum, and <code>log (ce_loss)</code> is <code>-inf</code>, its minimum.</p>
<p>Whether you train with <code>ce_loss</code> or <code>log (ce_loss)</code>, the optimization<br>
drives both towards their corresponding minima, namely <code>0.0</code> for  <code>ce_loss</code><br>
and <code>-inf</code> for <code>log (ce_loss)</code>.  Nothing special happens when <code>ce_loss</code><br>
becomes <code>1/N</code> (or <code>1</code> or any other value other than zero), and the fact that<br>
<code>log (ce_loss)</code> happens to be <code>0.0</code> when <code>ce_loss = 1/N</code> is irrelevant;<br>
the optimization simply continues to drive <code>log (ce_loss)</code> to algebraically<br>
smaller values, and thus to negative values on down towards <code>-inf</code>.</p>
<p>(Consider what would happen if one were to use <code>log (ce_loss) + 17.2</code><br>
or <code>log (ce_loss) - 105.9</code> as the loss function for training.  Would the<br>
gradients change?  Would the value <code>17.2</code> or <code>105.9</code> have any effect on the<br>
result of training?)</p>
<p>Best.</p>
<p>K. Frank</p>

username: my3bikaht
cooked: <p>Right, thanks for catching this.<br>
I mentally flipped the gradient direction like there was abs )</p>

username: mMagmer
cooked: <p>i think by using second log, you are only changing effective learning rate. i.e. if sum ce is around <code>10</code> then your effective learning rate is <code>.1*lr</code>.</p>