Weights in cross-entropy loss

sezer: <p>Hi,</p>
<p>Cross-entropy with weights is defined as follows [1]:<br>
loss(x,class) = weight[class](−x[class] + log(∑_j ​exp(x[j])))</p>
<p>Why the normalization term (denominator of softmax regression) is weighted by weight[class], too? Shouldn’t it be the sum of weighted exponentials as below?</p>
<p>loss(x,class) = −weight[class]*x[class] + log( ∑_j (weight[j] * exp(x[j])))</p>
<p>[1] <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="inline-onebox" rel="noopener nofollow ugc">CrossEntropyLoss — PyTorch 1.7.0 documentation</a></p>

InnovArul: <p>In my understanding, <code>weight</code> is used to reweigh the losses from different classes (to avoid class-imbalance scenarios), rather than influencing the softmax logits.</p>
<p>Consider that the loss function is independent of softmax.<br>
That is, In the cross-entropy loss function,<br>
<code>L_i(y, t) = -t_ij log y_ij</code> (here <code>t_ij=1</code>)</p>
<p><code>y_i</code> is the probability vector that can be obtained by any other way than just softmax. With <code>weight</code>, we are trying to reweigh <code>L_i(y, t)</code> based on the ground truth class.</p>

sezer: <p>But the original form of the cross-entropy loss is exactly the negative log-likelihood of softmax regression:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cda6a7b64cf6de53b21cace5277eeb479dff26e2.png" data-download-href="https://discuss.pytorch.org/uploads/default/cda6a7b64cf6de53b21cace5277eeb479dff26e2" title="resim"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cda6a7b64cf6de53b21cace5277eeb479dff26e2.png" alt="resim" data-base62-sha1="tlgWbdmZ4xnTYvgZUgABKmvSEEi" width="690" height="79" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">resim</span><span class="informations">786×90 8.71 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The denominator (the log term in the first equation) seems to be weighted because of the logarithm, I think I missed this.</p>
<p>I get why this works, but still I’m not sure if it gives the correctly normalized probabilities for each class.</p>

InnovArul: <aside class="quote no-group" data-username="sezer" data-post="3" data-topic="109810">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/cab0a1/48.png" class="avatar"> sezer:</div>
<blockquote>
<p>But the original form of the cross-entropy loss is exactly the negative log-likelihood of softmax regression:</p>
</blockquote>
</aside>
<p>Technically, Cross-entropy (CE) is independent of softmax and a generic concept to measure distances/differences between two probability distributions. It so happens that one of the ways to get probability distributions in deep learning (especially for classification problems) is to use softmax. Ideally, CE loss can be realized by <code>log_softmax + NLL loss</code>. However, for numerical stability reasons, CE loss in pytorch is coupled with Softmax directly.</p>
<aside class="quote no-group" data-username="sezer" data-post="3" data-topic="109810">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/cab0a1/48.png" class="avatar"> sezer:</div>
<blockquote>
<p>The denominator (the log term in the first equation) seems to be weighted because of the logarithm, I think I missed this.</p>
</blockquote>
</aside>
<p>Can you elaborate on this a bit more?</p>

sezer: <aside class="quote no-group" data-username="InnovArul" data-post="4" data-topic="109810">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar"> InnovArul:</div>
<blockquote>
<p>Technically, Cross-entropy (CE) is independent of softmax and a generic concept to measure distances/differences between two probability distributions. It so happens that one of the ways to get probability distributions in deep learning (especially for classification problems) is to use softmax. Ideally, CE loss can be realized by <code>log_softmax + NLL loss</code>. However, for numerical stability reasons, CE loss in pytorch is coupled with Softmax directly.</p>
</blockquote>
</aside>
<p>I understand that, but in this case aren’t they the same? <code>log_softmax + NLL loss</code> should be the same thing as the negative log-likelihood of multinomial logistic regression (which I called softmax regression before). The equation in my previous post shows that the loss is defined as the negative log-likelihood of multinomial logistic regression (I suppose x[class] is class_weights*input).</p>
<p>If I were to define a prior distribution over class labels, I would write something like exp(w_k * x)*q_k / \sum_j (exp(w_j * x)*q_j) where x is the input, w_k is the parameter vector of class k, and q_k is the weight of class k. I think, this way I would get the correctly normalized probabilities considering the weights as well. And I would end up something different than the weighted loss given in [1]. Btw my first attempt (second equation in my first post) seems to be wrong.</p>

InnovArul: <aside class="quote no-group" data-username="sezer" data-post="5" data-topic="109810">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/cab0a1/48.png" class="avatar"> sezer:</div>
<blockquote>
<p>If I were to define a prior distribution over class labels, I would write something like exp(w_k * x)*q_k / \sum_j (exp(w_j * x)*q_j) where x is the input, w_k is the parameter vector of class k, and q_k is the weight of class k.</p>
</blockquote>
</aside>
<p>I am a bit lost about this prior distribution concept.<br>
However, I have some counterpoints with your formulation:</p>
<ol>
<li>
<p>when you multiply <code>q_k</code> in the numerator and take <code>log</code> (for CE loss), <code>q_k</code> will be simply ignored as <code>log q_k</code> is a constant.</p>
</li>
<li>
<p>By trying to multiply <code>q_x</code> with <code>exp()</code> terms (i.e., <code>exp(w_j * x)*q_j</code>), it seems like you are trying to add a constant bias (i.e., <code>exp(w_j * x)*q_j</code> = <code>exp(w_j * x)*exp(log q_j)</code> = <code>exp(w_j * x + log q_j)</code>. The network might learn to pump up the logits to avoid this bias (I am not sure!).</p>
</li>
</ol>
<p>What do you think?</p>