Passing the weights to CrossEntropyLoss correctly

ivan-bilan: <p>Hi,</p>
<p>I just wanted to ask how the mechanism of passing the weights to CrossEntropyLoss works. Currently, I have a list of class labels that are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10].</p>
<p>I create the loss function in the <strong>init</strong> and pass the weights to the loss:</p>
<pre><code class="lang-auto">weights = [0.5, 1.0, 1.0, 1.0, 0.3, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
class_weights = torch.FloatTensor(weights).cuda()
self.criterion = nn.CrossEntropyLoss(weight=class_weights)
</code></pre>
<p>Then in the update step, I pass the labels of my current batch to the loss function:</p>
<pre><code class="lang-auto">self.model.train()
self.optimizer.zero_grad()
logits, _ = self.model(inputs)
loss = self.criterion(logits, labels)
</code></pre>
<p>Now the labels can be something like this: [2, 4, 2, 1, 0, 4, 5]</p>
<p>So, since the weights are passed as a list, will they be automatically mapped to the current labels passed? like so:  [2: 1.0, 4: 0.3,  2: 1.0, 1: 1.0, 0: 0.5, 4: 0.3, 5: 1.0]?</p>
<p>The reason I am asking this is that I would be much more comfortable passing a dictionary of weights to the loss function, instead of a list. So that I am sure the weights are mapped to the proper classes. Can someone elaborate on the way weights are passed to the loss function here?</p>

richard: <p>Your interpretation is correct, that is how the class weights will work with CrossEntropyLoss.</p>
<p>There’s a little more detail on the docs on how this is done: <a href="http://pytorch.org/docs/master/nn.html?highlight=nll%20loss#torch.nn.NLLLoss">http://pytorch.org/docs/master/nn.html?highlight=nll%20loss#torch.nn.NLLLoss</a></p>

amy: <p>how are we getting the weights? Like if the true distribution while testing is 10:1 (for sample frequency) and in training it is 1:1 (sample frequency). What should the weights be in that case?</p>

soltius: <p>You shouldn’t have such variations in your distributions between train and test anyhow.</p>

zezhishao: <p>Hi,<br>
I noticed that the <code>KLDivLoss</code> class does not have the parameter <code>weight</code>, so how can I passing the weight to it?</p>

BikashgG: <p>Hi, shouldn’t your weights/class_weights vector be normalised (i.e. the total adds up to 1) before passing it to the CrossEntropyLoss function?</p>

MariosOreo: <p>I think we don’t need to normalize the weights of classes, using invert class sampling count is okay.</p>

ptrblck: <p>Small addition to <a class="mention" href="/u/mariosoreo">@MariosOreo</a>’s answer:<br>
if your loss function uses <code>reduction='mean'</code>, the loss will be normalized by the sum of the corresponding weights for each element. If you are using <code>reduction='none'</code>, you would have to take care of the normalization yourself.<br>
Here is a small example:</p>
<pre><code class="lang-python">x = torch.randn(10, 5)
target = torch.randint(0, 5, (10,))

weights = torch.tensor([1., 2., 3., 4., 5.])
criterion_weighted = nn.CrossEntropyLoss(weight=weights)
loss_weighted = criterion_weighted(x, target)

criterion_weighted_manual = nn.CrossEntropyLoss(weight=weights, reduction='none')
loss_weighted_manual = criterion_weighted_manual(x, target)
loss_weighted_manual = loss_weighted_manual.sum() / weights[target].sum()

print(loss_weighted == loss_weighted_manual)
</code></pre>

balloch: <p>This is actually exactly related to my problem; working in a continual learning setting, the number of classes changes through time; If I want to use weight balancing, but the weights vector changes through time, what do I do here? Do I have to reinitialize my loss for every new task to use weight balancing?</p>

gunshi_gupta: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> will there be gradients backpropp-ed through these weights if we constructed the weights from the logits themselves (as in focal losses for object detection (<a href="https://arxiv.org/pdf/1708.02002.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1708.02002.pdf</a>))</p>

ptrblck: <p>No, as the derivative for <code>weights</code> is not implemented.<br>
You could check it by passing a <code>weights</code> tensor using <code>requires_grad=True</code>, which will raise the exception:</p>
<pre><code class="lang-python">RuntimeError: the derivative for 'weight' is not implemented
</code></pre>
<p>However, have a look at <a href="https://github.com/mbsariyildiz/focal-loss.pytorch/blob/6551bd3e433ce41020b6bc8d99221eb6cd10ae17/focalloss.py#L33" rel="nofollow noopener">this implementation</a> of the focal loss.</p>

gunshi_gupta: <p>I see, thanks.<br>
Alternatively, even if i construct this weights matrix and matrix multiply it with the loss matrix i get back when using the loss with reduction = ‘none’, it should be equivalent right?</p>

ptrblck: <p>I think so, but would need to see the code to be sure. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
However, I’m not sure, how you are constructing the weights and if they should also be trainable or not.</p>

gunshi_gupta: <p>In my case the final focal loss computation looks like the code below (focal loss is supposed to backprop the gradients even through the weights as i understand, since none of the repos i referenced including the one mentioned above, calls detach() on these weights for which backward() is well defined):</p>
<pre><code>    alpha_factor = torch.ones(targets.shape).cuda() * alpha
    alpha_factor = torch.where(targets_eq_1, 1. - alpha_factor, alpha_factor)             
    focal_weight = torch.where(targets_eq_1, 1. - classification, classification)
    focal_weight = alpha_factor * torch.pow(focal_weight, gamma)
    cls_loss = F.binary_cross_entropy_with_logits(classification, targets, size_average=False, reduction = 'none')
    final_loss = focal_weight*cls_loss</code></pre>

ptrblck: <p>You might be right, but wouldn’t you recreate <code>focal_weight</code> in each iteration (based on the targets)?<br>
Where would the gradient in <code>focal_weight</code> be used in this case?</p>

gunshi_gupta: <p>Yes, this focal weight would change every iteration. I myself wasn’t entirely sure if focal weights should be backpropagated into, but it seems that way from open-source implementations (the original paper doesn’t have a clarification <a href="https://arxiv.org/abs/1708.02002" rel="nofollow noopener">https://arxiv.org/abs/1708.02002</a>)<br>
In this case, the gradients would flow from the focal weights into/through the ‘classification’ vector (which are the logits) I guess.</p>

John_Grabner: <p>I’m missing something. Loss (Error on rare class) same as Loss (Error on common class)?</p>
<blockquote>
<p>weight_inbalance=torch.tensor([1000.0,1.0]) # very rare first class<br>
loss_inbalance = torch.nn.CrossEntropyLoss(weight=weight_inbalance,reduction=‘mean’)</p>
<ul>
<li>batch of 2 samples, toggle only first sample<br>
inbal_err_on_rare = loss_inbalance(torch.tensor([[-1.,-1],[-1,1]]), torch.tensor([0,1]))<br>
inbal_err_on_com = loss_inbalance(torch.tensor([[1.,1],[-1,1]]), torch.tensor([0,1]))<br>
print(inbal_err_on_rare, inbal_err_on_com)<br>
tensor(0.6926) tensor(0.6926)</li>
</ul>
</blockquote>
<p>The same experiment on a ballance class yeilds the same equivalence, just offset in value.</p>
<blockquote>
<p>weight_balance=torch.tensor([1.0,1.0])<br>
loss_balance = torch.nn.CrossEntropyLoss(weight=weight_balance,reduction=‘mean’)<br>
bal_err_on_not_rare = loss_balance(torch.tensor([[-1.,-1],[-1,1]]), torch.tensor([0,1]))<br>
bal_err_on_com = loss_balance(torch.tensor([[1.,1],[-1,1]]), torch.tensor([0,1]))<br>
print(bal_err_on_not_rare, bal_err_on_com)<br>
tensor(0.4100) tensor(0.4100)</p>
</blockquote>
<p>I generated the contours, and they look the same, just shifted.</p>
<blockquote>
<p>weight_inbalance=torch.tensor([1000.0,1.0]) # very rare first class<br>
weight_balance=torch.tensor([1.0,1.0])<br>
loss_inbalance = torch.nn.CrossEntropyLoss(weight=weight_inbalance,reduction=‘mean’)<br>
loss_balance = torch.nn.CrossEntropyLoss(weight=weight_balance,reduction=‘mean’)<br>
X = np.arange(-1., 1.1, 0.1)<br>
Y = np.arange(-1., 1.1, 0.1)<br>
Z_inbalance = np.zeros((X.shape[0],Y.shape[0]))<br>
Z_balance = np.zeros((X.shape[0],Y.shape[0]))<br>
<span class="hashtag-raw">#print</span>(X.shape, Y.shape, Z.shape)</p>
<p>i = -1<br>
for x in X:<br>
i+=1<br>
j=-1<br>
for y in Y:<br>
j+=1<br>
batch=torch.tensor([[x, y], [-1, 1]], dtype=torch.float32)</p>
<pre><code>    X[i], Y[j], Z_inbalance[i, j] =x,y,loss_inbalance(batch, torch.tensor([0,1]))
    X[i], Y[j], Z_balance[i, j] =x,y,loss_balance(batch, torch.tensor([0,1]))
</code></pre>
<p>fig, ax = plt.subplots(2)</p>
<p>CS0 = ax[0].contour(Y,X,Z_inbalance,cmap=cm.coolwarm)<br>
ax[0].clabel(CS0, inline=1, fontsize=14)<br>
<span class="hashtag-raw">#ax</span>[0].set_title(‘Inbalanced Classes’)<br>
ax[0].set_ylabel(‘true class is rare’)<br>
<span class="hashtag-raw">#ax</span>[0].set_xlabel(“not selected class is Common”)</p>
<p>CS1 = ax[1].contour(Y,X,Z_balance,cmap=cm.coolwarm)<br>
ax[1].clabel(CS1, inline=1, fontsize=14)<br>
<span class="hashtag-raw">#ax</span>[1].set_title(‘Balanced Classes’)<br>
ax[1].set_ylabel(‘true class is common’)<br>
ax[1].set_xlabel(“not selected class is Common”)</p>
</blockquote>
<p>So either I am missing the point of weights, or made a newbie error. Either way, I would appreciate some comments.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/9/293dcdf39dc322c3db509fe4cb7e9b0e1ffa8b05.png" alt="image" data-base62-sha1="5SPZuxu3SUNt0W0Ut2Jui81RVOt" width="426" height="285"></p>

ptrblck: <p>Note that in your first example both output tensors predict class1 only, so it’s expected to get the same loss value.<br>
You can check it via:</p>
<pre><code class="lang-python">torch.argmax(torch.tensor([[-1.,-1],[-1,1]]), 1)
&gt; tensor([1, 1])

torch.argmax(torch.tensor([[1.,1],[-1,1]]), 1)
&gt; tensor([1, 1])
</code></pre>
<p>I haven’t looked into the second experiment as the formatting is unclear, but you could rerun it with another reduction (e.g. <code>reduction='sum'</code>), since the default <code>reduction='mean'</code> would normalize the loss with the applied weights.</p>

John_Grabner: <p>Thank you, again.<br>
Not following how the argmax relates CrossEntropyLoss. Please elaborate.</p>
<p>You are correct, somehow the “mean” nullifies the weight bias.</p>
<p>Experiment 1 shows an error on any logit when classes are balanced results in the same loss.<br>
Experiment 2 shows if the rare class is the true class, the losses are amplified when any error occurs.</p>
<p><strong>Experiment 1:</strong></p>
<ul>
<li>balanced 2 classes,</li>
<li>batch of 2 samples,</li>
<li>introduce  a single logit error and observe loss for true class = 0, 0 and 1, 1</li>
</ul>
<blockquote>
<p>loss =      torch.nn.CrossEntropyLoss(weight=torch.tensor([1.,1.]),reduction=‘sum’)<br>
print(“2 balanced class, reduction=sum”)</p>
<p>true_class_index = torch.tensor([0,0]) # so perfect prediction is [1,-1], [1,-1]<br>
incorrect_sample_0_class_0 = loss(torch.tensor([[-1., -1],[1.,-1]]), true_class_index)<br>
incorrect_sample_0_class_1 = loss(torch.tensor([[1., 1],[1.,-1]]), true_class_index)<br>
incorrect_sample_1_class_0 = loss(torch.tensor([[1., -1],[-1.,-1]]), true_class_index)<br>
incorrect_sample_1_class_1 = loss(torch.tensor([[1., -1],[1.,1]]), true_class_index)<br>
print(f"all have one logit wrong {incorrect_sample_0_class_0:0.2f}, {incorrect_sample_0_class_1:0.2f}, {incorrect_sample_1_class_0:0.2f}, {incorrect_sample_1_class_1:0.2f}")</p>
<ul>
<li>all have one logit wrong 0.82, 0.82, 0.82, 0.82</li>
</ul>
<p>true_class_index = torch.tensor([1,1]) # so perfect prediction is [1,-1], [1,-1]<br>
incorrect_sample_0_class_0 = loss(torch.tensor([[1., 1],[-1.,1]]), true_class_index)<br>
incorrect_sample_0_class_1 = loss(torch.tensor([[-1., -1],[-1.,1]]), true_class_index)<br>
incorrect_sample_1_class_0 = loss(torch.tensor([[-1., 1],[1.,1]]), true_class_index)<br>
incorrect_sample_1_class_1 = loss(torch.tensor([[-1., 1],[-1.,-1]]), true_class_index)<br>
print(f"all have one logit wrong {incorrect_sample_0_class_0:0.2f}, {incorrect_sample_0_class_1:0.2f}, {incorrect_sample_1_class_0:0.2f}, {incorrect_sample_1_class_1:0.2f}")</p>
<ul>
<li>all have one logit wrong 0.82, 0.82, 0.82, 0.82</li>
</ul>
</blockquote>
<p><strong>Experiment 2:</strong></p>
<ul>
<li>unbalanced 2 classes,</li>
<li>batch of 2 samples,</li>
<li>introduce  a single logit error and observe loss for true class = 0, 0 and 1, 1</li>
</ul>
<blockquote>
<p>loss =      torch.nn.CrossEntropyLoss(weight=torch.tensor([10000.,1.]),reduction=‘sum’)<br>
print(“2 unbalanced class, reduction=sum”)</p>
<p>true_class_index = torch.tensor([0,0]) # so perfect prediction is [1,-1], [1,-1]<br>
incorrect_sample_0_class_0 = loss(torch.tensor([[-1., -1],[1.,-1]]), true_class_index)<br>
incorrect_sample_0_class_1 = loss(torch.tensor([[1., 1],[1.,-1]]), true_class_index)<br>
incorrect_sample_1_class_0 = loss(torch.tensor([[1., -1],[-1.,-1]]), true_class_index)<br>
incorrect_sample_1_class_1 = loss(torch.tensor([[1., -1],[1.,1]]), true_class_index)<br>
print(f"all have one logit wrong {incorrect_sample_0_class_0:0.2f}, {incorrect_sample_0_class_1:0.2f}, {incorrect_sample_1_class_0:0.2f}, {incorrect_sample_1_class_1:0.2f}")<br>
<strong>&gt; - all have one logit wrong 8200.75, 8200.75, 8200.75, 8200.75</strong></p>
<p>true_class_index = torch.tensor([1,1]) # so perfect prediction is [1,-1], [1,-1]<br>
incorrect_sample_0_class_0 = loss(torch.tensor([[1., 1],[-1.,1]]), true_class_index)<br>
incorrect_sample_0_class_1 = loss(torch.tensor([[-1., -1],[-1.,1]]), true_class_index)<br>
incorrect_sample_1_class_0 = loss(torch.tensor([[-1., 1],[1.,1]]), true_class_index)<br>
incorrect_sample_1_class_1 = loss(torch.tensor([[-1., 1],[-1.,-1]]), true_class_index)<br>
print(f"all have one logit wrong {incorrect_sample_0_class_0:0.2f}, {incorrect_sample_0_class_1:0.2f}, {incorrect_sample_1_class_0:0.2f}, {incorrect_sample_1_class_1:0.2f}")</p>
<ul>
<li>all have one logit wrong 0.82, 0.82, 0.82, 0.82</li>
</ul>
</blockquote>
<p>btw, is there a way to post code snippets and not loose indentation on this site?</p>

John_Grabner: <p>An excellent summary for CrossEntropy <a href="https://youtu.be/ErfnhcEV1O8" rel="nofollow noopener">https://youtu.be/ErfnhcEV1O8</a></p>