Custom Loss penalizing similar categegories more strong

username: jonas15
cooked: <p>Hi everyone,</p>
<p>I’m designing at the moment my network, which shall classify images relating how many people are in there. So my output dimension is 6 - 0 … 5 people. I’m observing now that the most errors are happening in cases where 4 people are on the image but the network classifies it as 3 or 5. As one idea I would like to define my loss (at the moment I’m using CrossEntropyLoss) in a manner that these cases are more weighted in compare to other cases.</p>
<p>How can you do this?</p>
<p>Thanks and best regards<br>
Jonas</p>

username: beaupreda
cooked: <p>Hello,</p>
<p>If you look at the CrossEntropyLoss <a href="https://pytorch.org/docs/1.3.0/nn.html#crossentropyloss" rel="nofollow noopener">documentation</a>, there is a parameter <code>weight</code> that you can specify to give more weight to certain classes. For example,</p>
<pre><code class="lang-auto">    weights = torch.tensor([0.1, 0.1, 0.1, 0.3, 0.3, 0.2], dtype=torch.float32)
    criterion = nn.CrossEntropyLoss(weight=weights)
</code></pre>
<p>which would give more weight to classes 3, 4, 5.</p>
<p>Hope this helps!</p>

username: jonas15
cooked: <p>Hi,</p>
<p>Thanks for you fast reply! I’ve already seen this weight-option, but it doesn’t completely fit to my idea, because I’m more thinking about a “dynamic”-weighting, meaning if</p>
<ul>
<li>the target is 4, I would like to have a weighting of [0.1, 0.1, 0.1, 0.3, 0.3, 0.3]</li>
<li>the target is 2, I would like to have a weighting of [0.1, 0.3, 0.3, 0.3, 0.1, 0.1]</li>
</ul>
<p>Is something like this possible?</p>

username: beaupreda
cooked: <p>You could modify the weights of the criterion like this:</p>
<pre><code class="lang-auto">criterion.weight = torch.tensor([0.3, 0.3, 0.3, 0.1, 0.1, 0.1], dtype=torch.float32)
</code></pre>
<p>So I guess you could have a <code>if</code> condition on the target value and change the <code>weight</code> attribute as you want. Is this what you want?</p>

username: jonas15
cooked: <p>Yep, good hint thanks!</p>
<p>I tried to implement a first example but struggling now with the batch-size. I tried making an array for the weight variable like this</p>
<pre><code class="lang-auto">        w = torch.tensor(np.matlib.repmat([0.3, 0.3, 0.3, 0.1, 0.1, 0.1], args.train_batch_size,1),dtype=torch.float32)
</code></pre>
<p>However the Weight-Vector can be only a 1D-Array, how would you face this problem?</p>

username: beaupreda
cooked: <p>Sorry for the late reply.</p>
<p>If I understand correctly, you want to consider the neighbors of the target when computing the loss. <code>nn.CrossEntropyLoss</code> does not permit that, but I implemented a custom loss function that takes into account the 2 neighbors of the current target.</p>
<pre><code class="lang-auto">class CustomLoss(nn.Module):
    def __init__(self, weights):
        super(CustomLoss, self).__init__()
        self.weights = weights

    def forward(self, logits, targets):
        loss = torch.tensor([0], dtype=torch.float32)
        log_probabilities = F.log_softmax(logits)
        for i in range(targets.shape[0]):
            low = int(max(0, targets[i] - 1))
            high = int(min(log_probabilities.shape[1], targets[i] + 2))
            p = log_probabilities[i, low:high]
            loss += -torch.sum(torch.mul(p, self.weights[(low - low):(high - low)]))
        return loss

we = torch.tensor([0.2, 0.5, 0.2], dtype=torch.float32)
criterion1 = CustomLoss(we)
</code></pre>
<p>So basically, the target has a weight of 0.5 while its two neighbors (if they both exists) have weight of 0.2.<br>
Let me know if it helps!</p>

username: jonas15
cooked: <p>Hi,</p>
<p>No worries, thanks for your effort and the code!<br>
This helps me quite a lot, also to see how to implement such custom loss function <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> So far it didn’t worked for my problem, I have to rethink some stuff but now I’ve a great possibility to move on <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>