Own loss function for multi class classifikation

roxor: <p>Hey everyone,<br>
Currently I am developing an multiclass classifikator. I try to replace the nn.CrossEntropyLoss  with my own loss function. The background is that I want to penalize some missclassification stronger than others. For example I have 4 classes, then the following matrix will describe the penalty:</p>
<p>0 , 1 , 4 , 5<br>
1 , 0 , 3 , 8<br>
4 , 3 , 0 , 2<br>
5 , 8 , 2 , 0</p>
<p>As you can see, a misclassification from a sample from class 1 as class 2 is not so fatal as class 4. In the first case, my loss function should return 1 and in the second case 5.  I started to implement the function as follow:</p>
<pre><code class="lang-auto">class MyLoss(nn.Module):
    def __init__(self):
         super(MyLoss, self).__init__()
        
    def forward(self, output, target):
           _, predicted = torch.max(output, axis=1)
           loss = []
           for i in range(len(predicted):
                   loss.append(self._loss_matrix[predicted[i].cpu().numpy()][target[i].cpu().numpy()]
           return torch.mean(loss)
</code></pre>
<p>I figured out that the loss can not be backpropagated properly since the torch.max function destroys the backpropagation graph. Has anybody any idea how to replace the torch.max function with a differentiable function. I know that I can use softmax(output)[1] for a 2 class classification problem. Is there something similar for the n class problem?</p>
<p>Thanks in advance!!</p>

KFrank: <p>Hi roxor!</p>
<aside class="quote no-group" data-username="roxor" data-post="1" data-topic="115448" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/4da419/48.png" class="avatar"> roxor:</div>
<blockquote>
<p>I want to penalize some missclassification stronger than others.</p>
</blockquote>
</aside>
<p>One approach is to use cross-entropy with probabilistic or <em>soft</em> targets.</p>
<p>This post has further discussion and details:</p>
<aside class="quote quote-modified" data-post="2" data-topic="114208">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/label-smoothing-for-only-a-subset-of-classes/114208/2">Label smoothing for only a subset of classes</a> 
  </div>
  <blockquote>
    Hi macazinc! 

Here’s what I think you’re asking: 
You have a multi-class (30 classes) classification problem.  You know 
that for most of your classes, your ground-truth target labels are 
correct, but your labels sometimes mix up two of your classes, say 4 
and 9,  Let’s say that a sample labelled 4 is actually a 9 25% of the 
time, and that a sample labelled 9 is actually a 4 10% of the time. 
You will (most likely) want to use cross-entropy loss, but pytorch only 
provides a version that tak…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

roxor: <p>Hi KFrank,<br>
thanks for your fast reply!<br>
I read the post you mentioned and try to implement it. The first aspect I recognised was that my target vector must have the shape of (Batchsize, C) now instead of (Batchsize, ). So instead of the ground truth value, I have to pass in the corresponding line of the distance matrix. Did I get it right?</p>
<p>Attached you can find my first draft: I made some modifications from your code since, as far as I know the usage of  Variabel is deprected:</p>
<pre><code class="lang-auto">def softXEnt (input, target):
    logprobs = torch.nn.functional.log_softmax (input, dim = 1)
    return  -(target * logprobs).sum() / input.shape[0]

# input values are logits
# 4 possible classes and 2 samples 
inputBad = torch.tensor([[3,5,2.5,6], [7,5.5,2,0]])
inputGood = torch.tensor([[0,0,0,1], [1,0,0,0]], dtype=torch.float)

# ground truth is [4,1]
# Recap the distance matrix is:
# 0 , 1 , 4 , 5
# 1 , 0 , 3 , 8
# 4 , 3 , 0 , 2
# 5 , 8 , 2 , 0

# For softmax we need a float tensor
penalize_target = torch.tensor([[5 , 8 , 2 , 0], [0 , 1 , 4 , 5]], dtype=torch.float)
target = torch.nn.functional.softmax(penalize_target, dim = 1)

print("SoftXEnt result bad prediction")
print(softXEnt (inputBad, target))
print("SoftXEnt result good prediction")
print(softXEnt(inputGood, target))
</code></pre>
<p>And the output is.</p>
<pre><code class="lang-auto">SoftXEnt result bad prediction
tensor(4.0218)
SoftXEnt result good prediction
tensor(1.7411)
</code></pre>
<p>As we can see, the pad predication is stronger penalized than the good prediction of the classificator. However, shouldn’t the good prediction result in a tensor(0.0) since the prediction is equal to the ground truth?</p>
<p>Thanks! Looking forward to your opinion!<br>
Best<br>
roxor</p>

KFrank: <p>Hi roxor!</p>
<aside class="quote no-group" data-username="roxor" data-post="3" data-topic="115448" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/4da419/48.png" class="avatar"> roxor:</div>
<blockquote>
<pre><code class="lang-auto"># input values are logits
# 4 possible classes and 2 samples 
inputBad = torch.tensor([[3,5,2.5,6], [7,5.5,2,0]])
inputGood = torch.tensor([[0,0,0,1], [1,0,0,0]], dtype=torch.float)
</code></pre>
</blockquote>
</aside>
<p>Your <code>inputBad</code> and <code>inputGood</code> don’t represent the predictions you<br>
think they do.  As your comment notes, they are logits.  To make them<br>
a little easier to understand, let’s turn them into probabilities with<br>
<code>softmax()</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.nn.functional.softmax (inputBad, dim = 1)
tensor([[3.4387e-02, 2.5408e-01, 2.0857e-02, 6.9067e-01],
        [8.1249e-01, 1.8129e-01, 5.4745e-03, 7.4090e-04]])
&gt;&gt;&gt; torch.nn.functional.softmax (inputGood, dim = 1)
tensor([[0.1749, 0.1749, 0.1749, 0.4754],
        [0.4754, 0.1749, 0.1749, 0.1749]])
</code></pre>
<p>First note that <code>inputGood</code> assigns your ground-truth label (class-4<br>
and class-1 for your two samples, respectively) of 47.5% for both<br>
samples.  Your ground-truth label does have the highest probability,<br>
but it does not have a probability of 100%.</p>
<p>To repeat this, the input vector, <code>[0, 0, 0, 1]</code>, does <em>not</em> mean a<br>
probability of 100% for class-4 – it is a vector of logits that translates<br>
to a probability of 47.5% for class-4.  So it is <em>not</em> a pure class-4<br>
prediction.</p>
<p>Second, <code>inputBad</code> translates to ground-truth probabilities of 69.1%<br>
and 81.2% for the first and second samples, respectively.  So<br>
<code>inputBad</code> actually gives <em>stronger</em> predictions for the ground truth<br>
than does <code>inputGood</code>.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">penalize_target = torch.tensor([[5 , 8 , 2 , 0], [0 , 1 , 4 , 5]], dtype=torch.float)
target = torch.nn.functional.softmax(penalize_target, dim = 1)
</code></pre>
</blockquote>
</aside>
<p>Now let’s look at <code>penalize_target</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.nn.functional.softmax(penalize_target, dim = 1)
tensor([[4.7299e-02, 9.5003e-01, 2.3549e-03, 3.1870e-04],
        [4.8372e-03, 1.3149e-02, 2.6410e-01, 7.1791e-01]])
</code></pre>
<p>In both cases your target probability is <em>smallest</em> for your ground-truth<br>
entry (class-4 and class-1, respectively).  The problem is that a small<br>
number in your distance matrix (small is “good,” and zero is “perfect”)<br>
translates into a small probability – backwards of what you want.</p>
<p>A couple of comments:</p>
<p>First, although you could reverse-engineer soft-cross-entropy to give<br>
you a distance-matrix-like loss function by appropriately adjusting<br>
your soft targets, you don’t really want a distance-matrix loss.  You<br>
really do want something like cross-entropy, with its logarithmic<br>
divergences.  In practice, it seems to train better.</p>
<p>Take a look at the numerical details of the class-4 / class-9 example<br>
in the post I linked to.  That shows concretely how to penalize certain<br>
misclassifications more than others.</p>
<p>Best.</p>
<p>K. Frank</p>

roxor: <p>Hi KFrank,<br>
thanks for the explenation about the logit and all the details about my code.</p>
<p>Actually, all I want is the distance-matrix loss.</p>
<p>I read the class-4 / class-9 example again. However, I don’t get how to use it in my example since my data are labeled correctly. I want to penalize the predictions based on the matrix and I don’t get it how to do it with the approach you recommended in the example. How would you do this?</p>
<p>Thanks!<br>
Best,<br>
roxor</p>

KFrank: <p>Hi roxor!</p>
<aside class="quote no-group" data-username="roxor" data-post="5" data-topic="115448" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/4da419/48.png" class="avatar"> roxor:</div>
<blockquote>
<p>Actually, all I want is the distance-matrix loss.</p>
</blockquote>
</aside>
<p>Two comments:</p>
<p>First, as I said initially, cross-entropy with soft targets it a way to penalize<br>
some misclassifications differently from others.  But it is not the same as<br>
a difference-matrix loss.</p>
<p>Second, you should think hard about whether you really want to use a<br>
difference-matrix loss.  What you want in practice is a loss that causes<br>
your network to train effectively and then give accurate predictions.<br>
A cross-entropy-like loss is likely to work better in this regard.  In any<br>
event, you shouldn’t just assume that a distance-matrix loss will work<br>
better – at a minimum, you should try both and see which gives better<br>
results.</p>
<p>Best.</p>
<p>K. Frank</p>