Create a Variation of Cross Entropy Loss with Per Pair of Classes Weights

DavidHere: <p>I am after creating a Cross Entropy Loss with the addition ow weighing per pair of classes.</p>
<p>For instance, let’s say I have 5 classes, I would like to have greater penalty for the case the input class is <code>1</code> and the output prediction class is <code>5</code> then the case the output is <code>2</code>.</p>
<p>In classic classification it is done by a loss matrix $D$ where $D_{ij}$ is the penalty factor for classifying class <code>i</code> as <code>j</code>.</p>
<p>How can one achieve this in PyTorch using custom loss?<br>
I am asking both in regard to speed optimization and numeric stability.</p>

KFrank: <p>Hi David!</p>
<aside class="quote no-group" data-username="DavidHere" data-post="1" data-topic="157369" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/838e76/48.png" class="avatar"> DavidHere:</div>
<blockquote>
<p>In classic classification it is done by a loss matrix $D$ where $D_{ij}$ is the penalty factor for classifying class <code>i</code> as <code>j</code>.</p>
</blockquote>
</aside>
<p>If I understand your use case, you would not typically do this where<br>
your <code>target</code> (ground-truth labels) are categorical class labels.  This<br>
is because with a certain (“hard,” probability-of-one) class labels, cross<br>
entropy only depends on the predicted probability of the correct class<br>
(so there are no “pairs of classes”).</p>
<p>If you insist on using per-pair weights with categorical labels, then only<br>
the diagonal of your per-pair weight matrix, <code>D</code>, will matter and you can<br>
use pytorch’s built-in <code>CrossEntropyLoss</code> with its <code>weight</code> argument:</p>
<pre data-code-wrap="python"><code class="lang-python">loss_fn = torch.nn.CrossEntropyLoss (weight = torch.diag (D))
</code></pre>
<p>If you have probabilistic (“soft”) labels, then all elements of <code>D</code> will matter<br>
and you can implement per-pair-weighted, probabilistic-label cross entropy<br>
as follow:</p>
<pre data-code-wrap="python"><code class="lang-python"># assumes that input and target both have shape [nBatch, nClass]
def perPairWeightedCE (input, target, D):
    logprobs = torch.nn.functional.log_softmax (input, dim = 1)
    return -(target *  (logprobs @ D.T)).sum() / input.shape[0]
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

DavidHere: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a> ,<br>
Thank you for the assistance.</p>
<p>What I want is to be able to have different penalty for different mistakes.<br>
Let’s say we work on a sample with class <code>1</code>.<br>
I want that in case the net labeled the sample as <code>5</code> to have a bigger penalty than the case it was labeled as <code>2</code>.</p>
<p>Is that clearer?</p>
<p>Thank You.</p>

KFrank: <p>Hi David!</p>
<aside class="quote no-group" data-username="DavidHere" data-post="3" data-topic="157369" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/838e76/48.png" class="avatar"> DavidHere:</div>
<blockquote>
<p>What I want is to be able to have different penalty for different mistakes.</p>
</blockquote>
</aside>
<p>Cross entropy (by definition) doesn’t work this way.  But you can write<br>
some other loss function that does have “per-pair” penalties.</p>
<p>I give a couple of possibilities in the example script, below.  <code>misclassA()</code><br>
just weights the probability for an incorrect prediction by the per-pair<br>
weight given in your matrix <code>D</code>.</p>
<p>My guess is that such a loss function won’t train very well.  In my experience,<br>
the logarithmic divergence for bad predictions in cross entropy seems to<br>
be very helpful for training.</p>
<p><code>misclassB()</code> (which I have not tried out on any kind of training) puts in<br>
such a logarithmic divergence.  Maybe it will work better.</p>
<p>Lastly, it might make sense to use cross entropy as your “base” loss<br>
criterion to get its generally good training properties and then add on<br>
something like <code>misclassA()</code> or <code>misclassB()</code> to also incorporate your<br>
per-pair penalties.</p>
<p>Here is the script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

torch.manual_seed (2022)

# loss is weighted probablity of incorrect prediction
def misclassA (pred, targ, D):   # pred[nBatch, nClass], targ[nBatch], D[nClass, nClass]
    return  (D[targ] * pred).mean()

# give incorrect predictions a logarithmic divergence
# (note, it would be better to work with log-probalities than to use clamp)
def misclassB (pred, targ, D):   # pred[nBatch, nClass], targ[nBatch], D[nClass, nClass]
    return  (D[targ] * (-(pred - 1.0).clamp (min = 1.e-7).log())).mean()

nBatch = 3
nClass = 5

D = torch.rand (nClass, nClass)   # per-pair class weights
D -= D * torch.eye (nClass)       # no penalty for correct predictions

logits = torch.randn (nBatch, nClass)
pred = logits.softmax (dim = 1)
targ = torch.randint (nClass, (nBatch,))

lossA = misclassA (pred, targ, D)
lossB = misclassB (pred, targ, D)

print ('lossA =', lossA)
print ('lossB =', lossB)

lossCE = torch.nn.functional.cross_entropy (logits, targ)   # standard cross entropy for comparison

print ('lossCE =', lossCE)
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">1.12.0
lossA = tensor(0.0818)
lossB = tensor(7.3061)
lossCE = tensor(2.8050)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Royi: <p><a class="mention" href="/u/kfrank">@KFrank</a> , What do you think about something like:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/a/ca8338c3f8e8878feac2ce1d4a0a5f3dbc1bd9a9.png" data-download-href="https://discuss.pytorch.org/uploads/default/ca8338c3f8e8878feac2ce1d4a0a5f3dbc1bd9a9" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/a/ca8338c3f8e8878feac2ce1d4a0a5f3dbc1bd9a9.png" alt="image" data-base62-sha1="sTvABa2FYEERjvXA7ylZCOKnwy5" width="621" height="500" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">636×512 30.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The idea is adding weight to the Cross Entropy loss based on the matrix.<br>
Will it be friendly to the AD engine?</p>

MarkW: <p><a class="mention" href="/u/royi">@Royi</a> , I think you can even raise the numerator to the power to imitate different norm induced scaling.</p>