Custom loss function for NLP multiclass classification problem

username: Legolas
cooked: <p>I need help/advice/example regarding the approach in the development of PyTorch custom-loss function in NLP multiclass classification.<br>
The dataset looks something like this:<br>
TEXT LABEL<br>
text1 ‘AC’<br>
text2 ‘AD’<br>
text3 ‘BC’<br>
text4 ‘BC’<br>
text5 ‘BD’<br>
…the rest of the dataset…</p>
<p>Labels ‘AB’ or ‘CD’ are impossible from the business perspective and will not appear in the dataset. The first part of the label is always ‘A’ or ‘B’ and the second is always ‘C’ or ‘D’. So, there are only <strong>four possible classes</strong>: ‘AC’, ‘AD’, ‘BC’, BD’.</p>
<p>I <strong>need multiclass classification to four classes</strong>, ‘AC’, ‘AD’, ‘BC’, BD’, but I need a loss function that will <strong>include</strong> and learn/implement the relation between components A-B/C-D. For example, the penalization factor for the wrong prediction of the first label should be <strong>alpha</strong> and the second <strong>beta</strong>. It would be great to learn these parameters <strong>alpha</strong> and <strong>beta</strong>. Of course, if someone has a better suggestion, instead of these parameters, I would follow it.</p>
<p>For example, to simplify, let these labels present labels of sentiment analysis. A component means emotionally, B component means unemotionally, C component means positive, D component means negative (expression in the text).</p>
<p>What would be the best way to do that? Are there steps I should do in feature engineering, or it’s possible to implement just through the loss function (<strong>preferable</strong>)? Some code snippets are more than welcome.</p>
<p>Thank you</p>

username: KFrank
cooked: <p>Hi Ninoslav!</p>
<aside class="quote no-group" data-username="Legolas" data-post="1" data-topic="118168" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/b9bd4f/48.png" class="avatar"> Legolas:</div>
<blockquote>
<p>The first part of the label is always ‘A’ or ‘B’ and the second is always ‘C’ or ‘D’. So, there are only <strong>four possible classes</strong>: ‘AC’, ‘AD’, ‘BC’, BD’.</p>
</blockquote>
</aside>
<p>You should understand your problem to be a <em>multi-label, multi-class</em><br>
problem in the following sense:</p>
<p>It is multi-class because you have have multiple classes – in your case,<br>
two classes, “emotion” and “positivity.”  And is is multi-label because<br>
each sample is given not just one class label (i.e., “emotion” or<br>
“positivity,” but not both) but rather multiple labels, potentially one for<br>
each of your two classes.</p>
<p>That is, each of your samples can be labelled with neither, one, or<br>
both of your two classes.  Said in another (but equivalent) way, each<br>
sample carries two binary labels:  first binary label – “no-emotion”<br>
vs. “yes-emotion”; and second binary label – “no-positivity” vs.<br>
“yes-positivity.”</p>
<p>(It is true that you can structure this as a single-label, four-class problem<br>
with the four labels being your ‘AC’, ‘AD’, ‘BC’, ‘BD’, but it is <em>generally</em><br>
better to structure this kind of problem as a multi-label problem.)</p>
<p>The preferred loss function will be <a href="https://pytorch.org/docs/1.8.1/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">BCEWithLogitsLoss</a>.  Note that this<br>
loss function explicitly supports “the case of multi-label classification.”</p>
<aside class="quote no-group">
<blockquote>
<p>I <strong>need multiclass classification to four classes</strong>, ‘AC’, ‘AD’, ‘BC’, BD’, but I need a loss function that will <strong>include</strong> and learn/implement the relation between components A-B/C-D.</p>
</blockquote>
</aside>
<p>As noted above, you’ll be better of with <em>two</em> classes, but use the<br>
multi-label structure.  Just to be clear, your “A-B” is the first of two<br>
binary labels and “C-D” is the second.</p>
<aside class="quote no-group">
<blockquote>
<p>the penalization factor for the wrong prediction of the first label should be <strong>alpha</strong> and the second <strong>beta</strong>.</p>
</blockquote>
</aside>
<p>Here is one benefit of using <code>BCEWithLogitsLoss</code> with the multi-label<br>
approach.  Use <code>BCEWithLogitsLoss</code>’s <code>weight</code> constructor-argument<br>
with a tensor, <code>[[alpha, beta]]</code> (of shape <code>[1, 2]</code>).  (The leading<br>
singleton dimension of <code>weight</code> will be broadcast along the batch<br>
dimension, so that you’ll be using the same class weights for each<br>
sample in batch.)  This will weight the (binary-cross-entropy) contribution<br>
of your “emotion” class by <code>alpha</code> in your loss function and weight<br>
“positivity” with <code>beta</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>It would be great to learn these parameters <strong>alpha</strong> and <strong>beta</strong>.</p>
</blockquote>
</aside>
<p>I don’t think learning <code>alpha</code> and <code>beta</code> really makes sense.</p>
<p>Maybe you care more about getting “emotion” right than “positivity.”<br>
Or maybe the other way around.  I don’t see how your network can<br>
know this without you telling it.  You <em>tell</em> the network how much you<br>
care about “emotion” vs. “positivity” with your <code>alpha</code> and <code>beta</code><br>
parameters, so the network doesn’t have any way to “learn” this<br>
preference of yours.</p>
<aside class="quote no-group">
<blockquote>
<p>or it’s possible to implement just through the loss function (<strong>preferable</strong>)?</p>
</blockquote>
</aside>
<p>Yes, as outlined above, using just a loss function – specifically the<br>
multi-label case of <code>BCEWithLogitsLoss</code> – it is possible – and likely<br>
the best way – to implement your classifier.</p>
<p>(Just to be sure, I used your two-class example – “emotion” and<br>
“positivity” – for simplicity and to follow along with your post.  But<br>
there is nothing about this scheme that is restricted to two classes.<br>
You could, for example, label your natural-language phrases with five<br>
features, say “emotion,” “positivity,” “certainty,” “conciseness,” and<br>
“understandability,” and use <code>BCEWithLogitsLoss</code> to build a multi-label,<br>
five-class classifier.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: Legolas
cooked: <p>Hi Frank,</p>
<p>Thank you for such a complete reply and feedback. I appreciate it.</p>
<p>I’m aware that we can approach the problem as a multi-label, multi-class. I am trying to treat it as a multi-class problem (for the initial set of four classes, ‘AC’, ‘AD’, ‘BC’, ‘BD’) but would like to include ‘impact’ of class components. Do you have any ideas in that direction?</p>

username: KFrank
cooked: <p>Hi Ninoslav!</p>
<aside class="quote no-group" data-username="Legolas" data-post="3" data-topic="118168" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/b9bd4f/48.png" class="avatar"> Legolas:</div>
<blockquote>
<p>I am trying to treat it as a multi-class problem (for the initial set of four classes, ‘AC’, ‘AD’, ‘BC’, ‘BD’) but would like to include ‘impact’ of class components.</p>
</blockquote>
</aside>
<p>I have a few comments:</p>
<p>First, to reiterate, since you want to “look inside” you classes, drilling<br>
down into what you call the “class components,” it seems unnatural to<br>
me to treat this as a single-label, four-class problem rather than a<br>
multi-label, two-class problem.  By using your four classes, you’ve<br>
“hidden” their components that you want access to, just making things<br>
harder.</p>
<p>Now some thoughts about your approach:</p>
<p>The most common approach for a (single-label) four-class problem<br>
would be to have the final <code>Linear</code> layer of your model output four<br>
raw-score logits that you then pass into <code>CrossEntropyLoss</code>.  (The<br>
ground-truth <code>target</code> passed into <code>CrossEntropyLoss</code> will be a single<br>
integer class label (for each sample in the batch) that takes on the<br>
values <code>{0, 1, 2, 3}</code>.)</p>
<p>You can convert the logits into probabilities for your four classes by<br>
passing them through <code>softmax()</code>.  (<code>CrossEntropyLoss</code> has, in effect,<br>
<code>softmax()</code> built in.)</p>
<p>Note, the predictions coming out of your model are logits (that can be<br>
converted to probabilities) specifically because your <em>trained</em> them to be<br>
by using <code>CrossEntropyLoss</code> as the loss criterion you backpropagate.<br>
Were you to use some other loss criterion for training the meaning of<br>
your model outputs would likely change.</p>
<p>Nonetheless, let’s work under the assumption that your model outputs<br>
remain logits even as we modify the loss criterion.</p>
<p>So:  Convert your model outputs to probabilities, and compute the<br>
probabilities for your class “components”:</p>
<pre><code class="lang-plaintext">P(A) = P(AC) + P(AD)
P(B) = P(BC) + P(BD)
P(C) = P(AC) + P(BC)
P(D) = P(AD) + P(BD)
</code></pre>
<p>Note that <code>P(A) + P(B) = 1</code> (which is consistent because the<br>
first component is either in state <code>A</code> or state <code>B</code>), and, similarly,<br>
<code>P(C) + P(D) = 1</code>.</p>
<p>Now consider your ground-truth <code>target</code>:  <code>target = 1</code>, for example,<br>
means class <code>AD</code>, so 100% probability of the first component being<br>
in state <code>A</code> (so 0% probability in state <code>B</code>) and 0% probability of the<br>
second component being in state <code>C</code> (so 100% probability in state <code>D</code>).</p>
<p>You now have the both predicted and ground-truth probabilities of your<br>
first component being in state <code>A</code> so you can use the mismatch between<br>
these two probabilities (for example, the cross entropy) as your loss<br>
criterion for the first component of your class, and weight it with <code>alpha</code>.</p>
<p>Similarly, you could use the mismatch between the predicted and<br>
ground-truth probabilities for your second component being in state<br>
<code>C</code> as your second-component loss criterion, and weight it with <code>beta</code>.</p>
<p>You can now train your model with this combined <code>alpha</code> and <code>beta</code><br>
loss criterion.  Problem solved!</p>
<p>However …</p>
<p>What you’ve <em>actually</em> done here is built a (weighted) multi-label<br>
binary-cross-entropy loss criterion for the case where your multi-label<br>
predictions are encoded in an unusual way.</p>
<p>How can the four outputs of the final <code>Linear</code> layer of your model turn<br>
into multi-label predictions (with an unusual encoding) instead of being<br>
the four conventional single-label predictions for your four classes?</p>
<p>Because that’s what you’ve <em>trained</em> them to be.  (The first rule of neural<br>
networks is that the output of your model means what you train it to<br>
mean.)</p>
<p>There’s nothing wrong nor inconsistent about doing things this way<br>
(although it does seem like you would be standing on your head), but<br>
the <em>meaning</em> of your model’s predictions is no longer the same as<br>
if you had trained it as a conventional single-label, four-class model.<br>
Instead, your model is making multi-label, two-class predictions, albeit<br>
encoded in an unusual way.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Legolas
cooked: <p>Hi Frank,<br>
Thank you for your replies and suggestions. Regarding the following part:</p>
<blockquote>
<p>There’s nothing wrong nor inconsistent about doing things this way<br>
(although it does seem like you would be standing on your head), but<br>
the meaning of your model’s predictions is no longer the same as<br>
if you had trained it as a conventional single-label, four-class model.<br>
Instead, your model is making multi-label, two-class predictions, albeit<br>
encoded in an unusual way.</p>
</blockquote>
<p>Isn’t it true when I consider both ‘components’, I should push the model to train for predicting ‘both components’, in other words, a four-class model – this way through ‘components’?</p>
<p>Thank you</p>

username: KFrank
cooked: <p>Hi Ninoslav!</p>
<aside class="quote no-group" data-username="Legolas" data-post="5" data-topic="118168" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/b9bd4f/48.png" class="avatar"> Legolas:</div>
<blockquote>
<p>Isn’t it true when I consider both ‘components’, I should push the model to train for predicting ‘both components’, in other words, a four-class model – this way through ‘components’?</p>
</blockquote>
</aside>
<p>Yes,  you can look at it this way, and you can do it this way.</p>
<p>The details of training will be different, but there is an equivalence<br>
between a multi-label, two-class problem and a single-label,<br>
four-class problem.</p>
<p>Let’s say that in the first instance your classes are A and B, and in<br>
the second, 0, 1, 2, 3.</p>
<p>Then:</p>
<pre><code class="lang-plaintext">   A--no, B--no  --&gt;  0
   A-yes, B--no  --&gt;  1
   A--no, B-yes  --&gt;  2
   A-yes, B-yes  --&gt;  3
</code></pre>
<p>gives an explicit mapping between the two problems.</p>
<p>My intuition tells be you will get better training and better results treating<br>
this as a multi-label, two-class problem.  But you should also be able to<br>
get things to work treating this as a single-label, four-class problem.</p>
<p>Your call.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Legolas
cooked: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a>,</p>
<p>Thank you.<br>
I didn’t quite understand your explanation in this part:</p>
<p>Let’s say that in the first instance, your classes are A and B, and in<br>
the second, 0, 1, 2, 3.<br>
Then:<br>
A–no, B–no → 0<br>
A-yes, B–no → 1<br>
A–no, B-yes → 2<br>
A-yes, B-yes → 3</p>
<p>Could you be so kind as to explain it a little bit further?<br>
Also, do you have any suggestion regarding constructing the custom loss in this case? We should convert new calculated probabilities into logits again using F.sigmoid; am I right?<br>
Thank you</p>