Calculating loss for mapped logits

vcog: <p>I’d like to add a second loss term for a mapped class category (will explain) and I’m having trouble finding the appropriate way to do so.</p>
<p>My model produces logits for a multi-class classification problem and no issues using <code>binary_cross_entropy_with_logits</code> as normal.  I’d like to add a second loss term based upon another level in the class hierarchy.  So for each c_i, there’s a c_i → p_i class mapping, and similarly for the ground truth gt_c_i → gt_p_i (where the P != C naturally).  So functionally, I would like to map the logits to a distribution over C, and then call <code>cross_entropy</code> on that to produce the loss to sum.  How would I do that?  I’ve tried quite a few variations and I feel like I’m missing something obvious.</p>
<p>Thank you in advance!</p>

KFrank: <p>Hi George!</p>
<aside class="quote no-group" data-username="vcog" data-post="1" data-topic="189844" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/85e7bf/48.png" class="avatar"> vcog:</div>
<blockquote>
<p>My model produces logits for a multi-class classification problem and no issues using <code>binary_cross_entropy_with_logits</code> as normal.</p>
</blockquote>
</aside>
<p>Could you clarify what this first level of your use case is?</p>
<p>To me, “multi-class classification” means that each data sample belongs to<br>
exactly one of some number of classes.  That is, for example, a given image<br>
is an image of exactly one animal, say, “bat,” “bear,” or “dog” (but is not in more<br>
than one class at the same time).</p>
<p>If this is the case, your model should predict (for this three-class example)<br>
three unnormalized log-probabilities (These log-probabilities are technically<br>
not logits.) and you would use <code>CrossEntropyLoss</code> as your loss criterion.</p>
<p>Does this correctly describe your use case, or are you doing something<br>
different?</p>
<aside class="quote no-group">
<blockquote>
<p>I’d like to add a second loss term based upon another level in the class hierarchy.  So for each c_i, there’s a c_i → p_i class mapping</p>
</blockquote>
</aside>
<p>So, to extend my earlier example, you might also have images of “crow,”<br>
“pigeon,” and “robin.”  You would then like to group your six classes into<br>
two superclasses:</p>
<p>mammals:  bat, bear, dog<br>
birds:  crow, pigeon, robin</p>
<p>Is this what you mean by “mapped class category?”</p>
<aside class="quote no-group">
<blockquote>
<p>I would like to map the logits to a distribution over C, and then call <code>cross_entropy</code> on that to produce the loss to sum.</p>
</blockquote>
</aside>
<p>Given that logits (appropriate for <code>BCEWithLogitsLoss</code>) are not the same as<br>
unnormalized log-probabilities (appropriate for <code>CrossEntropyLoss</code>), there is<br>
not really a natural mapping from class-logits to superclass-log-probabilities.</p>
<p>First, could you clarify whether you actually mean logits here?</p>
<p>Second, what is your goal with adding a cross-entropy loss at your superclass<br>
level?</p>
<p>Third, is it your intention to use your superclass loss function *instead" of a<br>
subclass loss function, or do want to combine the to together into a single<br>
loss function?</p>
<p>Best.</p>
<p>K. Frank</p>

vcog: <p>The thorough response is greatly appreciated <a class="mention" href="/u/kfrank">@KFrank</a>!</p>
<p>Yes, you understand precisely and agreed on both the multi-class nature of the problem and the characterization of the superclasses as what I intended for “mapped class category”.</p>
<p>I was being a bit loose with my terminology, but I am using <code>F.binary_cross_entropy_with_logits</code> on the logits directly (without issue) currently.</p>
<p>The goal of adding the secondary loss at the superclass level is to penalize errors at that level, and the formulation will be somewhat different from straight BCE, but for these purposes let’s presume that’s the case and the intent is just to sum the loss from both levels to arrive at the loss to backprop.</p>
<p>Which is the crux of the question (I poorly communicated, so thanks for your patience!) - given that I know the appropriate weighting from the class to the secondary class and am comfortable generating a loss per class for the secondary class, how would I practically go about calculating and applying it?  And more specifically, what’s the right way to compute it to that I can pass back the gradient?</p>
<p>I was setting up a sparse NxM matrix W, where there are N elements in C and M elements in P and W_i_j is only non-zero if c_i is the child of p_j.  Then I should be able to take the logits (or perhaps more naturally as probabilities with a sigmoid), mult by W, and then run BCEWithLogitsLoss (or CrossEntropyLoss) vs the ground truth P. If those are tensors then, can I simply add that to the loss and be able to call backward there?  (It didn’t seem so, but I might have another issue.  I really should inspect the gradients more carefully here for a toy example.)</p>
<p>(Alternatively, I thought about having another layer with fixed weights and attempting to backprop through that, but that seemed problematic as well and I was likely missing something more fundamental.)</p>
<p>Appreciate any insight.  In the meantime, I’ll take another pass at this and post some code here shortly if that’s not clear.</p>
<p>Thanks again!</p>

KFrank: <p>Hi George!</p>
<aside class="quote no-group" data-username="vcog" data-post="3" data-topic="189844" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/85e7bf/48.png" class="avatar"> vcog:</div>
<blockquote>
<p>you understand precisely and agreed on both the multi-class nature of the problem</p>
</blockquote>
</aside>
<p>You keep talking as if this is a (single-label) multi-class classification problem<br>
(rather than a <em>multi-label,</em> multi-class problem).</p>
<aside class="quote no-group">
<blockquote>
<p>I am using <code>F.binary_cross_entropy_with_logits</code> on the logits directly</p>
</blockquote>
</aside>
<p>But you state that you are using <code>binary_cross_entropy_with_logits()</code>,<br>
which is not appropriate for the single-label case (but is the correct choice<br>
for a multi-label problem).</p>
<p>(Note, I explained what a (single-label) “multi-class classification” problem<br>
is in my first post.)</p>
<p>Before we can usefully address how (or whether) to add a superclass loss<br>
term to your overall loss, you will have to clarify whether you a performing<br>
a single-label or multi-label classification.</p>
<p>Best.</p>
<p>K. Frank</p>

vcog: <p>Yes, I am treating it as a multi-label, multi-class problem.  (There is in practice only one class typically, but since there are cases where multiple classes can occur in some datasets, I structure it as multi-label.  But if the single label case is much clearer here we could certainly proceed as if that is the case and I can adapt from there.)</p>
<p>Thanks again for your precision!</p>

KFrank: <p>Hi George!</p>
<aside class="quote no-group" data-username="vcog" data-post="5" data-topic="189844" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/85e7bf/48.png" class="avatar"> vcog:</div>
<blockquote>
<p>if the single label case is much clearer here we could certainly proceed as if that is the case</p>
</blockquote>
</aside>
<p>Regardless of whether yours is a multi-label problem or not, probably your<br>
best approach is to compute your loss just at the subclass level and not add<br>
a superclass term to it.  If you train to predict the correct subclasses (which<br>
is what you should be trying to do), you will also automatically be training to<br>
predict the correct superclasses.  You haven’t offered any explanation of <em>why</em><br>
you need or want to add a superclass term.</p>
<p>(You’ve told us more-or-less nothing about your concrete use case,* so its<br>
hard to offer useful advice as to what approach might be best.)</p>
<p>Having said that, let me answer your technical question in the single-label<br>
context.</p>
<p>Your network will predict unnormalized log-probabilities for your subclasses.<br>
These will typically be the output of your final <code>Linear</code> layer (without any<br>
subsequent non-linear activations).  You would normally pass these directly<br>
into <code>CrossEntropyLoss</code>.</p>
<p>Conceptually, you want to combine your subclass probabilities into superclass<br>
probabilities that you would then pass into a superclass cross-entropy loss<br>
function.</p>
<p>For numerical reasons (essentially the same reasons that <code>CrossEntropyLoss</code><br>
takes log-probabilities rather than plain probabilities), you should do all of this<br>
in log-space – that is, always work with log-probabilities without ever explicitly<br>
converting them to probabilities – combining your subclass log-probability<br>
predictions into superclass log-probabilities (that you would then pass into a<br>
superclass-level <code>CrossEntropyLoss</code>).</p>
<p>First pass the unnormalized subclass log-probabilities through <code>log_softmax()</code><br>
to convert them to normalized log-probabilities.</p>
<p>At this point, conceptually, you would use <code>exp()</code> to obtain subclass probabilities,<br>
you would sum the subclass probabilities within a given superclass to obtain the<br>
probability for that superclass, and then call <code>log()</code> to obtain that superclass’s<br>
log-probability.</p>
<p>But we wish to perform this manipulation is log-space, so, instead, we will select<br>
the subclass log-probabilities for the subclasses of a given superclass and “add”<br>
them together with <code>logsumexp()</code> to obtain that superclass’s log-probability<br>
“directly.”</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.1.0
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2023)
&gt;&gt;&gt;
&gt;&gt;&gt; # example with six subclasses and two superclasses
&gt;&gt;&gt;
&gt;&gt;&gt; #  subclass       superclass
&gt;&gt;&gt;
&gt;&gt;&gt; #  0: bat         1: mammal
&gt;&gt;&gt; #  1: bear        1: mammal
&gt;&gt;&gt; #  2: crow        0: bird
&gt;&gt;&gt; #  3: dog         1: mammal
&gt;&gt;&gt; #  4: pigeon      0: bird
&gt;&gt;&gt; #  5: robin       0: bird
&gt;&gt;&gt;
&gt;&gt;&gt; class_map = torch.tensor ([                       # membership of subclass (column) in superclass (row)
...     [False, False,  True, False,  True,  True],
...     [ True,  True, False,  True, False, False]
... ])
&gt;&gt;&gt;
&gt;&gt;&gt; subu = torch.randn (6)                            # unnormalized subclass log-probabilities
&gt;&gt;&gt; subn = subu.log_softmax (dim = 0)                 # normalized subclass log-probabilities
&gt;&gt;&gt;
&gt;&gt;&gt; subn
tensor([-3.0564, -1.2996, -2.2345, -1.1579, -2.5913, -1.6918])
&gt;&gt;&gt; subn.exp()
tensor([0.0471, 0.2727, 0.1070, 0.3141, 0.0749, 0.1842])
&gt;&gt;&gt;
&gt;&gt;&gt; supern = torch.empty (class_map.size (0))         # storage for superclass (normalized) log-probabilities
&gt;&gt;&gt; for  i in range (class_map.size (0)):             # compute log-probability for each superclass
...     supern[i] = subn[class_map[i]].logsumexp (dim = 0)
...
&gt;&gt;&gt; supern
tensor([-1.0047, -0.4559])
&gt;&gt;&gt; supern.exp()
tensor([0.3661, 0.6339])
</code></pre>
<p>*)  For example, what is the input data to your model?  Images?  Time series?<br>
Sets of disparate descriptive values?  And what does that data <em>mean?</em>  What<br>
are your classes?  Is your training data balanced or unbalanced?  How much<br>
training data do you have?  What are your most important performance metrics?</p>
<p>Best.</p>
<p>K. Frank</p>