BCEWithLogitsLoss pos_weights -- how to do re-weighting on inference

layman: <p>I am training a multilabel classifier on some imbalanced dataset where I am using pos weights on loss.</p>
<p>Dataset looks like</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>label</th>
<th>cat1</th>
<th>cat2</th>
<th>cat3</th>
</tr>
</thead>
<tbody>
<tr>
<td>a</td>
<td>100</td>
<td>100</td>
<td>100</td>
</tr>
<tr>
<td>b</td>
<td>100</td>
<td>200</td>
<td>400</td>
</tr>
<tr>
<td>c</td>
<td>100</td>
<td>400</td>
<td>800</td>
</tr>
<tr>
<td>d</td>
<td>100</td>
<td>800</td>
<td>1600</td>
</tr>
</tbody>
</table>
</div><p>And my corresponding loss</p>
<pre><code class="lang-auto"># One-hot encoded trinary labels hence 12-dim output for 4 labels
nn.BCEWithLogitsLoss(pos_weight=torch.tensor([1, 1, 1,
                                              4, 2, 1, 
                                              8, 2, 1, 
                                              16, 2, 1])
</code></pre>
<p>My question is, what is the mathematically correct way to correct for the expected data distribution for actual inference? I don’t think dividing the logits directly by weight is correct.</p>
<pre><code class="lang-auto">output = model(data.to(device))
output = output.detach().cpu() / np.array([1, 1, 1,
                                           4, 2, 1, 
                                           8, 2, 1, 
                                           16, 2, 1])
output = output.reshape((-1, 4, 3))
output = softmax(output)
</code></pre>

KFrank: <p>Hi Victor!</p>
<aside class="quote no-group" data-username="layman" data-post="1" data-topic="206415" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/layman/48/65651_2.png" class="avatar"> layman:</div>
<blockquote>
<p>I am training a multilabel classifier on some imbalanced dataset where I am using pos weights on loss.</p>
</blockquote>
</aside>
<p>Think through with care what you are trying to do here – things look a<br>
little bit garbled.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto"># One-hot encoded binary labels hence 8-dim output for 4 labels
nn.BCEWithLogitsLoss(pos_weight=torch.tensor([1, 1, 2, 1, 4, 1, 8, 1])
</code></pre>
</blockquote>
</aside>
<p>Most likely you don’t want one-hot encoding nor an eight-dimensional<br>
output.</p>
<p>A *multi-label" classifier consists of a set of binary classifiers, one for each<br>
of your four classes.  Thus “a – yes, no,” “b – yes, no,” etc.</p>
<p>When using pytorch and <code>BCEWithLogitsLoss</code>, you would want to have<br>
four (floating-point) labels for each sample – <code>0.0</code> vs. <code>1.0</code> for each of your<br>
four classes.  So <code>labels</code> should be a <code>FloatTensor</code> of shape <code>[nBatch, 4]</code>.</p>
<p>Your <code>output</code> would have the same type and shape and would be the<br>
predicted logits for each of your four classes.</p>
<aside class="quote no-group">
<blockquote>
<p>what is the mathematically correct way to correct for the expected data distribution for actual inference?</p>
</blockquote>
</aside>
<p>To me, “inference” means you take a sample and make a prediction.  For<br>
just this operation, you wouldn’t perform any re-weighting.  Any re-weighting<br>
would have occurred during training, with the goal of training your model to<br>
make better predictions.</p>
<p>(You may or may not wish to use re-weighting when computing performance<br>
metrics for your validation or test datasets.  If you compute a loss function<br>
for your validation dataset, you generally want it to be the same loss function<br>
as the one you compute for your training set so that the two will be directly<br>
comparable.  You would then typically use the same re-weighting for your<br>
validation-dataset loss function.)</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">output = softmax(output.detach().cpu() / np.array([1, 1, 2, 1, 4, 1, 8, 1]))
</code></pre>
</blockquote>
</aside>
<p>When performing “pure” binary classification, you do <em>not</em> ever want to<br>
use <code>softmax()</code>.  <code>softmax()</code> might be used for a multi-class, <em>single-label</em><br>
problem.</p>
<p>One possible point of confusion:  It is possible to recast a single-label binary<br>
problem as a two-class (single-label) problem, and train it as a multi-class<br>
(single-label) problem (that happens two have two classes) using<br>
<code>CrossEntropyLoss</code>.  (Your two-class labels would then be the one-hot<br>
encoded version of your original binary labels.)</p>
<p>It looks as though this approach might be being mixed into what you are<br>
doing.  While treating a binary problem as a two-class, multi-class problem<br>
is fully legitimate, it doesn’t play nicely with a multi-label problem (even<br>
though the multi-label problem can be looked at as being a set of binary<br>
problems).</p>
<p>Best.</p>
<p>K. Frank</p>

layman: <p>Hi K, thanks for the detailed response! One thing though – I actually posted it this way with binary labels to simplify things. In my actual problem, the labels are ordinal categories, hence how I ended up with the multi-hot encoded vector. What would you suggest in this case of multilabel+multiclass?</p>

KFrank: <p>Hi Victor!</p>
<aside class="quote no-group" data-username="layman" data-post="3" data-topic="206415" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/layman/48/65651_2.png" class="avatar"> layman:</div>
<blockquote>
<p>this case of multilabel+multiclass?</p>
</blockquote>
</aside>
<p>You will have to explain what you mean by “multilabel+multiclass” and<br>
illustrate it with a concrete (if contrived) example.</p>
<p>Best.</p>
<p>K. Frank</p>

layman: <p>Just updated the initial post as a multilabel trinary classification example.</p>
<p>For something more tangible, imagine something like a video classifier where the data labels are satisfaction surveys with 1-5 stars across multiple categories. Also imagine the distribution is skewed towards 5 stars by varying degrees, sometimes by multiple orders of magnitude.</p>
<p>My thinking here is, loss weighting should serve to amplify the gradients so that the model can learn something beyond just biases for the final linear layer. But I don’t really want to trade precision for recall in my use case either, so I want to recalibrate for when I am doing inference.</p>

KFrank: <p>Hi Victor!</p>
<aside class="quote no-group" data-username="layman" data-post="5" data-topic="206415" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/layman/48/65651_2.png" class="avatar"> layman:</div>
<blockquote>
<p>Just updated the initial post as a multilabel trinary classification example.</p>
</blockquote>
</aside>
<p>Okay, I understand what you’re trying to do now.</p>
<p>First off:</p>
<aside class="quote no-group" data-username="layman" data-post="1" data-topic="206415">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/layman/48/65651_2.png" class="avatar"> layman:</div>
<blockquote>
<p>And my corresponding loss</p>
<pre><code class="lang-auto"># One-hot encoded trinary labels hence 12-dim output for 4 labels
nn.BCEWithLogitsLoss(pos_weight=torch.tensor([1, 1, 1,
                                              4, 2, 1, 
                                              8, 2, 1, 
                                              16, 2, 1])
</code></pre>
</blockquote>
</aside>
<p>This isn’t right.  You are working with a not-binary, multi-class problem<br>
(which in your example, has three classes).  You <em>do not</em> want to be using<br>
<code>BCEWithLogitsLoss</code> nor any kind of one-hot encoding (even though<br>
your problem is “multi-label” in the sense that you have four three-class<br>
classifiers that share some upstream processing).</p>
<p>Now to your problem:</p>
<p>Let’s say you are rating videos across four categories (your four “labels”),<br>
say, “plot,” “dialog,” “acting,” and “makeup,” and each rating category has<br>
three classes, “one star,” two stars," and “three stars.”</p>
<p>You should treat each of these categories as a multi-class classification<br>
problem and each should have its own instance of a <code>CrossEntropyLoss</code><br>
<code>loss_fn</code>, <em>each instantiated with its own <code>weight</code> argument</em> that accounts<br>
for the class imbalance across that specific category’s number-of-stars<br>
classes.  (<code>weight</code> is <code>CrossEntropyLoss</code>’s analog of <code>BCEWithLogitsLoss</code>’s<br>
<code>pos_weight</code> constructor argument.)</p>
<p>Something like this:</p>
<p>Your model will have a shared “backbone” that all four categories use and<br>
four classifier “heads,” one for each category.  Let’s say that the last layer<br>
of your backbone is a <code>Linear</code> with <code>out_features = 100</code>.  Then:</p>
<pre data-code-wrap="python"><code class="lang-python"># your earlier backbone layers
self.last_backbone_layer = torch.nn.Linear (500, 100)
self.headA = torch.nn.Linear (100, 3)
self.headB = torch.nn.Linear (100, 3)
self.headC = torch.nn.Linear (100, 3)
self.headD = torch.nn.Linear (100, 3)
</code></pre>
<p>And <code>forward()</code> would look something like this:</p>
<pre data-code-wrap="python"><code class="lang-python">x = self.last_backbone_layer (x)
return  self.headA (x), self.headB (x), self.headC (x), self.headD (x)
</code></pre>
<p>Then</p>
<pre data-code-wrap="python"><code class="lang-python">outputA, outputB, outputC, outputD = model (input)
lossA = loss_fnA (outputA, labelsA)
lossB = loss_fnB (outputB, labelsB)
lossC = loss_fnC (outputC, labelsC)
lossD = loss_fnD (outputD, labelsD)
loss_total = lossA + lossB + lossC + lossD
</code></pre>
<p>Each of the “labels” tensors, e.g., <code>labelsA</code>, will be a <code>LongTensor</code> with<br>
shape <code>[nBatch]</code> (with no class dimension) and consist of integer categorical<br>
class labels whose values run over the values <code>0</code>, <code>1</code>, and <code>2</code>.</p>
<p>Make sure that you understand how to build and train a “conventional” (that<br>
is, not “multi-label”) multi-class classifier using <code>CrossEntropyLoss</code>, using<br>
its <code>weight</code> constructor argument to compensate for class imbalance.  This<br>
is really just the same except that you have four such classifiers that share<br>
a common backbone.</p>
<p>Best.</p>
<p>K. Frank</p>