BCEWithLogitLoss for multiclass, class imbalance

Jeguarko: <p>Help to understand how BCEWithLogitLoss works for a multiclass case with class imbalance (object detection, Yolov5 (<a href="https://github.com/ultralytics/yolov5/blob/master/utils/loss.py" class="inline-onebox" rel="noopener nofollow ugc">yolov5/loss.py at master · ultralytics/yolov5 · GitHub</a>)).</p>
<p>I have a dataset of 6595 images. Each image can have up to 5 different object classes. Suppose these are people, cars, billboards, trees, bicycles. Number of objects by class on the entire dataset:<br>
people: 614,<br>
cars: 947,<br>
billboards: 1628,<br>
trees: 2743,<br>
bicycles: 663</p>
<ol>
<li>I need to use the pos_weight argument and pass the weights tensor to it [9.74, 5.96, 3.05, 1.4, 8.94], using this recommendation?<br>
“For example, if a dataset contains 100 positive and 300 negative examples of a single class, then pos_weight for the class should be equal to 300/100=3. The loss would act as if the dataset contains 3×100=300 positive examples.”</li>
<li>Anyway, what happens when filling pos_weight for Multiclass?</li>
<li>what is the “weight” argument for and what happens when it is not empty?</li>
</ol>

KFrank: <p>Hi Egor!</p>
<aside class="quote no-group quote-modified" data-username="Jeguarko" data-post="1" data-topic="151892" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jeguarko/48/49113_2.png" class="avatar"> Jeguarko:</div>
<blockquote>
<p>I have a dataset of 6595 images, which consists of 5 classes:<br>
0: 614,<br>
1: 947,<br>
2: 1628,<br>
3: 2743,<br>
4: 663</p>
</blockquote>
</aside>
<p>This suggests to me that you are performing a <em>single-label,</em> multi-class<br>
classification.</p>
<p>That is, each image is either in class 0, or in class 1, or in class 2, etc.,<br>
but not in more than one class (and not in no class).</p>
<p>For single-label, multi-class problems, you would typically use<br>
<code>CrossEntropyLoss</code> (not <code>BCEWithLogitsLoss</code>).</p>
<aside class="quote no-group">
<blockquote>
<ol>
<li>I need to use the pos_weight argument</li>
</ol>
</blockquote>
</aside>
<p>For <code>CrossEntropyLoss</code>, you would use its <code>weight</code> constructor argument.<br>
(It plays a similar role <code>BCEWithLogitLoss</code>’s <code>pos_weight</code> argument.)</p>
<aside class="quote no-group">
<blockquote>
<p>and pass the weights tensor to it [9.74, 5.96, 3.05, 1.4, 8.94]</p>
</blockquote>
</aside>
<p>One would typically use the reciprocal of a class’s frequency for its<br>
weight, thus:</p>
<pre data-code-wrap="python"><code class="lang-python">weight = torch.tensor ([6595 / 614, 6595 / 947, ...])
</code></pre>
<p>Note, the exact values used for <code>weight</code> don’t matter – they’re just rough<br>
fudge factors to account for the class imbalance.</p>
<p>Also, in your case, your biggest class imbalance is about 4.5-to-1, which<br>
isn’t too bad, so you probably don’t need to use <code>weight</code>, although doing<br>
so won’t hurt, and could help at the margins.</p>
<p>Best.</p>
<p>K. Frank</p>

Jeguarko: <p>Thank you for the answer and sorry for the incorrect description of the problem. I have 5 classes of different objects, each of which can be in the image.</p>

KFrank: <p>Hi Egor!</p>
<aside class="quote no-group" data-username="Jeguarko" data-post="3" data-topic="151892" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jeguarko/48/49113_2.png" class="avatar"> Jeguarko:</div>
<blockquote>
<p>I have 5 classes of different objects, each of which can be in the image.</p>
</blockquote>
</aside>
<p>Yes, then this is a multi-label, multi-class problem for which<br>
<code>BCEWithLogitsLoss</code> would typically be the best choice.</p>
<p>Your proposed values for <code>pos_weight</code> look appropriate, given the<br>
class frequencies you posted.</p>
<p>When <code>pos_weight</code> is not <code>None</code>, it gets applied along the class dimension<br>
of your <code>input</code> (predictions) and <code>target</code> (labels) as shown in the formula<br>
in the <code>BCEWithLogitsLoss</code> documentation.  That is, the element of<br>
<code>pos_weight</code> that corresponds to the class in question multiplies the<br>
“predicted-positive” term in the per-sample, per-sample term cross-entropy<br>
expression.</p>
<p>The <code>weight</code> argument, if present, simply multiplies the loss contribution<br>
of each sample in the batch.</p>
<p>(More precisely, <code>input</code>, <code>target</code>, and <code>weight</code> are all mutually broadcast<br>
with respect to one another, and then combined together element-wise.<br>
This means that, for example, in a semantic-segmentation use case, you<br>
could use the <code>weight</code> argument to give different weights to each pixel in<br>
your image, not just to different samples in your batch.)</p>
<p>Best.</p>
<p>K. Frank</p>

Jeguarko: <p>Thank you for the detailed answer!</p>