Pos_weight attribute in BCEWithLogitLoss

username: chinmay5
cooked: <p>I am attempting to work on the following loss function. To my understanding, this is essentially a BCE loss function where we need to work with the weight parameter. I initially started with</p>
<ol>
<li>
<p>Counting the number of positive examples and then <code>weight =  total_samples / number_positive_per_class</code>. However, clearly this is not what the paper suggests.</p>
</li>
<li>
<p>I read through the documents and it seems I need to set the <code>pos_weight</code> parameter. However, there is nothing to set the <code>neg_weights</code> as might be needed in this case. So, can I use the ratio of <code>pos_weight = neg_samples/ pos_samples</code></p>
</li>
<li>
<p>If this should be the way, can I use the direct ratio or I need to do something else in order to ensure the loss values are correctly weighted?</p>
</li>
</ol>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30bdd1cf8740f6c711b8019bbaef66cf0f23f33f.png" alt="image" data-base62-sha1="6XbBwa0hZX8QgFuXvLaXFi1kGrt" width="677" height="346"></p>
<p>Any help would be highly appreciated. Thank you</p>

username: ptrblck
cooked: <p>I think you could transform the posted formula into the <code>pos_weight</code> formula used in <code>nn.BCEWithLogitsLoss</code> as given in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss">docs</a>.</p>
<p>As described, <code>pos_weight</code> is specified as <code>nb_neg/nb_pos</code> and is multiplied with the “positive” part of the loss (left summand). If you divide both summands by <code>lambda_0</code> in your formula, you would end up with the same loss formulation.</p>

username: chinmay5
cooked: <p>Thank you so much for the reply. Here is a followup though. I see there is also an implementation for the function <code>MultiLabelSoftMarginLoss</code>. I think in spirit it is similar to the <code>BCEWithLogitLoss</code> and since I have a multi label classification, I thought it would be better to use the <code>MultiLabelSoftMarginLoss</code> function. However, there is no <code>pos_weight</code> parameter in it. Is there a way I can use it or should I stick to <code>BCEWithLogitLoss</code>.</p>
<p>Also, just to be sure, should I use <code>pos_weight = neg_samples_of_each_class/ pos_samples_of_each_class</code> in the loss formulation?</p>

username: ptrblck
cooked: <p>I don’t know how <code>MultiLabelSoftMarginLoss</code> would compare to <code>BCEWithLogitsLoss</code>, so let’s wait for some experts to chime in.</p>
<p>In case you are dealing with a multi-label classification, <code>pos_weight</code> should get a value for each class, i.e. it should be a tensor containing <code>nb_classes</code> values defined as <code>[nb_neg_class0/nb_pos_class0, nb_neg_class1/nb_pos_class1, ...]</code>.</p>

username: chinmay5
cooked: <p>Thank you for the quick response. Just to confirm, even if I use only the <code>BCEWithLogitLoss</code>, it should be fine in the Multi-label scenario. I just want to make sure that the loss I am computing in this way, is not incorrect.</p>

username: ptrblck
cooked: <p>Yes, <code>nn.BCEWithLogitsLoss</code> can be used for binary and multi-label classification use cases.</p>