About weighted BCELoss

Jona: <p>Hi</p>
<p>I’m training a Fully Connected NN with Pytorch, and the model seems to perform very well. this is the model, and the hyper-parameters:</p>
<p><strong>The Model (I’ve not changed the name to <code>FullyConnectedNetworkClassifier</code>)</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fde25bfb56c4f642054f063c5a8b3c763645e7a7.png" data-download-href="https://discuss.pytorch.org/uploads/default/fde25bfb56c4f642054f063c5a8b3c763645e7a7" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fde25bfb56c4f642054f063c5a8b3c763645e7a7.png" alt="image" data-base62-sha1="AdXLTR8GLBaM1AR71RHzQlDmqLZ" width="472" height="500" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">714×756 37.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As you can see, the output layer has a <code>sigmoid</code>, so my model predicts a probability</p>
<p><strong>The hyper-parameters</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be3c936d8d3a007e96cdbbf7dd5d75f8becc6be2.png" alt="image" data-base62-sha1="r8Uva7fMx3PRGjsUYgt9IvcuywG" width="658" height="189"></p>
<p><strong>The results</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9e35f9849d4f1be681dd46349f3d1133ec141c4.png" data-download-href="https://discuss.pytorch.org/uploads/default/d9e35f9849d4f1be681dd46349f3d1133ec141c4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9e35f9849d4f1be681dd46349f3d1133ec141c4_2_690x378.png" alt="image" data-base62-sha1="v5wKTyn5FalPZg1rBpH9oAQdZ88" width="690" height="378" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9e35f9849d4f1be681dd46349f3d1133ec141c4_2_690x378.png, https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9e35f9849d4f1be681dd46349f3d1133ec141c4.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9e35f9849d4f1be681dd46349f3d1133ec141c4.png 2x" data-dominant-color="F3F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">724×397 42.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I made a terrible mistake. I didn’t realize that the both  dataset’s and unbalanced. The column target has two values, <code>True</code> or <code>False</code>, and, as I said, the unbalancing comes in the form that in both dataset’s the samples with <code>target=False</code> are the 90% (10% of samples with <code>target=True</code>) of the total (both in training and validation)</p>
<p>So my questions:</p>
<ol>
<li>My model, as it’s, … is it predicting the probability of the <code>True</code> class (<code>target=True</code>), or the False one (<code>target=False</code>)?</li>
<li>How can I parametrize <code>BCELoss</code> with <code>weights</code>? how much should be the magnitud of those <code>weights</code>? What is the in intuition of the <code>weight</code> parameter, exactly?</li>
<li>Could I use <code>BCEWithLogitsLoss</code> to work with my unbalanced dataset’s? what about its <code>weight</code> and <code>pos_weight</code> parameters?</li>
</ol>
<p>The Pytorch’docs<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html#torch.nn.BCELoss" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html#torch.nn.BCELoss</a><br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss</a></p>
<p>Best regards and thanks</p>

ptrblck: <ol>
<li>
<p>Most likely your model is predicting the majority class only, so I guess it’s predicting the <code>False</code> class. You can check it by printing the unique predictions for the datasets.</p>
</li>
<li>
<p>You could use <code>nn.BCEWithLogitsLoss</code>, remove the <code>sigmoid</code>, and set the <code>pos_weight</code> as <code>number_negative_samples / number_positive_samples</code>. If that doesn’t work, try to use <code>WeightedRandomSampler</code> as described <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2">here</a>.</p>
</li>
<li>
<p>see 2.</p>
</li>
</ol>
<p>Also, note that the accuracy can be misleading for an imbalanced dataset as described in the <a href="https://en.wikipedia.org/wiki/Accuracy_paradox">Accuracy paradox</a>.</p>

Jona: <blockquote>
<p>Most likely your model is predicting the majority class only, so I guess it’s predicting the  <code>False</code> class. You can check it by printing the unique predictions for the datasets.</p>
</blockquote>
<p>You mean, take the predictions and see their probability distributions?</p>
<blockquote>
<p>You could use  <code>nn.BCEWithLogitsLoss</code> , remove the  <code>sigmoid</code> , and set the  <code>pos_weight</code>  as  <code>number_negative_samples / number_positive_samples</code> . If that doesn’t work, try to use  <code>WeightedRandomSampler</code>  as described <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2">here</a>.</p>
</blockquote>
<p>I’ll do it and let you know</p>
<blockquote>
<p>Also, note that the accuracy can be misleading for an imbalanced dataset as described in the <a href="https://en.wikipedia.org/wiki/Accuracy_paradox" rel="noopener nofollow ugc">Accuracy paradox</a>.</p>
</blockquote>
<p>I’ll ckeck out both links</p>
<p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, best regards</p>

ptrblck: <aside class="quote no-group" data-username="Jona" data-post="3" data-topic="95708">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jona/48/27195_2.png" class="avatar"> Jona:</div>
<blockquote>
<p>You mean, take the predictions and see their probability distributions?</p>
</blockquote>
</aside>
<p>Yes, exactly. You could use a threshold of e.g. <code>0</code> to create the predictions and check the predicted class labels (similar to how you would calculate the accuracy). I assume <code>torch.unique</code> would return only the majority class.</p>

Jona: <p>Ok, right … one more thing to test this weekend:</p>
<ol>
<li>Do you think <a class="mention" href="/u/ptrblck">@ptrblck</a> that the use of embeddings and the fact that the model get trained in batches lessens the unbalanced situation? My model, as it’s, in the unseen test_set had a roc_auc of 86,7%, which puts me in third position in an internal competition in my job.</li>
</ol>
<p>Best regards</p>

ptrblck: <p>No, I don’t think using batches would reduce the imbalance.<br>
E.g. if you are passing the complete dataset in a single batch, the imbalance would still be there.<br>
Of course you could use the <code>WeightedRandomSampler</code> to create balanced batches.</p>

Jona: <p>Ok …</p>
<p>In a couple of hours more start my happy coding hours, which extends during the weekend. I’ll be testing all the things you’ve suggested me … so, stay tune pls <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Here in Chile is 20.16 PM … by the way … Where are you writting from?</p>
<p>Best regards <a class="mention" href="/u/ptrblck">@ptrblck</a></p>