BCEwithlogitsloss and loss functions in general

Leo2: <p>Hi there<br>
I’m working on my first deep learning project. When you define BCEwithlogitloss, you have the option to include pos_weights which defines how much importance should be placed on positive labels in a multilabel classification with lots of negative ones.</p>
<p>I’m using mini batch and I want to calculate pos_weight by following the suggestion of pos_weight = total_neg/total_pos.</p>
<p>Does that value need to be across all in the dataset or within each mini batch. Does it matter if i keep redefining the loss function for each mini batch iteration, Is any information lost by doing that?</p>
<p>thanks</p>

RaLo4: <aside class="quote no-group" data-username="Leo2" data-post="1" data-topic="132908">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leo2/48/37227_2.png" class="avatar"> Leo2:</div>
<blockquote>
<p>Does that value need to be across all in the dataset or within each mini batch.</p>
</blockquote>
</aside>
<p>I am pretty sure it is across all the dataset.<br>
Which is also what the <a href="https://pytorch.org/docs/1.9.1/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">documentation</a> says here:</p>
<blockquote>
<p>For example, if a dataset contains 100 positive and 300 negative examples of a single class, then pos_weight for the class should be equal to 300/100=3. The loss would act as if the dataset contains 3*100=300 positive examples.</p>
</blockquote>
<p>Calculating it for every mini-batch would also be problematic since theoretically you could have a mini-batch with 0 positive samples in which case you would be dividing by 0.</p>
<aside class="quote no-group" data-username="Leo2" data-post="1" data-topic="132908">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leo2/48/37227_2.png" class="avatar"> Leo2:</div>
<blockquote>
<p>Does it matter if i keep redefining the loss function for each mini batch iteration, Is any information lost by doing that?</p>
</blockquote>
</aside>
<p>As for this question I am not sure.<br>
Technically it <strong>is</strong> possible to change the loss function during each iteration and there might be some very specific cases where this is done.<br>
But for your application using <code>BCEwithlogitloss</code> I think the intended usage would be defining the loss function once with <code>pos_weight</code> being like you said here</p>
<aside class="quote no-group" data-username="Leo2" data-post="1" data-topic="132908">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leo2/48/37227_2.png" class="avatar"> Leo2:</div>
<blockquote>
<p>pos_weight = total_neg/total_pos</p>
</blockquote>
</aside>
<p>where <code>total</code> would mean total across the whole dataset.</p>
<p>Just one more thing to keep in mind.<br>
If you use <code>pos_weight</code> you need to also be carefully with using any <code>random sampler</code> or <code>random sub-sampler</code> where you would be randomly reducing the number of samples per epoch, since than the calculated <code>pos_weight</code> would no longer be true.</p>

Leo2: <p>thank you very much. Makes total sense with 0 pos cases example</p>