BCEWithLogitsLoss validation loss did not crease after applying pos_weight

Mahammad_Nabizade: <p>Hi everyone, I’m dealing a multilabel segmentation problem where I have 2 classes - Background class, Object class. The objects are so small so it means there is a class imbalance in data. Thus, I decided to use BceLossLogit with pos_weight.</p>
<ol>
<li>For the first epoch the training loss was so high - around 7. Is it normal to start with such a high loss - it only happens when I use pos_weight parameter.</li>
<li>From the second epoch the loss became normal but after that the validation loss did not decrease. In general what is the best case practice to choose a loss function in such tasks?</li>
</ol>
<p>The model outputs (also the target images) have this shape : [Batch, Number of Classes, H, W ]</p>
<pre><code class="lang-auto"> def _calculate_positive_weight(self, label_images) -&gt; torch.tensor:
        class_counts = label_images.sum(dim=(0, 2, 3))
        total_count = label_images.numel() / label_images.size(1)
        positive_weight = total_count / (2 * class_counts[1].float())
        positive_weight_tensor = torch.tensor([positive_weight]).to(self.device)
        return positive_weight_tensor

    def _initialize_criterion(self):
        for batch in self.training_data_loader:
            label_images = batch["label_image_tensor"].float().to(self.device)
            positive_weight_tensor = self._calculate_positive_weight(label_images)
            self.criterion = torch.nn.BCEWithLogitsLoss(pos_weight=positive_weight_tensor)
            break
</code></pre>
<p>I appreciate your support.</p>

KFrank: <p>Hi Mahammad!</p>
<p>The short story – making some assumptions about your use case and<br>
what your model is doing – is that you should modify your model to predict<br>
a single value per pixel and modify your target images accordingly.</p>
<aside class="quote no-group" data-username="Mahammad_Nabizade" data-post="1" data-topic="204088" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahammad_nabizade/48/69699_2.png" class="avatar"> Mahammad_Nabizade:</div>
<blockquote>
<p>I’m dealing a multilabel segmentation problem where I have 2 classes - Background class, Object class.</p>
</blockquote>
</aside>
<p>A note on terminology:  I think you mean you have a <em>multiclass</em> problem.<br>
That is you have a single-label, two-class problem.</p>
<p>By way of example, a <em>multilabel</em> problem would be something like where<br>
you classify images according to three attributes, say animal - cat, dog,<br>
rabbit (three classes); type - black and white, color (two classes); focus -<br>
blurry, sharp (two classes).</p>
<p>Please correct me if you really do have a multilabel problem – this matters<br>
for how the rest of your post should be understood.</p>
<aside class="quote no-group">
<blockquote>
<p>I decided to use BceLossLogit with pos_weight.</p>
</blockquote>
</aside>
<p>As I understand you, your model classifies each pixel to be either<br>
“Background” or “Object.”  So you have a two-class – which is to say<br>
binary – segmentation problem.</p>
<p>While conceptually the same, the pytorch infrastructure distinguishes<br>
between a binary problem and a general multiclass problem that happens<br>
to be a two-class (and hence conceptually binary) problem.</p>
<p>Specifically, if you structure your problem as binary, your model should<br>
predict a single output per pixel and that output should be (if you are<br>
using <code>BCEWithLogitsLoss</code>) the <em>logit</em> that corresponds to the probability<br>
of that pixel being in the “Object” class.</p>
<p>On the other hand, if you structure you problem as a multiclass problem,<br>
that in your case happens to be two-class, your model should predict two<br>
outputs per pixel that correspond to the unnormalized log-probabilities of<br>
that pixel being in each of the two classes, “Background” and “Object.”<br>
For such a multiclass problem – even in the case of two classes – you<br>
would typically use <code>CrossEntropyLoss</code> as your loss criterion.</p>
<aside class="quote no-group">
<blockquote>
<p>The model outputs (also the target images) have this shape : [Batch, Number of Classes, H, W ]</p>
</blockquote>
</aside>
<p>This says to me that your model is treating this as a multiclass problem<br>
with two classes.  But, while a legitimate approach, this will not work<br>
correctly with <code>BCEWithLogitsLoss</code> as your loss criterion.</p>
<p>So, to repeat, modify your model to predict a single output value per<br>
pixel (with shape <code>[Batch, H, W ]</code> – no class dimension), modify your<br>
target images analogously to have a single target value per pixel, <code>0.0</code><br>
for “Background” and <code>1.0</code> for “Object” (again with shape <code>[Batch, H, W ]</code>),<br>
use <code>BCEWithLogitsLoss</code>, and use its <code>pos_weight</code> constructor argument<br>
to compensate for your class imbalance.</p>
<p>(Note, <code>BCEWithLogitsLoss</code> will work correctly if both your predictions and<br>
target images carry a singleton class dimension (that is, a “trivial” class<br>
dimension of length one), but I prefer stylistically to leave such a class<br>
dimension out.)</p>
<p>Best.</p>
<p>K. Frank</p>

Mahammad_Nabizade: <p>Thank you so much for such a detailed answer.</p>
<p><strong>Problem Description</strong><br>
You are right I confused the terminology - the problem is a binary segmentation problem. To give you more info - this is a change detection model taking 2 images and resulting the change mask.<br>
I changed the model now.</p>
<ul>
<li>Input : rgb image converted to tensor <code>([1, 3, 224, 224])</code></li>
<li>Target : Mask with 2 colors (white background, with a foreground showing objects) - converted to <code>tensor ([1, 1, 224, 224])</code> - the values are binary (0 or 1)</li>
<li>Model output : <code>([1, 1, 224, 224])</code> tensor - the values inside the are not binary - to visualize the output I apply <strong>sigmoid</strong> function to make it binary</li>
</ul>
<pre><code class="lang-auto"> with torch.no_grad():
        prediction = model((test_image, reference_image))
    
    output = torch.nn.functional.sigmoid(prediction)
    output = (output &gt; 0.5).float()
</code></pre>
<p><strong>Current Problems:</strong></p>
<ol>
<li>The validation loss stops improving (aka decreasing) after a certain epoch(20-25th) so early - stopping happens .</li>
<li>Also, validation loss is less than training loss at the beginning for the first 5-6 epochs which I believe should be the opposite</li>
</ol>
<p>Does this mean <strong>overfitting</strong> , <strong>undeffitting</strong> or both ? Can it be because the model is too complex for the task and it learns quickly but can not generalize ? I used several techniques such as Weight Decay, Drop-out but none of them really worked.</p>
<p><strong>Questions:</strong></p>
<ol>
<li>
<p>Should I change the model architecture so that, the input and output will be [batch, H, W] or can I keep [batch, 1, H, W]. If I keep [batch, 1, H, W] - I can’t use CrossEntropy (the loss values become zero if I do)</p>
</li>
<li>
<p>If I use <strong>pos weight</strong> parameter, how should I calculate it ? As I know it is division of negative classes/positive classes - in my case this ratio is too high because usually the images contain a tiny blue part, so I think this is a huge class imbalance. For training set the average ratio is <strong>311</strong>, for validation set <strong>1179</strong>, for test set <strong>414</strong>. So should I use an intuitive general factor for this ratio (for example: pos weight = 4, meaning that there is a 80/20 ratio). As I understood this is a trade-off between precision and recall and if the <strong>pos weight</strong> is high, then the recall value starts from a too high (0.96 for example) value then decreases where all other metrics slowly increase. If I use the real ratio (which is so high) the <strong>validation loss stops improving</strong> after 6-7th epoch and early stopping happens.</p>
</li>
</ol>
<p>The questions can be really basic or noob, I’m sorry in advance. I’m just a beginner and I’d appreciate your help.</p>

KFrank: <p>Hi Mahammad!</p>
<aside class="quote no-group" data-username="Mahammad_Nabizade" data-post="3" data-topic="204088" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahammad_nabizade/48/69699_2.png" class="avatar"> Mahammad_Nabizade:</div>
<blockquote>
<ul>
<li>Target : Mask with 2 colors (white background, with a foreground showing objects) - converted to <code>tensor ([1, 1, 224, 224])</code> - the values are binary (0 or 1)</li>
<li>Model output : <code>([1, 1, 224, 224])</code> tensor - the values inside the are not binary</li>
</ul>
</blockquote>
</aside>
<p>This is fine.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto"> with torch.no_grad():
        prediction = model((test_image, reference_image))
</code></pre>
</blockquote>
</aside>
<p>For inference (e.g., validation), this is fine.  For training (where you would<br>
use backpropagation), you would not use <code>with torch.no_grad():</code>.</p>
<aside class="quote no-group">
<blockquote>
<ol>
<li>The validation loss stops improving (aka decreasing) after a certain epoch(20-25th) so early - stopping happens .</li>
</ol>
</blockquote>
</aside>
<p>You haven’t said how big your dataset is, so we don’t know how many<br>
samples you have in an epoch.  However 25 epochs of training is in<br>
general quite small.  Try training <em>much</em> longer.  It’s perfectly possible<br>
for your training loss to “plateau” and then start making progress again<br>
as you train more (and this can happen multiple times in a training run).</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>Also, validation loss is less than training loss at the beginning for the first 5-6 epochs which I believe should be the opposite</li>
</ol>
</blockquote>
</aside>
<p>This could just be “noise.”  Six epochs is basically nothing.  If the random<br>
initialization of your model happens to better match the samples randomly<br>
selected for your validation set, your validation loss might randomly be<br>
lower than your training loss (and you might expect to see this about half<br>
the time).  Only if your validation loss systematically remains lower than<br>
your training loss after you have trained <em>much longer</em> should you start to<br>
suspect that something fishy might be going on.</p>
<aside class="quote no-group">
<blockquote>
<p>Does this mean <strong>overfitting</strong> , <strong>undeffitting</strong> or both ?</p>
</blockquote>
</aside>
<p>Neither.  For any but the simplest toy problems, you haven’t trained long<br>
enough to reach any conclusions.</p>
<aside class="quote no-group">
<blockquote>
<ol>
<li>Should I change the model architecture so that, the input and output will be [batch, H, W] or can I keep [batch, 1, H, W]. If I keep [batch, 1, H, W]</li>
</ol>
</blockquote>
</aside>
<p>You can do it either way – the two are equivalent.  For purely stylistic<br>
reasons, I prefer to not have the singleton dimension, but you don’t<br>
need to change your model architecture to get rid of it – you can simply<br>
<code>.squeeze()</code> away any singleton dimensions that the output of your<br>
model might have (if you care – again, it doesn’t matter).</p>
<aside class="quote no-group">
<blockquote>
<p>I can’t use CrossEntropy (the loss values become zero if I do)</p>
</blockquote>
</aside>
<p>You don’t want to use <code>CrossEntropyLoss</code>.  You should be using<br>
<code>BCEWithLogitsLoss</code>.</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>If I use <strong>pos weight</strong> parameter, how should I calculate it ? As I know it is division of negative classes/positive classes - in my case this ratio is too high because usually the images contain a tiny blue part, so I think this is a huge class imbalance.</li>
</ol>
</blockquote>
</aside>
<p>I wouldn’t call this a <em>huge</em> imbalance, but it is large enough that you will<br>
want to compensate for it, for example, by using <code>pos_weight</code>.  (Some<br>
people suggest using an intersection-over-union (IoU) or Dice-coefficient<br>
loss for imbalanced segmentation problems.  My advice is to start with<br>
<code>pos_weight</code> and only <em>augment</em> <code>BCEWithLogitsLoss</code> with something<br>
like IoU if its clear that <code>pos_weight</code> isn’t working well enough.)</p>
<aside class="quote no-group">
<blockquote>
<p>For training set the average ratio is <strong>311</strong>, for validation set <strong>1179</strong>, for test set <strong>414</strong>. So should I use an intuitive general factor for this ratio (for example: pos weight = 4, meaning that there is a 80/20 ratio).</p>
</blockquote>
</aside>
<p>No, a value of 4 would be way to small to effectively compensate for your<br>
class imbalance.  A value of about 400 would be a good starting point.</p>
<aside class="quote no-group">
<blockquote>
<p>As I understood this is a trade-off between precision and recall</p>
</blockquote>
</aside>
<p>It’s up to you and your use case whether precision or recall is more<br>
important to you and therefore how you should tune that trade-off.</p>
<aside class="quote no-group">
<blockquote>
<p>If I use the real ratio (which is so high)</p>
</blockquote>
</aside>
<p>Start by using a value that is roughly equal to the real ratio.  Train for much<br>
longer so that your results are meaningful.  Then look at your precision and<br>
recall and adjust <code>pos_weight</code>, as appropriate, to achieve your desired<br>
precision / recall trade-off.</p>
<aside class="quote no-group">
<blockquote>
<p>the <strong>validation loss stops improving</strong> after 6-7th epoch</p>
</blockquote>
</aside>
<p>Again, 7 epochs of training is almost nothing.</p>
<aside class="quote no-group">
<blockquote>
<p>and early stopping happens.</p>
</blockquote>
</aside>
<p>Turn off early stopping (so that you can let your training run much longer).</p>
<p>Best.</p>
<p>K. Frank</p>