Weighted cross entropy

edshkim98: <p>Hello everyone, I am doing a deep learning project which has imbalanced class dataset.<br>
So, I am trying to use weighted cross entropy with soft dice loss.<br>
However, I have a question regarding use of weighted ce.<br>
I usually set my weights for classes as  1/no.instance which seems to be correct I think.<br>
This should work well as it counts every instances for each class but, this seems to be not working so well compare to when I approximately set the weights for each class.<br>
What could be a reason for my model performing worse when I weight classes by 1/number of occurrences in each image compare to giving weights by my random prediction?<br>
Thank you and I look forward to hearing from someone for the help!</p>

ptrblck: <p>I think setting the initial weights as <code>1/class_count</code> is a viable initial value, but not necessarily the most suitable for your use case and I think it’s the right approach to play around with these values to find a “sweet spot” which fits your use case well.</p>

edshkim98: <p>aha i see.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a>, just one more question about the loss function.<br>
For the case of multi class focal loss, is alpha which is the hyperparameter just the same weight as the weighted cross entropy?<br>
Thank you!</p>

ptrblck: <p>I’m not sure what <code>alpha</code> refers to, but the focal loss would be weigthed by <code>(1 - pred)**gamma</code>, if I remember it correctly. In this case the weighting isn’t a static value, but depends on the output probability of the model for the current target, such that “well-classified samples” get a lower loss than the wrongly classified ones.</p>

edshkim98: <p>Hi, thank you so much for the help.<br>
I just read the paper again and i think alpha is a list of weight for classes.<br>
Just one more question please.<br>
Currently, my dataset is hugely imbalanced and when I sometimes train my model, validation accuracy and iou stay exactly same for every epoch. Do you know what might have caused this and how to solve it?<br>
Could it be due to overfitting?</p>

ptrblck: <p>Is the training accuracy and IOU still decreasing while the validation metrics stay the same?<br>
In that case, yes it sounds like overfitting.</p>

edshkim98: <p>Thank you for the reply!<br>
The problem only occurs when I set learning rate scheduler.<br>
Could lr_scheduler cause sucb the problem?<br>
Also, Is the weight_decay parameter equivalent to L2 regularization?</p>

edshkim98: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="101933">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>ccuracy and IOU still decreasing while the validation metrics stay the same?<br>
In that case, yes it sounds like overfitting.</p>
</blockquote>
</aside>
<p>Actually training accuracy and IOU don’t decrease when this problem happens which is really weird.</p>

ptrblck: <p>Your learning rate scheduler might reduce the learning rate too far and maybe the training just gets stuck?<br>
Did you try to remove the scheduler and does the training benefit from it?</p>
<p><code>weight_decay</code> will add L2 regularization to all parameters for standard SGD. Note that for certain optimizers (such as <code>Adam</code>) the weight decay is not equal to L2 regularization as explained in <a href="https://arxiv.org/abs/1711.05101">Decoupled Weight Decay Regularization</a>, which is why <code>AdamW</code> was implemented.</p>

edshkim98: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you so much for the help!!<br>
I have been using the code below for optimizer:</p>
<pre><code class="lang-auto">optimizer = AdamW(model.parameters(), 0.01,weight_decay=1e-4)
lmbda = lambda epoch: 0.95
scheduler = torch.optim.lr_scheduler.MultiplicativeLR(optimizer, lr_lambda=lmbda)
</code></pre>
<p>I presume this lr_scheduler works by lr*0.95 for every epoch right?<br>
It works fine without scheduler, but when I add the scheduler, it starts to overfitting for some reason and it happens from epoch 1 so I am not sure whether this is a overfitting problem.<br>
Also, this problem is solved by adding weight_decay parameter in AdamW but I am not sure why.</p>
<p>Furthermore, I am currently facing a huge class imbalance problem in my semantic segmentation task.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/0/60c28614a9102978aacc47949b94c7c349d43424.png" alt="image" data-base62-sha1="dNYBlykd7IZk8ed5LUxFKtfN1Xe" width="375" height="264"><br>
The original data looks as above where 0-7 are classes.<br>
To solve this, other than using loss functions, I have also tried doing oversampling.<br>
This looks as below and unfortunately I can not make class 0 in same range as others as class 0 appears in every dataset.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6d32c816f6028c230e1f29ebf5170c21cf6c343.png" alt="image" data-base62-sha1="q5lkVnN7KD7Maj86VsC6rTjX3cn" width="381" height="264"><br>
Do you think the above could help to solve the imbalance problem?<br>
Sorry for keep asking questions.</p>

ptrblck: <aside class="quote no-group" data-username="edshkim98" data-post="10" data-topic="101933">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/edshkim98/48/29279_2.png" class="avatar"> edshkim98:</div>
<blockquote>
<p>I presume this lr_scheduler works by lr*0.95 for every epoch right?</p>
</blockquote>
</aside>
<p>Yes, that should be the case. You can double check it by printing the learning rates:</p>
<pre data-code-wrap="python"><code class="lang-python">[...]
optimizer.step()
scheduler.step()
print(optimizer.param_groups[0]['lr'])
print(scheduler.get_last_lr())
</code></pre>
<aside class="quote no-group" data-username="edshkim98" data-post="10" data-topic="101933">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/edshkim98/48/29279_2.png" class="avatar"> edshkim98:</div>
<blockquote>
<p>Do you think the above could help to solve the imbalance problem?</p>
</blockquote>
</aside>
<p>You could certainly try your oversampling approach, but currently you are creating a lot of class0 samples, so your model might overfit to this class even more now.</p>

edshkim98: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you for the reply.<br>
I know, the class 0 has increased even more now, but the problem is that the class 0 appears in every single image as it refers to background, so it is inevitable I think if I use oversampling(with a few augmentation methods) technique.<br>
I am currently training my model, but seems that the training iou has significantly increased but not so much with the validation iou yet.<br>
Do you think I shouldn’t use the oversampling with augmentation technique in my application and rather leave my original dataset as it was?</p>