Adding weights to the losses

username: duddal
cooked: <p>Hello,</p>
<p>I have a model with 5 different losses as shown below:</p>
<pre><code class="lang-auto">cls_loss_function = HeatmapLoss(reduction='mean')  # Custom Loss Function
txty_loss_function = nn.BCEWithLogitsLoss(reduction='none')
twth_loss_function = nn.SmoothL1Loss(reduction='none')
iou_loss_function = nn.SmoothL1Loss(reduction='none')
dep_loss_funciton = nn.SmoothL1Loss(reduction='none')
</code></pre>
<p>The losses are calculated and then the total_loss is returned as shown below.</p>
<p><code>total_loss = cls_loss + txty_loss + twth_loss + iou_loss + dep_loss total_loss.backward()</code></p>
<p>However, <code>txty_loss</code> dominates the <code>total_loss</code>. For example: After 50 Epochs with batch_size 3, Adam with lr=1e-4, GroupNorm with frozen BN layers of ResNet18 which is used as an Encoder, all the other losses except <code>txty_loss</code> are 0, but <code>txty_loss</code> ranges between 10 to 17. I think the model gives bad predictions because it is dominated by txty_loss. Could this be right?</p>
<p>Note: I have tried using SGD and Adam with different learning rates but the behaviour remains the same.</p>
<p>If so, how do I add weights to these losses, as in: Assigning coefficients to each loss but keeping the sum of those coefficients 1 ?. The concept should be similar to the weighted BCE, I think.</p>
<p>Feel free to correct if the concepts are wrong.</p>
<p>Thank You.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="duddal" data-post="1" data-topic="129908">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/duddal/48/39820_2.png" class="avatar"> duddal:</div>
<blockquote>
<p>all the other losses except <code>txty_loss</code> are 0, but <code>txty_loss</code> ranges between 10 to 17. I think the model gives bad predictions because it is dominated by txty_loss. Could this be right?</p>
</blockquote>
</aside>
<p>Wouldn’t this mean that the model training is respecting the other losses primarily (as they are reduced to 0), while <code>txty_loss</code> might increase?</p>
<aside class="quote no-group" data-username="duddal" data-post="1" data-topic="129908">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/duddal/48/39820_2.png" class="avatar"> duddal:</div>
<blockquote>
<p>If so, how do I add weights to these losses, as in: Assigning coefficients to each loss but keeping the sum of those coefficients 1 ?</p>
</blockquote>
</aside>
<p>Your approach sounds valid and you could assign coefficients to each loss term (and normalize them so that their sum equals <code>1</code>).<br>
E.g.:</p>
<pre data-code-wrap="python"><code class="lang-python">total_loss = 0.2 * cls_loss + 0.2 * txty_loss + 0.2 * twth_loss + 0.2 * iou_loss + 0.2 * dep_loss 
</code></pre>
<p>(of course you might want to change the values)</p>

username: duddal
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , Thank you for responding.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="129908">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Wouldn’t this mean that the model training is respecting the other losses primarily (as they are reduced to 0), while <code>txty_loss</code> might increase?</p>
</blockquote>
</aside>
<p>What does this mean exactly? But txty_loss decreases but then just remains kinda constant. If the model indeed respects the other losses than txty_loss, why though?</p>
<p>Regarding the values,<br>
so the txty_loss should have the highest weight?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="duddal" data-post="3" data-topic="129908">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/duddal/48/39820_2.png" class="avatar"> duddal:</div>
<blockquote>
<p>If the model indeed respects the other losses than txty_loss, why though?</p>
</blockquote>
</aside>
<p>I don’t know, but maybe the losses try to “pull” the model parameters into different directions?<br>
E.g. what would happen if the 4 losses expect positive output values of the model to reduce their loss while <code>txty_loss</code> expects negative outputs? This is of course a very simplified view of this use case, but could something similar happen in your training?</p>

username: duddal
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>Hmm, interesting thought. I hadn’t thought about this. But I think it can also be because of my data and/or using a batch size of 2 or 3 with GroupNorm. Also, my data is extremely complicated. Could that also be the reason?<br>
I am also training using only 2735 images. My belief is that I need more data.</p>

username: ptrblck
cooked: <p>It’s hard to tell what’s exactly causing the different loss values, but I would guess that the losses converging to zero might be easier to reduce for the model than the other (high) one.<br>
Yes, the number of samples seems to quite limited, but it also might depend on the used model, the complexity of the use case etc.</p>

username: duddal
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="129908">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>(of course you might want to change the values)</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> Sorry for the late reply. I was trying out a few things to check if the loss is decreased or not. I tried to complicate the model (increasing no. of parameters) but the reduction in loss was not observed.</p>
<p>Regarding the above quote,<br>
The loss with the highest value should have the lowest coefficient, right? and vice versa.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="duddal" data-post="7" data-topic="129908">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/duddal/48/39820_2.png" class="avatar"> duddal:</div>
<blockquote>
<p>The loss with the highest value should have the lowest coefficient, right?</p>
</blockquote>
</aside>
<p>I think this wouldn’t change the “behavior” of the training and the already high loss would be further “ignored”, no?</p>

username: duddal
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Oh yeah!!! So the highest loss value should have the highest coefficient. This way model would understand that this particular loss is the issue and should be worked on further, right?</p>

username: Jie_Xia
cooked: <p><a class="mention" href="/u/duddal">@duddal</a><br>
Hi there! I suffer from the same problem recently, did you figure it out, looking forward to hear from you!</p>