How to balance different loss term?

username: nsknsl
cooked: <p>For example,<br>
The network has two outputs, ouput1 is from an intermedia layer and output2 is from the last layer.<br>
And my total loss:<br>
<code>loss_total = loss_1 + loss_2</code><br>
where loss_1 is calculated using output1 and loss_2 is calculated using output2.<br>
Now the problem is the loss_total seems to be dominated by loss_1 , and loss_2 doesn’t play a role。<br>
I try to put some weights on loss_1, loss_2, like:<br>
<code>loss_total = loss_1 * 0.001 + loss_2 *0.01</code><br>
However, this is not likely to work.<br>
Does any one has some ideals on the similar problems?</p>

username: dpernes
cooked: <p>I have no idea which specific losses you are using. Anyway, the following may help solving your problem.<br>
Just define your <code>loss_total</code> as:</p>
<pre><code>loss_total = alpha*loss_1 + (1-alpha)*loss_2
</code></pre>
<p>and finetune the hyperparameter <code>alpha</code> in the interval (0,1).</p>
<p>If you say that <code>loss_1</code> dominates the <code>total_loss</code>, then you’ll have o choose an <code>alpha</code> that is closer to 0 than to 1. Maybe you can start with <code>alpha = 0.4</code> and, if it still does not work, decrease it by a fixed step of 0.05 or so, until you get reasonable results.</p>
<p>However, before doing this cross-validation, you may wish to try training with <code>alpha = 0</code>, just to make sure that you can in fact minimize <code>loss_2</code>. There could be a bug in the definition of <code>loss_2</code> and this procedure allows you to check that.</p>

username: Pongroc
cooked: <p>Hi <a class="mention" href="/u/dpernes">@dpernes</a>. I have met the same problem as above and appreciate your solution. And I hope the network can learn the alpha itself just like other hyperparameters of the network. But the parameters in the network are optimized by the optimizer like SGD. So how can I do to optimize the alpha automaticly ?</p>

username: dpernes
cooked: <p>Hi <a class="mention" href="/u/pongroc">@Pongroc</a>.</p>
<p>In principle, <code>alpha</code> is a hyperparameter, so it is not learned by gradient-based optimization, but rather by cross-validation. (You know what cross-validation is, right?)</p>
<p>You may ask why is that so - and that’s precisely what I am going to try to explain. Suppose that <code>loss_1</code> is typically much larger than <code>loss_2</code>. Then, if you try to learn <code>alpha</code> and the remaining parameters jointly, in such a way that the <code>loss_total</code> is minimized, most likely you will get a value for <code>alpha</code> that is very close to 0, i.e. the optimization simply removes the contribution of <code>loss_1</code> to your <code>loss_total</code>. This does not mean, however, that <code>loss_1</code> will be low - on the contrary, it will probably be high. If both <code>loss_1</code> and <code>loss_2</code> are measurements of how well your model behaves (in two different senses), then probably you want both to be relatively low instead of having one that is very low and the other one that is very high. The role of the hyperparameter <code>alpha</code> is, therefore, to weight these two losses in some way that optimizes the <em>performance</em> of your model, not necessarily minimizing <code>loss_total</code>.</p>
<p>If it makes it easier for you to understand, you may also think of it in the context of regularization. You usually do not learn your regularization hyperparameter together with the model parameters, do you?<br>
Note that the usual regularized loss has a form that is very similar to the loss I proposed:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/aa60343b9a5472ba9233c18f5c93f8a491440650.png" alt="image" width="204" height="32">,</p>
<p>where <img src="https://discuss.pytorch.org/uploads/default/original/2X/1/1fe42e7c29cb33223677863187be4de1c74cbaa1.png" alt="image" width="38" height="20"> is some regularization function (e.g. L-2 norm).<br>
Dividing the RHS by a constant (independent of theta) does not change the minimizer of the function, so I may equivalently redefine the loss as:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e522b6cce2bc74a09e2187e116737b42189f0a3e.png" alt="image" width="269" height="39">,</p>
<p>which, by setting</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/6741663439af6a80124bc2c8b3489f722dd33744.png" alt="image" width="80" height="39">,</p>
<p>has exactly the same form as the <code>loss_total</code> that I proposed in my first reply to this topic.<br>
Now, note that if we set <code>alpha</code> to a value that is very close to 0 (or, equivalently, if we set lambda to a very large value), our <code>loss_total</code> will be totally dominated by the regularization term, and so we’ll get a model with a very poor performance (high bias), because it basically learns nothing related with the specific problem we are trying to solve. On the other hand, if we set <code>alpha</code> to a value that is very close to 1 (or, equivalently, if we set lambda to a very small value), our <code>loss_total</code> will be almost unregularized, and so we are likely to have a model that does not generalize well (high variance).</p>

username: zangzh97
cooked: <p>I met another problem. Suppose that the value of loss_1 is optimized from 4.xx to 1.xx while the value of loss_2 is optimized from 100.xx to 10.xx. Is that necessary to balance them into the same magnitude? Like<br>
L_total = alpha * loss_1 + (1 - alpha) * loss_2, where alpha is like 0.9 or 0.95.<br>
Thanks.</p>