Combine Losses and Weight those

JanoschMenke: <p>I am training a multitask model in which I have some classification and some regression tasks.</p>
<p>So I am using two loss functions:</p>
<pre><code class="lang-auto">loss_function_reg = nn.MSELoss()
loss_function_clf = nn.BCEWithLogitsLoss()
</code></pre>
<p>and combine them:</p>
<pre><code class="lang-auto">loss_reg = loss_function_reg(prediction_reg, batch[1].cuda())
loss_clf = loss_function_clf(prediction_clf.flatten(), batch[2].cuda())
loss = loss_reg + loss_clf
loss.backward(loss)
</code></pre>
<p>I have way more regression tasks than I have classification tasks. Should I weight for that? Because the loss obtained from either loss function is the average across all tasks. So the sum of the two losses is “biased” towards the loss function with less variables/tasks. Or Did I just overthink that?</p>

ccoding: <p>You can merge them, that’s ok. And both in technical and mathematics,  it’s work.<br>
But adding them together is a simple way, you can add learning variable <strong>a</strong> to self-learning the “biased” of that two different loss.</p>
<pre><code class="lang-python">a = torch.tensor(0.5, requires_grad=True)
loss = (1-a)*loss_reg + a*loss_clf
</code></pre>

JanoschMenke: <p>But if a is learnable, would the netowkr not start minimizing the loss function which is easier to minimize and then just fit a to only value that loss function?</p>

KFrank: <p>Hi Janosch!</p>
<aside class="quote no-group" data-username="JanoschMenke" data-post="3" data-topic="108700" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/janoschmenke/48/15781_2.png" class="avatar"> JanoschMenke:</div>
<blockquote>
<p>But if a is learnable, would the netowkr not start minimizing the loss function which is easier to minimize and then just fit a to only value that loss function?</p>
</blockquote>
</aside>
<p>Conceptually, yes, <code>a</code> would train to prefer the easier loss.  But, as<br>
written, <code>a</code> could train to become a very large positive number or a<br>
very large negative number, with your loss becoming unbounded<br>
below.</p>
<p>(You could train <code>(1 - a)**2 * loss_reg + a**2 * loss_clf</code>,<br>
but now <code>a</code> will just train to 0 or 1 to ignore the “harder” loss.)</p>
<p>As to your original question, it does make sense to consider weighting<br>
the two losses, e.g., something like:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = loss_reg + alpha * loss_clf
</code></pre>
<p>But <code>alpha</code> should now be a non-trainable hyperparameter whose<br>
value you tune “by hand.”  (You could also contemplate using some<br>
automated hyperparameter-tuning scheme, but those are often more<br>
complicated than they are worth.)</p>
<p>Best.</p>
<p>K. Frank</p>

ccoding: <p>That’s not true.<br>
Cuz you could get loss back propagate to all elements.<br>
And they share the loss equally<br>
see the formula of loss</p>
<pre><code class="lang-python">loss = (1-a)*loss_reg + a*loss_clf
</code></pre>
<p>the derivatives of <strong>a</strong>( or <strong>alpha</strong>)<br>
is <code>loss_clf - loss_reg</code><br>
make them equal is not easy</p>

JanoschMenke: <p>Maybe I am missing your point. But my and kind of <a class="mention" href="/u/kfrank">@KFrank</a> point was that if you and learnable alpha it would bias your loss towards the easier predictions.</p>
<p>As in my example i have a few classification tasks( and the appropriate loss: loss_clf) and regression tasks(loss_reg)</p>
<p>If one of the two is easier to minimize ( yields a smaller loss) then alpha could just be set to only weight the smaller loss and ignore the “harder” task. The model would then just backproagate the loss of the one task but not the other effectively not training for the harder task</p>

ccoding: <p>Thank you for explaining details of “hard” task.<br>
I understood what you mean. So set <strong>a</strong> to a constant number, and harder task<br>
got larger number.<br>
That’s like weight loss, that is weight loss.</p>

JanoschMenke: <p>Yes for example. But I was more concerned about the need for weighting.</p>
<p>Lets says you have 45 regression tasks and only 1 classification task.<br>
If you combine the two losses (loss_reg+loss_clf), do you “overweight” the clf loss? I am not sure. And that is why I was wondering if you should reweight the regression loss</p>

ccoding: <p>You can’t think it as numbers’ adding.<br>
It‘s optimization of mathmatics, finding the minimum params apply to the loss function.<br>
So according to the <a href="https://en.wikipedia.org/wiki/Lagrange_multiplier" rel="noopener nofollow ugc">Lagrange multiplier</a>, you can find the most appropriate <strong>a</strong> to balance <code>loss_ctf</code> and <code>loss_reg</code>.</p>