How to handle Multiple Losses

Ahmed_Abdelaziz: <p>I have this model depicted in the figure. Model 1 and model 2 used to be two disjoint models such that they worked in a pipeline that we first train model 1 till convergence and feed the preprocessed outputs to model 2 as inputs. I am now training them end to end and I am struggling with how to integrate these two losses instead of just using <code>loss 2</code>. Other quesiton is, can I use two different optimizers for each of their parameters.</p>
<p>Should I do <code>loss1.backward()</code> first to update the gradients for the first model and then do <code>loss2.backward()</code> which will update the gradients for both model 1 and model 2 parameters. Do you think this is a good idea where gradients can be updated from both losses with a controlled learning rate so that I can force model 1 to learn more from <code>loss1</code> than from <code>loss2</code>?</p>
<p>Another idea that came to my mind is to sum both <code>loss1</code> and <code>loss2</code> (let’s call it <code>loss3</code>) and backpropagate. I have the initial idea that <code>loss3</code> will only backpropagate <code>loss2</code> until it reaches <code>c</code> and then backpropagated the weighted sum. Is that right?</p>
<p>Any ideas or references to the literature will be appreciated.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/616705d8873dcdae761bd1fe7434d65528e7548f.png" data-download-href="https://discuss.pytorch.org/uploads/default/616705d8873dcdae761bd1fe7434d65528e7548f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/1/616705d8873dcdae761bd1fe7434d65528e7548f.png" alt="image" data-base62-sha1="dTF2tUEMFDD9kSMir2iqzVi5Dyv" width="548" height="500" data-dominant-color="FBFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">551×502 4.59 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

KFrank: <p>Hello Ahmed!</p>
<aside class="quote no-group" data-username="Ahmed_Abdelaziz" data-post="1" data-topic="89563" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/958977/48.png" class="avatar"> Ahmed_Abdelaziz:</div>
<blockquote>
<p>Another idea that came to my mind is to sum both <code>loss1</code> and <code>loss2</code> (let’s call it <code>loss3</code>) and backpropagate. I have the initial idea that <code>loss3</code> will only backpropagate <code>loss2</code> until it reaches <code>c</code> and then backpropagated the weighted sum. Is that right?</p>
</blockquote>
</aside>
<p>Yes, this is, in effect, what will happen.</p>
<p>Summing the losses is the approach I would take.  I would probably<br>
include relative weights: <code>loss = w1 * loss1 + w2 * loss2</code>.</p>
<p>Note, that <code>loss1</code> does not depend on the weights in modules D and<br>
E, so backpropagating <code>loss1 + loss2</code> will have the same effect on<br>
modules D and E as backpropagating <code>loss2</code> alone.</p>
<p>But, as you correctly reason, both <code>loss1</code> and <code>loss2</code> will affect<br>
modules A, B, and C when backpropagating <code>loss1 + loss2</code>.</p>
<p>Be aware though, that including <code>loss1</code> could push push A, B, and<br>
C away from where they would provide the best performance for<br>
the output of your overall model (and lead to higher values of<br>
<code>loss2</code>).</p>
<p>On the other hand, including <code>loss1</code> could help prevent A, B, and<br>
C from overfitting your training set, potentially providing better<br>
performance on your test set.</p>
<p>Good luck.</p>
<p>K. Frank</p>

Ahmed_Abdelaziz: <p>Hi Frank,</p>
<p>Thanks for your elaborate answer. Your explanation makes sense. I am actually running an experiment with that setup and it seems that the loss is decaying. Thank you!</p>