Issues with combining loss together

username: PhilWallace
cooked: <p>Hi there.</p>
<p>I am trying to implement the multi-task training to my own model (implementation: <a href="https://github.com/Hui-Li/multi-task-learning-example-PyTorch/blob/master/multi-task-learning-example-PyTorch.ipynb" rel="nofollow noopener">https://github.com/Hui-Li/multi-task-learning-example-PyTorch/blob/master/multi-task-learning-example-PyTorch.ipynb</a>), and find some issues with combining the loss together.</p>
<p>I have three models to train, Model_1 and Model_2 are trained with nn.CrossEntropyLoss() and Model_3 is trained with nn.BCELoss().</p>
<p>how the training goes are like this:</p>
<pre><code class="lang-auto">optimizer.zero_grad()
loss = loss_1 + loss_2 + loss_3
loss = torch.mean(loss)
loss.backward()
optimizer.step()
</code></pre>
<p>But actually I found that Model_2 can not be trained properly, as loss_2 hardly decreases. However, when I try to train Model_2 separately with identical model and data, the loss can normally decrease and the model is learning quite well.</p>
<p>What I want to know is <strong>how loss.backward() specifically works in such situation?</strong> I also tried to manually give loss_2 a big weight value, or make loss_1 = loss_3 = 0, but these methods do not help.</p>
<p>Can someone please explain a little bit about that? Thanks a lot.  : )</p>

username: bolt25
cooked: <p>Which model parameters have you fed into the optimizer?<br>
Model_1, Model_2 or Model_3</p>

username: PhilWallace
cooked: <p>Parameters of all three models are fed into the optimizer, the trained model is actually a “LossWrapper” instance. How my code works is generally the same as in the link: <a href="https://github.com/Hui-Li/multi-task-learning-example-PyTorch/blob/master/multi-task-learning-example-PyTorch.ipynb" rel="nofollow noopener">https://github.com/Hui-Li/multi-task-learning-example-PyTorch/blob/master/multi-task-learning-example-PyTorch.ipynb</a></p>

username: googlebot
cooked: <p>This piece:</p>
<pre><code class="lang-auto">precision1 = torch.exp(-self.log_vars[0])
loss = torch.sum(precision1 * (targets[0] - outputs[0]) ** 2. + self.log_vars[0], -1)

precision2 = torch.exp(-self.log_vars[1])
loss += torch.sum(precision2 * (targets[1] - outputs[1]) ** 2. + self.log_vars[1], -1)
</code></pre>
<p>seems to be summing log-likelihoods of two normal distributions. But as their precisions differ, their densities are scaled differently and are thus not comparable. That loss works though, at least with no shared parameters, you just have to examine part losses separately.</p>
<p>Now, with classification losses, underlying distributions have no precision parameters, so simple unweighted sum of losses should work. Indeed, trainable loss weights may easily screw things up. Again, shared parameters may impede training, but otherwise for loss = loss1+loss2, autograd.backward([loss]) is the same as autograd.backward([loss1,loss2]) (i.e. independent backprops) as SumBackward doesn’t change gradients (and MeanBackward just divides gradients by a constant).</p>

username: PhilWallace
cooked: <p>Thanks for the reply. When I am training my demo, I actually did not add the ‘precision’ parameter, and the total loss is the torch.mean(torch.sum(loss_1 + loss_2 + loss_3)). And the training for loss_2 still does’t work.</p>
<p>My code has some differences with the code in the link: <a href="https://github.com/Hui-Li/multi-task-learning-example-PyTorch/blob/master/multi-task-learning-example-PyTorch.ipynb" rel="nofollow noopener">Multi-Task</a>. 1. My loss wrapper receives three separate inputs, and get three separate output and losses, while the model in the link shares one common input. 2. I found that loss_2 has different scale than loss_1 and loss_3, it is much smaller.</p>
<p>However, If the problem is 1: multi-input problem, loss_1 and loss_3 works normally; and if the problem is 2: scale problem, set loss_1 = loss_3 = 0 or simply abandon them still does not help.</p>
<p>I am still so confused and may need to check more details about “loss.backward()”. <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_head_bandage.png?v=9" title=":face_with_head_bandage:" class="emoji" alt=":face_with_head_bandage:"></p>

username: googlebot
cooked: <p>That doesn’t sound like backward() issue. Make sure your use same inputs &amp; targets as when separately training model2. If first epoch’s loss_2 is the same (with same parameters and inputs), investigate why optimizer does different things.</p>

username: PhilWallace
cooked: <p>Thanks a lot. It is the problem with the model_2. I changed the loss of model 2 from BCELoss to MSELoss, and the multi-task model can be trained normally.</p>