Total loss of multi-task model

nikiguo93: <p>Hi guys, I have a multi-task model which would do regression and classfication together. I use mse loss and cross entropy loss for regression problem and classification problem, respectively. Then the total loss is defined according to <a href="https://github.com/ywatanabe1989/custom_losses_pytorch/blob/master/multi_task_loss.py" class="inline-onebox" rel="noopener nofollow ugc">custom_losses_pytorch/multi_task_loss.py at master · ywatanabe1989/custom_losses_pytorch · GitHub</a><br>
import torch</p>
<p>class MultiTaskLoss(torch.nn.Module):<br>
def <strong>init</strong>(self, is_regression, reduction=‘none’):<br>
super(MultiTaskLoss, self).<strong>init</strong>()<br>
self.is_regression = is_regression<br>
self.n_tasks = len(is_regression)<br>
self.log_vars = torch.nn.Parameter(torch.zeros(self.n_tasks))<br>
self.reduction = reduction</p>
<p>def forward(self, losses):<br>
dtype = losses.dtype<br>
device = losses.device<br>
stds = (torch.exp(self.log_vars)<strong>(1/2)).to(device).to(dtype)<br>
self.is_regression = self.is_regression.to(device).to(dtype)<br>
coeffs = 1 / ( (self.is_regression+1)*(stds</strong>2) )<br>
multi_task_losses = coeffs*losses + torch.log(stds)</p>
<pre><code>if self.reduction == 'sum':
  multi_task_losses = multi_task_losses.sum()
</code></pre>
<p>With the total loss decreasing, the mse loss went up and the cross entropy loss declined. And the accuracy is not satisfactory. Can anybody give me some clues to improve that?<br>
total loss:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/3/4334329514c59155a9881a7b1bdea827e785fbfd.png" data-download-href="https://discuss.pytorch.org/uploads/default/4334329514c59155a9881a7b1bdea827e785fbfd" title="1649240879(1)"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/3/4334329514c59155a9881a7b1bdea827e785fbfd.png" alt="1649240879(1)" data-base62-sha1="9AvP2TO8RbsE6SBCMFmkIRYLblj" width="690" height="245" data-dominant-color="FAF9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1649240879(1)</span><span class="informations">856×305 17.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
classification accuracy:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf4e3d0fb072906ea4b277cfb9868bfd1b051c22.png" data-download-href="https://discuss.pytorch.org/uploads/default/bf4e3d0fb072906ea4b277cfb9868bfd1b051c22" title="1649240946(1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf4e3d0fb072906ea4b277cfb9868bfd1b051c22_2_287x500.png" alt="1649240946(1)" data-base62-sha1="rimOYcAzQzjToZFe8htveynVRJ0" width="287" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf4e3d0fb072906ea4b277cfb9868bfd1b051c22_2_287x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf4e3d0fb072906ea4b277cfb9868bfd1b051c22_2_430x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf4e3d0fb072906ea4b277cfb9868bfd1b051c22.png 2x" data-dominant-color="FAF9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1649240946(1)</span><span class="informations">435×757 26.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
mse loss for regression problem:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/7911666f6418d41ac8bdc6536a43587313a66763.png" data-download-href="https://discuss.pytorch.org/uploads/default/7911666f6418d41ac8bdc6536a43587313a66763" title="1649241139(1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/7911666f6418d41ac8bdc6536a43587313a66763_2_283x500.png" alt="1649241139(1)" data-base62-sha1="hh13jPAFtfiupOuIQOui82Hg0a7" width="283" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/7911666f6418d41ac8bdc6536a43587313a66763_2_283x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/7911666f6418d41ac8bdc6536a43587313a66763_2_424x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/9/7911666f6418d41ac8bdc6536a43587313a66763.png 2x" data-dominant-color="FAF9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1649241139(1)</span><span class="informations">428×754 28.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
cross entropy loss for classification problem:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d595b01435adc3e0922c71505dc94bb69519907d.png" data-download-href="https://discuss.pytorch.org/uploads/default/d595b01435adc3e0922c71505dc94bb69519907d" title="1649241170(1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d595b01435adc3e0922c71505dc94bb69519907d_2_286x500.png" alt="1649241170(1)" data-base62-sha1="utspa6QTswAFUnGjRXzlWqn7i9f" width="286" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d595b01435adc3e0922c71505dc94bb69519907d_2_286x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d595b01435adc3e0922c71505dc94bb69519907d_2_429x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/5/d595b01435adc3e0922c71505dc94bb69519907d.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1649241170(1)</span><span class="informations">436×762 27.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

KFrank: <p>Hi Nikiguo!</p>
<aside class="quote no-group quote-modified" data-username="nikiguo93" data-post="1" data-topic="148445" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikiguo93/48/30791_2.png" class="avatar"> nikiguo93:</div>
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">class MultiTaskLoss(torch.nn.Module):
    ...
    self.log_vars = torch.nn.Parameter(torch.zeros(self.n_tasks))
    ...
</code></pre>
<p>With the total loss decreasing, the mse loss went up and the cross entropy loss declined. And the accuracy is not satisfactory.</p>
</blockquote>
</aside>
<p>Your <code>MultiTaskLoss</code> is a scheme to “learn” the appropriate relative<br>
weighting for the two losses for the two dissimilar tasks – regression<br>
and classification – that you are training your network to perform at<br>
the same time.</p>
<p>I have three comments:</p>
<p><code>self.log_vars</code> is a “learnable” parameter that is to be trained by your<br>
optimizer.  Make sure that you have added <code>self.log_vars</code> to whatever<br>
optimizer you are using.  Otherwise it won’t get updated, and you’ll just<br>
be summing your two losses together, which could easily not be giving<br>
enough weight to your “mse loss.”</p>
<p>It’s possible that your use case isn’t well suited to the way that<br>
<code>MultiTaskLoss</code> works.  I generally advise trying the “simple” approach<br>
first, even if the fancier approach should (and maybe even does) work<br>
better.  Try leaving out <code>MultiTaskLoss</code> and just sum your two losses<br>
together with a weighting factor.  Can you get your network to train with<br>
both losses going down reasonably by adjusting your weighting factor<br>
appropriately?  (In essence, you would be doing by hand what<br>
<code>MultiTaskLoss</code> is supposed to do for you automatically.)</p>
<p>Your training looks fine.  I don’t see any evidence of your cross-entropy<br>
loss or accuracy plateauing or of overfitting.  The only problem is that<br>
your mse loss starts going up after falling initially.  This suggest that your<br>
mse loss is not weighted heavily enough in your total loss.  This could<br>
happen if you failed to include <code>self.log_vars</code> in your optimizer (or if<br>
<code>MultiTaskLoss</code> just doesn’t work for your use case).  But it could also<br>
happen if it takes a while for <code>self.log_vars</code> itself to train properly.<br>
Your charts run for about 650 “somethings,” which I imagine might be<br>
epochs,  For many substantive problems, this isn’t very much training<br>
at all, so you might try training significantly longer (after verifying that<br>
<code>self.log_vars</code> is being updated by your optimizer).  Perhaps the<br>
relative loss weight will start to be learned properly, your mse loss<br>
will be weighted more heavily, and mse loss will start going down.<br>
(And even if your regression problem doesn’t train well, maybe your<br>
classification problem will continue to train effectively and your<br>
classification accuracy will become “satisfactory.”)</p>
<p>Best.</p>
<p>K. Frank</p>