Summing the losses from different models produces different results

silvester: <p>Assuming two independent models <code>model0</code> and <code>model1</code>, when I train these two models individually with losses <code>loss0</code> and <code>loss1</code> correspondingly, I get different results (accuracy) compared to the time that I train the models together and with the loss <code>loss = loss0+loss1</code>.<br>
I appreciate any thought on this problem.<br>
Cheers</p>

InnovArul: <p>I think, that’s how it is supposed to work. No?<br>
For instance, softmax and triplet losses work this way too.</p>

silvester: <p>Could you please elaborate, as  <code>grad(loss) = grad(loss0) + grad(loss1)</code> and these two models have no parameters in common. So they should get same gradients as before when I trained them individually.</p>
<p>I again emphasize that the only difference is that I sum the losses and backpop on the summed loss.</p>

InnovArul: <p>I see. I missed that they are two independent models.</p>
<p>How do you initialize optimizers?<br>
Do you use the same optimizer for both?</p>

silvester: <p>The parameters for two models are put into list and an optimizer is setup based on the parameters in the list. So I assume the optimizer should work similar for both of the models.</p>

InnovArul: <aside class="quote no-group" data-username="silvester" data-post="3" data-topic="100449">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/6bbea6/48.png" class="avatar"> silvester:</div>
<blockquote>
<p><code>grad(loss) = grad(loss0) + grad(loss1)</code></p>
</blockquote>
</aside>
<p>As you said here, gradients are independent I suppose.<br>
So, I am not sure how to debug this. I do not know if the advanced optimizers (Like Adam) consider the parameters as independent entities or if they perform any operations to normalize the gradients somehow.</p>
<p>Did you try with simple SGD?</p>

silvester: <p>What I did is with SGD with decay rate and momentum, though the idea of normalization seems interesting to me, too. But it is not clear when this is happened. I am suspicious may it is also related to the number of parameters, as in the summed case the number of parameters is twice of the number of parameters in the individual training.</p>

InnovArul: <p>I am not sure about the case of the number of parameters. Do you have a concrete reason behind this thought? Is it related to overfitting?</p>
<p>As the gradients are independent, I would expect them to behave more or less similarly in both cases.<br>
I hope you do not have implementation-related issues.</p>

silvester: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I was hopping to get some leads from Pytorch community about this issue.</p>

silvester: <p>Does applying np.float32 rather than np.int64 for the labels lead to degradation in accuracy?</p>

ptrblck: <p>If the losses are independent and don’t share any parameters, which were used to compute them, you should get the same results.<br>
Are you able to reproduce this issue deterministically by seeding the code?<br>
If so, could you share a code snippet so that we could have a look?<br>
Here is a small example, which yields exact the same gradients:</p>
<pre><code class="lang-python"># separate losses
torch.manual_seed(2809)

model1 = models.resnet18()
model2 = models.resnet34()

x = torch.randn(2, 3, 224, 224)
target = torch.tensor([0, 999])
criterion = nn.CrossEntropyLoss()

output1 = model1(x)
output2 = model2(x)
loss1 = criterion(output1, target)
loss2 = criterion(output2, target)

loss1.backward()
loss2.backward()
grad1_ref = {name: p.grad.clone() for name, p in model1.named_parameters()}
grad2_ref = {name: p.grad.clone() for name, p in model2.named_parameters()}

model1.zero_grad()
model2.zero_grad()

# loss sum
torch.manual_seed(2809)

model1 = models.resnet18()
model2 = models.resnet34()

x = torch.randn(2, 3, 224, 224)
target = torch.tensor([0, 999])
criterion = nn.CrossEntropyLoss()

output1 = model1(x)
output2 = model2(x)
loss1 = criterion(output1, target)
loss2 = criterion(output2, target)

loss = loss1 + loss2
loss = loss.backward()
grad1 = {name: p.grad.clone() for name, p in model1.named_parameters()}
grad2 = {name: p.grad.clone() for name, p in model2.named_parameters()}

# compare
for name in grad1_ref:
    print('model1 grad diff for {}: {}'.format(
        name, (grad1_ref[name] - grad1[name]).abs().max()))
    print('model2 grad diff for {}: {}'.format(
        name, (grad2_ref[name] - grad2[name]).abs().max()))
</code></pre>