Is multiplying loss by 0.0 the same as not having the loss?

username: yxchng1
cooked: <p>I am meeting a strange problem where the loss multiplied by 0.0 still have effect on my model during training.</p>
<p>I do not want to have a lot of if else statements in my code, that’s why I decided to use multiplying by 0.0 as a switch.</p>

username: ptrblck
cooked: <p>Multiplying the loss with <code>0.0</code> will create zero gradients. However, your model could still “change” e.g. since running stats are updated in each forward pass in e.g. batchnorm layers during training (i.e. if the model is in <code>model.train()</code> mode). Also, the optimizer could still update parameters if its using running stats (e.g. <code>Adam</code>) and if the parameters were already updated (i.e. if the running stats are already set) even if the gradient is set to zero.</p>

username: yxchng1
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e80e5c2c8928e03f9cfa4bb81193eb9b4f0f71cc.png" data-download-href="https://discuss.pytorch.org/uploads/default/e80e5c2c8928e03f9cfa4bb81193eb9b4f0f71cc" title="Screenshot from 2022-11-03 13-51-41"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e80e5c2c8928e03f9cfa4bb81193eb9b4f0f71cc_2_690x345.png" alt="Screenshot from 2022-11-03 13-51-41" data-base62-sha1="x6RxrwH9obyLMnsEQeAXXeF4oeo" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e80e5c2c8928e03f9cfa4bb81193eb9b4f0f71cc_2_690x345.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e80e5c2c8928e03f9cfa4bb81193eb9b4f0f71cc_2_1035x517.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/8/e80e5c2c8928e03f9cfa4bb81193eb9b4f0f71cc.png 2x" data-dominant-color="EFEDEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-11-03 13-51-41</span><span class="informations">1108×555 82.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have not used BatchNorm in my model but I do indeed use Adam. However, based on the equation above, it seems like the optimizer only store the running stats of gradient, which for loss2 is always 0. (loss = loss1 + loss2, where loss2 is always multiplied by 0). Why will it still affect the training dynamics? Am I understanding Adam wrong?</p>

username: ptrblck
cooked: <p>The running stats are computed and updated in each <code>step()</code> and kept for the next update. Even if all gradients in the current <code>step()</code> are <code>0.0</code> the optimizer will still update them as seen here:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1, bias=False)
optimizer = torch.optim.Adam(model.parameters(), lr=1.)

x = torch.randn(1, 1)
out = model(x)
out.mean().backward()

print(model.weight)
# tensor([[-0.7789]], requires_grad=True)

optimizer.step()
print(model.weight)
# tensor([[-1.7789]], requires_grad=True)

optimizer.zero_grad()
print(model.weight.grad)
# tensor([[0.]])

optimizer.step()
print(model.weight)
# tensor([[-2.4490]], requires_grad=True)

# delete gradients
optimizer.zero_grad(set_to_none=True)
print(model.weight.grad)
# None

optimizer.step()
print(model.weight)
# tensor([[-2.4490]], requires_grad=True)
</code></pre>
<p>You would have to delete them (e.g. via <code>set_to_none=True</code>) so that <code>Adam</code> ignores them.</p>

username: J_Johnson
cooked: <p>Are you using dropout? This is randomly assigned during each batch.</p>

username: yxchng1
cooked: <p>I have drop path no drop out. Will drop path affect? How so?</p>

username: yxchng1
cooked: <p>This example is different from my case. My case has 2 losses, where second loss is always multiplied by 0, not saying the subsequent iteration has 0 gradient. So my question is will this second loss when multiplied with 0 the same as not adding it, in the case of Adam.</p>

username: ptrblck
cooked: <p>No, the second loss which is multiplied by <code>0</code> won’t have any effect on the first loss and the optimizer.</p>