Is AdamW will be invariant as if the loss function is time a positive number?

songyuc: <p>Hi, guys,<br>
According to the documentation of <code>AdamW</code> <a href="https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html#adamw" rel="noopener nofollow ugc">[doc]</a>, it seems that this implementation of <code>AdamW</code> will be invariant to the case of loss function time a positive number.<br>
But it does not behave like the documentation, in our test as,</p>
<pre><code class="lang-python">...
criterion1 = nn.CrossEntropyLoss()
criterion2 = nn.CrossEntropyLoss()
optimizer1 = optim.AdamW(net1.parameters(), lr=0.0001, betas=(0.1,0.1), weight_decay=0.9,eps=1e-08)
optimizer2 = optim.AdamW(net2.parameters(), lr=0.0001, betas=(0.1,0.1), weight_decay=0.9,eps=1e-08*1000)
# eps time a same scale
...

input2 = inputs.clone()
label2 = labels.clone()
# zero the parameter gradients
optimizer2.zero_grad()
# forward + backward + optimize
output2 = net2(input2)
loss2 = criterion2(output2, label2)*s
# loss2 time the same scale
loss2.backward()
optimizer2.step()
...
</code></pre>
<p><a href="https://colab.research.google.com/drive/1EzuOiwXMuxgl_w3Omwhvmju2GU2u-8St?usp=sharing" rel="noopener nofollow ugc">[Torch_ls.ipynb]</a><br>
So, how can we explain this behavior?</p>
<p>Your answer and guide will be appreciated!</p>

KFrank: <p>Hi Song!</p>
<aside class="quote no-group" data-username="songyuc" data-post="1" data-topic="146028" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a698b9/48.png" class="avatar"> songyuc:</div>
<blockquote>
<p>this implementation of <code>AdamW</code> will be invariant to the case of loss function time a positive number.<br>
But it does not behave like the documentation, in our test</p>
</blockquote>
</aside>
<p><code>AdamW</code> does appear to be invariant to the “scale” of the loss function.</p>
<p>Is it possible that you are not initializing <code>net1</code> and <code>net2</code> identically?<br>
Typically your network weights will be initialized randomly, and if you<br>
don’t reset the pseudorandom-number generator (or take other measures),<br>
<code>net1</code> and <code>net2</code> won’t start out the same.</p>
<p>Here is a self-contained, runnable script that illustrates <code>AdamW</code>’s<br>
invariance:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

_ = torch.manual_seed (2022)

inputs = torch.randn (10, 5)
labels = torch.randint (3, (10,))

_ = torch.manual_seed (12345)   # set seed used to initialize net1
net1 = torch.nn.Linear (5, 3)
_ = torch.manual_seed (12345)   # reset seed so that net2 is initialized identically to net1
net2 = torch.nn.Linear (5, 3)

print ('check weight:', torch.equal (net1.weight, net2.weight))
print ('check bias:', torch.equal (net1.bias, net2.bias))

scale = 1000.0

optimizer1 = torch.optim.AdamW (net1.parameters(), lr = 0.0001, betas = (0.1,0.1), weight_decay = 0.9, eps = 1e-08)
optimizer2 = torch.optim.AdamW (net2.parameters(), lr = 0.0001, betas = (0.1,0.1), weight_decay = 0.9, eps = scale * 1e-08)

for  i in range (5):
    optimizer1.zero_grad()
    optimizer2.zero_grad()
    loss1 = torch.nn.CrossEntropyLoss() (net1 (inputs), labels)
    loss2 = scale * torch.nn.CrossEntropyLoss() (net2 (inputs), labels)
    loss1.backward()
    loss2.backward()
    optimizer1.step()
    optimizer2.step()

print ('check weight:', torch.equal (net1.weight, net2.weight))
print ('check bias:', torch.equal (net1.bias, net2.bias))
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">1.10.2
check weight: True
check bias: True
check weight: True
check bias: True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

songyuc: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="146028">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p><code>scale * 1e-08</code></p>
</blockquote>
</aside>
<p>Thanks, Frank! I can get it from your code, but even if we changed my code following your setting, we still cannot get the equivalent results like yours.<br>
Could you please help me find out which operation may cause this difference?<br>
<a href="https://colab.research.google.com/drive/1EzuOiwXMuxgl_w3Omwhvmju2GU2u-8St?usp=sharing" rel="noopener nofollow ugc">[Torch_ls.ipynb]</a><br>
Appreciate it very much.</p>