torch.optim.SGD and hand-written SGD parameter updates are slightly different

mattlevine22: <p>I’m finding that during some iterations of SGD, the torch.optim.SGD yields a slightly different parameter update when compared to a manually written update.</p>
<p>So far, it looks like an issue of precision to me:</p>
<p>I’m comparing: <code>val.data - lr*val.grad.data</code><br>
vs <code>optimizer.step(); val.data</code></p>
<p>I’m comparing the norms of the weight matrices for simplicity (np.linalg.norm)</p>
<p>Here are two weight matrices which are QUITE close, but slightly different (this occurs a few steps into the 2nd epoch of SGD).<br>
optim_new =<br>
{‘B’: 0.842387760071843, ‘A’: 1.5769101725012014}</p>
<p>manual_new =<br>
{‘B’: 0.842387760071843, ‘A’: 1.5769101725012018}</p>
<p>I’m finding that my manual implementation outperforms optim.step, perhaps due to some unexpected numerical instability.</p>
<p>Any thoughts on this? Note that I’m training an RNN to fit a chaotic dynamical system, so these kinds of sensitivities accumulate and start to matter more than expected.</p>
<p>Note: I found the following similar thread, which seems to have suffered from a different problem (<a href="https://discuss.pytorch.org/t/losses-not-matching-when-training-using-hand-written-sgd-and-torch-optim-sgd/3467" class="inline-onebox">Losses not matching when training using hand-written SGD and torch.optim.SGD</a>)</p>
<p>UPDATE: I’m now finding that the discrepancy comes from the difference between:<br>
val.data - lr*val.grad.data<br>
and<br>
val.data.add_(val.grad, alpha=-lr) # this is used by optim.SGD.step()</p>
<p>SECOND UPDATE: this thread has been helpful…<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/14463" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/14463" target="_blank" rel="nofollow noopener">Tensor.add_() returns different values on different runs using some inputs (using CPU).</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-11-28" data-time="09:31:43" data-timezone="UTC">09:31AM - 28 Nov 18 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2019-12-16" data-time="20:11:17" data-timezone="UTC">08:11PM - 16 Dec 19 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/sakhar" target="_blank" rel="nofollow noopener">
          <img alt="sakhar" src="https://avatars2.githubusercontent.com/u/10168556?v=4" class="onebox-avatar-inline" width="20" height="20">
          sakhar
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
When using the CPU only (no CUDA), the in-place add function torch.Tensor.add_() returns different values on different calls (with trivial...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">high priority</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">topic: numerical-stability</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>Which PyTorch version are you using on which OS?<br>
I’m unsure, if this behavior wasn’t reproducible anymore at one point.</p>

mattlevine22: <p>PyTorch 1.5.0<br>
Python 3.7.6<br>
macOS Catalina 10.15.5</p>

ptrblck: <p>Unfortunately I don’t have access to a macOS machine, so feel free to update the linked issue with your code and some information.</p>

KDeja: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> I see that the issue is dead but I noticed exactly the same thing. There are slight differences between handwritten SGD and pytorch SGD. Is there any mechanism used to prevent numerical errors in the optimizer that I’m not aware of?<br>
Here is a snippet of the code to reproduce the issue.</p>
<pre><code class="lang-auto">import torch
import copy
h_sgd = torch.rand([100,100])
w = torch.rand([100,10])
lr = 1000
h_sgd.requires_grad = True
optim = torch.optim.SGD([h_sgd],lr=lr)
out = h_sgd@w
loss = torch.nn.CrossEntropyLoss()(out, torch.ones([100]).long())
loss.backward()
h_handwritten = copy.deepcopy((h_sgd - lr*h_sgd.grad).detach())
optim.step()
print((h_handwritten - h_sgd).std())
</code></pre>
<p>I’m using pytorch 1.12.1+cu113 on Ubuntu 18.04.6 LTS (Bionic Beaver). Seems like the difference is there both on CPU and GPU</p>

ptrblck: <p>Relative errors in the range <code>~1e-6</code> are expected for <code>float32</code> and in your case you are seeing:</p>
<pre><code class="lang-python">print((h_handwritten - h_sgd).abs().max())
# tensor(4.7684e-07, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p>which also fits.<br>
These errors are usually caused by a different order of operations and due to the limited floating point precision. Take a look at <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format">Wikipedia - Single-precision floating-point format</a> for more general information.<br>
Also note, that neither of these values is the “true” value and you cannot claim your custom implementation is the correct one while the built-in PyTorch method diverges. Both should show an initial error to the “exact” values (which are not representable in the used numerical format) and could later diverge entirely.<br>
I would expect the noise to be random and haven’t seen any custom implementation, which saw a benefit when it was executed multiple times.</p>

KDeja: <p>Thank you for the answer, of course I don’t expect any of the ways to be better than another. In fact with exactly the same sequence of operation as in SGD (h_sgd = h_sgd.add_(h_sgd.grad, alpha=-lr)) I get the same values. Thanks for clarification! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>