Gradient difference between autograd and manual backprop computation

kevin_zhang: <p>Hi,</p>
<p>I implemented a custom RNN in two ways:</p>
<ol>
<li>Implement the forward pass, and rely on autograd for backprop</li>
<li>Use the same forward pass, but write explicit, manually derived backprop</li>
</ol>
<p>Code for 2) can be found below, although forward implementation for both 1) and 2) are identical:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/kevinghst/scratch/blob/master/manual_backprop.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/kevinghst/scratch/blob/master/manual_backprop.py" target="_blank" rel="nofollow noopener">kevinghst/scratch/blob/master/manual_backprop.py</a></h4>
<pre><code class="lang-py">from torch import nn
from torch.autograd import Function
import torch
import torch.nn.functional as F

import pdb

def d_sigmoid(z):
    return (1 - z) * z

def d_tanh(z):
    return 1 - (z * z)

def d_softplus(x):
    return 1 / (1 + torch.exp(-x))

def d_cos(x):
    return -1 * torch.sin(x)


</code></pre>

  This file has been truncated. <a href="https://github.com/kevinghst/scratch/blob/master/manual_backprop.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>When I trained both versions on a language modeling task with the same seeds, however, the losses for 1) and 2) are different. Furthermore, 1) converges to a lower loss than 2). I also confirmed that when I ran 1) repeatedly, I do get the same losses. Which means that the seedings are supposed to produce deterministic behavior.</p>
<p>I checked the hidden state returned by the forward passes from 1) and 2), it is the same. So I know the forward pass works the same way in both versions.</p>
<p>When I checked the gradients after backpropagating once using loss.backward(), however, there appears to be a small difference between the weight gradient tensors. The Max absolute difference between any two indices is 4.7683716e-07. My best guess is that the difference in loss convergence is due to this small difference, which adds up across multiple backpropagations.</p>
<p>However, when I checked the gradient of my custom backpropagation model using torch.auto_grad.gradcheck, it tells me its OK.</p>
<p>The ONLY other difference between the 2 versions is that in my custom backpropagation, I returned None for input gradient, since I don’t think it’s required for my purpose anyway.</p>
<p>What do you think is the root cause for the difference in loss convergence?<br>
If you think it’s related to the small difference in gradient, how can I fix it?</p>
<p>Thanks,<br>
Kevin</p>

ptrblck: <p>The difference seems to come from the limited floating point precision.<br>
Note that <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.gradcheck"><code>gradcheck</code></a> will also use a tolerance value to check the gradients.</p>
<p>How reproducible is the convergence issue using these two approaches, i.e. are you able to get a lower loss using the first approach every time you run the code?</p>