Numerical instability in Google Colab

JonathanSum: <p>The torch allclose function on the grad value and the hand-calculated grad values are very far away on colab. However, the issue was gone when the same code ran locally.<br>
For more detail, you can see here.</p><aside class="onebox githubissue" data-onebox-src="https://github.com/karpathy/nn-zero-to-hero/issues/13">
  <header class="source">

      <a href="https://github.com/karpathy/nn-zero-to-hero/issues/13" target="_blank" rel="noopener nofollow ugc">github.com/karpathy/nn-zero-to-hero</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/karpathy/nn-zero-to-hero/issues/13" target="_blank" rel="noopener nofollow ugc">Numerical instability in Google Colab - Part 4 of Makemore</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2022-10-13" data-time="18:00:45" data-timezone="UTC">06:00PM - 13 Oct 22 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/sachag678" target="_blank" rel="noopener nofollow ugc">
          <img alt="sachag678" src="https://avatars.githubusercontent.com/u/19661393?v=4" class="onebox-avatar-inline" width="20" height="20">
          sachag678
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">I ran into an interesting issue in makemore 4 backpro ninja where the dhpreact w<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">as not exactly matching the hpreact.grad. 

However, this was only in the collab notebook because when I put the same code into a local jupyter notebook it works fine. 

Not sure why this would be the case but just an odd curiosity.</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
You can run the colab notebook here: <a href="https://colab.research.google.com/drive/1HmZ8bgtAfvyMaZyu3Sr1Bgxsj35jitTs?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/6/26846490a627368b402d236c5da8987ecfac46fc.png" data-download-href="https://discuss.pytorch.org/uploads/default/26846490a627368b402d236c5da8987ecfac46fc" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/26846490a627368b402d236c5da8987ecfac46fc.png" alt="image" data-base62-sha1="5uJNeWUuaNLSQjhscQi14xA5D7S" width="690" height="440" data-dominant-color="474747"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">707×451 11.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
But<br>
When you run it locally, you will see the torch allclose function says they are the same.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d00bf893ce65544426db9dee028324f72c66dfc4.png" data-download-href="https://discuss.pytorch.org/uploads/default/d00bf893ce65544426db9dee028324f72c66dfc4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d00bf893ce65544426db9dee028324f72c66dfc4_2_690x378.png" alt="image" data-base62-sha1="tGsXCryy8DR2rp4y2mS4SAGPMNu" width="690" height="378" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d00bf893ce65544426db9dee028324f72c66dfc4_2_690x378.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d00bf893ce65544426db9dee028324f72c66dfc4_2_1035x567.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d00bf893ce65544426db9dee028324f72c66dfc4_2_1380x756.png 2x" data-dominant-color="2B2F37"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1930×1058 142 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>These small numerical mismatches are usually expected due to the limited numerical precision. Are you checking your manual gradient calculation with the built-in <code>gradcheck</code> util.?</p>

JonathanSum: <p>But the problem is this only happens on colab, not in the local. This is weird.<br>
For the gradcheck function, according to the doc, it said it also uses allclose() function too, " The check between numerical and analytical gradients uses <a href="https://pytorch.org/docs/stable/generated/torch.allclose.html#torch.allclose" rel="noopener nofollow ugc"><code>allclose()</code></a>."</p>
<p>gradcheck: <a href="https://pytorch.org/docs/stable/generated/torch.autograd.gradcheck.html" class="inline-onebox" rel="noopener nofollow ugc">torch.autograd.gradcheck — PyTorch 1.12 documentation</a></p>