Does high learning rate produces NaN?

vymao: <p>I noticed that sometimes at high learning rate, my model produces NaN randomly in the test output:</p>
<pre><code class="lang-auto">ValueError: Input contains NaN, infinity or a value too large for dtype('float32').
</code></pre>
<p>That is to say, I am able to train it properly without error, but randomly during evaluating on the test set, the prediction from the model contains NaN.</p>
<p>Is this known to happen, and if so, why?</p>

nicofish: <p>Tou would have to specify what kind of model you are using. To answer as there could be some other cause.</p>
<p>However, from what you are saying it does seem like the learning rate is responsible for this. The reason this occurs is because of the way gradient descent works.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3bc08026c1335b23506eca473d3a03e7c9bbea6.png" alt="image" data-base62-sha1="yMaSACfxlincfsnL8MUR0ILXoOi" width="393" height="321"><br>
image from here:</p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/42332058/gradient-descent-thetas-not-converging">
  <header class="source">

      <a href="https://stackoverflow.com/questions/42332058/gradient-descent-thetas-not-converging" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/7589729/mark-hodges" target="_blank" rel="noopener nofollow ugc">
    <img alt="Mark Hodges" src="https://www.gravatar.com/avatar/3551cea93ee740cc4529ca55d545f521?s=256&amp;d=identicon&amp;r=PG&amp;f=y&amp;so-version=2" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/42332058/gradient-descent-thetas-not-converging" target="_blank" rel="noopener nofollow ugc">Gradient Descent: thetas not converging</a>
</h4>

<div class="tags">
  <strong>matrix, machine-learning, octave, linear-regression</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/7589729/mark-hodges" target="_blank" rel="noopener nofollow ugc">
    Mark Hodges
  </a>
  on <a href="https://stackoverflow.com/questions/42332058/gradient-descent-thetas-not-converging" target="_blank" rel="noopener nofollow ugc">07:49PM - 19 Feb 17 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Essentially your larger learning rate is causing you to overshoot and you are artificially causing something similar to the exploding gradient problem (which you can read more about here: <a href="https://machinelearningmastery.com/exploding-gradients-in-neural-networks/" rel="noopener nofollow ugc">https://machinelearningmastery.com/exploding-gradients-in-neural-networks/</a>)</p>
<p>In exploding gradient problem errors accumulate as a result of having a deep network and result in large updates which in turn produce infinite values or NaN’s. In your case your large updates are directly a result of having a large learning rate forcing a large update which causes your NaNs. Though if you have a large network it could be a result of both things combined.</p>