Loss: inf & Parameters: nan - Why?

username: mayurat22
cooked: <p>I am training a simple polynomial Model w2 * t_u ** 2 + w1 * t_u + b.</p>
<p>Implementation Details</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cb0e9b65904a361947e8edaa2263c338283db66.png" data-download-href="https://discuss.pytorch.org/uploads/default/6cb0e9b65904a361947e8edaa2263c338283db66" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cb0e9b65904a361947e8edaa2263c338283db66_2_609x500.png" alt="image" data-base62-sha1="fvwBgmD5W4wlD6Wu8zGJtWKjqho" width="609" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cb0e9b65904a361947e8edaa2263c338283db66_2_609x500.png, https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cb0e9b65904a361947e8edaa2263c338283db66.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cb0e9b65904a361947e8edaa2263c338283db66.png 2x" data-dominant-color="F0F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">874×717 54.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>After few epochs, the loss tends to inf and parameters move to nan as in below image</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edfaf3438c9428a7566ac94803ac57b423c185e1.png" data-download-href="https://discuss.pytorch.org/uploads/default/edfaf3438c9428a7566ac94803ac57b423c185e1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edfaf3438c9428a7566ac94803ac57b423c185e1_2_577x500.png" alt="image" data-base62-sha1="xXgOLX3wJiDTAu5AFEnmw7CRoI1" width="577" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edfaf3438c9428a7566ac94803ac57b423c185e1_2_577x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edfaf3438c9428a7566ac94803ac57b423c185e1_2_865x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/d/edfaf3438c9428a7566ac94803ac57b423c185e1.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">872×755 81.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can anyone explain why it happens and how to avoid it ?</p>

username: mayurat22
cooked: <p>I think the reason may be,<br>
Few reasons</p>
<ol>
<li>Parameters updates are too large and its overshooting the gradient. The optimization process is unstable, it diverges instead of converging to a minimum.</li>
<li>Since weights and bias are at extreme end after first epoch, it continues to fluctuate causing loss to move to inf. Solution is to normalize the X to [-1, 1] or [0,1].</li>
<li>I was using SGD, which is sensitive to scaling and makes the parameters to overshoot. If i use Adam optimizer i.e. adaptive learning rate the model tends to find convergence even without scaling down the X to [-1, 1] or [0, 1].</li>
</ol>