Werid Training Loss Curve

Finspire13: <p>Hi, I am training a simple LSTM for regression task and got following training loss curve.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/af84210c66d3a2e87a38f419f248fd79999d8c1a.png" data-download-href="https://discuss.pytorch.org/uploads/default/af84210c66d3a2e87a38f419f248fd79999d8c1a" title="Screenshot from 2017-10-18 21-05-42"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/af84210c66d3a2e87a38f419f248fd79999d8c1a_2_690x393.png" alt="Screenshot from 2017-10-18 21-05-42" data-base62-sha1="p2GDQbYwpovVJe8NaDhB1aAFkOS" width="690" height="393" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/af84210c66d3a2e87a38f419f248fd79999d8c1a_2_690x393.png, https://discuss.pytorch.org/uploads/default/original/2X/a/af84210c66d3a2e87a38f419f248fd79999d8c1a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/a/af84210c66d3a2e87a38f419f248fd79999d8c1a.png 2x" data-dominant-color="FDFCFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2017-10-18 21-05-42</span><span class="informations">701×400 11.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>While training, I logged the MSE computed on whole dataset at each epoch. The weird thing is that there is a flat region in the training curve. It seems that my network got stuck and learned nothing from epoch 2 to epoch 10. Then it got a ‘Aha moment’ at around epoch 11 and the loss dropped rapidly.</p>
<p>This really puzzled me. Any idea why this happened?</p>
<p>For code, please check <a href="https://github.com/Finspire13/Surgery-Skill-Assessment-Start" rel="noopener nofollow ugc">here</a>.</p>

ptrblck: <p>Your model might get into a saddle point, where the gradient is pretty low.<br>
After epoch 10 is could escape the saddle point and gained some speed.</p>
<p>This is a good illustration:<br>
          <a href="http://www.offconvex.org/assets/saddle/escapesmall.png" target="_blank" rel="nofollow noopener">
            <img src="http://www.offconvex.org/assets/saddle/escapesmall.png" width="459" height="368">
          </a>
<br>
Taken from: <a href="http://www.offconvex.org/2016/03/22/saddlepoints/" rel="nofollow noopener">http://www.offconvex.org/2016/03/22/saddlepoints/</a></p>

Finspire13: <p>Yeah, this makes sense. But I tried to repeat the training process several times and kept getting exactly similar ‘flat region’. Since dataset are randomly shuffled by DataLoader, it shouldn’t get into the saddle point every time, right?<br>
Or is this phenomenon caused by some intrinsic nature of the model / dataset used?</p>

mrTsjolder: <p>From my experience, this is actually pretty typical behaviour for an LSTM training curve. My intuition for this phenomena is as follows:</p>
<ol>
<li>the LSTM learns to predict the “average” signal by setting the bias parameters (first plateau)</li>
<li>the LSTM figures out how to use the (forward and recurrent) weight parameters to actually solve the task (second plateau)</li>
</ol>
<p>This can be interpreted as a saddle point (where the function is much steeper in bias dimensions than in the other dimensions).</p>