Autoencoder very weird loss spikes when training

username: BrigandRo
cooked: <p><strong>Introduction:</strong><br>
I am trying to make an autoencoder learn 32 features like position, velocity, etc in 32 time steps =&gt; 32x32 ‘image’.<br>
For this I just made a simple linear model that uses in every layer the Tanh function with an encoder and a decoder that are symmetric.<br>
During training I added my own version of dropout for just the input. (in the future I will use the nn.Dropout)</p>
<p><strong>Problem:</strong><br>
I get large spikes in loss function “sqrt(MSE)” at irregular intervals. (Batch_Size = 6000)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c5e553b96780ccd03faa5280d21fee7946565903.png" data-download-href="https://discuss.pytorch.org/uploads/default/c5e553b96780ccd03faa5280d21fee7946565903" title="1000"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5e553b96780ccd03faa5280d21fee7946565903_2_690x454.png" alt="1000" data-base62-sha1="seFolcwj78M6e44DcjlpTqjdrs7" width="690" height="454" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5e553b96780ccd03faa5280d21fee7946565903_2_690x454.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5e553b96780ccd03faa5280d21fee7946565903_2_1035x681.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/5/c5e553b96780ccd03faa5280d21fee7946565903.png 2x" data-dominant-color="FCF8F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1000</span><span class="informations">1080×711 65.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What I have tried: (small test, 1000 epochs max)</p>
<ol>
<li>clip_grad_norm_(model.parameters(), max_norm = 0.5).</li>
<li>ReLu and ELU activation function</li>
<li>Batch = N / 2 (I wanted to do N but the memory of my gpu was not enough)</li>
<li>Not adding noise or dropout (the noise/dropout I think helps but does not solve the problem)<br>
Nothing has worked.</li>
</ol>
<p>Can someone explain to me why this happens and how to fix it?<br>
Thanks a lot in advance, I really do appreciate.</p>
<p>Jupyter notebook:</p><aside class="onebox googledrive" data-onebox-src="https://drive.google.com/file/d/1wjjyFH-X-S7lEWAYcou6aIhwP4Nwa5Pt/view?usp=sharing">
  <header class="source">

      <a href="https://drive.google.com/file/d/1wjjyFH-X-S7lEWAYcou6aIhwP4Nwa5Pt/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">drive.google.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://drive.google.com/file/d/1wjjyFH-X-S7lEWAYcou6aIhwP4Nwa5Pt/view?usp=sharing" target="_blank" rel="noopener nofollow ugc"><span class="googledocs-onebox-logo g-drive-logo"></span></a>



<h3><a href="https://drive.google.com/file/d/1wjjyFH-X-S7lEWAYcou6aIhwP4Nwa5Pt/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">notebook.ipynb</a></h3>

<p>Google Drive file.</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Edit: Added code for getting <a href="https://i.stack.imgur.com/empKN.png" rel="noopener nofollow ugc">gradient’s norm over epochs graph</a> (without noise/dropout)</p>