How to go about finding cause of loss spike

divinho: <p>I’m curious to hear strategies about how to approach this problem which I think is a common one:</p>
<p>How do you go about figuring out what part of the model or loss function is causing a rare loss spike?</p>
<p>Pytorch has detect_anomaly , but that slows down training massively. One could add isfinite() checks to the forward pass, but that won’t detect stuff going wrong in the backward pass. One could also add hooks for the backward pass, but that’s a lot of effort.</p>
<p>What other approaches might be possible?</p>

J_Johnson: <p>You can make use of gradient clipping.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.clip_grad_norm_.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.utils.clip_grad_norm_.html</a></p>
<p>If it’s an RL problem, you might also try clipping the objective function.</p>
<p><a href="https://spinningup.openai.com/en/latest/algorithms/ppo.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://spinningup.openai.com/en/latest/algorithms/ppo.html</a></p>

divinho: <p>This reply does not answer the question of how to find the cause.</p>

J_Johnson: <p>As stated in direct message, absent gradient clipping, gradients get smaller the further up the model you go in the backward pass. It’s a consequence of how calculating the derivatives work.</p>
<p>When they get small enough, certain optimizers, such as RMSProp and Adam, which certain calculations from the gradients in the denominator, can cause the step change to jump when the gradient change approaches zero. And so it is likely that your first layer in the model is what produces the issue.</p>
<p>If you want to see what is what is occurring, you can print the average of the gradients of each layer.</p>
<pre><code class="lang-auto">#after calculating loss and running loss.backward() and optimizer.step()
for param in model.parameters():
    print(torch.mean(param.grad))
</code></pre>
<p>Also see here:</p>
<aside class="onebox stackexchange" data-onebox-src="https://datascience.stackexchange.com/questions/25024/strange-behavior-with-adam-optimizer-when-training-for-too-long">
  <header class="source">

      <a href="https://datascience.stackexchange.com/questions/25024/strange-behavior-with-adam-optimizer-when-training-for-too-long" target="_blank" rel="noopener nofollow ugc">datascience.stackexchange.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://datascience.stackexchange.com/users/42255/bai-li" target="_blank" rel="noopener nofollow ugc">
    <img alt="Bai Li" src="https://www.gravatar.com/avatar/9c4e43d1c358143a5cc388a586391142?s=256&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://datascience.stackexchange.com/questions/25024/strange-behavior-with-adam-optimizer-when-training-for-too-long" target="_blank" rel="noopener nofollow ugc">Strange behavior with Adam optimizer when training for too long</a>
</h4>

<div class="tags">
  <strong>perceptron, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://datascience.stackexchange.com/users/42255/bai-li" target="_blank" rel="noopener nofollow ugc">
    Bai Li
  </a>
  on <a href="https://datascience.stackexchange.com/questions/25024/strange-behavior-with-adam-optimizer-when-training-for-too-long" target="_blank" rel="noopener nofollow ugc">09:03PM - 22 Nov 17 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


divinho: <p>I was hoping to start a discussion. Thank you for making me realize this is not the right place.</p>