About gradients and gradient clipping on LSTM!

username: Laura_Montalvo
cooked: <p>Hi there, I’m implementing a custom LSTM with 3 hidden layers by using LSTMCells. I know that with RNN’s we must be careful about gradient exploding so we need to use gradient clipping technique but does this apply to LSTM too?</p>
<p>Also, how could I see gradient values so I know which value to set in gradient clipping? And how to know when the gradient is exploding?</p>
<p>Thanks!!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Laura_Montalvo" data-post="1" data-topic="131332">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/laura_montalvo/48/34770_2.png" class="avatar"> Laura_Montalvo:</div>
<blockquote>
<p>I know that with RNN’s we must be careful about gradient exploding so we need to use gradient clipping technique but does this apply to LSTM too?</p>
</blockquote>
</aside>
<p>In <a href="https://arxiv.org/abs/1409.3215">Sequence to Sequence Learning with Neural Networks</a> (which might be considered a bit old by now) the authors claim:</p>
<blockquote>
<p>Although LSTMs tend to not suffer from the vanishing gradient problem, they can have exploding gradients. Thus we enforced a hard constraint on the norm of the gradient [10,25] by scaling it when its norm exceeded a threshold. …</p>
</blockquote>
<p>So I would assume that LSTMs can also suffer from exploding gradients.</p>
<aside class="quote no-group" data-username="Laura_Montalvo" data-post="1" data-topic="131332">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/laura_montalvo/48/34770_2.png" class="avatar"> Laura_Montalvo:</div>
<blockquote>
<p>Also, how could I see gradient values so I know which value to set in gradient clipping?</p>
</blockquote>
</aside>
<p>You could register hooks for each parameter and check the gradient norm, min/max values etc. via:</p>
<pre data-code-wrap="python"><code class="lang-python">for param in model.parameters():
    param.register_hook(your_check_method)
</code></pre>
<aside class="quote no-group" data-username="Laura_Montalvo" data-post="1" data-topic="131332">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/laura_montalvo/48/34770_2.png" class="avatar"> Laura_Montalvo:</div>
<blockquote>
<p>And how to know when the gradient is exploding?</p>
</blockquote>
</aside>
<p>You could use the same approach as used in the linked paper, but you might also find more recent papers with more “modern” approaches.</p>

username: Vrushank98
cooked: <p>This code is to visualize the “Vanishing gradient Problem”. If you want to visualize when and where the gradients are exploding, you can play around with the top value of plt.ylim.</p>
<pre><code class="lang-auto">plt.ylim(bottom = -0.001, top = 0.02)
</code></pre>
<pre><code class="lang-auto">
import torch
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.lines import Line2D
@torch.no_grad()
def plot_grad_flow(named_params, path):

    avg_grads, max_grads, layers = [], [], []
    plt.figure(figsize = ((10,20)))
    
    for n, p in named_params:
        
        if (p.requires_grad) and ('bias' not in n):
            
            layers.append(n)
            avg_grads.append(p.grad.abs().mean())
            max_grads.append(p.grad.abs().max())
        
    plt.bar(np.arange(len(max_grads)), max_grads, alpha = 0.1, lw = 1, color = 'c')
    plt.bar(np.arange(len(max_grads)), avg_grads, alpha = 0.1, lw = 1, color = 'b')
    plt.hlines(0, 0, len(avg_grads) + 1, lw = 2, color = 'k')
    plt.xticks(range(0, len(avg_grads), 1), layers, rotation = 'vertical')
    plt.xlim(left = 0, right = len(avg_grads))
    plt.ylim(bottom = -0.001, top = 0.02) #Zoom into the lower gradient regions
    plt.xlabel('Layers')
    plt.ylabel('Average Gradients')
    plt.title('Gradient Flow')
    plt.grid(True)
    plt.legend([
        Line2D([0], [0], color = 'c', lw = 4),
        Line2D([0], [0], color = 'b', lw = 4),
        Line2D([0], [0], color = 'k', lw = 4),
        ],
        ['max-gradient', 'mean-gradient','zero-gradient'])
    
    plt.savefig(path)
    plt.close()
</code></pre>
<blockquote>
<p>Output:</p>
</blockquote>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/057abf197055cfc2d9513479bbffae2218324488.png" data-download-href="https://discuss.pytorch.org/uploads/default/057abf197055cfc2d9513479bbffae2218324488" title="grad (1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/057abf197055cfc2d9513479bbffae2218324488_2_250x500.png" alt="grad (1)" data-base62-sha1="MtmY70UQxYF7mwk282tPNoVOkg" width="250" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/057abf197055cfc2d9513479bbffae2218324488_2_250x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/057abf197055cfc2d9513479bbffae2218324488_2_375x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/057abf197055cfc2d9513479bbffae2218324488_2_500x1000.png 2x" data-dominant-color="F0F8FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grad (1)</span><span class="informations">720×1440 24.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: googlebot
cooked: <p>you can try LayerNorm instead of clipping, though output space becomes a bit odd with non-independent dimensions</p>