How to check for vanishing/exploding gradients

ml9951: <p>I’ve been training a model and have not been getting the results that I expect.  I have a suspicion that it might be due to vanishing/exploding gradients, but would like to verify this somehow.  How might I go about inspecting the gradients of my model to verify that this is in fact the case?</p>

unrahul: <p>I am not sure how to identify/verify exploding gradients, you could try gradient clipping using something like below that will prevent the gradients from going aboard:</p>
<p><code>torch.nn.utils.clip_grad_norm(model.parameters())</code> as shown in:<br>
<a href="https://github.com/pytorch/examples/blob/master/word_language_model/main.py#L138" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/word_language_model/main.py#L138</a> and see if that makes any difference.</p>

tom: <p>I think the canonical reference for finding bad gradients is this snippet by Adam Paszke:</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/apaszke/f93a377244be9bfcb96d3547b9bc424d" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/apaszke/f93a377244be9bfcb96d3547b9bc424d" target="_blank" rel="nofollow noopener">https://gist.github.com/apaszke/f93a377244be9bfcb96d3547b9bc424d</a></h4>
<h5>bad_grad_viz.py</h5>
<pre><code class="Python">from graphviz import Digraph
import torch
from torch.autograd import Variable, Function

def iter_graph(root, callback):
    queue = [root]
    seen = set()
    while queue:
        fn = queue.pop()
        if fn in seen:</code></pre>
This file has been truncated. <a href="https://gist.github.com/apaszke/f93a377244be9bfcb96d3547b9bc424d" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>It checks for NaN (by using <code>x!=x</code> if and only if x is NaN) and very large gradients, but you could easily adapt <code>is_bad_grad</code> to best fit your purpose.</p>
<p>Best regards</p>
<p>Thomas</p>

ml9951: <p>Thanks, I’ll definitely be taking a look into this.  Very helpful.</p>

jel158: <p>In my case, I use this simple code to see values of gradient, particularly for rnn:</p>
<pre><code class="lang-auto">for p,n in zip(rnn.parameters(),rnn._all_weights[0]):
    if n[:6] == 'weight':
        print('===========\ngradient:{}\n----------\n{}'.format(n,p.grad))
</code></pre>

ml9951: <p>That’s exactly what I was looking for.  Thanks!</p>

ianp: <p>This snippet appears not to work with Python 3.7 and PyTorch 1.5.0. Is there a new “canonical” reference?</p>

tom: <p>The new way of getting bad grads is the anomaly mode. But since you asked, I put up a Notebook of Adam’s bad_grad_viz adapted to modern PyTorch at <a href="https://github.com/t-vi/pytorch-tvmisc/blob/master/visualize/bad_grad_viz.ipynb" rel="noopener nofollow ugc">https://github.com/t-vi/pytorch-tvmisc/blob/master/visualize/bad_grad_viz.ipynb</a>.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/264156dd700f718ab2874dd5fdfb27a5e93d1643.png" alt="image" data-base62-sha1="5sq8ea7zegslXJEUkdNhWYQMeQP" width="272" height="377"></p>

saba: <p><a class="mention" href="/u/jel158">@jel158</a>  <a class="mention" href="/u/tom">@tom</a><br>
sorry if I want to see the gradient of my model as follow how I should use the code?</p>
<pre><code class="lang-auto">class Generator994(nn.Module):
    def __init__(self,ngpu,nz,ngf):
        super(Generator994, self).__init__()
        self.ngpu=ngpu
        self.nz=nz
        self.ngf=ngf
        self.l1= nn.Sequential(
            nn.ConvTranspose2d(self.nz, self.ngf * 8, 3, 1, 0, bias=False),
            nn.BatchNorm2d(self.ngf * 8),
            nn.ReLU(True),)
        self.l2=nn.Sequential(nn.ConvTranspose2d(self.ngf * 8, self.ngf * 4, 3, 1, 0, bias=False),
            nn.BatchNorm2d(self.ngf * 4),
            nn.ReLU(True),)
        self.l3=nn.Sequential(nn.ConvTranspose2d( self.ngf * 4, self.ngf * 2, 3, 1, 0, bias=False),
            nn.BatchNorm2d(self.ngf * 2),
            nn.Sigmoid(),)
        self.l4=nn.Sequential(nn.ConvTranspose2d( self.ngf*2, 1, 3, 1, 0, bias=False),nn.Sigmoid()
        )

    def forward(self, input):
        out=self.l1(input)
        out=self.l2(out)
        out=self.l3(out)
        out=self.l4(out)
        return out
</code></pre>

tom: <p>The idea of the linked code is to just run your model as usual to give some function <code>loss</code>.<br>
Then you add the following before and after the backward:</p>
<pre><code class="lang-python">get_dot = register_hooks(loss)
loss.backward()
dot = get_dot()
</code></pre>
<p>and then dot contains a dot graph object that you can display in Jupyter or render.<br>
Note that it might get large fast, but for a convnet like you posted it should be OK.</p>
<p>Then the red items are the ones that are problematic. (You could adapt the criterion by changing <code>is_bad_grad</code>.<br>
The other question to ask is, of course, why you’re having exploding gradient issue and what you would do if you knew “where” is happens.</p>
<p>Best regards</p>
<p>Thomas</p>

saba: <p>I appreciate your explanation. I get this  graph as my gradient flow, do you think the number of layers  should be cut? because the  gradient in the earliest layer are very small. what is the feature of good gradient flow?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/4/14a598045edb336bb0a6a07cfd0bc366f293e58c.jpeg" alt="epoch=042grads" data-base62-sha1="2WEkCe13v34rlJNuJXAJwEIQPO4" width="640" height="480"></p>

Alice_NL: <aside class="quote no-group" data-username="jel158" data-post="5" data-topic="9019">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jel158/48/25608_2.png" class="avatar"> jel158:</div>
<blockquote>
<pre><code class="lang-auto">for p,n in zip(rnn.parameters(),rnn._all_weights[0]):
    if n[:6] == 'weight':
        print('===========\ngradient:{}\n----------\n{}'.format(n,p.grad))
</code></pre>
</blockquote>
</aside>
<p>I get error ‘Net’ object has no attribute ‘all_weights’…</p>

ptrblck: <p><code>_all_weights</code> is an internal attribute of <code>RNNBase</code> and I would not recommend to rely on the usage of these internal objects, as they might change without any deprecation warning.<br>
That being said, what is your exact use case? If you want to filter out some parameters of the RNN you could try to use <code>model.rnn.named_parameters()</code> instead.<br>
Let me know, if this would work for you.</p>

Alice_NL: <p>Dear Piotr,</p>
<p>Oh it is clear then - I do not use RNNBase. I was just looking for the way to see what is happening to the gradients in my network to overall have an idea what is going on in it. If there is a quick pointer you could share, I would be grateful.</p>
<p>Best,<br>
Alice</p>

ptrblck: <p>You could manually check all gradients e.g. via:</p>
<pre><code class="lang-python">for name, param in model.named_parameters():
    print(name, param.grad.norm())
</code></pre>
<p>(or any other stats, if <code>norm</code> is not desired).<br>
However, this approach would be quite limited and more sophisticated algorithms for model interpretability can be applied by e.g. <a href="https://captum.ai/">Captum</a>.<br>
Also, you might want to plot the histograms of the gradients in TensorBoard or any other visualization tool, which should give you more insight into the model training.</p>

ZimoNitrome: <p>How am I supposed to interpret it that anomaly mode is pointing to <code>torch.nn.mse_loss()</code> as the place where the gradient goes bad?</p>
<p>Similarly the graphing code posted by <a class="mention" href="/u/tom">@tom</a> above show red nodes from beginning (after input variable) to end (mse loss function).</p>

tom: <p>You could check the input to mse_loss and its gradients (use out.retain_grad() to keep the gradient).</p>
<p>Best regards</p>
<p>Thomas</p>

ZimoNitrome: <p>Thanks!</p>
<p>You are right, it seems like the weights are updated such that an intermediate tensor becomes <code>nan</code> somewhere in the model.</p>

Aidan_Kehoe: <p>I’ve tried implementing this code roughly as follows:</p>
<pre><code class="lang-auto">loss = self.get_loss(x, x_hat)
get_dot = register_hooks(loss)
loss.backward()
dot = get_dot()
</code></pre>
<p>I’m able to generate graphs for a few batches, however after about 10 batches I receive an assertion error stemming from the fact that a backward gradient function is not in the function dictionary. When printing the size of the dictionary, I notice that the number of functions in the dictionary remains constant for a number of batches, and then drops suddenly. Do you have any idea why <code>register_grad</code> in <code>hook_cb</code> is not adding the full set of functions to the dictionary?<br>
Thanks!</p>

Aidan_Kehoe: <p>I would try batch normalization, it may help the gradient flow reach the early layers before dying out.<br>
Regarding what a good gradient flow looks like, recall that the gradient influences how much the model is able to learn from an instance of data. Thus, a healthy gradient flow should be non-zero (mostly) from the top layer all the way to the input layer. Otherwise, the weights in the earlier layers will not update at all.<br>
I find it more intuitive to examine the graph from right to left since this is actually how one would compute gradients during backpropagation.</p>