Check gradient flow in network

alwynmathew: <p>Is there a easy way to check that the gradient flow is proper in the network? Or is it broke somewhere in the network?</p>
<p>Will this <a href="https://github.com/pytorch/pytorch/blob/master/torch/autograd/gradcheck.py#L139" rel="nofollow noopener">gradcheck</a> be useful? How do I use it? Example?</p>

tom: <p>Gradcheck checks a single function (or a composition) for correctness, eg when you are implementing new functions and derivatives.<br>
For your application, which sounds more like “I have a network, where does funny business occur”, Adam Paszke’s script to find bad gradients in the computational graph might be a better starting point. Check out the thread below.</p>
<p>Best regards</p>
<p>Thomas</p>
<aside class="quote quote-modified" data-post="3" data-topic="9019">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-check-for-vanishing-exploding-gradients/9019/3">How to check for vanishing/exploding gradients</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    I think the canonical reference for finding bad gradients is this snippet by Adam Paszke: 


It checks for NaN (by using x!=x if and only if x is NaN) and very large gradients, but you could easily adapt is_bad_grad to best fit your purpose. 
Best regards 
Thomas
  </blockquote>
</aside>


alwynmathew: <p>Thanks <a class="mention" href="/u/tom">@tom</a></p>
<p>But when I use it in my network:</p>
<pre><code class="lang-auto">...
import bad_grad_viz as bad_grad
...
loss_G , loss_D = model(data_1, data_2)
get_dot = bad_grad.register_hooks(loss_1)
model.optimizer_network.zero_grad()
loss_G.backward()
loss_D.backward()
dot = get_dot()
dot.save('/path/to/dir/tmp.dot')
</code></pre>
<p>I get this error from <a href="https://gist.github.com/apaszke/f93a377244be9bfcb96d3547b9bc424d" rel="nofollow noopener">bad_grad_viz.py</a>:</p>
<pre><code class="lang-auto">grad_output = grad_output.data
AttributeError: 'NoneType' object had no attribute 'data'
</code></pre>
<p>More info:</p>
<p><code>model</code> is a GAN network.</p>
<pre><code class="lang-auto">print(type(loss_G))
&gt;&gt;&gt; &lt;class 'torch.autograd.variable.Variable'&gt;
print(type(loss_D))
&gt;&gt;&gt; &lt;class 'torch.autograd.variable.Variable'&gt;
print(loss_G.requires_grad)
&gt;&gt;&gt; True
print(loss_D.requires_grad)
&gt;&gt;&gt; True
</code></pre>
<p>Any suggestion <a class="mention" href="/u/apaszke">@apaszke</a> ?</p>

tom: <p>What if you test that condition and return False?</p>

alwynmathew: <p>Try 1:</p>
<pre><code class="lang-auto">def is_bad_grad(grad_output):
        if grad_output.requires_grad == True:
            print('grad_ouput have grad')
        grad_output = grad_output.data
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">if grad_output.requires_grad == True:
AttributeError: 'NoneType' object has no attribute 'requires_grad'
</code></pre>
<p>Try 2:</p>
<pre><code class="lang-auto">def is_bad_grad(grad_output):
        if grad_output.requires_grad == False:
            print('grad_ouput doesnt have grad')
        grad_output = grad_output.data
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">grad_ouput doesnt have grad
grad_ouput doesnt have grad
grad_ouput doesnt have grad
grad_ouput doesnt have grad
grad_ouput doesnt have grad
grad_ouput doesnt have grad
grad_ouput doesnt have grad
grad_ouput doesnt have grad
grad_ouput doesnt have grad
grad_ouput doesnt have grad
grad_ouput doesnt have grad
grad_ouput doesnt have grad
grad_ouput doesnt have grad
grad_ouput doesnt have grad
grad_ouput doesnt have grad
Traceback (most recent call last):
grad_ouput doesnt have grad
grad_ouput doesnt have grad
if grad_output.requires_grad == False:
AttributeError: 'NoneType' object has no attribute 'requires_grad'
</code></pre>
<p>Please give more details, so that i can debug this issue.</p>

tom: <p>It seems that something you makes your output not require grads as much as one would expect, this could happen due to networks being in <code>.eval()</code> instead of <code>.training()</code>, or setting requires_grad = False manually or volatile or something entirely different…<br>
If you had a minimal demo of how it happens, it would be easier to find out why it is not working.</p>
<p>Best regards</p>
<p>Thomas</p>

RoshanRane: <p>I use a simple trick. I record the average gradients per layer in every training iteration and then plotting them at the end. If the average gradients are zero in the initial layers of the network then probably your network is too deep for the gradient to flow.<br>
So this is how I do it -</p>
<ol>
<li>This API to plot</li>
</ol>
<pre><code class="lang-auto">def plot_grad_flow(named_parameters):
    ave_grads = []
    layers = []
    for n, p in named_parameters:
        if(p.requires_grad) and ("bias" not in n):
            layers.append(n)
            ave_grads.append(p.grad.abs().mean())
    plt.plot(ave_grads, alpha=0.3, color="b")
    plt.hlines(0, 0, len(ave_grads)+1, linewidth=1, color="k" )
    plt.xticks(range(0,len(ave_grads), 1), layers, rotation="vertical")
    plt.xlim(xmin=0, xmax=len(ave_grads))
    plt.xlabel("Layers")
    plt.ylabel("average gradient")
    plt.title("Gradient flow")
    plt.grid(True)
</code></pre>
<ol start="2">
<li>plug this API after the loss.backward() during the training as follows -</li>
</ol>
<pre><code class="lang-auto">loss = self.criterion(outputs, labels)  
loss.backward()
plot_grad_flow(model.named_parameters())
</code></pre>
<ol start="3">
<li>Results<br>
bad gradient flow -<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e4b19586eb7b68d94ba02fa9159f141c9f12e106.png" alt="bad_grad_flow" data-base62-sha1="wD7kNm4I9AeSZdu24xPDG5jbKBw" width="402" height="376"><br>
kinda good gradient flow -<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5af139d68e14dbdd9745952a744f105458d2caa7.png" alt="kinda_good_grad_flow" data-base62-sha1="cYvNnN8cHtu8uu39USo2ELwZSn5" width="402" height="376"><br>
good gradient flow -<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/5/545a14ac2527f5539c2b14e92bb7bd4a1f0bdefa.png" alt="mnist_grad_11layers" data-base62-sha1="c2d6ueddtZtxM0sRiOSXK6ZT826" width="403" height="356"><br>
PS : This may not be the best way though…</li>
</ol>

Abhishek_Das: <p>This is my training iteration. I need to check the gradient. I incorporated what you suggested.<br>
I am getting this error.</p>
<p>grad_output = grad_output.data**</p>
<p>atributeError: ‘NoneType’ object has no attribute ‘data’**</p>
<p>The code snippet:<br>
for i,batch in enumerate(train_loader):<br>
if (i%100)==0:<br>
print(epoch,i,len(train_loader))</p>
<h1>print(i,batch)</h1>
<pre><code>          #Clear the GRADIENT
          optimizer.zero_grad()
          #INPUT
          imgs,ques,ans = batch
          #imgs,ques,ans = imgs.cuda(),ques.cuda(),ans.cuda()
          imgs,ques,ans = imgs.to(device),ques.to(device),ans.to(device)
          #FORWARD PASS
          #OUTPUT
          
          ansout,queryout = net(imgs,ques,ans)
          
          #LOSS CALCULATED
          #Here answer is my target thus ".data" is used
          queryloss = loss(queryout, ansout.data)
          #Here query is my target 
          ansloss = loss(ansout, queryout.data)
          get_dot = check_grad.register_hooks(ansloss)
          #MSE Added
          #print(ansloss)
          sum_ans_train += ansloss.item()
          queryloss.backward(retain_graph=True)
        
          #BACKPROPAGATION
          ansloss.backward()
          dot = get_dot()
          dot.save('/home/Abhishek/plots/tmp.dot')</code></pre>

RoshanRane: <p>A much better implementation of the function <img src="https://discuss.pytorch.org/images/emoji/apple/sunny.png?v=9" title=":sunny:" class="emoji" alt=":sunny:"></p>
<pre><code class="lang-auto">
def plot_grad_flow(named_parameters):
    '''Plots the gradients flowing through different layers in the net during training.
    Can be used for checking for possible gradient vanishing / exploding problems.
    
    Usage: Plug this function in Trainer class after loss.backwards() as 
    "plot_grad_flow(self.model.named_parameters())" to visualize the gradient flow'''
    ave_grads = []
    max_grads= []
    layers = []
    for n, p in named_parameters:
        if(p.requires_grad) and ("bias" not in n):
            layers.append(n)
            ave_grads.append(p.grad.abs().mean())
            max_grads.append(p.grad.abs().max())
    plt.bar(np.arange(len(max_grads)), max_grads, alpha=0.1, lw=1, color="c")
    plt.bar(np.arange(len(max_grads)), ave_grads, alpha=0.1, lw=1, color="b")
    plt.hlines(0, 0, len(ave_grads)+1, lw=2, color="k" )
    plt.xticks(range(0,len(ave_grads), 1), layers, rotation="vertical")
    plt.xlim(left=0, right=len(ave_grads))
    plt.ylim(bottom = -0.001, top=0.02) # zoom in on the lower gradient regions
    plt.xlabel("Layers")
    plt.ylabel("average gradient")
    plt.title("Gradient flow")
    plt.grid(True)
    plt.legend([Line2D([0], [0], color="c", lw=4),
                Line2D([0], [0], color="b", lw=4),
                Line2D([0], [0], color="k", lw=4)], ['max-gradient', 'mean-gradient', 'zero-gradient'])
</code></pre>

trdavidson: <p>simply wanted to comment that this^ is a wonderfully written function to inspect gradients -&gt; highly recommend.</p>

Aaron_Smith: <p>I have a peculiar problem. Thanks to the function provided above I was able to see the gradient flow but to my dismay, the graphs show the gradient decreasing from right side to left side, which is as God intended. But, in my case the graphs show the gradient decreasing from left side to right side, which is clearly wrong, albeit, I will be highly grateful if somebody can tell me what’s going on with the network.</p>
<p>It has a convolutional block followed by an encoder and decoder. The network is fully convolutional.</p>
<p>I will be highly grateful for any help provided.</p>

Cagri_Kaplan: <p>I have a class of VGG16 and I wonder if named_parameters in your function refers to model.parameters()? model is an instance of class VGG16 by the way. If your response is ‘yes’ then I receive an error ‘too many values to unpack (expected 2)’ for command ‘for n, p in model.parameters():’. Do you see the reason?</p>

tom: <p>For any <code>nn.Module</code> instance, <code>m.named_parameters()</code> returns an iterator over pairs of <code>name, parameter</code>, while <code>m.parameters()</code> just returns one for the parameters.<br>
You should be able to use <code>m.named_parameters()</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

Cagri_Kaplan: <p>This helped a lot. Thank you very much.</p>
<p>Best Regards</p>

Zuanazzi: <p>There is a place in heaven for people like you! Any chance pytorch is integrating something alike soon?</p>

sdv4: <p><a class="mention" href="/u/roshanrane">@RoshanRane</a><br>
I used your code for plotting the gradient flow (thank you!), and obtained this output:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0715b5b68039e693e1f8435183f52b1260315d98.png" data-download-href="https://discuss.pytorch.org/uploads/default/0715b5b68039e693e1f8435183f52b1260315d98" title="36%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0715b5b68039e693e1f8435183f52b1260315d98_2_288x250.png" alt="36%20PM" data-base62-sha1="10FR205Lxng4mkP3THzrLf3bD7i" width="288" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0715b5b68039e693e1f8435183f52b1260315d98_2_288x250.png, https://discuss.pytorch.org/uploads/default/optimized/2X/0/0715b5b68039e693e1f8435183f52b1260315d98_2_432x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/0/0715b5b68039e693e1f8435183f52b1260315d98_2_576x500.png 2x" data-dominant-color="EFEFF9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">36%20PM</span><span class="informations">822×712 67.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>With the second implementation I obtain the following output:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/1e22d123be94a949277ba597c1aeb40bbd2cf474.png" data-download-href="https://discuss.pytorch.org/uploads/default/1e22d123be94a949277ba597c1aeb40bbd2cf474" title="15%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/1e22d123be94a949277ba597c1aeb40bbd2cf474_2_450x375.png" alt="15%20PM" data-base62-sha1="4iAUYYuRCUYdJxOLo5ZiLEUrSSw" width="450" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/1e22d123be94a949277ba597c1aeb40bbd2cf474_2_450x375.png, https://discuss.pytorch.org/uploads/default/optimized/2X/1/1e22d123be94a949277ba597c1aeb40bbd2cf474_2_675x562.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/1/1e22d123be94a949277ba597c1aeb40bbd2cf474.png 2x" data-dominant-color="C2DEED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">15%20PM</span><span class="informations">884×736 57.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is for a single layer GRU. I was surprised to see the gradient of the hidden state stay so small. The only thing I can think of as to why this would be the case is because the hidden state is re-initialized with each training example (and thus stays small), while the other gradients accumulate as a result of being connected to learned parameters. Does that seem correct? Is this what a plot of the gradient flow in a single layer GRU should typically look like?</p>
<p>Alternatively, for a 4 layer LSTM, I get the following output:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/242b8fb5ae029a1458a48124c1bde9c0ae594563.png" data-download-href="https://discuss.pytorch.org/uploads/default/242b8fb5ae029a1458a48124c1bde9c0ae594563" title="57%20AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/242b8fb5ae029a1458a48124c1bde9c0ae594563_2_279x250.png" alt="57%20AM" data-base62-sha1="59Ywa0qSMMZ8OapalumxtUMkYhR" width="279" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/242b8fb5ae029a1458a48124c1bde9c0ae594563_2_279x250.png, https://discuss.pytorch.org/uploads/default/optimized/2X/2/242b8fb5ae029a1458a48124c1bde9c0ae594563_2_418x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/2/242b8fb5ae029a1458a48124c1bde9c0ae594563_2_558x500.png 2x" data-dominant-color="F3F3F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">57%20AM</span><span class="informations">802×718 59.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Does that seem correct? Is this what a plot of the gradient flow in a multi-layer layer LSTM should typically look like? The larger gradient values are from the initial epochs. I am not sure when they are so much larger to start with. Thoughts?</p>

isalirezag: <aside class="quote no-group" data-username="RoshanRane" data-post="10" data-topic="15063">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/roshanrane/48/6205_2.png" class="avatar"> RoshanRane:</div>
<blockquote>
<p>Line2D</p>
</blockquote>
</aside>
<p>not sure what is <code>Line2D</code>in your code???</p>

zwacke: <p>Its a function from the matplotlib library, just add this line to the top of your script:</p>
<p>from matplotlib.lines import Line2D</p>

RoshanRane: <p>You can import it as <code>from matplotlib.lines import Line2D</code></p>

xvdp: <p>Just a comment. I’m trying to measure the behaviour of Generative nets, specifically a UNet, so I looked at this, Implemented something in tensorboard and realized that while yes, one does get gradient flow graphs similar to what you show, it isnt quite as straightforwards. The larger the number of weight parameters, the lower the gradient has to be so it does not explode. So you may want to look at the gradients in logscale. Here are 2 representations. The first is similar to the code above, where x:layer number (0 thru 28), y:abs mean gradient (or signed max), z: iteration; uses SummaryWriter.add_histogram_raw() the second x:iteration, y:absmean gradient; using .add_scalars() <img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f8a0dbecee194bbba81c16955b78eec24aae4584.png" alt="image" data-base62-sha1="ztt3cCgAd0HSzmBWnyYdsw5Ypda" width="564" height="187"> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/aaa9c75dbd768a0ae4d2955da736e36a58fabe7c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/aaa9c75dbd768a0ae4d2955da736e36a58fabe7c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/aaa9c75dbd768a0ae4d2955da736e36a58fabe7c_2_690x270.jpeg" alt="image" data-base62-sha1="olKUZSeQrZcLxvZSmLUl8ZdV0ra" width="690" height="270" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/aaa9c75dbd768a0ae4d2955da736e36a58fabe7c_2_690x270.jpeg, https://discuss.pytorch.org/uploads/default/original/2X/a/aaa9c75dbd768a0ae4d2955da736e36a58fabe7c.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/a/aaa9c75dbd768a0ae4d2955da736e36a58fabe7c.jpeg 2x" data-dominant-color="D7D5D9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">757×297 132 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Even though the middle gradients are close to zero, they still flow, and the net learns. The higher values are first and last layer which have 5e2 and 1e3 parameters, whereas the parameters of the center layers are~5e6 to 1e7<br>
I tried different inits and the shape of the gradients always ends up being similar, and the UNet learns well. It seems to me that this U shape is a consequence of the netowrk architecture, and shouldnt be necessarily interpreted as wrong.</p>