Unexpected output while printing gradients of a layer when backward hook is applied on it

puranjay_mishra: <p>Hi everyone!<br>
I’m trying to understand the working of backward hook and thus playing around with it to observe and learn.<br>
I applied a backward hook on a simple covnet. The hook looks something like this:</p>
<pre><code class="lang-auto">def hook_function(m,i,o):
    estimated_gradient = torch.ones_like(i[0])
    estimated_gradient = torch.unsqueeze(estimated_gradient, dim=0)
    return estimated_gradient
</code></pre>
<p>Here I’m return a tensor of 1s as the input gradient for the next layer to use. I print these values after doing the forward propagation (but before doing loss.backwards and optimizer.step)<br>
The code looks something like this -</p>
<pre><code class="lang-auto">#print(list(model.parameters())[6].shape) # I used this to identify the particular layer I want to print
print(list(model.parameters())[6].grad[0])
</code></pre>
<p>This prints the grads of the last layer in my covnet . The code where I apply these hooks on the layers of my model is as follows (I am applying these hooks on all the linear layers of my covnet-</p>
<pre><code class="lang-auto">def register_backward_hook_for_(Model):
    target_modules = []
    print(list(Model.modules()))
    for m in Model.modules():
        if isinstance(m,nn.Linear):
            target_modules.append(m)
    # print(target_modules)
    for modules in target_modules:
        modules.register_full_backward_hook(hook_function)
</code></pre>
<p>Expected output -  I expected these values to be 1, as I did not perform any backward computation (printing it before my loss.backward call)<br>
Actual output - Real valued outputs</p>
<p>What is going wrong? Or am I missing something about the working of these hooks?</p>

ptrblck: <aside class="quote no-group" data-username="puranjay_mishra" data-post="1" data-topic="146225">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/b5ac83/48.png" class="avatar"> puranjay_mishra:</div>
<blockquote>
<p>Expected output - I expected these values to be 1, as I did not perform any backward computation (printing it before my loss.backward call)</p>
</blockquote>
</aside>
<p>I don’t quite understand this statement as backward hooks will be called during the <code>.backward()</code> call.<br>
Could you explain what exactly you are printing? If no <code>backward()</code> call was used, all <code>.grad</code> attributes would be set to <code>None</code>.</p>

puranjay_mishra: <p>Thank you for the reply sir !<br>
Now I get why I’m not able to print 1s as the output of gradients for any particular layer. I did not realize this fact and also that the values of gradients being printed would be the ones stored in these tensors from the previous epoch.<br>
I really appreciate you helping me out, thank you!</p>

ptrblck: <p>Ah OK. Thanks for the update and good to hear the issue seems to be solved! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>