How do hooks work?

username: theairbend3r
cooked: <p>I’m trying to implement the GradCam paper which uses the gradient information flowing into the last convolutional layer of the CNN to assign importance values to each neuron for a particular decision of interest.</p>
<p>The paper says -<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc0aa3cbafdaee1535f96715af0c3a8da4af677d.png" data-download-href="https://discuss.pytorch.org/uploads/default/cc0aa3cbafdaee1535f96715af0c3a8da4af677d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc0aa3cbafdaee1535f96715af0c3a8da4af677d.png" alt="image" data-base62-sha1="t72cnCZ99DiIX3WyHDM2irV2YsB" width="690" height="473" data-dominant-color="EBEBEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">745×511 62.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>1.</strong> The first step to implementing GradCam would be obtaining the gradient wrt to the activation maps. My question is how do it obtain it? From what I’ve read online, we need hooks to get it. My question is: how is the value obtained using hooks different from the <code>cnnlayer.weight.grad</code>?</p>
<hr>
<p><strong>2.</strong> Now, let’s say we need hooks. How do we actually use hooks here? The docs say that there we can use the hooks on either tensors or modules. My question is, how are they different? Does a module refer to a single layer such as <code>nn.Conv2d(...)</code> or does it refer to the entire class shown below.</p>
<pre><code class="lang-auto">class CNNModel(nn.Module):
    def __init__(self):
        super(CNNModel, self).__init__()
        self.cnn1 = nn.Conv2d()
        self.cnn2 = nn.Conv2d()
        
        # cnn_out gives a (1,n) output where n is the number of classes.
        # We use cnn_out instead of a fc layer.
 
        self.cnn_out = nn.Conv2d()

    def forward(self, x):
        x = self.cnn1(x)
        x = self.cnn2(x)
        out = self.cnn_out(x)

        return out

</code></pre>
<p>ie. does module refer to <code>CNNModel</code> or <code>nn.Conv2d()</code>?</p>
<hr>
<p><strong>3.</strong> We can use hooks on a tensor by way of <code>register_hook()</code>. How do we use it? From what I’ve read online, we need to use hooks on a tensor to implement GradCAM. Let’s say my setup if the following -</p>
<p>I have a custom image classifier as follows. How do I use <code>register_hook</code> to obtain the gradients of <code>self.cnn2</code>? I’ve seen examples where <code>register_hook</code> is used inside the <code>forward()</code> method (<a href="https://medium.com/@stepanulyanin/implementing-grad-cam-in-pytorch-ea0937c31e82" rel="noopener nofollow ugc">Source</a>). Sometime, people make a separate class out of it (<a href="https://books.google.co.in/books?id=udCwDwAAQBAJ&amp;pg=PA120&amp;lpg=PA120&amp;dq=register+hook+vs+register+forward+hook+pytorch&amp;source=bl&amp;ots=ImnBxhJXD7&amp;sig=ACfU3U0q9wVNk1QWUj0c5YcMXpitFdoiNA&amp;hl=en&amp;sa=X&amp;ved=2ahUKEwiiraHjtdPpAhUS4nMBHRo-DeA4ChDoATACegQICRAB#v=onepage&amp;q=register%20hook%20vs%20register%20forward%20hook%20pytorch&amp;f=false" rel="noopener nofollow ugc">Source (pg132)</a>).</p>
<p>What is the correct way of doing it? Assume the trained model is stored in variable <code>model</code> as shown below.</p>
<pre><code class="lang-auto">class CNNModel(nn.Module):
    def __init__(self):
        super(CNNModel, self).__init__()
        self.cnn1 = nn.Conv2d()
        self.cnn2 = nn.Conv2d()
        
        # cnn_out gives a (1,n) output where n is the number of classes.
        # We use cnn_out instead of a fc layer.
 
        self.cnn_out = nn.Conv2d()

    def forward(self, x):
        x = self.cnn1(x)
        x = self.cnn2(x)

        # we return a (1,10) output because we have 10 classes in MNIST.
        out = self.cnn_out(x)

        return out


# Let's assume this model has been trained on MNIST (28 x 28 images)
model = CNNModel()

# Obtain prediction for a single image 
pred = model(single_test_img)
</code></pre>
<p>What do we do after we obtain the <code>pred</code>. Can you please show that by completing the pseudo-code I’ve written. Like, where does <code>register_hook</code> (define and run it) and everything fit in while we try to obtain the gradient mentioned above (at the top)?</p>
<hr>
<p><strong>4.</strong> Can we use <code>register_backward_hook()</code> or <code>register_forward_hook()</code> somehow? What I mean is, can these 2 functionalities by used to get the same output as <code>register_hook()</code>?</p>
<p>Can you give a small example of how we use <code>register_forward_hook()</code> using the <code>CNNModel</code> class I defined above?</p>
<p>I’ve also seen the docs say that <code>register_backward_hook()</code> is buggy, so we are advised not to use it.</p>
<hr>
<p>I am confused by how hooks work. Hooks in PyTorch do not receive a lot of attention (docs or blogs). Please feel free to answer individual chunks, I’ll follow up in the comments. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: albanD
cooked: <p>Hi,</p>
<ol>
<li>
<p>Hooks can allow you to get gradients wrt to intermediate results in the forward, not just the weights/biases.</p>
</li>
<li>
<p>For backward hooks, you should only use the Tensor version right now (the nn.Module version is not working properly at the moment).<br>
To you a module hook for the output of cnn1 for example, you would modify your forward method as follow:</p>
</li>
</ol>
<pre><code class="lang-auto"># Defined somewhere else
def my_hook(grad):
    # Do what you want with the grad


# In your model definition
    def forward(self, x):
        x = self.cnn1(x)
        x.register_hook(my_hook)
        x = self.cnn2(x)
        out = self.cnn_out(x)

        return out
</code></pre>
<ol start="3">
<li>As in the example above.</li>
</ol>

username: theairbend3r
cooked: <p>Hey <a class="mention" href="/u/alband">@albanD</a>, thanks for taking the time to answer. I have a few follow up questions.</p>
<ol>
<li>
<p>What do you mean when you say intermediate results? Gradients are calculated for a weight  (<code>dL/dW</code>). By intermediate results do you mean <code>dL/dA</code> where <code>A</code> is the activation for applied for a layer which has a weight <code>W</code>? Can show what these intermediate results are in the following image please?</p>
</li>
<li>
<p>How is <code>register_hook</code> applied on a tensor different from <code>register_backward_hook()</code> applied on a <code>nn.Module</code>? What is <code>nn.Module</code> here - a layer such as <code>nn.Conv2d</code> or an entire class like <code>CNNModel()</code>? I have this confusion because one example in the doc has this <code>net.conv2.register_backward_hook(printnorm)</code> (used on a layer conv2) but modules are supposed to be classes that have a <code>.forward()</code> function. Or is my understanding of a <code>nn.Module</code> incorrect?</p>
</li>
</ol>
<p>Can you highlight where <code>register_hook()</code>, <code>register_forward_hook()</code>, <code>register_backward_hook()</code> apply in the attached image please?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d34f014ede7d5bd594815a880ab509a53c96024.png" data-download-href="https://discuss.pytorch.org/uploads/default/4d34f014ede7d5bd594815a880ab509a53c96024" title="Sample Neural Net"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d34f014ede7d5bd594815a880ab509a53c96024_2_690x251.png" alt="Sample Neural Net" data-base62-sha1="b10bpdsoYXQwL9fqqbfTrYRjguE" width="690" height="251" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d34f014ede7d5bd594815a880ab509a53c96024_2_690x251.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d34f014ede7d5bd594815a880ab509a53c96024_2_1035x376.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d34f014ede7d5bd594815a880ab509a53c96024.png 2x" data-dominant-color="E4CDDA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Sample Neural Net</span><span class="informations">1170×426 56 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In the code example you wrote, <code>register_hook()</code> is applied after <code>self.cnn1</code>. So, the hook (it’s like a callback function right?) can now access the gradients flowing into <code>self.cnn1</code> during backprop?</p>

username: albanD
cooked: <p>Hi,</p>
<ol>
<li>
<p>In your image, the intermediate results will be z and a. And the correpsonding gradients for them are <code>dL/dz</code> and <code>dL/da</code>.<br>
Note that in your image, all the boxes correspond to functions. The purple function is what you define in your forward and the red functions are our backward Node.<br>
The Tensor containing values are on the arrows: the purple ones are the intermediate results you have when you write your forward.<br>
The green are what is saved for backward (you don’t see these).<br>
And the red are the gradients flowing back, containing the gradient for each intermediate result.<br>
You can see a Tensor hook as observing one of these red edges.</p>
</li>
<li>
<p>nn.Module builds on top of the autograd to have a nice neural network library.<br>
So a hook on the nn.Module is trying (and failing at the moment) to get the intermediate gradient for all the inputs/outputs of it’s forward function.<br>
Hooking on a Tensor is using directly the autograd construct. And will give you the gradient computed for that specific Tensor.</p>
</li>
</ol>