How to get gradients wrt inputs for intermediate layers?

username: francois_rousseau
cooked: <p>Hi there,</p>
<p>I’d like to compute the gradient wrt inputs for several layers inside a network. So far, I’ve built several intermediate models to compute the gradients of the network output wrt input using autograd.grad.</p>
<p>Can it be done more efficiently using hooks?</p>
<p>Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Using autograd.grad sounds is the best solution in general.</p>
<p>Not sure how you would use hooks. If the final result you want is a linear combination of all these gradient, then you might be able to use the linearity of the gradient to do that. But not sure. You will need to share more information about what you’re trying to do here/</p>

username: francois_rousseau
cooked: <p>Hi,</p>
<p>Thanks for the reply.</p>
<p>I’d like to compute the gradient of each layer, so basically I do a loop over the layers and call autograd. (the layers in the code below are stored in a list)</p>
<pre><code class="lang-auto">  var_x = torch.autograd.Variable(x, requires_grad=True)  
  
  for l in range(n):  
    mx = layers[l](var_x)
    
    gradients = torch.autograd.grad(outputs=mx, inputs=var_x,
                                      grad_outputs=torch.ones(mx.size()).to(device), 
                            create_graph=True, retain_graph=True, only_inputs=True)[0]
      
    norm_gradient_velocity[l] = torch.max( torch.norm( gradients.view(gradients.shape[0],-1), p=np.inf, dim=1 ) )

</code></pre>
<p>Because it’s long during training to use such loop, I was wondering if it could be done using hooks.</p>

username: albanD
cooked: <p>Ok, so not linear at all. So you wont be able to do that <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
In that case, I don’t think there is any other way to do this.<br>
Hooks would only allow you to call a function during the backward pass. But you still have to do all these autograd.grad.</p>

username: smiler96
cooked: <p>sorry to bother you. have you achieved to get gradients of the intermediate layers w.r.t the input data with hook in pytorch?<br>
Now I also met this issue and i want calculate the F-norm of Jacobi Matrix w.r.t  the every layers and input data in a network .<br>
Thanks for your reply!</p>

username: hamedB
cooked: <p>Hi<br>
I also face such challenge and I have tried several solutions that came in my mind.<br>
If you find the solution, I would greatly appreciate it if you could please share it with us.</p>

username: hamedB
cooked: <p>I have found a solution.<br>
My problem is to start the gradient process from a specific filter in a specific layer with respect to the input image.<br>
First, I have assigned a forward hook to all conv. layers in order to keep output of each filter (activation map). Second, I have assigned a backward hook to all layers to keep their gradient in the backward pass.<br>
Third, I fed the model with an input image to obtain all activation maps. Then, I called the backward function on the norm of one of the activation maps (remind that each activation map of each filter is similar to the output of loss). Finally, I have all gradients in a list variable where the last element of this list is the gradient of input image which should have the size of (1,3,224,224) for instance.</p>

username: sgmath12
cooked: <p>Did you use same functino for regestering the forward_hook and backward_hook?</p>
<p>After I assigned a backward hook, I get tuple for intermediate layer’s output.</p>

username: Avani_Gupta
cooked: <p>Hello,<br>
I also want to get gradient of output wrt activation of intermediate layer.<br>
I tried backward_hook for that layer, it returns tuple of what? could you explain? is it gradient of output wrt weight and bias of that layer? I want wrt activation(output) of that layer.</p>

username: shubham_vats
cooked: <p>Hi, were you able to find a solution to this? I’m trying to do the same but rather get None as output.</p>

username: AlphaBetaGamma96
cooked: <p><a class="mention" href="/u/shubham_vats">@shubham_vats</a>  You can either use <code>.retain_grad()</code> on the input value but a more consistent way of getting this value is to use hooks. You can use a <a href="https://pytorch.org/docs/stable/generated/torch.nn.modules.module.register_module_full_backward_hook.html" rel="noopener nofollow ugc">full backward hook</a> (not a backward hook as that’s deprecated and gives the wrong result). Also, check there’s no use of in-place operations (for example with ReLU) nor use of <code>.detach()</code> which breaks the gradient of your computational graph.</p>
<p><a class="mention" href="/u/avani_gupta">@Avani_Gupta</a> For an explanation of what <code>grad_input</code> and <code>grad_output</code> mean there’s an explanation on the forums <a href="https://discuss.pytorch.org/t/exact-meaning-of-grad-input-and-grad-output/14186/8">here</a>. The <code>grad_output</code> term is the gradient of the loss with respect to the output of that <code>nn.Module</code>.</p>

username: shubham_vats
cooked: <p><a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> thank you. I’ll look into this. I’m fairly new to pytorch and hooks didn’t just come naturally to me. I was hoping to use <a href="https://pytorch.org/vision/stable/feature_extraction.html" rel="noopener nofollow ugc">create_feature_extractor</a> to get those intermediate outputs and do something with them but apparently, the grad attribute is what I need for which retain_grad should be helpful. I also wanted to avoid hooks because as per my understanding I would have to retrain the model to use them? I am using load_state_dict to load a pretrained model for now.</p>

username: AlphaBetaGamma96
cooked: <p><a class="mention" href="/u/shubham_vats">@shubham_vats</a> You won’t have to retrain the model, you can just attach the hooks to the model perform a forward pass then call <code>loss.backward()</code>. That will trigger the full backward hooks and you’ll get your derivatives (although it will be for all samples so you’ll need to take a mean over that to get the correct shape)</p>

username: shubham_vats
cooked: <p>Thank you for getting back. I was able to get it to work without the hooks using retain_grad and backward. But, it’s always good to know an alternative approach. Thank you for all the help!</p>