Interpreting output of backward hook

hamedB: <p>Hi</p>
<p>I have a question about the output of backward hook on the Mixed_6e.branch1x1.conv layer of pytorch pre-trained inception_v3.<br>
(Mixed_6e): InceptionC(<br>
(branch1x1): BasicConv2d(<br>
(conv): Conv2d(768, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)<br>
(bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
…<br>
)<br>
I registered forward and backward hook on this layer. As a result:<br>
The shape of module_out in forward hook is [5,192,17,17] which I thinks is the activation maps of 5 images passing through this layer. However, the output of backward is not by any means clear for me. And I really appreciate it if anyone could please explain to me.<br>
The out put of module_in of backward hook:<br>
len of module_in  3<br>
shape of inside module_in torch.Size([5, 768, 17, 17])<br>
shape of inside module_in torch.Size([192, 768, 1, 1])<br>
None found for Gradient<br>
The output of module_out of hook:<br>
shape of inside module_out  torch.Size([5, 192, 17, 17])</p>

ptrblck: <p>In case you were using <code>register_backward_hook</code>, note that its behavior was broken for some modules which is why it’s deprecated now. Use <code>register_full_backward_hook</code> now. The <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=backward_hook#torch.nn.Module.register_full_backward_hook">docs</a> explain the expected inputs and outputs.</p>

hamedB: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for your response. I have read the link that you sent and I have a follow up question. What is the difference between grad_out as a result of registering <code>register_full_backward_hook</code> on a module such as conv2d and A.register_hook() (A as a output of a module that is called activation)? Are both equal but come from different modules?<br>
My purpose is computing the gradient respect to the outputs of filters in a specific layer. Which one of these functions, <code>register_full_backward_hook</code> or register_hook on a tensor do you suggest?</p>
<p>Thank you in advance for your attention given to my problem.</p>

ptrblck: <p><code>register_full_backward_hook</code> will give you the grad inputs and outputs, which are passed into this module and forwarded to the next module.<br>
<a href="https://pytorch.org/docs/stable/autograd.html?highlight=register_hook#torch.Tensor.register_hook"><code>register_hook</code></a> is used on parameters and will return the gradient w.r.t this parameter.</p>

hamedB: <p>Thank you again <a class="mention" href="/u/ptrblck">@ptrblck</a>  for your explanations. Sorry, the inputs and outputs that you mentioned means input activations to this module and the output activations of this module? And the parameter that you also mentioned means weights and bias of this module? Am I right? Because, based on the code snippet provided by <a class="mention" href="/u/alband">@albanD</a>  in this discussion <a href="https://discuss.pytorch.org/t/how-do-hooks-work/83109" class="inline-onebox">How do hooks work?</a>, the register_hook function is registered on the output activations of the module.</p>
<p>Thank you so much.</p>

ptrblck: <aside class="quote no-group" data-username="hamedB" data-post="5" data-topic="114357">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hamedb/48/35498_2.png" class="avatar"> hamedB:</div>
<blockquote>
<p>Sorry, the inputs and outputs that you mentioned means input activations to this module and the output activations of this module?</p>
</blockquote>
</aside>
<p>No, the gradient inputs and outputs created during the backward pass. The figures in <a href="https://cs231n.github.io/optimization-2/#grad">this post</a> give you a better idea of the forward activations and the gradients during the backward pass.</p>
<aside class="quote no-group" data-username="hamedB" data-post="5" data-topic="114357">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hamedb/48/35498_2.png" class="avatar"> hamedB:</div>
<blockquote>
<p>And the parameter that you also mentioned means weights and bias of this module? Am I right? Because, based on the code snippet provided by <a class="mention" href="/u/alband">@albanD</a> in this discussion <a href="https://discuss.pytorch.org/t/how-do-hooks-work/83109">How do hooks work? </a>, the register_hook function is registered on the output activations of the module.</p>
</blockquote>
</aside>
<p>Yes, I meant the parameters of the module such as <code>weight</code> and <code>bias</code>, but <code>register_hook</code> can be used generally on tensors, not only parameters.</p>