Correctly indexing the input tuple in a backward hook function

numpee: <p>Hi,</p>
<p>I’ve noticed something peculiar when using PyTorch backward hook functions. Perhaps this was designed so on purpose, but it makes using backward hooks a bit unintuitive.</p>
<p>First off, a backward hook function would look something like this:</p>
<pre><code class="lang-python">def hook_fn(module, inputs, outputs):
    # something here
</code></pre>
<p>where the function’s parameters are the module name, inputs (tuple), and outputs (tuple).</p>
<p><strong>This is the part that confuses me.</strong> The parameter <code>inputs</code> is a tuple, which contains gradients of the module, but the order of elements seems a bit… random. Not stochastic, but random. To elaborate, if we register this function on a <code>nn.Linear</code> layer with bias parameters, this tuple becomes:</p>
<pre><code class="lang-auto">(grad_bias, grad_preactivation, grad_weight) 
</code></pre>
<p>So why is the <code>grad_preactivation</code> (gradients for activation of the previous layer) between <code>grad_bias</code> and <code>grad_weight</code>?</p>
<p>What makes it worse is that for convolutional layers, this order is shuffled. For <code>nn.Conv2d</code> layers, the order seems to be:</p>
<pre><code class="lang-auto">(grad_preactivation, grad_weight, grad_bias)
</code></pre>
<p>Is there any documentation that explains this in depth (I can’t seem to find much <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=10" title=":cry:" class="emoji" alt=":cry:">), and are there any better ways to intercept gradients in the backward pass? For example, say I want to the modify the gradients of the weights in each layer, where some layers are <code>nn.Conv2d</code> and others are <code>nn.Linear</code>, and the <code>bias</code> could be either true of false. Then, would I need to hard-code my hook function to account for all possibilities, or is there a more elegant solution?</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

albanD: <p>Hi,</p>
<p>Are you using <code>Module.register_backward_hook()</code>?<br>
If so, this is a known issue and if you check the latest doc, they have been deprecated in favor of <code>Module.register_full_backward_hook()</code> that has the exact same API and doc (but actually do what the doc says they’re supposed to do <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"> ).</p>

numpee: <p>Hi,<br>
Thanks for the reply.<br>
I just tried out <code>register_full_backward_hook()</code>, and it looks like only the pre and post activation gradients are accessed. Thanks for your answer <img src="https://discuss.pytorch.org/images/emoji/apple/grinning_face_with_smiling_eyes.png?v=10" title=":grinning_face_with_smiling_eyes:" class="emoji" alt=":grinning_face_with_smiling_eyes:"> :</p>
<p>On a side note:<br>
I assume, for the gradients of model parameters, there is no need to access it mid-backprop? I guess these could be accessed by iterating over <code>model.named_parameters()</code> after the full backward pass.</p>

albanD: <p>Yes you can access them afterwards.<br>
If you want to access them in a hook, you can directly register a Tensor hook on these parameters directly: <code>mod.weight.register_hook(hook_fn)</code>.</p>

numpee: <p>Ok. Thanks for the thorough reply <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"></p>