Understanding of Backward Hook

basingse: <p>I recently got to know about <code>register_bcakward_hook</code> and <code>register_forward_hook</code> for <code>nn.Module</code>.</p>
<p>I have some queries about <code>register_backward_hook</code>. This is the sample code I am using</p>
<pre data-code-wrap="python3"><code class="lang-python3">class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(3, 4)
        self.relu1 = nn.ReLU()
        self.fc2 = nn.Linear(4, 1)

    def forward(self, x):
        o = self.fc1 (x)
        o = self.relu1 (o)
        o = self.fc2 (o)
        return o


forward_values = {}
backward_values = {}

# Define the forward hook function
def hook_fn_forward(module, inp, out):
    forward_values[module] = {}
    forward_values[module]["input"] = inp
    forward_values[module]["output"] = out
    
def hook_fn_backward(module, inp_grad, out_grad):
    backward_values[module] = {}
    backward_values[module]["input"] = inp_grad
    backward_values[module]["output"] = out_grad


model = Model()

modules = model.named_children()
for name, module in modules:
    module.register_forward_hook(hook_fn_forward)
    module.register_backward_hook(hook_fn_backward)

# batch size of 1 -&gt; shape = (1,3)
x = torch.tensor([[1.0, 1.0, 1.0]])
o = model(x)
o.backward()
</code></pre>
<p>I was checking the gradients of input and output for layer <code>self.fc2</code> (last layer). The output looks like this</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/8/28aa0c9a9b2c65991e6dee3ada8a3623844fa5d0.png" data-download-href="https://discuss.pytorch.org/uploads/default/28aa0c9a9b2c65991e6dee3ada8a3623844fa5d0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/28aa0c9a9b2c65991e6dee3ada8a3623844fa5d0_2_690x158.png" alt="image" data-base62-sha1="5NJqvMeOrgqebR6gbUHanNqC8ne" width="690" height="158" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/28aa0c9a9b2c65991e6dee3ada8a3623844fa5d0_2_690x158.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/28aa0c9a9b2c65991e6dee3ada8a3623844fa5d0_2_1035x237.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/8/28aa0c9a9b2c65991e6dee3ada8a3623844fa5d0.png 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1112×256 23 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Output gradient <code>1</code> makes sense. Why is the input gradient is <code>tuple</code> and why it has 3 values?<br>
Can anyone explain what these depict?</p>
<p>I am using PyTorch version <code>1.7.1</code></p>

albanD: <p>Hi,</p>
<p>Unfortunately the Module backward hooks have been broken forever for such “complex” model.<br>
If you’re using a recent version of pytorch, you can use the “full” versions <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=register_full_backward#torch.nn.Module.register_full_backward_hook" class="inline-onebox">Module — PyTorch 1.9.0 documentation</a> that will have the expected behavior.</p>

basingse: <p>Thanks. I updated to 1.9.0 and it works as expected.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/3/9369f7303d576823f368a324137e363e4e442025.png" data-download-href="https://discuss.pytorch.org/uploads/default/9369f7303d576823f368a324137e363e4e442025" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/9369f7303d576823f368a324137e363e4e442025_2_690x105.png" alt="image" data-base62-sha1="l25dxobZUtJ1uwWgekzMKuSAwrr" width="690" height="105" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/9369f7303d576823f368a324137e363e4e442025_2_690x105.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/9369f7303d576823f368a324137e363e4e442025_2_1035x157.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/3/9369f7303d576823f368a324137e363e4e442025.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1101×168 17.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>