Confused by shape of gradient on VGG16

Aditya1: <p>I am slightly confused by the shape of the gradient after the backward pass on a VGG16 Network.</p>
<p>The input dimension of the tensor is  <code>[1, 3, 224, 224]</code> , but on backward pass the gradient is  <code>[1 , 64, 224, 224]</code> , which corresponds to the output after the first convolution layer in the network.</p>
<p>Shouldn’t the gradient be the same as the size of the input?</p>

ptrblck: <p>How did you check the gradient shape? Could you post the code snippet you were using?<br>
If you want to create gradients for the input, you would have to set the <code>requires_grad</code> attribute of the <code>input</code> tensor to <code>True</code> and could check the gradient via <code>print(input.grad)</code> after the <code>backward</code> call.<br>
The gradient would have the same shape as the input in this case.</p>

ptrblck: <p><code>self.gradients</code> isn’t defined in the code snippet so I guess you have used some hooks to create it?</p>

Aditya1: <pre><code class="lang-auto">  def generate_gradients(self, input_image, target_class):
        import pdb; pdb.set_trace()
        input_image = input_image.cuda()
        input_image.requires_grad = True
        model_output = self.model(input_image)
        # Zero gradients
        self.model.zero_grad()
        # Target for backprop
        one_hot_output = torch.FloatTensor(1, model_output.size()[-1]).zero_()
        one_hot_output[0][target_class] = 1
        one_hot_output = one_hot_output.cuda()
        # Backward pass
        model_output.backward(gradient=one_hot_output, retain_graph = True)
        # Convert Pytorch variable to numpy array
        # [0] to get rid of the first channel (1,3,224,224)
        gradients_as_arr = self.gradients.data.detach().cpu().numpy()[0]
        return gradients_as_arr
</code></pre>
<p>Yes, I am using hooks to create it. When I try to set <code>requires_grad = True</code>, I get <code>RuntimeError: you can only change requires_grad flags of leaf variables.</code> The code is a bit large to put here so I am not able to put a completely reproducible example. My intention is to do guided back-propagation, i believe for this the gradient needs to flow all the back to the inputs so I can get the gradient in the shape of the input (correct me if I am wrong), do I need to set <code>required_grad = True</code> at the time of training then?</p>

ptrblck: <p>The error message would be raised, if <code>input_image</code> is not a leaf variable, i.e. if it was created by another operation. If that’s the case, it might already require gradients and you should be able to access them via <code>imput_image.grad</code>.</p>
<p>Since the current <code>self.gradients</code> shape is different than the <code>input_image.shape</code>, it seems you are registering the hook on a different parameter or layer.</p>

Aditya1: <p>I tried some little more debugging, now <code>input_image.grad</code> returns a <code>[1, 3, 224, 224]</code> tensor as expected and does not throw an error about the input not having grads, however  <code>self.gradients.data</code> is still <code>[1, 64, 224, 224]</code>.</p>
<pre><code class="lang-auto">    def hook_layers(self):
        def hook_function(module, grad_in, grad_out):
            self.gradients = grad_in[0]
        first_layer = list(self.model.features_1._modules.items())[0][1][0]
        first_layer.register_backward_hook(hook_function)
</code></pre>
<p>This is how I am hooking into the layers, as you can see the hook is at the first layer itself, so ideally speaking the shape flowing into the first layer should be equal to <code>[1, 3, 224, 224]</code> and not <code>[1, 64, 224, 224]</code> right?</p>

ptrblck: <p>I would recommend not to use <code>register_backward_hook</code>, as it might fail as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_backward_hook">docs</a>.<br>
Use <code>parameter.register_hook</code> or grab the <code>.grad</code> attribute directly.</p>

Aditya1: <p>Got it, but will <code>register_backward_hook</code> affect the <code>grad_in</code> of the module? I am correctly placing the hook in the first layer where it should be, the gradient flowing in should be the shape of the image <code>[1, 3, 224, 224]</code> but both <code>grad_in</code> and <code>grad_out</code> are <code>[1, 64, 224, 224]</code> any tips for this?</p>

ptrblck: <aside class="quote no-group" data-username="Aditya1" data-post="9" data-topic="94007">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aditya1/48/18016_2.png" class="avatar"> Aditya1:</div>
<blockquote>
<p>any tips for this?</p>
</blockquote>
</aside>
<p>Since it’s a knows issue, as described in the docs, my recommendation would be to use <code>register_hook</code> instead.</p>

Aditya1: <p>Thanks for your reply. Alternatively, since you said I can directly grab the gradient, I can also assign <code>self.gradient</code> in my code to the gradient of the input itself?</p>
<p>Also thanks a lot for your patience!</p>

ptrblck: <p>Yes, that should be possible.<br>
E.g. you could try to use:</p>
<pre><code class="lang-python">self.gradient = input.grad
</code></pre>
<p>Use <code>input.grad.clone()</code> if you want to assign a copy to <code>self.gradient</code>.</p>