RuntimeError: hook 'backward' has changed the size of value

ProGamerGov: <p>I’m trying to modify <code>grad_input</code> in my backwards hook function, but I am getting the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "st.py", line 260, in &lt;module&gt;
    optimizer.step(feval)
  File "/usr/local/lib/python2.7/dist-packages/torch/optim/lbfgs.py", line 101, in step
    orig_loss = closure()
  File "st.py", line 255, in feval
    loss.backward()
  File "/usr/local/lib/python2.7/dist-packages/torch/autograd/variable.py", line 167, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)
  File "/usr/local/lib/python2.7/dist-packages/torch/autograd/__init__.py", line 99, in backward
    variables, grad_variables, retain_graph)
RuntimeError: hook 'backward' has changed the size of value
</code></pre>
<pre><code class="lang-auto">    def backward(self, grad_input, grad_output):

        gradInputTuple = ()
        gradOutput = grad_output[0] 
        for input in grad_input:
          print(str(input.size()) + " " + str(self.target.size()))
          if input.nelement() == self.target.nelement():
             self.gradInput = mse_loss2(input, self.target)
          if self.normalize == 'True':
             self.gradInput = self.gradInput.div(torch.norm(self.gradInput, 1) + 1e-8) # Normalize Gradients
          self.gradInput = self.gradInput * self.strength
          self.gradInput = self.gradInput + gradOutput
          print("gradInput.size():")
          print(str(self.gradInput.size()))
          gradInputTuple = list(gradInputTuple)
          gradInputTuple.append(self.gradInput)
          gradInputTuple = tuple(gradInputTuple)
        else:
          self.target = gradOutput
        return gradInputTuple #self.gradInput
</code></pre>
<p>I am trying to replicate this Lua/Torch code:</p>
<pre><code class="lang-auto">function ContentLoss:updateGradInput(input, gradOutput)
  if self.mode == 'loss' then
    if input:nElement() == self.target:nElement() then
      self.gradInput = self.crit:backward(input, self.target)
    end
    if self.normalize then
      self.gradInput:div(torch.norm(self.gradInput, 1) + 1e-8)
    end
    self.gradInput:mul(self.strength)
    self.gradInput:add(gradOutput)
  else
    self.gradInput:resizeAs(gradOutput):copy(gradOutput)
  end
  return self.gradInput
end
</code></pre>
<p>What am I doing wrong here?</p>

jphreid: <p>I do have the same error message, but for a slightly different problem. In the end, did you fix it?</p>
<p>In my case, I would like opt.step() to use complete different gradient values.<br>
‘new_gradients’ is a dictionary for which the keys() have the same name as that the model parameters. For example,</p>
<p>block.fc1.weight<br>
block.fc1.bias<br>
etc…</p>
<p>Thus, we have that</p>
<pre><code>	f = model(x)

	loss = criterion(f, y)
	
	for (key, w) in model.named_parameters():
		w.register_hook(lambda grad: new_gradients[key])

	optimizer.zero_grad()
	loss.backward()
</code></pre>
<p>There is an error at the very last line,</p>
<p>RuntimeError: hook ‘lambda’ has changed the size of value</p>
<p>Help would be appreciated.</p>

albanD: <p>Hi,</p>
<p>Your error comes from the fact that one of your lambda returns a tensor which is not of the same size as what it got as input. You can replace the lambda by a real function and add some extra prints. In this case, I think the batch size is handled in a way you don’t expect: the gradient of the hook for a tensor of size HxW will be BxHxW and is then summed over the batch dimension if I remember correctly.</p>
<p>For your use case, if you just want to set the gradients, you don’t actually need to run a backward pass:</p>
<pre><code class="lang-auto">optimizer.zero_grad()
with torch.no_grad():
  for (key, w) in model.named_parameters():
    w.grad.copy_(new_gradients[key])

optimizer.step()
</code></pre>

HaFred: <p>Hi <a class="mention" href="/u/progamergov">@ProGamerGov</a>, sorry to bring this up. I ran into the same <code>RuntimeError: hook has changed the size</code> which makes me confused about the <code>grad_input</code> tuples. Correct me if I am wrong, the tuple should be <code>(act_grad, weight_grad, bias_grad)</code> right?<br>
So what was wrong with your previous code? Did you solve it? Thank you very much.</p>

ProGamerGov: <p>I never solved that particular issue, but I would recommend using a custom autograd function if you want to change things in the backward pass: <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html</a></p>