Hessian of output with respect to inputs

username: kkudus
cooked: <p>Is there any way to find the full hessian matrix of the inputs with respect to the outputs in a simple feed-forward fully-connected network? I know you can call .backwards twice to get the hessian-vector product, but is there an efficient way (without using a loop) to find the full matrix? Thanks in advance!</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can find an example in this gist of how to implement such function: <a href="https://gist.github.com/apaszke/226abdf867c4e9d6698bd198f3b45fb7" rel="nofollow noopener">https://gist.github.com/apaszke/226abdf867c4e9d6698bd198f3b45fb7</a></p>

username: kkudus
cooked: <p>Thanks for your quick response <a class="mention" href="/u/alband">@albanD</a>.</p>
<p>I have been implementing something very similar to the example you’ve posted here. What I’m wondering, however, is if there is some way to find the full hessian of the output with respect to the inputs with a single call to torch.autograd.grad?</p>

username: albanD
cooked: <p>I’m afraid not.<br>
This is a limitation of backpropagation and automatic differentiation in general. It does not allow you to compute a Jacobian, but a vector Jacobian product. This you need multiple calls to re-recreate the full Jacobian. (similarly for the Hessian).</p>

username: kkudus
cooked: <p>Thanks for confirming that this is the case!</p>

username: Li_Zheng
cooked: <p>Hi,</p>
<p>I’m using the same code by Adam to calculate the Hessian of a simple feed-forward fully-connected network’s outputs with respect to inputs. But I got a tensor full of zeros as hessian. Do you have any idea? Any help you could offer would be greatly appreciated!</p>
<p>Thanks,<br>
Li</p>

username: albanD
cooked: <p>Hi,</p>
<p>This is most likely because your function is Linear and so the Jacobian is constant and the hessian is 0 no?<br>
Can you share your function?</p>

username: Felix_Asanger
cooked: <p>Hi,</p>
<p>I am working on the same problem, I used the code of the link: <a href="https://gist.github.com/apaszke/226abdf867c4e9d6698bd198f3b45fb7" rel="nofollow noopener">https://gist.github.com/apaszke/226abdf867c4e9d6698bd198f3b45fb7</a>, which you posted earlier at this post.</p>
<p>I am also using a straight forward neuronal network as a toy example:</p>
<pre><code class="lang-auto">def jacobian(y, x, create_graph=True):                                                               
    jac = []                                                                                          
    flat_y = y.reshape(-1)
    #print("flat_y: ", flat_y)  

    grad_y = torch.zeros_like(flat_y)   
    #print("grad_y: ", grad_y)    

    for i in range(len(flat_y)):                                                                      
        grad_y[i] = 1.                                                                                
        grad_x, = torch.autograd.grad(flat_y, x, grad_y, retain_graph=True, create_graph=create_graph)
        #grad_x, = torch.autograd.grad(flat_y, x,retain_graph=True, create_graph=create_graph)
        jac.append(grad_x.reshape(x.shape))                                                           
        grad_y[i] = 0.                                                                                
    return torch.stack(jac).reshape(y.shape + x.shape)                                                
   
                                                                                                      
def hessian(y, x):                                                                                    
    return jacobian(jacobian(y, x, create_graph=True), x) 

if __name__ == '__main__':
    x = torch.ones(4, requires_grad=True) 
    y = Variable(Tensor([1.]), requires_grad=True)

    model = nn.Linear(4, 1)
    model.weight.data.fill_(0.5)
    model.bias.data.fill_(0.5)

    loss = torch.sum((y - model(x))**2)
    pred = model(x)
 
    print("type", loss)
    print("type(loss)", type(loss))
    print("loss.shape", loss.shape)
    print(hessian(loss, x))
    print("\n") 

    print("pred", pred[0])
    print("type(pred)", type(pred[0]))
    print("pred.shape", pred[0].shape)
    print("\n")   
    print(hessian(pred[0], x))     


</code></pre>
<p>And get the following result and error:</p>
<p>loss:  tensor(2.2500, grad_fn=&lt; SumBackward0 &gt;)<br>
type(loss):  &lt;class ‘torch.Tensor’&gt;<br>
loss.shape:  torch.Size([])<br>
tensor([[0.5000, 0.5000, 0.5000, 0.5000],<br>
[0.5000, 0.5000, 0.5000, 0.5000],<br>
[0.5000, 0.5000, 0.5000, 0.5000],<br>
[0.5000, 0.5000, 0.5000, 0.5000]], grad_fn=&lt; ViewBackward &gt;)</p>
<p>pred:  tensor(2.5000, grad_fn=&lt; SelectBackward &gt;)<br>
type(pred):  &lt;class ‘torch.Tensor’&gt;<br>
pred.shape:  torch.Size([])</p>
<p>Traceback (most recent call last):<br>
File “test_hfm.py”, line 58, in <br>
print(hessian(pred[0], x))<br>
File “test_hfm.py”, line 27, in hessian<br>
return jacobian(jacobian(y, x, create_graph=True), x)<br>
File “test_hfm.py”, line 19, in jacobian<br>
grad_x, = torch.autograd.grad(flat_y, x, grad_y, retain_graph=True, create_graph=create_graph)<br>
File “/Users/felixasanger/anaconda3/envs/dil_bmw/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 158, in grad<br>
inputs, allow_unused)<br>
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.</p>
<p>Why does everything work fine for the loss, but not if I try it with the prediction? Is there any difference?<br>
Any help is highly apperciated, I read a lot about this problem, but I just can´t solve it.</p>
<p>Best regards and thanks a lot,</p>
<p>Felix</p>

username: albanD
cooked: <p>Hi,</p>
<p>Same answer as above <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
The function that gives your prediction given the weights is linear (litterally just a Linear layer). And so the Jacobian is constant and so independent of the weights.<br>
Depending on the cases (internal implementation weirdness) we either return a matrix full of 0 and a message saying that the two are independent (meaning that the gradient is 0 everywhere).</p>
<p>Your loss function add a quadratic term and so makes the hessian non-zero. So it works fine.</p>

username: Felix_Asanger
cooked: <p>Hi,</p>
<p>thanks for your answer, but maybe you misunderstood my question. I didn´t ask why the Hessian would be 0. I was asking why I get an error, when I want to compute the Hessian of my prediction (NOT the loss) with respect to my input x.</p>
<p>I don´t get the error when I do pred = model(x)**1, then it is possible to compute the Hessian of pred with respect to the input. Do you have any idea why it only works when I add the **1 ?</p>
<p>Best,<br>
Felix</p>

username: albanD
cooked: <p>Ho, the answer is that because of how pytorch works, a gradient of 0 can be represented in 3 ways:</p>
<ul>
<li>A Tensor full of 0s</li>
<li>None being returned by autograd.grad(…, allow_unused=True) or the in the .grad field after .backward()</li>
<li>An error with autograd.grad(…, allow_unused=False)</li>
</ul>
<p>Note that we have no way to differentiate “is independent of” and “has a gradient of 0” and both can be represented in the three ways above.</p>

username: jarvico
cooked: <p>Hi albanD,</p>
<p>Actually I am also trying to compute hessian of a CNN model’s prediction output w.r.to input vector.</p>
<p>Let’s assume simple task of MNIST classification. We have 1x1x28x28 input vector and 1x10 output tensor</p>
<p>Now my aim is to compute the hessian of a specific class’s pred value w.r.to input .</p>
<p>I want to calculate Jacobian in the dimension of 784x1. For example, calculating Jacobian for classs 2’s pred value should yield a tensor of dim 784x1.</p>
<p>And calculating Hessian for class 2’s pred value should yield a matrix of dimension 784 x 784.</p>
<p>I tried to use the code you have provided but I couldn’t manage to change to the one I need.</p>
<p>Can you please suggest a way to calculate a Hessian matrix with dimension 784 x 784 for this specific case?Assuming I need to calculate hessian for prediction output value of a specific class.</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you use the functions in <code>torch.autograd.functional</code> to do this, then you will get a hessian of size <code>input_size + input_size</code>. So in your case <code>1x1x28x28x1x1x28x28</code>. But you can use <code>.view()</code> to make that <code>784x784</code> if that’s what you want. You just collapse the dimensions as if you were using a function with input size <code>784</code>.</p>

username: jarvico
cooked: <p>Hi,</p>
<p>I used below code to compute hessian matrix for a specific class. It seems to work but I need to double check. In below example I calculated the hessian matrix of output prediction for class 2 with respect to input tensor. I got an 4x4 hessian matrix in this way.</p>
<p>Can you please comment?</p>
<pre><code class="lang-auto">import torch

def jacobian(y, x, create_graph=False):
    jac = []
    flat_y = y.reshape(-1)
    grad_y = torch.zeros_like(flat_y)
    for i in range(len(flat_y)):
        grad_y[i] = 1.
        grad_x, = torch.autograd.grad(flat_y, x, grad_y, retain_graph=True, create_graph=create_graph)
        jac.append(grad_x.reshape(x.shape))
        grad_y[i] = 0.
    return torch.stack(jac).reshape(y.shape + x.shape)

def jacobian2(y, x, t, create_graph=False):
    flat_y = y.reshape(-1)
    grad_y = torch.zeros_like(flat_y)
    grad_y[t] = 1.
    grad_x, = torch.autograd.grad(flat_y, x, grad_y, retain_graph=True, create_graph=create_graph,allow_unused=True)
    return grad_x


def hessian(y, x, t):
    return jacobian(jacobian2(y, x, t, create_graph=True), x)


def f(x):
    return x * x * torch.arange(4, dtype=torch.float)


x = torch.ones(4, requires_grad=True) * 2

print(jacobian2(f(x), x, 2))
a = hessian(f(x), x, 2)
print(a)```</code></pre>

username: albanD
cooked: <p>The function has 4 inputs. So you get a hessian of size 4x4. That looks expected to me.</p>