Compute the Hessian matrix of a network

ToastCheng: <p>Hi, I am trying to compute Hessian matrix by calling twice <code>autograd.grad()</code> on a variable.<br>
It works fine in a toy example:</p>
<pre><code class="lang-python">a = torch.FloatTensor([1])
b = torch.FloatTensor([3])

a, b = Variable(a, requires_grad=True), Variable(b, requires_grad=True)

c = a + 3 * b**2
c = c.sum()

grad_b = torch.autograd.grad(c, b, create_graph=True)
grad2_b = torch.autograd.grad(grad_b, b, create_graph=True)

print(grad2_b)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">Variable containing:
 6
[torch.FloatTensor of size 1]
</code></pre>
<p>But here is the question, I want to compute the Hessian of a network,<br>
so I define a function:</p>
<pre><code class="lang-python">def calculate_hessian(loss, model):
    var = model.parameters()
    temp = []
    grads = torch.autograd.grad(loss, var, create_graph=True)[0]
    grads = torch.cat([g.view(-1) for g in grads])
    
    for grad in grads:
        grad2 = torch.autograd.grad(grad, var, create_graph=True)
        temp.append(grad2)
    return np.array(temp)
</code></pre>
<p>It returns an empty list <code>[]</code>. Seems like the gradient of grad cannot be computed.<br>
Any help?</p>
<p>Thanks,</p>

Wei_Deng: <p>Waiting for the same solution.</p>

paul_c: <p>I use follow code to evaluate Hessian matrix:</p>
<pre><code class="lang-auto"># eval Hessian matrix
def eval_hessian(loss_grad, model):
    cnt = 0
    for g in loss_grad:
        g_vector = g.contiguous().view(-1) if cnt == 0 else torch.cat([g_vector, g.contiguous().view(-1)])
        cnt = 1
    l = g_vector.size(0)
    hessian = torch.zeros(l, l)
    for idx in range(l):
        grad2rd = autograd.grad(g_vector[idx], model.parameters(), create_graph=True)
        cnt = 0
        for g in grad2rd:
            g2 = g.contiguous().view(-1) if cnt == 0 else torch.cat([g2, g.contiguous().view(-1)])
            cnt = 1
        hessian[idx] = g2
    return hessian.cpu().data.numpy()
</code></pre>
<p>where loss_grad can calculate like:  <code>autograd.grad(loss, net.parameters(), create_graph=True)</code></p>
<p>Note: it’s only for small network</p>

JIANG_GUOQING: <p>I’ve solved this problem. Hope below will also help your problem.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/Ageliss/For_shared_codes/blob/master/Second_order_gradients.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/Ageliss/For_shared_codes/blob/master/Second_order_gradients.py" target="_blank" rel="nofollow noopener">Ageliss/For_shared_codes/blob/master/Second_order_gradients.py</a></h4>
<pre><code class="lang-py"># Author: Guo-qing Jiang (jianggq@mit.edu)
# Pytorch second oder gradient calculation for the diagonal of Hessian matrix 
# feel free to copy
import torch


def get_second_order_grad(grads, xs):
    start = time.time()
    grads2 = []
    for j, (grad, x) in enumerate(zip(grads, xs)):
        print('2nd order on ', j, 'th layer')
        print(x.size())
        grad = torch.reshape(grad, [-1])
        grads2_tmp = []
        for count, g in enumerate(grad):
            g2 = torch.autograd.grad(g, x, retain_graph=True)[0]
            g2 = torch.reshape(g2, [-1])
            grads2_tmp.append(g2[count].data.cpu().numpy())
        grads2.append(torch.from_numpy(np.reshape(grads2_tmp, x.size())).to(DEVICE_IDS[0]))
        print('Time used is ', time.time() - start)
</code></pre>

  This file has been truncated. <a href="https://github.com/Ageliss/For_shared_codes/blob/master/Second_order_gradients.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


michaelklachko: <p>Hi <a class="mention" href="/u/jiang_guoqing">@JIANG_GUOQING</a> , looking at your code, in the line</p>
<p><code>g2 = torch.autograd.grad(g, x, retain_graph=True)[0]</code></p>
<p><code>g</code> is an individual gradient, but <code>x</code> is a vector of weights.<br>
Is that intended? Don’t you want to calculate individual second order gradients for each individual weight?</p>
<p>However, if I add <code>x = torch.reshape(x, [-1])</code> line I get</p>
<pre><code class="lang-auto">&gt;     g2 = torch.autograd.grad(g, x, retain_graph=True)[0]
&gt; RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
</code></pre>
<p>Anyone knows what this means?</p>
<p>EDIT: Oh I see that you pick g2[count] afterwards to get the diagonal of the hessian, but I’m still confused why I can’t calculate a gradient of a scalar in respect to a scalar.</p>

John1231983: <p>Interesting! Anyone can answer the purpose of hessian in cnn. Thanks</p>

michaelklachko: <p>I’m using it to penalize growth of second order gradients, so basically the same reason why anyone would penalize first order gradients. In my case it’s to improve noise robustness.</p>

John1231983: <aside class="quote no-group" data-username="michaelklachko" data-post="7" data-topic="15270">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michaelklachko/48/541_2.png" class="avatar"> michaelklachko:</div>
<blockquote>
<p>improve noise robustness.</p>
</blockquote>
</aside>
<p>So, it will be integrated in loss function or working as a layer of cnn? Do you have any reference paper using hessian to improve accuracy? I am also interested in classification/detection/segmentation</p>

michaelklachko: <p>What do you mean “working as a layer”?</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://www.semanticscholar.org/img/favicon.png" class="site-icon" width="16" height="16">
      <a href="https://www.semanticscholar.org/paper/Regularizing-Multilayer-Perceptron-for-Robustness-Dey-Nag/4ad1bbcaa6b038c7737d0430949a4321a7bbe193" target="_blank" rel="nofollow noopener">semanticscholar.org</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:113/59;"><img src="https://www.semanticscholar.org/img/semantic_scholar_og.jpg" class="thumbnail"></div>

<h3><a href="https://www.semanticscholar.org/paper/Regularizing-Multilayer-Perceptron-for-Robustness-Dey-Nag/4ad1bbcaa6b038c7737d0430949a4321a7bbe193" target="_blank" rel="nofollow noopener">Regularizing Multilayer Perceptron for Robustness - Semantic Scholar</a></h3>

<p>The weights of a multilayer perceptron (MLP) may be altered by multiplicative and/or additive noises if it is implemented in hardware. Moreover, if an MLP is implemented using analog circuits, it is prone to stuck-at 0 faults, i.e., link failures. In...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>However it’s extremely slow, and I’m about to give up trying to implement it efficiently.</p>

JIANG_GUOQING: <p>No, you don’t need to reshape x. Since the graph was memorized in the previous gradient calculation, you should not change x if you wanna calculate high order gradient.</p>
<p>Hope it makes sense to you.</p>
<p>Michael Klachko <a href="mailto:noreply@discuss.pytorch.org">noreply@discuss.pytorch.org</a> 于2019年1月13日周日 上午6:45写道：</p>

michaelklachko: <p>Did you use this code during training? I guess not, because switching to numpy would break autodiff…</p>

JIANG_GUOQING: <p>Yes, you are right. But actually I turned on the training mode because I needed to calculate the high order of ‘Gradient’.</p>
<p>Michael Klachko via PyTorch Forums <a href="mailto:noreply@discuss.pytorch.org">noreply@discuss.pytorch.org</a> 于2019年3月28日周四 上午3:29写道：</p>

nima_rafiee: <p>I think currently it’s only possible to calculate Hessian w.r.t input x not the parameters of the network.</p>

tengerye: <p>I have checked with solutions of <a class="mention" href="/u/jiang_guoqing">@JIANG_GUOQING</a> and <a class="mention" href="/u/paul_c">@paul_c</a>, overall, they are slow. The biggest problem stuck with the <code>autograd.grad()</code> can only work on single output. Does anyone has faster solution?</p>

albanD: <p>Hi,</p>
<p>Unfortunately, these operations are expected to be relatively slow. You can find <a href="https://gist.github.com/apaszke/226abdf867c4e9d6698bd198f3b45fb7" rel="nofollow noopener">here</a> a simple way to get Jacobian and Hessian matrices.</p>

tengerye: <p>Thank you for your kind reply. Unfortunately, the linked code is even slightly slower.</p>

Yaroslav_Bulatov: <p><a class="mention" href="/u/tengerye">@tengerye</a> Would the Gauss-Newton matrix work for you? It’s equal to the Hessian for Linear and ReLU networks. For other networks it can be viewed as a positive semidefinite approximation to the Hessian (ie, <a href="http://andrew.gibiansky.com/blog/machine-learning/gauss-newton-matrix/" rel="nofollow noopener">http://andrew.gibiansky.com/blog/machine-learning/gauss-newton-matrix/</a>)</p>
<p>Generic Hessian is expensive even for a theoretically optimal algorithm while Gauss-Newton is cheap.</p>

tengerye: <p><a class="mention" href="/u/yaroslav_bulatov">@Yaroslav_Bulatov</a> Sorry for the reply. I believe that is exactly I wanted. If possible, is there any tutorial with pytorch code or code demos that you may know? Thank you so much.</p>

Yaroslav_Bulatov: <p><a class="mention" href="/u/tengerye">@tengerye</a> even though Gauss-Newton is cheap to compute, the matrix will typically need too much storage to store explicitly, so you’ll additionally need some kind of structured approximation. Here’s an example of computing diagonal and KFAC approximations of Gauss-Newton for linear layers – <a href="https://github.com/cybertronai/autograd-lib#autograd_lib" rel="nofollow noopener">https://github.com/cybertronai/autograd-lib#autograd_lib</a></p>

kkudus: <p>Im looking to efficiently compute the hessian of my loss function with respect to my inputs (only inputs, not weights). Is this a suitable solution? Having some trouble understanding it.</p>