Hessian vector product

username: Lukas_Schroth
cooked: <p>Hey all,</p>
<p>I need to determine the largest eigenvalue of the Hessian of my loss-function. As my model is pretty large, i cannot compute the hessian directly, it’s too expensive. In my code I try to estimate the largest eigenvalue using the power iteration method and torch.autograd.grad to compute the hessian vector product.</p>
<p>However my function is currently incorrect, it sometimes converges to negative values (The Hessian is positive definite, the largest eigenvalue must be positive). Is the calculation of the Hessian vector product correct? I am especially unsure about the part where I flatten the output of torch.autograd.grad to obtain the vector.</p>
<p>Maybe this has numerial issues, so if somebody could confirm correctness of the Hessian vector product, it would be great.</p>
<p>Many thanks in advance!!</p>
<p>All the best,<br>
Lukas</p>
<pre><code class="lang-auto">def estimate_hessian_eigenvalue(self, loss, params, device, tol=1e-4, max_iter=100, mode="largest"):
    """estimates the largest singular value based on power iteration"""
    # get number of params
    num_param = sum(p.numel() for p in params)
    # Calculate the gradient of the loss with respect to the model parameters
    #print(params)
    grad_params = torch.autograd.grad(loss, list(params), create_graph=True)
    #print("grad_params unfalttened:",grad_params)
    grad_params = torch.cat([e.flatten() for e in grad_params]) # flatten
    #print("grad_params:",grad_params)
    # Compute the vector product of the Hessian and a random vector using the power iteration method
    v = torch.rand(num_param).to(device)
    v = v/torch.norm(v)
    #print(v)      
    Hv = torch.autograd.grad(grad_params, list(params), v, retain_graph=True)
    #print("Hv:",Hv)
    Hv = torch.cat([e.flatten() for e in Hv]) # flatten
    #print("Hv:",Hv)       
    # normalize Hv
    Hv = Hv /torch.norm(Hv)
    for i in range(max_iter):
        # Compute the vector product of the (inverse Hessian or) Hessian and Hv 
        w = torch.autograd.grad(grad_params, list(params), Hv, retain_graph=True)
        w = torch.cat([e.flatten() for e in w]) # flatten
        # Calculate the Rayleigh quotient to estimate the largest eigenvalue of the Hessian (inverse Hessian)
        eigenvalue = torch.dot(Hv, w)/ torch.dot(Hv, Hv) 
        # Check if the difference between consecutive estimates is below the tolerance level
        if i &gt; 0 and torch.abs(eigenvalue - last_eigenvalue) &lt; tol:
            print("tolerance reached")
            break
        last_eigenvalue = eigenvalue
        # Update Hv for the next iteration
        Hv = w/torch.norm(w)        
    return eigenvalue
</code></pre>

username: KFrank
cooked: <p>Hi Lukas!</p>
<aside class="quote no-group" data-username="Lukas_Schroth" data-post="1" data-topic="179361" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lukas_schroth/48/59548_2.png" class="avatar"> Lukas_Schroth:</div>
<blockquote>
<p>However my function is currently incorrect, it sometimes converges to negative values (The Hessian is positive definite, the largest eigenvalue must be positive).</p>
</blockquote>
</aside>
<p>It is not true, in general, that the Hessian is positive definite – it can certainly<br>
have negative eigenvalues.</p>
<p>(The Hessian is symmetric, so your computed Hessian should be symmetric up<br>
to numerical round-off error.  At a true (local or global) minimum, the Hessian<br>
will be positive semi-definite – it can have zero eigenvalues in special cases.<br>
However, even if you have trained well and you model is giving good results,<br>
you’re still not necessarily at a minimum of your loss and the Hessian could<br>
well have negative eigenvalues.)</p>
<p>Just to be clear, the “power iteration method” will converge to the eigenvalue<br>
that is largest in absolute value (rather than the algebraically largest).  So if<br>
your method returns a negative eigenvalue of <code>-1.002</code>, you could still have a<br>
positive eigenvalue of comparable, but slightly smaller magnitude, say, <code>1.001</code>.</p>
<p>I don’t see anything wrong offhand with your code, and, in particular, your<br>
flattening of the output of <code>grad()</code> seems sound.  Also, the power iteration<br>
method is pretty robust, so unless your problem is ridiculously large and / or<br>
has a lot of near degeneracy, it should be giving you the right answer.</p>
<p>Again, in general the Hessian can have negative eigenvalues.  If – taking this<br>
into account – you still think you’re having problems, apply your method to an<br>
intermediate-sized problem for which you can compute the full Hessian and<br>
its spectrum and compare the result of your method with something like that<br>
of <code>torch.linalg.eigh()</code>.</p>
<p>Best.</p>
<p>K. Frank</p>