Not getting symmetric hessian when calculating by using torch.autograd

Mrinmay_sen: <p>Hi,<br>
I am using following code to generate hessian. I have used L2 regularised categorical cross entropy as a loss function. So hessian should be symmetric. But i am getting symmetric.</p>
<p>import torch<br>
torch.set_default_dtype(torch.float64)<br>
import torchvision<br>
import torchvision.models as models<br>
from torchvision import transforms<br>
import torch.utils.data<br>
import torch.nn as nn<br>
import numpy as np<br>
device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)<br>
<span class="hashtag">#defining</span> data tranform<br>
transform = transforms.Compose([transforms.ToTensor(), transforms.Lambda(lambda x: x.repeat(3, 1, 1) ), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])</p>
<h1>
<a name="download-trainset-and-testset-1" class="anchor" href="#download-trainset-and-testset-1"></a>download trainset and testset</h1>
<p>trainset = torchvision.datasets.MNIST(root=‘data’, train=True, download=True, transform=transform)<br>
train_loader=torch.utils.data.DataLoader(trainset, batch_size=512, shuffle=True, num_workers=2)</p>
<p>testset = torchvision.datasets.MNIST(root=‘data’, train=False, download=True, transform=transform)<br>
testloader = torch.utils.data.DataLoader(testset, batch_size=128, shuffle=True, num_workers=2)</p>
<p><span class="hashtag">#defining</span> model<br>
cnn = models.resnet18(pretrained=False)<br>
classifier = nn.Linear(in_features=512, out_features=10, bias=True)<br>
cnn.fc=classifier<br>
cnn=cnn.to(device)</p>
<p>#<span class="hashtag">#loss</span> function<br>
loss_fn = nn.CrossEntropyLoss()</p>
<p>#<span class="hashtag">#flattering</span> parameters<br>
p= torch.cat([gi.data.view(-1) for gi in cnn.parameters()])</p>
<h2>
<a name="calculating-hessian-2" class="anchor" href="#calculating-hessian-2"></a>calculating hessian</h2>
<p>for batch, (X, y) in enumerate(train_loader):<br>
cnn.train()</p>
<pre><code> ###creating hessian matrix with zeros 
 H=torch.zeros(len(p), len(p)).to(device)
 X = X.to(device)
 y = y.to(device)
 pred=cnn(X)
 loss=loss_fn(pred, y) 
 l2_norm = sum(p.pow(2.0).sum() for p in cnn.parameters())
 loss = loss + l2_lambda * l2_norm

 env_grads = torch.autograd.grad(loss, cnn.parameters(), retain_graph=True, create_graph=True)
 g= torch.cat([gi.reshape(-1) for gi in env_grads])

 for i in range(len(p)):
       h_col=torch.autograd.grad(g[column_indx[i]], cnn.parameters(), retain_graph=True, create_graph=False)
       H[i]= torch.cat([gi.reshape(-1) for gi in h_col])
</code></pre>
<p>After creating H, i checked whether it is symmetric  or not… But not getting symmetric. Please suggest where is the problem.<br>
At first i used “torch.set_default_dtype(torch.float64)” but still not able to solve.<br>
I will not be able to use PyTorch inbuilt hessian function to calculate hessian as I need only some part  of hessian without calculation of whole hessian matrix.</p>

KFrank: <p>Hi Mrinmay!</p>
<aside class="quote no-group" data-username="Mrinmay_sen" data-post="1" data-topic="161508" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrinmay_sen/48/53252_2.png" class="avatar"> Mrinmay_sen:</div>
<blockquote>
<pre><code> env_grads = torch.autograd.grad(loss, cnn.parameters(), retain_graph=True, create_graph=True)
 g= torch.cat([gi.reshape(-1) for gi in env_grads])

 for i in range(len(p)):
       h_col=torch.autograd.grad(g[column_indx[i]], cnn.parameters(), retain_graph=True, create_graph=False)
       H[i]= torch.cat([gi.reshape(-1) for gi in h_col])
</code></pre>
</blockquote>
</aside>
<p>In the code you posted you are using <code>column_indx</code>  when you compute<br>
the second derivative, but not when you compute the first.  On its face,<br>
this will make your result for <code>H</code> not symmetric.  (I also note that <code>column_indx</code><br>
isn’t defined anywhere in your post.)</p>
<p>As an aside, you shouldn’t expect the Hessian you compute to be exactly<br>
numerically symmetric (even though it is mathematically symmetric); due<br>
to floating-point round-off error, it will be numerically slightly asymmetric.</p>
<p>Best.</p>
<p>K. Frank</p>

Mrinmay_sen: <p>Dear Frank,<br>
column_indx is  the list of columns which are randomly selected variable put of all the parameters. Only hessian with   these columns i need to caculate</p>
<p>p= torch.cat([gi.data.view(-1) for gi in cnn.parameters()])<br>
column_indx= np.random.choice(len(p), 10, replace= False)</p>

Mrinmay_sen: <p>But when i considered all the columns that is full hessian, it is also not symmetric.</p>
<p>cnn.train()<br>
p= torch.cat([gi.data.view(-1) for gi in cnn.parameters()])<br>
column_indx= np.random.choice(len(p), len(p), replace= False)<br>
for batch, (X, y) in enumerate(train_loader):<br>
cnn.train()<br>
X = X.to(device)<br>
y = y.to(device)<br>
pred=cnn(X)<br>
loss=loss_fn(pred, y)<br>
env_grads = torch.autograd.grad(loss, cnn.parameters(), retain_graph=True, create_graph=True)<br>
g= torch.cat([gi.reshape(-1) for gi in env_grads])<br>
C=torch.zeros(len(p), len(p)).to(device)<br>
cnn.zero_grad()<br>
for i in range(len(p)):<br>
h_col=torch.autograd.grad(g[column_indx[i]], cnn.parameters(), retain_graph=True, create_graph=False)<br>
C[i]= torch.cat([gi.reshape(-1) for gi in h_col])</p>
<p>This is the code which i have used to calculate  hessian.<br>
Kindly have a look on the last two line of this code. Please check whether it is the correct approach or not to calculate hessian</p>

KFrank: <p>Hi Mrinmay!</p>
<aside class="quote no-group" data-username="Mrinmay_sen" data-post="4" data-topic="161508" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrinmay_sen/48/53252_2.png" class="avatar"> Mrinmay_sen:</div>
<blockquote>
<pre><code>    h_col=torch.autograd.grad(g[column_indx[i]], cnn.parameters(), retain_graph=True, create_graph=False) 
</code></pre>
</blockquote>
</aside>
<p>As I alluded to in my first post, when you use <code>column_indx</code> just for the<br>
columns of your Hessian, you are, in effect permuting the columns of the<br>
Hessian but not permuting the rows.</p>
<p>If you permute just the columns of a symmetric matrix, it won’t be symmetric<br>
anymore.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.12.0
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt;
&gt;&gt;&gt; mat = torch.randn (5, 5)
&gt;&gt;&gt; mat = mat + mat.T   # make symmetric matrix
&gt;&gt;&gt; torch.equal (mat, mat.T)   # verify symmetric
True
&gt;&gt;&gt;
&gt;&gt;&gt; column_indx = torch.randperm (5)   # random permutation of column indices
&gt;&gt;&gt;
&gt;&gt;&gt; mat_cperm = mat[:, column_indx]   # permute columns
&gt;&gt;&gt; torch.equal (mat_cperm, mat_cperm.T)   # verify asymmetric
False
&gt;&gt;&gt;
&gt;&gt;&gt; mat_cperm_rperm = mat_cperm[column_indx, :]   # also permute rows
&gt;&gt;&gt; torch.equal (mat_cperm_rperm, mat_cperm_rperm.T)   # matrix is now symmetric again
True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Mrinmay_sen: <p>Hi Frank,</p>
<p>Thanks a lot for the solution.<br>
Now i came to know about the problem. Thanks a lot again Frank.</p>