The following is my code block for realizing the second-order derivative of the matrix. Why is the output second-order derivative different from the one I manually calculated

username: jinxin_wang
cooked: <p>z = torch.ones_like(y)</p>
<p>dy = torch.autograd.grad(y, X, grad_outputs=z, create_graph=True)[0]<br>
print(dy)</p>
<p>d2y = torch.autograd.grad(dy, X, grad_outputs=z, create_graph=True)[0]<br>
print(d2y)</p>

username: Sam-gege
cooked: <p>can you also provide X and y?<br>
anyways,<br>
import torch<br>
X=torch.tensor([1.,2.,3.],requires_grad=True)<br>
y=X**3<br>
z = torch.ones_like(y)<br>
dy = torch.autograd.grad(y, X, grad_outputs=z, create_graph=True)[0]<br>
print(dy)<br>
d2y = torch.autograd.grad(dy, X, grad_outputs=z, create_graph=True)[0]<br>
print(d2y)</p>
<p>The output of d2y is tensor([ 6., 12., 18.], grad_fn=&lt; MulBackward0 &gt;), which is just the second order derivative of y=x**3 isn’t it? y’’=6x</p>

username: jinxin_wang
cooked: <p>I am a newbie to pytorch. I recently tried to use pytorch to implement the PINN framework. The following is a simplified network layer I designed to verify the correctness of the second-order guide implementation of the program, but obviously, the output results are not very ideal. I also tried to use this form to test the Linear() and Sigmoid() layers separately, and the result was no problem. But I don’t know why there is a problem when combining them to find the second derivative. I hope you can help me find the problem.</p>
<p>import torch<br>
from torch import nn<br>
x=torch.tensor([[2.,1]],dtype=torch.float32).t()<br>
y=torch.tensor([[1,0]],dtype=torch.float32).t()<br>
net=nn.Sequential(nn.Linear(2,2),nn.Sigmoid())<br>
def init_ones(m):<br>
if type(m)==nn.Linear:<br>
nn.init.ones_(m.weight)<br>
nn.init.zeros_(m.bias)<br>
net.apply(init_ones)<br>
net[0].weight.data[1][0]=0.0<br>
net[0].weight,net[0].bias</p>
<pre><code class="lang-auto">(Parameter containing:
 tensor([[1., 1.],
         [0., 1.]], requires_grad=True),
 Parameter containing:
 tensor([0., 0.], requires_grad=True))
</code></pre>
<p>X=torch.cat((x,y),1)<br>
X.requires_grad_(True)<br>
y=net(X)</p>
<p>z = torch.ones_like(y)<br>
dy = torch.autograd.grad(y, X, grad_outputs=z,retain_graph=True, create_graph=True)[0]<br>
print(dy)</p>
<p>d2y = torch.autograd.grad(dy, X, grad_outputs=z, create_graph=False)[0]<br>
print(d2y)</p>

username: Sam-gege
cooked: <p>perhaps it has something to do with how <a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html" rel="noopener nofollow ugc">nn.Linear</a> is defined? i.e. y=x@A.T, where ‘@’ is matmul, not the usual y=A@X. so during backprop, dy/dx == grad @ A, whereas if y=A@X, then dy/dx==A.T@grad.</p>
<p>I myself is a new learner too, and I didn’t notice this until now, thanks dude!</p>
<p>BTW, you can test this by using:</p>
<p>import torch<br>
X=torch.tensor([[2., 1.],[1., 0.]],requires_grad=True)<br>
w=torch.tensor([[1., 1.],[0., 1.]],requires_grad=True)<br>
y=torch.nn.Sigmoid()(torch.matmul(X,w.T))<br>
print(f’weight:{w}\n’)<br>
print(f’input:{X}\n’)<br>
print(f’output:{y}\n’)<br>
z = torch.ones_like(y)<br>
dy = torch.autograd.grad(y, X, grad_outputs=z,retain_graph=True, create_graph=True)[0]<br>
print(f’X.grad is:{dy}’)</p>
<p>which gives the same result as your dy. whereas if you used y=torch.nn.Sigmoid()(torch.matmul(w,X)), then dy is the transpose of your dy</p>

username: jinxin_wang
cooked: <aside class="quote no-group" data-username="jinxin_wang" data-post="3" data-topic="133044">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jinxin_wang/48/37154_2.png" class="avatar"> jinxin_wang:</div>
<blockquote>
<p>d2y = torch.autograd.grad(dy, X, grad_outputs=z, create_graph=False)[0]<br>
print(d2y)</p>
</blockquote>
</aside>
<p>Oh, dude, your job just now was great.<br>
Maybe I did not express it clearly enough that you did not understand the point of my question. After my manual verification, the solution of the first-order derivative is no problem.<br>
But an unknown error occurred when <code>d2y = torch.autograd.grad(dy, X, grad_outputs=z, create_graph=False)[0]  print(d2y)</code> calculated the second derivative. You can manually calculate the second derivative of this implementation process. The correct result should be<code>  tensor([[-0.0409, -0.0909],         [-0.0909,  0.0000]])</code>  the output of the above program is indeed  <code>tensor([[-0.0818, -0.1726],         [-0.1817, -0.1817]])</code></p>

username: Sam-gege
cooked: <p>I believe pytorch is correct. Basically the problem is this:<br>
y=sigmoid(x @ w.T)<br>
dy/dx=y*(1-y) @ w<br>
where @ is matrix mul, * is element-wise mul<br>
to calculate second order derivatives d2y/dx2, you can draw a computational graph starting from x and ending at dy/dx, then backpropagates to x. The final answer is this:<br>
d2y/dx2 = z @ w.T * (1 - 2y) * y * (1-y) @ w<br>
Of course, when I tried to derive the above equations, I got the wrong answer too… if anyone is good at math please let me know how it’s derived…</p>

username: AlphaBetaGamma96
cooked: <p>This will only work if <code>X</code> is a scalar, what you need is to use <code>torch.autograd.functional.hessian</code>. The docs for that function are here: <a href="https://pytorch.org/docs/stable/generated/torch.autograd.functional.hessian.html?highlight=hessian#torch.autograd.functional.hessian" class="inline-onebox" rel="noopener nofollow ugc">torch.autograd.functional.hessian — PyTorch 1.9.1 documentation</a></p>

username: jinxin_wang
cooked: <p>Thank you for informing me of this function, but the pytorch I use is too low to support the implementation of this function. As far as I know, the implementation of my code above is also based on the Hessian matrix.</p>

username: AlphaBetaGamma96
cooked: <p>Can you not update to a higher version of pytorch?</p>
<p>If not, what’s the shape of <code>X</code> and <code>y</code>?</p>

username: jinxin_wang
cooked: <pre><code class="lang-auto">X=tensor([[2., 1.],
        [1., 0.]])
</code></pre>
<pre><code class="lang-auto">y=tensor([[0.9526, 0.7311],
        [0.7311, 0.5000]], grad_fn=&lt;SigmoidBackward&gt;)
</code></pre>

username: AlphaBetaGamma96
cooked: <p>So <code>X</code> and <code>y</code> are both shape <code>[2,2]</code>? Is this for a single sample or for all samples?</p>

username: jinxin_wang
cooked: <p>single sample .<br>
I use this simple program to verify the correctness of the code block function that I will write into the PINN deep training.</p>

username: AlphaBetaGamma96
cooked: <p>That example only works because the input arrays both hold scalars, for non-scalar arguments it won’t work which is why the <code>torch.autograd.functional</code> library exists!</p>

username: jinxin_wang
cooked: <p>Maybe you are right, dude</p>

username: AlphaBetaGamma96
cooked: <p>There’s probably a way in which to do this without the use of the <code>torch.autograd.functional</code> library, although I’m not 100% how to do this. You could have a look at the source code for both functions and see if you can reproduce them without upgrading pytorch?</p>
<p>If you can update pytorch I’d recommend you do it.</p>