Second order derivatives of loss function

username: keqiao_li
cooked: <p>My main question is how to calculate the second order derivatives of a loss function. But I started with a toy example as follows:</p>
<pre><code>import torch
x = torch.tensor(1., requires_grad = True)
y = 2*x**3 + 5*x**2 + 8
y.backward(retain_graph=True, create_graph=True)
x.grad
y.backward()
x.grad
</code></pre>
<p>My thought is by call “backward()” twice would give me the second order derivatives, but the answer is 32 which is not correct. It would be really appreciate if someone can explain where is wrong.</p>
<p>And second my goal is to calculate the second order derivatives of the following loss function. I have this following defined loss function, and I tried to use newton’s method to do optimization. Since newton’s method requires the first derivative and second derivative at the each iteration, so I tried to write some code as follows:</p>
<pre><code>loss.backward(retain_graph=True, create_graph=True)
first_derivative = w.grad
loss.backward()
second_derivative = w.grad
</code></pre>
<p>I guess what I’m doing here is wrong given the above result of the toy example. How can I get the second order derivatives?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d84e2ee9ceb5fef8c6a1ee0ea677f72ad32ada66.png" data-download-href="https://discuss.pytorch.org/uploads/default/d84e2ee9ceb5fef8c6a1ee0ea677f72ad32ada66" title="Screen Shot 2020-03-02 at 9.10.41 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d84e2ee9ceb5fef8c6a1ee0ea677f72ad32ada66_2_690x89.png" alt="Screen Shot 2020-03-02 at 9.10.41 PM" data-base62-sha1="uRwDHXAyh4mY2nnUKFwyMIO5vLg" width="690" height="89" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d84e2ee9ceb5fef8c6a1ee0ea677f72ad32ada66_2_690x89.png, https://discuss.pytorch.org/uploads/default/original/3X/d/8/d84e2ee9ceb5fef8c6a1ee0ea677f72ad32ada66.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/8/d84e2ee9ceb5fef8c6a1ee0ea677f72ad32ada66.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-03-02 at 9.10.41 PM</span><span class="informations">1014×132 6.75 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ziqipang
cooked: <p>You may find something here</p><aside class="quote" data-post="5" data-topic="15085">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wei_deng/48/3370_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-calculate-2nd-derivative-of-a-likelihood-function/15085/5">How to calculate 2nd derivative of a likelihood function</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Thanks you <a class="mention" href="/u/dpernes">@dpernes</a> for you inspiring comments, that works for that problem. 
But I found this still can not help me find the 2nd derivative of a likelihood function for each weight, do you know how to achieve that? 
Or do you know if we can do it based on the current pytorch version?
  </blockquote>
</aside>


username: albanD
cooked: <p>Hi,</p>
<p>If you have a single input and single output, you want to do the following:<br>
(note that using <code>.backward()</code> for higher order derivatives is discouraged because the .grad field becomes hard to reason about).</p>
<pre><code class="lang-auto">first_derivative = autograd.grad(loss, x, create_graph=True)[0]
# We now have dloss/dx
second_derivative = autograd.grad(first_derivative, x)[0]
# This computes d/dx(dloss/dx) = d2loss/dx2
</code></pre>

username: bb243
cooked: <p><a class="mention" href="/u/alband">@albanD</a> What should I do to take the second derivative of a 2D tensor (or list of 2D tensors)?<br>
Specifically, I need to get the second derivative w.r.t. all parameters in the network.<br>
Currently, I have set up the weights of the network  as a ParameterDict <code>weights</code> and am doing the following:<br>
<code>grads = torch.autograd.grad(criterion, weights.values(), create_graph=True</code><br>
and now I want to again take the derivative w.r.t. <code>weights.values()</code> but this time of <code>grads</code> which is a list of 2D tensors rather than of the scalar <code>criterion</code>.</p>

username: albanD
cooked: <p>Do you want the full hessian matrix? If so, you will need to “reconstruct” it by doing multiple times the second backward pass with a grad_output that will be a one-hot matrix for each entry in the 2D Tensors one after the other.</p>