Second derivatives of a function of two variables

Mikhail_RC: <p>My model takes two inputs x and y and returns T(x, y).<br>
How can I find the second derivatives of d_2_T / d_x_x and d_2_T / d_y_y?</p>
<p>Now I managed to find only the first derivatives in this way:</p>
<pre><code class="lang-auto"># g - tensor of input values
T = NN(g)
T_x_y = autograd.grad(T,g,torch.ones([g.shape[0], 1]).to(device), retain_graph=True, create_graph=True)[0]
</code></pre>

KFrank: <p>Hi Mikhail!</p>
<aside class="quote no-group" data-username="Mikhail_RC" data-post="1" data-topic="166462" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mikhail_rc/48/45668_2.png" class="avatar"> Mikhail_RC:</div>
<blockquote>
<p>How can I find the second derivatives of d_2_T / d_x_x and d_2_T / d_y_y?</p>
</blockquote>
</aside>
<p>Continue on with the code you have.  Your call to <code>autograd.grad()</code> computes<br>
the first derivative, but it also constructs the computation graph for the<br>
computation of the first derivative.  So you can simply call <code>autograd.grad()</code><br>
a second time to compute the second derivative.</p>
<p>Here is an illustration in the spirit of your code:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.13.0
&gt;&gt;&gt;
&gt;&gt;&gt; g = torch.tensor ([5.0, 7.0], requires_grad = True)
&gt;&gt;&gt;
&gt;&gt;&gt; T = g[0]**2 * g[1]**3
&gt;&gt;&gt; T_x_y = torch.autograd.grad (T, g, retain_graph=True, create_graph=True)[0]   # creates graph of first derivative
&gt;&gt;&gt;
&gt;&gt;&gt; T_d_xx = torch.autograd.grad (T_x_y[0], g, retain_graph = True)[0][0]         # compute second derivative (and save graph for T_d_yy computation)
&gt;&gt;&gt; T_d_yy = torch.autograd.grad (T_x_y[1], g)[0][1]                              # compute second derivative
&gt;&gt;&gt;
&gt;&gt;&gt; T_d_xx_B = 2 * g[1]**3        # check second derivative "by hand"
&gt;&gt;&gt; T_d_yy_B = g[0]**2 * 6*g[1]   # check second derivative "by hand"
&gt;&gt;&gt;
&gt;&gt;&gt; T_d_xx
tensor(686.)
&gt;&gt;&gt; T_d_xx_B
tensor(686., grad_fn=&lt;MulBackward0&gt;)
&gt;&gt;&gt; T_d_yy
tensor(1050.)
&gt;&gt;&gt; T_d_yy_B
tensor(1050., grad_fn=&lt;MulBackward0&gt;)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

buburakk: <p>It has been a while since the last reply given in this thread. However, I have got a similar question and associated error, and thought it would be more appropriate to post it here. In PyTorch version <code>2.1.0+cu118</code> I have set up two different ways, <strong>Code I</strong> and <strong>Code II</strong>, to calculate second derivatives of a multivariable function. How can I successfully carry out it as in <strong>Code II</strong>?</p>
<p><strong>Code I:</strong></p>
<pre><code class="lang-python">def f(x, t):

  return (3 * x ** 3) * t + (t ** 3) * x

x = torch.tensor([2.06, 5.7], requires_grad=True)
t = torch.tensor([1.5, 0.33], requires_grad=True)

y = f(x, t)

dy_dt = torch.autograd.grad(y.sum(), t, create_graph=True)[0]
dy_dx = torch.autograd.grad(y.sum(), x, create_graph=True)[0]
d2y_dx2 = torch.autograd.grad(dy_dx.sum(), x)[0]

loss = dy_dt + dy_dx - 0.01 * d2y_dx2

print(loss)
</code></pre>
<p><strong>Output I:</strong></p>
<p><code>tensor([100.2378, 653.6338], grad_fn=&lt;SubBackward0&gt;)</code></p>
<p><strong>Code II:</strong></p>
<pre><code class="lang-python">x = torch.tensor([2.06, 5.7], requires_grad=True)
t = torch.tensor([1.5, 0.33], requires_grad=True)

model = torch.nn.Linear(2, 1)

var_input = torch.stack([x, t], dim=1)

u = model(var_input)

du_dt = torch.autograd.grad(u.sum(), t, create_graph=True)[0]
du_dx = torch.autograd.grad(u.sum(), x, create_graph=True)[0]
d2u_dx2 = torch.autograd.grad(du_dx.sum(), x)[0]

loss = du_dt + du_dx - 0.01 * d2u_dx2

print(loss)
</code></pre>
<p><strong>Output II:</strong></p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-40-bee5e3fcde20&gt; in &lt;cell line: 12&gt;()
     10 du_dt = torch.autograd.grad(u.sum(), t, create_graph=True)[0]
     11 du_dx = torch.autograd.grad(u.sum(), x, create_graph=True)[0]
---&gt; 12 d2u_dx2 = torch.autograd.grad(du_dx.sum(), x)[0]
     13 
     14 loss = du_dt + du_dx - 0.01 * d2u_dx2

/usr/local/lib/python3.10/dist-packages/torch/autograd/__init__.py in grad(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused, is_grads_batched, materialize_grads)
    392         )
    393     else:
--&gt; 394         result = Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    395             t_outputs,
    396             grad_outputs_,

RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
</code></pre>

KFrank: <p>Hi Burak!</p>
<aside class="quote no-group quote-modified" data-username="buburakk" data-post="3" data-topic="166462" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/buburakk/48/63881_2.png" class="avatar"> buburakk:</div>
<blockquote>
<p><strong>Code II:</strong></p>
<pre data-code-wrap="python"><code class="lang-python">model = torch.nn.Linear(2, 1)

var_input = torch.stack([x, t], dim=1)

u = model(var_input)
</code></pre>
<p><strong>Output II:</strong></p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
---&gt; 12 d2u_dx2 = torch.autograd.grad(du_dx.sum(), x)[0]
...
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
</code></pre>
</blockquote>
</aside>
<p>In your second case, <code>u</code> is linear in <code>x</code> (and in <code>t</code>).  Therefore <code>du_dx</code> is a<br>
constant with respect to <code>x</code>, which is to say independent of <code>x</code>.  Autograd<br>
recognizes this and, for efficiency, removes <code>x</code> from <code>du_dx</code>’s computation<br>
graph (at least in effect).</p>
<p>To see what is going on, try <code>var_input = torch.stack([x**2, t], dim=1)</code>.<br>
<code>u</code> will no longer be linear in <code>x</code> and the second derivative of <code>u</code> with respect<br>
to <code>x</code> will no longer vanish.  Also setting <code>allow_unused = True</code> will let you<br>
use autograd to compute zero for the second derivative in the linear case.</p>
<p>As an aside, it probably would have been better to post this question in a<br>
new thread, rather than resurrecting a zombie.</p>
<p>Best.</p>
<p>K. Frank</p>

buburakk: <p>Thank you very much! This reply really made me realize that the simple linear model I gave in my question for demonstration is the cause of my issue. Since the intended model I have in hand is in fact nonlinear. In short, I have created an imaginary problem on my own.</p>

ParsaMD99: <p>Hey KFrank,</p>
<p>Just discovered your awesome answer from two years ago. Wanted to say a big thank you! It really saved the day near my project deadline. I’m sure your response has also been a lifesaver for many others :).</p>
<p>Cheers<br>
Parsa</p>