2nd Order Cross-derivative Matrix-Vector Product (Similar to Hessian-Vector Product)

pkoch: <p>Hi,</p>
<p>I am currently trying to find the product of a 2nd order cross-derivative matrix and a vector. That is, (similar to a hessian-vector product) I want to find<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/5/35dae25b1154e3fd38d0ccac77dbd71768025a5a.png" alt="Screenshot from 2024-05-08 12-57-16" data-base62-sha1="7GqgjDIMrUtpUCh2See3qPm6u0i" width="358" height="177"><br>
for arbitrary b and h some function.</p>
<p>Additionally, I want to do this in a way which doesn’t ever store the matrix directly, rather I just need the product (E.g. I don’t want to use torch.func.hessian(self.h, argnums=(0, 1))).</p>
<p>Here is my current attempt with a toy function h:</p>
<pre><code class="lang-auto">import torch

def h(x, y):
    return x**2 @ y + y**2 @ x

x = torch.tensor([1., 2., 3.], requires_grad=True)
y = torch.tensor([2., 3., 4.], requires_grad=True)
b = torch.tensor([1., 10., 100.,])

eval = h(x, y)
eval.backward()

eval_2 = (x.grad * b).sum()
eval_2.backward()

print(y.grad)
</code></pre>
<p>However I seem to get the following error:</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>Any help is much appreciated.</p>

KFrank: <p>Hi Pkoch!</p>
<aside class="quote no-group" data-username="pkoch" data-post="1" data-topic="202327" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/df705f/48.png" class="avatar"> pkoch:</div>
<blockquote>
<p>However I seem to get the following error:</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
</blockquote>
</aside>
<p>The short answer is that you when you use autograd to compute a second derivative,<br>
you need to use <code>create_graph = True</code> so that the first derivative you compute has<br>
a computation graph through which you can backpropagate.</p>
<p>Here is a script that illustrates this both with <code>.backward()</code> and <code>autograd.grad()</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

def h(x, y):
    return x**2 @ y + y**2 @ x

x = torch.tensor([1., 2., 3.], requires_grad=True)
y = torch.tensor([2., 3., 4.], requires_grad=True)
b = torch.tensor([1., 10., 100.,])

# first version -- doesn't work

eval = h (x, y)
eval.backward()                       # doesn't create computation graph for x.grad
print ('x.grad.requires_grad:', x.grad.requires_grad)
eval_2 = (x.grad * b).sum()
print ('eval_2.requires_grad:', eval_2.requires_grad)

# eval_2.backward()                   # fails -- no computation graph for eval_2

x.grad = None
y.grad = None

# fixed version

eval = h (x, y)
eval.backward (create_graph = True)   # creates computation graph for x.grad
y.grad = None                         # zero out unwanted d_eval / d_y
eval_2 = (x.grad * b).sum()
eval_2.backward()
print ('y.grad:', y.grad)

x.grad = None
y.grad = None

# autograd.grad version -- a little cleaner

eval = h (x, y)
x_grad = torch.autograd.grad (eval, x, create_graph = True)[0]
y_grad = torch.autograd.grad ((x_grad * b).sum(), y)[0]

print ('y_grad:', y_grad)
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">2.3.0
x.grad.requires_grad: False
eval_2.requires_grad: False
&lt;path_to_pytorch_install&gt;\torch\autograd\graph.py:744: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at C:\cb\pytorch_1000000000000\work\torch\csrc\autograd\engine.cpp:1208.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
y.grad: tensor([   6.,  100., 1400.])
y_grad: tensor([   6.,  100., 1400.])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>