Computing gradient of quadratic form using autograd

Xavier_Aramayo_Carra: <p>Hello, I am trying to compute the gradient of the following quadratic expression: x.T@A@x where x is a vector with n samples and m features and A is a matrix m times m, the direct derivative of this expression with respect to x is x@(A + A.T), I am trying to get this derivative using autograd function but I couldn’t do it, I implemented it this way:</p>
<pre><code class="lang-auto">n_samples = 5
n_features = 2
x = torch.randint(0, 5, (n_samples, n_features)).to(torch.float32).requires_grad_(True)
A  = torch.randint(0, 5, (n_features, n_features)).to(torch.float32)#.requires_grad_(True)

#Using Autograd
loss = (x @ A @ x.T).sum()
loss.backward()
grad = x.grad
grad2 = x @ (A + A.T)

print(grad)
print(grad2)
</code></pre>
<p>However, both results are different, the grad result corresponds to the sum over columns of grad2, for example:</p>
<pre><code class="lang-auto">grad = tensor([[96., 78.],
        [96., 78.],
        [96., 78.],
        [96., 78.],
        [96., 78.]])
grad2 = tensor([[12., 12.],
        [16., 14.],
        [20., 18.],
        [28., 22.],
        [20., 12.]], grad_fn=&lt;MmBackward0&gt;)
grad2.sum(dim=0) = tensor([96., 78.], grad_fn=&lt;SumBackward1&gt;)
</code></pre>
<p>I understand that this behavior is maybe due to the fact that I am using the .sum() operator before backward(), but I would like to know if I can get the same result as in the manual derivation. I appreciate any help and further explanation about this. Thanks in advance.</p>

KFrank: <p>Hi Xavier!</p>
<aside class="quote no-group" data-username="Xavier_Aramayo_Carra" data-post="1" data-topic="192220" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xavier_aramayo_carra/48/65623_2.png" class="avatar"> Xavier_Aramayo_Carra:</div>
<blockquote>
<p>I am trying to compute the gradient of the following quadratic expression: x.T@A@x where x is a vector with n samples and m features and A is a matrix m times m</p>
</blockquote>
</aside>
<p>It’s not entirely clear what you are trying to compute here.  (Note that<br>
here you have the transpose in the wrong place, but you have it correct<br>
in your sample code.)</p>
<p>Correcting the transpose, your expression <code>x @ A @ x.T</code> has shape<br>
<code>[n_samples, n_samples]</code>, so it’s unclear what you want to differentiate<br>
with respect to <code>x</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>the direct derivative of this expression with respect to x is x@(A + A.T)</p>
</blockquote>
</aside>
<p>This formula gives you a batch (of length <code>n_samples</code>) of gradients (each<br>
of which has length <code>n_features</code>) of scalar quadratic forms using <code>A</code>.  So I<br>
speculate that you want to compute <code>x[i] @ A @ x[i]</code> on a batch basis<br>
and then compute the gradients of these scalars with respect to <code>x[i]</code> on<br>
a  batch basis.</p>
<aside class="quote no-group">
<blockquote>
<p>I am trying to get this derivative using autograd function</p>
</blockquote>
</aside>
<p>You can use autograd to do what (I think) you want, but you have to<br>
somehow select out the individual batch elements that you want.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">#Using Autograd
loss = (x @ A @ x.T).sum()
loss.backward()
grad = x.grad
</code></pre>
</blockquote>
</aside>
<p>Consider this tweaked version of your example code:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

_ = torch.manual_seed (1066)

n_samples = 5
n_features = 2
x = torch.randint(0, 5, (n_samples, n_features)).to(torch.float32).requires_grad_(True)
A  = torch.randint(0, 5, (n_features, n_features)).to(torch.float32)#.requires_grad_(True)

#Using Autograd
loss = (x @ A @ x.T).sum()              # this includes  x[i] @ A @ x[j] terms
loss.backward()
grad = x.grad

x.grad = None
loss = (x @ A @ x.T).diagonal().sum()   # keep just the  x[i] @ A @ x[i] terms
loss.backward()
gradB = x.grad

grad2 = x @ (A + A.T)                   # batch of gradients for a batch of scalar quadratic forms

print(grad)
print(gradB)
print(grad2)

for  xi in x:                           # compute batch of gradients with explicit loop
    xi = xi.detach().requires_grad_()
    (xi @ A @ xi).backward()
    print (xi.grad)
</code></pre>
<p>Here is its output:</p>
<pre><code class="lang-plaintext">2.1.0
tensor([[68., 74.],
        [68., 74.],
        [68., 74.],
        [68., 74.],
        [68., 74.]])
tensor([[10., 20.],
        [18., 14.],
        [ 6.,  8.],
        [14., 12.],
        [20., 20.]])
tensor([[10., 20.],
        [18., 14.],
        [ 6.,  8.],
        [14., 12.],
        [20., 20.]], grad_fn=&lt;MmBackward0&gt;)
tensor([10., 20.])
tensor([18., 14.])
tensor([6., 8.])
tensor([14., 12.])
tensor([20., 20.])
</code></pre>
<p>This shows how you can use a loop to compute the batch of gradients<br>
and also how you can do it using a single, loop-free call to <code>.backward()</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>I understand that this behavior is maybe due to the fact that I am using the .sum() operator before backward()</p>
</blockquote>
</aside>
<p>Yes, the <code>.sum()</code> is your problem (based on my assumption that you want<br>
to focus on the individual scalar quadratic forms, <code>x[i] @ A @ x[i]</code>).  In your<br>
expression, <code>.sum()</code> is also summing over the cross terms <code>x[i] @ A @ x[j]</code>.<br>
There are various ways to deal with this, one being to use <code>.diagonal()</code> to<br>
pluck out just the terms you want.  The loop version in my example script<br>
verifies that <code>.diagonal().sum()</code> does, in fact, give you what (I think) you<br>
want.</p>
<p>Best.</p>
<p>K. Frank</p>

Xavier_Aramayo_Carra: <p>That is what I was willing to do, thank you so much!</p>