Obtaining derivatives with PyTorch

sguysc: <p>I am trying to obtain hard-to-symbolically-compute derivatives of matrices, so I thought that I can use PyTorch’s autograd features.</p>
<p>I have a vector <em>y</em>=f(<em>x</em>) where <em>y</em> is a vector (n,1) and <em>x</em> is a matrix of (p,q). <em>y</em> is composed of a lot of operations involving x (multiplications, vertical stacking, etc …).</p>
<p>I’m trying to find the gradient of d<em>y</em>/d<em>x</em> (which I suppose needs to have the dimensions [<em>n</em> by <em>p</em> by <em>q</em>]. Not exactly sure what <a href="https://pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html#optional-reading-tensor-gradients-and-jacobian-products" rel="noopener nofollow ugc"><em>v</em></a> to write in the y.backward(<em>v</em>) function to get the correct results because if I use <code>v = torch.ones_like(y)</code> then I get the gradient <code>x.grad</code> as a matrix [p,q] where I would expect it to become [n,p,q] (and then again, maybe I have a different problem?).</p>
<p>Here’s a sample code:</p>
<pre><code class="lang-auto">import torch

n, p, q = 9, 2, 3
x = torch.randn(p, q, requires_grad=True)
A = torch.randn(q, p)
z = torch.randn(q, 1)

f = A@x
y = None
for k in range(int(n/q)):
    if(y is None):
        y = f @ z
    else:
        y = torch.vstack( ( y, f @ z ) )

    # prepare for next time step
    f = (A@x) @ f

v = torch.ones_like(y)
y.backward(v, retain_graph=True)
dydx = x.grad.clone()

print('dydx with autograd: \n', dydx)
&gt;&gt;&gt; tensor([[13.4633, -3.9368,  7.8990],
            [21.1977, -4.3395, -5.6655]])
</code></pre>

eqy: <p>What is being done here is that the autograd engine expects the starting point for backward to be a scalar, and if it is not a scalar, the <code>gradient</code> parameter of <code>backward</code> is used to compute the gradient with respect to the inputs (in this case the inputs are x). When the elements of x are used in multiple parts of the graph, the total derivative is computed (the sum of the gradients for each of the uses). You can observe this as the magnitude of <code>x.grad</code> will increase the larger you make n.</p>
<p>If you expect these to be separate, you might want to try calling backward on individual elements of <code>y</code> and manually constructing your <code>[n, p, q]</code> gradient tensor (e.g., with <code>torch.stack</code>).</p>

sguysc: <p>Ok, thank you, I see. So in this case, I would have a couple of follow-up questions.<br>
If I understand you correctly, I would have to create dy/dx[k,i,j] step by step:</p>
<ol>
<li>
<p>This would involve taking the gradient of y[k] with respect to every element of x[i,j].<br>
If I define x as in the above code, x[i,j].grad is None. If I manually create x11,x12 … and then construct the x from it using stack, then I can get the gradient, but it is very gawky. Is there a way to define x as a matrix but then compute the derivative of each individual matrix item?</p>
</li>
<li>
<p>Would I then call y[k].backward() every time before looping through all (i,j) to get the gradients?</p>
</li>
</ol>
<p>Updated code with the questions in the comments:</p>
<pre><code class="lang-auto">import torch

n, p, q = 9, 2, 3

# this is the gawky part
x11 = torch.tensor(1.1815, requires_grad=True)
x12 = torch.tensor(-0.2189, requires_grad=True)
x13 = torch.tensor(-0.9923, requires_grad=True)
x21 = torch.tensor(0.2705, requires_grad=True)
x22 = torch.tensor(0.9501, requires_grad=True)
x23 = torch.tensor(-0.4003, requires_grad=True)
X_lst = [[x11, x12, x13], [x21,x22,x23]]

x1 = x11
x1 = torch.hstack((x1, x12))
x1 = torch.hstack((x1, x13))
x2 = x21
x2 = torch.hstack((x2, x22))
x2 = torch.hstack((x2, x23))
x  = torch.vstack((x1,x2))
# just to get the x

A = torch.randn(q, p)
z = torch.randn(q, 1)

y = None
for k in range(int(n/q)):
    if(y is None):
        f = A@x
        y = f @ z
    else:
        f = A@x @ f
        y = torch.vstack( ( y, f @ z ) )

# need to do this? here?
# for k in range(n):
#     y[k].backward(retain_graph=True)

dY_dX = torch.zeros((n,p,q))
for i in range(p):
    for j in range(q):
        for k in range(n):
            # or do it here?
            y[k].backward(retain_graph=True)
            xx = X_lst[i][j]
            xx.retain_grad()
            dyk_dxij = xx.grad.clone()
            xx.grad.zero_()

            # because x[i,j].grad gives me None
            # dyk_dxij = x[i,j].grad

            dY_dX[k, i, j] = dyk_dxij


print('dydx with autograd: \n', dY_dX)
</code></pre>
<p>Is this the way to go?</p>

eqy: <p>I think something simpler can be done e.g.,</p>
<pre><code class="lang-auto">import torch

n, p, q = 9, 2, 3
x = torch.randn(p, q, requires_grad=True)
A = torch.randn(q, p)
z = torch.randn(q, 1)

f = A@x
y = None
for k in range(int(n/q)):
    if(y is None):                                                                                                                  y = f @ z
    else:                                                                                                                           y = torch.vstack( ( y, f @ z ) )                                                                                                                                                                                                                    # prepare for next time step                                                                                                f = (A@x) @ f
dydx = list()
for i in range(n):
    x.grad = None
    y[i, 0].backward(retain_graph=True)
    dydx.append(x.grad.clone())
dydx = torch.stack(dydx)
print('dydx with autograd: \n', dydx)
print(dydx.shape)
</code></pre>

sguysc: <p>This works very well, thank you, but I have another follow-up question.<br>
suppose now that y is not a vector (<em>n</em>), but a matrix (<em>n,n</em>). So now I need to calculate:</p>
<pre><code class="lang-auto">for k in range(n):
  for l in range(n):
    x.grad = None
    y[k,l].backward(retain_graph=True)
    dydx[k, l, :, :] = x.grad.clone()
</code></pre>
<p>It works, but it’s running really slow, even for a relatively low n (=80). I guess it’s because it is re-calculating the whole computation graph for every element <em>k,l</em> ?<br>
Is there any way to make it faster?</p>

eqy: <p>Yes, calling backward 6400 times is likely to be slow. I don’t know the best solution here as it is not a use case I am familiar with but it might be useful to check the amount of overhead that is due to Python.</p>