Directly getting gradients

username: yoonholee
cooked: <p>Is there a way for me to directly compute the gradient of a variable w.r.t. another variable, like tf.gradients()?</p>

username: Tudor_Berariu
cooked: <p>Let’s say you compute variable <code>y</code> as a function of variable <code>x</code>.</p>
<p>When you call <code>y.backward(dL_dy)</code> you’ll get the value of <code>dL_dx</code> in <code>x.grad</code> (which is <code>dL_dy * dy_dx</code>). If you just put a tensor full of ones instead of <code>dL_dy</code> you’ll get precisely the gradient you are looking for.</p>
<pre><code>import torch
from torch.autograd import Variable

x = Variable(torch.ones(10), requires_grad=True)
y = x * Variable(torch.linspace(1, 10, 10), requires_grad=False)
y.backward(torch.ones(10))
print(x.grad)
</code></pre>
<p>produces</p>
<pre><code>Variable containing:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
[torch.FloatTensor of size 10]</code></pre>

username: yoonholee
cooked: <p>Thanks for the reply:)<br>
I should have been more precise with my question, though.</p>
<p>I’m trying to look for a way to compute second-order derivatives.<br>
In tensorflow, I would do something like:<br>
tf.gradients(tf.gradients(y, x), x)</p>
<p>As far as I can tell, there’s no way of accomplishing this using y.backward(). That’s why I was wondering if there was a standalone function that differentiated.</p>

username: alexis-jacq
cooked: <p>The second order derivative of pytorch functions are not “known” by pytorch, while the first order are coded “by hand” in the <code>backward</code> methods. If you want the second derivative of your whole graph, I’m afraid that you need at least to extend all the functions you use with a method similar to <code>backward</code> for second order derivative, by hands.</p>

username: smth
cooked: <p>we are working on enabling this. This will be in the next major pytorch release 2 months away. It wont be a full 2nd derivative but it will be the Hessian-vector product (whether it’s tf.gradients or in pytorch). We are tracking this feature as double-backprop / double backwards.</p>

username: vsahil
cooked: <p><a class="mention" href="/u/smth">@smth</a> has this been implemented? If yes, how can I use it?</p>

username: BruceShakeham
cooked: <p>Dear PyTorch dev,</p>
<p>I am a professor in one of the US Universities working on data-driven scientific computing using PyTorch now. The ability to get gradients allows for some amazing new scientific computing algorithms. You can see from <a href="https://www.sciencedirect.com/science/article/pii/S0021999118307125?via%3Dihub" rel="nofollow noopener">this paper</a>, and this <a href="https://github.com/maziarraissi/PINNs/blob/master/main/continuous_time_identification%20(Navier-Stokes)/NavierStokes.py" rel="nofollow noopener">github link</a> (e.g., starting on line 121, “u = tf.gradients(psi, y)”), the ability to get gradients between two variables is in Tensorflow and is becoming one of the major differentiator between platforms in scientific computing. This paper is published in 2019 and has gained 168 citations, very high in the realm of scientific computing. I see many people are following suite and using this algorithm. We are now using PyTorch but might be forced to migrate to TF if such functionality is not available with PyTorch. Really hoping it could be released…</p>

username: yf225
cooked: <p><a class="mention" href="/u/alband">@albanD</a> Would the work involved in <a href="https://github.com/pytorch/pytorch/issues/30632" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/30632</a> resolve this feature request?</p>

username: albanD
cooked: <p>Yes it will. Hopefully this will be done in the next weeks !</p>

username: amitoz
cooked: <p>Is this issue has be solved?</p>

username: albanD
cooked: <p>Yes,<br>
It was launched with v1.5</p>

username: rodsveiga
cooked: <p>Is it working? I tried that with v1.6.</p>
<blockquote>
<blockquote>
<blockquote>
<p>x = Variable(torch.Tensor([1, 2, 3, 4, 5, 6]), requires_grad= True)<br>
y = Variable(torch.Tensor([2, 4, 6, 8, 10, 12]), requires_grad= False)<br>
torch.autograd.grad(y, x)<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “anaconda3/envs/updated/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py”, line 190, in grad<br>
return Variable._execution_engine.run_backward(<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn`</p>
</blockquote>
</blockquote>
</blockquote>
<p>Same message with older versions.</p>

username: albanD
cooked: <p>Hi,</p>
<p>This is quite unrelated. The problem here is just that <code>y</code> does not require gradients. So no gradient can be computed.</p>