how to get gradients that still have requires_grad True

yash11: <p>Let <code>w</code> and <code>phi</code> be two parameters</p>
<pre><code class="lang-auto">w = Parameter(T.tensor([2.2]))
phi = Parameter(T.tensor([1.5]))
wp = w*phi
wp.backward()
grd = phi.grad
print(grd)
</code></pre>
<p>Printed:</p>
<pre><code class="lang-auto">tensor([2.2000])
</code></pre>
<p>I want:</p>
<pre><code class="lang-auto">tensor([2.2000], requires_grad=True)
</code></pre>
<p>i.e. I want <code>phi.grad</code> as <code>w</code> which is a parameter of larger network, should have <code>requires_grad=True</code> so I can do</p>
<pre><code class="lang-auto">grd.backward()
w.grad
</code></pre>
<p>I don’t know how to seperate these two computaion graph.</p>

soulitzer: <p>If I’m understanding correctly, you want to compute the second-order gradients wrt w. So you’d like to have <code>phi.grad</code> itself to have an autograd graph accumulating into <code>w</code> (and thus have <code>requires_grad=True</code>).</p>
<p>You could do this by doing <code>wp.backward(create_graph=True)</code>.</p>

yash11: <p>Thanks for your reply.</p>
<p>I actually want gradient of <code>wp</code> only wrt <code>phi</code>, so this worked for me,</p>
<pre><code class="lang-auto">w = Parameter(T.tensor([2.2]))
phi = Parameter(T.tensor([1.5]))
wp = w*phi
grd = T.autograd.grad(wp, phi, create_graph=True)[0]
print(grd)
grd.backward()
w.grad
print(w.grad)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">tensor([2.2000], grad_fn=&lt;MulBackward0&gt;)
tensor([1.])
</code></pre>
<p>Using modified last method</p>
<pre><code class="lang-auto">w = Parameter(T.tensor([2.2]))
phi = Parameter(T.tensor([1.5]))
wp = w*phi
wp.backward(create_graph=True)
grd = phi.grad
print(grd)
grd.backward()
w.grad
print(w.grad)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">tensor([2.2000], grad_fn=&lt;CopyBackwards&gt;)
tensor([2.5000], grad_fn=&lt;CopyBackwards&gt;)
</code></pre>
<p>I don’t know what is going on with last method.<br>
Also I found a Quote, which suggest against using <code>.grad</code> in such cases.</p>
<aside class="quote no-group" data-username="albanD" data-post="4" data-topic="71797">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"><a href="https://discuss.pytorch.org/t/second-order-derivatives-of-loss-function/71797/4">Second order derivatives of loss function</a></div>
<blockquote>
<p>If you have a single input and single output, you want to do the following:<br>
(note that using <code>.backward()</code> for higher order derivatives is discouraged because the .grad field becomes hard to reason about).</p>
<pre><code class="lang-auto">first_derivative = autograd.grad(loss, x, create_graph=True)[0]
# We now have dloss/dx
second_derivative = autograd.grad(first_derivative, x)[0]
# This computes d/dx(dloss/dx) = d2loss/dx2
</code></pre>
</blockquote>
</aside>

soulitzer: <p>I believe the quote is saying that <code>.backward()</code> is hard to reason about (not <code>.grad()</code>). <code>.grad()</code> is actually the preferred alternative because (by default) it’s more explicit about what inputs its computing gradients for. Its also returns the gradient instead of performing a side effect like updating <code>.grad</code>.</p>