Losing the grad

username: junyuphybies
cooked: <p>I am a beginner of pytorch, and wish to use the autograd function of the system. I wish to ask the following problem.</p>
<p>Now I wish to implement a list of variables that contain another variable with derivatives. I wish the list could maintain the derivative. For instance, here is the code:</p>
<p>import numpy as np<br>
import torch<br>
import numpy.linalg as linalg</p>
<p>Now I define a variable phi with derivative:<br>
phi =(torch.randn(2))<br>
phi.requires_grad = True</p>
<p>And I wish to define a function such that<br>
def Afunction(phi):<br>
return torch.tensor([torch.cos(phi[0]),torch.sin(phi[1])])</p>
<p>I find when I define it in the above way, the derivative of A is lost.</p>
<p>print(phi)<br>
tensor([ 0.2366, -0.5896], requires_grad=True)<br>
print(Afunction(phi))<br>
tensor([ 0.9721, -0.5561])</p>
<p>There might be some other ways to make the input matrix better. However, I have to do it in this way since the actual list I have, beyond this example, [[…]], is printed from some other codes, and it is very long. I have no other ways to transform it to python.</p>
<p>Is there any resolution to this problem?</p>

username: KFrank
cooked: <p>Hi junyuphybies!</p>
<aside class="quote no-group" data-username="junyuphybies" data-post="1" data-topic="118084" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/d2c977/48.png" class="avatar"> junyuphybies:</div>
<blockquote>
<p>def Afunction(phi):<br>
return torch.tensor([torch.cos(phi[0]),torch.sin(phi[1])])</p>
<p>I find when I define it in the above way, the derivative of A is lost.</p>
</blockquote>
</aside>
<p>I am not an expert on autograd, but (if I understand your question)<br>
I don’t believe that there is a convenient way of doing what you want.</p>
<p>The problem, as I see it, is that the <code>grad_fn</code> of a tensor is attached<br>
to the tensor as a whole, and is not attached to individual elements<br>
of the tensor.</p>
<p>The proper way to accomplish you goal would be to write a <em>custom<br>
autograd function</em> that has its own <code>backward()</code> method that knows<br>
how to compute the derivative of its <code>forward()</code> method.</p>
<p>See <a href="https://pytorch.org/docs/1.8.1/notes/extending.html#extending-torch-autograd" rel="noopener nofollow ugc">Extending torch.autograd</a> for the details.</p>
<p>However, in the special case that you really do want <code>cos()</code> and <code>sin()</code><br>
(and they’re not just for the purposes of a simple example), you can<br>
use pytorch’s (recent) support for complex autograd, together with<br>
the fact that the (complex) exponential function has <code>cos()</code> and <code>sin()</code><br>
inside of it.</p>
<p>Here is a script that implements this approach:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

def Afunction (phi):   # your version
    return torch.tensor ([torch.cos (phi[0]), torch.sin (phi[1])])

def Bfunction (phi):   # version using complex exp() with autograd
    z = torch.exp (1.0j * phi)
    w = torch.tensor ([1.0 + 0.0j, 0.0 - 1.0j]) * z
    f = w.real
    return f

phi = torch.tensor ([ 0.2366, -0.5896], requires_grad = True)

print (phi)
print (Afunction (phi))
print (Bfunction (phi))

Bfunction (phi).backward (gradient = torch.tensor ([1.0, 1.0]))
print (phi.grad)

print (-torch.sin (phi[0]), torch.cos (phi[1]))   # check phi.grad
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">1.9.0.dev20210412
tensor([ 0.2366, -0.5896], requires_grad=True)
tensor([ 0.9721, -0.5560])
tensor([ 0.9721, -0.5560], grad_fn=&lt;SelectBackward&gt;)
tensor([-0.2344,  0.8312])
tensor(-0.2344, grad_fn=&lt;NegBackward&gt;) tensor(0.8312, grad_fn=&lt;CosBackward&gt;)
</code></pre>
<p>Good luck.</p>
<p>K. Frank</p>