How to use autograd in the C++ api to compute a gradient of a multivalued function?

username: tmaric
cooked: <p>I would like to calculate y = sin(x), where x is a vector of length N and compute the corresponding derivative vector y’ = (dy_i / dx_i), i = 1,2,3,…,N using autograd.</p>
<p>This is what I ended up doing:</p>
<pre><code class="lang-auto">    // Input vector x 
    auto x = torch::linspace(
        0, M_PI, 100, torch::requires_grad()
    ); 

    // sin(x) as a vector-valued function 
    auto sinx = torch::sin(x); 
    // backpropagate so that the gradient value is stored at x 
    sinx.backward(torch::ones_like(x));
    // x.grad() is dsin(x_i)/dx_i for some reason and not sinx.grad() 
    auto xgrad = x.grad(); 
    // sin'(x) = cos(x) for comparison
    auto cosx = torch::cos(x); 

</code></pre>
<p>and it seems to work:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7a108105903674444c8ae6ac1a008ca6f1b7a923.png" data-download-href="https://discuss.pytorch.org/uploads/default/7a108105903674444c8ae6ac1a008ca6f1b7a923" title="autograd-derivative"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7a108105903674444c8ae6ac1a008ca6f1b7a923_2_690x443.png" alt="autograd-derivative" data-base62-sha1="hpPBSOcQc4TbkECeN9bWprIkjZh" width="690" height="443" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7a108105903674444c8ae6ac1a008ca6f1b7a923_2_690x443.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7a108105903674444c8ae6ac1a008ca6f1b7a923_2_1035x664.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/a/7a108105903674444c8ae6ac1a008ca6f1b7a923.png 2x" data-dominant-color="F6F7F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">autograd-derivative</span><span class="informations">1074×691 42.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is this correct and are there other ways (<code>torch::autograd::grad</code>) to calculate the gradient?</p>
<p>I have two questions.</p>
<ol>
<li>
<p>Why do I have to give the argument <code>torch::ones_like(x)</code> to the <code>backward</code> function of <code>sinx</code> ? I understand the backpropagation in a network, there’s the output given by the model, and the difference between the output and the target is proportional to the gradient at this layer with respect to the weights and biases, this is used to correct the network parameters, etc. Here there is simply x → sin(x),there aren’t any weights and biases in this “network” (AD graph).</p>
</li>
<li>
<p>x → sin(x) is an acyclic directed graph, so why is the gradient of sin(x) stored at x? Why not at the sinx node? x.grad() somehow doesn’t intuitively mean sin’(x). Is x.grad() here the gradient of the root node (sinx) with respect to x, or did I misunderstand something?</p>
</li>
</ol>

username: albanD
cooked: <p>Hi,</p>
<p>I think the best way to see what happens here is to write the Jacobian of your function and see what happens when you do the vector jacobian product (backward pass).<br>
For your sin function, it just applies the sin element-wise to the input. So the Jacobian matrix is a diagonal matrix with on each diagonal element being the gradient of a single y_i wrt to x_i.</p>
<p>Now when you do the backward pass with a vector of all ones (from your <code>torch::ones_like(x)</code>). Then you do a matrix product between a vector of ones and a diagonal matrix.<br>
This will return to you a vector containing the diagonal of the matrix.<br>
And this new vector happens to be exactly what you’re looking for.</p>
<p>Note that this won’t be true anymore as soon as your Jacobian is not diagonal anymore though!</p>

username: tmaric
cooked: <p>Thanks a lot for the hint! I thought that a full Jacobian is always computed by backward, as described in the documentation on <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#gradients" rel="noopener nofollow ugc">Vector Calculus using <code>autograd</code></a>. Is there any documentation besides the source code of <code>backward</code> that describes the calculation of the Jacobian?</p>

username: albanD
cooked: <p>That documentation that you linked explicitly states that it computes a vector jacobian product <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>If you want to get the full jacobian in general, you will need to do as many backward as there are outputs to your function.<br>
In the special case where it is diagonal though (like here), you can compute it in one go as done in your example.</p>

username: tmaric
cooked: <p>Cool, thanks!</p>
<p>So reading the line in the documentation linked above, that states</p>
<blockquote>
<p>Equivalently, we can also aggregate Q into a scalar and call backward implicitly, like <code>Q.sum().backward()</code> .</p>
</blockquote>
<p>This is just a trick to use the <code>sum</code> to have a real-valued output function and not vector-valued output function, and by doing so simplify the call to backward and populate grad objects with data?</p>

username: albanD
cooked: <p>Doing the sum() like that is the same as calling backward with a vector full of ones like you do.</p>
<p>But why it works is because it makes the overall function have a single output and so the overall jacobian has a single row. And multiplying it by a vector of [1] will return that row in one backward pass.</p>

username: tmaric
cooked: <p>I played a bit with <code>torch::autograd::grad</code>.</p>
<p>if x,y in R^3 and f = dot(x,y), then f : R^3 -&gt; R.</p>
<p>f = x1y1 + x2y2 + x3y3</p>
<p>Now I want grad_x f (sensitivity of f with respect to x), which is a <a href="https://en.wikipedia.org/wiki/Matrix_calculus#Derivatives_with_vectors" rel="noopener nofollow ugc">scalar by vector derivative</a>, and I expect to get</p>
<p>grad_x f = [ df / dx1 df/dx2 df/dx3 ]^T = [y1 y2 y3]^T</p>
<p>which makes sense because dot(x,y) is linear in x, and y is kept constant with grad_x f.</p>
<p>Here’s the code with x = y,</p>
<pre><code class="lang-auto">    auto x = torch::linspace(0, 1, 3, torch::requires_grad()); 
    auto y = torch::linspace(0, 1, 3, torch::requires_grad()); 
    auto f = dot(x,y); 

    std::cout &lt;&lt; "x = \n" &lt;&lt; x &lt;&lt; std::endl;
    std::cout &lt;&lt; "y = \n" &lt;&lt; y &lt;&lt; std::endl;
    std::cout &lt;&lt; "f = dot(x,y) = " &lt;&lt; f &lt;&lt; std::endl;
    
    auto grad_f = torch::autograd::grad(
        {f}, {x}, {torch::ones_like(x)}
    );

    std::cout &lt;&lt; "grad_f = \n" &lt;&lt; grad_f &lt;&lt; std::endl; 
</code></pre>
<p>and the output for grad_f is</p>
<pre><code class="lang-auto">x = 
 0.0000
 0.5000
 1.0000
[ CPUDoubleType{3} ]
y = 
 0.0000
 0.5000
 1.0000
[ CPUDoubleType{3} ]
f = dot(x,y) = 1.25
[ CPUDoubleType{} ]
grad_f = 
 0.0000
 1.5000
 3.0000
[ CPUDoubleType{3} ]

</code></pre>
<p>which is not y, why? I seem to still be misunderstanding the J^T v calculation done in autograd.</p>

username: albanD
cooked: <p>Is the code you shared actually running? The grad output you give should be the same size as the output so f here and not x.<br>
I am not sure why this actually runs without error…</p>

username: tmaric
cooked: <p>Yep, it’s running. The grad output should be the same as x… f is linear in x_i components. Gradient of a scalar function is a vector. Vector derivative of a scalar function is a vector…</p>

username: albanD
cooked: <p>The grad input should be the same size as the input (x here)<br>
But the grad output should be the same size as the output (scalar f here)</p>

username: tmaric
cooked: <p><img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I think I have to read more about reverse mode automatic differentiation…</p>

username: albanD
cooked: <p>In this case, if you have your jacobian as a 2D matrix of size (nb_output, nb_input). And backward mode AD allows you to compute the vector jacobian product between a vector of size nb_output (grad output) with the jacobian to obtain a vector of size nb_input (grad input)</p>

username: tmaric
cooked: <p>If f  = dot(x,y), then f = f (x,y), so the Jacobian should be J = [[df/dx1, df/dx2, df/dx3],[df/dy1, df/dy2, df/dy3]], if I want to compute the grad with output f and input {x,y}, right?</p>

username: albanD
cooked: <blockquote>
<p>If f = dot(x,y), then f = f (x,y), so the Jacobian should be J = [[df/dx1, df/dx2, df/dx3],[df/dy1, df/dy2, df/dy3]]</p>
</blockquote>
<p>No <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
To be able to write the jacobian as a 2D matrix, you need all inputs and outputs to be 1D Tensors.<br>
So you will need to have a single output that is the concatenation of x and y. And then you can write the Jacobian as  [[df/dx1, df/dx2, df/dx3, df/dy1, df/dy2, df/dy3]]</p>

username: tmaric
cooked: <p>Thanks! Can you point me to a source of documentation beyond what I’ve linked, that describes how exactly <code>autograd::grad</code> works? Should I go into the source code?</p>
<p>Here’s one more even simpler example</p>
<pre><code class="lang-auto">        std::vector&lt;double&gt; e {1, 0, 0}; 
        auto x = torch::from_blob(e.data(),3,torch::requires_grad());
        auto f = dot(x,x); 
        std::cout &lt;&lt; "x = \n" &lt;&lt; x &lt;&lt; std::endl;
        std::cout &lt;&lt; "f = dot(x,x) = " &lt;&lt; f &lt;&lt; std::endl;

        auto partial_f_x = torch::autograd::grad({f}, {x});
        std::cout &lt;&lt; "partial_f_x = \n" &lt;&lt; partial_f_x &lt;&lt; std::endl; 

</code></pre>
<p>with the output:</p>
<pre><code class="lang-auto">[ CPUDoubleType{1} ]
x = 
 1
 0
 0
[ CPUDoubleType{3} ]
f = dot(x,x) = 1
[ CPUDoubleType{} ]
partial_f_x = 
 2
 0
 0
[ CPUDoubleType{3} ]

</code></pre>
<p>so if x in R^3, partial_x (dot(x,x))=2x.  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: albanD
cooked: <aside class="quote no-group" data-username="tmaric" data-post="15" data-topic="108753">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/f19dbf/48.png" class="avatar"> tmaric:</div>
<blockquote>
<p>Can you point me to a source of documentation beyond what I’ve linked, that describes how exactly <code>autograd::grad</code> works?</p>
</blockquote>
</aside>
<p>autograd.grad does reverse mode automatic differentiation (also called backpropagation), between the output and input that are given, using <code>grad_output</code> as first gradient (which default to 1 for scalar outputs).<br>
You can find some work in progress doc here: <a href="https://alband.github.io/doc_view/notes/autograd.html" class="inline-onebox">Autograd mechanics — PyTorch master documentation</a></p>
<blockquote>
<p>Should I go into the source code?</p>
</blockquote>
<p>I don’t think that will help as it involves quite a lot of components and they interact in non obvious ways</p>
<blockquote>
<p>so if x in R^3, partial_x (dot(x,x))=2x. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
</blockquote>
<p>Yes that works as expected.<br>
Note that for a first example, I would avoid re-using the same variable as it introduces implicit accumulations in the backward pass.</p>

username: tmaric
cooked: <p>Thanks a lot! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I’ll read the in-progress doc around 20-30 times, maybe that’ll help hehe.  I’ve read the PyTorch papers, one is pointing towards the AD book “Evaluating Derivatives<br>
Principles and Techniques of Algorithmic Differentiation, Second Edition” by Andreas Griewank and Andrea Walther. I’ll try to pick up the reverse mode AD from there also.</p>
<p>I want to work with higher derivatives of the models, so I really must be 100% sure about what grad is doing and how to build up on that.</p>

username: pancras
cooked: <p>Hello,my name is Pancras,I see you solve a related problem.So I want to ask you for help.</p>
<p>When I was training the model, loss always appeared nan. I don’t know what is the reason for nan? Can you help me analyze it? I used code monitoring and reported an error:RuntimeError: Function ‘CudnnBatchNormBackward’ returned nan values in its 0th output.looking forward to your reply</p>