Very confused with changes in autograd

username: kgizdov
cooked: <p>Hi,</p>
<p>I’m having a weird problem with autograd in recent PyTorch. I’m trying to make a simple example following the official documentation, but instead of in Python, I want to do it in C++ for basically having a test case. Here’s my code:</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;

int main() {
  if (torch::cuda::is_available()) {
    std::cout &lt;&lt; "CUDA device is available.\n";
  } else {
    std::cout &lt;&lt; "CUDA device is not available.\n";
  }
  try {
    auto test = torch::ones({2, 2}, at::requires_grad()).cuda();
    if (test.device().type() == torch::kCUDA) {
      std::cout &lt;&lt; "We are using CUDA\n";
    } else if (test.device().type() == torch::kCPU) {
      std::cout &lt;&lt; "Oops... we are supposed to run on CUDA GPU, but we are running on CPU.";
    } else {
      std::cout &lt;&lt; "Uuuhm, we are supposed to run on CUDA GPU, but we are not even running on CPU.";
    }
    auto a = torch::ones({2, 2}, torch::TensorOptions().dtype(torch::kFloat).requires_grad(true)).cuda();
    auto b = torch::randn({2, 2}, torch::TensorOptions().dtype(torch::kFloat)).cuda();
    std::cout &lt;&lt; "a: " &lt;&lt; a &lt;&lt; "\n";
    auto c = a + b;
    std::cout &lt;&lt; "c: " &lt;&lt; c &lt;&lt; "\n";
    auto cgrad = c.grad();
    std::cout &lt;&lt; "cgrad : " &lt;&lt; cgrad &lt;&lt; "\n";
    // tried this - doesn't work either
    // torch::Tensor a_grad = torch::tensor({1.0, 0.1, 1.0, 0.01}, torch::TensorOptions().dtype(torch::kFloat)).cuda().view({2,2});
    torch::Tensor a_grad = torch::tensor({1.0, 1.0, 1.0, 1.0}, torch::TensorOptions().dtype(torch::kFloat)).cuda().view({2,2});
    std::cout &lt;&lt; "a_grad: " &lt;&lt; a_grad &lt;&lt; "\n";
    // tried this as well, also doesn't work
    // c.sum().backward(a_grad);  // a.grad() will now hold the gradient of c w.r.t. a.
    c.sum().backward();  // a.grad() will now hold the gradient of c w.r.t. a.
    std::cout &lt;&lt; "a.grad(): "&lt;&lt; a.grad() &lt;&lt; "\n";
    std::cout &lt;&lt; "a_grad: "&lt;&lt; a_grad &lt;&lt; "\n";
    torch::Tensor g_loss = torch::binary_cross_entropy(a, b);
    g_loss.backward();
    std::cout &lt;&lt; "g_loss: " &lt;&lt; g_loss &lt;&lt; "\n";
    std::cout &lt;&lt; "a.grad(): " &lt;&lt; a.grad() &lt;&lt; "\n";
  } catch (const c10::Error&amp; e) {
    // this was the original code (for CPU) that worked fine on PyTorch 1.2
    std::cout &lt;&lt; e.what() &lt;&lt; "\n";
    auto test = torch::ones({2, 2}, at::requires_grad());
    if (test.device().type() == torch::kCPU) {
      std::cout &lt;&lt; "We are using CPU\n";
    } else if (test.device().type() == torch::kCUDA) {
      std::cout &lt;&lt; "Oops... we are supposed to run on CPU, but we are running on CUDA GPU.";
    } else {
      std::cout &lt;&lt; "Uuuhm, we are supposed to run on CPU, but we are not even running on CUDA GPU.";
    }
    auto a = torch::ones({2, 2}, at::requires_grad());
    auto b = torch::randn({2, 2});
    auto c = a + b;
    c.backward(a); // a.grad() will now hold the gradient of c w.r.t. a.
    std::cout &lt;&lt; c &lt;&lt; "\n";
  }
  return 0;
}
</code></pre>
<p>I’ve played around with different versions of it. You can see the CPU specific code section is showing the previous structure I used before PyTorch 1.2 (it’s showing CPU variants, but was basically equivalent for GPU as well). The new style I’m trying to write currently only sits in the GPU section. This is done for illustration purposes here, but I’m actually trying the same for both. This is to account for the change in how autograd deals with passing the dispatch &amp; Jacobian to construct the gradients (<a href="https://github.com/ROCmSoftwarePlatform/pytorch/commit/6e3687341ef211b5ee5c5a3680b49d02475d5377" rel="nofollow noopener">link</a>).</p>
<p>That’s all good, but since then I’ve not been able to get any gradients to work. I am probably doing something wrong, but can’t put my finger on it (or found a weird bug, no idea). The output I get is this:</p>
<pre><code class="lang-auto">$ make -C pytorch/autograd run
make[1]: Entering directory '/home/gizdov/Git/arch-package-tests/pytorch/autograd'
g++ autograd.cpp -I/usr/include/torch/csrc/api/include -I/usr/include/python3.8 -I/usr/include/python3.8m -L/usr/lib/pytorch -L/opt/cuda/lib -lc10 -ltorch -lnvrtc -lcuda -o autograd
./autograd
CUDA device is available.
We are using CUDA
a:  1  1
 1  1
[ CUDAFloatType{2,2} ]
c:  2.8679  0.9870
 0.8103  1.9954
[ CUDAFloatType{2,2} ]
cgrad : [ Tensor (undefined) ]
a_grad:  1  1
 1  1
[ CUDAFloatType{2,2} ]
a.grad(): [ Tensor (undefined) ]
a_grad:  1  1
 1  1
[ CUDAFloatType{2,2} ]
g_loss: 9.25194
[ CUDAFloatType{} ]
a.grad(): [ Tensor (undefined) ]
make[1]: Leaving directory '/home/gizdov/Git/arch-package-tests/pytorch/autograd'
</code></pre>
<p>My gradients are always undefined or I get a runtime error from c10 for passing the wrong dispatch tensor if <code>a_grad</code> is missing or malformed.</p>
<p>Could someone please help?<br>
Thanks.</p>
<p><strong>EDIT</strong>: This is on PyTorch 1.4.1, CUDA 10.2 Compute Capability 5.2, GCC 8.4.0, Python 3.8.2</p>

username: richard
cooked: <p>Please see the comment here: <a href="https://discuss.pytorch.org/t/valueerror-cant-optimize-a-non-leaf-tensor/21751/2" class="inline-onebox">ValueError: can't optimize a non-leaf Tensor?</a>. The main problem you are running into is that<br>
<code>auto a = torch::ones({2, 2}, torch::TensorOptions().dtype(torch::kFloat).requires_grad(true)).cuda(); </code><br>
is NOT a leaf node, so autograd does not accumulate gradients into <code>a.grad</code>. This is because you’re calling <code>.cuda()</code> on the actual leaf node, returning a new tensor. If you directly construct the tensor on CUDA, then you can get gradients on it:</p>
<pre><code class="lang-auto">auto a = torch::ones({2, 2}, torch::TensorOptions().dtype(torch::kFloat).requires_grad(true).device(torch::kCUDA));
auto c = (a + a).sum();
c.backward()
a.grad() // now has something
</code></pre>

username: kgizdov
cooked: <p>Hi,</p>
<p>Thanks. Indeed that solved part of the issue. However, does that mean I cannot get intermediate gradients? For example, I cannot seem to get <code>c = a + b</code> to get anything for <code>c.grad()</code>.</p>

username: albanD
cooked: <p>In python, you can use <code>c.retain_grad()</code> for this to be populated for non-leafs.<br>
Not sure if it is available on the cpp API yet? cc <a class="mention" href="/u/yf225">@yf225</a></p>
<p>Otherwise, you can use autograd::grad and provide the inputs you want gradient for, where you can specify any Tensor that requires gradients (no need for them to be leafs).</p>

username: kgizdov
cooked: <p>Thanks.</p>
<p>Well, I did indeed get it working to a point, but the C++ API does not seem to implement this. Firstly, <code>retain_grad()</code> is not defined. Secondly, when using <code>auto cgrad = torch::autograd::grad({c}, {a, b}, {a_dispatch});</code>, I get the following:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'std::runtime_error'
  what():  the derivative for 'target' is not implemented
</code></pre>
<p>So this is also not implemented. I guess for now I will stick with what works.<br>
Cheers.</p>
<p><strong>EDIT</strong>: wait, no, this seems to be caused by requiring a gradient on <code>b</code> as well to satisfy <code>autograd::grad()</code> call, which then breaks <code>torch::binary_cross_entropy(a, b)</code>. So that is what is not implemented. Not sure how to work around it - maybe give it another <code>const</code> for the time being.</p>

username: yf225
cooked: <p><code>retain_grad()</code> should already work on <code>torch::Tensor</code>, we have a test here: <a href="https://github.com/pytorch/pytorch/blob/64a6faa2c8435a9e96743b078d9f2ae2b7ef1cc3/test/cpp/api/autograd.cpp#L129-L153" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/64a6faa2c8435a9e96743b078d9f2ae2b7ef1cc3/test/cpp/api/autograd.cpp#L129-L153</a></p>