Is retain_grad() supported in new C++ API?

username: cndn
cooked: <p>I want to use retain_grad to get the gradient for non-leaf variables in C++. Hook registration is implemented in <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/python_cpp_function.cpp" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/python_cpp_function.cpp</a>. But I didn’t find retain_grad() implementation in C++. Does it mean I need to implement it with registerFunctionHook()? Thanks!</p>

username: SimonW
cooked: <p>Yes, but as you said, you would need to manually create a <code>FunctionPreHook</code> (defined <a href="https://github.com/pytorch/pytorch/blob/17776db2ee8cb9248233c22f33810345859352b8/torch/csrc/autograd/function_hook.h#L12">here</a>) and then use <code>variable.add_hook</code> (see <a href="https://github.com/pytorch/pytorch/blob/17776db2ee8cb9248233c22f33810345859352b8/torch/csrc/autograd/variable.h#L551">here</a>) to register it.</p>

username: cndn
cooked: <p>Thanks Simon will do that! Just curious do you have plans for having a retain_grad() explicitly in C++? as I imagine it could be a very useful helper hook…LMK if I can be of any help here.</p>

username: cndn
cooked: <p>I implemented a FunctionPreHook and use variable.add_hook to “register”, but at runtime the hook is not invoked at all. I checked the length of pre_hooks_ for Functions at runtime and they are all 0s.</p>
<p>This is where I checked length of pre_hooks_, inside engine.cpp</p>
<pre><code class="lang-cpp">static variable_list call_pre_hooks(Function&amp; fn, variable_list inputs) {
  std::cout &lt;&lt; fn.pre_hooks().size() &lt;&lt; std::endl;
  for (const auto&amp; hook : fn.pre_hooks()) {
    inputs = (*hook)(inputs);
  }
  return inputs;
}
</code></pre>
<p>I checked code under csrc/autograd but didn’t find where add_pre_hook() is called in cpp, or where hooks in variables are added to pre_hooks_ in Function.</p>
<p>Did I miss something? Thanks!</p>
<p>This is a minimal example, where “here” is expected to be printed out twice.</p>
<pre><code class="lang-cpp">using namespace torch::autograd;
struct DummyHook : public FunctionPreHook {
  DummyHook(Variable* x) : v(x) {};
  variable_list operator()(const variable_list&amp; grads) {
    std::cout &lt;&lt; "here" &lt;&lt; std::endl;
    return grads;
  }
  Variable* v;
};

int main(int argc, char** argv) {
  torch::Tensor x = torch::tensor({1.0}, at::requires_grad()); // CPUDoubleType
  Variable y = x * x;
  y.add_hook(std::shared_ptr&lt;DummyHook&gt;(new DummyHook(&amp;y)));
  torch::Tensor z = 2 * y;
  for (auto hook: y.hooks()) {
    (*hook.get())({});
  }
  z.backward();
}
</code></pre>

username: SimonW
cooked: <p>I think <code>add_hook</code> only works on the leaves. For intermediate variables you can try <code>variable.grad_fn().add_pre_hook</code></p>

username: cndn
cooked: <p>add_pre_hook in grad_fn works, thanks Simon!</p>

username: capuzz
cooked: <p>Hi Ning,<br>
how do you use add_pre_hook?<br>
I was trying to replicate your code but if I print y.grad() after the backward function I got</p>
<blockquote>
<p>[ Tensor (undefined) ]</p>
</blockquote>
<p>How can I resolve this problem?<br>
Thanks</p>