Question about tensors in autograd function

G.M: <p>I have this autograd function.</p>
<pre><code class="lang-auto">class raw(autograd.Function):
    @staticmethod
    def forward(ctx, inp):
        ctx.a = inp * inp + 1
        print(inp.requires_grad, ctx.a.requires_grad)
        return inp * ctx.a.pow(-0.5)

    @staticmethod
    def backward(ctx, grad_output):
        return grad_output * ctx.a.pow(-1.5)
</code></pre>
<p>when I ran this function like <code>raw.apply(tc.randn(1, requires_grad=True))</code>, the output is <code>True False</code>. In this case, <code>ctx.a</code> does not require grad. But don’t tensors that require grad produce tensors that also requires grad? For example</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a=tc.randn(1,requires_grad=True)
&gt;&gt;&gt; b=a*a+1
&gt;&gt;&gt; b.requires_grad
True
</code></pre>
<p>I’m quite confused about what is happening during the process of calling <code>raw.apply</code>(I don’t know where to find the implementation of <code>.apply</code> either).<br>
Can someone explain this process to me?<br>
Thank u very much.</p>

albanD: <p>Hi,</p>
<p>The requires_grad field is used to track operation and Tensors that require gradients. Here since you are inside a autograd.Function. You do no use the autograd anymore to get gradients, you code the backward yourself. So the Tensors that are created in during the forward pass do not require gradients as the autograd engine does not need to track operations on them to compute gradients.</p>

G.M: <p>Thanks for ur reply, that solved my problem.<br>
Just another question, do u know where is the source code of the <code>apply</code> of autograd functions?</p>

albanD: <p>Sure, it is <a href="https://github.com/pytorch/pytorch/blob/29e6f13cd1c309cf3dfd8ba45a6ca199359422b5/torch/csrc/autograd/python_function.cpp#L675-L731">here</a>. The part that disable the gradient is the <code>GradMode(false);</code> that is equivalent to <code>with torch.no_grad()</code> in python.<br>
Let me know if you have more question about this !</p>

G.M: <p>Thanks! Then is there a way implement an autograd function in C++ and use it directly in Python(not writing the forward and backward separately and wrap it in Python)?</p>

albanD: <p>You can use the cpp api equivalent: <code>torch::autograd::Function</code>. And you will need to bind the <code>::apply()</code> method properly to python.</p>
<p>EDIT: You can find <a href="https://github.com/pytorch/pytorch/blob/master/test/cpp/api/autograd.cpp#L127-L142">here</a> an example of the cpp Function. You can then bind its apply method just like any function.</p>

G.M: <p>Thanks. I tried your suggestion, but I got the following compiling error. I used a function based on the one in your example, and the C++ <code>apply</code> worked fine.<br>
Here’s my code for binding the methods:</p>
<pre><code class="lang-auto">namespace py = pybind11;
PYBIND11_MODULE(actv, m) {
	py::class_&lt;CPTanh&gt;(m, "CPTanh")//line 25
		.def_static("forward", &amp;CPTanh::forward)
		.def_static("backward", &amp;CPTanh::backward)
		.def_static("apply",&amp;CPTanh::apply); //line 28

}
</code></pre>
<p>Here’s the error:</p>
<pre><code class="lang-auto">actv.cpp(29): error C2672: 'pybind11::class_&lt;CPTanh&gt;::def_static': no matching overloaded function foun
d
actv.cpp(26): error C2783: 'pybind11::class_&lt;CPTanh&gt; &amp;pybind11::class_&lt;CPTanh&gt;::def_static(const char *
,Func &amp;&amp;,const Extra &amp;...)': could not deduce template argument for 'Func'
</code></pre>
<p>It seems that <code>apply</code> can’t be bind in the common way, do u have any idea about this(I’m quite new in C++)?</p>

albanD: <p>You don’t need to bind the whole class in python. Just the apply function will work. And if you can’t bind <code>CPTanh::apply</code> you can create a local function that then calls it.<br>
Just like the python version, you should never call <code>forward</code> and <code>backward</code> yourself nor instantiate the class. Only the static method apply is needed.</p>

G.M: <p>Thanks, that solved the problem.</p>