Is it possible to look up the gradient of native functions?

username: tillahoffmann
cooked: <p>Is it possible to obtain the gradient function for builtin functions, such as <code>tanh</code> similar to the <a href="https://pytorch.org/docs/stable/generated/torch.autograd.Function.backward.html#torch.autograd.Function.backward" rel="noopener nofollow ugc"><code>backward</code> method of <code>Function</code> objects</a>? E.g. for <code>tanh</code>, I would like to obtain a function that evaluates <code>lambda x: 1 - x.tanh().square()</code>.</p>

username: googlebot
cooked: <p>afaik, there is no way to obtain a symbolic form / python code. depending on your needs, you can generate code with sympy or fully fledged CAS software, but that’s somewhat manual approach.</p>

username: tillahoffmann
cooked: <p>Derivatives for elementary functions are available in the source, e.g. the tanh derivative is implemented <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cpu/BinaryOpsKernel.cpp#L814" rel="noopener nofollow ugc">here</a>. I’m just not sure how to get at it from the python code.</p>
<p>In other words, I don’t have a requirement for a symbolic form or a python representation, just a black-box function that evaluates the gradient.</p>

username: googlebot
cooked: <ol>
<li>
</li>
</ol>
<pre><code class="lang-auto">torch.ops.aten.tanh_backward(torch.ones(3), torch.tanh(torch.full((3,), 2.0)))
</code></pre>
<ol start="2">
<li>
</li>
</ol>
<pre><code class="lang-auto">f = lambda x : x.tanh()
def df(x, grad_out=None):
	x = x.detach().clone().requires_grad_()
	return torch.autograd.grad(f(x),x, grad_out or torch.ones_like(x))
</code></pre>

username: tillahoffmann
cooked: <p>Neat, the <code>tanh_backward</code> is what I’m looking for. Unfortunately, bindings for <code>sin_backward</code> don’t exist (because it’s just <code>cos</code>). But of course pytorch “knows” that <code>cos</code> is the derivative of the builtin function <code>sin</code>. What I’d like to do is extract that knowledge. Pseudocode below</p>
<pre><code class="lang-python">def get_grad(builtin_function): ...
    """
    Returns a function that evaluates the gradient of the 
    `builtin_function`. Raises `TypeError` if the function
    does not have a registered gradient.

    &gt;&gt;&gt; get_grad(th.sin)  # Returns a function.
    &gt;&gt;&gt; get_grad(lambda x: x.exp().sin())  # Raises TypeError.
    """
</code></pre>

username: googlebot
cooked: <p>well, pytorch does code generation from tools/autograd/derivatives.yaml:</p>
<pre><code class="lang-auto">- name: sin(Tensor self) -&gt; Tensor
  self: grad * self.cos().conj()

- name: tanh(Tensor self) -&gt; Tensor
  self: tanh_backward(grad, result)
</code></pre>
<p>note that some backward functions of y=F(x) need x (self) and some y (result) as input</p>
<p>Generated Functions are in torch/csrc/autograd/generated/Functions.cpp, but I’m not sure whether their interface is exported to python.</p>

username: AlphaBetaGamma96
cooked: <p>If you want to be able to differentiate a function directly within PyTorch you’ll want to have a look at the <a href="https://github.com/pytorch/functorch" rel="noopener nofollow ugc">FuncTorch</a> library and use either <code>grad</code> or <code>jacrev</code>/<code>jacfwd</code> commands on a function to return a functional form of the derivative.</p>
<p>Here’s an example,</p>
<pre><code class="lang-auto">import torch
from functorch import grad, vmap

x = torch.randn(10) #samples

def f(x):
    return torch.sin(x)

grad_f = grad(f)                     #differentiate f(x) w.r.t x
df_dx = vmap(grad_f, in_dims=(0))(x) #vmap over samples 

exact = torch.cos(x) #analytical result

print("Check: ",torch.allclose(exact, df_dx)) #compare results (returns True)
</code></pre>