How to compute the gradient of a function without explicit expression?

Sishou_Zhou: <p>I am trying to compute the gradient of mittag_leffler function with respect to its first two parameters. When I ran the below codes:</p>
<pre><code class="lang-auto">import torch
import mittag_leffler as ml
import numpy as np
from torch.autograd import Variable


# Create a tensor
x = torch.tensor([0.,1.,2.,3.,4.,5.,6.,7.,8.,9.], requires_grad=True)
b = torch.tensor([.1,.2,.3,.4,.5,.6,.7,.8,.9,0.9], requires_grad=True)

# Define a function

y = torch.sum(torch.tensor(ml.ml(-x,b)))

# Compute gradients
y.backward()

# Access gradients
print(x.grad)
print(b.grad)
</code></pre>
<p>it shows error: RuntimeError: Can’t call numpy() on Variable that requires grad. Use var.detach().numpy() instead.<br>
When I add detach as below:</p>
<pre><code class="lang-auto">import torch
import mittag_leffler as ml
import numpy as np
from torch.autograd import Variable


# Create a tensor
x = torch.tensor([0.,1.,2.,3.,4.,5.,6.,7.,8.,9.], requires_grad=True)
b = torch.tensor([.1,.2,.3,.4,.5,.6,.7,.8,.9,0.9], requires_grad=True)

# Define a function

x = x.detach()
b = b.detach()
y = torch.sum(torch.tensor(ml.ml(-x,b)))

# Compute gradients
y.backward()

# Access gradients
print(x.grad)
print(b.grad)
</code></pre>
<p>it shows error: RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn.<br>
So, what should I do next ? Any help or comments are highly appreciated.<br>
the mittag-leffler codes see<br>
<a href="https://github.com/khinsen/mittag-leffler" rel="noopener nofollow ugc">ml</a></p>

ptrblck: <p>Autograd won’t track <code>numpy</code> operations so the issues are expected.<br>
You could try to rewrite the <code>ml</code> function in PyTorch. If that’s not possible, you would need to implement a custom <code>autograd.Function</code> including the <code>backward</code> pass as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a>.</p>

Sishou_Zhou: <p>I want to " implement a custom <code>autograd.Function</code> including the <code>backward</code> pass" and try the codes below:</p>
<pre><code class="lang-auto">import torch
import math
import mittag_leffler as ml

class CustomML(torch.autograd.Function):

    @staticmethod
    def forward(ctx, x, beta):
        ctx.save_for_backward(x,beta)
        return ml.ml(-x,beta)

    @staticmethod
    def backward(ctx, grad_output):
        gbeta = ((ml.ml(x,beta+0.00001)-ml.ml(x,beta))/0.00001 + (ml.ml(x,beta-0.00001)-ml.ml(x,beta))/(-0.00001))/2
        input, = ctx.saved_tensors
        return grad_output * gbeta

dtype = torch.float
device = torch.device("cpu")

x = torch.linspace(0, 3, 2000, device=device, dtype=dtype)
beta_o = torch.linspace(0.8, 1.0, 2000, device=device, dtype=dtype)
y = ml.ml(-x, beta_o)

beta = torch.full((), 0.7, device=device, dtype=dtype, requires_grad=True)

learning_rate = 5e-6
for t in range(2000):
    # To apply our Function, we use Function.apply method. We alias this as 'P3'.
    CML = CustomML.apply

    y_pred = CML(-x,beta)

    # Compute and print loss
    loss = (y_pred - y).pow(2).sum()
    if t % 100 == 99:
        print(t, loss.item())

    # Use autograd to compute the backward pass.
    loss.backward()

    # Update weights using gradient descent
    with torch.no_grad():
        
        beta -= learning_rate * beta.grad
        
        # Manually zero the gradients after updating weights
        beta.grad = None

print(f'beta = {beta}')
</code></pre>
<p>but the error is the same :  RuntimeError: Can’t call numpy() on Variable that requires grad. Use var.detach().numpy() instead.<br>
Could you help me improve the codes above ?</p>

ptrblck: <p>Assuming the error is raised in <code>ml.ml(-x,beta)</code>, call <code>x.detach().numpy()</code> before passing it to <code>ml</code> as the error message suggests.</p>