How to self define a backward function for a net in libtorch? (I tested some code, but failed.)

username: Snade_Qian
cooked: <p>I’m using libtorch (the frontend of pytorch in c++) to train a network. To monitor the gradient, I created checker to catch the backward process, but it does not enter the self-defined backward() function.</p>
<p>I compile the libtorch from sources on Ubuntu16.04.</p>
<p>The checkBP class is below.</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;

#include &lt;cstddef&gt;
#include &lt;cstdio&gt;
#include &lt;iostream&gt;
#include &lt;string&gt;
#include &lt;vector&gt;

class checkBP: public torch::nn::Module
{
public:
    checkBP(int show, std::string label):show(show), label(label) {std::cout&lt;&lt;" created checkBP"&lt;&lt;std::endl; }

    torch::Tensor forward(torch::Tensor inputs)
    {
        std::cout&lt;&lt;"\n\n\n\n checkBP forward passed\n\n\n\n"&lt;&lt;std::endl;
        return inputs.clone();
    }
    torch::Tensor backward(torch::Tensor grad_output){
        std::cout&lt;&lt;" In checkBP backward."&lt;&lt;std::endl;
        auto grad_mean = grad_output.abs().mean().data&lt;float&gt;();
        float grad_mean_float = *grad_mean;

        if ( show == 1)
        {
            std::cout&lt;&lt;"!!!! checkBP grad of "&lt;&lt;label&lt;&lt;" is(show=1): "&lt;&lt;grad_mean_float&lt;&lt;std::endl;
        }

        return grad_output;
    }

public:
    int show;
    std::string label;
};

class Net_model: public nn::Module
{
public:
    Net_model(int n_input,
              int n_hidden,
              int n_out){
        fc1 = register_module("fc1_net",torch::nn::Linear(n_input, n_hidden));
        fc2 = register_module("fc2_net",torch::nn::Linear(n_hidden, n_out));
    }

    nn::Linear fc1{nullptr}, fc2{nullptr};
    torch::Tensor forward(torch::Tensor input){
        auto x = fc1-&gt;forward(input);
        x = checkBP(/*show*/1,"net_model_checkBP").forward(x);
        x = fc2-&gt;forward(x);
        return x;
    }
};

int main()
{

    int n_input = 10, n_hidden = 8, n_out = 2, n_sample =  3;
    torch::Tensor inputs = torch::ones({n_sample, n_input});
    torch::Tensor target = torch::zeros({n_sample, n_out});

    auto net = Net_model(n_input,n_hidden,n_out);
    std::cout&lt;&lt;net&lt;&lt;std::endl;
    torch::optim::SGD optimizer(
        net.parameters(), torch::optim::SGDOptions(0.01).momentum(0.5));
    for( int i = 0; i &lt; 10; ++ i)
    {

        auto res = net.forward(inputs);
        optimizer.zero_grad();
        auto loss = torch::mse_loss(res, target);
        loss.backward();
        optimizer.step();
    }
return 0;
}
</code></pre>
<p>In the training loop: after executing “loss.backward()”, it seems that the checkBP::backward() does not execute. What is the correct definition of backward in libtorch then?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The ‘nn.Module’ only use the autograd to compute gradients.<br>
If you want to specify gradients for an operation you need to use a ‘Function’.</p>

username: Snade_Qian
cooked: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="49002">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>autograd</p>
</blockquote>
</aside>
<p>Thank you for in-time reply.</p>
<p>I have test your suggestion in Python env by creating a function first as below.</p>
<pre><code class="lang-auto">#submodules.py
import numpy as np
import torch
from torch.autograd import Function
class CheckBP(Function):

    def __init__(self, label, show):
        super(CheckBP, self).__init__()
        self.label = label
        self.show = show

    def forward(self, input):
        # print(self.label + ': forward passed.')
        # print(input)
        return input.clone()

    def backward(self, grad_output):
        grad_mean = grad_output.abs().mean()
        if self.show == 1:
            print('grad_' + self.label + ': ' + str(grad_mean))
        return grad_output
</code></pre>
<p>Then,  a class is used to wrap the function as below:</p>
<pre><code class="lang-auto"># modules
import numpy as np
import torch
import torch.nn as nn
from torch.autograd import Variable
import submodules as F

class CheckBP(nn.Module):

    def __init__(self, label='a', show=1):
        super(CheckBP, self).__init__()
        self.check_bp = F.CheckBP(label, show)

    def forward(self, input):
        return self.check_bp(input)
</code></pre>
<p>Testing the CheckBP in Python is ok.</p>
<p>However, in c++(libtorch),</p>
<pre><code class="lang-auto">class checkBP :public torch::autograd::Function
{
...(something)
}
</code></pre>
<p>I got this error " error: invalid use of incomplete type ‘struct torch::autograd::Function<br>
struct checkBP : public torch::autograd::Function "</p>
<p>I checked “torch::autograd::Function” in   /torch/csrc/autograd/edge.h . There is forward() or backward() to inherit.</p>
<p>So, how to define the checkBP module in c++ like in python with self-defined backward()?</p>
<p>Thank for your reply again!</p>

username: albanD
cooked: <p>Hi,</p>
<p>Note that on the python side, the Function have changed slightly as you can see in the <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="nofollow noopener">tuto</a>.</p>
<p>For cpp it is a bit more complex. a Function does only one way and its “apply” method should be implemented. It is either implemented in pure autograd by performing operations on Variables or the output should be wrapped and the backward Function specified.<br>
You will need 2 functions if you want a custom backward. For example <a href="https://github.com/pytorch/pytorch/blob/8cde4c4d223d3eb1179f87fd6336d25c189acb98/torch/csrc/autograd/functions/basic_ops.cpp#L18" rel="nofollow noopener">here</a>, “DelayedError” is the forward function and “Error” is the backward.</p>

username: yf225
cooked: <p><a class="mention" href="/u/snade_qian">@Snade_Qian</a> This is currently work-in-progress tracked in <a href="https://github.com/pytorch/pytorch/pull/23020" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/23020</a>. We will provide documentation / tutorial for this once we have the API ready.</p>