Getting a zero grad in non-linear regression

username: scollazo
cooked: <p>Hello everybody! I am working on a code that tries to fit a theoretical curve to a set of observational data points. This could be easily achieved through least squares, but I have noticed that a very good and sophisticated way of doing it is through an algorithm of non-linear regression in PyTorch. The problem is that I don’t know any example code like this. The idea came to my mind when I was watching a video about linear regression in PyTorch, but this is the only way of doing a regression I know.  What I want to do is to get the best free parameters so the theoretical curve is the best option to model the data.</p>
<p>Well, let’s explain the code. The class that defines the model is</p>
<pre><code class="lang-auto">class Mass_and_v(nn.Module):
    def __init__(self, param):
        super(Mass_and_v, self).__init__()
        self.parameter = nn.Parameter(data=param, requires_grad=True)
        
    # get v_circ_tot
    def __call__(self, x):
        r, mass_rar, _ = rar.model(self.parameter[3:], gradient_tracking=True)
        r_max = r[-1]
        mass_max = torch.max(mass_rar)
        x_spline = x[x &lt;= r_max]
        mass_spline = chs.interp(r, mass_rar, x_spline)
        v_circ = torch.where(x &lt;= r_max,
                             torch.sqrt(v_circ_b(x, self.parameter[0])**2 + v_circ_MN(x, self.parameter[1:3])**2 + G_u*mass_spline/x),
                             torch.sqrt(v_circ_b(x, self.parameter[0])**2 + v_circ_MN(x, self.parameter[1:3])**2 + G_u*mass_max/x))
        return v_circ

class FitModel(nn.Module):
    def __init__(self):
        super(FitModel, self).__init__()
        self.nonlinear_stack = Mass_and_v(param_t)
        
    def forward(self, x):
        output = self.nonlinear_stack(x)
        return output
</code></pre>
<p>Then, I instantiate the model object through</p>
<pre><code class="lang-auto">model = FitModel()
</code></pre>
<p>The external functions that I use are <strong>rar.model</strong>, <strong>v_circ_b</strong> and <strong>v_circ_MN</strong>, all of which are correctly coded using torch (let me know if you need to see the code of these functions).</p>
<p>So, when I run</p>
<pre><code class="lang-auto">for i in range(n_epoch):
    pred = model(inputs)
    # The loss function is a nn.MSELoss()
    loss_fn = loss(pred, targets)
    # We apply the backpropagation method using a torch.optim.Adam optimizer
    optimizer.zero_grad()
    """ The next step is to backpropagate this error through the network. Backward
    propagation is kicked off when we call .backward() on the error tensor. Autograd
    then calculates and stores the gradients for each model parameter in the parameter’s 
    .grad attribute."""
    loss_fn.backward()
    """ We call .step() to initiate Adam. The optimizer adjusts each parameter by
    its gradient stored in .grad."""
    optimizer.step()
</code></pre>
<p>I get a null grad for the <strong>self.parameter[4:]</strong> parameters used inside <strong>rar.model</strong>. Do you know what could be the problem?</p>

username: ptrblck
cooked: <p>Check, if any of the internal functions is (accidentally) detaching a tensor from the computation graph (e.g. by using a non-differentiable operation) by checking the <code>.grad_fn</code> of intermediate tensors.<br>
If one of them shows <code>None</code> it would mean that this particular tensor is not attached to a graph. Often users try to fix this by rewrapping a tensor and setting <code>requires_grad=True</code> again, which won’t fix the issue.</p>

username: scollazo
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
Thank you for your answer! I will check that.</p>