Why don't model parameter learn?

username: Dongsu_Lee
cooked: <p>Hi, everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"><br>
I am studying Inverse Rational Control.<br>
I want to make an experimentalist that infers the real parameter of the agent.</p>
<p>But,<br>
Don’t update the model parameter.<br>
Below my code<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fbfbca947c35c3d2144e5ea9b02ec45ce6002e50.png" data-download-href="https://discuss.pytorch.org/uploads/default/fbfbca947c35c3d2144e5ea9b02ec45ce6002e50" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fbfbca947c35c3d2144e5ea9b02ec45ce6002e50.png" alt="image" data-base62-sha1="zX9iWoSzCVypF1gS4iJU7sSzdiU" width="672" height="500" data-dominant-color="2F2F2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">775×576 44.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>coef is a model parameter I want to find.<br>
coef is a tensor size 2</p>
<p>after backward(), coef.grad is None and loss.grad is always [1.]</p>
<p>Does someone help me?</p>

username: Dongsu_Lee
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edee3775dce673b7f904fe409eda187d37e156dd.png" alt="image" data-base62-sha1="xWPxjnh6ybVGdsSBM8o2PXAa6Pr" width="569" height="50"></p>

username: ptrblck
cooked: <p>You are detaching the <code>loss</code> tensor by wrapping it into the deprecated <code>Variable</code>, so remove this line of code.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Dongsu_Lee
cooked: <p>Thanks reply <a class="mention" href="/u/ptrblck">@ptrblck</a> .<br>
Of course, I have tried it, but coefficients(parameter) do not change</p>
<pre><code class="lang-auto">def exp_inverse(true_coef, arg, env, agent, x_traj, a_traj, true_loss, filename, n, part_coef=False):
    tic = time.time()

    # Parameterization of coefficient to apply optimizer
    coef = nn.Parameter(reset_coef(arg.desvel_range, arg.uns4IDM_range))
    ini_coef = coef.data.clone()

    loss_log = deque(maxlen=arg.NUM_IT)
    loss_log_recent = deque(maxlen=100)
    coef_log = deque(maxlen=arg.NUM_IT)
    coef = Variable(coef, requires_grad=True)

    # Set-up Optimizer &amp; Scheduler
    opt = torch.optim.Adam([coef], lr=arg.ADAM_LR, eps=arg.ADAM_eps, amsgrad=False)
    Sched = torch.optim.lr_scheduler.StepLR(opt, step_size=10, gamma=0.99)

    print('===========================================')
    print('Finding Coefficient')
    print('===========================================')

    for it in tqdm(range(arg.NUM_IT)):
        # Calculate loss function
        loss, _ = getLoss(agent, x_traj, a_traj, coef, env, arg.PI_STD, arg.NUM_SAM)
        # loss = Variable(loss, requires_grad=True)
        loss.requires_grad = True
        # Initialize the gradients calculated previous step; Clears old gradients
        opt.zero_grad()
        # print(loss.grad)

        # Collect the data for plotting tendency
        loss_log.append(loss.data)
        loss_log_recent.append(loss.data)

        # Backpropagation
        loss.backward(retain_graph=True)
        # opt.param_groups[0]["params"][0].grad
        print(coef.grad)
        # Optimizer and Scheduler
        opt.step() # change coefficients
        Sched.step()
</code></pre>
<p>what is the problem? could you help me again?</p>

username: Dongsu_Lee
cooked: <p>if totally delete the line ‘loss.requires_grad =True’, i got the error message like below</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/bmil/flow-autonomous-driving/SPRING2021/Code/Inverse_LC/Inverse/Trajectory_collect.py", line 324, in &lt;module&gt;
    Trajectory_collect(args)
  File "/home/bmil/flow-autonomous-driving/SPRING2021/Code/Inverse_LC/Inverse/Trajectory_collect.py", line 240, in Trajectory_collect
    exp_inverse(true_coef, arg, env, agent, x_traj, a_traj, true_loss, f_name, 2)
  File "/home/bmil/flow-autonomous-driving/SPRING2021/Code/Inverse_LC/Inverse/Inverse_exp.py", line 50, in exp_inverse
    loss.backward(retain_graph=True)
  File "/home/bmil/anaconda3/envs/flow/lib/python3.7/site-packages/torch-1.4.0-py3.7-linux-x86_64.egg/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/bmil/anaconda3/envs/flow/lib/python3.7/site-packages/torch-1.4.0-py3.7-linux-x86_64.egg/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>

username: ptrblck
cooked: <p>It seems that <code>loss</code> is already detached when it’s returned from the model, which might happen, if you rewrap it into a tensor, use a non-differentiable operation, or use a 3rd party library (e.g. numpy) without writing a custom <code>autograd.Function</code>.<br>
To debug it further, you can check the <code>.grad_fn</code> of the intermediate tensors and locate which tensor returns a <code>None</code> value.</p>

username: Dongsu_Lee
cooked: <p>Thanks for your reply. <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I reckon that the problem is  line ‘state = np.concatenate([only_state, desvel.detach().numpy(), uns4IDM.detach().numpy()])’.<br>
but i can’t use just .numpy() or torch.tensor() because this framework accepts only numpy type array.<br>
What should I do in this case?</p>
<pre><code class="lang-auto">def getLoss(agent, x_traj, a_traj, c, env, PI_STD, NUM_SAM):
    # getLoss
    logPr = torch.zeros(1)
    logPr_act = torch.zeros(1)
    logPr_lc = torch.zeros(1)
    logPr_acc = torch.zeros(1)

    desvel, uns4IDM = torch.split(c.view(-1),1)

    env.desvel = desvel
    env.uns4IDM = uns4IDM

    for num_it in range(NUM_SAM):
        logPr_ep = torch.zeros(1)
        logPr_act_ep = torch.zeros(1)
        logPr_lc_ep = torch.zeros(1)
        logPr_acc_ep = torch.zeros(1)

        t = torch.zeros(1)
        state = env.reset()
        only_state = np.delete(state, (14, 15))
        state = np.concatenate([only_state, desvel.detach().numpy(), uns4IDM.detach().numpy()])


        print('\ncalculating loss')
        for it, next_x in tqdm(enumerate(x_traj[1:])):
            action = agent.compute_action(state)
            action_t = torch.Tensor(action)


            lc_loss = ((action_t[1] - a_traj[it][1]) ** 2) / (2 * (PI_STD ** 2)) + ((action_t[2] - a_traj[it][2]) ** 2) / (2 * (PI_STD ** 2)) + ((action_t[3] - a_traj[it][3]) ** 2) / (2 * (PI_STD ** 2))# policy

            logPr_acc_ep = acc_loss.sum() + logPr_acc_ep
            logPr_lc_ep = lc_loss.sum() + logPr_lc_ep
            logPr_act_ep = acc_loss.sum() + lc_loss.sum() + logPr_act_ep
            logPr_ep = logPr_ep + acc_loss.sum() + lc_loss.sum()

            next_state, _, _, _ = env.step(action)

            t += 1
            on_state = np.delete(next_state, (14, 15))
            state = np.concatenate([on_state, desvel.detach().numpy(), uns4IDM.detach().numpy()])

        logPr_acc += logPr_acc_ep
        logPr_lc += logPr_lc_ep
        logPr_act += logPr_act_ep
        logPr += logPr_ep
        print("acc:{}, lc:{}, logPr:{}, logPr_act:{}".format(logPr_acc, logPr_lc, logPr_ep, logPr_act))
    return logPr/NUM_SAM, logPr_act/NUM_SAM
</code></pre>

username: ptrblck
cooked: <p>If you need to use numpy operations, you would have to implement the backward operation manually using a custom <code>autograd.Function</code> as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a>.</p>

username: Dongsu_Lee
cooked: <p>Thank you for your time! <a class="mention" href="/u/ptrblck">@ptrblck</a></p>