Optimizer not updating weights C++ Frontend

Ruslan_Mukhamadiarov: <p>Hello! I am trying to implement Actor Critic algorithm in C++, and for some reason the weights of my Actor and Critic networks are not being updated.</p>
<p>My Actor.h</p>
<pre><code class="lang-auto">#pragma once
#include &lt;torch/torch.h&gt;

class ActorImpl : public torch::nn::Module {
    public:
        ActorImpl(int input_dims, int hidden_size, int n_actions);
        torch::Tensor forward(torch::Tensor state);
        torch::Tensor choose_action(torch::Tensor state);
        torch::Tensor calculate_loss(double gamma);

    private:
        torch::nn::Linear input;
        torch::nn::Linear pi;
};
TORCH_MODULE(Actor);
</code></pre>
<p>My Actor.cpp</p>
<pre><code class="lang-auto">#include "Actor.h"
#include &lt;torch/torch.h&gt;

ActorImpl::ActorImpl(int input_dims, int hidden_size, int n_actions)
    : input(input_dims, hidden_size), pi(hidden_size, n_actions){
    register_module("input", input);
    register_module("pi", pi);
}

torch::Tensor ActorImpl::forward(torch::Tensor state) {
    auto x = F::relu(input-&gt;forward(state));
    return pi-&gt;forward(x);
}

torch::Tensor ActorImpl::calculate_loss(double gamma){
    vector&lt;torch::Tensor&gt; Pi, V;
    for(auto i = 0; i &lt; _states.size(); i++){ // _states.size() has the same length as the episode length
        auto pi_i = forward(_states[i]);
        CriticImpl critic_instance(_input_dims, _hidden_size);
        auto v_i = critic_instance.forward(_states[i]);;
        Pi.push_back(pi_i);
        V.push_back(v_i);
    }

    double R = 0;
    vector&lt;double&gt; returns;
    for(auto i = 0; i &lt; _rewards.size(); i++){
        R += pow(gamma,i)*_rewards[i];
        returns.push_back(R);
    }

    vector&lt;double&gt; log_probs;
    for(auto i = 0; i &lt; _actions.size(); i++){
        auto probs = F::softmax(Pi[i], F::SoftmaxFuncOptions(1)); // converts list of logits to list of probabilities
        int action = _actions[i];
        double log_prob = log(probs[0][action].item&lt;double&gt;());
        log_probs.push_back(log_prob);
    }

    vector&lt;double&gt; total_loss;
    for(auto i = 0; i &lt; _states.size(); i++){
        double actor_loss  = -log_probs[i]*(returns[i] - V[i][0][0].item&lt;double&gt;());
        double critic_loss = pow(returns[i] - V[i][0][0].item&lt;double&gt;(), 2);
        total_loss.push_back(actor_loss + critic_loss);
    }
    auto av_total_loss = accumulate(total_loss.begin(), total_loss.end(), 0) / (1.0*total_loss.size());
    torch::Tensor return_loss_as_tensor = torch::tensor(av_total_loss, torch::requires_grad());

    return return_loss_as_tensor;
}

</code></pre>
<p>My main.cpp</p>
<pre><code class="lang-auto">    // some parameters definitions here
    // ...
    auto cuda_available = torch::cuda::is_available();
    torch::Device device(cuda_available ? torch::kCUDA : torch::kCPU);
    cout &lt;&lt; (cuda_available ? "CUDA available. Training on GPU." : "Training on CPU.") &lt;&lt; '\n';

    Actor actor(input_dims, hidden_size, n_actions);
    actor-&gt;to(device); 
    torch::optim::Adam optimizer_actor(actor-&gt;parameters(), torch::optim::AdamOptions(lr));
    cout &lt;&lt; "actor parameters" &lt;&lt; endl &lt;&lt; actor-&gt;parameters() &lt;&lt; endl;

    // in between i do some stuff, pass observation state and compute the loss

    // now update the weights and biases
    torch::Tensor loss = actor-&gt;calculate_loss(gamma); 
    optimizer_actor.zero_grad(); // Reset gradients
    loss.backward(); // backpropagate loss
    optimizer_actor.step(); // update network parameters
    actor-&gt;clear_memory(); 
    cout &lt;&lt; "updated actor parameters" &lt;&lt; endl &lt;&lt; actor-&gt;parameters() &lt;&lt; endl;
</code></pre>
<p>No runntime errors, but the parameters just do not change, no matter what I do with the learning rate. I know that the issue somehow must be with how I define my Actor class, because I have tried doing backdrop on some simple linear regression models, and everything worked.</p>
<p>The complete project is here, in case if you are interested:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://gitlab.com/mruslani/random-walker-who-is-afraid-of-periodic-bc/-/tree/main/C%2B%2B/non-interacting-AC">
  <header class="source">
      <img src="https://gitlab.com/assets/favicon-yellow-018213ceb87b472388095d0264be5b4319ef47471dacea03c83ecc233ced2fd5.png" class="site-icon" width="32" height="32">

      <a href="https://gitlab.com/mruslani/random-walker-who-is-afraid-of-periodic-bc/-/tree/main/C%2B%2B/non-interacting-AC" target="_blank" rel="noopener nofollow ugc">GitLab</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:531/500;"><img src="https://gitlab.com/uploads/-/system/project/avatar/42620926/Screenshot_2023-01-16_at_14.48.26.png" class="thumbnail" width="531" height="500"></div>

<h3><a href="https://gitlab.com/mruslani/random-walker-who-is-afraid-of-periodic-bc/-/tree/main/C%2B%2B/non-interacting-AC" target="_blank" rel="noopener nofollow ugc">C++/non-interacting-AC · main · Ruslan Mukhamadiarov / Random walker who is...</a></h3>

  <p>My first attempt to put together a code which implements RL Actor-Critic algorithm, by training a random walker to avoid the periodic boundary of 1d chain.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>I don’t see where <code>actor-&gt;calculate_loss</code> is defined so I would assume you might be using another class providing this method?<br>
If it’s missing from your code snippets, check if you are detaching the computation graph in this method somehow.</p>

Ruslan_Mukhamadiarov: <p>Thank you for your reply! I have added the missing function. It is currently inside the Actor class. I just tried to keep things short, sorry.</p>

ptrblck: <p>Thanks for the missing code!<br>
These lines of code:</p>
<pre><code class="lang-auto">auto av_total_loss = accumulate(total_loss.begin(), total_loss.end(), 0) / (1.0*total_loss.size());
torch::Tensor return_loss_as_tensor = torch::tensor(av_total_loss, torch::requires_grad());
</code></pre>
<p>are detaching <code>av_total_loss</code> from the computation graph by re-wrapping it into a new <code>torch::tensor</code>.<br>
Remove the last line of code and see if this helps. If you are getting another error and used the re-wrapping as a workaround, please post this error as you would need to fix it properly.</p>

Ruslan_Mukhamadiarov: <p>Thank you very much! Will try it out!</p>