Error in MmBackward0 during backpropagation in C++ PyTorch

username: umur
cooked: <p>Hello,</p>
<p>I am encountering a runtime error during the backpropagation step of my training loop, specifically when calling <code>actor_loss.backward()</code>. The error message, first with warning is:</p>
<p>[W anomaly_mode.cpp:54] Warning: Error detected in MmBackward0. Traceback of forward call that caused the error:<br>
(function print_stack)<br>
An exception occurred during backward pass: one of the variables needed for gradient computation has been modified by an inplace operation: [CPUFloatType [1, 9]] is at version 1216; expected version 1197 instead.</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">// Update the update_policy function to handle batched data
void update_policy(std::shared_ptr&lt;Actor&gt; actor, std::shared_ptr&lt;Critic&gt; critic,
    const std::vector&lt;torch::Tensor&gt;&amp; batch_states,
    const std::vector&lt;torch::Tensor&gt;&amp; batch_actions,
    const std::vector&lt;torch::Tensor&gt;&amp; batch_log_probs,
    const std::vector&lt;double&gt;&amp; batch_rewards,
    const std::vector&lt;torch::Tensor&gt;&amp; batch_next_states,
    double clip_param = 0.01)
{
    torch::autograd::AnomalyMode::set_enabled(true);

    torch::Tensor states = torch::stack(batch_states);
    torch::Tensor actions = torch::stack(batch_actions);
    torch::Tensor old_log_probs = torch::stack(batch_log_probs);
    torch::Tensor rewards = torch::tensor(batch_rewards);
    torch::Tensor next_states = torch::stack(batch_next_states);

    torch::Tensor returns = compute_returns(rewards, 1); 
    
    torch::Tensor values = critic-&gt;forward({ states }).squeeze(-1);

    torch::Tensor advantages = returns - values.detach();

    // Normalize advantages
    advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-10);

    // Evaluate current policy
    auto [means, stds] = actor-&gt;forward(states);
    torch::Tensor curr_log_probs = calculate_log_probs(actions, means, stds);

    // Compute ratio
    torch::Tensor ratios = torch::exp(curr_log_probs - old_log_probs);;

    // Compute surrogate losses
    torch::Tensor surr1 = ratios * advantages;
    torch::Tensor surr2 = torch::clamp(ratios, 1.0 - clip_param, 1.0 + clip_param) * advantages;

    // Compute actor and critic losses
    torch::Tensor actor_loss = torch::min(surr1, surr2).mean();

    // To be sure, values is not implemented below:
    torch::Tensor critic_loss = torch::mse_loss(critic-&gt;forward({ states }).squeeze(-1), returns);

    //std::cout &lt;&lt; "actor_loss: " &lt;&lt; actor_loss &lt;&lt; std::endl;
    //std::cout &lt;&lt; "critic_loss: " &lt;&lt; critic_loss &lt;&lt; std::endl;


    // Perform backpropagation 
    critic-&gt;optimizer.zero_grad();
    critic_loss.backward();
    critic-&gt;optimizer.step();

    actor-&gt;optimizer.zero_grad();
    actor_loss.backward();    // Error and warning here
    actor-&gt;optimizer.step();

    std::cout &lt;&lt; "Update - Actor Loss: " &lt;&lt; actor_loss.item&lt;float&gt;()
        &lt;&lt; ", Critic Loss: " &lt;&lt; critic_loss.item&lt;float&gt;() &lt;&lt; std::endl;
}
</code></pre>
<p>I believe I use an unsupported action while calculating one of the tensors defining actor_loss, but I cannot find it. Any support is appreciated, thanks!</p>