PyTorch RuntimeError: Trying to backward through the graph a second time

username: Bernardoolisan
cooked: <p>I am attempting to implement the Proximal Policy Optimization (PPO) algorithm using PyTorch. My implementation is based on my understanding of how the PPO algorithm works. At a high level, the process of my implementation looks something like this:</p>
<ol>
<li>First, we need to initialize the Policy Network with random parameters.</li>
<li>Then, at each time step of the current episode, we need to compute the advantage function. This involves  computing the Value functions <em>V(s)</em> and <em>Q(s,a)</em> in order to get <em>A(s,a) = V(s) - Q(s,a)</em>. In my implementation, each value function is a function approximator that optimizes both of them at each time step using a simple gradient descent approach.</li>
<li>Additionally, at each time step of the episode, I compute the clipped surrogate values at the current step.</li>
<li>At the end of the episode, I calculate the expected value of all the clipped surrogate values in order to compute the SGA (Stochastic Gradient Ascent) for optimizing the Policy Network parameters.</li>
</ol>
<p>My Value functions and Policy Network are implemented as classes. The Value functions classes each contain PyTorch initialization and forward pass methods, as well as MSE loss and SGD methods. The Policy Network class includes initialization and forward pass methods, along with the clipped surrogate value method for computation at each time step, and lastly the SGA method. You can find the full code at <a href="https://github.com/BernardoOlisan/PPO-Clip/blob/main/ppo.py" rel="noopener nofollow ugc">https://github.com/BernardoOlisan/PPO-Clip/blob/main/ppo.py</a></p>
<p>Everything is working, but when I try to compute SGA for the policy network at the end of the episode, this <strong>error shows up</strong>:</p>
<p>RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward()  or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</p>
<p>Some people suggested that the issue might be due to my use of <code>loss.backward()</code> in each class (value functions and policy network), and advised me to resolve it by setting <code>retain_graph=True</code> in each <code>loss.backward()</code> instance. However, this solution is not effective.</p>
<p>I am unsure what is causing the error in my code. Why is the Policy Network SGA throwing this error? How can I fix it? Am I making a mistake?</p>
<p>Here is my policy rollouts:</p>
<pre><code>def rollouts(self, policy, state_value_function, action_value_function):
    old_policy = copy.deepcopy(policy)

    for episode in range(self.EPISODES):
        print(f"###################################### EPISODE [{episode}] ######################################")

        (initial_state, _) = self.env.reset()
        self.env.render()
        state = initial_state
        clipped_surrogated_values = []

        for step in range(self.EPISODE_STEPS):
            print(f"---------------------- STEP [{step}] ----------------------")

            policy_output = policy.forwardPropagation(torch.from_numpy(state))
            old_policy_output = old_policy.forwardPropagation(torch.from_numpy(state))
            action = torch.distributions.Categorical(policy_output).sample().item()
            old_action = torch.distributions.Categorical(old_policy_output).sample().item()

            state, reward, terminated, truncated, info = self.env.step(action)

            target_value = torch.tensor([reward], dtype=torch.float32)

            predicted_state_value = state_value_function.forwardPropagation(torch.from_numpy(state))
            loss_state_value = state_value_function.MSE(predicted_state_value, target_value)
            state_value_function.SGD(loss_state_value, VALUE_LEARNING_RATE)

            predicted_action_value = action_value_function.forwardPropagation(torch.from_numpy(state), torch.tensor([action], dtype=torch.float32))
            loss_action_value = action_value_function.MSE(predicted_action_value, target_value)
            action_value_function.SGD(loss_action_value, VALUE_LEARNING_RATE)

            advantage = predicted_state_value - predicted_action_value
            ratio = torch.tensor((action + 1e-6) / (old_action + 1e-6), dtype=torch.float32)
            clipped_surrogate_objective = policy.clippedSurrogateObjective(ratio, advantage)
            clipped_surrogated_values.append(clipped_surrogate_objective)

            old_policy = copy.deepcopy(policy)

            # Policy Network Rollout
            print("Policy Network (π) Rollout:")
            print(f"Policy NN output: {policy_output}")
            print(f"Old Policy NN output: {old_policy_output}")
            print(f"Selected action: {action}")
            print(f"Reward received: {reward}\n")

            # State Value Function (V(s))
            print("State Value Function (V(s)):")
            print(f"Predicted Value: {predicted_state_value}")
            print(f"Loss (V(s)): {loss_state_value}\n")

            # Action Value Function (Q(s, a))
            print("Action Value Function (Q(s, a)):")
            print(f"Predicted Value: {predicted_action_value}")
            print(f"Loss (Q(s, a)): {loss_action_value}\n")

            # Clipped Surrogate Objective at t (J(θ_t))
            print("Clipped Surrogate Objective at t (J(θ_t)):")
            print(f"Ratio: {ratio}")
            print(f"Advantage: {advantage}")
            print(f"Result: {clipped_surrogate_objective}\n")

            if terminated:
                time.sleep(1)
                break
            time.sleep(0.1)

        expected_surrogate_objective = sum(clipped_surrogated_values)
        # Here is where the error shows up
        policy.SGA(expected_surrogate_objective, NN_LEARNING_RATE) 

        print(f"\nEPISODE DETAILS:")
        print(f"Expected Surrogate Objective: {expected_surrogate_objective}")

    self.env.close()
</code></pre>
<p>Policy Network class is this one:</p>
<p>class PolicyNetwork(nn.Module):<br>
def <strong>init</strong>(self, input_dim, hidden_dim, output_dim):<br>
super(PolicyNetwork, self).<strong>init</strong>()<br>
self.fc1 = nn.Linear(input_dim, hidden_dim)<br>
self.fc2 = nn.Linear(hidden_dim, output_dim)</p>
<pre><code>def forwardPropagation(self, X):
    Y = torch.relu(self.fc1(X))
    Z = self.fc2(Y)
    z_hat = F.softmax(Z, dim=-1)
    return z_hat

def clippedSurrogateObjective(self, ratio, advantage):
    first_value = ratio * advantage
    second_value = torch.clamp(ratio, 1 - EPSILON, 1 + EPSILON) * advantage
    return torch.min(first_value, second_value)

def SGA(self, expected_surrogate_objective, learning_rate):
    ''' Error in here...
        RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after 
        they have already been freed). Saved intermediate values of the graph are freed when you call .backward() 
        or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if 
        you need to access saved tensors after calling backward.
    '''
    optimizer = optim.Adam(self.parameters(), lr=learning_rate)
    optimizer.zero_grad()

    loss = -expected_surrogate_objective
    loss.backward()
    optimizer.step()
</code></pre>
<p>The complete code can be found at <a href="https://github.com/BernardoOlisan/PPO-Clip/blob/main/ppo.py" rel="noopener nofollow ugc">https://github.com/BernardoOlisan/PPO-Clip/blob/main/ppo.py</a>. It’s not a very large code.</p>
<p>Thank you in advance.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Bernardoolisan" data-post="1" data-topic="191396">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bernardoolisan/48/65282_2.png" class="avatar"> Bernardoolisan:</div>
<blockquote>
<p>Some people suggested that the issue might be due to my use of <code>loss.backward()</code> in each class (value functions and policy network), and advised me to resolve it by setting <code>retain_graph=True</code> in each <code>loss.backward()</code> instance. However, this solution is not effective.</p>
</blockquote>
</aside>
<p>If you have multiple <code>backward</code> passes using connected computation graphs, the claim is correct and the second <code>backward</code> pass will fail since intermediates were already freed in the first call.<br>
Use <code>retain_grah=True</code> in the first <code>backward</code> pass or accumulate the losses before calling <code>backward</code> a single time. I also don’t know what “this solution is not effective” means.</p>

username: Bernardoolisan
cooked: <p>Apologies, using <code>retain_graph=True</code> is not effective in solving the issue because I encounter an error when setting <code>retain_graph=True</code> in all instances of <code>loss.backward()</code></p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [32, 1]], which is output 0 of AsStridedBackward0, is at version 20; expected version 19 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>Also, they are 3 different Neural Networks that each of them has a <code>loss.backward()</code></p>
<p>any idea <a class="mention" href="/u/ptrblck">@ptrblck</a> ?</p>

username: KFrank
cooked: <p>Hi Bernardo!</p>
<aside class="quote no-group" data-username="Bernardoolisan" data-post="3" data-topic="191396" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bernardoolisan/48/65282_2.png" class="avatar"> Bernardoolisan:</div>
<blockquote>
<p>I encounter an error when setting <code>retain_graph=True</code> in all instances of <code>loss.backward()</code></p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [32, 1]], which is output 0 of AsStridedBackward0, is at version 20; expected version 19 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
</blockquote>
</aside>
<p>I assume that the outputs of your Value functions depend somehow on<br>
your Policy Network and that you need to take these dependencies into<br>
account when you optimize your Policy Network.  (If not, you could get<br>
rid of the <code>retain_graph  =True</code> and <code>.detach()</code> the outputs of your Value<br>
functions before you use them in anything that has to do with your Policy<br>
Network.  Doing so would solve the “backward a second time” error you<br>
reported in your first post.)</p>
<p>The problem is (if you leave the dependency of your Policy Network on<br>
your Value functions in place) that when you call <code>.backward()</code> on your<br>
Policy-Network loss, you will also backpropagate through those Value<br>
functions.  But you have already done something along the lines of calling<br>
<code>optimizer.step()</code> on your Value functions and doing so modifies the<br>
<code>Parameter</code>s of your Value functions inplace, leading to the error you see.</p>
<p>Assuming that this is what is going on, you should modify the forward<br>
passes of your Value functions so that their <code>Parameter</code>s (that are being<br>
modified inplace) are not, themselves, needed in Policy Network’s<br>
backward pass.  You can probably achieve this with something like:</p>
<pre data-code-wrap="python"><code class="lang-python"># in ValueFunction
linA = torch.nn.Linear (in_features, out_features)
...
# instead of
# y = linA (x)
# use
y = torch.nn.functional.linear (x, linA.weight.clone(), linA.bias.clone())
</code></pre>
<p>The point is that when backpropagating through <code>ValueFunction</code>, the<br>
<code>clone()</code>s of <code>linA</code>’s <code>Parameter</code>s are used and those <code>clone()</code>s haven’t<br>
been modified by calling <code>optimizer.step()</code>.</p>
<p>For further insight into what is going on in this situation I described or for<br>
some suggestions on how to debug your issue if it’s something else that<br>
is going on, please see this post:</p>
<aside class="quote quote-modified" data-post="7" data-topic="171826">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-torch-floattensor-64-1-which-is-output-0-of-asstridedbackward0-is-at-version-3-expected-version-2-instead-hint-the-backtrace-further-a/171826/7">"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Fahmyadan and Sangyoon! 

Here are some suggestions about how to track down (and maybe fix) 
inplace-modification errors.  Note that an inplace modification in the forward 
pass is not necessarily* an error – it depends on whether and how the tensor 
that was modified is used in the backward pass.  Note that inplace operations 
can be useful for saving memory – if you replace an innocent inplace operation 
with an out-of-place equivalent, your training will use more memory (and, to a 
minor e…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

username: Bernardoolisan
cooked: <aside class="quote no-group quote-modified" data-username="KFrank" data-post="4" data-topic="191396">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p><a class="mention" href="/u/kfrank">@KFrank</a> I assume that the outputs of your Value functions depend somehow on<br>
your Policy Network and that you need to take these dependencies into<br>
account when you optimize your Policy Network. (If not, you could get<br>
rid of the <code>retain_graph  =True</code> and <code>.detach()</code> the outputs of your Value<br>
functions before you use them in anything that has to do with your Policy<br>
Network. Doing so would solve the “backward a second time” error you<br>
reported in your first post.)</p>
</blockquote>
</aside>
<p>That’s interesting because there’s actually a relationship between the Value Functions networks and the Policy Network. The Policy Network uses an objective function as the loss (clipped surrogate objective function from PPO algorithm), this objective function uses an Advantage Function in order to be computed. The Advantage Function is calculated by the difference between the two Value Functions Networks outputs, so in theory that is the relation or connection that Value Networks and Policy Network have.</p>
<p>Passing the output of the Value Networks directly to compute the Advantage function, which serves to calculate the Policy Network objective, also passes the gradients <code>grad_fn</code> that the Value Networks produced. This issue can be resolved by using <code>.detach()</code>.</p>
<p>I didn’t think the advantage function was causing the issue, as the connection between Value Networks and Policy Networks was difficult to see.</p>
<p>Thanks, K. Frank</p>