Multi-Agent Advantage calculation is leading to in-place gradient error

_aadharna: <p>I am working on some multi-agent RL training using PPO.</p>
<p>As part of that, I need to calculate the advantage on a per-agent basis which means that I’m taking the data generated by playing the game and masking out parts of it at a time.</p>
<p>This has led to an <code>in-place</code> error that’s killing the gradient and pytorch’s anomaly detection = True stack trace shows me the value function output from my NN.</p>
<p>Here’s a gist of the appropriate code with the learning code separated out: <a href="https://gist.github.com/aadharna/fe6d4c6cd18d090757247fc679ebc308" class="inline-onebox" rel="noopener nofollow ugc">cleanRL · GitHub</a></p>
<p>I found this <a href="https://discuss.pytorch.org/t/why-my-linear-layer-is-considered-as-an-inplace-operation/168333">post</a> where they were getting a similar error and fixed it by saving intermediate data into a list and then making that list of tensors into the complete tensor at the end. However, that solution is not working.</p>
<p>Here’s (what I think is) the offending code:</p>
<pre><code class="lang-python">    with torch.no_grad():
        sort_list = []
        advantages = []
        returns = []
        indices = torch.arange(0, rewards.shape[0]).long().to(device)
        next_value = learner.get_value(torch.FloatTensor(next_obs).unsqueeze(0).to(device))
        for player in ['player_0', 'player_1']:
            mask = np.array(rollouts.agent_id) == player
            masked_inds = list(indices[mask])
            lastgaelam = 0
            for t in reversed(range(mask.sum())):
                if t == mask.sum() - 1:
                    nextnonterminal = 1.0 - next_done
                    nextvalues = next_value
                else:
                    nextnonterminal = 1.0 - dones[mask][t + 1]
                    nextvalues = values[mask][t + 1]
                delta = rewards[mask][t] + gamma * nextvalues * nextnonterminal - values[mask][t]
                lastgaelam = delta + gamma * gae_lambda * nextnonterminal * lastgaelam
                advantages.append(lastgaelam)
                sort_list.append(masked_inds[t])
                returns.append(lastgaelam + values[mask][t])

        advantages = torch.cat(advantages)[torch.LongTensor(sort_list)].to(device)
        returns = torch.cat(returns)[torch.LongTensor(sort_list)].to(device)
</code></pre>
<p>Here’s how the code looks when doing single-agent learning and there’s still indexing and setting into the advantage tensor:</p>
<pre><code class="lang-python">        with torch.no_grad():
            next_value = agent.get_value(next_obs).reshape(1, -1)
            advantages = torch.zeros_like(rewards).to(device)
            lastgaelam = 0
            for t in reversed(range(args.num_steps)):
                if t == args.num_steps - 1:
                    nextnonterminal = 1.0 - next_done
                    nextvalues = next_value
                else:
                    nextnonterminal = 1.0 - dones[t + 1]
                    nextvalues = values[t + 1]
                delta = rewards[t] + args.gamma * nextvalues * nextnonterminal - values[t]
                advantages[t] = lastgaelam = delta + args.gamma * args.gae_lambda * nextnonterminal * lastgaelam
            returns = advantages + values
</code></pre>
<p>I guess it’s the masking that is the issue, but I don’t see how else to compute what I need without it.</p>
<hr>
<p>Here’s an example of multi-agent PPO where all the agents move simultaneously which then means that you can vectorize the advantage calculations along the agent dimension of the data buffers: pettingzoo. farama. org/tutorials/cleanrl/implementing_PPO/</p>
<p>but that doesn’t work for extensive form games where the data is naturally interleaved, hence the masking solutions to independently calculate things.</p>
<hr>
<p>Full error if anyone wants it:</p>
<pre><code class="lang-auto">/home/roque/miniconda3/envs/mapo/lib/python3.9/site-packages/torch/autograd/__init__.py:197: UserWarning: Error detected in AddmmBackward0. Traceback of forward call that caused the error:
  File "/mnt/d/PycharmProjects/ubc/mapo/c4_train.py", line 264, in &lt;module&gt;
    next_done, batch_reward_main, batch_reward_opp, batch_opponents_scores) = generate_data(
  File "/mnt/d/PycharmProjects/ubc/mapo/c4/ppo.py", line 55, in generate_data
    rollout_results = rollout(env, learner, opponent, f"main_v{opponent_id}", device)
  File "/mnt/d/PycharmProjects/ubc/mapo/c4/utils.py", line 138, in rollout
    action, logprob, entropy, value, logits = main_policy.get_action_and_value(observation)
  File "/mnt/d/PycharmProjects/ubc/mapo/c4_train.py", line 184, in get_action_and_value
    return action, probs.log_prob(action), probs.entropy(), self.critic(x), logits.cpu().detach()
  File "/home/roque/miniconda3/envs/mapo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/roque/miniconda3/envs/mapo/lib/python3.9/site-packages/torch/nn/modules/container.py", line 204, in forward
    input = module(input)
  File "/home/roque/miniconda3/envs/mapo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/roque/miniconda3/envs/mapo/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/roque/miniconda3/envs/mapo/lib/python3.9/site-packages/torch/fx/traceback.py", line 57, in format_stack
    return traceback.format_stack()
 (Triggered internally at /opt/conda/conda-bld/pytorch_1670525551200/work/torch/csrc/autograd/python_anomaly_mode.cpp:114.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "/mnt/d/PycharmProjects/ubc/mapo/c4_train.py", line 279, in &lt;module&gt;
    loss, entropy_loss, pg_loss, v_loss, explained_var, approx_kl, meanclipfracs, old_approx_kl = update_model(
  File "/mnt/d/PycharmProjects/ubc/mapo/c4/ppo.py", line 248, in update_model
    loss.backward(retain_graph=True)
  File "/home/roque/miniconda3/envs/mapo/lib/python3.9/site-packages/torch/_tensor.py", line 488, in backward
    torch.autograd.backward(
  File "/home/roque/miniconda3/envs/mapo/lib/python3.9/site-packages/torch/autograd/__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [256, 1]], which is output 0 of AsStridedBackward0, is at version 4; expected version 3 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>

KFrank: <p>Hi Aaron!</p>
<aside class="quote no-group quote-modified" data-username="_aadharna" data-post="1" data-topic="183172" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/_aadharna/48/61756_2.png" class="avatar"> _aadharna:</div>
<blockquote>
<p>This has led to an <code>in-place</code> error that’s killing the gradient and pytorch’s anomaly detection = True stack trace shows me the value function output from my NN.</p>
<p>…</p>
<p>Full error if anyone wants it:</p>
<pre><code class="lang-auto">/home/roque/miniconda3/envs/mapo/lib/python3.9/site-packages/torch/autograd/__init__.py:197: UserWarning: Error detected in AddmmBackward0. Traceback of forward call that caused the error:
...
  File "/mnt/d/PycharmProjects/ubc/mapo/c4_train.py", line 184, in get_action_and_value
    return action, probs.log_prob(action), probs.entropy(), self.critic(x), logits.cpu().detach()
...
  File "/mnt/d/PycharmProjects/ubc/mapo/c4/ppo.py", line 248, in update_model
    loss.backward(retain_graph=True)
...
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [256, 1]], which is output 0 of AsStridedBackward0, is at version 4; expected version 3 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
</blockquote>
</aside>
<p>Starting with the forward-call traceback, look at the last couple of lines<br>
in your code (that are then followed by calls into pytorch infrastructure).</p>
<p>I would certainly drill down into what <code>self.critic (x)</code> is doing.</p>
<p>[Edit:  Some further words of clarification / explanation:  As I’ve come to<br>
understand it, anomaly detection’s forward-call traceback flags the<br>
operation in the forward pass whose backward pass is being blocked<br>
by the inplace modification of some tensor required by the backward<br>
pass (rather than flagging the operation that modifies that tensor).</p>
<p>Given that you are using <code>retain_graph = True</code>, I speculate that you<br>
are doing something like:</p>
<pre data-code-wrap="python"><code class="lang-python">critic_loss.backward (retain_graph = True)
critic_optimizer.step()   # modifies critic's parameters inplace
...
actor_loss.backward()     # where actor loss depends on critic
actor_optimizer.step()
</code></pre>
<p>If so, you will try to backpropagate again through <code>critic</code>, which has had<br>
its parameters modified inplace by its optimizer.  Whether or not modifying<br>
a tensor inplace will cause an inplace-modification error depend on the<br>
details of whether that tensor is needed in the backward-pass computation,<br>
but it is likely that at least some of <code>critic</code>’s parameters will be needed in<br>
the backward pass.</p>
<p>If you have identified the tensor that is causing the inplace-modification<br>
error – likely one of <code>critic</code>’s parameters – print out its <code>._version</code> before<br>
and after calling <code>critic_optimizer.step()</code>.  If you can’t identify the tensor<br>
in question, you can still test this theory by commenting out the call to<br>
<code>critic_optimizer.step()</code> and see if this particulate inplace-modification<br>
error goes away.  (There may be others.)]</p>
<p>Also, look at the inplace-modification error itself.  It is telling you that a<br>
<code>FloatTensor</code> of shape <code>[256, 1]</code> is the tensor that is being modified<br>
inplace.  Where in your code do you have a tensor of that shape (that<br>
occurs somewhere in the forward pass)?  Look closely at how it is being<br>
used and if you can see an inplace modification.</p>
<p>Note that the error message is complaining that it should be of “version 3”<br>
rather than of “version 4.”  Print out the tensor’s <code>._version</code> property at<br>
various strategic places in your code.  The inplace modification is occurring<br>
somewhere between <code>._version</code> values of <code>3</code> and <code>4</code>.  You can insert<br>
intermediate <code>._version</code> print statements to perform a binary search to<br>
locate exactly where the inplace modification is occurring.</p>
<p>For example, you might try printing out <code>._version</code> just before and just<br>
after the call to <code>self.critic (x)</code> upon which the forward-call traceback<br>
casts some suspicion.</p>
<p>Note also that you are calling <code>.backward (retain_graph = True)</code>.<br>
First, make sure that this is correct logic for your use case.  If it is, be<br>
aware that calling <code>optimizer.step()</code> performs an inplace modification<br>
of the parameters being optimized by <code>optimizer</code>.  Again, you can check<br>
this by printing out <code>._version</code> for the problematic tensor before and after<br>
the call to <code>optimizer.step()</code>.</p>
<p>For some examples that illustrate these inplace-modification debugging<br>
techniques, see this post:</p>
<aside class="quote quote-modified" data-post="7" data-topic="171826">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-torch-floattensor-64-1-which-is-output-0-of-asstridedbackward0-is-at-version-3-expected-version-2-instead-hint-the-backtrace-further-a/171826/7">"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Fahmyadan and Sangyoon! 

Here are some suggestions about how to track down (and maybe fix) 
inplace-modification errors.  Note that an inplace modification in the forward 
pass is not necessarily* an error – it depends on whether and how the tensor 
that was modified is used in the backward pass.  Note that inplace operations 
can be useful for saving memory – if you replace an innocent inplace operation 
with an out-of-place equivalent, your training will use more memory (and, to a 
minor e…
  </blockquote>
</aside>

<p>Good luck!</p>
<p>K. Frank</p>

_aadharna: <p>The retain_graph=True was there because the previous error told to do that to get more info along with setting anomaly_detection mode on. It’s not usually on in my code.</p>
<p>Here’s the agent architecture, and the only place I have a [256, 1] tensor is <strong>the last layer of the critic</strong>:</p>
<pre><code class="lang-auto">class Agent(nn.Module):
    def __init__(self, envs):
        super().__init__()
        self.critic = nn.Sequential(OrderedDict([
            ('fc1', layer_init(nn.Linear(envs.single_observation_space.shape[0], 512))),
            ('relu1', nn.ReLU()),
            ('fc2', layer_init(nn.Linear(512, 256))),
            ('relu2', nn.ReLU()),
            ('critic_out', layer_init(nn.Linear(256, 1), std=1.0)),
        ]))
        self.actor = nn.Sequential(OrderedDict([
            ('fc1', layer_init(nn.Linear(envs.single_observation_space.shape[0], 512))),
            ('relu1', nn.ReLU()),
            ('fc2', layer_init(nn.Linear(512, 256))),
            ('relu2', nn.ReLU()),
            ('actor_out', layer_init(nn.Linear(256, envs.single_action_space.n), std=0.01)),
        ]))

    def get_value(self, x):
        return self.critic(x)

    def get_action_and_value(self, x, action=None):
        if x.ndim == 1:
            x = x.unsqueeze(0)
        logits = self.actor(x)
        probs = torch.distributions.Categorical(logits=logits)
        if action is None:
            action = probs.sample()
        return action, probs.log_prob(action), probs.entropy(), self.critic(x), logits.cpu().detach()
</code></pre>
<hr>
<p>So this is wild.</p>
<p>I started tracking the version number of the final linear layer of the critic network (and of the tensors that were the output of the value function and all things derived from the value function) and they’re all showing 1’s and 0’s as their versions.</p>
<p>To make things go faster, I shrunk the number of games that get played to create the on-policy dataset to 10, and now things just work. But once that number gets big enough (~25 games) then the error appears.</p>
<p>Here’s the code (via a zip to preserve file structure – it’s just the files in the gist structured appropriately): <a href="https://drive.google.com/file/d/1sr9EcyGr_F2xQcgdQD3bWf3JIhOE94gC/view?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">ppo_wip.zip - Google Drive</a></p>
<hr>
<p>Just saw your edit:</p>
<p>I am updating the entire network with one backward call.</p>
<pre><code class="lang-auto">            # Total loss
            loss = pg_loss - ent_coef * entropy_loss + v_loss * vf_coef + novelty_loss * 0.01
            # right now the novelty loss is 0 (locked behind an if statement that will always evaluate to False

            optimizer.zero_grad()
            # print("pre backward: ", learner.critic.critic_out._version)
            loss.backward(retain_graph=True)
            # print("post backward: ", learner.critic.critic_out._version)
            nn.utils.clip_grad_norm_(learner.parameters(), max_grad_norm)
            # print("post clip pre step: ", learner.critic.critic_out._version)
            optimizer.step()
            # print("post step: ", learner.critic.critic_out._version)
</code></pre>

KFrank: <p>Hi Aaron!</p>
<aside class="quote no-group" data-username="_aadharna" data-post="3" data-topic="183172" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/_aadharna/48/61756_2.png" class="avatar"> _aadharna:</div>
<blockquote>
<p>The retain_graph=True was there because the previous error told to do that to get more info</p>
</blockquote>
</aside>
<p>Try getting rid of <code>retain_graph = True</code>.  Whether or not it helps in<br>
debugging, an unnecessary <code>retain_graph = True</code> could well be the<br>
cause of your problem.  (Even a necessary <code>retain_graph = True</code><br>
could cause trouble, but then fixing things can be more involved.)</p>
<aside class="quote no-group">
<blockquote>
<p>Here’s the agent architecture, and the only place I have a [256, 1] tensor is <strong>the last layer of the critic</strong>:</p>
<pre><code class="lang-auto">            ('critic_out', layer_init(nn.Linear(256, 1), std=1.0)),
</code></pre>
</blockquote>
</aside>
<p>This makes sense.  The <code>.weight</code> property  of a <code>Linear (256, 1)</code> has<br>
shape <code>[1, 256]</code>, and, as I understand things, its transpose, with shape<br>
<code>[256, 1]</code>, is used in the backward pass, so that all agrees with your<br>
error messages.</p>
<aside class="quote no-group">
<blockquote>
<p>To make things go faster, I shrunk the number of games that get played to create the on-policy dataset to 10, and now things just work. But once that number gets big enough (~25 games) then the error appears.</p>
</blockquote>
</aside>
<p>Well, that’s a bit of a puzzle.</p>
<p>But, nevertheless, try printing out<br>
<code>critic.critic_out.weight._version</code>  before and after calling<br>
<code>optimizer.step()</code> and see if it increases consistently with the<br>
error message (at least in those runs where the error occurs).</p>
<aside class="quote no-group">
<blockquote>
<p>I am updating the entire network with one backward call.</p>
<pre><code class="lang-auto">            optimizer.zero_grad()
            # print("pre backward: ", learner.critic.critic_out._version)
            loss.backward(retain_graph=True)
            # print("post backward: ", learner.critic.critic_out._version)
            nn.utils.clip_grad_norm_(learner.parameters(), max_grad_norm)
            # print("post clip pre step: ", learner.critic.critic_out._version)
            optimizer.step()
            # print("post step: ", learner.critic.critic_out._version)
</code></pre>
</blockquote>
</aside>
<p>This should be fine with the exception of <code>retain_graph = True</code>.  Since<br>
your code logic doesn’t reuse the graph of <code>loss</code>, you should be fine<br>
getting rid of it.</p>
<p>(If you keep <code>retain_graph = True</code>, you do reuse parts of <code>loss</code>’s graph<br>
the next time through your training loop, and parts of that graph<br>
(presumably <code>critic_out.weight</code>) will have been modified inplace<br>
by <code>optimizer.step()</code>.)</p>
<p>So try <code>loss.backward()</code> without <code>retain_graph = True</code> and see if that<br>
fixes your error.</p>
<p>(As an aside, I’m not familiar with <code>clip_grad_norm_()</code>, but it does modify<br>
your <em>gradients</em> inplace.  Maybe the gradients only actually get clipped if<br>
the <code>grad_norm</code> exceeds <code>max_grad_norm</code> and maybe that only happens<br>
when you run 25 games rather than 10.  Whether this has anything to do<br>
with your actual error, I don’t know.)</p>
<p>Best.</p>
<p>K. Frank</p>

_aadharna: <p>So this just keeps getting wilder. Good news, I know how to run the code. Bad news, it makes no sense to me why I need to do what I need to do to get it to run.</p>
<p>I played around with the batch sizes and it seems to be that when I have multiple minibatches is when everything breaks. If I increase the number of games played to make up each update’s dataset such that it becomes big enough that I need to split it into minibatches, the error turns up.</p>
<p>But if I also increase the batch size it all works.</p>
<p>Alternatively, if I increase the number of optimization steps I can take on a single batch, that works fine. This is very odd.</p>
<pre><code class="lang-auto">def update_model(learner, learner_id, optimizer, env, data, n_steps, n_opt_steps, minibatch_size, gamma, gae_lambda,
                 clip_coef, norm_adv, clip_vloss, max_grad_norm, ent_coef, vf_coef, target_kl, device,
                 opponent=None, archive=None, ArchiveKL: Optional["ArchiveKL"]=None):
    # get data for the model we want to update
    rollouts = data[learner_id]
    obs = rollouts.obs
    actions = rollouts.actions
    logprobs = rollouts.logprobs
    rewards = rollouts.rewards
    dones = rollouts.dones
    values = rollouts.values
    logits = rollouts.logits
    next_obs = data['next_obs']
    next_done = data['next_dones']

    # print("extracted from storage pre GAE: ", values._version)
    # print("start of opt: ", learner.critic.critic_out._version)

    # bootstrap value if not done
    with torch.no_grad():
        sort_list = []
        advantages = []
        returns = []
        indices = torch.arange(0, rewards.shape[0]).long().to(device)
        next_value = learner.get_value(torch.FloatTensor(next_obs).unsqueeze(0).to(device))
        for player in ['player_0', 'player_1']:
            mask = np.array(rollouts.agent_id) == player
            masked_inds = list(indices[mask])
            lastgaelam = 0
            for t in reversed(range(mask.sum())):
                if t == mask.sum() - 1:
                    nextnonterminal = 1.0 - next_done
                    nextvalues = next_value
                else:
                    nextnonterminal = 1.0 - dones[mask][t + 1]
                    nextvalues = values[mask][t + 1]
                delta = rewards[mask][t] + gamma * nextvalues * nextnonterminal - values[mask][t]
                lastgaelam = delta + gamma * gae_lambda * nextnonterminal * lastgaelam
                advantages.append(lastgaelam)
                sort_list.append(masked_inds[t])
                returns.append(lastgaelam + values[mask][t])

        advantages = torch.cat(advantages)[torch.LongTensor(sort_list)].to(device)
        returns = torch.cat(returns)[torch.LongTensor(sort_list)].to(device)

    # print("extracted from storage post GAE: ", values._version)
    # print("return version: ", returns._version)

    # print("after gae: ", learner.critic.critic_out._version)

    # flatten the batch
    b_obs = obs.reshape((-1,) + env.single_observation_space.shape)
    b_logprobs = logprobs.reshape(-1)
    b_actions = actions.reshape((-1,) + env.single_action_space.shape)
    b_advantages = advantages.reshape(-1)
    b_returns = returns.reshape(-1)
    b_values = values.reshape(-1)
    b_logits = logits.reshape((-1,) + (env.single_action_space.n,))

    # print("flattened: ", b_values._version)

    # Optimizing the policy and value network
    batch_size = b_obs.shape[0]
    print(batch_size, minibatch_size)
    print(list(range(0, batch_size, minibatch_size)))
    b_inds = np.arange(batch_size)
    clipfracs = []
    for epoch in range(n_opt_steps):
        np.random.shuffle(b_inds)
        for start in range(0, batch_size, minibatch_size):
            end = start + minibatch_size
            mb_inds = b_inds[start:end]

            _, newlogprob, entropy, newvalue, _ = learner.get_action_and_value(b_obs[mb_inds].unsqueeze(1),
                                                                               b_actions.long()[mb_inds])

            # print("getting new log probs: ", learner.critic.critic_out._version)

            logits_archive = []
            if ArchiveKL is not None:
                # get logits for observations from each agent in the novelty archive
                for policy_id, policy in archive.items():
                    if policy_id in ["main", "greedy_main"]:
                        continue
                    # load policy into random agent
                    opponent.load_state_dict(policy)
                    logits_archive.append(opponent.get_action_and_value(b_obs[mb_inds].unsqueeze(1),
                                                                        b_actions.long()[mb_inds])[4])
                logits_archive = torch.stack(logits_archive, dim=0).detach()

            logratio = newlogprob - b_logprobs[mb_inds]
            ratio = logratio.exp()

            with torch.no_grad():
                # calculate approx_kl http://joschu.net/blog/kl-approx.html
                old_approx_kl = (-logratio).mean()
                approx_kl = ((ratio - 1) - logratio).mean()
                clipfracs += [((ratio - 1.0).abs() &gt; clip_coef).float().mean().item()]

            mb_advantages = b_advantages[mb_inds]
            if norm_adv:
                mb_advantages = (mb_advantages - mb_advantages.mean()) / (mb_advantages.std() + 1e-8)

            # Policy loss
            pg_loss1 = -mb_advantages * ratio
            pg_loss2 = -mb_advantages * torch.clamp(ratio, 1 - clip_coef, 1 + clip_coef)
            pg_loss = torch.max(pg_loss1, pg_loss2).mean()

            # Value loss
            newvalue = newvalue.view(-1)
            if clip_vloss:
                v_loss_unclipped = (newvalue - b_returns[mb_inds]) ** 2
                v_clipped = b_values[mb_inds] + torch.clamp(
                    newvalue - b_values[mb_inds],
                    -clip_coef,
                    clip_coef,
                )
                v_loss_clipped = (v_clipped - b_returns[mb_inds]) ** 2
                v_loss_max = torch.max(v_loss_unclipped, v_loss_clipped)
                v_loss = 0.5 * v_loss_max.mean()
            else:
                v_loss = 0.5 * ((newvalue - b_returns[mb_inds]) ** 2).mean()

            # Entropy loss
            entropy_loss = entropy.mean()

            # Novelty loss
            # try to maximize the KL divergence between the current policy and the archive
            novelty_loss = 0
            if ArchiveKL is not None:
                novelty_loss = ArchiveKL.forward(b_logits[mb_inds].unsqueeze(-1), logits_archive).mean()

            # Total loss
            loss = pg_loss - ent_coef * entropy_loss + v_loss * vf_coef + novelty_loss * 0.01

            optimizer.zero_grad()
            # print("pre backward: ", learner.critic.critic_out._version)
            loss.backward()
            # print("post backward: ", learner.critic.critic_out._version)
            nn.utils.clip_grad_norm_(learner.parameters(), max_grad_norm)
            # print("post clip pre step: ", learner.critic.critic_out._version)
            optimizer.step()
            # print("post step: ", learner.critic.critic_out._version)

        if target_kl is not None:
            if approx_kl &gt; target_kl:
                break

    # print("b_returns final version: ", b_returns._version)
    y_pred, y_true = b_values.cpu().detach().numpy(), b_returns.cpu().detach().numpy()
    var_y = np.var(y_true)
    explained_var = np.nan if var_y == 0 else 1 - np.var(y_true - y_pred) / var_y

    return (loss.item(), entropy_loss.item(), pg_loss.item(), v_loss.item(),
            explained_var, approx_kl.item(), np.mean(clipfracs), old_approx_kl.item()
            )
</code></pre>

KFrank: <p>Hi Aaron!</p>
<aside class="quote no-group" data-username="_aadharna" data-post="5" data-topic="183172" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/_aadharna/48/61756_2.png" class="avatar"> _aadharna:</div>
<blockquote>
<p>This is very odd.</p>
</blockquote>
</aside>
<p>Under some conditions, the error occurs; under others, it doesn’t.</p>
<p>This raises the question of whether you’re seeing the same error every<br>
time, or different, but similar, errors.</p>
<p>To be systematic about this, when the error does occur, do you get<br>
essentially the same error message, including the part in the forward-call<br>
traceback?</p>
<p>Does the same tensor get modified inplace and can you use <code>._version</code><br>
to track down where the tensor is being modified and verify that it is<br>
being modified in the same place?</p>
<p><code>Agent.get_action_and_value()</code> calls <code>self.critic (x)</code>, but it<br>
appears that you have multiple kinds of <code>Agent</code>s.  From the snippets<br>
you have posted, I see <code>main_policy.get_action_and_value()</code>,<br>
<code>learner.get_action_and_value()</code>, and<br>
<code>opponent.get_action_and_value()</code>.</p>
<p>Just speculation, but if, for example, you randomly choose an <code>opponent</code><br>
from a pool of <code>opponent</code>s, then maybe your batch size or the number of<br>
games played changes the probability that you choose an <code>opponent</code> that<br>
has already been used and has a <code>self.critic</code> whose parameters have<br>
already been modified inplace by an optimization step, so that when you<br>
backward through the <code>opponent.get_action_and_value()</code> call (and<br>
therefore backward through its call to <code>self.critic (x)</code>), you get the<br>
inplace-modification error.</p>
<p>Or something like that.</p>
<p>In any event, you need to determine which specific tensor is being modified<br>
inplace and which specific line of code is making that inplace modification in<br>
order to reliably fix the problem (and to figure out whether you have just one<br>
such error or a cluster of similar errors).</p>
<p>It does look like the <code>self.critic.critic_out.weight</code> tensor (whose<br>
transpose has shape <code>[256, 1]</code>) is being modified inplace.  But, since<br>
you have multiple <code>Agent</code>s, which <code>Agent</code>’s <code>critic</code> is being modified?</p>
<p>Good luck!</p>
<p>K. Frank</p>

_aadharna: <p>Fair. I definitely need to clean up the naming terminology.</p>
<p><code>main_policy</code> and <code>learner</code> are the same network (learner is the name of the model passed into the update_model function so that I can pass in other NNs to optimization code) and the <code>opponent</code> is never getting called (locked behind if statements that evaluate to False while I get the normal learning working).</p>
<p>Thanks for all your pointers!</p>