Trying to update parameters in a loop gives following error: One of the variables needed for gradient computation has been modified by an inplace operation

username: Dagenham
cooked: <p>Hi, I’m doing a policy gradient method in PyTorch and wanted to move the network update into the loop but it stopped working. I’m still a PyTorch newbie so sorry if the explanation is obvious. I’m using PyTorch 1.6 (built from source).</p>
<p>Here is the original code that works fine:</p>
<pre><code class="lang-auto">self.policy.optimizer.zero_grad()
G = T.tensor(G, dtype=T.float).to(self.policy.device) 

loss = 0
for g, logprob in zip(G, self.action_memory):
    loss += -g * logprob
                                 
loss.backward()
self.policy.optimizer.step()
</code></pre>
<p>And this is a change I was trying to make that doesn’t work:</p>
<pre><code class="lang-auto">G = T.tensor(G, dtype=T.float).to(self.policy.device) 

loss = 0
for g, logprob in zip(G, self.action_memory):
    loss = -g * logprob
    self.policy.optimizer.zero_grad()
                                 
    loss.backward()
    self.policy.optimizer.step()
</code></pre>
<p>This second snippet gives the following error:</p>
<pre><code class="lang-auto">File "g:\VScode_projects\pytorch_shenanigans\policy_gradient.py", line 86, in learn
    loss.backward()
  File "G:\Anaconda3\envs\pytorch_env\lib\site-packages\torch\tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "G:\Anaconda3\envs\pytorch_env\lib\site-packages\torch\autograd\__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128, 4]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>I read that this RuntimeError often has to do with having to clone something, because we’re using the same tensor to <a href="https://nieznanm.medium.com/runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-85d0d207623" rel="noopener nofollow ugc">compute itself</a> but I can’t make heads of tails of what is wrong in my case.</p>

username: ptrblck
cooked: <p>I’m not familiar with your use case and don’t know how all tensors are calculated.<br>
However, based on the error I guess you might be updating the parameters and trying to reuse an already “stale” forward activation as described in <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">this post</a>.</p>

username: Dagenham
cooked: <p>Thanks.</p>
<p>Colloquially speaking the forward passes are “precalculated” in self.action_memory in my case.<br>
Algorithm often calls the snippet below during an RL episode and the training loop in my question only takes place afterwards.</p>
<pre><code class="lang-auto">probabilities = F.softmax(self.policy.forward(observation))
action_probs = T.distributions.Categorical(probabilities)
action = action_probs.sample()
log_probs = action_probs.log_prob(action)
self.action_memory.append(log_probs)    
</code></pre>
<p>So in other words after one step(), the next backward() call doesn’t make much sense because the loss was calculated using old parameters? Am I understanding it correctly?</p>
<p>If so, one way to fix this error would be to execute forward passes in the loop as well, right?</p>
<p>Sorry if my train of thought (or wording) is confusing.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Dagenham" data-post="3" data-topic="111801">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dagenham/48/34654_2.png" class="avatar"> Dagenham:</div>
<blockquote>
<p>So in other words after one step(), the next backward() call doesn’t make much sense because the loss was calculated using old parameters? Am I understanding it correctly?</p>
</blockquote>
</aside>
<p>Yes, that’s correct. Additionally to this the intermediate activations were also calculated using the “old” parameters. The backward pass would use the new parameters with the old intermediates to calculate the gradients for the new parameters, which would be wrong.</p>
<aside class="quote no-group" data-username="Dagenham" data-post="3" data-topic="111801">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dagenham/48/34654_2.png" class="avatar"> Dagenham:</div>
<blockquote>
<p>If so, one way to fix this error would be to execute forward passes in the loop as well, right?</p>
</blockquote>
</aside>
<p>Yes, that would be an alternative solution to your first approach (calling <code>backward</code> and <code>step</code> after the loop).</p>
<aside class="quote no-group" data-username="Dagenham" data-post="3" data-topic="111801">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dagenham/48/34654_2.png" class="avatar"> Dagenham:</div>
<blockquote>
<p>Sorry if my train of thought (or wording) is confusing.</p>
</blockquote>
</aside>
<p>Don’t worry, it’s not at all confusing. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>