Help spotting inplace operation error

username: jsa378
cooked: <p>I am trying to implement a policy gradient method (using reward-to-go and a state value baseline). I’m given a template and my job is to fill in the gaps in the template.</p>
<p>When I try to run the code after filling in the gaps, I get an inplace error (shown at end of post). The template should not have errors, so the problems I encounter should be due to the code I’ve written.</p>
<p><strong>Essentially, I asking if anybody can spot an inplace operation (and provide a workaround) in the code that I show below.</strong></p>
<p>The parts I have filled in are shown below.</p>
<p><strong>1. A function to calculate the return-to-go, given a string of returns over multiple episodes</strong></p>
<pre><code class="lang-auto">    def _calc_return(self, r, done):
        '''
        TODO 2.1: Given a tensor of per-timestep rewards (r), and a tensor (done)
        indicating if a timestep is the last timestep of an episode, Output a
        tensor (return_t) containing the return (i.e. reward-to-go) at each timestep.
        '''
        
        gamma = self._discount
        done_indices = torch.nonzero(done, as_tuple=False)
        indices = torch.cat((torch.zeros(1), torch.squeeze(torch.transpose(done_indices, 0, 1)), torch.Tensor([len(r) - 1])))
        return_t_list = []

        for i in range(len(indices) - 1):
            if i == 0:
                episode = r[int(indices[i]):int(indices[i + 1] + 1)]
                episode_indices = torch.arange(0, len(episode))
                for j in range(len(episode_indices)):
                    episode_to_go = episode[j:]
                    indices_to_go  = episode_indices[j:]
                    gammas = torch.full([len(episode_to_go)], gamma)
                    exponents = episode_indices[:len(episode_indices) - j]
                    discounts = torch.pow(gammas, exponents)
                    discounted_sum = torch.dot(episode_to_go, discounts)
                    return_t_list.append(discounted_sum)
            if i != 0:
                episode = r[int(indices[i] + 1):int(indices[i + 1] + 1)]
                episode_indices = torch.arange(0, len(episode))
                for j in range(len(episode_indices)):
                    episode_to_go = episode[j:]
                    indices_to_go  = episode_indices[j:]
                    gammas = torch.full([len(episode_to_go)], gamma)
                    exponents = episode_indices[:len(episode_indices) - j]
                    discounts = torch.pow(gammas, exponents)
                    discounted_sum = torch.dot(episode_to_go, discounts)
                    return_t_list.append(discounted_sum)

        return_t = torch.stack(return_t_list)

        return return_t
</code></pre>
<p><strong>2. A function to calculate the “advantage” (reward-to-go minus state value)</strong></p>
<pre><code class="lang-auto">    def _calc_adv(self, norm_obs, ret):
        '''
        TODO 2.2: Given the normalized observations (norm_obs) and the return at
        every timestep (ret), output the advantage at each timestep (adv).
        '''
        
        value = self._model.eval_critic(norm_obs)
        value = torch.squeeze(torch.transpose(value, 0, 1))
        adv = ret - value
        return adv
</code></pre>
<p><strong>3. A function to calculate the critic loss, i.e. the loss used to train the value function</strong></p>
<pre><code class="lang-auto">    def _calc_critic_loss(self, norm_obs, tar_val):
        '''
        TODO 2.3: Given the normalized observations (norm_obs) and the returns at
        every timestep (tar_val), compute a loss for updating the value
        function (critic).
        '''

        value = self._model.eval_critic(norm_obs)
        value = torch.squeeze(torch.transpose(value, 0, 1))
        squared_diff = (tar_val - value)**2
        loss = squared_diff.mean()
        return loss
</code></pre>
<p><strong>4. A function to calculate the actor loss, i.e the loss used to train the policy</strong></p>
<pre><code class="lang-auto">    def _calc_actor_loss(self, norm_obs, norm_a, adv):
        '''
        TODO 2.4: Given the normalized observations (norm_obs), normalized
        actions (norm_a), and the advantage at every timestep (adv), compute
        a loss for updating the policy (actor).
        '''
        
        policy = self._model.eval_actor(norm_obs)
        policy_a = policy.log_prob(norm_a)
        loss = -(adv * policy_a).mean()
        return loss
</code></pre>
<p>Now, when I run all the code, I get the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "run.py", line 103, in &lt;module&gt;
    main(sys.argv)
  File "run.py", line 93, in main
    train(agent=agent, max_samples=max_samples, out_model_file=out_model_file, 
  File "run.py", line 50, in train
    agent.train_model(max_samples=max_samples, out_model_file=out_model_file, 
  File "/Users/jesse/rl_assignments/learning/base_agent.py", line 57, in train_model
    train_info = self._train_iter()
  File "/Users/jesse/rl_assignments/learning/base_agent.py", line 226, in _train_iter
    train_info = self._update_model()
  File "/Users/jesse/rl_assignments/a2/pg_agent.py", line 144, in _update_model
    actor_info = self._update_actor(actor_batch)
  File "/Users/jesse/rl_assignments/a2/pg_agent.py", line 187, in _update_actor
    loss.backward()
  File "/Users/jesse/anaconda3/envs/rl/lib/python3.8/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/Users/jesse/anaconda3/envs/rl/lib/python3.8/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 41; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>Essentially, I am guessing that I’ve done an inplace operation somewhere that is messing with the gradient computation. However, I do not know what to look for, and <code>torch.autograd.set_detect_anomaly(True)</code> doesn’t seem helpful.</p>
<p>I appreciate any help.</p>

username: vmoens
cooked: <p>Hey! Can you also provide the part of the code where you call the backwards?<br>
Thanks!</p>

username: jsa378
cooked: <p>Hello, thanks for replying. I assume you are referring to the code surrounding the <code>loss.backward()</code> call. Here is the function containing that call:</p>
<pre><code class="lang-auto">    def _update_actor(self, batch):
        norm_obs = batch["norm_obs"]
        norm_a = batch["norm_action"]
        adv = batch["adv"]
        
        loss = self._calc_actor_loss(norm_obs, norm_a, adv)
        
        info = {
            "actor_loss": loss
        }
        
        if (self._action_bound_weight != 0):
            a_dist = self._model.eval_actor(norm_obs)
            action_bound_loss = self._compute_action_bound_loss(a_dist)
            if (action_bound_loss is not None):
                action_bound_loss = torch.mean(action_bound_loss)
                loss += self._action_bound_weight * action_bound_loss
                info["action_bound_loss"] = action_bound_loss.detach()

        self._actor_optimizer.zero_grad()
        loss.backward()
        self._actor_optimizer.step()
        
        return info
</code></pre>
<p>Let me know if you would like to see anything else. Thanks.</p>

username: eqy
cooked: <p><code>loss += self._action_bound_weight * action_bound_loss</code> looks problematic as that would be an in-place operation.</p>
<p>Does changing it to an out-of-place one via<br>
<code>loss = loss + ...</code><br>
work?</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; loss = torch.ones(10, requires_grad=True)
&gt;&gt;&gt; loss += 1
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.
&gt;&gt;&gt; loss = torch.ones(10, requires_grad=True)
&gt;&gt;&gt; loss = loss + 1 #OK
&gt;&gt;&gt; loss.sum().backward()
&gt;&gt;&gt;
</code></pre>

username: jsa378
cooked: <p>Thanks for replying. In the <code>_update_actor</code> function, I commented out <code>loss += self._action_bound_weight * action_bound_loss</code> and put in <code>loss = loss + self._action_bound_weight * action_bound_loss</code>.</p>
<p>Looks like I get a very similar (possibly identical) error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "run.py", line 103, in &lt;module&gt;
    main(sys.argv)
  File "run.py", line 93, in main
    train(agent=agent, max_samples=max_samples, out_model_file=out_model_file, 
  File "run.py", line 50, in train
    agent.train_model(max_samples=max_samples, out_model_file=out_model_file, 
  File "/Users/jesse/rl_assignments/learning/base_agent.py", line 57, in train_model
    train_info = self._train_iter()
  File "/Users/jesse/rl_assignments/learning/base_agent.py", line 226, in _train_iter
    train_info = self._update_model()
  File "/Users/jesse/rl_assignments/a2/pg_agent.py", line 144, in _update_model
    actor_info = self._update_actor(actor_batch)
  File "/Users/jesse/rl_assignments/a2/pg_agent.py", line 188, in _update_actor
    loss.backward()
  File "/Users/jesse/anaconda3/envs/rl/lib/python3.8/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/Users/jesse/anaconda3/envs/rl/lib/python3.8/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 41; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>I am not sure if this is helpful, but in <code>_calc_actor_loss</code>, if I comment out the code I shared in my original post and just use</p>
<pre><code class="lang-auto">    def _calc_actor_loss(self, norm_obs, norm_a, adv):
        loss = torch.zeros(1)
        return loss
</code></pre>
<p>then the code runs without error, although obviously the computations will be garbage.</p>
<p>Thanks for your help.</p>

username: eqy
cooked: <p>Well, it’s difficult to say where the problem is without any visibility into how<br>
<code>policy.log_prob</code> and <code>self._model.eval_actor</code> are implemented.</p>
<p>The last line doesn’t seem to be obviously wrong</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.ones(10, requires_grad=True)
&gt;&gt;&gt; b = torch.ones(10, requires_grad=True)
&gt;&gt;&gt; loss = -(a * b).mean()
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt;
</code></pre>

username: jsa378
cooked: <p>The code is spread across a number of files, so I’m not absolutely sure these are the correct functions:</p>
<pre><code class="lang-auto">    def eval_actor(self, obs):
        h = self._actor_layers(obs)
        a_dist = self._action_dist(h)
        return a_dist
</code></pre>
<pre><code class="lang-auto">    def log_prob(self, x):
        diff = x - self._mean
        logp = -0.5 * torch.sum(torch.square(diff / self._std), dim=-1)
        logp += -0.5 * self._dim * np.log(2.0 * np.pi) - torch.sum(self._logstd, dim=-1)
        # logp = logp + -0.5 * self._dim * np.log(2.0 * np.pi) - torch.sum(self._logstd, dim=-1)
        return logp
</code></pre>
<p>Using the <code>logp = logp + ...</code> line in <code>log_prob</code> didn’t make any difference. I suppose this means that I have to try to dig into <code>eval_actor</code>.</p>
<p>The whole thing is a bit strange, though, because I’m not supposed to have to modify the <code>.py</code> files containing the above functions. Hence my focus on the four functions I provided in my original post.</p>

username: KFrank
cooked: <p>Hi jsa!</p>
<p>Two comments:  The first analyzes the backward-pass error message; and the<br>
second concerns <code>set_detect_anomaly (True)</code>.</p>
<aside class="quote no-group" data-username="jsa378" data-post="1" data-topic="181369" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/bc79bd/48.png" class="avatar"> jsa378:</div>
<blockquote>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 41; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
</blockquote>
</aside>
<p>This error message gives two useful pieces of information:</p>
<p>First, it tells you the type and shape of the tensor being modified inplace,<br>
<code>torch.FloatTensor [64, 1]</code>; and second it tells you that the tensor is being<br>
modified inplace forty times, <code>version 41; expected version 1 instead.</code></p>
<p>Where in your code do you have tensors of that shape?  Take a look at them<br>
and the inplace modification might become apparent.  Where might such a<br>
tensor be modified many times?  Do you have a loop or other repetitive structure<br>
in your forward pass?  Look there.</p>
<p>You can use your tensor’s <code>_version</code> property to track down where the inplace<br>
modification occurs with a divide-and-conquer scheme.  Look at the candidate<br>
tensors, <code>t</code>, that have the reported shape and print out <code>t._version</code> at various<br>
places in your forward pass.  When you find a section of code where <code>_version</code><br>
increases, you’ve found the section that contains your inplace modification.<br>
Now print out <code>t._version</code> at various places in that section of code to further<br>
narrow down the location.  If need be, keep doing this until you’ve bracketed<br>
a single line of code with <code>t._version</code>s that show <code>_version</code> increasing.  That<br>
line is your problem.</p>
<p>It might be the case that you have more than one line of code that makes<br>
an inplace modification (that matters to your backward pass) – if so, you<br>
will need to find and fix them all.</p>
<p>(You may have multiple candidate tensors with the same shape and type.<br>
You can print out <code>t._version</code> for all of them where they’re first created and<br>
then again at the end of your forward pass to determine which one is the<br>
culprit.)</p>
<aside class="quote no-group">
<blockquote>
<p><code>torch.autograd.set_detect_anomaly(True)</code> doesn’t seem helpful.</p>
</blockquote>
</aside>
<p><code>detect_anomaly (True)</code> is often very helpful.  Look in the “Traceback of<br>
forward call” that <code>detect_anomaly</code> causes to be printed out when the inplace<br>
modification error is detected in the backward pass.  Look for lines in that<br>
traceback that reference <em>your</em> code (rather than pytorch internals).  This<br>
typically leads you to the line of code that is making the inplace modification.</p>
<p>As an aside, I’ve noticed a number of posts where inplace-modification errors<br>
crop up in policy / actor-critic code.    Maybe there’s something about these<br>
algorithms that is conducive to such errors, or maybe there’s some sample code<br>
out there with landmines in it.  In any event, if you find your bug, please post<br>
what and where it was, how you found it, and how it was caused by / related to<br>
actor-critic (if indeed it was).  This could be a big help to those who might have<br>
similar problems in the future.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: jsa378
cooked: <p>K. Frank: Thanks for your reply. I will try to deal with it when I have more time.</p>
<p>I went to get some help with my code from a teaching assistant. The TA suggested that I change the <code>_calc_adv</code> function from the way I had it, i.e.</p>
<pre><code class="lang-auto">        value = self._model.eval_critic(norm_obs)
        value = torch.squeeze(torch.transpose(value, 0, 1))
        adv = ret - value
        return adv
</code></pre>
<p>to the following:</p>
<pre><code class="lang-auto">        value = self._model.eval_critic(norm_obs)
        value = torch.squeeze(value).detach()
        adv = ret - value
        return adv
</code></pre>
<p>Now the code runs without error.</p>
<p>I am not totally convinced this is a good solution. I’m wondering if this will mess up the critic loss computation. I have done a little bit of training, and the critic loss seems very high compared to the actor loss, and the test return seems kind of low, but perhaps I just haven’t trained enough. (I’m also not an expert in any of this.)</p>
<p>I will train more and see what happens.</p>

username: vmoens
cooked: <p>FYI we have a page dedicated to this in torchrl doc:<br>
<a href="https://pytorch.org/rl/reference/generated/knowledge_base/PRO-TIPS.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/rl/reference/generated/knowledge_base/PRO-TIPS.html</a></p>

username: KFrank
cooked: <p>Hi jsa!</p>
<aside class="quote no-group quote-modified" data-username="jsa378" data-post="9" data-topic="181369" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/bc79bd/48.png" class="avatar"> jsa378:</div>
<blockquote>
<pre><code class="lang-auto">        value = self._model.eval_critic(norm_obs)
        value = torch.squeeze(value).detach()
</code></pre>
<p>I am not totally convinced this is a good solution. I’m wondering if this will mess up the critic loss computation.</p>
</blockquote>
</aside>
<p>I think your skepticism about this “fix” is well founded.  I think it more likely<br>
that this is sweeping the real problem under the rug rather than correcting it.</p>
<p><code>detach()</code> disconnects the result of <code>squeeze()</code> from the computation graph<br>
so that backpropagation doesn’t continue on back up through <code>eval_critic()</code>.<br>
If, for example, <code>eval_critic()</code> contained the bad inplace modification, you<br>
haven’t fixed it – you just don’t backpropagate through it, so you suppress the<br>
error message and hide the real problem.</p>
<p>This could explain why your critic loss seems high.  If you don’t backpropagate<br>
through <code>eval_critic()</code>, the parameters that feed into <code>eval_critic()</code> and<br>
presumably contribute to your “critic loss” don’t get trained, so your critic loss<br>
doesn’t get better.</p>
<p>As an aside, it appears that your TA may be a practitioner of what I sometimes<br>
call “Monte Carlo programming” – change this, change that, add this, delete<br>
that until your program “works.”  The problem with this is that your program<br>
probably doesn’t actually work – you’ve just masked the most obvious<br>
consequences of your bugs, but the bugs are still there.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: jsa378
cooked: <p>If anyone is waiting for a denouement, I asked the professor about the use of <code>.detach()</code>, specifically whether this would mess up the critic loss computations, and sweep the real issue under the rug. I was told that it was correct to use <code>.detach()</code> as suggested in order to cut off the gradient computation in the right place. (I did not press for a detailed explanation.)</p>
<p>Presumably the professor knows this code better than anybody, so I guess that’s that. Thanks for the help.</p>

username: Dear_Deer
cooked: <p>I was following your issue. So, did you get the results with detach() ?</p>

username: jsa378
cooked: <p>The code ran without error, and the test return was more or less what it I was told it should be. So as far as I know, things worked properly.</p>

username: ptrblck
cooked: <p>You should note, however, that <a class="mention" href="/u/kfrank">@KFrank</a>’s explanation is correct and indeed detaching <code>value</code> would cut the computation graph at this point and the parameters involved to calculate <code>value</code> might not be trained.<br>
Also, <a class="mention" href="/u/jsa378">@jsa378</a> explained “I did not press for a detailed explanation”, which I think is also concerning especially when also claiming that “presumably the processor knows this code better than anybody”.</p>

username: jsa378
cooked: <p>Commenting on ptrblck’s last paragraph: It’s a slightly tricky situation for me. You guys maybe/probably know PyTorch better than my professor, but I cannot dump all the code here and expect anybody to comb through all of it.</p>
<p>So I cannot be completely sure whether you guys or my professor are correct about this. All I can say is that after implementing <code>.detach()</code> as suggested, the code runs and performs roughly as the professor said it would. Perhaps using <code>.detach()</code> as suggested is not best practice - or maybe it’s the right thing to do, given the context of the code.</p>
<p>(To anybody reading this: This is not advice. I am not an authority on PyTorch.)</p>
<p>Thanks for everyone’s help.</p>

username: KFrank
cooked: <p>Hi jsa!</p>
<p>The following is meant as an honest suggestion, not as a rhetorical comment.</p>
<aside class="quote no-group" data-username="jsa378" data-post="16" data-topic="181369" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/bc79bd/48.png" class="avatar"> jsa378:</div>
<blockquote>
<p>Perhaps using <code>.detach()</code> as suggested is not best practice - or maybe it’s the right thing to do, given the context of the code.</p>
</blockquote>
</aside>
<p>First, you have to do whatever is best to keep your TA and professor happy<br>
in the context of your project.</p>
<p>However, since you’re trying to learn pytorch and neural networks, if and when<br>
you have a free moment, you might try to track down where in your code the<br>
inplace modification is occurring.</p>
<p>Then look at the logic of your model and training algorithm and form your<br>
<em>own</em> opinion about whether or not that inplace modification is “legitimate”<br>
and whether the use of <code>.detach()</code> is or is not an appropriate solution in<br>
the context of your specific use case.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: jsa378
cooked: <p>K. Frank,</p>
<p>Thanks for the advice. I will see if I have time when the course is over.</p>