Cannot find where's the in-place operation error

ntuce002: <p>I’m trying to train the MADDPG model; however, it has occurred an in-place operation error.<br>
Here’s the traceback, and I’ve taken some excerpts that I think are critical:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [16, 2]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
</blockquote>
<blockquote>
<p>[W …\torch\csrc\autograd\python_anomaly_mode.cpp:104] Warning: Error detected in AddmmBackward0. Traceback of forward call that caused the error:</p>
</blockquote>
<blockquote>
<p>c_loss, a_loss = all_agents.learn(memory)<br>
File “C:\Users\chhuang\new_dens_speed_offset\maddpg.py”, line 107, in learn<br>
pi = self.agents[idx - 1].actor(car_dens_state[‘%s’ % idx], scooter_dens_state[‘%s’ % idx],<br>
File “C:\Users\chhuang\anaconda3_1\lib\site-packages\torch\nn\modules\module.py”, line 1102, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “C:\Users\chhuang\new_dens_speed_offset\networkv4.py”, line 73, in forward<br>
xyzw = self.mu(xyzw)</p>
</blockquote>
<p>It showed the error had occurred in networkv4.py, line 73.<br>
The code <code>xyzw = self.mu(xyzw)</code> is in the <code>forward()</code> method, under the <code>Actor()</code> class. The whole code of the <code>Actor()</code> class is down below:</p>
<pre><code class="lang-auto">class Actor(nn.Module):
    def __init__(self, alpha, name, chkpt_dir):
        super(Actor, self).__init__()
        
        self.chkpt_file = os.path.join(chkpt_dir, name)

        self.fc1 = nn.Linear(4, 16)
        self.fc2 = nn.Linear(64, 64)
        self.fc3 = nn.Linear(64, 32)
        self.fc4 = nn.Linear(32, 16)
        
        self.bn1 = nn.LayerNorm(16)
        self.bn2 = nn.LayerNorm(64)
        self.bn3 = nn.LayerNorm(32)
        self.bn4 = nn.LayerNorm(16)
        
        self.mu = nn.Linear(16, 2)
        
        self.optimizer = optim.Adam(self.parameters(), lr=alpha)
        self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
        self.to(self.device)

    def forward(self, car_D, scooter_D, car_S, scooter_S):
        torch.autograd.set_detect_anomaly(True)
        x = self.fc1(car_D)
        x = self.bn1(x)
        x = F.relu(x).flatten(1)
        
        y = self.fc1(scooter_D)
        y = self.bn1(y)
        y = F.relu(y).flatten(1)
        
        z = self.fc1(car_S)
        z = self.bn1(z)
        z = F.relu(z).flatten(1)
        
        w = self.fc1(scooter_S)
        w = self.bn1(w)
        w = F.relu(w).flatten(1)
        
        xyzw = torch.cat([x, y, z, w], dim=1)
        xyzw = self.fc2(xyzw)
        xyzw = self.bn2(xyzw)
        xyzw = F.relu(xyzw)
        
        xyzw = self.fc3(xyzw)
        xyzw = self.bn3(xyzw)
        xyzw = F.relu(xyzw)
        
        xyzw = self.fc4(xyzw)
        xyzw = self.bn4(xyzw)
        xyzw = F.relu(xyzw)
        xyzw = self.mu(xyzw)
        xyzw = torch.sigmoid(xyzw)
        return xyzw

</code></pre>
<p>And here’s the link to the whole code of <a href="https://github.com/chhuang17/MADDPG-trafflc_signal/blob/main/maddpg.py" rel="noopener nofollow ugc">maddpg.py</a>. They are a little bit ugly so I uploaded them to the github instead of posting them here.</p>
<p>I’ve stuck with this problem all day long, and still couldn’t find out where’s the bug. Hope someone can give me some directions to modify my code properly.</p>

ptrblck: <p>I guess the issue might be raised by using</p>
<pre><code class="lang-python">critic_loss.backward(retain_graph=True)
...
actor_loss.backward(retain_graph=True)
...
self.agents[idx - 1].actor.optimizer.step()
self.agents[idx - 1].update_network_parameters()
</code></pre>
<p>Using <code>retain_graph=True</code> won’t release the computation graph which often yields these types of errors (e.g. if the parameters were already updated and thus the forward activations are stale as described <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">here</a>.<br>
Could you explain why <code>retain_graph=True</code> is used?</p>

zanga: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a>, I know this is a little outdated but I came across the same error.<br>
The github link doesn’t work no more so I cannot see the whole code but if I’m guessing it right, during learning <a class="mention" href="/u/ntuce002">@ntuce002</a> has to make multiple passes through the graph (1 for each agent) so it needs to be retained.</p>
<p>Maybe I’m totally wrong and my understanding of the mechnisms behind it are as well but how can it work if we don’t retain the graph ?</p>
<p>Thank you.</p>

ptrblck: <p>I don’t know enough anymore about this use case and was speculating what might be causing the issue. Unfortunately, <a class="mention" href="/u/ntuce002">@ntuce002</a> didn’t follow up so I don’t know if my guess was right.</p>

zanga: <p>Thank you for the reply, I think I’ll open a new post if you have time to check my problem.</p>

ptrblck: <p>Sure, you can create a new topic for your issue. If possible, post a minimal and executable code snippet which would reproduce the error, so that I could directly debug it.</p>