"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a

Sangyoon1207: <p>I am trying to make a Multi-Agent Deep reinforcement learning (Soft Actor-Critic) algorithm in pytorch.<br>
And I got the following error:<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
<p>I found some people asked the similar questions in this website.<br>
I did some ways people recommeded and it did not worked.</p>
<p>Error appears when actor network is in update process.</p>
<p>Following codes are the learning process of my code.<br>
The error happens in the following code.</p>
<p>It would be very appreciate if you help me.</p>
<pre><code>def learn(self, memory):
    if not memory.ready():
        return

    actor_state, state, action, reward, actor_new_state, state_, done = memory.sample_buffer()

    state = T.tensor(state, dtype=T.float)
    action = T.tensor(action, dtype=T.float)
    reward = T.tensor(reward)
    state_ = T.tensor(state_, dtype=T.float)
    done = T.tensor(done)

    agents_actions = []

    actions_for_value_update = []
    log_probs_for_value_update = []
    for agent_idx, agent in enumerate(self.agents):
        a_state = T.tensor(actor_state[agent_idx], dtype=T.float)
        v_action, v_log_probs = agent.actor.sample_normal(a_state, reparameterize=False)
        v_log_probs = v_log_probs.view(-1)

        actions_for_value_update.append(v_action)
        log_probs_for_value_update.append(v_log_probs)

    actions_for_value_update = T.hstack((actions_for_value_update[:]))

    for agent_idx, agent in enumerate(self.agents):
        agents_actions.append(action[agent_idx])

        value = agent.value(state).view(-1)

        q1_new_policy = agent.critic1.forward(state=state, action=actions_for_value_update)
        q2_new_policy = agent.critic2.forward(state=state, action=actions_for_value_update)
        critic_value = T.min(q1_new_policy, q2_new_policy)
        critic_value = critic_value.view(-1)

        agent.value.optimizer.zero_grad()
        value_target = -log_probs_for_value_update[agent_idx].to(T.float32) + critic_value
        value_loss = 0.5 * F.mse_loss(value, value_target.detach())

        value_loss.backward(retain_graph=True)
        agent.value.optimizer.step()

        agent.update_network_parameters()

    actions = T.cat([acts for acts in agents_actions], dim=1)

    actions_for_actor_update = []
    log_probs_for_actor_update = []


    for agent_idx, agent in enumerate(self.agents):

        a_state = T.tensor(actor_state[agent_idx], dtype=T.float)
        a_action, a_log_probs = agent.actor.sample_normal(a_state.clone(), reparameterize=True)
        a_log_probs = a_log_probs.view(-1)
        actions_for_actor_update.append(a_action)
        # actions_for_actor_update.append(a_action)
        log_probs_for_actor_update.append(a_log_probs)

    actions_for_actor_update = (T.hstack((actions_for_actor_update[:]))).float()

    for agent_idx, agent in enumerate(self.agents):
        #print(agent_idx)
        q1_new_policy = agent.critic1.forward(state=state, action=actions_for_actor_update) 
        q2_new_policy = agent.critic2.forward(state=state, action=actions_for_actor_update)
        critic_value = T.min(q1_new_policy.clone(), q2_new_policy.clone())
        critic_value = critic_value.view(-1).clone()

        actor_loss = log_probs_for_actor_update[agent_idx].to(T.float32).clone() - critic_value
        actor_loss = T.mean(actor_loss)
        agent.actor.optimizer.zero_grad()
        actor_loss.backward(retain_graph=True)
        agent.actor.optimizer.step()

        value_ = agent.target_value(state_).view(-1)
        value_[done[:, 0]] = 0.0

        agent.critic1.optimizer.zero_grad()
        agent.critic2.optimizer.zero_grad()
        q_hat = 1.0 * reward[:, agent_idx].to(T.float32) + agent.gamma * value_

        q1_old_policy = agent.critic1.forward(state=state, action=actions).view(-1)
        q2_old_policy = agent.critic2.forward(state=state, action=actions).view(-1)
        critic1_loss = 0.5 * F.mse_loss(q1_old_policy, q_hat)
        critic2_loss = 0.5 * F.mse_loss(q2_old_policy, q_hat)

        critic_loss = critic1_loss + critic2_loss
        critic_loss.backward()
        agent.critic1.optimizer.step()
        agent.critic2.optimizer.step()
</code></pre>

Sangyoon1207: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hello. In the other links, I saw you solved the errors similar to my one (Above error).<br>
I did tried a lot of things, but it didn’t worked.<br>
I may ask you for a help in the error I faced. Thank you.</p>

ptrblck: <p>Often these issues can be raised by using <code>retain_graph=True</code> as a workaround for another error:</p>
<pre><code class="lang-python">value_loss.backward(retain_graph=True)
</code></pre>
<p>Could you explain why <code>retain_graph=True</code> is used here, please?</p>

Sangyoon1207: <p>First, thanks for replying.</p>
<p>I found that <code>retain_graph=True</code>  could be erased in updating <strong>value network</strong> (i.e., <code>value_loss.backward()</code>).</p>
<p>However, if I erase<br>
<code>retain_graph=True</code><br>
when updating <strong>actor network</strong> (i.e.,<code> actor_loss.backward()</code>), following error comes:<br>
<em><strong>“RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.”</strong></em></p>
<p><strong>The above error comes when in Multi-agent environment. It does not appear when Single-agent environment.</strong></p>

fahmyadan: <p>Hello <a class="mention" href="/u/sangyoon1207">@Sangyoon1207</a></p>
<p>I have been faced with the same error in a MultiAgent environment. I am currently looking for a solution but I strongly suspect it would be something related to <a href="https://discuss.pytorch.org/t/inplace-operation-errors-when-implementing-a2c-algorithm/145406/5">this post</a>. Hopefully you find it helpful</p>

KFrank: <p>Hi Fahmyadan and Sangyoon!</p>
<aside class="quote no-group" data-username="fahmyadan" data-post="6" data-topic="171826" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fahmyadan/48/55168_2.png" class="avatar"> fahmyadan:</div>
<blockquote>
<p>I have been faced with the same error</p>
</blockquote>
</aside>
<p>Here are some suggestions about how to track down (and maybe fix)<br>
inplace-modification errors.  Note that an inplace modification in the forward<br>
pass is not <em>necessarily</em>* an error – it depends on whether and how the tensor<br>
that was modified is used in the backward pass.  Note that inplace operations<br>
can be useful for saving memory – if you replace an innocent inplace operation<br>
with an out-of-place equivalent, your training will use more memory (and, to a<br>
minor extent, take more time).</p>
<p>First, understand what can be causes of inplace modifications.  These include<br>
inplace tensor functions, indicated by names with trailing underscores, such<br>
as <code>.zero_()</code>, <code>.copy_()</code>, and <code>.tanh_()</code>.  There are also some functions that<br>
can be called with a boolean <code>inplace</code> flag that tell them to operate inplace,<br>
such as <code>torch.nn.functional.relu (x, inplace = True)</code>.</p>
<p>Writing into a tensor using indices, e.g., <code>t[0, 1] = 10.0</code>, modifies the tensor<br>
inplace, as does, for example, <code>.scatter_()</code>.</p>
<p>Fixing an inplace error can be as easy as replacing an inplace function with<br>
an out-of-place version:</p>
<pre data-code-wrap="python"><code class="lang-python"># instead of
s.tanh_()
# use
t = t.tanh()
</code></pre>
<p>or <code>clone()</code> the tensor before modifying it:</p>
<pre data-code-wrap="python"><code class="lang-python"># instead of
s[0, 1] = 10.0
# use
t = t.clone()
t[0, 1] = 10.0
</code></pre>
<p>Let’s look at how to track down inplace errors – here’s an example script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

_ = torch.manual_seed (2023)

class DummyResNet (torch.nn.Module):
    def __init__ (self, printVersions = False):
        super().__init__()
        self.printVersions = printVersions
        self.lin1 = torch.nn.Linear (5, 10)
        self.lin2 = torch.nn.Linear (10, 10)
        self.lin3 = torch.nn.Linear (10, 1)
        self.relu = torch.nn.ReLU()
    def forward (self, x):
        y1 = self.lin1 (x)    # shape [3, 10]
        sc = self.relu (y1)   # shape [3, 10]
        y2 = self.lin2 (sc)   # shape [3, 10]
        y3 = self.relu (y2)   # shape [3, 10]
        y3 += sc   # skip connection
        y = self.lin3 (y3)    # shape [3, 1]
        if  self.printVersions:
            print ('versions:')
            print ('y1:', y1._version)
            print ('sc:', sc._version)
            print ('y2:', y2._version)
            print ('y3:', y3._version)
            print ('y: ', y._version)
        return y

drn = DummyResNet()
inp = torch.randn (3, 5)   # batch size of 3

print ('\nrun model and backward pass ...')
try:
    out = drn (inp)
    loss = out.sum()
    loss.backward()
except Exception as e:
    print ('%s: %s' % (type (e).__name__, e))

print ('\nrun forward pass with versions ...')
drn_versions = DummyResNet (True)
out = drn_versions (inp)

print ('\nrun again with anomaly detection ...')
with torch.autograd.detect_anomaly():
    out = drn (inp)
    loss = out.sum()
    loss.backward()
</code></pre>
<p>This is a toy version of a resnet that implements its skip connection with an<br>
inplace add:</p>
<pre data-code-wrap="python"><code class="lang-python">        y3 += sc   # skip connection
</code></pre>
<p>Here’s the output from the first part of the script:</p>
<pre><code class="lang-plaintext">1.13.1

run model and backward pass ...
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 10]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>The inplace-modification error gives us two useful pieces of information:<br>
First is the shape of the modified tensor (and its version number, see below);<br>
and second is that <code>ReluBackward0</code> is complaining about the modification.</p>
<p>Our toy model has two <code>relu()</code>s (both of which produce tensors of shape<br>
<code>[3, 10]</code>).  The first looks safe, while the second has its output modified<br>
inplace by <code>y3 += sc</code>, and this is indeed the cause of the error.</p>
<p>But, as suggested by the error message, we can gather more information by<br>
using autograd’s anomaly detection.</p>
<p>Later in the example script we rerun the forward / backward pass wrapped in<br>
a <code>with torch.autograd.detect_anomaly():</code> block.  Here is the result:</p>
<pre><code class="lang-plaintext">run again with anomaly detection ...
&lt;string&gt;:46: UserWarning: Anomaly Detection has been enabled. This mode will increase the runtime and should only be enabled for debugging.
&lt;path_to_pytorch_install&gt;\torch\autograd\__init__.py:197: UserWarning: Error detected in ReluBackward0. Traceback of forward call that caused the error:
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;string&gt;", line 47, in &lt;module&gt;
  File "&lt;path_to_pytorch_install&gt;\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "&lt;string&gt;", line 18, in forward
  File "&lt;path_to_pytorch_install&gt;\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "&lt;path_to_pytorch_install&gt;\torch\nn\modules\activation.py", line 102, in forward
    return F.relu(input, inplace=self.inplace)
  File "&lt;path_to_pytorch_install&gt;\torch\nn\functional.py", line 1457, in relu
    result = torch.relu(input)
  File "&lt;path_to_pytorch_install&gt;\torch\fx\traceback.py", line 57, in format_stack
    return traceback.format_stack()
 (Triggered internally at C:\cb\pytorch_1000000000000\work\torch\csrc\autograd\python_anomaly_mode.cpp:119.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;string&gt;", line 49, in &lt;module&gt;
  File "&lt;path_to_pytorch_install&gt;\torch\_tensor.py", line 488, in backward
    torch.autograd.backward(
  File "&lt;path_to_pytorch_install&gt;\torch\autograd\__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 10]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>In this, the “traceback of the forward call” shows us where the inplace<br>
modification is being made.  The parts relevant to the example script are:</p>
<pre><code class="lang-plaintext">File "&lt;string&gt;", line 47, in &lt;module&gt;
</code></pre>
<p>which is</p>
<pre data-code-wrap="python"><code class="lang-python">    out = drn (inp)
</code></pre>
<p>where we call the forward pass of the model, and</p>
<pre><code class="lang-plaintext">File "&lt;string&gt;", line 18, in forward
</code></pre>
<p>which is</p>
<pre data-code-wrap="python"><code class="lang-python">        y3 += sc   # skip connection
</code></pre>
<p>and is the inplace add that leads to the inplace-modification error.</p>
<p>So anomaly detection’s forward-pass traceback has led us rather directly<br>
to the source of the error.</p>
<p>(In this case the error can be fixed, at the cost of memory, by replacing<br>
<code>y3 += sc</code> with <code>y3 = y3 + sc</code>.)</p>
<p>Last, sometimes, even if you know what tensor is being modified, it can<br>
be difficult to tell where the modification occurs, especially if you are<br>
calling into third-party code or the tensor is being modified through an<br>
alias or view.</p>
<p>Pytorch checks for inplace modifications by updating (at least in this current<br>
pytorch version, 1.13.1) a tensor’s <code>_version</code> property every time a modification<br>
is made.  You can track this <code>_version</code> property to see if and when a tensor is<br>
being modified.  This is also illustrated by the example script:</p>
<pre><code class="lang-plaintext">run forward pass with versions ...
versions:
y1: 0
sc: 0
y2: 0
y3: 1
y:  0
</code></pre>
<p>Here we print out the <code>_version</code> of all of the tensors at the end of the<br>
model’s <code>forward()</code> method and see that only <code>y3</code> is being modified.<br>
We didn’t do it, but we could have printed out <code>y3._version</code> just before<br>
and after <code>y3 += sc</code> and seen that, indeed, this is the line that modifies<br>
<code>y3</code> inplace.</p>
<p>A rather different cause of inplace-modification errors can be the following:</p>
<p><code>optimizer.step()</code> is also an inplace operation in that <code>optimizer</code> updates<br>
the parameters it is training inplace.  Normally these parameters are used<br>
in a new forward pass that creates a <em>new</em> computation graph for which the<br>
updated parameters are appropriate for the subsequent backward pass<br>
(hence no inplace-modification error).</p>
<p>However, if <code>retain_graph = True</code> is used when performing a backward<br>
pass, <code>optimizer.step()</code> is called, and then a second backward pass is<br>
performed, the modified parameters can trigger an inplace-modification<br>
error in the second backward pass.</p>
<p>Note that although <code>retain_graph = True</code> has legitimate use cases, there is<br>
a lot of code floating around on the internet that uses <code>retain_graph = True</code><br>
incorrectly or unnecessarily.</p>
<p>This script uses a toy generative adversarial network to illustrate how<br>
<code>retain_graph = True</code> can cause inplace-modification errors and how to<br>
track them down:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

_ = torch.manual_seed (2023)

nFeatures = 5   # number of "features" input to generator
length = 10     # length of 1d "image"

# not meant to illustrate how a GAN should actually be implemented

class DummyGen (torch.nn.Module):
    def __init__ (self):
        super().__init__()
        nHidden = 20   # number of hidden features
        self.lin1 = torch.nn.Linear (nFeatures, nHidden)
        self.relu = torch.nn.ReLU()
        self.lin2 = torch.nn.Linear (nHidden, length)
    def forward (self, x):
        y1 = self.lin1 (x)
        y2 = self.relu (y1)
        y3 = self.lin2 (y2)
        return y3

class DummyDisc (torch.nn.Module):
    def __init__ (self):
        super().__init__()
        nHidden = 20   # number of hidden features
        self.lin1 = torch.nn.Linear (length, nHidden)
        self.relu = torch.nn.ReLU()
        self.lin2 = torch.nn.Linear (nHidden, 1)
    def forward (self, x):
        y1 = self.lin1 (x)
        # y1 = torch.nn.functional.linear (x, self.lin1.weight.clone(), self.lin1.bias)
        y2 = self.relu (y1)
        y3 = self.lin2 (y2)
        # fix inplace error by using:
        # y3 = torch.nn.functional.linear (y2, self.lin2.weight.clone(), self.lin2.bias)
        return y3

gen = DummyGen()
dsc = DummyDisc()
g_opt = torch.optim.SGD (gen.parameters(), lr = 0.01)
d_opt = torch.optim.SGD (dsc.parameters(), lr = 0.01)

loss_fn = torch.nn.BCEWithLogitsLoss()

fake = gen (torch.randn (nFeatures))   # fake "image" from random noise
real = torch.randn (length)            # dummy data for "real" image

fake_pred = dsc (fake)
real_pred = dsc (real)

print ('\nrun discriminator backward pass and optimizer step ...')
d_loss = loss_fn (fake_pred, torch.tensor ([0.0])) + loss_fn (real_pred, torch.tensor ([1.0]))
d_opt.zero_grad()
d_loss.backward (retain_graph = True)
d_opt.step()

print ('\nrun generator backward pass and optimizer step ...')
try:
    g_loss = loss_fn (fake_pred, torch.tensor ([0.0])) + loss_fn (real_pred, torch.tensor ([1.0]))
    g_opt.zero_grad()
    g_loss.backward()
    g_opt.step()
except Exception as e:
    print ('%s: %s' % (type (e).__name__, e))

print ('\nrun again with anomaly detection ...')
with torch.autograd.detect_anomaly():
    fake = gen (torch.randn (nFeatures))   # fake "image" from random noise
    real = torch.randn (length)            # dummy data for "real" image
    
    fake_pred = dsc (fake)
    real_pred = dsc (real)

    print ('\nrun discriminator backward pass and optimizer step ...')
    d_loss = loss_fn (fake_pred, torch.tensor ([0.0])) + loss_fn (real_pred, torch.tensor ([1.0]))
    d_opt.zero_grad()
    d_loss.backward (retain_graph = True)
    d_opt.step()
    
    print ('\nrun generator backward pass and optimizer step ...')
    g_loss = loss_fn (fake_pred, torch.tensor ([0.0])) + loss_fn (real_pred, torch.tensor ([1.0]))
    g_opt.zero_grad()
    g_loss.backward()
    g_opt.step()
</code></pre>
<p>Here’s the output from the first part of the script;</p>
<pre><code class="lang-plaintext">1.13.1

run discriminator backward pass and optimizer step ...

run generator backward pass and optimizer step ...
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [20, 1]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>The modified tensor has shape <code>[20, 1]</code>.  The model does not obviously have any<br>
tensors of this shape.  However <code>DummyDisc.lin2.weight</code> has shape <code>[1, 20]</code><br>
(and is indeed the tensor that is causing the inplace-modification error).  I assume<br>
that the source of this shape comes from the backward pass using a transposed<br>
view of the <code>weight</code> matrix.</p>
<p>Anomaly detection confirms this:</p>
<pre><code class="lang-plaintext">run again with anomaly detection ...
&lt;string&gt;:69: UserWarning: Anomaly Detection has been enabled. This mode will increase the runtime and should only be enabled for debugging.

run discriminator backward pass and optimizer step ...

run generator backward pass and optimizer step ...
&lt;path_to_pytorch_install&gt;\torch\autograd\__init__.py:197: UserWarning: Error detected in MmBackward0. Traceback of forward call that caused the error:
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;string&gt;", line 74, in &lt;module&gt;
  File "&lt;path_to_pytorch_install&gt;\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "&lt;string&gt;", line 35, in forward
  File "&lt;path_to_pytorch_install&gt;\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "&lt;path_to_pytorch_install&gt;\torch\nn\modules\linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
  File "&lt;path_to_pytorch_install&gt;\torch\fx\traceback.py", line 57, in format_stack
    return traceback.format_stack()
 (Triggered internally at C:\cb\pytorch_1000000000000\work\torch\csrc\autograd\python_anomaly_mode.cpp:119.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;string&gt;", line 85, in &lt;module&gt;
  File "&lt;path_to_pytorch_install&gt;\torch\_tensor.py", line 488, in backward
    torch.autograd.backward(
  File "&lt;path_to_pytorch_install&gt;\torch\autograd\__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [20, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>The relevant lines are:</p>
<pre><code class="lang-plaintext">  File "&lt;string&gt;", line 74, in &lt;module&gt;
  File "&lt;string&gt;", line 35, in forward
  File "&lt;path_to_pytorch_install&gt;\torch\nn\modules\linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
</code></pre>
<p>which are:</p>
<pre><code class="lang-auto">    real_pred = dsc (real)
        y3 = self.lin2 (y2)
</code></pre>
<p>and the <code>forward()</code> of pytorch’s <code>Linear</code>.</p>
<p>This again shows how autograd’s backward-pass <code>RuntimeError</code>, especially<br>
together with anomaly detection, can help pin down the specific causes of<br>
inplace-modification errors.</p>
<p>In this particular example <code>DummyDisc.lin1</code> will also cause an inplace-modification<br>
error – it just doesn’t show up in the output because autograd raises an error and<br>
stops when the <code>DummyDisc.lin2</code> error occurs.</p>
<p>There are a number of ways to fix these inplace-modification errors. If we<br>
choose not to rework the logic of how the forward / backward passes and<br>
optimization steps work with one another, probably the most direct fix is to<br>
clone the <code>weight</code>s of <code>lin1</code> and <code>lin2</code> and call the functional form of <code>Linear</code>,<br>
that is:</p>
<pre data-code-wrap="python"><code class="lang-python">        y1 = torch.nn.functional.linear (x, self.lin1.weight.clone(), self.lin1.bias)
        y3 = torch.nn.functional.linear (y2, self.lin2.weight.clone(), self.lin2.bias)
</code></pre>
<p>rather than:</p>
<pre data-code-wrap="python"><code class="lang-python">        y1 = self.lin1 (x)
        y3 = self.lin2 (y2)
</code></pre>
<p>*)  See the following post for some explanation of when inplace modifications<br>
do and don’t lead to inplace-modification errors:</p>
<aside class="quote quote-modified" data-post="4" data-topic="170133">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/gradient-computation-issue-due-to-inplace-operation-unsure-how-to-debug-for-custom-model/170133/4">Gradient computation issue due to inplace operation, unsure how to debug for custom model</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Sahil! 

An inplace modification of a tensor that is in the computation graph does 
not necessarily cause an error – it depends on whether the that tensor’s 
value is actually used in the backward pass.  For some explanation, please 
see this post: 



From your first post: 

inter is also in the computation graph and is also being modified inplace 
(assignment to indexed elements).  Because cloning inter fixes your inplace 
error, modifying inter via indexing was likely the cause of your err…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

Hadi_Ali: <p>In the name of Allah</p>
<p>Hi, this explanation is very good and solved my problem using “.clone()”.<br>
Thank you</p>

seyedali_hamedi: <p>Hey there, i’m facing a similar problem here, i’m training an A2C agent for a scheduling problem(assigning tasks to processing elements) but the agent runs multiple episodes at the same time and update itself after each episode is finished by calculating the returns.</p>
<p>but the problem is the episodes my overlap; example:<br>
episode 1 starts with initial weights (w0)<br>
episode 2 starts with initial weights (w0)<br>
episode 2 finishes sooner than episode1 and the weights get updated(w1)<br>
then at last episode 1 finishes and when i want to update the parameters they are changed;</p>
<p>i wanted to know if anyone has encountered  anything similar or has any advice on the matter, or maybe i’m just mistaking and the problem is somewhere else.</p>
<p>here is the code snippit (agent.py):</p>
<pre><code class="lang-auto">    def run(self):
        queue = Preprocessing().get_agent_queue()

        print(f"Agent  queue: {queue}")
        for job_ID in queue.keys():
            task_queue = queue[job_ID]
            self.schedule(task_queue, job_ID)

    def schedule(self, task_queue, job_id):
        if len(task_queue) == 0:
            return

        if job_id not in self.main_log_probs:
            self.main_log_probs[job_id] = []
            self.sub_log_probs[job_id] = []
            self.rewards[job_id] = []
            self.time[job_id] = []
            self.energy[job_id] = []
            self.fail[job_id] = []
            self.selected_devices[job_id] = []

        job_state, pe_state = State().get()

        current_task_id = task_queue.pop(0)
        Preprocessing().remove_from_queue(current_task_id)

        current_task = Database().get_task(current_task_id)
        # input_state = get_input(current_task, pe_state)
        input_state = get_input(current_task, {})
        input_state = torch.tensor(input_state, dtype=torch.float32)

        # option_logits = self.device.agent(input_state)
        option_logits = self.device(input_state)
        # current_devices = self.device.agent.get_devices(input_state)
        option_dist = torch.distributions.Categorical(
            F.softmax(option_logits, dim=-1))
        option = option_dist.sample()

        selected_device = self.devices[option.item()]
        selected_device_index = option.item() 
reward, fail_flag, energy, time = State().apply_action(selected_device_index,
                                                               selected_core_index, dvfs[0], dvfs[1], current_task_id)
        self.main_log_probs[job_id].append(option_dist.log_prob(option))

        self.rewards[job_id].append(reward)
        self.time[job_id].append(time)
        self.energy[job_id].append(energy)
        self.fail[job_id].append(fail_flag)
        self.selected_devices[job_id].append(selected_device_index)
        job_state = State().get_job(job_id)
        if job_state and len(job_state["remainingTasks"]) == 1:
            total_loss = self.update(self.main_log_probs[job_id], self.sub_log_probs[job_id],
                                     self.rewards[job_id], self.selected_devices[job_id])

            Monitor().add_agent_log(
                {
                    'loss': total_loss.item(),
                    'reward': sum(self.rewards[job_id]) / len(self.rewards[job_id]),
                    'time': sum(self.time[job_id]) / len(self.time[job_id]),
                    'energy': sum(self.energy[job_id]) / len(self.energy[job_id]),
                    'fail': sum(self.fail[job_id]) / len(self.fail[job_id]),
                }
            )

            del self.main_log_probs[job_id]
            del self.sub_log_probs[job_id]
            del self.rewards[job_id]
            del self.time[job_id]
            del self.energy[job_id]
            del self.fail[job_id]
            del self.selected_devices[job_id]
</code></pre>
<p>and here is my update function which used the collected log probs and rewards and …(i haven’t implemented the critic yet)</p>
<pre data-code-wrap="gamma"><code class="lang-gamma">
        returns = []
        G = 0
        for reward in reversed(rewards):
            G = reward + gamma * G
            returns.insert(0, G)

        returns = torch.tensor(returns, dtype=torch.float32)

        main_log_probs = torch.stack(main_log_probs)

        main_loss = -torch.sum(main_log_probs.mul(returns))

        self.device_optimizer.zero_grad()
        main_loss.backward()
        self.device_optimizer.step()
</code></pre>

bano4ka: <p>I’m facing a similar problem. It was strange that trace led to a line with no inplace operations, but it turned out to be quite simple. I used rnn layer and passed it its past hidden state. I detached the tensor as needed, but I did it this way:</p>
<pre><code class="lang-auto">output, hid = encoder(input, hid)
hid.detach().to(dev)
</code></pre>
<p>And I should have done it like this:<br>
<code>hid = hid.detach().to(dev)</code></p>

Huidong_Jin: <p>When using retain_graph and you are optimizing with two different optimizers but they are based on intersected model parameters (typically in gan traininng), you can’t do sth like this:</p>
<pre><code class="lang-auto">optimizer_a.zero_grad()
a_loss.backward(retain_graph=True)
optimizer_a.step()
optimizer_b.zero_grad()
b_loss.backward()
optimizer_b.step()
</code></pre>
<p>because line 3 implicitly replace the original parameters after steping on the gradient. Simply step optimizer_a at the end instead to resolve. Note that if you don’t want, say model_a to learn from loss_b, then you’d need to lock the gradient of model_a before backwarding loss_b.<br>
Just in case someone else’s still encountering the same problem.</p>

basavyr: <p>Sir, I just wanted to give you a personal appreciation for such a detailed explanation on how PyTorch handles in-place operations, especially in the context of backpropagation.</p>
<p>You have helped me with an important issue! Cheers!</p>