Gradient computation sees a variable modified but it doesn't seem o

username: iftg
cooked: <p>I get the following error:<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64, 1, 6]], which is output 0 of TanhBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
<p>My forward functions is:</p>
<pre><code class="lang-auto">def forward(self, x: torch.Tensor):
        # x: [Batch_size, seq_len, input_size]
        batch_size = x.size(0)
        
        # Flatten the input across the sequence and input size
        x = x.reshape(batch_size, -1) # x: [Batch_size, seq_len * input_size]
        
        # Apply the linear layer
        x = self.Linear(x) # x: [Batch_size, pred_len * predict_size]
        
        # Reshape the output to [Batch_size, pred_len, predict_size]
        x = x.reshape(batch_size, self.pred_len, self.predict_size)

        # Apply activation function if specified
        x = self.activation(x)
        
        return x # [Batch_size, pred_len, predict_size]

</code></pre>
<p>self.Linear is defined as</p>
<pre><code class="lang-auto">self.Linear = nn.Linear(self.seq_len * self.input_size, self.pred_len * self.predict_size)
</code></pre>
<p>Using view() instead of reshape() doesn’t help.<br>
self.activation can be any: relu, tanh,… If I remove  x = self.activation(x) the code works fine.<br>
Could you please advise what’s wrong here?</p>

username: KFrank
cooked: <p>Hi iftg!</p>
<aside class="quote no-group quote-modified" data-username="iftg" data-post="1" data-topic="207841" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/ecccb3/48.png" class="avatar"> iftg:</div>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64, 1, 6]], which is output 0 of TanhBackward0, is at version 1; expected version 0 instead.<br>
…</p>
<p>My forward functions is:</p>
<pre><code class="lang-auto">def forward(self, x: torch.Tensor):
        ...
        return x # [Batch_size, pred_len, predict_size]
</code></pre>
</blockquote>
</aside>
<p>Most likely, the output of <code>forward()</code> is being modified inplace after<br>
<code>forward()</code> has run.</p>
<aside class="quote no-group">
<blockquote>
<p>If I remove  x = self.activation (x) the code works fine.</p>
</blockquote>
</aside>
<p><code>self.activation(x)</code> isn’t causing the inplace modification error, per se.<br>
Rather, the presence of <code>self.activation (x)</code> in the computation graph<br>
is causing an inplace modification – that exists with or without the call<br>
to <code>self.activation (x)</code> –  to matter.</p>
<p>There is a reasonable chance that adding a <code>.clone()</code>, specifically<br>
<code>x = self.activation (x).clone()</code>, will fix your problem.</p>
<p>Or ask pytorch to <a href="https://pytorch.org/docs/2.4/autograd.html#torch.autograd.graph.allow_mutation_on_saved_tensors" rel="noopener nofollow ugc">sweep this inplace modification error under the rug</a><br>
for you.</p>
<p>Sometimes these errors are a symptom of something incorrect or<br>
sub-optimal in what you are doing.  If you want to track down the root<br>
cause, take a look at the debugging techniques in the following post:</p>
<aside class="quote quote-modified" data-post="7" data-topic="171826">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-torch-floattensor-64-1-which-is-output-0-of-asstridedbackward0-is-at-version-3-expected-version-2-instead-hint-the-backtrace-further-a/171826/7">"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Fahmyadan and Sangyoon! 

Here are some suggestions about how to track down (and maybe fix) 
inplace-modification errors.  Note that an inplace modification in the forward 
pass is not necessarily* an error – it depends on whether and how the tensor 
that was modified is used in the backward pass.  Note that inplace operations 
can be useful for saving memory – if you replace an innocent inplace operation 
with an out-of-place equivalent, your training will use more memory (and, to a 
minor e…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

username: iftg
cooked: <p>Your suggestion solves the problem. Perhaps, a better way, which also does the trick it to swap two lines:</p>
<pre><code class="lang-auto">        # Apply activation function if specified
        x = self.activation(x)

        # Reshape the output to [Batch_size, pred_len, predict_size]
        x = x.reshape(batch_size, self.pred_len, self.predict_size)   
</code></pre>