How to debug "gradient computation has been modified by an inplace operation" errors?

krishnab: <p>Hello folks. So there are a lot of posts on the forum about debugging this particular error. The error seems to occur when a user make some inplace change to a tensor, and this apparently messes up the gradient computation for that tensor. That all makes sense.</p>
<p>My question was how should users debug these types of errors. So I have a model and training loop. The training loop will generate this error when executing the <code>loss.backward()</code> step, which is very far from the actual source of the error. Also, the traceback of the error masks the actual location of the error, and just shows the error at <code>loss.backward()</code>.</p>
<p>Now the error message indicates that I should just the <code>torch.autograd.detect_anomaly</code> function or context manager to debug the error. But I have not found a realistic example of using this tool. The example is a toy example of writing a function <code>MyFunc()</code> and then checking its gradients. So it is not a realistic example of how to debug these errors.</p>
<p>Can I use something like <code>pdb</code> to debug these types of errors? In my case I created a Sequence-to-Sequence model with Attention. Now the original sequence-to-sequence code worked and it seems to have a lot of inplace operations. However, when I added a global Attention mechanism to the code, then it started to fail. So it is not clear how adding attention suddenly started causing this error.</p>
<p>If it is helpful, here is the full error message. It gives me a sense of which tensor is causing the issue, but not exactly where.</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace 
operation: [torch.cuda.FloatTensor [64, 1, 7]], which is output 0 of UnsqueezeBackward0, is at version 
729; expected version 725 instead. Hint: enable anomaly detection to find the operation that failed to 
compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>Any suggestions would be helpful</p>

googlebot: <ol>
<li>Identify the tensor. set_detect_anomaly will show the line where it was used, “output 0 of UnsqueezeBackward0” refers to earlier creation, that may be not as useful as shape.</li>
<li>
<strong>Manually</strong> search for further mutating uses of that tensor. Its _version attribute will change somewhere.</li>
<li>Yes, this is annoying and cumbersome.</li>
</ol>

krishnab: <p><a class="mention" href="/u/googlebot">@googlebot</a> haha, thanks for the tips. This is helpful, since I was not sure if that UnsqueezeBackward0 was something that I could trace or not. Seems like the answer is “not”, based on your answer.</p>
<p>I was able to trace through the model and put <code>.clone()</code> on some assignments, which seemed to make the code work now–albeit slower. I probably have more clones than I need, so I can now go through and remove them until the model breaks again. Haha, yeah definitely a cumbersome thing, but not surprising.</p>
<p>Haha, it is funny because from a numerical computing standpoint, we are taught to prefer in-place operations because they are non-allocating and accelerate the code. So it is funny to ignore that instinct.</p>
<p>The odd thing was that all of the examples I saw of this error occurred when a user assigned something to itself, for examples <code>u[i] = 2*u[i-1]</code> or something like that. In my case I did not have anything like that, so the real issue is probably buried somewhere in the lowered code of one of the function calls or such. But at least I know what to do for these kinds of errors now. Thanks again for helping to decipher the message.</p>

ume_habiba: <p><a class="mention" href="/u/krishnab">@krishnab</a> Did you find the solution ? I’m facing the same problem from couple of days, the training loop is showing the error at <code>loss.backward</code> . I trace backed all the tensors which can cause this error vain. Even I didn’t  find the variable which has the dimensions   [96, 2048] with relu activation function.</p>
<p><code>one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [96, 2048]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</code></p>

KFrank: <p>Hi Ume!</p>
<aside class="quote no-group quote-modified" data-username="ume_habiba" data-post="4" data-topic="107362" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ume_habiba/48/43774_2.png" class="avatar"> ume_habiba:</div>
<blockquote>
<p><code>one of the variables needed for gradient computation has been modified by an inplace operation:!</code></p>
</blockquote>
</aside>
<p>In addition to Alex’s above post – in particular his suggestion to use<br>
<code>set_detect_anomaly()</code> – here is a post that illustrates some ways<br>
to debug these kinds of errors:</p>
<aside class="quote quote-modified" data-post="7" data-topic="171826">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-torch-floattensor-64-1-which-is-output-0-of-asstridedbackward0-is-at-version-3-expected-version-2-instead-hint-the-backtrace-further-a/171826/7">"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Fahmyadan and Sangyoon! 

Here are some suggestions about how to track down (and maybe fix) 
inplace-modification errors.  Note that an inplace modification in the forward 
pass is not necessarily* an error – it depends on whether and how the tensor 
that was modified is used in the backward pass.  Note that inplace operations 
can be useful for saving memory – if you replace an innocent inplace operation 
with an out-of-place equivalent, your training will use more memory (and, to a 
minor e…
  </blockquote>
</aside>

<p>Good luck!</p>
<p>K. Frank</p>