RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation [torch.cuda.FloatTensor [64, 192, 16, 16]], which is output 0 of AsStridedBackward0, is at version 3; expected version 0 instead

username: Oussama_Bouldjedri
cooked: <p>Hi I am runing a vision transformer variation and I get this error:</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64, 192, 16, 16]], which is output 0 of AsStridedBackward0, is at version 3; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
<p>Does anyone have an idea ?thanks</p>

username: KFrank
cooked: <p>Hi Oussama!</p>
<aside class="quote no-group" data-username="Oussama_Bouldjedri" data-post="1" data-topic="205871" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oussama_bouldjedri/48/64172_2.png" class="avatar"> Oussama_Bouldjedri:</div>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64, 192, 16, 16]], which is output 0 of AsStridedBackward0, is at version 3; expected version 0 instead.</p>
</blockquote>
</aside>
<p>Depending on your use case, you might be able to “automatically” fix your<br>
issue by using pytorch’s <a href="https://pytorch.org/docs/2.3/autograd.html#torch.autograd.graph.allow_mutation_on_saved_tensors" rel="noopener nofollow ugc">sweep-inplace-modification-errors-under-the-rug</a><br>
context manager.</p>
<p>If you want to track down what is causing your issue, you can find some<br>
techniques for debugging inplace-modification errors in this post:</p>
<aside class="quote quote-modified" data-post="7" data-topic="171826">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-torch-floattensor-64-1-which-is-output-0-of-asstridedbackward0-is-at-version-3-expected-version-2-instead-hint-the-backtrace-further-a/171826/7">"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Fahmyadan and Sangyoon! 

Here are some suggestions about how to track down (and maybe fix) 
inplace-modification errors.  Note that an inplace modification in the forward 
pass is not necessarily* an error – it depends on whether and how the tensor 
that was modified is used in the backward pass.  Note that inplace operations 
can be useful for saving memory – if you replace an innocent inplace operation 
with an out-of-place equivalent, your training will use more memory (and, to a 
minor e…
  </blockquote>
</aside>

<p>Just speculation, but your <code>FloatTensor [64, 192, 16, 16]</code> could<br>
be the <code>.weight</code> of a <code>Conv2d (192. 64, 16)</code> or maybe a batch of 64<br>
192-channel 16 x 16 image patches.</p>
<p>Good luck!</p>
<p>K. Frank</p>