What caused this in-place modification error?

JimW: <p>I am having this in-place modification error:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [128, 1]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
</blockquote>
<p>So I went back to my code and narrowed down the cause to this piece of codes:</p>
<blockquote>
<p><span class="hashtag-raw">#my_layer</span> = nn.Sequential(torch.nn.Linear(200, 1), torch.nn.Sigmoid())<br>
my_layer = nn.Sequential(torch.nn.Linear(200, 128), torch.nn.Linear(128, 1), torch.nn.Sigmoid())<br>
my_layer.double()<br>
my_layer.train()<br>
my_optim = torch.optim.Adam(<br>
my_layer.parameters(),<br>
lr=0.001,<br>
betas=(0.9, 0.999),<br>
weight_decay=1e-4)</p>
</blockquote>
<p>The error disappeared when I used only 1 single linear layer in my network. But if I invert an additional linear layer, the error came back.</p>
<p>Could someone please provide some guidance what is the root cause of such errors?  Thanks.</p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="JimW" data-post="1" data-topic="207421">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jimw/48/38000_2.png" class="avatar"> JimW:</div>
<blockquote>
<p>my_layer.double()</p>
</blockquote>
</aside>
<p>Perhaps calling <code>.double()</code> counts as an in-place operation? Also run your code with a <code>torch.autograd.set_detect_anomaly</code> context manager, docs here: <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.set_detect_anomaly" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 2.4 documentation</a></p>

JimW: <p>Thank you for your response.<br>
I converted all the doubles to back to float, and removed the my_layer.double().<br>
The error is still there:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [128, 1]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
<p>Fatal Python error: gilstate_tss_set: failed to set current tstate (TSS)</p>
</blockquote>
<p>I added <code>with torch.autograd.set_detect_anomaly(True):</code> as you suggested.</p>

KFrank: <p>Hi Jim!</p>
<aside class="quote no-group quote-modified" data-username="JimW" data-post="1" data-topic="207421" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jimw/48/38000_2.png" class="avatar"> JimW:</div>
<blockquote>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [128, 1]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead.</p>
</blockquote>
<p>So I went back to my code and narrowed down the cause to this piece of codes:</p>
<blockquote>
<p><span class="hashtag-raw">#my_layer</span> = nn.Sequential(torch.nn.Linear(200, 1), torch.nn.Sigmoid())<br>
my_layer = nn.Sequential(torch.nn.Linear(200, 128), torch.nn.Linear(128, 1), torch.nn.Sigmoid())<br>
my_layer.double()<br>
my_layer.train()</p>
</blockquote>
</blockquote>
</aside>
<p>It is most likely the case that the <code>.weight</code> of your <code>Linear (128, 1)</code><br>
is being inappropriately modified inplace.  You are probably doing<br>
something like:</p>
<pre data-code-wrap="python"><code class="lang-python">some_loss.backward (retain_graph = True)
my_optim.step()        # counts as inplace modifications of my_layer.parameters()
...
some_loss.backward()   # triggers error
</code></pre>
<p>You could verify this by printing out <code>my_layer[1].weight._version</code><br>
before and after you call <code>my_optim.step()</code> and see if <code>._version</code><br>
changes from <code>1</code> to <code>2</code>.</p>
<p>One fix could be pytorch’s <a href="https://pytorch.org/docs/2.4/autograd.html#torch.autograd.graph.allow_mutation_on_saved_tensors" rel="noopener nofollow ugc">“allow mutation” context manager.</a>  You could<br>
also call <a href="https://pytorch.org/docs/2.4/generated/torch.nn.functional.linear.html" rel="noopener nofollow ugc">torch.nn.functional.linear()</a> with clones of the <code>Linear (128, 1)</code>'s<br>
<code>.weight</code> and <code>.bias</code> parameters.</p>
<aside class="quote no-group">
<blockquote>
<p>The error disappeared when I used only 1 single linear layer in my network. But if I invert an additional linear layer, the error came back.</p>
</blockquote>
</aside>
<p>With your two-<code>Linear</code> <code>Sequential</code>, pytorch needs the unmodified<br>
<code>.weight</code> of the second <code>Linear</code> to compute gradients with respect to<br>
the first <code>Linear</code>.  When you use just a one-<code>Linear</code> <code>Sequential</code>, you<br>
are still modifying that <code>Linear</code>’s parameters inplace, but those<br>
parameters are not needed in order to compute any gradients, so<br>
the inplace-modification is not an error.</p>
<p>As an aside, the two <code>Linear</code>s in a row in your <code>Sequential</code> (without<br>
any intervening non-linear “activation”) collapse, in effect, into a<br>
single <code>Linear (200, 1)</code>.  Therefore, you would probably want to<br>
add something like a <code>ReLU()</code> or <code>Sigmoid()</code> in between them.</p>
<p>Best.</p>
<p>K. Frank</p>