'inplace operation' error, with an nn.funciontioal

Ruhollah_Majdoddin: <p>If I have</p>
<blockquote>
<p>loss = F.binary_cross_entropy_with_logits(input=pred_logits[0:1, 0:1], target=torch.tensor([[1.]]), reduction=‘none’)</p>
</blockquote>
<p>I get <code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 1]], which is output 0 of AsStridedBackward0, is at version 1; expected version 0 instead.</code></p>
<p>But if I change it to</p>
<blockquote>
<p>loss = F.binary_cross_entropy_with_logits(input=pred_logits.clone()[0:1, 0:1], target=torch.tensor([[1.]]), reduction=‘none’)</p>
</blockquote>
<p>I get no errors. Is it a pytorch bug? After all, slicing just creates a new Tensor, where is the inplace operation then?</p>

ptrblck: <p>Could you post a minimal, executable code snippet to reproduce the issue, please?</p>

KFrank: <p>Hi Ruhollah!</p>
<aside class="quote no-group" data-username="Ruhollah_Majdoddin" data-post="1" data-topic="169401" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ruhollah_majdoddin/48/53235_2.png" class="avatar"> Ruhollah_Majdoddin:</div>
<blockquote>
<p>After all, slicing just creates a new Tensor,</p>
</blockquote>
</aside>
<p>Slicing doesn’t really create an entirely new tensor.</p>
<p>In a technical sense, slicing creates a new tensor in that a new pytorch<br>
<code>Tensor</code> python object is created, but this <code>Tensor</code> is a wrapper object<br>
that contains the same underlying data as the original <code>Tensor</code>.  We say<br>
that slicing returns a <em>view</em> into the original <code>Tensor</code>.</p>
<p>You can use python’s <code>id()</code> function to see that a new <code>Tensor</code> is created,<br>
but then use <code>.storage().data_ptr()</code> to see that both <code>Tensor</code>s reference<br>
the same underlying data.  Furthermore, modifying the sliced <code>Tensor</code><br>
modifies the shared data so those modifications are reflected in the<br>
original <code>Tensor</code>.</p>
<p>Here is an illustration of these points:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.0'
&gt;&gt;&gt; t = torch.arange (5.)
&gt;&gt;&gt; t
tensor([0., 1., 2., 3., 4.])
&gt;&gt;&gt; u = t[1:2]
&gt;&gt;&gt; u
tensor([1.])
&gt;&gt;&gt; id (t)
1803859016768
&gt;&gt;&gt; id (u)
1803859016448
&gt;&gt;&gt; t.storage().data_ptr()
1803813400832
&gt;&gt;&gt; u.storage().data_ptr()
1803813400832
&gt;&gt;&gt; u[0] = 666.
&gt;&gt;&gt; t
tensor([  0., 666.,   2.,   3.,   4.])
</code></pre>
<aside class="quote no-group">
<blockquote>
<p>where is the inplace operation then?</p>
</blockquote>
</aside>
<p>The inplace operation could be hiding most anywhere.  Try running both your<br>
forward and backward pass inside of a <code>with autograd.detect_anomaly():</code><br>
block and see if the additional information this provides points you to the<br>
problematic inplace operation.</p>
<p>Best.</p>
<p>K. Frank</p>

Ruhollah_Majdoddin: <p><a class="mention" href="/u/kfrank">@KFrank</a> Thank you very much for the explanation. Now I understand it better.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> rechecking my code, I figured out a later inplace operation:</p>
<blockquote>
<p>pred_logits[0:1, 0:1] = torch.Tensor[[1]]</p>
</blockquote>
<p>commenting it out, eliminates the error. So the issue is resolved. Thank you very much!</p>