[Autograd] In-place operations, clone and slicing?

technick: <p>Hello, I would appreciate if somebody could confirm that my understanding about autograd and inplace operations is correct and my code does not run into issues.</p>
<ol>
<li>
<p>I train a transformer model based on DETR. When I sample tensors from my dataset and do preprocessing, they have:<br>
<code>requires_grad=False</code><br>
I do not change that field before feeding this data to the PyTorch transformer and assume that is fine. I understand that when the samples get processed in the transformer, the results automatically have requires_grad = True as the network weights probably are configured that way.</p>
</li>
<li>
<p>During my pre-processing and also during post-processing I currently use in-place operations, e.g.<br>
<code>tensor[b ,idx, 0] = 2 * ((tensor[b, idx, 0] - x_min) / (x_max - x_min)) - 1</code><br>
I assume this is fine in pre-processing, as the grad flag is still false. During post-processing I want to change my network output before calculating the loss function. Here I figured, that I need to clone my output tensor first before reverting the inplace operation above. I then want to write the changed tensor values into the original tensor. Does this work fine with autograd?</p>
</li>
</ol>
<pre><code class="lang-auto">tensor_temp = tensor.clone()
tensor_temp[b, idx, 0] = (tensor_temp[b, idx, 0] + 1) * 0.5 * (x_max - x_min) + x_min
tensor = tensor_temp
</code></pre>
<p>Or do I need to find some workaround to avoid the slicing and inplace operation by for example creating a zero tensor and filling it with the new values, to then use e.g. torch.where() to merge it with the original tensor.</p>
<ol start="3">
<li>Generally, can I not use slicing when using autograd? E.g.:</li>
</ol>
<pre><code class="lang-auto">tensor[:, 0] = 5
</code></pre>
<p>or<br>
<code>tensor[mask] = 5</code><br>
It seems very unintuitive to me to then always create a zero tensor with e.g. the value 5 and then adding it to the original one.</p>
<p>This leads me to my current understanding of autograd:</p>
<ul>
<li>use clone() when I want to do inplace operations on my tensor with grad history, which I want to keep. I can also assign my cloned tensor to the original one, as it has the same grad history.</li>
<li>use detach().clone() when I want to have a copy of my tensor that uses new memory and has no grad history</li>
<li>inplace operations without clone() can cause issues with autograd and my grad results could be wrong even though there is no grad error</li>
</ul>
<p>Thank you!</p>

KFrank: <p>Hi Nick!</p>
<aside class="quote no-group" data-username="technick" data-post="1" data-topic="171456" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/technick/48/56898_2.png" class="avatar"> technick:</div>
<blockquote>
<p>This leads me to my current understanding of autograd:</p>
<ul>
<li>use clone() when I want to do inplace operations on my tensor with grad history, which I want to keep. I can also assign my cloned tensor to the original one, as it has the same grad history.</li>
<li>use detach().clone() when I want to have a copy of my tensor that uses new memory and has no grad history</li>
<li>inplace operations without clone() can cause issues with autograd and my grad results could be wrong even though there is no grad error</li>
</ul>
</blockquote>
</aside>
<p>Let me make some comments that are relevant to your questions:</p>
<p>Inplace operations <em>can</em> cause “inplace-modification” errors during<br>
backpropagation, but they don’t <em>necessarily</em> cause such errors.</p>
<p>The following post of mine tries to explain what is going on with some<br>
examples:</p>
<aside class="quote quote-modified" data-post="3" data-topic="169583">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/when-inplace-operation-are-allowed-and-when-not/169583/3">When inplace operation are allowed and when not</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Jie (and Srishti)! 

The part that can be confusing is that a given tensor used in the forward 
pass may or may not be used in the backward pass.  It depends on the math 
of the gradient computation (and the details of how the relevant backward() 
function is implemented).  It’s not always obvious. 
Here are some explanatory examples: 
The derivative of exp (x) is exp (x).  So it makes sense that pytorch saves 
the output of exp (x) when it is computed during the forward pass so that it 
can …
  </blockquote>
</aside>

<p>Note, your “grad results” won’t be wrong because if they would have been<br>
wrong, autograd <em>will</em> raise a “grad error.”</p>
<p>Tensors without a “grad history” (that is, with <code>requires_grad = False</code>)<br>
are not immune to inplace-modification errors.  A tensor which is not<br>
having its own gradient computed can still be used in the computation<br>
of some other tensor.  Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.0'
&gt;&gt;&gt; a = torch.tensor ([2])   # a tensor with requires_grad = False
&gt;&gt;&gt; t = torch.ones (1, requires_grad = True)
&gt;&gt;&gt; l = a * t
&gt;&gt;&gt; l.backward()
&gt;&gt;&gt; t.grad
tensor([2.])
&gt;&gt;&gt; t.grad = None
&gt;&gt;&gt; l = a * t
&gt;&gt;&gt; a[0] = 99   # modify inplace
&gt;&gt;&gt; l.backward()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;path_to_pytorch_install&gt;\torch\_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "&lt;path_to_pytorch_install&gt;\torch\autograd\__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.LongTensor [1]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>Yes, if you need to modify a tensor inplace <em>and</em> doing so causes an<br>
inplace-modification error during backpropagation, then cloning the<br>
tensor before modifying it is often an appropriate solution.</p>
<p>Best.</p>
<p>K. Frank</p>

technick: <p>Hello K. Frank,</p>
<p>thank you for your answer and examples, it helped a lot.<br>
I take the following away from this, please correct me if I am wrong:</p>
<ul>
<li>If PyTorch does not throw an error regarding inplace-modifications, there is nothing for me to worry about. Even if I use inplace-operations.</li>
<li>When an inplace-error does happen I can circumvent it by cloning the tensor, applying my in-place operations and then writing the cloned and modified tensor to the variable that used contain my original tensor, without messing anything up regarding the gradient. Just like I do it in my example.</li>
<li>It is generally a good style in PyTorch to prevent the usage of in-place operations for tensors with “requires-grad = True”. Better use more code lines and use e.g. torch.where() to replace values in tensors.</li>
</ul>

KFrank: <p>Hi Nick!</p>
<aside class="quote no-group" data-username="technick" data-post="3" data-topic="171456" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/technick/48/56898_2.png" class="avatar"> technick:</div>
<blockquote>
<ul>
<li>If PyTorch does not throw an error regarding inplace-modifications, there is nothing for me to worry about. Even if I use inplace-operations.</li>
</ul>
</blockquote>
</aside>
<p>Correct.</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>When an inplace-error does happen I can circumvent it by cloning the tensor, applying my in-place operations and then writing the cloned and modified tensor to the variable that used contain my original tensor,</li>
</ul>
</blockquote>
</aside>
<p>Correct.  But to clarify, you have a named “variable” in your python script<br>
that refers (as a python reference) to some tensor that autograd needs<br>
(and autograd also keeps its own reference to this tensor).  You clone<br>
the tensor and your reference (the named “variable”) and autograd’s<br>
reference both still refer to the original tensor.  When you “write the cloned<br>
and modified tensor to the variable” you are telling the python interpreter<br>
to reuse that name and set it to now refer to a different tensor – the cloned<br>
and modified tensor.  (Note, you could have set that “variable” to refer, for<br>
example, to some python list that has nothing to do with pytorch.)</p>
<p>Setting your named “variable” to refer to something else (or, for that matter,<br>
to <code>None</code>) has no effect on the original tensor nor on autograd’s reference to<br>
the original tensor.</p>
<p>It’s up to you whether you want to use a new variable name or reuse the<br>
original variable name to refer to the cloned and modified tensor – it doesn’t<br>
affect the logic of the code.  I would make that choice based on what makes<br>
the code the most readable.</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>It is generally a good style in PyTorch to prevent the usage of in-place operations for tensors with “requires-grad = True”.</li>
</ul>
</blockquote>
</aside>
<p>I wouldn’t necessarily agree.  Creating a new tensor to avoid an inplace<br>
operation costs memory.  Furthermore, if your cloned and modified tensor will<br>
be part of the backward pass, then it too will carry <code>requires_grad = True</code>.<br>
So your rule would have prevented you from modifying it inplace.</p>
<p>My general approach is to go ahead and use inplace modifications (where<br>
appropriate) unless I think it will lead to an inplace-modification error during<br>
the backward pass.  If I overlook such an inplace-modification error, autograd<br>
warns me and I go back and fix it.</p>
<aside class="quote no-group">
<blockquote>
<p>use e.g. torch.where() to replace values in tensors.</p>
</blockquote>
</aside>
<p><code>torch.where()</code> is a useful tool, but be aware that it has a booby-trap in it.<br>
If the “path not taken” in the <code>torch.where()</code> call has <code>nan</code>s in it, those <code>nan</code>s<br>
will pollute the result of <code>torch.where()</code>, even though they don’t <em>logically</em><br>
need to.</p>
<p>Best.</p>
<p>K. Frank</p>