Torch.exp() is modified by an inplace operation

chwan-rice: <p>My torch version is 1.5.1. I found a strange behavior of torch.exp():</p>
<blockquote>
<p>import torch<br>
x = torch.ones(2, 1, requires_grad=True)<br>
x = torch.exp(x)<br>
x[0] = 0<br>
out = x.mean()<br>
out.backward()</p>
</blockquote>
<p>After running this code, I got a running error:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [2, 1]], which is output 0 of ExpBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
</blockquote>
<p>However, if I replace <code>torch.exp()</code> with <code>torch.log()</code> or <code>torch.sin()</code> in the code, it works very well. Can anyone help me to find the solution? Thank you very much.</p>
<p><strong>Edit:</strong> One thing I forgot to mention. If I add <code>x=x+0</code> before <code>x[0]=0</code>, the code works well too.</p>

tom: <p>So I covered some of that along with many “little mysteries” in a <a href="https://twitter.com/ThomasViehmann/status/1282555652112621568" rel="nofollow noopener">course</a> recently (slides 28-35). From the summary slide:</p>
<p>There are two main situations where inplace operations cannot be used:</p>
<ul>
<li>When operating on leaf tensors, the leaf tensor would be moved into the graph, which would be bad.</li>
<li>When the operation before the inplace wants to have its result to compute the<br>
backward. Whether this is the case is not easy to tell from the outside,<br>
unfortunately.</li>
</ul>
<p>You are in the second case. What happens is that exp wants its output to compute the backward (which multiplies by the output) while e.g. log wants its input (because the backward divides by the input). This is the “not always easy to tell from the outside part”.<br>
If you do <code>x = x + 0</code> you are creating a new tensor (which happens to be numerically identical to the old x) and then assign that to x and modify it inplace, but exp finds its output unmodified. This is, btw. the purpose of the <code>ctx.save_for_backward(...)</code>, <code>ctx.saved_tensors</code> exrcise in <code>autograd.Function</code>s.</p>
<p>PDF of the slides (and Jupyter Notebook to run the code) are <a href="https://github.com/t-vi/acdl2020/" rel="nofollow noopener">on github</a>, but apparently not enough interest for video.</p>
<p>Of course, the authoritative in-depth reference is <a class="mention" href="/u/ptrblck">@ptrblck</a>’s  and my imaginary book from which I made the slides for the talk.</p>
<p>Best regards</p>
<p>Thomas</p>

chwan-rice: <p>Hello Thomas. Thanks for your detailed response. It is really helpful to me.</p>

kshitij: <aside class="quote no-group" data-username="tom" data-post="2" data-topic="90216">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>PDF of the slides (and Jupyter Notebook to run the code) are <a href="https://github.com/t-vi/acdl2020/" rel="noopener nofollow ugc">on github </a>, but apparently not enough interest for video.</p>
</blockquote>
</aside>
<p>+1 for the video if possible:)</p>