Inplace Errors with Dropout layers with PyTorch 1.9, but not with PyTorch 1.10

username: Stanley_C
cooked: <p>I’m currently working with the 3detr repo(<a href="https://github.com/facebookresearch/3detr" rel="noopener nofollow ugc">https://github.com/facebookresearch/3detr</a>), and it is only officially working with Pytorch 1.9. When I switched to Pytorch 1.10, I got an error:</p>
<p><code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [256, 1, 256]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True)</code></p>
<p>Turning on anomaly detection, I manage to trace errors to dropout layers, which I changed to work with Pytorch 1.10, like this :<br>
<code>self.dropout = nn.Dropout(dropout, inplace=False) # Inplace Originally True, set to False for Pytorch 1.10 Compatibility</code><br>
Since I posting large sections of the 3dter repo won’t be easy to read(and I would have to post a lot of code), instead, I created a pull request on the 3detr repo with the changes: (<a href="https://github.com/facebookresearch/3detr/pull/18/commits/9c36709d57b7d6e300d839b3394951debcb1acc6" rel="noopener nofollow ugc">Pull Request Here</a>)<br>
However, the dropout layers would returns errors on lines like this: <code>src2 = self.linear2(self.dropout(self.activation(self.linear1(src2))))</code></p>
<p>My first question is what changed between Pytorch 1.9 and Pytorch 1.10 that would cause errors like this to occur in Pytorch 1.10, but not in 1.9?<br>
My second question is however would I fix this errors without setting inplace to False(this is undesirable for me since it increases train time per batch by around 2x!) or rolling back to 1.9, and if this is not possible to do, is there any way to ignore the error Runtime Errors?</p>

username: ptrblck
cooked: <p>I would guess <a href="https://github.com/pytorch/pytorch/issues/63027">this PR</a> changing the <code>nn.ReLU</code> backward pass might have disallowed the following <code>dropout</code> layer to manipulate the outputs inplace, as they are now used during the <code>backward()</code>.</p>
<p>It’s strange to hear you are seeing a 2X slowdown. Could you give more information about the used device and setups, as I think the slowdown might be unrelated to the PR.</p>

username: Stanley_C
cooked: <p>Of course, my hardware setup is a 6 core CPU(8400), and a 1060 GPU with 3 gb of vrams, so a tad bit limited in compute power and vram.</p>
<p>Any ideas on how to solve this issue will still running inplace=True? Switching inplace to False required me to scale down the model layers, since inplace reduced the vram usage by a decent amount for me, so that was nice to have. I could get rid of the dropout errors but that is a bit of a non ideal solution.</p>

username: ptrblck
cooked: <p>The corresponding <a href="https://github.com/pytorch/pytorch/pull/63089">PR</a> was merged on Aug 26th, so you could compare the performance using the nightly binary from the day before and after the merge. If you are still seeing the issue, this could narrow it down to this PR.<br>
It’s still strange that you needed to scale down the model layers (I assume you needed to reduce the memory usage), as the PR should save memory and naively I would assume the inplace dropout would save the same amount of memory (haven’t looked into the code deeply yet).</p>

username: tom
cooked: <p>A short while ago I wrote a little blog about the <a href="https://lernapparat.de/pytorch-inplace/" rel="noopener nofollow ugc">nature of memory savings from inplace</a>. The synopsis is that inplace will only save temporary allocations, so it should not matter. You might be running into the case mentioned in sidenote 2 that you wan to run <code>gc.collect()</code>. Another option could be to use the JIT to take the computation out of Python.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Stanley_C
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="137544">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>The corresponding <a href="https://github.com/pytorch/pytorch/pull/63089" rel="noopener nofollow ugc">PR</a> was merged on Aug 26th, so you could compare the performance using the nightly binary from the day before and after the merge. If you are still seeing the issue, this could narrow it down to this PR.</p>
</blockquote>
</aside>
<p>I see, I’m not quite sure how to install nightly build from a specific day, any guide/instructions on how this could be done?</p>

username: Stanley_C
cooked: <p>Thank you for your advice, I’ll will try running gc.collect and see if that helps with memory, but would JIT make up for the 2x batch times I’m seeing? Personal I didn’t expect such a big difference with inplace versus no inplace.</p>

username: tom
cooked: <p>The JIT might eliminate the need to use gc.collect() to free the memory. It won’t help with the timing. For that, I would suggest to use the profiler to see what exactly is getting slower. It is not clear to me that it is necessarily linked.</p>
<p>Best regards</p>
<p>Thomas</p>

username: redamancy012
cooked: <p>I met this error which troubled me many times, I just set inplace == False,but I don’t konw how it happened, can you figure out it? Thanks</p>

username: tom
cooked: <p>It happens when the operation just before the dropout needs the output to compute the gradient.<br>
Unfortunately, it is not trivial to know which operations need the output. However, you can inspect <code>.grad_fn</code>s for <code>_saved_result</code>:</p>
<p>Sqrt needs the result to compute the backward, so using dropout with inplace on it will cause errors.</p>
<pre><code class="lang-auto">x =(torch.tensor((5., 5.), requires_grad=True).sqrt())
x.grad_fn._saved_result
</code></pre>
<p>Pow saves the base and the exponent for the backward, but not the result, so using dropout with inplace works.</p>
<pre><code class="lang-auto">x =(torch.tensor((5., 5.), requires_grad=True) ** 0.5)
x.grad_fn._saved_result  # error
</code></pre>
<p>And there you also see that it is highly implementation specific.</p>
<p>Best regards</p>
<p>Thomas</p>