Moving tensors around without violating the gradient computation restrictions

username: Zhenya_Brichkova
cooked: <p>Hello,</p>
<p>I am getting the following error:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
</code></pre>
<p>What I am currently doing is the following: I need to calculate the forward passes of my network in advance (that is, all of them before doing any backward pass). I want to store these tensors in memory, keeping their gradient graphs intact, so that when I am doing my training phase, I can rely on them.</p>
<p>A bit of more detail. As I perform the forward passes, I am storing the outputs of the networks in a dictionary which at every key contains a bag / group of outputs from my network:</p>
<pre><code class="lang-auto">d[bag_i] = torch.cat((d[bag_i], outputs[i]))
</code></pre>
<p>After I am done with this, I do the following reshaping:</p>
<pre><code class="lang-auto">for i in d:
    d[i] = d[i].view(-1, 100)
</code></pre>
<p>For training, I am sampling elements from these bags, and I want to use the tensors here to compute the loss for each batch, without running the forward passes while I am doing it.</p>
<p>It looks like I might be performing an inplace operation somewhere, so I am wondering if this way of concatenating the tensors is legal to keep the graph properly.</p>

username: Caruso
cooked: <p>It is totally possible to store you forward passes and calculate the gradients later, as long as you keep the graph of the outputs (so no <code>.detach()</code>).</p>
<p>I think the error with the inplace operation has another origin. Do you have any ReLUs with inplace=True? Maybe turn them off.</p>

username: Zhenya_Brichkova
cooked: <p>Thank you for the answer. But no, I don’t use neither detach, nor ReLU with the param inplace=True.</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>I think the problem is that you are updating <code>d[bag_i]</code> before computing backpropagation so when autograd tries to do it, the values already are modified.<br>
Note that <code>a = a+b</code> is considered <code>inplace</code> operation.</p>
<p>One possible solution is to clone <code>d[bag_i]</code>s using <code>.clone()</code> function and update cloned version. Other approach is defining new tensor for concatenation.</p>
<p>Please see these posts. <a href="https://discuss.pytorch.org/t/why-this-code-will-throw-out-an-inplace-error/51574">[1]</a>, <a href="https://discuss.pytorch.org/t/setting-part-of-the-training-value-into-zero/52044/2">[2]</a></p>
<p>Bests</p>

username: albanD
cooked: <p>What is <code>d</code> here? A python dictionary?<br>
If it is, then this should be fine and is not considered an inplace operation.</p>
<p>A good way to track this is to enable anomaly mode: <code>torch.autograd.set_detect_anomaly(True)</code> at the beginning of your code to get a more helpful stack trace.<br>
Can you report here what it points to and the code surrounding the faulty bit?</p>

username: Zhenya_Brichkova
cooked: <p>Hi,<br>
I have enabled the anomaly mode and this is what I get in the stack trace:</p>
<pre><code class="lang-auto">File "/Users/evgeniya/PycharmProjects/svp/Main.py", line 34, in &lt;module&gt;
    do_EM()
  File "/Users/evgeniya/PycharmProjects/svp/Main.py", line 31, in do_EM
    EM(loader, get_device()).em(epochs=550)
  File "/Users/evgeniya/PycharmProjects/svp/EM.py", line 163, in em
    b, d = self.e(model)
  File "/Users/evgeniya/PycharmProjects/svp/EM.py", line 39, in e
    idx, d, n, cr = self.eval_images(model)
  File "/Users/evgeniya/PycharmProjects/svp/EM.py", line 152, in eval_images
    d = torch.cat((d, model(data.view(-1, 1, 64, 64).float()).view(data.size()[0], 100)))
  File "/Users/evgeniya/PycharmProjects/svp/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/evgeniya/PycharmProjects/svp/Net.py", line 116, in forward
    x = self.layer12(x)
  File "/Users/evgeniya/PycharmProjects/svp/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/evgeniya/PycharmProjects/svp/venv/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/Users/evgeniya/PycharmProjects/svp/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/evgeniya/PycharmProjects/svp/venv/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 106, in forward
    exponential_average_factor, self.eps)
  File "/Users/evgeniya/PycharmProjects/svp/venv/lib/python3.6/site-packages/torch/nn/functional.py", line 1923, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
</code></pre>
<p>Should I assume, that the torch.cat operation is responsible for the error or it could be caused by something before?</p>
<pre><code class="lang-auto">    d = torch.cat((d, model(data.view(-1, 1, 64, 64).float()).view(data.size()[0], 100)))
</code></pre>
<p>I have checked the posts regarding this problem and I have found out, that in most cases torch.cat was not the problem.</p>

username: albanD
cooked: <p>The problem on this line is not the <code>torch.cat</code> but the <code>model(data.view(-1, 1, 64, 64).float())</code> as you can see that the rest of the stack trace points inside the model forward and to the batchnorm.</p>
<p>So you want to double check that input and output of the batchnorm and make sure they are not updated inplace (by a ReLU(inplace=True) for example).</p>

username: Zhenya_Brichkova
cooked: <p>The layers of my network are of the form:</p>
<pre><code class="lang-auto">        self.layer11 = nn.Sequential(
            nn.Conv2d(in_channels=1024, out_channels=1024, kernel_size=1),
            nn.BatchNorm2d(1024, track_running_stats=False),
            nn.ReLU()
        )
</code></pre>
<p>with some max pooling layers.</p>
<p>My forward function looks like this:</p>
<pre><code class="lang-auto">    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        ...
        x = self.layern(x)
        return x
</code></pre>

username: albanD
cooked: <p>That seems to work fine for me:</p>
<pre><code class="lang-python">import torch
from torch import nn

model = nn.Sequential(
    nn.Conv2d(in_channels=1024, out_channels=1024, kernel_size=1),
    nn.BatchNorm2d(1024, track_running_stats=False),
    nn.ReLU()
)

inp = torch.rand(1, 1024, 5, 5)

model(inp).sum().backward()
</code></pre>

username: Zhenya_Brichkova
cooked: <p>This is my learning part:</p>
<pre><code class="lang-auto">for bag in d_bags:
                    loss = - self.loss(d_bags[bag], b_bags[bag])
                    loss.backward(retain_graph=True)
                    opt.step()
</code></pre>
<p>It successfully performs the first iteration and then fails on loss.backward. Without opt.step like this:</p>
<pre><code class="lang-auto">for bag in d_bags:
                    loss = - self.loss(d_bags[bag], b_bags[bag])
                    loss.backward(retain_graph=True)
</code></pre>
<p>It does not fail.</p>

username: albanD
cooked: <p>Ho interesting.<br>
Then the issue is with the fact that the <code>opt.step()</code> is modifying inplace the weights of the batchnorm (for the affine part).<br>
And in the second iteration, the original weights are still needed to compute the backward hence the error you see.</p>
<p>You can move the step() outside of the for loop to accumulate all the gradients before doing the backward.<br>
Or even better, accumulate the loss inside the for-loop and do a single backward call outside on the accumulated loss.</p>

username: Zhenya_Brichkova
cooked: <p>I did the following and now it got me totally lost:<br>
I moved the backward, step outside the loop as you said</p>
<pre><code class="lang-auto">loss = 0
for bag in d_bags:
      loss = loss - self.loss(d_bags[bag], b_bags[bag])
loss.backward()
opt.step()
</code></pre>
<p>It was giving the same error.<br>
Then I tried to remove the batchnorm layer totally. With the anomaly flag off it is giving still the same error of inplaced operations:</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time. (apply at ../torch/csrc/autograd/generated/Functions.cpp:3969)

</code></pre>
<p>Then I turned the anomaly flag on torch.autograd.set_detect_anomaly(True) and it gave a different error:</p>
<pre><code class="lang-auto">RuntimeError: Function 'MulBackward0' returned nan values in its 1th output.
</code></pre>

username: albanD
cooked: <blockquote>
<p>With the anomaly flag off it is giving still the same error of inplaced operations:</p>
</blockquote>
<p>The error you linked is not the same actually !<br>
Try adding a retain_graph=True to see if it is better.</p>
<blockquote>
<p>Then I turned the anomaly flag on torch.autograd.set_detect_anomaly(True) and it gave a different error:</p>
</blockquote>
<p>This can happen for few reasons but most likely here because you multiply something infinite with 0. Do you get infinite values in the forward for the mul that is pointed to by anomaly mode?</p>