In-place operation error - PyTorch1.6.0

username: learner47
cooked: <p>This is my class definition:</p>
<pre><code class="lang-auto">class Tracker(nn.Module):
    def __init__(self):
        super(Tracker, self).__init__()
        self.bigru = nn.GRU(input_size=2, hidden_size=100, batch_first=True, bidirectional=True)
        self.fc1 = nn.Linear(200, 32)
        self.fc2 = nn.Linear(32, 2)
    def forward(self, inputs):
        x, states = self.bigru(inputs)
        x = self.fc1(x[:, -1, :])
        x = self.fc2(x)
        return x
</code></pre>
<p>While training I use <code>loss.backward(retain_graph=True)</code>. But I get the error</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation.</p>
</blockquote>
<p>I went through couple of discussions on the topic and realized it was due to the in-place operation of a tensor.So I switched to the following code.</p>
<pre><code class="lang-auto">    def forward(self, inputs):
        x, states = self.bigru(inputs)
        k = x.clone()
        y = self.fc1(k[:, -1, :])
        z = self.fc2(y.clone())
        return z
</code></pre>
<p>The error still persists. Can anyone tell me where am I going wrong? Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>Can you enable anomaly mode? That will show you which op had its input changed<br>
<a href="https://pytorch.org/docs/stable/autograd.html#anomaly-detection" class="onebox" target="_blank">https://pytorch.org/docs/stable/autograd.html#anomaly-detection</a></p>

username: learner47
cooked: <p>Hi,<br>
I have done that but forgot to attach the error. Here it is:</p>
<pre><code class="lang-auto">File "venv\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "src/del.py", line 76, in forward
    z = self.fc2(y.clone())
  File "venv\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "venv\lib\site-packages\torch\nn\modules\linear.py", line 91, in forward
    return F.linear(input, self.weight, self.bias)
  File "venv\lib\site-packages\torch\nn\functional.py", line 1674, in linear
    ret = torch.addmm(bias, input, weight.t())
</code></pre>

username: albanD
cooked: <p>So that means that an input to your linear layer was modified inplace.<br>
Since from the stack trace you clone the input to make sure it is not this one, then I guess it is the weights, which are most like modified by your optimizer when you do <code>optimizer.step()</code> ?</p>

username: learner47
cooked: <pre><code class="lang-auto">            tracker_opt.zero_grad()
            tracker_out = tracker_model(tracker_in.float())
            loss = nn.MSELoss()
            loss_tracker = loss(tracker_out, torch.as_tensor(np.concatenate([np.real(temp)[:, None],
                                                                             np.imag(temp)[:, None]], axis=1)).float())
            loss_tracker.backward(retain_graph=True)
            tracker_opt.step()
            tracker_in = tracker_out[:, None, :]
</code></pre>
<p>This is my training loop. Before I do <code>optimizer.step()</code>, I do not make any changes I believe. Do you spot anything wrong?</p>

username: albanD
cooked: <p>What I mean is that <code>optimizer.step()</code> is an inplace operation. So if you try to backward again after doing the optimizer step without re-doing the forward, you will see this error.<br>
Is your code doing something like:</p>
<pre><code class="lang-auto">out = model(inp)
first_loss = bar(out, label)
first_loss.backward(retain_graph=True)
opt.step()

opt.zero_grad()
second_loss = baz(out, label)
second_loss.backward()  # This will fail if model has a Linear
# Because the Linear's weight were modified inplace above and
# this backward cannot be done anymore.

</code></pre>

username: learner47
cooked: <p>Oh I see.<br>
My training method and loop is attached below</p>
<pre><code class="lang-auto">def train(tracker_model, tracker_opt, epoch, _):
    global train_tracker
    if train_tracker:
        global tracker_inputs

        tracker_in = torch.as_tensor(np.concatenate([np.real(tracker_inputs)[:, None, None],
                                                     np.imag(tracker_inputs)[:, None, None]], axis=2))
        for i in range(epoch):
            tracker_model.train()
            tracker_opt.zero_grad()
            tracker_out = tracker_model(tracker_in.float())
            loss = nn.MSELoss()
            temp = generate_ar_sequence(16)
            loss_tracker = loss(tracker_out, torch.as_tensor(np.concatenate([np.real(temp)[:, None],
                                                                             np.imag(temp)[:, None]], axis=1)).float())
            loss_tracker.backward(retain_graph=True)
            tracker_opt.step()
            tracker_in = tracker_out[:, None, :]
        train_tracker = False
        tracker_model.eval()
</code></pre>

username: albanD
cooked: <p>This looks ok I think.<br>
Note that you don’t need <code>retain_graph=True</code> here right? You don’t actually plan to backward on that graph again?</p>

username: learner47
cooked: <p>Ok, maybe I don’t understand <code>retain_graph</code> correctly. Isn’t <code>loss_tracker.backward()</code> required at every iteration?</p>

username: albanD
cooked: <p><code>retain_graph</code> is only needed if you call backward again without calling forward again. If you only call things once, there is no need for it.</p>