Inplace Operation Error using LSTM/GRU

arash_javanmard: <p>Hi,</p>
<p>I have sequence data, which are technically measurements from an IMU, which are taken between each camera image frame. I want to pass them through a many-to-one  LSTM-Module.<br>
The sequence has the dimension [S_out x B x S_in x N], S_out are the number of frames, B is the batch size, S_in is the number of measurements between each image frame and at least N are the actual measurement values. In this particular case we have:</p>
<blockquote>
<p>S_out = 4<br>
B = 8<br>
S_in = 10<br>
N = 6</p>
</blockquote>
<p>the pseudo-code for training is:</p>
<pre><code class="lang-auto">preds = [outer-seq-size]
for s=1 : outer-seq-size:
     preds[s] = model(x[s])
loss = clac_loss(preds, gts)
loss.backward()
....
model.reset_hidden_states()
</code></pre>
<p>My LSTM-Module is:</p>
<pre data-code-wrap="python"><code class="lang-python">class ImufeatRNN0(BaseImuFeatNet):
    def __init__(self, cfg):
        self.rnn = nn.LSTM(input_size=6, hidden_size=128,
                           num_layers=2, bidirectional=False, dropout=0.)

        self.num_dir = 1
        self.reset_hidden_states()

    def reset_hidden_states(self):
        self.h = None
        self.c = None

    def forward(self, x):
        x_padded = nn.utils.rnn.pad_sequence(x)
        s, b, n = x_padded.shape  # seq

        if self.h is None:
            out, (self.h, self.c) = self.rnn(x_padded, None)
        else:
            out, (self.h, self.c) = self.rnn(x_padded, (self.h, self.c))

        out = out.view(s, b, self.num_dir, 128)
        out = out[-1, :, 0].contiguous().squeeze() # many-to-one
        return out
</code></pre>
<p>The fig. below should make the situation more clear.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e90e5a0a8715cb6071706e2af0c505217def320f.png" data-download-href="https://discuss.pytorch.org/uploads/default/e90e5a0a8715cb6071706e2af0c505217def320f" title="lstm_imu"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e90e5a0a8715cb6071706e2af0c505217def320f_2_690x187.png" alt="lstm_imu" data-base62-sha1="xfHZWj7tSgZ25zkxDH7EJAFZRZd" width="690" height="187" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e90e5a0a8715cb6071706e2af0c505217def320f_2_690x187.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e90e5a0a8715cb6071706e2af0c505217def320f_2_1035x280.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e90e5a0a8715cb6071706e2af0c505217def320f_2_1380x374.png 2x" data-dominant-color="9ABF78"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lstm_imu</span><span class="informations">1480×403 55.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But every time I run the code I get this error:</p>
<blockquote>
<p>File “/home/ajava/Projects/DeepLIO/deeplio/models/nets/imu_feat_nets.py”, line 86, in forward<br>
out, (self.h, self.c) = self.rnn(x_padded, (self.h, self.c))<br>
File “/opt/miniconda3/envs/env_master/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/opt/miniconda3/envs/env_master/lib/python3.7/site-packages/torch/nn/modules/rnn.py”, line 559, in forward<br>
self.dropout, self.training, self.bidirectional, self.batch_first)<br>
(print_stack at /opt/conda/conda-bld/pytorch_1579022060824/work/torch/csrc/autograd/python_anomaly_mode.cpp:57)<br>
Traceback (most recent call last):<br>
File “/home/ajava/Projects/DeepLIO/deeplio/train.py”, line 67, in <br>
trainer.run()<br>
File “/home/ajava/Projects/DeepLIO/deeplio/models/trainer.py”, line 135, in run<br>
self.train(epoch)<br>
File “/home/ajava/Projects/DeepLIO/deeplio/models/trainer.py”, line 250, in train<br>
loss.backward(retain_graph=False)<br>
File “/opt/miniconda3/envs/env_master/lib/python3.7/site-packages/torch/tensor.py”, line 195, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/opt/miniconda3/envs/env_master/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 99, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [11, 8, 128]], which is output 0 of CudnnRnnBackward, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
</blockquote>
<p><strong>Update</strong><br>
The problem is that changing the shape of the ‘out’-variable  is an in-place operation, detaching it removes the error message:</p>
<pre data-code-wrap="python"><code class="lang-python">        out = out.view(s, b, self.num_dir, self.hidden_size)
        y = out[-1, :, 0].contiguous().squeeze().detach() # many-to-one
        return y
</code></pre>
<p>But can anyone plz tell me, is that the correct way? What happens to the loss which is calc. using detached output? Is the computational graph still valid?</p>
<p>Thanks</p>

googlebot: <p>No, .view() is fine, your problem is somewhere beyond this block, where memory is rewritten. And detach() is not valid, as subgraph leading to detached tensor won’t be trained.</p>