GRU model:one of the variables needed for gradient computation has been modified by an inplace operation

username: sigma_x
cooked: <p>This happens in a GRU model, specifically with the GRU layer processing the hidden input, judging by the error report.</p>
<p>GRU layer has two inputs: data (embedding layer, size [sequence length, batch size, embedding features]) and hidden features from the previous step, size [1, batch size, hidden features].</p>
<p><code>output, hidden = self.gru(x, hidden)</code></p>
<p>The error trace stack is</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [21, 3, 256]], which is output 0 of CudnnRnnBackward, is at version 1; 
expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>The size of the float tensor points to the hidden state of the GRU. I definitely didn’t use an inplace operation on it anywhere. Other tensors and variables seem to be fine.</p>
<p>So I don’t quite understand what to do here.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Given that it is output 0, isn’t it the output?<br>
Did you make sure that you don’t modify the output inplace either?</p>
<p>Could you share a small code sample that reproduces the issue if you can’t find the problem?</p>

username: sigma_x
cooked: <p>hi <a class="mention" href="/u/alband">@albanD</a>, there are two networks here, both  GRU: encoder and decoder. It is the decoder that returns this error. Here’s the code for the encoder:</p>
<pre><code class="lang-auto">class EncoderGRU(nn.Module):

     # seq2seq model
     def __init__(self, vocab_size, embedding_size, hidden_size, device, num_layers=1):
         super(EncoderGRU, self).__init__()
         self.embedding = nn.Embedding(vocab_size, embedding_size)
         self.gru = nn.GRU(embedding_size, hidden_size, num_layers, batch_first=False)
         self.device = device

     def forward(self, input_dialogue, hidden=None):
         x = self.embedding(input_dialogue)
         x = x.view(x.size()[1], x.size()[0],-1)
         output, hidden = self.gru(x, hidden)
         return output, hidden
</code></pre>
<p>The error refers to the third line in the forward method, because <code>[torch.cuda.FloatTensor [21, 3, 256]]</code> refers to the hidden state (sequence length:21, batch size: 3, number of features: 256), not the embedding vector. Where do I modify hidden inplace?</p>

username: sigma_x
cooked: <p>Also, looking at the error report, I’m not sure I understand what sort of ‘version’ the exception refers to: ‘is at version 1, expected version 0’. What does this mean?</p>

username: albanD
cooked: <p>Versions are a way to track inplace operations. every time a tensor is modified inplace, its version counter is incremented by 1.<br>
You might be modifying the output or hidden inplace after it is returned by the encoder gru. In the rest of your forward pass.</p>

username: sigma_x
cooked: <p>Right! It was modified (unsqueeze) before input inthe decoder, and I forgot to remove the _ inplace symbol! So given all these problem with autograd handling inplace operations, when can I actually use them? Just for the dataset manipulation?</p>

username: albanD
cooked: <p>You can use them in the autograd. But they are not all be allowed. Unfortunately, it is a bit hard to track down what caused the error in general.</p>

username: sigma_x
cooked: <p>well <code>unsqueeze_</code> is definitely one of the more uself in-place operations, and if it is not allowed, I’m not sure when to use it. The one I had a problem with was not the <code>hidden</code> state, as it seemed from the error stack trace: it was the output of the encoder, the full history of the feedforward, size <code>sequence_length x batch_size x hidden_state_size</code>.</p>

username: albanD
cooked: <p>The thing is that a single operation is never disallowed, it is a combinations of operations that can be.</p>
<pre><code class="lang-auto">c = a + b
a.div_(2)
(c + a).sum().backward()
</code></pre>
<p>This will work, because the value of a is not needed to compute the gradients in the sum operation.</p>
<pre><code class="lang-auto">c = a * b
a.div_(2)
(c + a).sum().backward()
</code></pre>
<p>This won’t work because the value of a is needed to compute the gradient wrt b in the multiplication.</p>
<p>So a given inplace operation can be allowed or not, depending on the surrounding code.</p>