Having problem with In place operation

milhouse74: <p>Good afternoon everyone,</p>
<p>I am new to PyTorch and I am having trouble with my code. The problem is that it seems that the operation that I am using is inplace, but I cant seem to understand how to fix it.</p>
<pre><code class="lang-auto">        # Create a tensor to store outputs during the Forward
        logits = torch.zeros(self.seq_len, self.batch_size, self.vocab_size).to(device)

        # initialize tensors needed for hidden calculation
        reset = torch.zeros(self.num_layers, self.batch_size, self.hidden_size)
        forget = torch.zeros(self.num_layers, self.batch_size, self.hidden_size)
        memory_temp = torch.zeros(self.num_layers, self.batch_size, self.hidden_size)
        one_matrix = torch.zeros(self.num_layers, self.batch_size, self.hidden_size)

        # https: // discuss.pytorch.org / t / how - to - copy - a - variable - in -a - network - graph / 1603 / 6
        # For each time step
        # https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py
        for timestep in range(self.seq_len):
            # Apply dropout on the embedding result
            input_ = self.dropout(embed_out[timestep])
            for layer in range(self.num_layers):
                # operations
                reset[layer] = torch.sigmoid(self.r[layer](torch.cat([input_, hidden[layer]], 1)))
                forget[layer] = torch.sigmoid(self.z[layer](torch.cat([input_, hidden[layer]], 1)))
                memory_temp[layer] = torch.tanh(self.h[layer](torch.cat([input_, torch.mul(reset[layer], one_matrix[layer])], 1)))
                hidden[layer] = torch.mul((one_matrix[layer]-forget[layer]), one_matrix[layer]) + torch.mul(reset[layer], memory_temp[layer])

                # Apply dropout on this layer, but not for the recurrent units
                input_ = self.dropout(hidden[layer])

            # Store the output of the time step
            logits[timestep] = self.out_layer(input_)
</code></pre>
<p>The error message is as follow:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [20, 200]], which is output 0 of SelectBackward, is at version 70; expected version 69 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

</code></pre>
<p>Thanks a lot for helping me out!</p>
<p>Francois</p>

albanD: <p>Hi,</p>
<p>The inplace operation happen when you do <code>reset[layer] = XXX</code>. This writes s subset of the Tensor inplace.<br>
You can avoid doing this by just having <code>reset</code> being a list.<br>
And if later you need to convert the content of that list into a single Tensor, you can do <code>torch.stack(reset, dim=0)</code>.</p>

milhouse74: <p>Thanks Alban for the quick answer!</p>
<p>It solves some of my problems, that is perfect. I did some changes to my code to fix this. However, I know that I am still doing an in-place operation on the memory and/or hidden line of code. I dont have a clue of how to fix it.</p>
<pre><code class="lang-auto"># operations
reset = torch.sigmoid(self.r[layer](torch.cat([input_, hidden[layer]], 1)))
forget = torch.sigmoid(self.z[layer](torch.cat([input_, hidden[layer]], 1)))
memory_temp = torch.tanh(self.h[layer](torch.cat([input_, torch.mul(reset, hidden[layer])], 1)))
hidden[layer] = torch.mul((one_matrix-forget), hidden[layer]) + torch.mul(reset, memory_temp)
</code></pre>
<p>Would the solution to have the formula of memory_temp and hidden[layer] be dependant on the clone() of hidden would make sense? I did clone just before those operation and it seems to work. However, would that still work with backprop?</p>
<p>Thanks again for your help!</p>

albanD: <p>If the autograd does not raise an error, then the backprop will work <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Here, you can have <code>hidden</code> to be a list and just append to it no?</p>