RuntimeError : one of the variables needed for gradient computation has been modified by an inplace operation

kyeongjun1007: <p>Hi.</p>
<p>I’m training self-supervision RNN with time-series data and sliding window.<br>
but an inplace operation error occurs when training model.</p>
<p>the following is my code.</p>
<pre><code class="lang-auto">import torch
import numpy as np
from torch.utils.data import DataLoader
from torch import nn

data = torch.randn(100,9)

class TimeseriesDataset(torch.utils.data.Dataset):   
    def __init__(self, X, y, seq_len=1):
        self.X = X
        self.y = y
        self.seq_len = seq_len

    def __len__(self):
        return self.X.__len__() - (self.seq_len-1)

    def __getitem__(self, index):
        return (self.X[index:index+self.seq_len], self.y[index:index+self.seq_len])
    
data = torch.tensor(np.array(data), dtype = torch.float32)

train_dataset = TimeseriesDataset(data[:-1], data[1:], seq_len=5)
train_loader = DataLoader(train_dataset, batch_size = 1, shuffle = False)

input_size = 9
hidden_size = 9
num_layers = 3

rnn = nn.RNN(input_size= input_size, hidden_size = hidden_size, num_layers = num_layers, batch_first = True)

num_epochs = 5
learning_rate = 0.01

optimizer = torch.optim.Adam(rnn.parameters(), lr=learning_rate)

criterion = nn.CrossEntropyLoss()

hidden = None

for epoch in range(num_epochs) :
  for i, d in enumerate(train_loader) :        
      out, hidden = rnn.forward(d[0], hidden)

      loss = criterion(d[1], out)
      
      optimizer.zero_grad()
      loss.backward(retain_graph=True)
      optimizer.step()
      
      print(loss.item())
</code></pre>
<p>result in the following error</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [9, 9]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>

ptrblck: <p>This error is often caused if <code>retain_graph=True</code> is used while it might not be needed. Could you explain why you are using this argument and if you’ll get another error otherwise?</p>

kyeongjun1007: <p>If I don’t use retain_graph =True, following error occurs.</p>
<p>Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</p>
<p>I found cause of this error! the error comes from input of rnn model.<br>
I have to do &gt;&gt; out, hidden = rnn(x, hidden_init) or out, hidden = rnn(x) &lt;&lt; these.<br>
but &gt;&gt; out, hidden = rnn(x, hidden) &lt;&lt; this.</p>
<p>because output of model is used to input (variable ‘hidden’ is replaced over iteration), inplace operation is occured…</p>

MH_Rosenberg: <p>Lol I don’t actually understand the details of your solution but thank you!!! Your final statement allowed me to fix my code. I fixed my issue by cloning the hidden layer before I passed it to the model. My hypothesis is that the optimizer doesn’t want any variables to take on different values per iteration of the training loop. In my case, I needed to call the forward pass of the model several times per training loop iteration because (within each training loop iteration) I iterate over all of the time points (columns) of a single input (matrix) to generate a single output (scalar) that is ultimately compared to a true label (scalar).</p>
<p>I would be interested if you or others have any additional insights here. I suspect that I’m not doing this the way pytorch intends the package to be used.</p>

kyeongjun1007: <p>In pytorch nn.RNN official documentation, written as follows.<br>
<strong>out, hidden_n = rnn(x, hidden_0)</strong><br>
when your input data length is N, out = (out1, out2, … , outN)<br>
and outN is determined by W_out * (tahn(hidden_(N-1) * W_hidden + x_N * W_input + bias))</p>
<p>hidden_0 is initialization of the hidden state (0th hidden) and hidden_n is the nth hidden state (out before multiplication with output weight(W_out).  tahn(hidden_(N-1) * W_hidden + x_N * W_input + bias))</p>
<p>if you solve your ploblem using following code,<br>
out, hidden = model(x, hidden.clone())<br>
it could work because you can avoid inplace operation by making hidden into leaf tensor (hidden.clone())<br>
but it means your model initialize the hidden state to nth hidden state value of prior iteration i think…</p>
<p>check this page!<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.RNN.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.RNN.html</a></p>