Inplace operation error with sliding window data loader

username: kyeongjun1007
cooked: <p>Hi.</p>
<p>I’m training self-supervision RNN with time-series data and sliding window.<br>
But an inplace operation error occurs when training model…</p>
<p>the following is my code.</p>
<pre><code class="lang-auto">import torch
import numpy as np
from torch.utils.data import DataLoader
from torch import nn

data = torch.randn(100,9)

class TimeseriesDataset(torch.utils.data.Dataset):   
    def __init__(self, X, y, seq_len=1):
        self.X = X
        self.y = y
        self.seq_len = seq_len

    def __len__(self):
        return self.X.__len__() - (self.seq_len-1)

    def __getitem__(self, index):
        return (self.X[index:index+self.seq_len], self.y[index:index+self.seq_len])
    
data = torch.tensor(np.array(data), dtype = torch.float32)

train_dataset = TimeseriesDataset(data[:-1], data[1:], seq_len=5)
train_loader = DataLoader(train_dataset, batch_size = 1, shuffle = False)

input_size = 9
hidden_size = 9
num_layers = 3

rnn = nn.RNN(input_size= input_size, hidden_size = hidden_size, num_layers = num_layers, batch_first = True)

num_epochs = 5
learning_rate = 0.01

optimizer = torch.optim.Adam(rnn.parameters(), lr=learning_rate)

criterion = nn.CrossEntropyLoss()

hidden = None

for epoch in range(num_epochs) :
  for i, d in enumerate(train_loader) :        
      out, hidden = rnn.forward(d[0], hidden)

      loss = criterion(d[1], out)
      
      optimizer.zero_grad()
      loss.backward(retain_graph=True)
      optimizer.step()
      
      print(loss.item())
</code></pre>
<p>resullt in the following error</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [9, 9]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>

username: srishti-git1110
cooked: <p>Hi,<br>
I’m not very sure about why the error is occurring, but do you have a reason to use <code>retain_graph=True</code> in the backward call?</p>
<p>It sometimes causes the error that you are facing.</p>

username: kyeongjun1007
cooked: <p>If loss.backward function is executed multiple times in one epoch, retain_graph option have to be True as i know…</p>
<p>if I run that code without retain_graph option, result in the following error… T T</p>
<p>Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</p>

username: kyeongjun1007
cooked: <p>I found cause of this problem…!</p>
<aside class="quote quote-modified" data-post="1" data-topic="163910">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kyeongjun1007/48/53615_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation/163910">RuntimeError : one of the variables needed for gradient computation has been modified by an inplace operation</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi. 
I’m training self-supervision RNN with time-series data and sliding window. 
but an inplace operation error occurs when training model. 
the following is my code. 
import torch
import numpy as np
from torch.utils.data import DataLoader
from torch import nn

data = torch.randn(100,9)

class TimeseriesDataset(torch.utils.data.Dataset):   
    def __init__(self, X, y, seq_len=1):
        self.X = X
        self.y = y
        self.seq_len = seq_len

    def __len__(self):
        return self.X.…
  </blockquote>
</aside>