Cannot find inplace operation using set_detect_anomaly(True)

Pepi: <p>I am building an LSTM model for time series prediction. We start with a pandas DataFrame and load that into a TensorDataset and train the model.</p>
<p>I have received this error a few times and have managed to fix it in the past, however this time I just cannot find the in-place operation in question so I would really appreciate a second eye.</p>
<p>Before forming the Tensor dataset I did have to perform some operations to construct the dataset in pandas but I don’t think that should affect autograd, right?</p>
<p>Code snippets:</p>
<pre><code class="lang-auto">leads_df = leads_df.resample('6H').sum() # just a resample of df
leads_df_var = leads_df.iloc[:, :10]
</code></pre>
<pre><code class="lang-auto">from torch.utils.data import TensorDataset, DataLoader
from torch.utils.data.dataset import random_split

def generate_sequences(df, tw):
    data = list()
    L = len(df)
    for i in range(L-tw):
        sequence = df[i:i+tw].values
        target = df[i+tw:i+tw+1].values
        data.append((sequence, target))
    return data

def list_of_tensors_to_dataset(data):
    tensor_x = torch.Tensor([x[0] for x in data])
    tensor_y = torch.Tensor([x[1] for x in data])
    return TensorDataset(tensor_x, tensor_y)
</code></pre>
<pre><code class="lang-auto">seq_len = 4 
data = generate_sequences(leads_df_var, seq_len)
dataset = list_of_tensors_to_dataset(data)

num_features = dataset[0][0].shape[1]
output_size = dataset[0][0].shape[1]
hidden_size = 128
batch_size = 16

train_len = int(len(dataset)*split)
lens = [train_len, len(dataset)-train_len]
dataloader = DataLoader(dataset, batch_size=batch_size)
</code></pre>
<p>Model:</p>
<pre><code class="lang-auto">class LeadsPredictor(nn.Module):
    
    def __init__(self, num_features, seq_len, hidden_size, output_size):
        super().__init__()

        self.num_features = num_features
        self.hidden_size = hidden_size
        self.seq_len = seq_len
        self.output_size = output_size
        self.n_layers = 1
        self.lstm = nn.LSTM(self.num_features,
                            self.hidden_size,
                            batch_first=True)

        self.fc = nn.Linear(self.hidden_size*self.seq_len, self.output_size)

        hidden_state = torch.zeros(self.n_layers, batch_size, self.hidden_size)
        cell_state = torch.zeros(self.n_layers, batch_size, self.hidden_size)

        self.hidden = (hidden_state, cell_state)
            
    def forward(self, x):
        x, self.hidden = self.lstm(x, self.hidden)
        x = x.contiguous().view(batch_size,-1)
        x = self.fc(x)
        return x
</code></pre>
<p>Train function:</p>
<pre><code class="lang-auto">import time
import matplotlib.pyplot as plt
def train(model, dataloader, num_epochs):
    with torch.autograd.set_detect_anomaly(True):
        model.train()
        losses = list()
        ts = time.time()
        for epoch in tqdm(range(num_epochs)):
            epoch_losses = list()
            for idx, (seq, label) in enumerate(dataloader):

                optimizer.zero_grad()
                out = model(seq)
                loss = criterion(out, label.squeeze())
                loss.backward(retain_graph=True)
                optimizer.step()

        te = time.time()
        fig, ax = plt.subplots()
        ax.plot(range(num_epochs), losses)
        plt.show()
        mins = int((te-ts) / 60)
        secs = int((te-ts) % 60)
        print('Training completed in {} minutes, {} seconds.'.format(mins, secs))
        return losses, model
</code></pre>
<p>Error Message:<br>
<code>one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [128, 512]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</code></p>
<p>I would really reaally appreciate some help with this</p>

albanD: <blockquote>
<p>Before forming the Tensor dataset I did have to perform some operations to construct the dataset in pandas but I don’t think that should affect autograd, right?</p>
</blockquote>
<p>No that won’t</p>
<p>Can you share the traceback reported by the detect anomaly? Where does it point?</p>

Pepi: <pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-244-b5623f61a7a3&gt; in &lt;module&gt;
----&gt; 1 losses, model = train(model, dataloader, 5)

&lt;ipython-input-242-54990647c901&gt; in train(model, dataloader, num_epochs)
     13                 out = model(seq)
     14                 loss = criterion(out, label)
---&gt; 15                 loss.backward(retain_graph=True)
     16                 optimizer.step()
     17 

~\anaconda3\envs\MarketingAnalytics\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)
    183                 products. Defaults to ``False``.
    184         """
--&gt; 185         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    186 
    187     def register_hook(self, hook):

~\anaconda3\envs\MarketingAnalytics\lib\site-packages\torch\autograd\__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    123         retain_graph = create_graph
    124 
--&gt; 125     Variable._execution_engine.run_backward(
    126         tensors, grad_tensors, retain_graph, create_graph,
    127         allow_unreachable=True)  # allow_unreachable flag

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [128, 512]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p><a class="mention" href="/u/alband">@albanD</a> This is the full traceback</p>
<p>​</p>

albanD: <p>Don’t you have a second traceback (as a warning) just above this one?</p>

Pepi: <p><a class="mention" href="/u/alband">@albanD</a> No there’s nothing above that. Maybe I’m filtering out the warnings, although I haven’t added a command that filters them. This was my main confusion too as I was expecting a pointer to the operation.</p>

albanD: <p>That is very surprising…<br>
You can still check for inplace ops on the result of a transpose op.<br>
Common ones happen in the Linear forward.</p>
<p>Why do you set <code>retain_graph=True</code> in your code btw?</p>
<p>Doing things like this would cause this error:</p>
<pre><code class="lang-auto">model = nn.Linear(...)
opt = optim.SGD(model.parameters(), ...)

out = model(inp)
out.backward(retain_graph=True)
opt.step()
out.backward() # Here, the linear weights were modified inplace

</code></pre>

Pepi: <p><a class="mention" href="/u/alband">@albanD</a></p>
<p>So I do have a linear layer in my network. Would this correspond to that? What might be the correct way to do it?</p>
<p>I noticed in your snippet you call backward twice, if this is in reference to my retain_graph=True parameter, when I remove it I get the following error:</p>
<p><code>RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.</code></p>
<p>I understand that this is not regular so is there something I should fix there?</p>
<p>Also, thank you so much for your help so far</p>

albanD: <p>Ho I just noticed that you update <code>self.hidden</code> in your forward function.<br>
That means that the next iteration will depend on the previous one.</p>
<p>I am not sure how you want to handle this. But you should either <code>.detach()</code> if you want to keep the updated value without having gradients flow back all the way to the previous iteration.<br>
Or if the value of the original hidden state should be fixed to a constant value, then you need to re-initialize it with a Tensor with no history.</p>

Pepi: <p>Ah I think that worked! Thank you so much!</p>
<p>I have changed the line:</p>
<pre><code class="lang-auto">x, self.hidden = self.lstm(x, self.hidden)
</code></pre>
<p>to:</p>
<pre><code class="lang-auto">x, _ = self.lstm(x, self.hidden)
</code></pre>
<p>So that it does not update <code>self.hidden</code></p>