Encounter in-place operation in delayed online learning

username: Joseph_Kao
cooked: <p>I’m working on a delayed online learning scheme where I want to do the optimization step after certain delay, and I’m encountering the in-place operation when doing slicing. Here is a rough pseudo code of what I’m doing.<br>
Let E be the number of episodes, T be the time horizon of each episode, and D be the delay.</p>
<pre><code class="lang-auto">input=torch.zeros(E,T)
output=torch.zeros(E,T)
output_target=torch.zeros(E,T)
for e in range(E):
   for t in range(T):
      input[e,t] = some outer environment source
      output[e,t] = model( input[e,t] )  # model is my defined neural net from nn.Module
      Act in the outer environment with output[e,t]
      output_target[e,t] = from outer environment source
   if e&gt;=D:
      for t in range(T):
         loss += loss_function( output_target[e-D,t] , output[e-D,t] )
      optimizer.zero_grad()
      loss.backward(retain_graph=True)
      optimizer.step()
</code></pre>
<p>And I got the in-place operation error: “RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation:”<br>
It seems that it’s because I keep filling the slice of output with new results, so the version of output also changes. What is the efficient way to program the thing I want to do then?</p>
<p>Thanks!</p>

username: Joseph_Kao
cooked: <p>Here is a minimal working example that produces the error:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
import matplotlib.pyplot as plt

class DNN(nn.Module):
    def __init__(self,dim):
        super(DNN,self).__init__()
        self.fc=nn.Sequential(
            nn.Linear(dim,dim),
            nn.Tanh(),
            nn.Linear(dim,dim)
        )
    def forward(self,data):
        return self.fc(data)

def main():
    E=20
    T=20
    D=5
    dim=3
    model=DNN(dim)
    optimizer=torch.optim.Adam(model.parameters())
    loss_fn=nn.MSELoss()
    data_in=torch.zeros(E,T,dim)
    data_out=torch.zeros(E,T,dim)
    data_target=torch.zeros(E,T,dim)
    for e in range(E):
        for t in range(T):
            data_in[e,t]=torch.from_numpy(np.random.rand(dim))
            data_out[e,t]=model(data_in[e,t])
            data_target[e,t]=torch.from_numpy(np.random.rand(dim))
        if e&gt;=D:
            loss=0.0
            for t in range(T):
                loss += loss_fn(data_target[e-D,t],data_out[e-D,t])
            optimizer.zero_grad()
            loss.backward(retain_graph=True)
            optimizer.step()
    
if __name__=='__main__':
    main()
</code></pre>
<p>Error message:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 3]] is at version 120; expected version 119 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>Basically the idea is, I want to train the model one by one with the old samples while testing/generating the new samples, i.e. in an online manner. In particular, at episode e, I want to train the model with the sample (output and target) from episode (e-D) while generating the new output of episode e. I have no idea how to do this other than slicing, but slicing (I believe) causes the in-place operation error.</p>

username: ptrblck
cooked: <p>Could you remove the inplace modification of the <code>loss</code> tensor and use:</p>
<pre><code class="lang-python">loss = loss + loss_fn(...)
</code></pre>
<p>instead?<br>
Based on the posted code snippet it seems to be the first place I would start to debug the issue.</p>

username: Joseph_Kao
cooked: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> , thank you for your reply. But after changing to “loss=loss+loss_fn(data_target[e-D,t],data_out[e-D,t])” I just got the same error:</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 3]] is at version 120; expected version 119 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>

username: Joseph_Kao
cooked: <p>Updated code with the modification mentioned above that still produces the same error:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np

class DNN(nn.Module):
    def __init__(self,dim):
        super(DNN,self).__init__()
        self.fc=nn.Sequential(
            nn.Linear(dim,dim),
            nn.Tanh(),
            nn.Linear(dim,dim)
        )
    def forward(self,data):
        return self.fc(data)

def main():
    E=20
    T=20
    D=5
    dim=3
    model=DNN(dim)
    optimizer=torch.optim.Adam(model.parameters())
    loss_fn=nn.MSELoss()
    data_in=torch.zeros(E,T,dim)
    data_out=torch.zeros(E,T,dim)
    data_target=torch.zeros(E,T,dim)
    for e in range(E):
        for t in range(T):
            data_in[e,t]=torch.from_numpy(np.random.rand(dim))
            data_out[e,t]=model(data_in[e,t])
            data_target[e,t]=torch.from_numpy(np.random.rand(dim))
        if e&gt;=D:
            loss=0.0
            for t in range(T):
                loss=loss+loss_fn(data_target[e-D,t],data_out[e-D,t])
            optimizer.zero_grad()
            loss.backward(retain_graph=True)
            optimizer.step()

if __name__=='__main__':
    main()
</code></pre>

username: tom
cooked: <p>There are several aspects of wrong here. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
One source of error is that the model needs the unmodified data_in but the versioning does not capture regions of <code>data_in</code>. Depending on your setup, you could either use a (nested) list of tensors or generate the inputs outside the loop.</p>
<p>If you fix this ad-hoc by using <code>model(data_in[e,t].clone())</code>, there will be another error from backpropagation with the weights that is triggered when you backprob inside the loop. I would venture that the <code>retain_graph=True</code> is an error here (it almost universally is an error unless you know and can articulate the precise reason why it is not) and you would want to structure your computation to separate forward and backward passes here.<br>
This is my general advice: Don’t keep stuff from the previous training (= forward + backward + optimization) step unless you have a good reason. Personally, I really try to articulate with some precision why things need to be carried over.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Joseph_Kao
cooked: <p>Hi <a class="mention" href="/u/tom">@tom</a> , it took me some time to digest what you wrote, especially what you meant by another error from backprop inside the loop.</p>
<p>But my question still remains. Indeed, I’m exactly asking how should I program if (forward + backward + opt) can’t be done in one step – the good reason is for application consideration. In online learning I have to generate good prediction (hence the forward pass) in each step; in the problem I consider, I’d like to add a delay for the backward and optimization parts due to application reasons.</p>
<p>Let me give a concrete flow of I want. Let’s assume for simplicity the delay D=2. I’ll use<br>
In[s] for input of step s, Out[s] for output of step s, and M_i for the i-th version of the model.</p>
<ol>
<li>The initial model is M_0</li>
<li>Generate Out[1] = M_0(In[1])</li>
<li>Generate Out[2] = M_0(In[2])</li>
<li>Generate Out[3] = M_0(In[3])</li>
<li>Backward and optimize with Out[1] and In[1] to generate M_1</li>
<li>Generate Out[4] = M_1(In[4])</li>
<li>Backward and optimize with Out[2] and In[2] to generate M_2</li>
<li>Generate Out[5] = M_2(In[5])<br>
…etc</li>
</ol>
<p>Note that in my case, deep copying another M_0 and feeding in In[1] will not necessarily produce Out[1] as the procedure contains randomness, so that Out[1] is not reproducible basically.</p>
<p>What is the correct way to program this then?</p>
<p>Thanks in advance!</p>

username: tom
cooked: <p>This is a decidedly difficult pattern. Would it be possible to move taking the gradient (but not the optimizer step) to where the output is generated? That would make it much easier because you only need to keep the gradients around, not the computational graph. The problem here is that you can only take gradients between the forward and changing the weights (so 7 would fail because 5 changed the weights from what they were in 3). If this isn’t an option you could probably replace all the modules to call <code>.clone()</code> on all parameters before using them. This is a bit tedious because you need to re-implement/subclass all the nn.Modules you are using, but it would be feasible.</p>
<p>So the other thing we touched upon is that for this kind of pattern, you probably want to use a list to store the input and output tensors, not a single large tensor you index into.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Joseph_Kao
cooked: <p>Hi <a class="mention" href="/u/tom">@tom</a>, thank you so much for your reply. Yes, taking the gradient first is totally fine for my application. I have revised my (actual) code based on your suggestions, and it is now working great.</p>
<p>This is how the minimal working example should look like based on the suggestions:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np

class DNN(nn.Module):
    def __init__(self,dim):
        super(DNN,self).__init__()
        self.fc=nn.Sequential(
            nn.Linear(dim,dim),
            nn.Tanh(),
            nn.Linear(dim,dim)
        )
    def forward(self,data):
        return self.fc(data)

def main():
    E=50
    T=20
    D=5
    dim=4
    model=DNN(dim)
    optimizer=torch.optim.Adam(model.parameters())
    loss_fn=nn.MSELoss()
    data_in=[[torch.zeros(dim) for t in range(T)] for e in range(E)]
    data_out=[[torch.zeros(dim) for t in range(T)] for e in range(E)]
    data_target=[[torch.zeros(dim) for t in range(T)] for e in range(E)]
    optimizer_list=[None for e in range(E)]
    for e in range(E):
        for t in range(T):
            data_in[e][t]=torch.from_numpy(np.random.rand(dim)).float()
            data_out[e][t]=model(data_in[e][t])
            data_target[e][t]=torch.from_numpy(np.random.rand(dim)).float()
        loss=0.0
        for t in range(T):
            loss=loss_fn(data_target[e][t],data_out[e][t])
        optimizer_list[e]=torch.optim.Adam(model.parameters())
        optimizer_list[e].zero_grad()
        loss.backward()
        if e&gt;=D:
            optimizer_list[e-D].step()
    
if __name__=='__main__':
    main()
</code></pre>
<p>I have been struggling on this for several weeks, you really save my life <a class="mention" href="/u/tom">@tom</a>. Thanks again!</p>

username: Joseph_Kao
cooked: <p>The code from my last post is wrong (doesn’t work as mentioned). Keeping a list of optimizers is useless, we should keep a (actually two) list of models and copy the gradients. The following is the correct code.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np

class DNN(nn.Module):
    def __init__(self,dim):
        super(DNN,self).__init__()
        self.fc=nn.Sequential(
            nn.Linear(dim,dim),
            nn.Tanh(),
            nn.Linear(dim,dim)
        )
    def forward(self,data):
        return self.fc(data)

def main():
    E=50
    T=20
    D=5
    dim=4
    model=DNN(dim)
    optimizer=torch.optim.Adam(model.parameters())
    loss_fn=nn.MSELoss()
    data_in=[[torch.zeros(dim) for t in range(T)] for e in range(E)]
    data_out=[[torch.zeros(dim) for t in range(T)] for e in range(E)]
    data_target=[[torch.zeros(dim) for t in range(T)] for e in range(E)]
    model_list=[None for e in range(E)]
    model_update_list=[None for e in range(E)]
    for e in range(E):
        if e==0:
            model_list[e]=DNN(dim)
        else:
            model_list[e]=type(model_update_list[e-1])(dim)
            model_list[e].load_state_dict(model_update_list[e-1].state_dict())
        for t in range(T):
            data_in[e][t]=torch.from_numpy(np.random.rand(dim)).float()
            data_out[e][t]=model_list[e](data_in[e][t])
            data_target[e][t]=torch.from_numpy(np.random.rand(dim)).float()
        loss=0.0
        for t in range(T):
            loss=loss_fn(data_target[e][t],data_out[e][t])
        optimizer=torch.optim.Adam(model_list[e].parameters())
        optimizer.zero_grad()
        loss.backward()
        model_update_list[e]=type(model_list[e])(dim)
        model_update_list[e].load_state_dict(model_list[e].state_dict())
        optimizer_update=torch.optim.Adam(model_update_list[e].parameters())
        if e&gt;=D:
            for p1,p2 in zip(model_update_list[e].named_parameters(),model_list[e-D].named_parameters()):
                p1[1].grad=p2[1].grad
            optimizer_update.step()
    
if __name__=='__main__':
    main()
</code></pre>