Why my linear layer is considered as an inplace operation?

momomo: <p>My model architecture is defined as the following:</p>
<pre><code class="lang-auto">class NN(torch.nn.Module):
    def __init__(self):
        super(NN, self).__init__()
        self.linear1 = nn.Linear(3, 9)
        self.act1 = nn.ReLU(inplace=False)
        self.linear2 = nn.Linear(9, 3)
        
    def forward(self, x):
        x = self.linear1(x)
        x = self.act1(x)
        x = self.linear2(x)
        return x
</code></pre>
<p>And I am trying to train a model by using codes similar to the following:</p>
<pre><code class="lang-auto">    model = NN()
    model = model.double()
    loss_fn = nn.MSELoss()
    optimizer=torch.optim.Adam(model.parameters(),lr=1e-3)

    epoches = 100
    batch_size = 5
    iterations = total_num / batch_size

    for epoch in range(1,epoches+1):
      train_a, ans = dataset_init(A, B, ratio)
      preditions = torch.tensor(np.zeros((total_num, 2, 3, 1, 3)))
      
      for iteration in range(0,iterations): 
          loss_train = torch.empty(batch_size)
          for i in range(0+batch_size*iteration, batch_size*(iteration+1)):
              prediction[i] = model(torch.unsqueeze(train_a[i].to(torch.float64),0))
              loss_train[i-batch_size*iteration] = loss_fn(prediction[i].clone(),ans[i])
          loss_train_avg = torch.mean(loss_train.clone())
          optimizer.zero_grad()
          loss_train_avg.backward(retain_graph=True)
          optimizer.step()
</code></pre>
<p>After running the code, I got:</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [9, 3]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
<p>However, I did not find inplace operations in my codes. And the  ‘torch.DoubleTensor [9, 3]’ in the error message looks like something related to my 2nd Linear layer, so I deleted it to check, then the codes run smoothly without the error. I have also tried to use ‘x = self.linear2(x.clone())’, but the codes also report the same error message above.</p>
<p>Now I have no idea how to modify it, and I dont know why my linear layer is considered as an inplace operation. Can anybody help me?</p>

soulitzer: <p>The inplace comes from doing optimzer.step(). Why do you need retain_graph=True, removing that should resolve this error.</p>

momomo: <p>Hi, thanks for your reply.</p>
<p>I add ‘retain_graph=True’ to solve another error. If I remove it, my codes will report the following error message <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</p>
</blockquote>

KFrank: <p>Hi Momo!</p>
<aside class="quote no-group" data-username="momomo" data-post="3" data-topic="168333" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/momomo/48/55709_2.png" class="avatar"> momomo:</div>
<blockquote>
<p>I add ‘retain_graph=True’ to solve another error. If I remove it, my codes will report the following error message</p>
</blockquote>
</aside>
<p>As <a class="mention" href="/u/soulitzer">@soulitzer</a> noted, using <code>retain_graph = True</code> can often lead to<br>
inplace-modification errors.  (Based just on the code you posted, I don’t<br>
see why you would need <code>retain_graph = True</code>, so you should try to<br>
get rid of it.)</p>
<p>However, I don’t think that this is your main issue.</p>
<aside class="quote no-group" data-username="momomo" data-post="1" data-topic="168333">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/momomo/48/55709_2.png" class="avatar"> momomo:</div>
<blockquote>
<pre><code class="lang-auto">          for i in range(0+batch_size*iteration, batch_size*(iteration+1)):
              prediction[i] = model(torch.unsqueeze(train_a[i].to(torch.float64),0))
              loss_train[i-batch_size*iteration] = loss_fn(prediction[i].clone(),ans[i])
          loss_train_avg = torch.mean(loss_train.clone())
  
</code></pre>
</blockquote>
</aside>
<p>Using indices to assign into a tensor is an inplace operation, and the<br>
tensors <code>prediction</code> and <code>loss_train</code> are in your computation graph,<br>
so your <code>for i in range():</code> loop is making such inplace modifications,<br>
hence your error.</p>
<p>Without digging into perhaps better ways to organize your code, you could<br>
do something like:</p>
<pre data-code-wrap="python"><code class="lang-python">        loss_train_list = []
        for i in range(batch_size):
            prediction_i = model (train_a[i].to(torch.float64))
            loss_train_i = loss_fn(prediction_i.clone(),ans[i])
            loss_train_list.append (loss_train_i)
            prediction[i] = prediction_i   # if needed
            loss_train[i] = loss_train_i   # if needed
        loss_train_stack = torch.stack (loss_train_list)
        loss_train_avg = torch.mean(loss_train_stack)
</code></pre>
<p>This way you don’t index-assign into your <code>loss_train</code> tensor; rather, you<br>
collect the <code>loss_train</code> slices computed in your for-loop into a list, and<br>
then pass the <code>loss_train_stack</code> version of your <code>loss_train</code> tensor to<br>
<code>torch.mean()</code>.</p>
<p>(If for some reason you still need to index-assign into <code>prediction</code> and / or<br>
<code>loss_train</code>, you can safely do so because they are no longer in the<br>
computation graph that leads to <code>loss_train_avg</code> on which you call<br>
<code>.backward()</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>

momomo: <p>Hi, Frank</p>
<p>Thanks for your reply!<br>
My problem was solved after modifying accroding to your suggestion! <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face.png?v=12" title=":smiling_face:" class="emoji" alt=":smiling_face:" loading="lazy" width="20" height="20"></p>