Problems training a model once, the interpreter says that a tensor required for the backpropagation was modified but i dont see where

ranon-rat: <p>Hello im trying to build a d3qn agent but i have problems when i try to train the model.4<br>
For some reason in the first iteration there is no problem but in the second one it gives me the next error:</p>
<pre data-code-wrap="py"><code class="lang-py">    264     # some Python versions print out the first line of a multi-line function
    265     # calls in the traceback and some print out the last line
--&gt; 266     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    267         tensors,
    268         grad_tensors_,

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [448, 112]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>Here are the functions that I build for training the model</p>
<pre data-code-wrap="py"><code class="lang-py">  def train_batch(self, batch):
    i=0
    with torch.autograd.detect_anomaly():

      for indicator,winned,hidden, action, reward, next_indicator,next_winned, done in batch:
        print(i)

        self.train_once(indicator,winned,hidden,action,reward,next_indicator,next_winned,done)
  # esto sera utilizado para el train short memory y train long memory
  def train_once(self,indicator,winned,hidden,action,reward,next_indicator,next_winned,done):

    indicator = torch.tensor(indicator)
    winned = torch.tensor(winned)
    next_indicator = torch.tensor(next_indicator)
    next_winned = torch.tensor(next_winned)

    pred, hidden = self.model(indicator, winned, hidden)
    target = pred.clone()
    
    with torch.no_grad():
        Q_new = reward
        if not done:
            out, _ = self.model(next_indicator, next_winned, hidden)
            Q_new = reward + self.gamma * torch.max(out)
    
    target[0][action] =torch.tensor( Q_new)
    self.optimizer.zero_grad()
    
    loss = F.mse_loss(pred,target)
    loss.backward()
    self.optimizer.step()
</code></pre>
<p>Here is how the model is</p>
<pre data-code-wrap="py"><code class="lang-py">class Model(nn.Module):

  def __init__(self,interval_input,number_indicators,kernel_length=8,kernels=16,lstm_num_layers=40,custom_lstm_size=False,lstm_hidden_length_custom=80):

    super(Model,self).__init__()
    length,convolutional_layers=get_number_cnn(interval_input,kernel_length)
    #esto sera para la lstm es importante
    lstm_hidden_length= lstm_hidden_length_custom if custom_lstm_size else length*kernels+number_indicators*2+20
    #esto lo guardaremos para cuando necesitemos calcular ciertas cosas
    self.lstm_num_layers=lstm_num_layers
    self.lstm_hidden_length=lstm_hidden_length
    #definimos la red neuronal convolucional
    self.conv=nn.Sequential(nn.Conv1d(number_indicators,kernels,kernel_length),
                            nn.ReLU(),
                            *([nn.Conv1d(kernels,kernels,kernel_length),nn.ReLU()]*(convolutional_layers)),
                            )


    self.lstm=nn.LSTM(length*kernels+number_indicators*2+2,lstm_hidden_length,num_layers=lstm_num_layers)
    #difference for improving its learning
    self.value=nn.Sequential( 
        nn.Linear(in_features=int(lstm_hidden_length),out_features=int(lstm_hidden_length*0.7)),
        nn.ReLU(),
        nn.Linear(in_features=int(lstm_hidden_length*0.7),out_features=int(lstm_hidden_length*0.7*0.7)),
        nn.ReLU(),
        nn.Linear(in_features=int(lstm_hidden_length*0.7*0.7),out_features=1)
        )
    
    #number of actions
    self.adv=nn.Sequential( 
        nn.Linear(in_features=int(lstm_hidden_length),out_features=int(lstm_hidden_length*0.7)),
        nn.ReLU(),
        nn.Linear(in_features=int(lstm_hidden_length*0.7),out_features=int(lstm_hidden_length*0.7*0.7)),
        nn.ReLU(),
        nn.Linear(in_features=int(lstm_hidden_length*0.7*0.7),out_features=3)
        )

  def empty_lstm_attributes(self):
    h0 = torch.zeros(self.lstm_num_layers, self.lstm_hidden_length,dtype=torch.float32)
    c0 = torch.zeros(self.lstm_num_layers, self.lstm_hidden_length,dtype=torch.float32)
    return (h0,c0)

  def forward(self,indicators,winned,hidden,future=0):
    output=self.conv(indicators)
    output=torch.cat(
        (
          winned, #i join the two inputs for the lstm
        output.view(output.size(0), -1)
        ), #here is the result of the convolutional neural network
         dim=-1)
    output,new_hidden=self.lstm(output,hidden)
    adv=self.adv(output)
    value=self.value(output)
    q = value + adv - adv.mean()

    return (q,new_hidden)

</code></pre>

KFrank: <p>Hi Ranon!</p>
<aside class="quote no-group quote-modified" data-username="ranon-rat" data-post="1" data-topic="201291" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ranon-rat/48/69418_2.png" class="avatar"> ranon-rat:</div>
<blockquote>
<p>in the first iteration there is no problem but in the second one it gives me the next error:</p>
<pre data-code-wrap="py"><code class="lang-py">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [448, 112]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>…</p>
<pre data-code-wrap="py"><code class="lang-py">train_once(self,indicator,winned,hidden,action,reward,next_indicator,next_winned,done):
    ...
    pred, hidden = self.model(indicator, winned, hidden)
    ...
    loss.backward()
    self.optimizer.step()
</code></pre>
<p>…</p>
<pre data-code-wrap="py"><code class="lang-py">    output,new_hidden=self.lstm(output,hidden)
</code></pre>
</blockquote>
</aside>
<p>I am going to speculate as follows:</p>
<p>The first time through, <code>hidden</code> doesn’t depend on the parameters of <code>lstm</code>.<br>
However, the second time through the (old) <code>hidden</code> depends on the parameters<br>
of <code>lstm</code> <em>before</em> they were updated by <code>optimizer.step()</code>.</p>
<p>In general, <code>optimizer.step()</code> modifies inplace the parameters of the model<br>
being optimized.</p>
<p>When you call <code>.backward()</code> the second time, <code>loss</code>, through <code>target</code>, depends<br>
on the old <code>hidden</code>, which depends on the old <code>lstm</code> parameters, so you get<br>
the inplace-modification error.</p>
<p>Would it make sense for your use case (assuming that this is the cause of your<br>
error) to <code>.detach()</code> the old <code>hidden</code> from the computation graph?  You would<br>
still pass the old <code>hidden</code> into <code>model</code> so the new <code>pred</code> and <code>hidden</code> would<br>
depend on the old <code>hidden</code>’s values, but those values would be <code>.detach()</code>ed<br>
from their dependency on the old values of <code>lstm</code>’s parameters.</p>
<p>Aside from this speculation, please see the suggestions for locating and fixing<br>
inplace-modification errors given in the following post:</p>
<aside class="quote quote-modified" data-post="7" data-topic="171826">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-torch-floattensor-64-1-which-is-output-0-of-asstridedbackward0-is-at-version-3-expected-version-2-instead-hint-the-backtrace-further-a/171826/7">"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Fahmyadan and Sangyoon! 

Here are some suggestions about how to track down (and maybe fix) 
inplace-modification errors.  Note that an inplace modification in the forward 
pass is not necessarily* an error – it depends on whether and how the tensor 
that was modified is used in the backward pass.  Note that inplace operations 
can be useful for saving memory – if you replace an innocent inplace operation 
with an out-of-place equivalent, your training will use more memory (and, to a 
minor e…
  </blockquote>
</aside>

<p>Good luck!</p>
<p>K. Frank</p>

ranon-rat: <p>It works now, thank you frank god bless you.</p>