How to find the inplace operation error during the backward?

username: Jeijgenhuijsen
cooked: <p>I build a custom LSTM model (it’s quite large), and somewhere in there there is an inplace operation which gives me an error during training.</p>
<p>error:</p>
<pre><code class="lang-auto"> File "c:\Users\jobei\Desktop\scriptie msc\code\models\train_model.py", line 46, in closure
    loss.backward()
  File "C:\Users\jobei\anaconda3\envs\machinelearning\lib\site-packages\torch\_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "C:\Users\jobei\anaconda3\envs\machinelearning\lib\site-packages\torch\autograd\__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [16, 16]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>I’ve already set torch.autograd.set_detect_anomaly(True), but it doesn’t really give me any useful info. Just trying out every small part of the model, but this takes a long time. Hope there is something better I can do.</p>
<p>The only 16x16 tensors in the model are the weights of the gate networks if I’m correct, but I never touch those myself as far as I’m aware.</p>
<p>this is the github link to the files if anyone is interested:</p><aside class="onebox githubfolder" data-onebox-src="https://github.com/DonGob/LOB-dynamic-sim-model/tree/custom_lstm/models">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/DonGob/LOB-dynamic-sim-model/tree/custom_lstm/models" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/DonGob/LOB-dynamic-sim-model/tree/custom_lstm/models" target="_blank" rel="noopener nofollow ugc">LOB-dynamic-sim-model/models at custom_lstm · DonGob/LOB-dynamic-sim-model</a></h3>

  <p><a href="https://github.com/DonGob/LOB-dynamic-sim-model/tree/custom_lstm/models" target="_blank" rel="noopener nofollow ugc">custom_lstm/models</a></p>

  <p><span class="label1">dynamic model for simulating a limit order book. Contribute to DonGob/LOB-dynamic-sim-model development by creating an account on GitHub.</span></p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>training is where I put the training loops.<br>
simple_linear_networks contains the LSTM gate networks etc,<br>
hawkes loss is a custom loss function I need for this network<br>
sd_PNHP is the network itself. it is a stacked continuous time lstm with two cell states per cell.<br>
it consists of a cellstate class, a class that stacks 4 cellstates onto eachother and finally a layer class that loops through these stacked cells.</p>

username: KFrank
cooked: <p>Hi J!</p>
<aside class="quote no-group" data-username="Jeijgenhuijsen" data-post="1" data-topic="181711" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jeijgenhuijsen/48/61134_2.png" class="avatar"> Jeijgenhuijsen:</div>
<blockquote>
<p>I’ve already set torch.autograd.set_detect_anomaly(True), but it doesn’t really give me any useful info.</p>
</blockquote>
</aside>
<p>In fact, <code>set_detect_anomaly (True)</code> is generally quite useful. Look at the<br>
lines of code in  The “Traceback of forward call” that detect-anomaly causes<br>
to be printed.  (Look at your lines of code – the internal pytorch code is less<br>
likely to be helpful.)  One of those lines of code is likely to be causing the inplace<br>
modification.</p>
<aside class="quote no-group">
<blockquote>
<p>The only 16x16 tensors in the model are the weights of the gate networks if I’m correct, but I never touch those myself as far as I’m aware.</p>
</blockquote>
</aside>
<p>Print out <code>._version</code> for those 16x16 weight tensors before and after your<br>
forward pass.  Is <code>._version</code> increasing for any of them, and, in particular,<br>
ending up with a value of <code>2</code>?  If so, they are being modified somewhere and<br>
that is likely to be the cause of your problem.  If not, you have some other<br>
16x16 tensors somewhere.  Find them and check if their <code>._version</code>s are<br>
showing inplace modifications.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: Jeijgenhuijsen
cooked: <p>Thanks alot Frank! I’ve found the problem, Silly mistake. Forgot to clear a list after every run</p>