Model training script emits RuntimeError when upgraded from 1.4.0 to 1.6-nightly

username: ResidentMario
cooked: <p>I tried to upgrade a model training script from a 1.4.0 pytorch environment, in which it is working, to a 1.6.0-nightly pytorch environment, in which it is not.</p>
<p>The fact there are no code changes leads me to believe that there’s some broken behavior in 1.6.0-nightly, which is why I’ve filed this issue as a bug report:</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/39085" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/39085" target="_blank" rel="nofollow noopener">Model training script emits RuntimeError when upgraded from 1.4.0 to 1.6-nightly</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-05-27" data-time="20:41:26" data-timezone="UTC">08:41PM - 27 May 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/ResidentMario" target="_blank" rel="nofollow noopener">
          <img alt="ResidentMario" src="https://avatars2.githubusercontent.com/u/3466341?v=4" class="onebox-avatar-inline" width="20" height="20">
          ResidentMario
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
I have a relatively straightforward feed-forward model training script written in and verified working using pytorch 1.4.0. I upgraded my...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Maybe I’m wrong though? In case I am, would love for someone here to double check my layers. I’m getting a <code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</code> and but I’m <em>pretty</em> sure that none of my operations are in-place? Would love to get a second pair of eyes on this. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
<p>Here’s a gist with the model training code: <a href="https://gist.github.com/ResidentMario/d3767d5f3944ca95218f83e5ec0f5b44" rel="nofollow noopener">https://gist.github.com/ResidentMario/d3767d5f3944ca95218f83e5ec0f5b44</a>.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can check the release note for 1.5. In particular the part about fixing the inplace detection code for the optimizer.step() functions for the builtin optimizers.</p>
<p>The issue you have is that the optimizer.step() modifies the weights inplace. But if these weights are needed to compute the backward later, you’ll see the error you see right now.<br>
I am not sure why you do the optimizer.step() at the beginning of the loop in your code, but you should move it to just after the backward if possible. (or at least not between the forward and the backward).</p>

username: ResidentMario
cooked: <p>Wow, thanks for the quick response!</p>
<p>The reason that they were in that order was that I got a nag message from <code>OneCycleLR</code> for placing the optimizer step function ahead of the learning rate scheduler step function. To reverse their order, I copy-pasted the optimizer up to its current position in the script.</p>
<p>After updating the training loop so everything goes at the end:</p>
<pre><code class="lang-auto">        for epoch in range(self.n_epochs):
            lvs = []
            for i, (X_batch, y_batch) in enumerate(batches):
                X_batch = X_batch.cuda()
                y_batch = y_batch.cuda()
                
                y_pred = model(X_batch).squeeze()
                loss = self.loss_fn(y_pred, y_batch)
                optimizer.zero_grad()
                
                loss.backward()
                
                lv = loss.detach().cpu().numpy()
                lvs.append(lv)
                if i % 100 == 0:
                    print(f"Epoch {epoch + 1}/{self.n_epochs}; Batch {i}; Loss {lv}")

                optimizer.step()
                scheduler.step()
            print(
                f"Epoch {epoch + 1}/{self.n_epochs}; Average Loss {np.mean(lvs)}"
            )
</code></pre>
<p>It works. Thanks for the help! Will definitely be more careful about the ordering of these steps going forward.</p>