Pytorch Inplace Operation Error

FezTheImmigrant: <p>When running the below code, I get the following issue:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [8, 1]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>The code below performs two forward passes, with one backward propagation:</p>
<pre><code class="lang-auto">for epoch in range(10):
    count = 0
    for i in range(0, len(X), batch_size):
        Xbatch = X[i:i+batch_size]
        y_pred = model(Xbatch)
        ybatch = y[i:i+batch_size]

        loss = loss_fn(y_pred, ybatch)

        if count % 2 == 0:
            if count != 0:
                optimizer.zero_grad()
                # Pytorch doesn't like this
                ((loss_1 + loss_2) / 2).backward()
                optimizer.step()
            loss_1 = loss
        else:
            loss_2 = loss
       count += 1
</code></pre>
<p>but if I slightly alter the code like so, it works fine:</p>
<pre><code class="lang-auto">for epoch in range(10):
    count = 0
    for i in range(0, len(X), batch_size):
        Xbatch = X[i:i+batch_size]
        y_pred = model(Xbatch)
        ybatch = y[i:i+batch_size]

        loss = loss_fn(y_pred, ybatch)



        if (count+1) % 2 == 0:
            optimizer.zero_grad()
            ((loss_1 + loss) / 2).backward()
            optimizer.step()
        else:
            loss_1 = loss

        count += 1
</code></pre>
<p>Why does the first version cause this issue?</p>

ptrblck: <p>I cannot reproduce the issue using:</p>
<pre data-code-wrap="python"><code class="lang-python">model = nn.Linear(1, 1)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.MSELoss()


for epoch in range(10):
    count = 0
    for i in range(0, 10):
        x = torch.randn(1, 1)
        output = model(x)
        target = torch.randn_like(x)
        
        loss = loss_fn(output, target)

        if count % 2 == 0:
            if count != 0:
                optimizer.zero_grad()
                # Pytorch doesn't like this
                ((loss_1 + loss_2) / 2).backward()
                optimizer.step()
            print("updating loss_1")
            loss_1 = loss
        else:
            print("updating loss_2")
            loss_2 = loss
        count += 1
</code></pre>
<p>Could you check if my code reproduces it on your side and if not add the missing pieces to your code making it executable?</p>

KFrank: <p>Hi Sergio!</p>
<aside class="quote no-group" data-username="FezTheImmigrant" data-post="1" data-topic="200183" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/f19dbf/48.png" class="avatar"> FezTheImmigrant:</div>
<blockquote>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [8, 1]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
</blockquote>
</aside>
<p><code>optimizer.step()</code> modifies the parameters of your model inplace, and, in<br>
your first version, causes the inplace-modification error.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">for epoch in range(10):
    count = 0
    for i in range(0, len(X), batch_size):
        Xbatch = X[i:i+batch_size]
        y_pred = model(Xbatch)
        ybatch = y[i:i+batch_size]

        loss = loss_fn(y_pred, ybatch)

        if count % 2 == 0:
            if count != 0:
                optimizer.zero_grad()
                # Pytorch doesn't like this
                ((loss_1 + loss_2) / 2).backward()
                optimizer.step()
            loss_1 = loss
        else:
            loss_2 = loss
       count += 1
</code></pre>
</blockquote>
</aside>
<p>For <code>count = 0</code>, you compute <code>loss</code> and assign it to <code>loss_1</code>.  <code>loss_1</code> is<br>
linked to the parameters of <code>model</code> by the computation graph.</p>
<p>For <code>count = 1</code>, you compute a new value of <code>loss</code> and assign it to <code>loss_2</code>.</p>
<p>For <code>count = 2</code>, you compute a new value of <code>loss</code>, backpropagate<br>
<code>loss_1</code> and <code>loss_2</code>, call <code>optimizer.step()</code>, which modifies the<br>
parameters of model inplace, and assign <code>loss</code> to <code>loss_1</code>  At this point<br>
<code>loss_1</code> depends on <code>model</code>’s <em>old</em> parameter values because it was<br>
computed (as <code>loss</code>) prior to <code>optimizer.step()</code>.</p>
<p>For <code>count = 3</code>, you compute a new value of <code>loss</code> and assign it to<br>
<code>loss_2</code> (but do not call <code>.backward()</code> nor <code>optimizer.step()</code>).</p>
<p>Then for <code>count = 4</code>, you call <code>.backward()</code> on <code>loss_1 + loss_2</code>.<br>
Backpropagating <code>loss_1</code> backpropagates through the <code>model</code> parameters,<br>
but <code>loss_1</code> depends on the <em>old</em> values of the <code>model</code> parameters,<br>
triggering the inplace-modification error.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">for epoch in range(10):
    count = 0
    for i in range(0, len(X), batch_size):
        Xbatch = X[i:i+batch_size]
        y_pred = model(Xbatch)
        ybatch = y[i:i+batch_size]

        loss = loss_fn(y_pred, ybatch)

        if (count+1) % 2 == 0:
            optimizer.zero_grad()
            ((loss_1 + loss) / 2).backward()
            optimizer.step()
        else:
            loss_1 = loss

        count += 1
</code></pre>
</blockquote>
</aside>
<p>In this case you either backpropagate the newly-computed <code>loss</code> and<br>
<code>loss_1</code> and call <code>optimizer.step()</code> (which modifies your <code>model</code> inplace)<br>
or you assign the newly-computed <code>loss</code> to <code>loss_1</code>.  But when you iterate<br>
further and next call <code>.backward()</code> both <code>loss_1</code> and <code>loss</code> are newly<br>
computed in the sense that both have been computed after the most recent<br>
call to <code>optimizer.step()</code>.  So they both backpropagate through the versions<br>
of <code>model</code>’s parameters that they were computed with (so no inplace error.)</p>
<p>Best.</p>
<p>K. Frank</p>

KFrank: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="200183" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I cannot reproduce the issue using:</p>
<pre data-code-wrap="python"><code class="lang-python">model = nn.Linear(1, 1)
</code></pre>
</blockquote>
</aside>
<p>At issue is the use of a single <code>Linear</code>.  When you backpropagate through<br>
<code>Linear.weight</code> <em>in order to compute <code>Linear.weight</code>’s gradient,</em> you<br>
don’t need the value of <code>Linear.weight</code> itself, so it doesn’t’ matter that<br>
it’s been modified inplace.  (<code>x</code> doesn’t carry <code>requires_grad = True</code>.)</p>
<p>But you do need the value of <code>Linear.weight</code> to backpropagate through<br>
the input to <code>Linear</code>.</p>
<p>Try:</p>
<pre data-code-wrap="python"><code class="lang-python">model = torch.nn.Sequential (
    torch.nn.Linear (1, 1),
    torch.nn.Linear (1, 1)
)
</code></pre>
<p>This will trigger Sergio’s error.</p>
<p>Now you do need the value of the second <code>Linear</code>’s weight because you<br>
need to backpropagate through the first <code>Linear</code> to compute gradients for<br>
the first <code>Linear</code>’s parameters.</p>
<p>Best.</p>
<p>K. Frank</p>

FezTheImmigrant: <p>This makes perfect sense. Thank you <a class="mention" href="/u/kfrank">@KFrank</a> and <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>

ptrblck: <p>Ah yes, of course! Thanks for pointing out this mistake <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>