Strange error: one of the variables needed for gradient computation has been modified by an inplace operation

Rancho_Xia: <p>Hi, everyone. I’m training a transformer for my research work. Here’s the <code>ScaledDotAttention</code> module:</p>
<pre><code class="lang-auto">    def forward(self, Q, K, V, d_k, attn_mask=None):
        scores = torch.matmul(Q, K.transpose(-1, -2)) / np.sqrt(d_k)
        if attn_mask is not None:
            assert scores.shape == attn_mask.shape
            scores.masked_fill_(attn_mask, -1e20)
        attns = self.dropout(torch.softmax(scores, dim=-1))d_v) -&gt; (batch_size, n_heads, seq_len, d_v)
        # if attn_mask is not None:
        #     attns.masked_fill_(attn_mask, 0.0)
        context = torch.matmul(attns, V)
        return context, attns
</code></pre>
<p>The above code works just fine. However, when I commented out <code>attns.masked_fill_(attn_mask, 0.0)</code>, an error occurs: <em>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</em></p>
<p>masked_fill attentions with 0.0 using attn_mask isn’t necessary since corresponding scores have already been masked. But I’m wondering why adding such an operation causes this error. Is anyone have an idea? Thanks!</p>
<p>ps: There’s too many codes in this project, and I can’t post all the codes here or create a minimal error-reproduction snippet. Sorry for that.</p>

Rancho_Xia: <p>btw, I’m 100 percent sure that that line is where the error stems from.</p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="Rancho_Xia" data-post="1" data-topic="139967">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rancho_xia/48/44725_2.png" class="avatar"> Rancho_Xia:</div>
<blockquote>
<pre><code class="lang-auto">            scores.masked_fill_(attn_mask, -1e20)
</code></pre>
</blockquote>
</aside>
<p>Any methods/functions in PyTorch that end in an underscore represent an inplace version of such operation. For example, multiplication can be done via <code>mul</code> or its inplace equivalent  <code>mul_</code></p>

Rancho_Xia: <p>Thanks for your reply. I know <code>*_</code> operations are in place, but what bothers me is that why this in place operation leads to this strange error, since this operation happens in the forward pass and <code>loss.backward()</code> happens in the backward pass.</p>

AlphaBetaGamma96: <p>Because you can’t differentiate in-place operations</p>

Rancho_Xia: <p>I don’t think that’s why this error occurs. There’re lots of in-place operations in Transformer, such as the line above <code>scores.masked_fill_(attn_mask, -1e20)</code>, but none of them triggers this error. Is there something that I’ve missed?</p>

ptrblck: <aside class="quote no-group" data-username="Rancho_Xia" data-post="6" data-topic="139967">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rancho_xia/48/44725_2.png" class="avatar"> Rancho_Xia:</div>
<blockquote>
<p>Is there something that I’ve missed?</p>
</blockquote>
</aside>
<p>You would have to check if the manipulated tensor is needed in its original form for the gradient calculation, which disallows inplace operations on it.<br>
<a href="https://discuss.pytorch.org/t/runtime-error-in-gradient-computation/139243/2">This post</a> shows you a simple example.</p>