Gradient computation modified by an inplace operation

Daniel_Maier: <p>Hi, I have a forward method of an encoder and at the end I want to calculate the euclidean distance between each sequence element (like self-attention but the attention scores are the L2 norm). I have tried for hours but no matter how I calculate this autograd is not having it.</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 128, 128]], which is output 0 of SqrtBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

</code></pre>
<p>I know that it is this calculation that causes the error because if I return before it, the error doesn’t occur. Why does this error occur in my calulation?</p>
<pre><code class="lang-auto">def forward(self, src, src_pos):
        src_mask = (src == -1)
        src = self.src_embed(src, src_pos)
        mem = self.encoder(src, src_key_padding_mask=src_mask)


        query = mem.unsqueeze(2)    # (B, M, 1, d_m)
        key = mem.unsqueeze(1)      # (B, 1, N, d_m)

        # Compute the squared Euclidean distance
        squared_distance = torch.sum((query - key) ** 2, dim=-1)  # (B, N, N)

        # Take the square root to get the Euclidean distance
        euclidean_distance = torch.sqrt(squared_distance)
        return euclidean_distance
</code></pre>

KFrank: <p>Hi Daniel!</p>
<aside class="quote no-group quote-modified" data-username="Daniel_Maier" data-post="1" data-topic="191242" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/daniel_maier/48/58768_2.png" class="avatar"> Daniel_Maier:</div>
<blockquote>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 128, 128]], which is output 0 of SqrtBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>…</p>
<pre><code class="lang-auto">        squared_distance = torch.sum((query - key) ** 2, dim=-1)  # (B, N, N)

        # Take the square root to get the Euclidean distance
        euclidean_distance = torch.sqrt(squared_distance)
        return euclidean_distance
</code></pre>
</blockquote>
</aside>
<p>The error message you posted gives two useful clues:  First, the shape of<br>
the tensor being modified inplace is <code>[1, 128, 128]</code>, and second, it is the<br>
output of a <code>sqrt()</code> operation.</p>
<p>Such an error could occur in parts of your code that you haven’t posted, but<br>
it could also be happening to <code>euclidean_distance</code>, the result of a <code>sqrt()</code><br>
operation.  Is the shape of <code>euclidean_distance</code> <code>[1, 128, 128]</code>?</p>
<p>I would look at what happens to <code>euclidean_distance</code> <em>after</em> it is returned<br>
from the <code>forward()</code> function you posted.  Do you see any inplace operations<br>
being applied to it in the code that is subsequently called?</p>
<p>If this is the issue, then returning a <code>clone()</code> of <code>euclidean_distance</code>, e.g.,<br>
<code>return euclidean_distance.clone()</code>, might be sufficient to fix the problem.</p>
<p>For further suggestions about how to debug such issues, please see this post:</p>
<aside class="quote quote-modified" data-post="7" data-topic="171826">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-torch-floattensor-64-1-which-is-output-0-of-asstridedbackward0-is-at-version-3-expected-version-2-instead-hint-the-backtrace-further-a/171826/7">"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Fahmyadan and Sangyoon! 

Here are some suggestions about how to track down (and maybe fix) 
inplace-modification errors.  Note that an inplace modification in the forward 
pass is not necessarily* an error – it depends on whether and how the tensor 
that was modified is used in the backward pass.  Note that inplace operations 
can be useful for saving memory – if you replace an innocent inplace operation 
with an out-of-place equivalent, your training will use more memory (and, to a 
minor e…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

Daniel_Maier: <p>Hi Frank,<br>
thanks a lot. You were right, returning euclidean_distance.clone() fixes the problem but I don’t understand why. After it is returned it only gets modified in a custom loss function (her it is ‘input’). The problem seems to be the sigmoid + clamping line which is strange because I used this loss for normal attention and it worked fine.</p>
<pre><code class="lang-auto">class DistanceLoss(nn.Module):
    '''
    Focal loss for binary classification with or without logits. Targets that are neither exactly 0 nor 1 are ignored.
    Occurences of the two classes are auto-balanced when using reduction='mean'
    '''
    def __init__(self, gamma=2, sigmoid_clamp: float = 1e-4, reduction='mean'):
        super().__init__()
        self.gamma = gamma
        self.sigmoid_clamp = sigmoid_clamp
        self.reduction = reduction

    def forward(self, input, target):
        input_s = torch.clamp(input.sigmoid_(), min=self.sigmoid_clamp, max=1-self.sigmoid_clamp)
        pos_idx = (target == 1)
        neg_idx = (target == 0)
        pos_loss = torch.pow(1 - input_s[pos_idx], self.gamma) * torch.log(input_s[pos_idx])
        neg_loss = torch.pow(input_s[neg_idx], self.gamma)     * torch.log(1 - input_s[neg_idx])

        if self.reduction == 'mean':
            n_pos = pos_idx.sum()
            n_neg = neg_idx.sum()
            if n_pos == 0:
                return -neg_loss.sum()/n_neg
            else:
                return -pos_loss.sum()/n_pos - neg_loss.sum()/n_neg
        elif self.reduction == 'sum':
            return -pos_loss.sum() - neg_loss.sum()
        return -pos_loss, -neg_loss
</code></pre>
<p>Edit: changing sigmoid_() to sigmoid() fixes the problem without needing euclidean_distance.clone(). Still, why isn’t this a problem with normal attention. Another problem is now that the loss is a number but some model parameters are set to NaN after the optimizer step. It is like the gradient can’t reach them. Could this be related to this?</p>
<pre><code class="lang-auto"> query = mem.unsqueeze(2)    # (B, M, 1, d_m)
 key = mem.unsqueeze(1)      # (B, 1, N, d_m)
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Daniel_Maier" data-post="3" data-topic="191242">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/daniel_maier/48/58768_2.png" class="avatar"> Daniel_Maier:</div>
<blockquote>
<p>Edit: changing sigmoid_() to sigmoid() fixes the problem without needing euclidean_distance.clone(). Still, why isn’t this a problem with normal attention.</p>
</blockquote>
</aside>
<p>Disallowing inplace operations (e.g. here the inplace <code>sigmoid_</code> usage) depends on the used operations and which tensors are needed in their original for gradient computation. <code>torch.sqrt</code> uses its <code>result</code> as seen <a href="https://github.com/pytorch/pytorch/blob/674c104d122797644805b07151850e1dffb10fa2/tools/autograd/derivatives.yaml#L1553-L1555">here</a> and your “normal attention” approach might use other operations not depending on the result for the gradient calculation.</p>