How to avoid "one of the variables needed for gradient computation has been modified by an inplace operation"

username: abhisek
cooked: <p>Trying to implement Tversky metric as follows:</p>
<pre><code class="lang-python">import torch
def tversky_score(
        predictions,
        targets,
        alpha=0.5,
        beta=0.5,
        eps=1e-16,
        encode_target=True
):
    """
    Ref A: https://arxiv.org/abs/1706.05721
    alpha = beta = 0.5 : Dice coefficient
    alpha = beta = 1 : Tanimoto coefficient / Jaccard Index
    alpha + beta = 1 : Produces set of F*-scores
    :param predictions:
    :param targets:
    :param alpha:
    :param beta:
    :param eps:
    :param encode_target:
    :return:
    """
    if encode_target:
        # noinspection PyArgumentList
        gt = predictions.clone().zero_()
        # noinspection PyArgumentList
        assert (targets.ndim == 4) and (
                targets.size(1) == 1
        ), "Only single channel images can be encoded!"
        gt = gt.scatter_(1, targets.to(dtype=torch.int64), 1)
        # gt.requires_grad = True
    else:
        gt = targets.clone()
        # gt.requires_grad = True
    # flatten label and prediction tensors
    predictions = predictions.view(-1)
    gt = gt.view(-1)

    # True Positives, False Positives &amp; False Negatives
    tp = (predictions * gt).sum()
    fp = ((1 - gt) * predictions).sum()
    fn = (gt * (1 - predictions)).sum()

    tversky = (tp + eps) / (
            tp + alpha * fp + beta * fn + eps
    )
    return tversky
</code></pre>
<p>My loss functions are:</p>
<pre><code class="lang-python">class FocalTverskyLoss(nn.Module):
    def __init__(self, alpha=0.5, beta=0.5, gamma=1, smooth=1e-16):
        """
        Ref A: https://arxiv.org/abs/1706.05721
        alpha = beta = 0.5, gamma = 1 : Dice coefficient
        alpha = beta = 1, gamma = 1 : Tanimoto coefficient / Jaccard Index
        alpha + beta = 1, gamma = 1 : Produces set of F*-scores
        :param alpha:
        :param beta:
        :param gamma:
        :param smooth:
        """
        super(FocalTverskyLoss, self).__init__()
        self.alpha = alpha
        self.beta = beta
        self.gamma = gamma
        self.eps = smooth

    def forward(
        self,
        predictions: torch.Tensor,
        targets: torch.Tensor,
        encode_target: bool = True
    ):
        tversky = tversky_score(
            predictions=predictions,
            targets=targets,
            encode_target=encode_target,
            alpha=self.alpha,
            beta=self.beta,
            eps=self.eps
        )
        focal_tversky = (1 - tversky) ** self.gamma
        return focal_tversky

class PCHuberLoss(nn.Module):
    def __init__(self, reduction='mean', delta=1.0):
        super(PCHuberLoss, self).__init__()
        self.net = nn.HuberLoss(
            reduction=reduction, delta=delta
        )

    def forward(self, predictions, targets):
        return self.net(predictions.F, targets.F)

class HybridLoss(nn.Module):
    def __init__(
            self,
            alpha=0.5,
            beta=0.5,
            gamma=1,
            smooth=1e-16,
            reduction='mean',
            delta=1.0,
            p_factor=0.5,
            i_factor=0.5,
            encode_label=True
    ):
        super(HybridLoss, self).__init__()
        self.im_criterion = FocalTverskyLoss(
            alpha=alpha,
            beta=beta,
            gamma=gamma,
            smooth=smooth
        )
        self.pc_criterion = PCHuberLoss(
            reduction=reduction,
            delta=delta
        )
        self.p_f = p_factor
        self.i_f = i_factor
        self.img_encode = encode_label

    def forward(self, predictions, targets):
        loss = self.im_criterion(
            predictions=predictions['label_tensor'],
            targets=targets['label_tensor'],
            encode_target=self.img_encode
        )
        if 'label' in predictions.keys():
            loss = (self.i_f * loss) + (
                self.p_f * self.pc_criterion(
                    predictions=predictions['point_tensor'],
                    targets=targets['point_tensor']
                )
            )
        return loss

</code></pre>
<p>I use <code>HybridLoss</code> to calculate loss. In <code>loss.backward()</code> call I encounter <code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</code>. Possibly this is due to <code>zero_()</code> call since it is an <code>in place</code> operation. However, I am not sure how can I fix this. I would appreciate some pointers.</p>

username: KFrank
cooked: <p>Hi Abhisek!</p>
<aside class="quote no-group quote-modified" data-username="abhisek" data-post="1" data-topic="129408" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abhisek/48/38962_2.png" class="avatar"> abhisek:</div>
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">        gt = predictions.clone().zero_()
</code></pre>
<p>Possibly this is due to <code>zero_()</code> call since it is an <code>in place</code> operation.</p>
</blockquote>
</aside>
<p>This is likely the problem.  (<code>.clone()</code> without <code>.detach()</code> is still part<br>
of the computation graph, and you then <code>.zero_()</code> the cloned tensor<br>
in place.)</p>
<aside class="quote no-group">
<blockquote>
<p>However, I am not sure how can I fix this.</p>
</blockquote>
</aside>
<p>It looks like you want to initialize <code>gt</code> as a zero-tensor of the same<br>
shape as <code>predictions</code>.  Try:</p>
<pre data-code-wrap="python"><code class="lang-python">gt = torch.zeros_like (predictions)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: abhisek
cooked: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="129408">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<pre><code class="lang-auto">gt = torch.zeros_like (predictions)
</code></pre>
</blockquote>
</aside>
<p>Thanks for the suggestion <a class="mention" href="/u/kfrank">@KFrank</a>. I already tried that as below:</p>
<pre data-code-wrap="python"><code class="lang-python">def tversky_score(
        predictions,
        targets,
        alpha=0.5,
        beta=0.5,
        eps=1e-16,
        encode_target=True
):
    if encode_target:
        # noinspection PyArgumentList
        # gt = predictions.clone()  # .zero_()
        gt = torch.zeros_like(predictions)
        # noinspection PyArgumentList
        assert (targets.ndim == 4) and (
                targets.size(1) == 1
        ), "Only single channel images can be encoded!"
        gt = gt.scatter_(1, targets.to(dtype=torch.int64), 1)
        gt.requires_grad = True
    else:
        gt = targets.clone()
        gt.requires_grad = True
    # flatten label and prediction tensors
    predictions = predictions.view(-1)
    gt = gt.view(-1)

    # True Positives, False Positives &amp; False Negatives
    tp = (predictions * gt).sum()
    fp = ((1 - gt) * predictions).sum()
    fn = (gt * (1 - predictions)).sum()

    tversky = (tp + eps) / (
            tp + alpha * fp + beta * fn + eps
    )
    return tversky
</code></pre>
<p>but I still get the exact same runtime error (<code>one of the variables needed for gradient computation has been modified by an inplace operation</code>). Is it because of the line</p>
<pre data-code-wrap="python"><code class="lang-python"> gt = gt.scatter_(1, targets.to(dtype=torch.int64), 1)
</code></pre>
<p>Is it necessary to perform <code>gt.requires_grad = True</code> since an entirely separate tensor is being created outside the graph?</p>

username: KFrank
cooked: <p>Hi Abhisek!</p>
<aside class="quote no-group" data-username="abhisek" data-post="3" data-topic="129408" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abhisek/48/38962_2.png" class="avatar"> abhisek:</div>
<blockquote>
<p>I still get the exact same runtime error</p>
</blockquote>
</aside>
<p>Could you post a trimmed-down, runnable, fully-self-contained script<br>
that reproduces the error?  (Please have the script print out the version<br>
of pytorch you’re using, <code>torch.__version__</code>.)</p>
<p>Please provide hard-coded (or random) sample data (with size and<br>
dimensionality as small as possible while still reproducing the error).<br>
Also delete any comments not relevant to the issue and see if you can<br>
can get rid of the <code>assert</code> and <code>if encode</code> branch, and  simplify or leave<br>
out some of the calculations without losing the error.</p>
<aside class="quote no-group">
<blockquote>
<p>Is it because of the line</p>
<pre data-code-wrap="python"><code class="lang-python"> gt = gt.scatter_(1, targets.to(dtype=torch.int64), 1)
</code></pre>
</blockquote>
</aside>
<p>In isolation, this in-place  <code>scatter_()</code> shouldn’t break anything.</p>
<p>(Note that the assignment to <code>gt</code> is redundant because <code>gt</code> is modified<br>
in-place and <code>gt.scatter_()</code> merely returns the original <code>gt</code> object, as<br>
modified.)</p>
<aside class="quote no-group">
<blockquote>
<p>Is it necessary to perform <code>gt.requires_grad = True</code> since an entirely separate tensor is being created outside the graph?</p>
</blockquote>
</aside>
<p>No, this shouldn’t be necessary, and, in general, you don’t want to<br>
be doing this.  In this specific case, doing so shouldn’t break anything,<br>
but, more generally, you could be asking for trouble down the line.</p>
<p>As a general rule, if you find you need to set <code>requires_grad = True</code><br>
for anything other than an “upstream” tensor whose value you want<br>
optimize, it’s a sign that you’re doing something wrong.</p>
<p>Best.</p>
<p>K. Frank</p>

username: abhisek
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a> Thanks for your suggestions. Following your recommendations, I narrowed down the part causing this issue. Turns out in a layer block I used <code>torch.nn.ReLU(True)</code> without paying much attention to it. The issue occurred since I explicitly performed this <code>inplace ReLU</code> operation.</p>