Auto-grad backward issue with combined loss function with the same inputs

vincent024: <p>Hi,<br>
I have a loss function that combines loss functions that use the same inputs. However, even though I removed all “in-place operations” and tried different things from “.clone()” or “.clone().detatch()” - I still got an "in-place operation error " with the autograd for the second part (and I can’t find where the problem is).</p>
<p>Also, these 2 parts can be used alone without any “inplace operation errors”, but when used together, it doesn’t work anymore.</p>
<p>The main part related to the problem is:</p>
<pre><code class="lang-auto">def forward(self, segin, edgein, segmask, edgemask):
        main_loss  = 0;
        if self.seg_weight &gt; 0:
            seg_loss =  self.seg_weight * self.seg_loss(segin, segmask)
            main_loss = main_loss + seg_loss;
        #end

        if self.att_weight &gt; 0:
            a =  self.edge_attention(segin.detach().clone(), segmask.detach().clone(), edgein.detach().clone());
            att_loss = self.att_weight * a;
            main_loss = main_loss + att_loss;
        #end
        main_loss = torch.mean(main_loss); #main_loss.mean();
        
        return main_loss;
#end

def edge_attention(self, input, target, edge): # seg_pred, seg_gt, edge_pred
        filler = torch.ones_like(target)
        targets = torch.where(edge &gt; 0.5, target, filler);
        return self.seg_loss(input.clone(), targets);
#end
</code></pre>
<p>Where seg_loss is a “Dice loss” calculation which also works fine when used alone OR when only one of these 2 parts is used (but not when these 2 parts are used together).</p>
<p>Does anyone have any idea what could be causing the “inplace operation” error and how I could do to fix it ?</p>
<p>I also try to use “torch.autograd.set_detect_anomaly(True)”, but eighter I don’t know where I need to set it or it’s doesn’t explain to me where exactly is the issue.</p>

soulitzer: <p>Are you running into the error “variables needed for gradient computation has been modified by an inplace operation”</p>

vincent024: <p>Yes, I am getting that kind of error…</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 21, 1]] is at version 21; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
<p>I’m also strangely getting the same error with another loss function (which also sums multiple losses).<br>
When I enable “torch.autograd.set_detect_anomaly(True)” it indicates an error on the line of the “cardinality” calculation in the main function that I use for the calculation of the Jaccard Loss function which is this function (it was also saying the same thing before):</p>
<pre><code class="lang-auto">def soft_jaccard_score_with_weight(output: torch.Tensor, target: torch.Tensor, weight: torch.Tensor,
                                   smooth: float = 0.0, eps: float = 1e-7, dims=None) -&gt; torch.Tensor:
    assert output.size() == target.size()
    
    if dims is not None:
        intersection = torch.sum(output * target * weight, dim=dims)
        cardinality = torch.sum( (output + target)*weight, dim=dims)
    else:
        intersection = torch.sum(output * target * weight)
        cardinality = torch.sum( (output + target)*weight)
    #end
    union = cardinality - intersection
    jaccard_score = (intersection + smooth) / (union + smooth).clamp_min(eps)
    return jaccard_score
#end
</code></pre>
<p>However, as we can see the “torch.sum” is not done as an “in place operation” so I don’t understand why it is pointing there.</p>
<p>Also, “weight” and “target” don’t have gradients - only the “output” variable (which is the model’s prediction) has gradients.</p>
<p>I don’t know if this could be useful, but I am using PyTorch 1.11.0 (maybe it’s related to a bug of that PyTorch version ?)</p>

vincent024: <p><strong>Well, I finally solved my problem.</strong></p>
<p>The problem came from the fact that the “soft_jaccard_score_with_weight” function is used in the “forward” of a class which calculates the “dynamic weights” to be used at each iteration/training batch.</p>
<p>However, since I was using the same “class Object” (of this loss function class) several times in a row, the code for calculating the gradient probably saw it as that this weight (yet without gradient) had been modified in an “inplace” way, thus causing this problem.</p>
<p>Just using different “loss objects” instead, so that each one is separate from each other, solved my problem.</p>