About loss.backward error

Cverlpeng: <p>hi,</p>
<p>my network  have two branch , seg branch and dt branch<br>
I use outputs of two branch  to cmpute KLDivLoss，but I get error:one of the variables needed for gradient computation has been modified by an inplace operation , when I loss.backward</p>
<pre><code class="lang-python">class SegmentationMultiLosses(nn.CrossEntropyLoss):
    """2D Cross Entropy Loss with Multi-L1oss"""
    def __init__(self, nclass=-1, weight=None,size_average=True, ignore_index=-1):
        super(SegmentationMultiLosses, self).__init__(weight, size_average, ignore_index)
        self.nclass = nclass


    def forward(self, *inputs):
        
        out, target_img, target_exist, target_dt= tuple(inputs)
       
        out_img, out_dt_img, out_exist = out
        
        loss_seg = super(SegmentationMultiLosses, self).forward(out_img, target_img)
        loss_exist = nn.BCELoss().forward(out_exist, target_exist)
        loss_dt = nn.MSELoss().forward(out_dt_img, target_dt)

        out_dt_norm = self.dt_norm(out_dt_img)
        out_seg_sm  = F.softmax(out_img, dim=1)
        loss_fuse = nn.KLDivLoss().forward(out_seg_sm[:,1:,:,:], out_dt_norm.clone())

      
        print('loss_seg: {}  loss_exit: {}  loss_dt: {} loss_fuse: {} '.format(loss_seg.item(), 
                                          loss_exist.item()*0.1, loss_dt.item(), loss_fuse.item()))
        loss = loss_seg + loss_dt + 0.1*loss_exist + loss_fuse
        return loss

    def dt_norm(self, dt_out):
        # clamp a tensor [0, 10]   and Normalization
        max = 10
        min = 0
        for i in range(dt_out.shape[0]):
            for j in range(dt_out.shape[1]):
                dt_out[i,j,:,:] = torch.div(torch.clamp(dt_out[i,j,:,:], min, max), max)
        return dt_out


</code></pre>
<p>dt_norm function is an inplace operation ？<br>
thanks</p>

tom: <aside class="quote no-group" data-username="Cverlpeng" data-post="1" data-topic="51036">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cverlpeng/48/13070_2.png" class="avatar"> Cverlpeng:</div>
<blockquote>
<pre><code class="lang-auto">for i in range(dt_out.shape[0]):
    for j in range(dt_out.shape[1]): 
        dt_out[i,j,:,:] = torch.div(torch.clamp(dt_out[i,j,:,:], min, max), max)
</code></pre>
</blockquote>
</aside>
<p>This is bad! use the non-inplace <code>dt_out = x.clamp(min, max) / max</code> instead.</p>
<p>Best regards</p>
<p>Thomas</p>

Cverlpeng: <pre><code class="lang-python">def dt_norm(self, dt_out):
        # clamp a tensor [0, 10]   and Normalization
        max = 10
        min = 0
        dt_out = x.clamp(min, max) / max
        return dt_out
</code></pre>
<p>Is that right? Are the two functions the same?</p>
<p>and</p>
<p>loss_fuse = nn.KLDivLoss().forward(out_seg_sm[:,1:,:,:], out_dt_norm.clone())<br>
loss_fuse &lt; 0 why?<br>
thank you very much</p>

tom: <aside class="quote no-group" data-username="Cverlpeng" data-post="3" data-topic="51036">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cverlpeng/48/13070_2.png" class="avatar"> Cverlpeng:</div>
<blockquote>
<p>Is that right? Are the two functions the same?</p>
</blockquote>
</aside>
<p>It doesn’t have the side-effect of modifing its input, you could compare results.</p>
<pre><code class="lang-auto">loss_fuse = nn.KLDivLoss().forward(out_seg_sm[:,1:,:,:], out_dt_norm.clone())
loss_fuse &lt; 0 why?
</code></pre>
<p>The typical way to get negative kl_div is because your arguments are not probability distributions, and indeed <code>out_dt_norm</code> has little reason to be. You could do something like</p>
<pre><code class="lang-auto">dt_out_unnormalized = dt_out.clamp(min, max)
dt_out = dt_out_unnormalized. / dt_out_unnormalized.sum((2, 3))
</code></pre>
<p>with <code>(2, 3)</code> replaced by whatever dimensons you want the normalization to cover.</p>
<p>(You should be using <code>torch.nn.functional.kl_div</code>, too, and don’t need the clone.)</p>
<p>I must admit I don’t know what you are trying to achieve in terms of modelling and so I cannot say if it is mathematically sound or not.</p>
<p>Best regards</p>
<p>Thomas</p>

Cverlpeng: <p>hi, Thomas V<br>
thanks</p>
<pre><code class="lang-python">class SegmentationMultiLosses(nn.CrossEntropyLoss):
    """2D Cross Entropy Loss with Multi-L1oss"""
    def __init__(self, nclass=-1, weight=None,size_average=True, ignore_index=-1):
        super(SegmentationMultiLosses, self).__init__(weight, size_average, ignore_index)
        self.nclass = nclass


    def forward(self, *inputs):
        
        out, target_img, target_exist, target_dt= tuple(inputs)
       
        out_img, out_dt_img, out_exist = out
       
        loss_seg = super(SegmentationMultiLosses, self).forward(out_img, target_img)
        loss_exist = nn.BCELoss().forward(out_exist, target_exist)
        loss_dt = nn.MSELoss().forward(out_dt_img, target_dt)

        out_dt_norm = self.dt_norm(out_dt_img)
        out_seg_sm  = F.softmax(out_img, dim=1)
        loss_fuse = nn.KLDivLoss().forward(out_seg_sm[:,1:,:,:], out_dt_norm)

        print('loss_seg: {}  loss_exit: {}  loss_dt: {} loss_fuse: {} '.format(loss_seg.item(), 
                                loss_exist.item()*0.1, loss_dt.item(), loss_fuse.item()))
        loss = loss_seg + loss_dt + 0.1*loss_exist + loss_fuse
        return loss
    def dt_norm(self, dt_out):
        max = 10
        min = 0
        dt_out = dt_out.clamp(min, max) / max
        return dt_out
</code></pre>
<p>it work, but I get loss_fuse(kl loss) &lt; 0</p>
<p>I want to segmentation a target. One branch of the network is trained using the cross entropy loss function. The other branch is a regression branch that predicts the weight of the target. Then I normalize this weight and guide the segmentation branch through klloss.</p>
<p>out_seg_sm[:,1,:,:]  : output of softmax<br>
out_dt_norm: normalize  [0-1]  ，Indicates the importance of each pixel on the target</p>
<p>thank you very much</p>

tom: <p>Ah, sorry the input must be <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.KLDivLoss" rel="nofollow noopener">log probs and the targets probs</a>. So logsumexp(inputs, -1) should be 0 and sum(targets, -1) should be 1. I would recommend checking that for a few cases.</p>
<p>Best regards</p>
<p>Thomas</p>

Cverlpeng: <p>hi,<br>
I don’t understand much，Why is the kl_div() input and target sum must be equal to 0, the sum on the entire feature map is  0, or the sum of a pixel in the feature map in the different channels is 0 ?<br>
thanks</p>

Cverlpeng: <p>hi,<br>
out_dt_img : is the discrete value of the regression branch prediction. Should I turn it into a probability distribution like this? Is there any pytorch function that can directly normalize a 4D tensor [b,c,h,w] in the [b,c,:,:] dimension.That is, normalize the pixels on each channel</p>
<p>thanks</p>

tom: <p>I think you want <code>x = x.reshape(n, c ,h * w).(log_)softmax(-1).view(n, c, h, w)</code> if you want logsoftmax or <code>x = x / x.sum((-1, -2), keepdim=True)</code> if you have positive values and want to normalize.</p>
<p>Best regards</p>
<p>Thomas</p>