About Dice loss, Generalized Dice loss

banikr: <p>Hello All,<br>
I am running multi-label segmentation of 3D data(<code>batch x classes x H x W x D</code>). The target is 1-hot encoded[all 0s and 1s].<br>
I have broad questions about the loss to be used.<br>
In V-net paper: <a href="https://arxiv.org/pdf/1606.04797.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1606.04797.pdf</a><br>
The dice loss used as:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e023d216f0d8c83ae3b7dcc6c880f36c3877482d.png" alt="image" data-base62-sha1="vYPHkaGKy4J0vBhmr2Eg9jplMM5" width="223" height="75"><br>
which varies from 0~1, and intended to maximize. Like regular IoU/Dice.<br>
In Generalized dice paper: <a href="https://arxiv.org/pdf/1707.03237.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1707.03237.pdf</a><br>
GDL loss is:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/8/58155759df1378c82e379f6ec7cfa36b347640c2.png" alt="image" data-base62-sha1="czdJVpxVi1Emxvg3CeZtFHrNYSS" width="341" height="86"><br>
and the author says about the weight:</p>
<blockquote>
<p>when choosing the GDLv weighting, the contribution of each label is<br>
corrected by the inverse of its volume, thus reducing the well-known correlation<br>
between region size and Dice score</p>
</blockquote>
<p>which I understand very well.</p>
<p>So, when I implement both losses with the following code from: <a href="https://github.com/rogertrullo/pytorch/blob/rogertrullo-dice_loss/torch/nn/functional.py#L708" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/nn/functional.py at rogertrullo-dice_loss · rogertrullo/pytorch · GitHub</a></p>
<pre><code class="lang-auto">torch.manual_seed(1001)
out = Variable(torch.randn(3, 9, 64, 64, 64)) 
print &gt;&gt; tensor(5.2134) tensor(-5.4812)
seg = Variable(torch.randint(0,2,[3,9,64,64, 64])) #target is in 1-hot-encoded format

def dice_loss(prediction, target, epsilon=1e-6):
    """
    prediction is a torch variable of size BatchxnclassesxHxW representing log probabilities for each class
    target is a 1-hot representation of the groundtruth, shoud have same size as the prediction
    """
    assert prediction.size() == target.size(), "prediction sizes must be equal."
    assert prediction.dim() == 5, "prediction must be a 4D Tensor."
    uniques = np.unique(target.numpy())
    assert set(list(uniques)) &lt;= set([0, 1]), "target must only contain zeros and ones"

    probs = F.softmax(prediction, dim=1) # channel/classwise
   
    num = probs * target # b,c,h,w--p*g
    num = torch.sum(num, dim=4)
    num = torch.sum(num, dim=3)
    num = torch.sum(num, dim=2) # b,c
    
    den1 = probs * probs # --p^2
    den1 = torch.sum(den1, dim=4) #b,c,h
    den1 = torch.sum(den1, dim=3) #b,c,h
    den1 = torch.sum(den1, dim=2) 
  
    
    den2 = target * target # --g^2
    den2 = torch.sum(den2, dim=4)
   
    den2 = torch.sum(den2, dim=3)
    den2 = torch.sum(den2, dim=2) 
    den = (den1+den2+epsilon)#.clamp(min=epsilon)
    dice=2*(num/den)
    dice_eso=dice #[:,1:]#we ignore background dice val, and take the foreground
    dice_total=torch.sum(dice_eso)/dice_eso.size(0)#divide by batch_sz

    return dice_total

dice_loss(prediction=out, target=seg)
</code></pre>
<p>I get a dice score of <code>1.9096</code> why is it beyond 1 and when using it as loss criterion during training, what limit it has to be maximized?</p>
<p>And when using GDL from: <a href="https://github.com/wolny/pytorch-3dunet/blob/eafaa5f830eebfb6dbc4e570d1a4c6b6e25f2a1e/pytorch3dunet/unet3d/losses.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch-3dunet/pytorch3dunet/unet3d/losses.py at eafaa5f830eebfb6dbc4e570d1a4c6b6e25f2a1e · wolny/pytorch-3dunet · GitHub</a></p>
<pre><code class="lang-auto">class GeneralizedDiceLoss(_AbstractDiceLoss):
    """Computes Generalized Dice Loss (GDL) as described in https://arxiv.org/pdf/1707.03237.pdf.
    """

    def __init__(self, normalization='sigmoid', epsilon=1e-6):
        super().__init__(weight=None, normalization=normalization)
        self.epsilon = epsilon

    def dice(self, prediction, target, weight):
        assert prediction.size() == target.size(), "'prediction' and 'target' must have the same shape"
        prediction = flatten(prediction) #flatten all dimensions except channel/class
        target = flatten(target)
        target = target.float()

        if prediction.size(0) == 1:
            # for GDL to make sense we need at least 2 channels (see https://arxiv.org/pdf/1707.03237.pdf)
            # put foreground and background voxels in separate channels
            prediction = torch.cat((prediction, 1 - prediction), dim=0)
            target = torch.cat((target, 1 - target), dim=0)
        w_l = target.sum(-1)
        w_l = 1 / (w_l * w_l).clamp(min=self.epsilon)
        w_l.requires_grad = False

        intersect = (prediction * target).sum(-1)
        print(intersect.shape)
        intersect = intersect * w_l

        denominator = (prediction + target).sum(-1)
        print(denominator)
        denominator = (denominator * w_l).clamp(min=self.epsilon)

        return 1 - (2 * (intersect.sum() / denominator.sum()))

GeneralizedDiceLoss(normalization='softmax').dice(prediction=out, target=seg, weight=None)
</code></pre>
<p>I get a dice score with the same <code>1.0013</code></p>
<p>Though I don’t know how GDL varies compared to the V-net Dice but can they be compared to some extent in terms of optimization?</p>
<p>Also, how do I use this in training?</p>
<p>For example,</p>
<pre><code class="lang-auto">            img, seg = img.to(device), seg.to(device)
            out = model(img)
            optimizer.zero_grad()
            loss = dice_loss_3d(out, seg).cuda()
            # print(loss)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()
</code></pre>
<p>should work?</p>

tkepp: <p>The generalized Dice loss is implemented in the MONAI framework. Take a look here: <a href="https://docs.monai.io/en/stable/_modules/monai/losses/dice.html#GeneralizedDiceLoss" class="inline-onebox" rel="noopener nofollow ugc">monai.losses.dice — MONAI 1.2.0 Documentation</a></p>