Year: 2021

Abstract:
Most segmentation losses are arguably variants of the Cross-Entropy (CE) or Dice losses. In the abundant segmentation literature, there is no clear consensus as to which of these losses is a better choice, with varying performances for each across different benchmarks and applications. In this work, we develop a theoretical analysis that links these two types of losses, exposing their advantages and weaknesses. First, we provide a constrained-optimization perspective showing that CE and Dice share a much deeper connection than previously thought: They both decompose into label-marginal penalties and closely related ground-truth matching penalties. Then, we provide bound relationships and an informationtheoretic analysis, which uncover hidden label-marginal biases: Dice has an intrinsic bias towards specific extremely imbalanced solutions, whereas CE implicitly encourages the ground-truth region proportions. Our theoretical results explain the wide experimental evidence in the medical-imaging literature, whereby Dice losses bring improvements for imbalanced segmentation. It also explains why CE dominates natural-image problems with diverse class proportions, in which case Dice might have difficulty adapting to different label-marginal distributions. Based on our theoretical analysis, we propose a principled and simple solution, which enables to control explicitly the label-marginal bias. Our loss integrates CE with explicit L1 regularization, which encourages label marginals to match target class proportions, thereby mitigating class imbalance but without losing generality. Comprehensive experiments and ablation studies over different losses and applications validate our theoretical analysis, as well as the effectiveness of our explicit label-marginal regularizers.

heading: 1 INTRODUCTION
text: Semantic segmentation is one of the most investigated problems in computer vision, and has been impacting a breadth of applications, from natural-scene understanding (Cordts et al., 2016; Kirillov et al., 2019) to medical image analysis (Litjens et al., 2017; Dolz et al., 2018). In the recent years, deep learning methods have dominated the field, as a result of the great capacity of Convolutional Neural Networks (CNN) (He et al., 2016) to automatically learn representations from large-scale data sets (Long et al., 2015; Ronneberger et al., 2015; Zhao et al., 2017; Chen et al., 2018; Yuan et al., 2020). Semantic segmentation is often stated as a pixel-wise classification task, following the optimization of a loss function expressed with summations over the ground-truth regions, as in the standard Cross-Entropy (CE) loss. A relevant aspect of segmentation problems is class imbalance, i.e., unequal proportions of the segmentation regions, which may cause large-region terms in the objective to completely dominate small-region ones. A representative example is the popular Cityscapes dataset (Cordts et al., 2016), where the average proportions of some classes, such as motorcycle or bicycle, are below 1%. In these scenarios, besides specifically designed CNN architectures or training schemes (Tao et al., 2020; Bao et al., 2021), the loss function to be minimized during learning plays a critical role, and has triggered a large body of research works in the last years (Lin et al., 2017; Wong et al., 2018; Kervadec et al., 2021b; Milletari et al., 2016; Sudre et al., 2017; Kervadec et al., 2021a).
There exists a great diversity of loss functions for image segmentation, which can be categorized into two main families, and are arguably variants of CE, the Dice loss (Ma et al., 2021; Yeung et al., 2021), or combinations of both (Wong et al., 2018; Taghanaki et al., 2019). The first family is mo-
tivated by distribution measures, i.e., CE and its variants, and is directly adapted from classification tasks. To deal with class imbalance, various extensions of CE have been investigated, such as increasing the relative weights for minority classes (Ronneberger et al., 2015), or modifying the loss so as to account for performance indicators during training, as in the popular Focal loss (Lin et al., 2017) or TopK loss (Wu et al., 2016). The second main family of losses is inspired by geometrical metrics. In this category, the most popular losses are linear Dice (Milletari et al., 2016) and its extensions, such as the logarithmic (Wong et al., 2018) or generalized (Sudre et al., 2017) Dice loss. Borrowing the idea of the weighted CE, the latter introduces class weights to increase the contributions of the minority classes. These loss functions are motivated by the geometric Dice coefficient, which measures the overlap between the ground-truth and predicted segmentation regions.
In the literature, to our knowledge, there is no clear consensus as to which category of losses is better, with the performances of each varying across data sets and applications. It has been empirically argued that the Dice loss and its variants are more appropriate for extreme class imbalance, and such empirical observations are the main motivation behind the wide use and popularity of Dice in medical-imaging applications (Milletari et al., 2016; Jha et al., 2020). CE, however, dominates most recent models in the context of natural images (Chen et al., 2018; Zhao et al., 2017; 2018; Yuan et al., 2020). Therefore, beyond experimental evidence, there is a need for a theoretical analysis that clarifies which segmentation loss to adopt for a given task, a decision that affects performance significantly.
On the surface, these two categories of losses (i.e., geometry-based vs. distribution-based) seem unrelated. Furthermore, an important body of work in the literature suggests that CE and Dice are complementary, which has motivated composite losses integrating both, e.g., (Wong et al., 2018; Taghanaki et al., 2019). Such composite losses perform very competitively in extremely imbalanced segmentation, as shown by (Wong et al., 2018; Taghanaki et al., 2019), among several other recent works (Isensee et al., 2018; Ma et al., 2021).
In this paper, we provide a constrained-optimization perspective showing that, in fact, CE and Dice share a much deeper connection than previously thought: They both decompose into label-marginal penalties and closely related ground-truth matching penalties. Our theoretical analysis highlights encoded hidden label-marginal biases in Dice and CE, and shows that the main difference between the two types of losses lies essentially in those label-marginal biases: Dice has an intrinsic bias preferring very small regions, while CE implicitly encourages the right (ground-truth) region proportions. Our results explain the wide experimental evidence in the medical-imaging literature, whereby using or adding Dice losses brings improvements for imbalanced segmentation with extremely small regions. It also explains why CE dominates natural-image problems with diverse class proportions, in which case Dice might have difficulty adapting to different label-marginal distributions (see Table
3 and the examples depicted in Fig. 1). Based on our theoretical analysis, we propose principled and simple loss functions, which enable to control explicitly the label-marginal bias. Our solution integrates the benefits of both categories of losses, mitigating class imbalance but without losing generality, as shown in the examples in Fig. 1.
Our contributions are summarized as follows:
• Showing through an explicit bound relationship (Proposition 1) that the Dice loss has a hidden label-marginal bias towards specific extremely imbalanced solutions, preferring small structures, while losing the flexibility to deal effectively with arbitrary class proportions.
• Providing an information-theoretic perspective of CE, via Monte-Carlo approximation of the entropy of the learned features (Proposition 2). This highlights a hidden label-marginal bias of CE, which encourages the proportions of the predicted segmentation regions to match the ground-truth proportions.
• Introducing new loss functions to control label-marginal biases: Our losses integrate CE with explicit regularization terms based on L1 or the KL divergence, which encourage label marginals to match target class proportions.
• Comprehensive experiments and ablation studies over different losses and applications, including natural and medical-imaging data, validate our theoretical analysis, as well as the effectiveness of our explicit label-marginal regularizers.

heading: 2 FORMULATION
text: Semantic segmentation is often stated as a pixel-wise classification task, following the optimization of a loss function for training a deep network. In Table 1, we present the notations, formulations and approximations used in our subsequent discussions. Besides the basic notations of the task (such as networks predictions), we explicitly include the loss functions, label-marginal regularizers and information-theoretic quantities that will be discussed in the following sections. We note that, to facilitate the reading of our analysis, we write the CE and Dice losses in a non-standard way using summations over the ground-truth segmentation regions, rather than as functions of the labels. Also, while we provide the CE loss for all segmentation regions, we give Dice for a single region. This is to accommodate two variants of the Dice loss in the literature: in the binary case, Dice is typically
used for the foreground region only (Milletari et al., 2016); in the multi-region case, it is commonly used over all the regions (Wong et al., 2018). Finally, to simplify notation, we give all the loss functions for a single training image, without summations over all training samples (as this does not lead to any ambiguity, neither does it alter the analysis hereafter). In the training iterations, we use the mean values across all the training samples via standard mini-batch optimization.

heading: 2.1 DEFINITION OF LABEL-MARGINAL BIASES AND PENALTY FUNCTIONS
text: In the following, we analyse the label-marginal biases inherent to CE and Dice losses, and show that the main difference between the two types of losses lies essentially in those label-marginal biases. To do so, we provide a constrained-optimization perspective of the losses: We define a label-marginal bias as a soft penalty function for the hard equality constraint p = t, where t is a given (fixed) target distribution. Such a penalty encourages the predicted label-marginal p to match a given target distribution t. In the general context of constrained optimization, penalty functions are widely used (Bertsekas, 1995). In general, penalty methods replace equality constraints of the form p = t by adding a term g(p) to the main objective being minimized. Such a penalty function g increases when p deviates from target t. By definition, for the constraint p = t, with the domain of p being probability simplex ∆K , a penalty g(p) is a continuous and differentiable function, which reaches its global minimum when the constraint is satisfied, i.e., it verifies: g(t) ≤ g(p)∀p ∈ ∆K .

heading: 2.2 THE LINK BETWEEN CROSS ENTROPY AND DICE
text: To ease the discussion in what follows, we will start by analyzing the link between CE and the logarithmic Dice, along with the label-marginal bias of the latter (Proposition 1). Then, we discuss a bounding relationship between the different Dice variants. Finally, we will provide an informationtheoretic analysis, which highlights the hidden label-marginal bias of CE (Proposition 2).
Let us consider the logarithmic Dice loss in the multi-class case. This loss decomposes (up to a constant) into two terms, a ground-truth matching term and a label-marginal bias:
− K∑
k=1
log(Dicek) c = − K∑ k=1 log  1 |Ωk| ∑ i∈Ωk pik  ︸ ︷︷ ︸
Ground-truth matching: DF
+ K∑ k=1
log (p̂k + ŷk)︸ ︷︷ ︸ Label-marginal bias: DB
(1)
where c= stands for equality up to an additive and/or non-negative multiplicative constant. The ground-truth matching term in Eq. (1) is a lower bound on the cross-entropy loss (CE) due to Jensen’s inequality and the convexity of function − log(x): DF ≤ CE. Therefore, minimizing CE could be viewed as a proxy for minimizing term DF that appears in the logarithmic Dice. In fact, from a constrained-optimization perspective, DF and CE are very closely related and could be viewed as two different penalty functions enforcing the same equality constraints: pik = 1, ∀i ∈ Ωk, ∀k. Both DF and CE are monotonically decreasing functions of each softmax and reach their global minimum when these equality constraints are satisfied. Therefore, they encourage softmax predictions pik for each region Ωk to reach their target ground-truth values of 1. Of course, this does not mean that penalties CE and DF yield exactly the same results. The difference in the results that they may yield is due to the optimization technique (e.g., different gradient dynamics in the standard training of deep networks as the penalty functions have different forms).

heading: 2.3 THE HIDDEN LABEL-MARGINAL BIAS OF DICE
text: The following proposition highlights how the label-marginal term DB in Eq. (1) encourages specific extremely imbalanced solutions. Proposition 1. Let t = ( t̂j ) 1≤j≤K ∈ {0, 1} K denote the simplex vertex verifying: t̂j = 1 when ŷj = max1≤k≤K ŷk and t̂j = 0 otherwise. For variables p = (p̂k)1≤k≤K and fixed distribution y = (ŷk)1≤k≤K , the label-marginal term in Eq. (1) reaches its minimum over the simplex at t:
K∑ k=1 log ( t̂k + ŷk ) ≤ K∑ k=1 log (p̂k + ŷk) ∀p ∈ ∆K (2)
Proof. The details of the proof are deferred to Appendix A.1. The main technical ingredient is based on Jensen’s inequality and the concavity of penalty DB with respect to simplex variables p.
Inequality (2) means that the label-marginal term in Dice in Eq. (1) is a penalty function for constraint p = t, where t is the simplex vertex given in Proposition 1. Therefore, it encourages extremely imbalanced segmentations, where a specific region includes all the pixels and the remaining regions are empty. All in all, the logarithmic Dice loss integrates a hidden label-marginal prior preferring extremely imbalanced segmentations, which is optimized jointly with a ground-truth matching term similar to CE. It is worth noting that, in the two-class (binary) segmentation case, Dice might be used for the foreground region only, as in the popular work in (Milletari et al., 2016), for instance. Similarly to the multi-class case discussed above, a single Dice term also decomposes into a ground-truth matching term and label-marginal penalty, with the latter encouraging extremely imbalanced binary segmentations. We provide more details for this case in Appendix B.

heading: 2.4 ON THE LINK BETWEEN THE DIFFERENT VARIANTS OF DICE
text: The label-marginal analysis we discussed above is based on the standard logarithmic Dice loss. Here, we argue that both logarithmic and linear Dice are very closely related and, hence, the linear Dice also hides a class-imbalance bias. In fact, from a constrained-optimization perspective, the two losses could be viewed as different penalty functions for imposing constraints: Dicek = 1∀k. Both functions – log(x) and (1 − x) are monotonically decreasing in [0, 1] and achieve their minimum in [0, 1] at x = 1. Furthermore, the logarithmic Dice is an upper bound on the linear one. This follows directly from: − log (t) ≥ 1 − t ∀t > 0. Of course, this does not mean that optimizing these two variants leads to exactly the same results. The differences in their results might be due to optimization (i.e., different gradient dynamics stemming from logarithmic and linear penalties).

heading: 2.5 THE HIDDEN LABEL-MARGINAL BIAS OF CE
text: In the following, we give an information-theoretic perspective of CE, via a generative view of network predictions and a Monte-Carlo approximation of the entropy of the learned features given the labels. This highlights a hidden label-marginal bias of CE, which encourages the proportions of the predicted segmentation regions to match the ground-truth proportions.
Proposition 2. Let F and K denote the random variables associated with the learned features and the labels, respectively, and H(F|K) the conditional entropy of learned features given the labels, estimated via Monte-Carlo :
H(F|K) ≈ K∑ k ykH(F|K = k) ≈ − 1 |Ω| K∑ k ∑ i∈Ωk log(P(fθi |k)) (3)
where H(F|K = k) is the empirical estimate of the conditional entropy of features given a specific class k (expression in Table 1) and P(fθi |k) denotes the probability of the learned features given class k. We have the following generative view of CE:
CE c= H(F|K)︸ ︷︷ ︸ Ground-truth matching + DKL(y||p)︸ ︷︷ ︸ Label-marginal bias
(4)
The detailed proof is deferred to Appendix A.2. The approximation of H(F|K = k) in the second line of Eq. (3) is based on the well-known Monte-Carlo estimation (Kearns et al., 1997; Tang et al., 2019). Then the relationship in Eq. (4) follows from Eq. (3), after some manipulations, using Bayes rule P(fθi |k) ∝ pik p̂k and ∑ i∈Ωk log(p̂k) = |Ωk| log(p̂k).
This information-theoretic view of CE shows that the latter has an implicit (hidden) label-marginal bias towards the ground-truth region proportions (the KL term). This bias competes with the entropy term, which encourages low uncertainty (variations) within each ground-truth segmentation region Ωk. The entropy term could be viewed as a ground-truth matching term: it reaches its global minima when the feature embedding is constant within each region. If used alone, the entropy term may lead to trivial imbalanced solutions. The label-marginal KL term avoids such trivial solutions by matching the ground-truth class proportions. Note that there is no mechanism in CE to control the relative contributions of those two competing terms as they are implicit in CE.

heading: 2.6 OUR SOLUTION
text: Our analysis shows that Dice, CE and their combinations, e.g., CE − log(Dice), are closely related and enforce two types of competing constraints : ground-truth matching and label-marginal constraints. However, there is no clear consensus in the literature as to which loss is better, with the performances of each varying across data sets and applications. This variability in performances could be explained by two fundamental factors:
• The difference in the label-marginal prior. The label-marginal priors are different as Dice has an intrinsic bias preferring very small regions, while CE encourages the right (ground-truth) region proportions. This might explain the wide experimental evidence in the medical imaging literature, where using or adding Dice losses brings improvements for imbalanced segmentation with extremely small regions.
• Weighting the contribution of the bias term. Our analysis suggests that CE should be preferred over Dice in all cases and applications (both balanced/imbalanced segmentation, or segmentation problems with high variability in region proportions) as it promotes the right label-marginal distribution. While this seems to be widely the case in natural image segmentation, where Dice is uncommon, the extensive experimental evidence in the medical-image segmentation literature suggests otherwise, especially in extremely imbalanced problems. We argue that this is due to the relative contribution of the label-marginal term in the overall objective. Controlling such label-marginal contribution is very important in imbalanced problems. In particular, it mitigates the difficulty that the ground-truth matching terms differ by several orders of magnitude across regions, as in CE, which causes large-region terms to completely dominate small-region ones. This analysis also resonates with the fact that combo losses such as CE - λ log (Dice) perform very competitively in imbalanced segmentation, as shown by (Wong et al., 2018; Taghanaki et al., 2019), among several other recent works. In this case, controlling the relative contribution of each of these terms indirectly controls the weight of the label-marginal bias. Note that such control is not possible when using CE alone or Dice alone, as the label-marginal biases in these losses are hidden (implicit).
We propose a principled and simple solution, which enables to control explicitly the labelmarginal bias, via regularization losses that encourage the correct class proportions and are used in conjunction with CE :
CE + λR(y; p) (5)
Our label-marginal regularizers increase the contribution of the minority classes in imbalanced problems, but, unlike Dice, do not lose adaptability to problems with various class proportions. Our extensive experiments and ablation studies over different losses and applications demonstrate the effectiveness of our explicit label-marginal regularizers. We investigate different forms of regularization, including the L1 norm, i.e.,R(y; p) = L1(y,p), and the KL divergence, i.e.,R(y; p) = DKL(y||p); see Table 1 for the expressions of DKL and L1. In Fig. 2, we depict our different regularizers as functions of the label-marginal distribution for a binary-segmentation case, with the
foreground-region proportion set to 0.1, along with the bias terms in Dice. While our DKL and L1 regularizers may deliver comparable performances (see the experimental section), L1 might be a better option for extremely imbalanced segmentations, due to its gradient properties and stability at the vicinity of 0, i.e., when the label-marginal probability p̂1 is close to 0. Notice that, at the vicinity of zero, both first and second derivatives of the regularizer are unbounded for DKL, but bounded and constant for L1. Our experiments on imbalanced medical image segmentation confirm the effectiveness of the L1 regularizer.

heading: 3 EXPERIMENTS
text: Datasets. We report results on both medical and natural image segmentation benchmarks, i.e., Retinal Lesions (Wei et al., 2020) and Cityscapes (Cordts et al., 2016). Retinal Lesions is a large collection of color fundus images, as shown in Fig. 3 and Fig. G.1. It is noted that a panel of 45 experienced ophthalmologist was formed to label this dataset and each image was assigned to at least three annotators to get trustworthy pixel-level annotations(Wei et al., 2020). In our experiments, we employ its public version1, containing 1, 593 samples, and conduct our experiments in the binary setting to segment the lesion region versus background. The dataset is randomly divided into training (70%), validation (10%) and test (20%) set, whose images are resized to 512 × 512. We tune the hyper-parameters of each model on the validation set, and report the results on the test set. The natural scene dataset, Cityscapes, is a large-scale dataset with high quality pixel-level annotations of 5, 000 images across 19 categories, containing both stuff and objects with high variation in the class proportions distribution. We use the official data split, which contains 2, 975 samples for training, 500 samples for validation and 1, 512 samples for testing. All the input images are resized to 512× 1024 for training, and 1024 × 2048 for testing. Fig. C.1 in Appendix C shows the distribution of region proportions for Retinal Lesions and the 19 classes in Cityscapes, highlighting the challenging class imbalance and high diversity on target sizes.
Backbones. It is worth stressing that the theoretical discussion in this paper is model-agnostic. Thus, in our experiments we employ Res34-FPN, Res50-FPN (Lin et al., 2017) and Res34-Unet, Res50-Unet (Ronneberger et al., 2015) on Retinal Lesions, while we show results with Res50-FPN and Res101-FPN on Cityscapes, whose implementations are publicly available2.
Training. The parameters of the encoder are initialized with the pre-trained weights on Imagenet, while those in the decoder head are randomly initialized. On Retinal Lesions, we train the model during 60 epochs, with a batch size of 8, via the Adam optimizer. The initial learning rate is set to 1e-4, and halved if the loss on the validation set does not decrease within 5 epochs. On Citycapes, the main setting is adopted from the state-of-the-art library3. Specifically, the SGD optimizer is used for training, with an initial learning rate set to 0.01 and a batch size of 8. During the 100 training epochs, we use an iteration-wise polynomial strategy to linearly scale the learning rate down to a minimum of 1e-4. For all the models, we tune the hyper-parameters on the validation and report the best results on the corresponding test set. Conventional data augmentations, such as random cropping, mirror flipping and changes in brightness, are employed on both datasets.
Losses. We evaluate the proposed loss function in Eq. (5) with two regularizing terms, i.e., DKL and L1. In our implementation, we use a modified soft-max function with a temperature parameter when computing the predicted region proportion, as this enables a better estimate of the actual region proportion (refer to Appendix D for details). We compare our results with the following baselines: CE, Focal loss (Lin et al., 2017), Dice related losses, i.e., linear/logarithmic/generalized Dice, and the composite loss combining CE and logarithmic Dice.
Evaluation metrics. On Retinal Lesions, we use two standard metrics in medical image segmentation, i.e., Dice Similarity Coefficient (DSC) and modified Hausdorff Distance (HD-95). Particularly, HD-95 represents the 95th percentile of the symmetric Hausdorff Distance (HD) between the binary objects in two images. On Cityscapes, we resort to the standard Intersection-Over-Union (IoU) score, which is widely employed in natural image segmentation.
Results on Retinal Lesions. Table 2 reports the quantitative comparison between the proposed loss, with the two different regularizers, and baselines on the Retinal Lesions dataset. Regarless of the backbone network, the proposed CE+L1 loss consistently outperforms all the others with a DSC score of 54.5% for Res50-FPN and 54.3% for Res50-Unet. Compared to CE, our best model, i.e., L1 as the label-marginal regularizer, brings nearly 3.0% improvement in terms of DSC and between 5.6% and 8.6% in terms of HD-95, depending on the backbone. Even though these differences are reduced with respect to the best baseline, this gap is still significant, particularly in terms of HD-95. It is noteworthy to mention that while DSC is more sensitive to the internal filling of the target region, the HD-95 is more sensitive to the segmentation boundary. Thus, the proposed loss is more effective
1https://github.com/WeiQijie/retinal-lesions 2https://github.com/qubvel/segmentation_models.pytorch 3https://github.com/open-mmlab/mmsegmentation
to predict better boundaries than existing losses. Furthermore, as shown in Fig. C.1 in Appendix C, the average region proportion is considerably low, i.e., 3.8%, and varies significantly (the standard deviation is 6.6% with a maximum of 57.8%). Under this highly heterogeneous scenario in terms of class region proportions, the proposed losses present a more stable performance across different runs and backbones, which is reflected in their lower variances. This can be explained by their better adaptability to different target sizes and better gradient dynamics at the vicinity of label marginal p̂k = 0 (as shown in Fig. 2). Another interesting finding is that, by combining CE with the bias term of Dice (CE + DB1), we obtain results close to combo loss CE − log(Dice1)(Wong et al., 2018). This validates our theoretical insight that the fundamental difference between CE and Dice lies in their distinct hidden biases.
On the balancing weight in the composite losses. In this section we evaluate the impact of the balancing weight λ of the proposed loss in Eq. (5), which balances the effects of the regularization term, as well as the balancing weight in CE− log(Dice1). In our experiments, we empirically found that the best λ values fordifferentpenaltyterms are : 1.0 for CE+L1, 0.1 for CE+DKL, 0.01 for CE + DB1. These values are consistent with the given arguments in Sec. 2.6, which emphasize that L1 has a better gradient evolution for large learning rates and relatively high weighting, especially at the beginning of training. Regarding the widely used combo loss of CE− log(Dice1), setting its balancing weight to 1.0 yielded consistent performance across the datasets and networks we used. Details of this experimental study can be found in Appendix E. Therefore, with the same hyperparameter budget, CE + L1 performed better than all the other related composite loss functions. Note that we use the best empirical values of λ found on Retinal Lesions for the experiments on Cityscapes.
Qualitative results on the Retinal Lesions dataset. As shown in Fig. 3, we can observe that the model trained with Dice variants (third and fourth column) tends to undersegment large and medium target regions, while ignoring several small lesions and generating false positives. In contrast, the proposed solution enables a better trade-off between finding small regions, reducing the number of false positives and matching the size of the larger targets. Furthermore, one can notice that our loss yields better consistencies with the target region proportions than CE.
Results on Cityscapes. Table 3 reports the comparative per-class IoU and mean IoU (mIoU) on the validation set of Cityscapes with two network architectures. First, we can observe that in this multi-class dataset, regardless of the network, the proposed learning objectives outperform all the evaluated losses in terms of mIoU. Then, by investigating the relationship between mRegProp (second row in Table 3) and the corresponding segmentation performance across small region classes, we can observe that Dice-related losses have a hidden label-marginal bias towards extremely imbalanced solutions, preferring small structures. In particular, the linear Dice often obtains the highest mIoU for the smallest structures. This bias comes at the cost of less flexibility when dealing with arbitrary class proportions, which is reflected in its poor average mIoU (right column).
Table 4: mIoU on Cityscapes test set.
Loss Res50-FPN Res101-FPN
CE 66.97 69.57 Focal loss 64.50 66.79 − log(Dice) 59.77 62.62 CE− log(Dice) 67.08 69.58 CE +DKL (Ours) 68.35 70.07 CE + L1 (Ours) 68.35 70.11
Quantitative evaluation on the Cityscapes test set is reported in Table 4. The observations in this table are consistent with the results on the validation set. Similarly to the validation set, our method achieves better results on both architectures. For the per-class scores on the test set, please refer to Appendix F.
It is noteworthy to highlight that both linear Dice and logarithmic Dice perform relatively poorly on Cityscapes, on average, empirically showing why Dice is rarely adopted in natural-image segmentation tasks. As we mentioned earlier, this might be due to its inherent label-marginal bias, which is inappropriate for segmenting regions with arbitrary class proportions. Furthermore, our simple solution yielded an improvement over CE by nearly 1.5%, regardless of the backbone. These empirical observations suggest that the proposed formulation in Eq. (5) results in better region-proportion guidance and training stability than existing segmentation losses. Finally, we can observe that employing L1 as regularizer consistently results in better performance than using the DKL term. Hence, L1 might be a better option in extremely imbalanced segmentations, due to its gradient properties and stability when the label-marginal probability is close to 0.

heading: 4 CONCLUSION
text: We provided a detailed theoretical analysis of the two most popular semantic segmentation losses, i.e., Cross-entropy and Dice, which revealed non-obvious bounding relationships and hidden labelmarginal biases, suggesting that CE is a better option in general. Then, we showed how both loss functions could be written within a common formulation, containing a ground-truth matching term and a label-marginal bias. The implicit bias in Dice prefers small regions, improving its performance in highly imbalanced conditions, as in medical-imaging applications. The bias hidden in CE encourages the ground-truth region proportion, which makes it a generally better option in complex scenarios with diverse class proportions. Furthermore, we proposed a principled solution, which enables to control the label-marginal bias via L1 and KL regularizers that encourage the target class proportions, while improving training stability. Our flexible formulation enables the minority classes to have better influence on training, without losing adaptability to medium-to-large regions. Extensive experiments on a natural and medical imaging datasets validate the theoretical analysis in this paper, as well as the effectiveness of the presented solution.

heading: A PROOFS
text: A.1 PROPOSITION 1
Proof. Let us write the label-marginal bias term in the logarithmic Dice as a vector-valued function of probability simplex vector p:
g(p) = 1>K log(p + y) for p = (p̂k)1≤k≤K ∈ ∆K (6)
where symbol > denotes transpose and 1K is the K-dimensional vector of ones. Function g is concave because its Hessian is a negative semi-definite matrix: The Hessian of g is a diagonal matrix whose diagonal elements are given by− 1
p̂2k and, hence, are all non-positive. Therefore, using
Jensen’s inequality and the fact that p is within the simplex, we have the following lower bound on penalty g in Eq. (6):
g(p) = g ( K∑ k=1 p̂kek ) ≥ K∑ k=1 p̂kg(ek) (7)
where ek ∈ {0, 1}K denote the k-th vertex of the simplex: the k-th component of ek is equal to 1 while the other components are all equal to 0. Now, recall the definition of simplex vector t = ( t̂j ) 1≤j≤K : t̂j = 1 when ŷj = max1≤k≤K ŷk and t̂j = 0 otherwise. Given this definition, one could easily verify the following fact:
g(ek) ≥ g(t) ∀k (8)
To see this, let j denotes the integer verifying ŷj = max1≤k≤K ŷk and let k 6= j. Then, we have: g(ek)− g(t) = log ( 1 + 1
ŷk
) − log ( 1 + 1
ŷj
) ≥ 0. (9)
This is due to the fact that function log ( 1 + 1x ) is monotonically decreasing in [0, 1] and ŷk ≤ ŷj .
Now, combining inequalities (7) and (8), and using the fact that ∑K k=1 p̂k=1, we obtain:
g(p) ≥ g(t) (10)
A.2 PROPOSITION 2
Proof. Considering a generative view of the prediction model, random variable F associated with the learned features is continuous, while the random variable describing the labels, i.e., K, takes its possible values in a finite set {1, . . . ,K}. Then, the marginal distribution of the labels could be empirically estimated by the GT proportion of each segmentation region (as listed in Table 1) (Kim et al., 2005) :
P(K = k) ≈ ŷk = |Ωk| |Ω|
(11)
Also, we express the conditional entropy of the learned features as follows :
H(F|K) = K∑ k P(K = k)H(F|K = k)
≈ 1 |Ω| K∑ k |Ωk|H(F|K = k) (12)
with eachH(F|K = k) given by : H(F|K = k) = − ∫
fθ P(fθ|K = k) logP(fθ|K = k)dfθ (13)
Hereafter, for notation simplicity, we omit K and use H(F|k) instead of H(F|K = k). Also, we use P(fθ|k) instead of P(fθ|K = k).
To estimate the conditional entropy in Eq. (13), let us refer to the following well known Monte-Carlo estimation (Kearns et al., 1997; Tang et al., 2019) :
Monte-Carlo estimation. For any discrete set of points S ⊂ Ω, any function g and any feature embedding f , we have : ∫
f g(f)P(f |S) ≈ 1 |S| ∑ i∈S g(fi) (14)
where fi denotes a feature vector at point i, and P(f |S) stands for the density of {fi, i ∈ S}. Therefore, applying Montre-Carlo toH(F|k) in Eq. (13), we can re-write Eq. (12) as follows :
H(F|K) ≈ − 1 |Ω| K∑ k ∑ i∈Ωk log(P(fθi |k)) (15)
Furthermore, using Bayes rule P(fθi |k) ∝ pik p̂k , in addition to the fact that ∑ i∈Ωk log(p̂k) = |Ωk| log(p̂k), we obtain :
H(F|K) ≈ − 1 |Ω| K∑ k ∑ i∈Ωk log ( pik p̂k )
= − 1 |Ω| K∑ k ∑ i∈Ωk log (pik) + 1 |Ω| K∑ k ∑ i∈Ωk log (p̂k)
= CE + 1
|Ω| K∑ k |Ωk| log(p̂k)
= CE + K∑ k ŷk log(p̂k) (16)
Finally, due to the definition of the label-marginal KL divergence, we have :
DKL(y||p) = K∑ k=1 ŷk log ( ŷk p̂k ) c = − K∑ k ŷk log(p̂k) (17)
This yields :
CE c= H(F|K) +DKL(y||p) (18)
In summary, we give an information-theoretic prospective of CE. The entropy term can be considered as a ground-truth matching term, while the label-marginal KL term avoids trivial solutions and encourages the proportions of the predicted segmentation regions to match the ground-truth proportions.

heading: B THE BINARY SEGMENTATION CASE
text: In the two-class (binary) segmentation case, Dice might be used for the foreground region only (Milletari et al., 2016). Similarly to the multi-class case discussed in the paper, a single Dice term also decomposes into a ground-truth matching term and label-marginal penalty, with the latter encouraging extremely imbalanced binary segmentations. For this specific case, the logarithmic Dice and CE could be written as summations over the foreground and background segmentation regions:
− log(Dice1) c = − log
( 1
|Ω1| ∑ i∈Ω1 pi1 ) ︸ ︷︷ ︸
Foreground matching: DF1
+ log (∑ i∈Ω pi1 + |Ω1| ) ︸ ︷︷ ︸
Label-marginal bias: DB1
(19)
CE = − 1 |Ω1| ∑ i∈Ω1
log pi1︸ ︷︷ ︸ Foreground matching: CE1
− 1 |Ω2| ∑ i∈Ω2
log(1− pi1)︸ ︷︷ ︸ Background matching: CE2
(20)
In Eq. (19), the term DB1 can be expressed, up to an additive constant, as a function of the marginal probability of the foreground class (k = 1) as follows:
DB1 c = log(p̂1 + ŷ1) (21)
Clearly, the marginal probability p̂1 measures the predicted proportion of pixels within the foreground region. This term reaches its minimum when the foreground region is empty (pi1 = 0∀i). Therefore, since log is monotonically increasing, minimizing term DB1 in Eq. (19) introduces a bias preferring small foreground structures. Note that this label-marginal term in the logarithmic Dice loss is important to avoid trivial solutions: when using the foreground-matching term alone, the model may assign all the pixels in the image to the foreground region.
The foreground-matching terms, CE1 and DF1, are closely related, with the former being and upper bound on the latter, due to Jensen’s inequality: DF1 ≤ CE1. Both foreground-matching terms are monotonically decreasing functions of each softmax and reach their global minimum when all the softmax predictions in the ground-truth foreground are equal to 1 (i.e., reach their target). Hence, the matching terms in Dice and CE can be viewed as two different penalty functions for imposing the same equality constraints, pi1 = 1,∀i ∈ Ω1, thereby encouraging the predicted foreground to include the ground-truth foreground.

heading: C DISTRIBUTION OF REGION PROPORTIONS ACROSS CLASSES
text: In Fig. C.1, we present the distribution of region portions for each class in Cityscapes and Retinal Lesions. Cityscapes has highly variable region sizes (ranging from an average of 1.0% for rider and motorbike to an average of 32.6% for road). In Retinal Lesions, although the average region proportion is 3.8%, there are some examples with large region proportions (above 50.0%), and the standard deviation is 6.6%. Thus, these plots present the significant heterogeneity on different classes found across these two data sets.

heading: D THE TEMPERATURE SCALING
text: In our implementation, we employ a modified soft-max function with a temperature scaling parameter when computing the predicted region proportion p̂k (also referred to as the predicted labelmarginal probability, as in Table 1) :
s(z)i = eτ ·zi∑K j e τ ·zj (22)
where z = (zi)1≤i≤K is the input vector of the soft-max function, and τ > 0 acts as the temperature hyper-parameter. High values of τ > 0 yield high confidence of the soft-max prediction, as shown in Fig. D.1: They push the softmax vector towards the vertices of the simplex, with prediction values approaching either 0 or 1. As a result, this enables a better estimate of the actual region proportion (or relative size). Throughout all our experiments, we set τ to 10.

heading: E STUDY OF THE TRADE-OFF WEIGHT IN THE COMPOSITE LOSSES
text: DS C
Figure E.1: The performances of different composite losses on the test set of Retinal Lesions with different values of the trade-off hyper-parameter λ. The network we used is fixed to Res50-FPN.
The proposed method is a type of composite loss. Thus, the trade-off hyper-parameter λ could influence the final performance. Fig. E.1 shows the performances of the different composite losses examined in the paper, with different trade-off hyper-parameter values. While we plot the performances on the test set of Retinal Lesions, the behaviour on the validation set is similar, with the best trade-off parameters reached at the same value for both the validation and test sets.
From the curves, we can notice that L1 is a better choice than KL divergence for the proposed method because of its better stability. This may relate to its gradient properties and stability at the vicinity of 0 (see Fig. 2 in the main document). Thus, we can use a relatively larger weighting values for CE+L1 (best performance achieved when λ = 1.0). Observe that CE+DKL leads to comparable results at its best when λ = 0.1, but it seems more sensitive to the trade-off hyperparameter. For the loss integrating CE with the Dice bias term, i.e., CE+DB1, we demonstrate that it can achieve performances similar to CE-log(Dice), but it drops significantly with high weighting values (λ ≥ 0.1). This might be due to its gradient characteristics. Comparing to the widely suggested composite loss of CE and Dice, our method deliver better performance with the same hyper-parmeter budget.

heading: F PER-CLASS RESULTS ON CITYSCAPES TEST SET
text: Table F.1: Per-class Results on Cityscapes test set. (Top: Res50-FPN, Bottom: Res101-FPN) The second row indicates the average region proportion (mRegProp) for each class. mIoU denotes the mean IoU score over all classes.
road swalk build. wall fence pole tlight sign veg. terrain sky person rider car truck bus train mbike bike mIoU mRegProp (%) 32.6 5.4 20.2 0.6 0.8 1.1 0.2 0.5 14.1 1.0 3.6 1.1 0.1 6.2 0.2 0.2 0.2 0.1 0.4
CE 97.5 79.3 90.3 45.0 46.3 53.5 61.6 67.3 92.1 70.1 93.5 78.4 53.3 93.7 41.4 48.8 38.2 55.2 67.1 66.97 Focal loss 97.6 79.1 89.8 41.3 43.6 49.9 57.7 63.8 91.7 68.4 94.1 75.7 47.3 93.0 37.8 44.9 35.6 50.5 63.8 64.50 CE− log(Dice) 97.4 79.2 90.3 43.8 45.0 55.1 67.0 70.3 91.8 70.0 94.0 79.4 56.0 93.3 39.7 50.8 32.2 51.2 68.1 67.08 CE + KL (Ours) 97.6 79.6 90.4 45.2 45.0 53.1 61.3 66.9 92.0 70.5 94.1 78.5 54.7 93.6 48.8 59.1 45.8 55.5 67.0 68.35 CE + L1 (Ours) 97.6 79.4 90.4 44.1 45.8 53.7 61.3 67.2 92.1 70.2 94.3 78.8 54.4 93.7 47.8 58.4 46.8 55.4 67.3 68.35 CE 97.8 81.3 90.9 45.6 50.2 55.7 64.2 67.1 92.3 70.9 94.6 79.7 57.6 94.2 51.7 58.0 43.5 58.1 68.5 69.57 Focal loss 97.9 81.3 90.6 46.4 46.9 52.4 59.9 65.7 91.9 69.4 94.4 77.3 53.0 93.6 42.7 48.7 35.9 54.5 66.5 66.79 CE− log(Dice) 97.7 80.0 90.5 43.8 48.5 55.8 68.1 70.9 91.7 70.5 94.2 80.6 61.4 93.7 46.9 57.6 40.6 59.2 70.4 69.58 CE + KL (Ours) 97.9 81.4 91.2 48.7 49.7 55.5 63.7 68.6 92.4 70.6 94.6 79.9 57.1 94.3 54.5 61.1 43.5 58.3 68.4 70.07 CE + L1 (Ours) 98.0 81.9 91.2 48.7 50.3 55.7 63.8 68.3 92.3 70.7 94.5 79.8 57.5 94.2 52.9 60.3 44.0 59.2 68.7 70.11
In Table F.1, we report the detailed per-class results on the Cityscapes test set. The proposed methods outperform related losses on both network settings. In term of mIoU, we achieve 68.35% and 70.11% on Res50-FPN and Res101-FPN, respectively. Compared to the baseline CE, our method yields improvement for most of the minority classes, like fence, train and motorbike, while it shows better adaptability for diverse classes than the Dice loss.

heading: G MORE QUALITATIVE RESULTS ON THE RETINAL LESIONS DATASET
text: 0.781 0.776 0.798 0.818
GT CE 1 - Dice CE - log( Dice ) CE + L1 (Ours)1
0.792 0.757 0.804 0.812
0.631 0.120 0.540 0.667
0.539 0.698 0.647 0.726
Figure G.1: Additional visual results on the Retinal Lesions dataset. The ground-truth is depicted in the first column. At the bottom of each image, we indicate the corresponding obtained DSC score.
Fig. G.1 gives more qualitative examples from Retinal Lesions Dataset. It is shown that our method is able to adapt to the highly variant segmenting regions from small to large, while Dice loss obviously degrades in the case of large regions like the example in the third row.

Review summaries:
The paper presents an analysis of the cross-entropy (CE) and dice loss (DSC) for segmentation tasks,  in terms of the label-marginal bias.  According to the analysis, DSC prefers small regions, while CE encourages a prediction that has a similar proportion to the ground truth. Considering this analysis, the work proposes an alternative loss function consisting of the cross-entropy augmented with a regularization term (L1 or KL divergence) that operates on the prediction of the network. Different experiments compare the presented loss function of previous proposals.

This work presents a decomposition of two commonly used segmentation losses (cross entropy/CE and Dice losses) into two competing components. The two components are interpreted as ground truth matching and label-marginal penalties. This work argues that the label-marginal penalties in Dice loss favour extreme  class imbalances, and hence their widespread use in medical imaging segmentation tasks which encounter class imbalance more commonly. The label marginal terms in CE is argued to be better as this term matches the label marginals to the ground truth distributions. The scaling between the two terms in CE is presented as the problem and this work proposes an L1 regularization to overcome the label bias problem in CE. Experiments on two segmentation tasks show interesting influence of these regularizations. 

This paper proposed theoretical analysis showing Dice loss has a bias preferring small structures, while losing flexibility of dealing with arbitrary class proportions. On the hand hand, the cross-entropy loss encourages proportions of the predicted segmentation to match the ground-truth proportions. A new loss function to control the label-marginal biases was proposed as sum of CE and weighted L1/KL divergence regularization.