The hidden label-marginal biases of segmentation losses
Authors: Bingyuan Liu
Jose Dolz
Adrian Galdran
Riadh Kobbi
Ismail Ben Ayed
Venue: ICLR 2022 Submitted (2022)

Abstract: Most segmentation losses are arguably variants of the Cross-Entropy (CE) or Dice losses. In the abundant segmentation literature, there is no clear consensus as to which of these losses is a better choice, with varying performances for each across different benchmarks and applications. In this work, we develop a theoretical analysis that links these two types of losses, exposing their advantages and weaknesses. First, we provide a constrained-optimization perspective showing that CE and Dice share a much deeper connection than previously thought: They both decompose into label-marginal penalties and closely related ground-truth matching penalties. Then, we provide bound relationships and an information-theoretic analysis, which uncover hidden label-marginal biases: Dice has an intrinsic bias towards specific extremely imbalanced solutions, whereas CE implicitly encourages the ground-truth region proportions. Our theoretical results explain the wide experimental evidence in the medical-imaging literature, whereby Dice losses bring improvements for imbalanced segmentation. It also explains why CE dominates natural-image problems with diverse class proportions, in which case Dice might have difficulty adapting to different label-marginal distributions. Based on our theoretical analysis, we propose a principled and simple solution, which enables to control explicitly the label-marginal bias. Our loss integrates CE with explicit ${\cal L}_1$ regularization, which encourages label marginals to match target class proportions, thereby mitigating class imbalance but without losing generality. Comprehensive experiments and ablation studies over different losses and applications validate our theoretical analysis, as well as the effectiveness of our explicit label-marginal regularizers.

Under review as a conference paper at ICLR 2022
THE HIDDEN LABEL-MARGINA, BIASES OF SEGMEN- TATION LOSSES
Anonymous authors Paper under double-blind review
ABSTRACT
Most segmentation losses are arguably variants of the Cross-Entropy (CE) or Dice losses, In the abundant segmentation literature, there is no clear consensus as which of these losses is a better choice, with varying performances for each across different benchmarks and applications. In this work, we develop theo- retical analysis that links these two types of losses, exposing their advantages and perspective showing
weaknesses. First, we provide a constrained-optimizati that CE and Dice share much deeper connection than previously thought: They both decompose into label-marginal penalties and closcly related ground-truth matching penalties. Then, we provide bound relationships and an information- theoretic analysis, which uncover hidden label-marginal biases: Dice has intrin- sie bias towards specific extremely imbalanced solutions, whereas CE implicitly encourages the ground-truth region proportions. Our theoretical results explain the wide experimental evidence in the medical-imaging literature, whereby Dice losses bring improvements for imbalanced segmentation. It also explains why CE dominates natural -imurell-mmgeeeeee problems with diverse class proportions, in which case Dice might have difficulty adapting to different label-marginal distributions. Based on our theoretical analysis, we propose principled and simple solution, which enables to control explicitly the label-marginal bias. Our loss integrates CE with explicit Ci regularization, which encourages label marginals to match target class proportions, thereby mitigating class imbalance but without losing gener- ality. Comprehensive experiments and ablation studies over different losses and applications validate our theoretical analysis, as well as the effectiveness of our explicit label-marginal regularizers.
INTRODUCTION
Semantic segmentation is one of the most investigated problems in computer vision, and has been impacting a breadth of applications, from natural-scene understanding (Cordts et al. 2016 Kirillov et al. 2019) to medical image analysis (Litjens et al. 2017 Dolz et al. 2018 In the recent years, deep learning methods have dominated the field, as a result of the great capacity of Convolutional Neural Networks (CNN) (He et al. 2016) to automatically learn representations from large-scale data sets (Long et al., 2015 RRnneberger et al.) 2015) Zhao et al. 2017 Chen et al., 2018 Yuan let al.) (020).. Semantic segmentation often stated pixel-wise classification task, following the optimization of a loss function expressed with summations over the ground-truth regions, as in the standard Cross-Entropy (CE) loss. A relevant aspect of segmentation problems is class imbal- ance, i.e., unequal proportions of the segmentation regions, which may cause large-region terms in the objective completely dominate small-region ones. A representative example the popular Cityscapes dataset (Cordis et al.) 2016), where the average proportions of some classes, such as motorcycle or bicycle, are below 1%. In these scenarios, besides specifically designed CNN archi- tectures or training schemes (Tao et al., 2020) Bao et al.) [202I), the loss function to be minimized during learning plays a critical role, and has triggered a large body of research works in the last years
(Lin et al. 2017 Wong et al. 2018: Kervadec et al. 2021b Milletari al. 2016 Sudre al. 2017
Kervadec et al. 2021a).
There exists great diversity of loss functions for image segmentation, which can be categorized into two main families, and are arguably variants of CE, the Dice loss (Ma al. 2021 Yeung et al.]
2021), or combinations of both (Wong al. 2018: Taghanaki et al. 2019). The first family is mo-

---

Under review as a conference paper at ICLR 2022
GT
CE
1- Dice
CE logiDkel
CE L1 (Ours)
0.5H4
0.308
6.70
0.767
2.000
C.4C0
0.443
0.507
The visual improvement of our solution compared to various losses under different classes in Cityscapes. The ground-truth is given in the left-most column. A magnified contour of the segmentation is also provided alongside the original target within each image. The numeric below each image indicates the IoU score for the corresponding prediction result. The examples show how Dice has a bias towards small regions, while CE is more stable for different scenarios. Our solution capable of leveraging the best aspects of both. See Sec. 2 for the explicit theoretical analysis and Sec. 3 for the detailed experimental results. Best seen in color.
Figure| 1:
tivated by distribution measures, i.e., CE and its variants, and is directly adapted from classification tasks. To deal with class imbalance, various extensions of CE have been investigated, such as in- creasing the relative weights for minority classes (Ronneberger al. 2015), or modifying the loss so as to account for performance indicators during training, as in the popular Focal loss (Lin et al. 2017) Topk loss (Wu et al. 2016). The second main family of losses is inspired by geometrical metrics. In this category, the most popular losses are linear Dice (Milletari et al. 2016) and its ex- tensions, such as the logarithmic (Wong et al., 2018) or generalized (Sudre et al. (017)) Dice loss. Borrowing the idea of the weighted CE, the latter introduces class weights to increase the contribu- tions of the minority classes. These loss functions are motivated by the geometric Dice coefficient, which measures the overlap between the ground- -ruut and predicted segmentation regions.
In the literature, to our knowledge, there is no clear consensus as to which category of losses is better, with the performances of each varying across data sets and applications. It has been empirically argued that the Dice loss and its variants are more appropriate for extreme class imbalance, and such empirical observations are the main motivation behind the wide use and popularity of Dice in medical-imaging applications (Milletari et al. 2016 Jha et al. 2020). CE, however, dominates most recent models in the context of natural images (Chen et al. 2018 Zhao et al., 2017 2018 Yuan et al. 2020). Therefore, beyond experimental evidence, there is a need for a theoretical analysis that clarifies which segmentation loss to adopt for given task, decision that affects performance significantly.
On the surface, these two categories of losses (i.e., geometry-based vs. distribution-based) seem unrelated. Furthermore, an important body of work in the literature suggests that CE and Dice are complementary, which has motivated composite losses integrating both, e.g. (Wong et al. 2018 Taghanaki et al.] 2019). Such composite losses perform very competitively in extremely imbalanced segmentation, shown by (Wong et al. 2018 Taghanaki al. 2019), among several other recent
works (Isensee et al. 2018 Ma et al.) (021)).
perspective showing that, in fact, CE and Dice share a much deeper connection than previously thought: They both decompose into label-marginal penalties and closely related ground-truth matching penalties. Our theoretical analysis highlights en- coded hidden label-marginal biases in Dice and CE, and shows that the main difference between the two types of losses lies essentially in those label-marginal biases: Dice has an intrinsic bias prefer- ring very small regions, while CE implicitly encourages the right (ground-truth) region proportions. Our results explain the wide experimental evidence in the medical -mmaaing literature, whereby us- ing or adding Dice losses brings improvements for imbalanced segmentation with extremely small regions. It also explains why CE dominates natural- natural-image problems with diverse class proportions, in which case Dice might have difficulty adapting different label-marginal distributions (see Table
In this paper, we provide a constrained-optimizzzin
2

---

to Under review as a conference paper at ICLR 2022
3 and the examples depicted in Fig. Db. Based theoretical analysis, propose principled and simple loss functions, which enable to control explicitly the label-marginal bias. Our solution integrates the benefits of both categories of losses, mitigating class imbalance but without losing generality, as shown the examples in Fig. [
Our contributions summarized follows:
Showing through an explicit bound relationship (Proposition) T that the Dice loss has a hid- den label-marginal bias towards specific extremely imbalanced solutions, preferring small structures, while losing the flexibility to deal effectively with arbitrary class proportions. perspective of CE, via Monte-Carlo approximation of
Providing an ceoomation-thhorede the entropy of the learned features (Proposition 2) This highlights hidden label- marginal bias of CE, which encourages the proportions of the predicted segmentation regions to match the ground-truth proportions.
Introducing new loss functions to control label- -arginal biases: Our losses integrate CE with explicit regularization terms based on L1 or the KI. divergence, which encourage label marginals match target class proportions.
Comprehensive experiments and ablation studies over different losses and applications, including natural and medical-imaginn data, validate our theoretical analysis, as well as the effectiveness of our explicit label- borl-iaa regularizers.
FORMULATION
Table 1: Notations, formulations and approximations used in this paper. and KC denotes the random variables associated with the learned features and the labels, respectively. P denotes probability. |.| denotes cardinality when the input a set and the standard absolute value when the input is a scalar. Note that network parameters 0 are omitted in the prediction quantities, simplify notations, as this does not lead to ambiguity.
Modeling
Dataset
Concept
Formula
Concept
Formula
Model parameters
Indices/numher of classes
S & K N < R²
Feature embedding at pixel 9
I
Spatial image domain
Soltmax predictions at pixel f
Pa (P(EFF")
Labels of pixel $2
Sha [0.
Predicted proportion of class le
Pa To EieaPa
GT region
e Blyne 1}
GT proportoon of region k
iu
[Ba
Predicted label-marginal prob.
p (At)
(K 1)-iipplx
AK {pe (0, 1K ExA==
GT lahel-marginal prob.
(Jt Min)g
Losses, mahel-marginal regularizers and information tthooretic quantities
Concept
Formula
Weighted cross-entropy
CE ΣK_ NET Eico, log(ps))
22nn,, Pie
Dice coefficient for region
Dice; E.en Puo
Label-mant marbeiall KL divergence
Pku(y||p) >K ix log(
Label marginal Ci distance
Li p)) Σk lim P
Monte-Card estimate of the entropy of features given region k H(FIKC = b) = [ Σiea. (P
Semantic segmentation is often stated as a pixel-wise classification task, following the optimization of loss function for training a deep network. In Table we present the notations, formulations and approximations used in our subsequent discussions. Besides the basic notations of the task (such as networks predictions), we explicitly include the loss functions, label-marginal regularizers and information-theorrtiee quantities that will be discussed in the following sections. We note that, to facilitate the reading of our analysis, we write the CE and Dice losses in a non-standen nta-standard way using summations the ground- truth segmentation regions, rather than functions of the labels. Also, while we provide the CE loss for all segmentation regions, we give Dice for a single region. This is accommodate two variants of the Dice loss in the literature: in the binary case, Dice is typically
3

---

Under review as a conference paper at ICLR 2022
used for the foreground region only (Milletari et al. 2016); in the multi-region case, it is commonly used over all the regions (Wong et al. 2018). Finally, to simplify notation, we give all the loss functions for single training image. without summations over all training samples (as this does not lead to any ambiguity, neither does it alter the analysis hereafter). In the training iterations, we use the mean values across all the training samples via standard mini-batch optimization.
2.1 DEFINITION OF
BIASES AND PENALTY FUNCTIONS
In the following, we analyse the the main difference between the two types of losses lies essentially in those maael-marginal biases. To do so, we provide bias as a soft penalty function for the hard equality constraint = t, where t is a given (fixed) target distribution. Such a penalty encourages the predicted label-marginal p to match a given target distribution t. In the general context constrained optimization, penalty functions are widely used (Bertsekas) 1995). In general, penalty methods replace equality constraints of the form p by adding a term (()) to the main objective being minimized. Such penalty function increases when p deviates from target t. By definition, for the constraint p t, with the domain of p being probability simplex Ak. penalty g(p) is a continuous and differentiable function, which reaches its global minimum when the constraint satisfied, i.e., it verifies: g(t) S g(p) Vp AK.
biases inherent to CE and Dice losses, and show that
petrrained-opimmizatinn perspective of the losses: We define a label-marginal
2.2 THE LINK BETWEEN CROSS ENTROPY AND DICE
To ease the discussion in what follows, we will start by analyzing the link between CE and the logarithmic Dice, along with the label-marginal bias of the latter (Proposition [. Then, we discuss a bounding relationship between the diffferent Dice variants. Finally, we will provide information- theoretic analysis, which highlights the hidden marbinaa bias of CE (Proposition 2)
Let consider the logarithmic Dice loss in the multi-class case. This loss decomposes (up constant) into two terms, a ground -ruth matching term and a label-marginal bias:
log (og(Dic)s))
log
Pik
log (pe + 0%)
(1)
12
SE
Grand-truth matching:
Label-marginal bias: DB
where - stands for equality up to an additive and/or non-negative multiplicative constant. The ground-truth matching term in Eq. (I) is a lower bound on the cross-entropy loss (CE) due to Jensen's inequality and the convexity of function - llo((z): DF < CE. Therefore, minimiz- ing CE could be viewed as a proxy for minimizing term DF that appears in the logarithmic Dice. In fact, from constrained-optimization perspective, DF and CE are very closely related and could be viewed as two different penalty functions enforcing the same equality constraints: Pik = 1, Vi E S2y Vk. Both DF and CE monotonically decreasing functions of each softmax and reach their global minimum when these equality constraints are satisfied. Therefore, they en- courage softmax predictions Piu: for each region $22 reach their target ground-truth values of 1. Of course, this does not mean that penalties CE and DF yield exactly the same results. The difference in the results that they may yield due to the optimization technique (e.g., different gradient dynamics in the standard training of deep networks as the penalty functions have different forms).
2.3 THE HIDDEN LABEL-MARGINAL BIAS OF DICE
The following proposition highlights how the label-marginal term DB in Eq. T encourages specific extremely imbalanced solutions.
Proposition 1. Let t (G)) ISiSk E [0,1]K denote the simplex vertex verifying: f, when 0 = max s Uy and Eg = O otherwise. For variables = (fw) and fixed distribution y (úx I the label-marginal term in Eq. [ reaches its minimum over the simplex at t;
K
K
log (fx + +x) $
log (px + 9k) Vp Ak
(2)
4

---

Under review as a conference paper at ICLR 2022
Proof. The details of the proof are deferred to Appendix[A.I The main technical ingredient is based on Jensen's 's inequality and the concavity of penalty DB with respect to simplex variables p.
Inequality 2 means that the label-marginal term in Dice in Eq. (T) is penalty function for con- straint = t, where t is the simplex vertex given in Proposition [ Therefore, it encourages ex- tremely imbalanced segmentations, where a specific region includes all the pixels and the remaining regions are empty. All in all, the logarithmic Dice loss integrates hidden label -mrginall prior pre- ferring extremely imbalanced segmentations, which is optimized jointly with a ground-truth match- ing term similar to CE. It is worth noting that, in the two-class (binary) segmentation case, Dice might be used for the foreground region only, as the popular work in (Milletari al. 2016), for instance. Similarly to the multi-class case discussed above, single Dice term also decomposes into a ground-truth matching term and babel-marginal penalty, with the latter encouraging extremely imbalanced binary segmentations. We provide more details for this case in Appendix B
2.4 ON THE LINK BETWEEN THE DIFFERENT VARIANTS OF DICE
The label-marginal analysis we discussed above based on the standard logarithmic Dice loss. Here, argue that both logarithmic and linear Dice very closely related and, hence, the linear Dice also hides claas-imballncee bias. In fact, from a constrained -iteraaindd-oiimizzaiio perspective, the two losses could be viewed as different penalty functions for imposing constraints: Dice; 1 vk. Both functions (z) and (1 - x) are monotonically decreasing in [0, 1] and achieve their minimum Furthermore, the logarithmic Dice is upper bound on the linear one. This follows directly from: log (1) 2 - 1I > 0. Of course, this does not mean that optimizing these two variants leads to exactly the same results. The differences in their results might be due to optimization (i.e., different gradient dynamics stemming from logarithmic and linear penalties).
in [0, 1] at x
2.5 THE HIDDEN
BIAS OF CE
In the following, give an -nformation-theoreticc perspective of CE, via generative view of network predictions and Monte-Carlo approximation of the entropy of the learned features given the labels. This highlights hidden label-ma morainngi bias of CE, which encourages the proportions of the predicted segmentation regions to match the ground-truth proportions.
Proposition 2. Let F and K. denote the random variables associated with the learned features and the labels, respectively, and H(F)K) the conditional entropy of learned features given the labels, estimated via Monte-Cario:
K
K
H(FIK)) R Σ U
(o)(
Ω
where H(FFC = k) is the empirical estimate of the conditional entropy of features given a specific class k (expression in Table [ and (k) denotes the probability of the learned features given class k. We have the following generative view CE:
CE:
CE -
H(FIK)
+ (yI(y
Ground-truth matching
Label-morginal bias
The approximation of H(FK = k) in the second line Eq. 3 based on the well-known Monte- Carlo estimation (Kearns et al. 1997 Tang al., 2019). Then the relationship in Eq. (4) follows from Eq. (3), after some manipulations, using Bayes Pik and Σien, log(Pk) |S2g log(
The detailed proof is deferred to Appendix A.2
rule (ff Pt
This information- theoretic view of CE shows that the latter has an implicit (hidden) label-marginal bias towards the ground-truth region proportions (the KL term). This bias competes with the entropy term, which encourages low uncertainty (variations) within each ground-truth segmentation region Sr The entropy term could be viewed as ground-truth matching term: it reaches its global minima when the feature embedding is constant within each region. If used alone, the entropy term may lead to trivial imbalanced solutions. The by matching the ground- ground-truth class proportions. Note that there no mechanism in CE to control the relative contributions of those two competing terms they are implicit in CE.
KL term avoids such trivial solutions
5

---

Under review as a conference paper at ICLR 2022
2.6 OUR SOLUTION
Our analysis shows that Dice, CE and their combinations, e.g., CE log(Dice), are closely related and enforce types of competing constraints: ground-truth matching and label-marginal con- straints. However, there is no clear consensus in the literature as to which loss is better, with the performances of each varying across data sets and applications. This variability performances could be explained by two fundamental factors:
The difference in the label-marginal prior. The label-marginal priors are different Dice has an intrinsic bias preferring very small regions, while CE encourages the right (ground-truth) region proportions. This might explain the wide experimental evidence in the medical imaging literature, where using or adding Dice losses brings improvements for imbalanced segmentation with extremely small regions.
Weighting the contribution of the bias term. Our analysis suggests that CE should be preferred over Dice cases and applications (both balanced/ibbllanced segmentation, or segmentation problems with high variability in region proportions) as it promotes the right label-marginal distri- bution. While this seems to be widely the case in natural image segmentation, where Dice is un- common, the extensive experimental evidence in the medical-image segmentation literature suggests otherwise, especially in extremely imbalanced problems. We argue that this due to the relative contribution of the label-marginal term in the overall objective. Controlling such label-marginal con- tribution very important in imbalanced problems. In particular, it mitigates the difficulty that the ground-truth matching terms differ by several orders of magnitude across regions, as in CE, which causes large-region terms to completely dominate small-region ones. This analysis also resonates with the fact that combo losses such as CE A log (Dice) perform very competitively imbalanced segmentation, as shown by (Wong et al. 2018 Taghanaki et al., (019). among several other recent works. In this case, controlling the relative contribution of each of these terms indirectly controls the weight of the label-marginal bias. Note that such control is not possible when using CE alone or Dice alone, as the label- marrinaa biases these losses are hidden (implicit).
We propose a principled and simple solution, which enables to control explicitly the label- marginal bias, via regularization losses that en- courage the correct class proportions and are used in conjunction with CE
CE xR(y:)
(5)
Our label-marginal regularizers increase the contribution of the minority classes in imbal- anced problems, but, unlike Dice, do not lose adaptability to problems with various class pro- portions. Our extensive experiments and abla- tion studies over different losses and applica- tions demonstrate the effectiveness of our ex- plicit label-marginal regularizers. We investi- gate different forms of regularization, includ- ing the Ci norm, i.e., R(y;p) (y, and the KL divergence, i.e., R(y:p) DRI.(y (ylp)); see Table I for the expressions of DxL and L1- In Fig. [ we depict our different regulariz- ers as functions of the aabel-marginal distribu- tion for mnnary-segmentatioon case, with the foreground-region proportion set to 0.1, along with the bias terms in Dice. While our DxL. and C, regularizens may deliver comparable performances (see the experimental section), L₁ might be better option for extremely imbalanced segmentations, due to its gradient properties and stability at the vicinity of 0, i.e., when the label-margee marginal probability P1 is close to O. Notice that, at the vicinity of zero, both first and second derivatives of the regularizer unbounded for DRL. but bounded and constant for L, Our experiments on imbalanced medical image segmentation confirm the effectiveness of the Li regularizer.
D2c
D81
0.0
0.1
0.3
0.5
0.7
0.9
Figure 2: Different label-marginal biases. The ground-truth foreground region proportion set to 0.1. The expression of penalty DB, is pro- vided Appendix B and corresponds to the two- class (binary) variant of Dice, where the loss is used over the foreground region only. Penalty Li presents better gradient dynamics at the vicinity of label marginal & 0. Best seen in color.

---

Under review as a conference paper at ICLR 2022
3 EXPERIMENTS
Datasets. We report results on both medical and natural image segmentation benchmarks, i.e. Retinal Lesions (Wei et al.) 2020) and Cityscapes (Cordts et al. collection of color fundus images, as shown in Fig. 3 Iandd Fig. G.I It is noted that a panel of 45 experienced ophthalmologist was formed to label this dataset and each image was assigned to at least three annotators to get trustworthy pixel-level annotations WWii et al. 2020). In our experiments, we employ its public version" containing 1, 593 samples, and conduct our experiments in the binary setting to segment the lesion region versus background. The dataset is randomly divided into training (70%), validation (10%) and test (20%) set, whose images are resized to 512 512. We tune the hyper-parameters of each model on the validation set, and report the results on the test set. The natural scene dataset, Cityscapes, is large-scale dataset with high quality pixel-level annotations of 5, 0000 images across 19 categories, containing both stuff and objects with high variation in the class proportions distribution. We use the official data split, which contains 2, 975 samples for training, 500 samples for validation and 1, 512 samples for testing, All the input images are resized to 512 x 1024 for training, and 1024 x 2048 for testing. Fig. C.I in Appendix C shows the distribution of region proportions for Retinal Lesions and the 19 classes in Cityscapes, highlighting the challenging class imbalance and high diversity on target sizes.
2016). Retinal Lesions is a large
Backbones. It is worth stressing that the theoretical discussion in this paper is model-agnostic. Thus, in our experiments we employ Res34-FPN, Res50-FPN (Lin et al., 2017) and Res34-Unet, Res50-Unet (Ronneberger et al. 2015) on Retinal Lesions, while we show results with Res50-FPN and Ress01--FFF on Cityscapes, whose implementations are publicly available"
Training. The parameters of the encoder are initialized with the pre-trained weights on Imagenet, while those in the decoder head are randomly initialized. On Retinal Lesions, we train the model during 60 epochs, with batch size of 8, via the Adam optimizer. The initial learning rate is set to 1e-4, and halved if the loss on the validation set does not decrease within 5 epochs. On Citycapes, the main setting adopted from the state-of-the-art library] Specifically, the SGD optimizer is used for training, with an initial learning rate set to 0.01 and a batch size of 8. During the 100 training epochs, we use an iteration- wise polynomial strategy to linearly scale the learning rate down a minimum of le-4. For all the models, we tune the srpeer-praamterss on the validation and report the best results on the corresponding test set. Conventional data augmentations, such as random cropping, mirror flipping and changes in brightness, are employed on both datasets.
Losses. We evaluate the proposed loss function in Eq. (5) with two regularizing terms, i.e., DKL and L1 In our implementation, we use modified soft-max function with a temperature parameter when computing the predicted region proportion, as this enables a better estimate of the actual region proportion (refer to Appendix [D] for details). We compare our results with the following baselines: CE, Focal loss (Lin et al. 2017). Dice related losses, i.e., ameear/larrthmic/geeeriizzz Dice, and the composite loss combining CE and logarithmic Dice.
Evaluation metrics. On Retinal Lesions, we use two standard metrics in medical image segmenta- tion, i.e., Dice Similarity Coefficient (DSC) and modified Hausdorff Distance (HD-95). Particularly, HD-95 represents the 95th percentile of the symmetric Hausdorff Distance (HD) between the binary objects in two images. On Cityscapes, we resort the standard Intersection-OOvr--nnie (IoU) score, which is widely employed in natural image segmentation.
Results on Retinal Lesions. Table 2 reports the quantitative comparison between the proposed loss. with the two different regularizers, and baselines on the Retinal Lesions dataset. Regarless of the backbone network, the proposed CE+ L. loss consistently outperforms all the others with a DSC score of 54.5% for Res50-FPN and 54.3% for Res50-Unet, Compared to CE, our best model, i.e., Ci as the label-marginal regularizer, brings nearly 3.0% improvement in terms of DSC and between 5.6% and 8.6% in terms of HD-95, depending on the backbone. Even though these differences are reduced with respect to the best baseline, this gap is still significant, particularly in terms of HD-95. is noteworthy to mention that while DSC is more sensitive to the internal filling of the target region, the HD-95 is more sensitive to the segmentation boundary. Thus, the proposed loss is more effective
htt.aa/s//tttbbb.ccoo/m///////er mod_ateeeelmmmm___eee_ s.pytorch
7

---

Under review as a conference paper at ICLR 2022
Table 2: Quantitative evaluations of different losses Retinal Lesions. Average DSC and HD95 values (and standard deviation over three independent runs) achieved on the test set are reported. Note that Dice, is implemented for all the Dice related losses for the binary setting on this dataset, and DB, refers to label-marginal bias for the binary Dice (details can be found in Appendix B
Re534-FFM
Rostc-Unet
Res50-EPK
Rec50-Unet
Loss
(S)
HD-25
DSC(%))
HDD-55
DSC(SS)
HDD-55
DSC(SS)
HD-95
(0.1)
85.50
(0.5)
85.46 (3.71)
(2.63)
81.27 (4.12)
Focal loes Lingetal 2017
31.2 (0.2)
84.38 (4.86)
31.2 (0.8)
88. (4.90)
(2.9(0.4)
80.87 (2.75)
51.8 (0.6)
84:41 (1.93)
Dicce Mclleunt 24 ai 2016
52.0 (0.7)
(3.30
52.3 (0.7)
89.19 (2.44)
(2.7
82:20 (4.01)
53.2 (0.1)
81.93 (4.00)
log[Dscc Weez ef 2018
51.7 (0.9)
$6.02 (3.44)
52.7 (0.5)
85.84 (0.18)
(2.0
81,69 (3.57)
53.5 (0.5)
82.15 (2.76)
GDice (Sudic or 2013
51.9 (0.7)
86.70 (2.72)
52. (0.5)
89.99 (1.28)
53.2 (0.5)
80.79 (2.36)
53.7 (0.4)
86.84 (3.20)
CE laz(iee)
Wong
2018
52.2
81.00
84.13
78.38 (2.06)
(3.4.0.3)
82.55 (2.11)
82.72
86.07 (1.83)
78.77 (0.4)
83.04 (0:46)
Dau (Ouns)
52.7 (0.3)
83:31 (2.40)
5288 (0.3)
82.66 (2.40)
53.8 (0.1)
(1.82(1182))
(0.4)
80.17 (1:28)
CE + 2) (Ours)
52.8 (0.3)
90.68 (2.49)
53.1 (0.2)
81.15 (2.75)
54.5 (0.2)
74,99 (1.96)
54.3 (0.2)
77.03 (1.51)
to predict better boundaries than existing losses. Furthermore, as shown in Fig. C.1 in Appendix C the average region proportion is considerably low, i.e., 3.8%, and varies significantly (the standard deviation 6.6% with a maximum of 57 38).. Under this highly heterogeneous scenario in terms of class region proportions, the proposed losses present more stable performance across different runs and backbones, which is reflected their lower variances. This can be explained by their better adaptability to different target sizes and better gradient dynamics at the vicinity of label marginal Pk = 0 (as shown in Fig. 2. Another interesting finding is that, by combining CE with the bias term of Dice (CE + DB we obtain results close combo loss CE log( (Dice; )(Wong et al. 2018). This validates our theoretical insight that the fundamental difference between CE and Dice lies in their distinet hidden biases.
On the balancing weight in the composite losses. In this section we evaluate the impact of the balancing weight of the proposed loss in Eq. 5), which balances the effects of the regularization term, as well as the balancing weight in CE (ogce))))), ). In our experiments, we empirically found that the best values fordi f ferentpenaltyterme are: 1.0 for CE+ 0.1 for CE+ 0.01 for CE + D These values are consistent with the given arguments in Sec. 2.6 which emphasize that L1 has a better gradient evolution for large learning rates and relatively high weighting, especially at the beginning of training. Regarding the widely used combo loss of CE - log(Dicez), setting its balancing weight to 1.0 yielded consistent performance across the datasets and networks we used. Details of this experimental study can be found in Appendix D Therefore, with the same hyper- parameter budget, CE + Li performed better than all the other related composite loss functions. Note that use the best empirical values of X found Retinal Lesions for the experiments on Cityscapes.
GT
CE
1- Dici
CE - log( Dice;) CE + L1 (Ours)
0.677
0.727
0.734
0.774
0.591
0.659
0.574
0.756
Visual results on the Retinal Lesions dataset. The ground-truth is depicted in the first column. At the bottom of each image, we indicate the corresponding obtained DSC score.
Figure 3:
Qualitative results on the Retinal Lesions dataset. As shown in Fig. 3 we can observe that the model trained with Dice variants (third and fourth column) tends to undersegment large and medium target regions, while ignoring several small lesions and generating false positives. In contrast, the proposed solution enables better trade-off between finding small regions, redueing the number of false positives and matching the size of the larger targets. Furthermore, one can notice that our loss yields better consistencies with the target region proportions than CE.
8

---

Under review as a conference paper at ICLR 2022
Table 3: Results on Cityscapes validation set. (Top: Res50-FPN, Bottom: Res10I-FPN) The second row indicates the average region proportion (mRegPropp) for each class. mloU denotes the mean IoU score over all classes.
road swalk build. wall
ience
pole
tlight
sign
veg.
lerrain
sky
person
rider
car
Iruck
hus
train
mhike
bike mloo
mRegPrrp (%)
32.6
5.4
20.2
0.6
0.8
1.1
0.2
0.5
14.1
1.0
3.6
1.1
0.1
6.2
0.2
0.2
0.2
0.1
0.4
CB
97.2
80.6
90.0
38.2
51.3
55.7
60.5
72.2
91.0
58.6
92.3
75.6
48.5
92.5
50.5
67.7
56.3
50.2
71.6
68.45
Rocal loss
97.1
78.8
89.5
37.9
48,9
51.8
56.9
67.6
90.6
56.9
92.6
73.1
42.8
47.5
57.5
46.1
49.2
68.4
65.55 05.53
Dice
95.9
77.6
87.7
41.1
47.6
56.0
66.2
7488
90.0
59.6
91.1
77.3
55.8
91.1
10.0
63.9
14.6
53.6
73.6
65.77
log; Dice
94.0
70.8
85.6
33.2
39.1
50.6
62.2
69.6
88,4
52.1
74.2
50.7
89
34,8
55.3
31.7
48.0
70.7
62.54
CR (ngeDice)
96.9
78.6
89.8
404
48.3
57.1
65.4
74.6
90.7
57.2
92.5
77.0
52.7
92.4
47.9
65.0
50.6
51.4
72.6
68.48
CE + (Our)))))
97.2
79.6
90.0
39.6
51.2
54.8
60.3
71.5
91,0
89.7
75.2
49.0
92.7
55.6
12.5
65.6
51.3
70.9
69.52
CE + C. (Ours)
97.4
80.2
90.1
37.9
51.9
55.7
61.0
72.1
91.0
58.9
93.5
75.7
49.8
92.8
54.9
70.2
62.8
54.0
71.6
69.55
CB
97.8
82.4
91.0
46.5
55.3
62.6
72.3
91.5
76.1
68.4
77.2
64.8
54,9
72.6
72.10
Focal loss
97.7
81.9
90.8
48.2
54.1
54.4
59.0
69.4
91.0
60.2
93.7
74.7
47.8
93.0
59.3
67.3
$1.9
52.1
70,5
69.32
Dice
79.6
88.1
50.6
76.0
0.0
26.8
59.42
log Dice
95.2
73.9
86.4
31.5
39.2
52.4
63.3
70.4
88.6
53.0
92.2
74.6
52
89
37
57.3
31.1
42.7
71.5
63.26
CB Dioe)
97.2
79.9
42.5
52.2
74.8
71.7
72.1
58.9
71.00
CE + Piu (Ours)
97.8
82.6
91.1
45.4
56.8
57.4
63.4
72.5
91.5
61.3
94.0
76.7
52.4
93.7
09.2
78.3
64.5
$7.1
72.8
72.56
CE+ + (Ours)
97.8
82,9
91.2
56.8
72.7
80.0
67.1
57.2
73.44
Results on Cityscapes. Table 3 reports the comparative per-class IoU and mean IoU (mloU) on the validation set of Cityscapes with two network architectures. First, we can observe that in this multi-class dataset, regardless of the network, the proposed learning objectives outperform all the evaluated losses in terms of mloU, Then, by investigating the relationship between mRegProp (sec- ond row in Table 3 and the corresponding segmentation performance across small region classes, we can observe that Dice-related losses have a hidden label-marginal bias towards extremely imbal- anced solutions, preferring small structures. In particular, the linear Dice often obtains the highest mloU for the smallest structures. This bias comes at the cost of less flexibility when dealing with arbitrary class proportions, which reflected in its poor average mloU (right column).
Quantitative evaluation on the Cityscapes test set is re- ported in Table The observations in this table are con- sistent with the results on the validation set. Similarly to the validation set, our method achieves better results on both architectures. For the per-class scores the test set, please refer to Appendix [F]
Table 4: mIoU on Cityscapes test set.
Loss
Res50-FPN
Res101-FPN
CE
66.97
69.57
Focal loss
64,50
66.79
(Dic))
59.77
62.62
log (Dice)
67.00 08
69.58
CE DKL. (Ours)
68.35
70.07
It is noteworthy to highlight that both linear Dice and log- arithmic Dice perform relatively poorly on Cityscapes, on average, empirically showing why Dice is rarely adopted in natural-image segmentation tasks. As we mentioned earlier, this might be due to its inherent label-marginal bias, which is inappropriate for segmenting regions with arbitrary class proportions. Furthermore, our simple solution yielded an improvement over CE by nearly 1.5%, regardless of the backbone. These empirical observations suggest that the proposed formulation in Eq. (5) results in better aggion-proportion guidance and training stability than existing segmentation losses. Finally, we can observe that employing L1 as regularizer consistently results in better performance than using the DKL. term. Hence, Li might be a better option in extremely imbalanced segmentations, due to its gradient properties and stability when the marginal probability is close to O.
CE L1 (Ours)
68.35
70.11
4 CONCLUSION
We provided a detailed theoretical analysis of the two most popular semantic segmentation losses, i.e. Cross-entropy and Dice, which revealed non-obvious bounding relationships and hidden label- marginal biases, suggesting that CE is a better option in general. Then, we showed how both loss functions could be written within a common formulation, containing a ground-trutt -ruth matching term and a label-marginal bias. The implicit bias in Dice prefers small regions, improving its perfor- applications. The bias hidden in CE
mance in highly imbalanced conditions, as in medical-imagin encourages the ground-truth region proportion, which makes it generally better option in complex scenarios with diverse class proportions. Furthermore, we proposed a principled solution, which enables to control the label-marginal bias via Li and KL regularizers that encourage the target class proportions, while improving training stability. Our flexible formulation enables the minority classes to have better influence on training, without losing adaptability to medium-to -oiiluummtt---ggee large regions. Exten- sive experiments on a natural and medical imaging datasets validate the theoretical analysis in this paper, as well as the effectiveness of the presented solution.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image transformers. arXiv preprint arXiv:2106. 2025444466444 2021.
D.P. Bertsekas. Nonlinear Programming. Athena Scientific, Belmont, MA, 1995.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. ECCV, 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Jose Dolz, Christian Desrosiers, and Ismail Ben Ayed. 3d fully convolutional networks for subcor- tical segmentation in mri: large-scale study. Neurolmage, 170:456 4700 2018.
K. He, X. Zhang, Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
Fabian Isensee, Jens Petersen, André Klein, David Zimmerer, Paul F. Jaeger, Simon Kohl. Jakob Wasserthal, Gregor Köhler. Tobias Norajitra, Sebastian J. Wirkert, and Klaus H. Maier-Hein. nuu-net: Self-adapting framework for u-net-based medical image segmentation. arXiv preprint arXiv: 150300233,. 2018.
D. Jha, M. A. Riegler, D. Johansen, P. Halvorsen, and H. D. Johansen. Doubleu-net: A deep convolutional neural network for medical image segmentation. In CBMS, 2020.
Michael J. Kearns, Yishay Mansour, and Andrew Ng. An information and soft assignment methods for clustering. In UAI, 1997.
teformation-theoreticc theoretic analysis of hard
Hoel Kervadec, Houda Bahig, Laurent Letoumeau- Jose Dolz, and Ismail Ben Ayed. Be- yond pixel-wise supervision for segmentation: A few global shape descriptors might be surpris- ingly good! In MIDI,, 2021a.
Hoel Kervadec, Jihene Bouchtiba, Christian Desrosiers, Eric Granger, Jose Dolz, and Ismail Ben Ayed. Boundary loss for highly unbalanced segmentation. Medical Image Analysis, 67:10185) 2021b.
Junmo Kim, John W. Fisher III, Anthony J. Yezzi, Müjdat Çetin, and Alan S. Willsky. A nonpara- metric statistical method for image segmentation using information theory and curve evolution. 2005.
IEEE Transactions on Image Processing, (4(10):14866
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollar. Panoptic segmen- tation. In CVPR, June 2019.
T. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. CVPR, 2017.
Tsung-Y Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Araad Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen A. W. M. van der Laak, Bram van Ginneken, and Clara I. Sánchez. survey on deep learning in medical image analysis. Medical Image Analysis, 42: 60-88, 2017.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Jun Ma, Jianan Chen, Matthew Ng. Rui Huang, Yu Chen Li, Xiaoping Yang, and Anne L. Martel. Loss odyssey in medical image segmentation. Medical Image Analysis, 71:102035, 2021.
Fausto Milletari, Nassir Navab, and Seyed-Ahman Ahmadi. V-net: Fully convolutional neural net- works for volumetric medical image segmentation. In 3DV, 2016.
10

---

Under review as a conference paper at ICLR 2022
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, 2015.
Carole Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and Manuel Jorge Cardoso. Gen- eralised dice overlap as deep learning loss function for highly unbalanced segmentations. In DIMIAMI-CDDS MICCAI, 2017.
Saeid Asgari Taghanaki, Yefeng Zheng. S Kevin Zhou, Bogdan Georgescu, Puneet Sharma, Daguang Xu, Dorin Comaniciu, and Ghassan Hamarneh. Combo loss: Handling input and out- put imbalance in multi-organ segmentation. Computerized Medical Imaging and Graphics, 75: 24-33, 2019.
Meng Tang, Dmitrii Marin, Ismail Ben Ayed, and Yuri Boykov. Kernel cuts: Kernel and spectral clustering meet regularization. International Journal of Computer Vision, 12765):477-511, 2019.
Andrew Tao, Karan Sapra, and Bryan Catanzaro. Hierarchical multi-scale attention for semantic segmentation. arXiv preprint arXiv:2005. 10821, 2020.
Qijie Wei, Xirong Li, Weihong Yu, Xiao Zhang, Yongpeng Zhang, Bojie Hu, Bin Mo, Di Gong, Ning Chen, Dayong Ding, and Youxin Chen. Learn segment retinal lesions and beyond. In ICPR, 2020.
Ken C. L. Wong, Mehdi Moradi, Hui Tang, and Tanveer F. Syeda-Mahmood. 3d segmentation with exponential logarithmic loss for highly unbalanced object sizes. In MICCAI, 2018.
Zifeng Wu, Chunhua Shen, and Anton van den Hengel. Bridging category-level and instance-level semantic image segmentation. arXiv preprint A555::6005008885 2016.
Michael Yeung, Evis Sala, Carola-Bibiane Schönlieb, and Leonardo Rundo. mixed focal loss function for handling class imbalanced medical image segmentation. arXiv preprint arXiv:2102.04525, 2021.
Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic seg- mentation. In ECCV, 2020.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia. Psanet: Point-wise spatial attention network for scene parsing. ECCV, 2018.

---

Under review as a conference paper at ICLR 2022
A PROOFS
A.1 PROPOSITION
Proof. Let write the label-marginal bias term in the logarithmic Dice as a vector-valued function of probability simplex vector p:
s(p) k log(p + y) for (f*)ISKSK E AAK
(6)
where symbol T denotes transpose and 1 is the K-dimensional vector of ones. Function g is concave because its Hessian is negative semi-definite matrix: The Hessian of y is a diagonal matrix whose diagonal elements given by - and, hence, are all non-positive. Therefore, using Jensen's inequality and the fact that is within the simplex, we have the following lower bound on penalty in Eq. (6):
g()) -
Σ Pree
PRg(es)
where e₁ E {0, 1] denote the k-th vertex of the simplex: the k-th component of e₁ is equal 1 while the other components all equal O. Now, recall the definition of simplex vector t t= (t)) &g = 1 when 0y maxisksk Oh and Eg = 0 otherwise. Given this definition, one could easily verify the following fact:
((x)) 2 g() Ak
(8)
To see this, let j denotes the integer verifying Úú = maxisa5k ūn and let k: f Then, we have:
g(cx) - g(t) = log 1 + ( log 1
93 20 0.
This is due to the fact that function log (1 1) is monotonically decreasing in [0, 1] and ÿk Új-
Now, combining inequalities and 8 and using the fact that Σ 1==1, we obtain:
(()) 2 g(t)
(10)
L
A.2 PROPOSITION 2
Proof. Considering generative view of the prediction model, random variable F associated with the learned features continuous, while the random variable describing the labels, i.e., K., takes its possible values in a finite set {l...., K}. Then, the marginal distribution of the labels could be empirically estimated by the GT proportion of each segmentation region (as listed in Table 1) (Ki
al. 2005)
P(CC k) = 0x
(11)
[2
Also, we express the conditional entropy of the learned features follows:
K
H(F(K) = Σ ()
1
Sa H(FKK k)
(12)
Ω
with each H(FK k) given by:
H(FI = k) =
((f KC = k) log ((F) = k)dF
(13)
Hereafter, for notation simplicity, we omit IC and use H(F(k) instead of H(FFKC = 1b). Also, we use IP((*Ik)) instead of (P(F KC k).
12

---

Under review as a conference paper at ICLR 2022
To estimate the conditional entropy in Eq. (13) let refer to the following well known Monte-Care Carlo estimation (Kearn et al. 1997 Tang et al. 2019))
Monte-Carlo estimation. For any discrete set of points S L2 any function g and any feature embedding f, we have
J ((f))(((S))))))))))))))))))))(
g(()) g(.)
(14)
S
iES
where f denotes feature vector at point i, and P([[S) stands for the density of {f,, E S}.
Therefore, applying Montre-Carlo to ()) in Eq. we can re-write Eq. as follows:
H(FIK)
(og(P((())))
(15)
S2 iEf
Furthermore, using Bayes rule P(f (k) [2x log(fs), we obtain
Pik in addition the fact that Eien, log(fx)
P
Pik Pke
H(FIKC [22
log
k
inf
K
K Σ
log (pix)
+
Σ
log
(px)
[22]
[2
k
k
iEs,
CE
Σ
S2 log(Es)
K
CE +
Σ
DK log(
(16)
Finally, due to the definition of the label-marging KL divergence, we have
K
DKL(y/lp)
Uh log
yk
c
Σ In log(De)
(17)
This yields:
CE E M(FIKC + (xl(y))p)
(18)
In summary, we give an information-theoreticc prospective of CE. The entropy term can be con- sidered as a ground-truth matching term, while the label-marginal KL term avoids trivial solutions and encourages the proportions of the predicted segmentation regions match the ground-truth proportions.
B THE BINARY SEGMENTATION CASE
In the two-class (binary) segmentation case, Dice might be used for the foreground region only (Mil- letari et al. 2016). Similarly to the multi-class case discussed in the paper, a single Dice term also decomposes into ground-truth matching term and mabel-marging ing extremely imbalanced binary segmentations. For this specific case, the logarithmic Dice and CE could be written summations over the foreground and background segmentation regions:
penalty, with the latter encourag-
(og(Dice))
log
log Σ pa + 12
(19)
Foreground matching: DF,
Label marginal bias: DB,
13

---

Under review as a conference paper at ICLR 2022
CE
log Pa
log(1 Pa)
(20)
12
Enn
SEf
Foreground matching: CEi
Background matching: CEa
In Eq. (19) the term DB, can be expressed, up to an additive constant, as a function of the marginal probability of the foreground class (k 1) as follows:
DB1 log(Lı + 51)
(21)
Clearly, the marginal probability Pi measures the predicted proportion of pixels within the fore- ground region. This term reaches minimum when the foreground region empty (P,1 = 0v4).
Therefore, since log is monotonically increasing, minimizing term DB₁ in Eq. (19) introduces bias preferring small foreground structures. Note that this label-margnn marginal term in the logarithmic Dice loss is important avoid trivial solutions: when using the penground-matchiin term alone, the model may assign all the pixels in the image to the foreground region.
The foreground maatrhhnmmmmthhnngg terms, CE₁ and DF1, are closely related, with the former being and upper bound on the latter, due to Jensen's 's inequality: DF1 S CE2, Both foreground-matching terms are monotonically decreasing functions of each softmax and reach their global minimum when all the softmax predictions in the ground-truth foreground are equal to 1 (ie. reach their target). Hence, the matching terms Dice and CE can be viewed as two different penalty functions for imposing the same equality constraints, Pı1 = 1, Vi E 22,, thereby encouraging the predicted foreground to include the ground truth foreground.
C DISTRIBUTION OF REGION PROPORTIONS ACROSS CLASSES
Cityscapes
Retinal Lesions
miplerrtt
Región
0.1
rerson
Pal
20.2
30.0
80.0
Region proportion (%6
Figure C.I: The distribution of region proportions for the 19 classes in Cityscapes dataset (Left) and Retinal Lesions dataset (Right). The dashed line in the right figure indicates the average region proportion for Retinal Lesions. Best seen in color.
In Fig. C.I we present the distribution of region portions for each class in Cityscapes and Retinal Lesions. Cityscapes has highly variable region sizes (ranging from an average of 1.0% for rider and motorbike proportion is 3.8%. there are some examples with large region proportions (above 50.0%), and the standard deviation is 6.6%. Thus, these plots present the significant heterogeneity on different classes found across these two data sets.
average of 32.6% for road). In Retinal Lesions, although the average region
D THE TEMPERATURE SCALING
In our implementation, we employ a modified soft-max function with temperature scaling param- eter when computing the predicted region proportion Ωx (also referred to as the predicted label- marginal probability, as in Table I):
T-
s(z);
(22)
Σ5 ET
14

---

Under review as a conference paper at ICLR 2022
10-
58 10
2=
02
Figure D.1: Comparison of soft-max functions with different temperature scaling paramter т. This shows that the output confidence increases with larger т. Best seen in color.
where = (zi) K is the input vector of the soft-max function, and >> O acts the temperature hhper-pprammae parameter. High values of r > yield high confidence the soft-max prediction, as shown in Fig. D.I] They push the softmax vector towards the vertices of the simplex, with prediction values approaching either 0 or 1. As a result, this enables better estimate of the actual region proportion (or relative size). Throughout all experiments, set 10.
STUDY OF THE TRADE-OFF WEIGHT IN THE COMPOSITE LOSSES
0.55
0.45
DSC
0.35
CE++1
0.30
CE+KL
CE+DB;
0.25
CE-log(Dice,)
0.20
10-3
10
10-1
10a
Figure E.1: The performances of different composite losses the test set of Retinal Lesions with different values the trade-off apper-parameter A. The network we used fixed to Res50-FPN,
The proposed method is type of composite loss. Thus, the trade-off hyper-parameter X could in- fluence the final performance. Fig. E.11 shows the performances of the different composite losses examined in the paper, with different trade-off' hyper-parameter values. While we plot the perfor- mances the test of Retinal Lesions, the behaviour on the validation set is similar, with the best trade-off parameters reached at the same value for both the validation and test sets.
From the curves, we can notice that C1 is a better choice than KL divergence for the proposed method because of its better stability. This may relate to its gradient properties and stability at the vicinity of 0 (see Fig. 2 in the main document). Thus, we can use a relatively larger weighting values for CE+Li (best performance achieved when À = 1.0). Observe that CE+DkL leads to comparable results at its best when À 0.1, but it seems more sensitive to the trade-off hyper- parameter. For the loss integrating CE with the Dice bias term, i.e., CE+DB,, we demonstrate that can achieve performances similar to CE-log (Dice), but it drops significantly with high weighting values (A > 0.1). This might be due its gradient characteristics. Comparing to the widely suggested composite loss of CE and Dice, our method deliver better performance with the same hyper-parmeter budget.
15

---

Under review as a conference paper at ICLR 2022
F PER-CLASS RESULTS ON CITYSCAPES TEST SET
Table F.I: Per-class Results on Cityscapes test set. (Top: Res50-FPN, Bottom: Res10I-FPN) The second row indicates the average region proportion (mRegPProp) for each class. mloU denotes the mean IoU score over all classes.
road swalk build. wall fence pole
tlight
sign
lerrain
sky
person
rider
car
truck
bus
train mhike bike mloo
mRegProp (%)
CE
97.5
79.3
90.3
45.0
46.3
53.5
61.6
67.3
92.1
70.1
93.5
78.4
53.3
93.7
41.4
48.8
38.2
55.2
67.1
66.97
Focal loss
97.6
79.1
89.8
41.3
43.6
49.9
57.7
63.8
91.7
68.4
94.1
75.7
47.3
93.0
37.8
44.9
35.6
50.5
63.8
64.50
CE- (Dge))
97.4
79.2
90.3
43.8
45.0
55.1
67.0
70.3
91.8
70.0
94.0
79.4
56.0
93.3
39.7
50.8
32.2
51.2
68.1
67.08
CE + KL. (Ours)
97.6
79.6
90.4
45.2
45.0
53.1
61.3
66.9
92.0
70.5
94.1
78.5
54.7
93.6
48.8
59.1
45.8
55.5
67.0
68.35
CE + Cj (Ours)
97.6
79.4
90.4
44.1
45.8
53.7
61.3
67.2
92.1
70.2
94.3
78.8
54.4
93.7
47.8
58.4
46.8
55.4
67.3
68.35
CE
97.8
81.3
90.9
45.6
50.2
55.7
64.2
67.1
92.3
70.9
94.6
79.7
57.6
94,2
51.7
58.0
41.5
58.1
68.5
69.57
Rocal loss
97.9 81.3
90.6
46.4
46.9
52.4
59.9
65.7
94.4
77.3
53.0
93.6
48.7
35.9
54.5
66.5
66.79
CE (icee)
97.7
80.0
90.5
43.8
48.5
55.8
68.1
70.9
91.7
70.5
94.2
80.6
61.4
93.7
46.9
57.6
40.6
59.2
70.4
69.58
CE KL (Ours)
97.9
81.4
91.2
48.7
49.7
55.5
63.7
70.6
94.6
79.9
57.1
94.3
54.5
61.1
43.5
58.3
68.4
70.07
CE + 4 (Ours)
98.0
81.9
91.2
48.7
50.3
55.7
63.8
68.3
92.
70.7
94.5
79.8
57.5
94.2
52,9
60.3
44.0
59.2
68.7
70.11
In Table F.1 we report the detailed per-class results on the Cityscapes test set. The proposed meth- ods outperform related losses both network settings. In term of mloU, achieve 68.35% and 70.11% on Res50-FPN and Res101-FPN, respectively. Compared to the baseline CE, our method yields improvement for most of the minority classes, like fence, train and motorbike, while it shows better adaptability for diverse classes than the Dice loss.
G MORE QUALITATIVE RESULTS ON THE RETINAL LESIONS DATASET
CE
Dice
CE - logl Dice,)
CE + L1 (Ours)
4
0
2
0.781
0.776
0.798
0.818
0.792
0.757
0.804
0.812
0.631
0.120
0.540
0.667
0.539
0.698
0.847
0.726
Figure G.1: Additional visual results the Retinal Lesions dataset. The ground-truth is depicted in the first column. At the bottom of each image, we indicate the corresponding obtained DSC score.
Fig. gives more qualitative examples from Retinal Lesions Dataset. It shown that our method able to adapt to the highly variant segmenting regions from small to large, while Dice loss obvi- ously degrades in the case of large regions like the example in the third row.
16


Reviews:
[{"review_id": "nRJbmIEqP-H", "reviewer": "Reviewer_P7HM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The paper presents an interesting analysis for the CE and Dice loss function, indicating their relationships in terms of the called label-marginal bias. Base on this, the paper proposes an alternative variation of the CE loss function for segmentation. \n\nWeaknesses\n- Regarding the performance on the medical dataset, are the performance gains similar in other medical datasets, for example, organ segmentation (where some structures can be considerably smaller compared with the background, especially when considering 3D input data)? It is possible to discuss the applicability to different datasets, and in what cases, can the proposed function more/less appropiate\n\n- Some points that might not be clear are the relation of the CE, DSC, and the label marginal concerning the performance of models in computer vision, and medical images. In this sense, it would be interesting to elaborate on this regard. \n\n- Do the CE-based functions used in the experiments correspond to the weighted versions?", "summary_of_the_paper": "The paper presents an analysis of the cross-entropy (CE) and dice loss (DSC) for segmentation tasks,  in terms of the label-marginal bias.  According to the analysis, DSC prefers small regions, while CE encourages a prediction that has a similar proportion to the ground truth. Considering this analysis, the work proposes an alternative loss function consisting of the cross-entropy augmented with a regularization term (L1 or KL divergence) that operates on the prediction of the network. Different experiments compare the presented loss function of previous proposals.", "summary_of_the_review": "Segmentation loss functions are employed in different medical and real-world datasets, particularly the medical domain can present a wide variability in available datasets.  It might be interesting to discuss results on additional datasets, compared with the DSC (as the paper discussed, it is widely used in the medical domain). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VFrvmJixzSN", "reviewer": "Reviewer_QsrX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n+ The question addressed in this work is inherently an interesting one. Studying the relation between two popular loss functions which are understood to be somehow complementary to each other and attempting to make connections between them can be an important contribution.\n+ The decomposition of the loss functions into the ground truth penalty and label bias terms is reasonable, and the proofs provided in A1. and A2 are mostly adequate. \n+ Experiments with the L1 regularization for the class imbalance dataset (retinal segmentation) show promising results. \n\nWeaknesses: \n\n\n- One of my major concerns is in the proposed strategies to modulate the weighting of the label bias term in CE. The key argument is that the relative contribution of the label-marginal term is of smaller magnitude in class imbalance cases. This is not obvious from the Propositions 2. How is this conclusion drawn? \n\n- Two regularization strategies are proposed; one of which is KLD term D(y||p). But, in  Eq. 4 CE already decomposes into entropy and the same KLD term. Are these two KLD terms the same? If so, what is the influence of the additional regularization? Perhaps it only serves as a scaling of the label bias term. In that case, how does the L1 regularization influence the scaling of the label bias term. \n\n- L1 regularization is posited to be better than KLD regularization, and one of the reasons cited is its \"gradient properties and stability\".  This is somewhat unconvincing. Is this in comparison with KLD? \n\n- The specific choice of the two types of regularization terms is simply stated. Why were these two types of regularization schemes chosen? What was the motivation? How were these choices made? Is there a theoretical justification for using the L1 term? \n\n - The decomposition of the losses into two competing terms and their interpretations are reasonable. However, for Dice loss, the arguments presented for why the label marginal bias term in Eq. 1 end up favouring extreme class imbalance is unclear. Additional explanation can be helpful here.\n\nMinor comments: \n- An important point in the paper is the competing nature of the ground truth penalty and label bias term. Their competing nature should be elucidated further. It is not immediately obvious in case of the CE (Eq.4). and can be helpful to readers. ", "summary_of_the_paper": "This work presents a decomposition of two commonly used segmentation losses (cross entropy/CE and Dice losses) into two competing components. The two components are interpreted as ground truth matching and label-marginal penalties. This work argues that the label-marginal penalties in Dice loss favour extreme  class imbalances, and hence their widespread use in medical imaging segmentation tasks which encounter class imbalance more commonly. The label marginal terms in CE is argued to be better as this term matches the label marginals to the ground truth distributions. The scaling between the two terms in CE is presented as the problem and this work proposes an L1 regularization to overcome the label bias problem in CE. Experiments on two segmentation tasks show interesting influence of these regularizations. ", "summary_of_the_review": "The decomposition of the losses into two terms is interesting and largely rigorously studied. However, the regularization schemes proposed are not well justified.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EIl60MUeQ-g", "reviewer": "Reviewer_3tvn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: Good motivation, Good organization, Readable \nWeakness: The paper analyzes Dice and CE losses while there are many recent losses ignored. \nThe proposed loss is mainly based on CE loss and adds one more regularization term. \nThe proposed loss is tested on Retinal Lesions dataset which groundtruth RoI is labeled as an oval which is not accurate for the segmentation task.\nCompare to other standard loss, i.e. Dice and CE, the proposed loss does not provide good results and improve with a tiny gap.", "summary_of_the_paper": "The paper provides an explicit bound relationship an information theoretic analysis, which uncover hidden label-marginal biases\nThe paper proposes LABEL-MARGINAL BIASES which combines CE loss with KL divergence or L1 norm to address class imbalance but without losing generality.\n\nThe evaluation is conducted on Retina dataset and Cityscapes.", "summary_of_the_review": "Novelty\nClarity\nSignificance", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "AcUDiS2eYOK", "reviewer": "Reviewer_njoP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1. This paper provides theoretical perspective about why Dice loss is better for smaller structure and CE loss is better at matching prediction distribution to training data distribution. While both phenomena are straightforward intuitively, a theoretical perspective from label marginal bias is appreciated. \n2. This paper proposes a novel loss function that uses CE loss with a regularization term to correct label proportions. \n3. Experiments on both natural and medical image segmentation problems with imbalanced classes were conducted to show that the proposed loss achieves better accuracy than Dice or CE loss themselves. \n\nWeakness: \n1. The proposed loss regularization term needs a hyper-parameter \\lambda to control the weight of regularization. It is understandable that the optimal \\lambda value is dataset-dependent due to different class distributions in different datasets. However, whether such \\lambda is also network-dependent is not clear. \n\nIn this paper, only ResNet-FPN and ResNet-Unet are used in the experiment. More extensive experiment on different architectures using same lambda values may be needed. Or theoretical argument about why \\lambda is only dataset-dependent and not network-dependent would be very helpful. \n\n2. A useful baseline to compare to is \"CE+w * Dice\", where w is a weighting coefficient. As the proposed loss is \"CE+\\lambda * L1\", the \"CE+w * Dice\" can provide probably similar performance with the same amount of hyper-parameter to tune. \n\n3. Only two datasets, Cityscape and retinal lesions, are presented. It would be great if more public datasets with different class distribution characteristics can be used, so that we can gain more insights about how to tune \\lambda to match datasets with different class distributions. \n\n4. I agree with other reviewers on that the \"segmentation GT\" of the retinal lesion dataset is so poor that it cannot even be called segmentation annotation, but rather just detection annotation. In fact, without proper medical knowledge, I would say CE in Figure 3 is returning a better segmentation as they differentiate \"normal\" from \"not-so-normal\" more closely. Thus, higher DSC on this dataset is likely due to segmentation shape being smoother/more elliptical. \n\n5. The Dice-related losses (\"1-Dice\" and \"-log Dice\") on **validation set** were highlighted by authors in Table 3, with extensive discussions in the first paragraph on page 9. However, Dice losses (\"1-Dice\" and \"-log Dice\") on **test set** were excluded in Table 4, without any discussion. I am not sure whether last paragraph of Sec.3 was for validation or test set. But either way, I think including a detailed comparison of Dice-related losses on test set is important, as the whole paper is essentially studying Dice loss and CE loss. \n", "summary_of_the_paper": "This paper proposed theoretical analysis showing Dice loss has a bias preferring small structures, while losing flexibility of dealing with arbitrary class proportions. On the hand hand, the cross-entropy loss encourages proportions of the predicted segmentation to match the ground-truth proportions. A new loss function to control the label-marginal biases was proposed as sum of CE and weighted L1/KL divergence regularization. ", "summary_of_the_review": "The theoretical perspectives states the intuitively well-understood fact that Dice loss is better at capturing small objects while CE provides prediction distribution that matches training data. Thus, although it is an alternative perspective from label marginal bias, it did not lead to significant new knowledge about both losses. The proposed new loss, although achieving better accuracy on two datasets with two different networks, is not sufficiently validated in terms of how \\lambda choice should be made regarding to different datasets and network structures. The comparison against \"CE + w*Dice\" is also strongly recommended, both theoretically and experimentally. The lack of Dice-related losses results on Cityscape dataset test set needs to be addressed. The poor quality of GT makes the retinal lesion dataset inappropriate to conduct segmentation experiments on.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission evaluates the relationship between (logarithmic) Dice loss and cross-entropy loss, arguing for a similar decomposition into ground truth and "hidden label-marginal biases."  The submission received mixed reviews, with two reviewers voting for rejection, and two feeling that it is marginally above the acceptance threshold.  Setting aside the numerical scores, there are reasons to believe that this submission, while interesting, has shortcomings that limit its relevance to the wider ICLR community.  These include
- Very many losses have been proposed for imbalanced classification / (medical) image segmentation, such as Jaccard and Tversky index or ranking measures, although admittedly Dice is probably the most popular in the medical imaging literature due to historic reasons.  Arguably, Dice is less well-behaved from a theoretical perspective compared to other options (e.g. it does not even form a metric), and may not be the most relevant point of departure for a representation learning conference.  The literature review misses many relevant papers on such losses, including papers that specifically are focused on the relationship between Dice and cross-entropy, e.g. Eelbode et al., IEEE-TMI 2020 and citations therein.
- The empirical results do not show substantially improved results compared to baselines.

On the balance, this does not cross the threshold for acceptance to a competitive venue such as ICLR.
Decision: Reject