The hidden label-marginal biases of segmentation losses

1 INTRODUCTION . Semantic segmentation is one of the most investigated problems in computer vision , and has been impacting a breadth of applications , from natural-scene understanding ( Cordts et al. , 2016 ; Kirillov et al. , 2019 ) to medical image analysis ( Litjens et al. , 2017 ; Dolz et al. , 2018 ) . In the recent years , deep learning methods have dominated the field , as a result of the great capacity of Convolutional Neural Networks ( CNN ) ( He et al. , 2016 ) to automatically learn representations from large-scale data sets ( Long et al. , 2015 ; Ronneberger et al. , 2015 ; Zhao et al. , 2017 ; Chen et al. , 2018 ; Yuan et al. , 2020 ) . Semantic segmentation is often stated as a pixel-wise classification task , following the optimization of a loss function expressed with summations over the ground-truth regions , as in the standard Cross-Entropy ( CE ) loss . A relevant aspect of segmentation problems is class imbalance , i.e. , unequal proportions of the segmentation regions , which may cause large-region terms in the objective to completely dominate small-region ones . A representative example is the popular Cityscapes dataset ( Cordts et al. , 2016 ) , where the average proportions of some classes , such as motorcycle or bicycle , are below 1 % . In these scenarios , besides specifically designed CNN architectures or training schemes ( Tao et al. , 2020 ; Bao et al. , 2021 ) , the loss function to be minimized during learning plays a critical role , and has triggered a large body of research works in the last years ( Lin et al. , 2017 ; Wong et al. , 2018 ; Kervadec et al. , 2021b ; Milletari et al. , 2016 ; Sudre et al. , 2017 ; Kervadec et al. , 2021a ) . There exists a great diversity of loss functions for image segmentation , which can be categorized into two main families , and are arguably variants of CE , the Dice loss ( Ma et al. , 2021 ; Yeung et al. , 2021 ) , or combinations of both ( Wong et al. , 2018 ; Taghanaki et al. , 2019 ) . The first family is mo- tivated by distribution measures , i.e. , CE and its variants , and is directly adapted from classification tasks . To deal with class imbalance , various extensions of CE have been investigated , such as increasing the relative weights for minority classes ( Ronneberger et al. , 2015 ) , or modifying the loss so as to account for performance indicators during training , as in the popular Focal loss ( Lin et al. , 2017 ) or TopK loss ( Wu et al. , 2016 ) . The second main family of losses is inspired by geometrical metrics . In this category , the most popular losses are linear Dice ( Milletari et al. , 2016 ) and its extensions , such as the logarithmic ( Wong et al. , 2018 ) or generalized ( Sudre et al. , 2017 ) Dice loss . Borrowing the idea of the weighted CE , the latter introduces class weights to increase the contributions of the minority classes . These loss functions are motivated by the geometric Dice coefficient , which measures the overlap between the ground-truth and predicted segmentation regions . In the literature , to our knowledge , there is no clear consensus as to which category of losses is better , with the performances of each varying across data sets and applications . It has been empirically argued that the Dice loss and its variants are more appropriate for extreme class imbalance , and such empirical observations are the main motivation behind the wide use and popularity of Dice in medical-imaging applications ( Milletari et al. , 2016 ; Jha et al. , 2020 ) . CE , however , dominates most recent models in the context of natural images ( Chen et al. , 2018 ; Zhao et al. , 2017 ; 2018 ; Yuan et al. , 2020 ) . Therefore , beyond experimental evidence , there is a need for a theoretical analysis that clarifies which segmentation loss to adopt for a given task , a decision that affects performance significantly . On the surface , these two categories of losses ( i.e. , geometry-based vs. distribution-based ) seem unrelated . Furthermore , an important body of work in the literature suggests that CE and Dice are complementary , which has motivated composite losses integrating both , e.g. , ( Wong et al. , 2018 ; Taghanaki et al. , 2019 ) . Such composite losses perform very competitively in extremely imbalanced segmentation , as shown by ( Wong et al. , 2018 ; Taghanaki et al. , 2019 ) , among several other recent works ( Isensee et al. , 2018 ; Ma et al. , 2021 ) . In this paper , we provide a constrained-optimization perspective showing that , in fact , CE and Dice share a much deeper connection than previously thought : They both decompose into label-marginal penalties and closely related ground-truth matching penalties . Our theoretical analysis highlights encoded hidden label-marginal biases in Dice and CE , and shows that the main difference between the two types of losses lies essentially in those label-marginal biases : Dice has an intrinsic bias preferring very small regions , while CE implicitly encourages the right ( ground-truth ) region proportions . Our results explain the wide experimental evidence in the medical-imaging literature , whereby using or adding Dice losses brings improvements for imbalanced segmentation with extremely small regions . It also explains why CE dominates natural-image problems with diverse class proportions , in which case Dice might have difficulty adapting to different label-marginal distributions ( see Table 3 and the examples depicted in Fig . 1 ) . Based on our theoretical analysis , we propose principled and simple loss functions , which enable to control explicitly the label-marginal bias . Our solution integrates the benefits of both categories of losses , mitigating class imbalance but without losing generality , as shown in the examples in Fig . 1 . Our contributions are summarized as follows : • Showing through an explicit bound relationship ( Proposition 1 ) that the Dice loss has a hidden label-marginal bias towards specific extremely imbalanced solutions , preferring small structures , while losing the flexibility to deal effectively with arbitrary class proportions . • Providing an information-theoretic perspective of CE , via Monte-Carlo approximation of the entropy of the learned features ( Proposition 2 ) . This highlights a hidden label-marginal bias of CE , which encourages the proportions of the predicted segmentation regions to match the ground-truth proportions . • Introducing new loss functions to control label-marginal biases : Our losses integrate CE with explicit regularization terms based on L1 or the KL divergence , which encourage label marginals to match target class proportions . • Comprehensive experiments and ablation studies over different losses and applications , including natural and medical-imaging data , validate our theoretical analysis , as well as the effectiveness of our explicit label-marginal regularizers . 2 FORMULATION . Semantic segmentation is often stated as a pixel-wise classification task , following the optimization of a loss function for training a deep network . In Table 1 , we present the notations , formulations and approximations used in our subsequent discussions . Besides the basic notations of the task ( such as networks predictions ) , we explicitly include the loss functions , label-marginal regularizers and information-theoretic quantities that will be discussed in the following sections . We note that , to facilitate the reading of our analysis , we write the CE and Dice losses in a non-standard way using summations over the ground-truth segmentation regions , rather than as functions of the labels . Also , while we provide the CE loss for all segmentation regions , we give Dice for a single region . This is to accommodate two variants of the Dice loss in the literature : in the binary case , Dice is typically used for the foreground region only ( Milletari et al. , 2016 ) ; in the multi-region case , it is commonly used over all the regions ( Wong et al. , 2018 ) . Finally , to simplify notation , we give all the loss functions for a single training image , without summations over all training samples ( as this does not lead to any ambiguity , neither does it alter the analysis hereafter ) . In the training iterations , we use the mean values across all the training samples via standard mini-batch optimization . 2.1 DEFINITION OF LABEL-MARGINAL BIASES AND PENALTY FUNCTIONS . In the following , we analyse the label-marginal biases inherent to CE and Dice losses , and show that the main difference between the two types of losses lies essentially in those label-marginal biases . To do so , we provide a constrained-optimization perspective of the losses : We define a label-marginal bias as a soft penalty function for the hard equality constraint p = t , where t is a given ( fixed ) target distribution . Such a penalty encourages the predicted label-marginal p to match a given target distribution t. In the general context of constrained optimization , penalty functions are widely used ( Bertsekas , 1995 ) . In general , penalty methods replace equality constraints of the form p = t by adding a term g ( p ) to the main objective being minimized . Such a penalty function g increases when p deviates from target t. By definition , for the constraint p = t , with the domain of p being probability simplex ∆K , a penalty g ( p ) is a continuous and differentiable function , which reaches its global minimum when the constraint is satisfied , i.e. , it verifies : g ( t ) ≤ g ( p ) ∀p ∈ ∆K . 2.2 THE LINK BETWEEN CROSS ENTROPY AND DICE . To ease the discussion in what follows , we will start by analyzing the link between CE and the logarithmic Dice , along with the label-marginal bias of the latter ( Proposition 1 ) . Then , we discuss a bounding relationship between the different Dice variants . Finally , we will provide an informationtheoretic analysis , which highlights the hidden label-marginal bias of CE ( Proposition 2 ) . Let us consider the logarithmic Dice loss in the multi-class case . This loss decomposes ( up to a constant ) into two terms , a ground-truth matching term and a label-marginal bias : − K∑ k=1 log ( Dicek ) c = − K∑ k=1 log 1 |Ωk| ∑ i∈Ωk pik ︸ ︷︷ ︸ Ground-truth matching : DF + K∑ k=1 log ( p̂k + ŷk ) ︸ ︷︷ ︸ Label-marginal bias : DB ( 1 ) where c= stands for equality up to an additive and/or non-negative multiplicative constant . The ground-truth matching term in Eq . ( 1 ) is a lower bound on the cross-entropy loss ( CE ) due to Jensen ’ s inequality and the convexity of function − log ( x ) : DF ≤ CE . Therefore , minimizing CE could be viewed as a proxy for minimizing term DF that appears in the logarithmic Dice . In fact , from a constrained-optimization perspective , DF and CE are very closely related and could be viewed as two different penalty functions enforcing the same equality constraints : pik = 1 , ∀i ∈ Ωk , ∀k . Both DF and CE are monotonically decreasing functions of each softmax and reach their global minimum when these equality constraints are satisfied . Therefore , they encourage softmax predictions pik for each region Ωk to reach their target ground-truth values of 1 . Of course , this does not mean that penalties CE and DF yield exactly the same results . The difference in the results that they may yield is due to the optimization technique ( e.g. , different gradient dynamics in the standard training of deep networks as the penalty functions have different forms ) .

Summary: This paper proposed theoretical analysis showing Dice loss has a bias preferring small structures, while losing flexibility of dealing with arbitrary class proportions. On the hand hand, the cross-entropy loss encourages proportions of the predicted segmentation to match the ground-truth proportions. A new loss function to control the label-marginal biases was proposed as sum of CE and weighted L1/KL divergence regularization.