The hidden label-marginal biases of segmentation losses

1 INTRODUCTION . Semantic segmentation is one of the most investigated problems in computer vision , and has been impacting a breadth of applications , from natural-scene understanding ( Cordts et al. , 2016 ; Kirillov et al. , 2019 ) to medical image analysis ( Litjens et al. , 2017 ; Dolz et al. , 2018 ) . In the recent years , deep learning methods have dominated the field , as a result of the great capacity of Convolutional Neural Networks ( CNN ) ( He et al. , 2016 ) to automatically learn representations from large-scale data sets ( Long et al. , 2015 ; Ronneberger et al. , 2015 ; Zhao et al. , 2017 ; Chen et al. , 2018 ; Yuan et al. , 2020 ) . Semantic segmentation is often stated as a pixel-wise classification task , following the optimization of a loss function expressed with summations over the ground-truth regions , as in the standard Cross-Entropy ( CE ) loss . A relevant aspect of segmentation problems is class imbalance , i.e. , unequal proportions of the segmentation regions , which may cause large-region terms in the objective to completely dominate small-region ones . A representative example is the popular Cityscapes dataset ( Cordts et al. , 2016 ) , where the average proportions of some classes , such as motorcycle or bicycle , are below 1 % . In these scenarios , besides specifically designed CNN architectures or training schemes ( Tao et al. , 2020 ; Bao et al. , 2021 ) , the loss function to be minimized during learning plays a critical role , and has triggered a large body of research works in the last years ( Lin et al. , 2017 ; Wong et al. , 2018 ; Kervadec et al. , 2021b ; Milletari et al. , 2016 ; Sudre et al. , 2017 ; Kervadec et al. , 2021a ) . There exists a great diversity of loss functions for image segmentation , which can be categorized into two main families , and are arguably variants of CE , the Dice loss ( Ma et al. , 2021 ; Yeung et al. , 2021 ) , or combinations of both ( Wong et al. , 2018 ; Taghanaki et al. , 2019 ) . The first family is mo- tivated by distribution measures , i.e. , CE and its variants , and is directly adapted from classification tasks . To deal with class imbalance , various extensions of CE have been investigated , such as increasing the relative weights for minority classes ( Ronneberger et al. , 2015 ) , or modifying the loss so as to account for performance indicators during training , as in the popular Focal loss ( Lin et al. , 2017 ) or TopK loss ( Wu et al. , 2016 ) . The second main family of losses is inspired by geometrical metrics . In this category , the most popular losses are linear Dice ( Milletari et al. , 2016 ) and its extensions , such as the logarithmic ( Wong et al. , 2018 ) or generalized ( Sudre et al. , 2017 ) Dice loss . Borrowing the idea of the weighted CE , the latter introduces class weights to increase the contributions of the minority classes . These loss functions are motivated by the geometric Dice coefficient , which measures the overlap between the ground-truth and predicted segmentation regions . In the literature , to our knowledge , there is no clear consensus as to which category of losses is better , with the performances of each varying across data sets and applications . It has been empirically argued that the Dice loss and its variants are more appropriate for extreme class imbalance , and such empirical observations are the main motivation behind the wide use and popularity of Dice in medical-imaging applications ( Milletari et al. , 2016 ; Jha et al. , 2020 ) . CE , however , dominates most recent models in the context of natural images ( Chen et al. , 2018 ; Zhao et al. , 2017 ; 2018 ; Yuan et al. , 2020 ) . Therefore , beyond experimental evidence , there is a need for a theoretical analysis that clarifies which segmentation loss to adopt for a given task , a decision that affects performance significantly . On the surface , these two categories of losses ( i.e. , geometry-based vs. distribution-based ) seem unrelated . Furthermore , an important body of work in the literature suggests that CE and Dice are complementary , which has motivated composite losses integrating both , e.g. , ( Wong et al. , 2018 ; Taghanaki et al. , 2019 ) . Such composite losses perform very competitively in extremely imbalanced segmentation , as shown by ( Wong et al. , 2018 ; Taghanaki et al. , 2019 ) , among several other recent works ( Isensee et al. , 2018 ; Ma et al. , 2021 ) . In this paper , we provide a constrained-optimization perspective showing that , in fact , CE and Dice share a much deeper connection than previously thought : They both decompose into label-marginal penalties and closely related ground-truth matching penalties . Our theoretical analysis highlights encoded hidden label-marginal biases in Dice and CE , and shows that the main difference between the two types of losses lies essentially in those label-marginal biases : Dice has an intrinsic bias preferring very small regions , while CE implicitly encourages the right ( ground-truth ) region proportions . Our results explain the wide experimental evidence in the medical-imaging literature , whereby using or adding Dice losses brings improvements for imbalanced segmentation with extremely small regions . It also explains why CE dominates natural-image problems with diverse class proportions , in which case Dice might have difficulty adapting to different label-marginal distributions ( see Table 3 and the examples depicted in Fig . 1 ) . Based on our theoretical analysis , we propose principled and simple loss functions , which enable to control explicitly the label-marginal bias . Our solution integrates the benefits of both categories of losses , mitigating class imbalance but without losing generality , as shown in the examples in Fig . 1 . Our contributions are summarized as follows : • Showing through an explicit bound relationship ( Proposition 1 ) that the Dice loss has a hidden label-marginal bias towards specific extremely imbalanced solutions , preferring small structures , while losing the flexibility to deal effectively with arbitrary class proportions . • Providing an information-theoretic perspective of CE , via Monte-Carlo approximation of the entropy of the learned features ( Proposition 2 ) . This highlights a hidden label-marginal bias of CE , which encourages the proportions of the predicted segmentation regions to match the ground-truth proportions . • Introducing new loss functions to control label-marginal biases : Our losses integrate CE with explicit regularization terms based on L1 or the KL divergence , which encourage label marginals to match target class proportions . • Comprehensive experiments and ablation studies over different losses and applications , including natural and medical-imaging data , validate our theoretical analysis , as well as the effectiveness of our explicit label-marginal regularizers . 2 FORMULATION . Semantic segmentation is often stated as a pixel-wise classification task , following the optimization of a loss function for training a deep network . In Table 1 , we present the notations , formulations and approximations used in our subsequent discussions . Besides the basic notations of the task ( such as networks predictions ) , we explicitly include the loss functions , label-marginal regularizers and information-theoretic quantities that will be discussed in the following sections . We note that , to facilitate the reading of our analysis , we write the CE and Dice losses in a non-standard way using summations over the ground-truth segmentation regions , rather than as functions of the labels . Also , while we provide the CE loss for all segmentation regions , we give Dice for a single region . This is to accommodate two variants of the Dice loss in the literature : in the binary case , Dice is typically used for the foreground region only ( Milletari et al. , 2016 ) ; in the multi-region case , it is commonly used over all the regions ( Wong et al. , 2018 ) . Finally , to simplify notation , we give all the loss functions for a single training image , without summations over all training samples ( as this does not lead to any ambiguity , neither does it alter the analysis hereafter ) . In the training iterations , we use the mean values across all the training samples via standard mini-batch optimization . 2.1 DEFINITION OF LABEL-MARGINAL BIASES AND PENALTY FUNCTIONS . In the following , we analyse the label-marginal biases inherent to CE and Dice losses , and show that the main difference between the two types of losses lies essentially in those label-marginal biases . To do so , we provide a constrained-optimization perspective of the losses : We define a label-marginal bias as a soft penalty function for the hard equality constraint p = t , where t is a given ( fixed ) target distribution . Such a penalty encourages the predicted label-marginal p to match a given target distribution t. In the general context of constrained optimization , penalty functions are widely used ( Bertsekas , 1995 ) . In general , penalty methods replace equality constraints of the form p = t by adding a term g ( p ) to the main objective being minimized . Such a penalty function g increases when p deviates from target t. By definition , for the constraint p = t , with the domain of p being probability simplex ∆K , a penalty g ( p ) is a continuous and differentiable function , which reaches its global minimum when the constraint is satisfied , i.e. , it verifies : g ( t ) ≤ g ( p ) ∀p ∈ ∆K . 2.2 THE LINK BETWEEN CROSS ENTROPY AND DICE . To ease the discussion in what follows , we will start by analyzing the link between CE and the logarithmic Dice , along with the label-marginal bias of the latter ( Proposition 1 ) . Then , we discuss a bounding relationship between the different Dice variants . Finally , we will provide an informationtheoretic analysis , which highlights the hidden label-marginal bias of CE ( Proposition 2 ) . Let us consider the logarithmic Dice loss in the multi-class case . This loss decomposes ( up to a constant ) into two terms , a ground-truth matching term and a label-marginal bias : − K∑ k=1 log ( Dicek ) c = − K∑ k=1 log 1 |Ωk| ∑ i∈Ωk pik ︸ ︷︷ ︸ Ground-truth matching : DF + K∑ k=1 log ( p̂k + ŷk ) ︸ ︷︷ ︸ Label-marginal bias : DB ( 1 ) where c= stands for equality up to an additive and/or non-negative multiplicative constant . The ground-truth matching term in Eq . ( 1 ) is a lower bound on the cross-entropy loss ( CE ) due to Jensen ’ s inequality and the convexity of function − log ( x ) : DF ≤ CE . Therefore , minimizing CE could be viewed as a proxy for minimizing term DF that appears in the logarithmic Dice . In fact , from a constrained-optimization perspective , DF and CE are very closely related and could be viewed as two different penalty functions enforcing the same equality constraints : pik = 1 , ∀i ∈ Ωk , ∀k . Both DF and CE are monotonically decreasing functions of each softmax and reach their global minimum when these equality constraints are satisfied . Therefore , they encourage softmax predictions pik for each region Ωk to reach their target ground-truth values of 1 . Of course , this does not mean that penalties CE and DF yield exactly the same results . The difference in the results that they may yield is due to the optimization technique ( e.g. , different gradient dynamics in the standard training of deep networks as the penalty functions have different forms ) .

Summary: This work presents a decomposition of two commonly used segmentation losses (cross entropy/CE and Dice losses) into two competing components. The two components are interpreted as ground truth matching and label-marginal penalties. This work argues that the label-marginal penalties in Dice loss favour extreme  class imbalances, and hence their widespread use in medical imaging segmentation tasks which encounter class imbalance more commonly. The label marginal terms in CE is argued to be better as this term matches the label marginals to the ground truth distributions. The scaling between the two terms in CE is presented as the problem and this work proposes an L1 regularization to overcome the label bias problem in CE. Experiments on two segmentation tasks show interesting influence of these regularizations.