Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper provides an explicit bound relationship an information theoretic analysis, which uncover hidden label-marginal biases
The paper proposes LABEL-MARGINAL BIASES which combines CE loss with KL divergence or L1 norm to address class imbalance but without losing generality.

The evaluation is conducted on Retina dataset and Cityscapes.
main_review: Strength: Good motivation, Good organization, Readable 
Weakness: The paper analyzes Dice and CE losses while there are many recent losses ignored. 
The proposed loss is mainly based on CE loss and adds one more regularization term. 
The proposed loss is tested on Retinal Lesions dataset which groundtruth RoI is labeled as an oval which is not accurate for the segmentation task.
Compare to other standard loss, i.e. Dice and CE, the proposed loss does not provide good results and improve with a tiny gap.
summary_of_the_review: Novelty
Clarity
Significance
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents an analysis of the cross-entropy (CE) and dice loss (DSC) for segmentation tasks,  in terms of the label-marginal bias.  According to the analysis, DSC prefers small regions, while CE encourages a prediction that has a similar proportion to the ground truth. Considering this analysis, the work proposes an alternative loss function consisting of the cross-entropy augmented with a regularization term (L1 or KL divergence) that operates on the prediction of the network. Different experiments compare the presented loss function of previous proposals.
main_review: Strengths
- The paper presents an interesting analysis for the CE and Dice loss function, indicating their relationships in terms of the called label-marginal bias. Base on this, the paper proposes an alternative variation of the CE loss function for segmentation. 

Weaknesses
- Regarding the performance on the medical dataset, are the performance gains similar in other medical datasets, for example, organ segmentation (where some structures can be considerably smaller compared with the background, especially when considering 3D input data)? It is possible to discuss the applicability to different datasets, and in what cases, can the proposed function more/less appropiate

- Some points that might not be clear are the relation of the CE, DSC, and the label marginal concerning the performance of models in computer vision, and medical images. In this sense, it would be interesting to elaborate on this regard. 

- Do the CE-based functions used in the experiments correspond to the weighted versions?
summary_of_the_review: Segmentation loss functions are employed in different medical and real-world datasets, particularly the medical domain can present a wide variability in available datasets.  It might be interesting to discuss results on additional datasets, compared with the DSC (as the paper discussed, it is widely used in the medical domain).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents a decomposition of two commonly used segmentation losses (cross entropy/CE and Dice losses) into two competing components. The two components are interpreted as ground truth matching and label-marginal penalties. This work argues that the label-marginal penalties in Dice loss favour extreme  class imbalances, and hence their widespread use in medical imaging segmentation tasks which encounter class imbalance more commonly. The label marginal terms in CE is argued to be better as this term matches the label marginals to the ground truth distributions. The scaling between the two terms in CE is presented as the problem and this work proposes an L1 regularization to overcome the label bias problem in CE. Experiments on two segmentation tasks show interesting influence of these regularizations. 
main_review: Strengths: 
+ The question addressed in this work is inherently an interesting one. Studying the relation between two popular loss functions which are understood to be somehow complementary to each other and attempting to make connections between them can be an important contribution.
+ The decomposition of the loss functions into the ground truth penalty and label bias terms is reasonable, and the proofs provided in A1. and A2 are mostly adequate. 
+ Experiments with the L1 regularization for the class imbalance dataset (retinal segmentation) show promising results. 

Weaknesses: 


- One of my major concerns is in the proposed strategies to modulate the weighting of the label bias term in CE. The key argument is that the relative contribution of the label-marginal term is of smaller magnitude in class imbalance cases. This is not obvious from the Propositions 2. How is this conclusion drawn? 

- Two regularization strategies are proposed; one of which is KLD term D(y||p). But, in  Eq. 4 CE already decomposes into entropy and the same KLD term. Are these two KLD terms the same? If so, what is the influence of the additional regularization? Perhaps it only serves as a scaling of the label bias term. In that case, how does the L1 regularization influence the scaling of the label bias term. 

- L1 regularization is posited to be better than KLD regularization, and one of the reasons cited is its "gradient properties and stability".  This is somewhat unconvincing. Is this in comparison with KLD? 

- The specific choice of the two types of regularization terms is simply stated. Why were these two types of regularization schemes chosen? What was the motivation? How were these choices made? Is there a theoretical justification for using the L1 term? 

 - The decomposition of the losses into two competing terms and their interpretations are reasonable. However, for Dice loss, the arguments presented for why the label marginal bias term in Eq. 1 end up favouring extreme class imbalance is unclear. Additional explanation can be helpful here.

Minor comments: 
- An important point in the paper is the competing nature of the ground truth penalty and label bias term. Their competing nature should be elucidated further. It is not immediately obvious in case of the CE (Eq.4). and can be helpful to readers. 
summary_of_the_review: The decomposition of the losses into two terms is interesting and largely rigorously studied. However, the regularization schemes proposed are not well justified.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed theoretical analysis showing Dice loss has a bias preferring small structures, while losing flexibility of dealing with arbitrary class proportions. On the hand hand, the cross-entropy loss encourages proportions of the predicted segmentation to match the ground-truth proportions. A new loss function to control the label-marginal biases was proposed as sum of CE and weighted L1/KL divergence regularization. 
main_review: Strengths: 
1. This paper provides theoretical perspective about why Dice loss is better for smaller structure and CE loss is better at matching prediction distribution to training data distribution. While both phenomena are straightforward intuitively, a theoretical perspective from label marginal bias is appreciated. 
2. This paper proposes a novel loss function that uses CE loss with a regularization term to correct label proportions. 
3. Experiments on both natural and medical image segmentation problems with imbalanced classes were conducted to show that the proposed loss achieves better accuracy than Dice or CE loss themselves. 

Weakness: 
1. The proposed loss regularization term needs a hyper-parameter \lambda to control the weight of regularization. It is understandable that the optimal \lambda value is dataset-dependent due to different class distributions in different datasets. However, whether such \lambda is also network-dependent is not clear. 

In this paper, only ResNet-FPN and ResNet-Unet are used in the experiment. More extensive experiment on different architectures using same lambda values may be needed. Or theoretical argument about why \lambda is only dataset-dependent and not network-dependent would be very helpful. 

2. A useful baseline to compare to is "CE+w * Dice", where w is a weighting coefficient. As the proposed loss is "CE+\lambda * L1", the "CE+w * Dice" can provide probably similar performance with the same amount of hyper-parameter to tune. 

3. Only two datasets, Cityscape and retinal lesions, are presented. It would be great if more public datasets with different class distribution characteristics can be used, so that we can gain more insights about how to tune \lambda to match datasets with different class distributions. 

4. I agree with other reviewers on that the "segmentation GT" of the retinal lesion dataset is so poor that it cannot even be called segmentation annotation, but rather just detection annotation. In fact, without proper medical knowledge, I would say CE in Figure 3 is returning a better segmentation as they differentiate "normal" from "not-so-normal" more closely. Thus, higher DSC on this dataset is likely due to segmentation shape being smoother/more elliptical. 

5. The Dice-related losses ("1-Dice" and "-log Dice") on **validation set** were highlighted by authors in Table 3, with extensive discussions in the first paragraph on page 9. However, Dice losses ("1-Dice" and "-log Dice") on **test set** were excluded in Table 4, without any discussion. I am not sure whether last paragraph of Sec.3 was for validation or test set. But either way, I think including a detailed comparison of Dice-related losses on test set is important, as the whole paper is essentially studying Dice loss and CE loss. 

summary_of_the_review: The theoretical perspectives states the intuitively well-understood fact that Dice loss is better at capturing small objects while CE provides prediction distribution that matches training data. Thus, although it is an alternative perspective from label marginal bias, it did not lead to significant new knowledge about both losses. The proposed new loss, although achieving better accuracy on two datasets with two different networks, is not sufficiently validated in terms of how \lambda choice should be made regarding to different datasets and network structures. The comparison against "CE + w*Dice" is also strongly recommended, both theoretically and experimentally. The lack of Dice-related losses results on Cityscape dataset test set needs to be addressed. The poor quality of GT makes the retinal lesion dataset inappropriate to conduct segmentation experiments on.
review_title: null

Meta-review: The submission evaluates the relationship between (logarithmic) Dice loss and cross-entropy loss, arguing for a similar decomposition into ground truth and "hidden label-marginal biases."  The submission received mixed reviews, with two reviewers voting for rejection, and two feeling that it is marginally above the acceptance threshold.  Setting aside the numerical scores, there are reasons to believe that this submission, while interesting, has shortcomings that limit its relevance to the wider ICLR community.  These include
- Very many losses have been proposed for imbalanced classification / (medical) image segmentation, such as Jaccard and Tversky index or ranking measures, although admittedly Dice is probably the most popular in the medical imaging literature due to historic reasons.  Arguably, Dice is less well-behaved from a theoretical perspective compared to other options (e.g. it does not even form a metric), and may not be the most relevant point of departure for a representation learning conference.  The literature review misses many relevant papers on such losses, including papers that specifically are focused on the relationship between Dice and cross-entropy, e.g. Eelbode et al., IEEE-TMI 2020 and citations therein.
- The empirical results do not show substantially improved results compared to baselines.

On the balance, this does not cross the threshold for acceptance to a competitive venue such as ICLR.
Decision: Reject