Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors present an analysis of the often used performance metrics for image segmentation (Accuracy, Dice) for noisy target labels. They performed a theoretical analysis of the problem and evaluation on a real-world data set. In their results, they provide a characterization of upper and lower bounds on the volume of optimal segmentation.
strengths_and_weaknesses: The paper is very well written and very detailed. The paper provides a detailed investigation of how noise labels influence the volume of optimal segmentation. This provides interesting insides on how the quality of the manual segmentations influences the final segmentation performance in relation to the state-of-the-art evaluation metrics Dice and accuracy.
questions: None
limitations: Yes, the authors have addressed the limitations of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: It is a theoretical paper that studies two popular performance metrics in medical image segmentation, i.e., Accuracy and Dice while considering the target labels are noisy/random. The noisy labels scenario would be common in segmentation, for example, when different annotators give different boundaries to differ two classes. The authors theoretically prove several theorems to provide a form of optimal solutions to Accuracy and Dice metrics, and derive multiple properties of the optimal solutions, including 1) bounds on the volume of optimal segmentation, 2) relation between the volume of optimal solutions to two metrics. Further, the authors use an experiment to show that the optimal segmentations with respect to Accuracy and Dice deviate from the expected volume of the target.
strengths_and_weaknesses: Originality:
* Strengths: The theoretical analysis on Accuracy and Dice looks original to the reviewer.

Quality:
* Strengths: The main contributions of the paper, i.e., theorems are well-stated and written in a good format, and lemmas and theorems are rigorously proved.

Clarity:
* Strengths: The main ideas of the paper are clearly presented.
* Weaknesses: As the measure theory is widely used in the paper for strictness and generality, sometimes the readers would not be able to get the true meaning/insight of some notations corresponding to their use in the medical segmentation. It is highly recommended that examples (even in an informally speaking way) are given to describe the notations when they first appear in the paper. For example, add some description that $\Omega$ is informally the continuous space for voxels from the data when $n=3$, and give some concrete examples for noisy segmentation $L$. Also, adding a non-measure-theory-style description for Accuracy and Dice would also help understand the paper. 

Significance:
* Weaknesses: Authors may need to highlight how their work could help with current medical image segmentation work, and how their insights from the theorems can help the people/scholars in this area, for example, selecting metrics etc.
questions: Suggestions:
1. As mentioned in Strengths And Weaknesses, the reviewer suggests adding more "informal" descriptions to the notations defined and giving some concrete examples when introducing the theory.
2.  As mentioned in Strengths And Weaknesses, the reviewer suggests adding more information about the significance of the work.

Questions:
1. The theory developed in the paper seems to focus more on the performance evaluation step when using these metrics. Could the current theory help with improving the training steps in some ways?
2. How do we make use of these properties to do better image segmentation? Maybe some examples could be provided.



limitations: 1. I agree with the limitations mentioned by the authors, i.e., the amount of experiments included is limited in the paper, because to demonstrate the usefulness and generality of the theory, results from different medical image segmentation applications may be needed to draw a conclusion.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors provided a theoretical analysis of how the optimal segmentation to the metrics Accuracy and Dice can deviate regarding the segmentation volume. In addition, the upper and lower bounds on the volume of optimal segmentation are provided. Unfortunately, although the authors attempt to show the relevance of the theoretical observation in a real dataset, the experiments are not well-introduced and contain many flaws (listed in the weakness below), which makes the experimental results less convincing.   
strengths_and_weaknesses: + A theoretical analysis of how the volume of optimal segmentation with respect to the performance of Accuracy and Dice could deviate is presented.

Weakness:
- The experiments are not well-introduced, and the results are not well-illustrated. For example, how exactly the marginal function m is computed by "taking the pixel-wise average of the different segmentations" is not clearly introduced? In addition, it is unknown what's the optimal segmentations in the provided dataset (with labels from multiple annotators).
- The selected dataset is small, with only 18 patients, and the ROIs in each case are annotated with unknown medical practitioners. 
- It remains unclear how the level of noise in the annotation will affect the evaluation capability of metrics. A conclusion of " the noise has a significant effect on the volume of optimal segmentations" is not practically valuable for detailed applications.  
- The authors also mentioned that the shape of the target is related to the effect of noise. Still, only a vague clue is given that spherical structures will be less affected. 
- FCN and U-Net-based segmentation methods have been around and popular since 2015, not "during the last years," as stated at the beginning of the Introduction section.

questions: See the weakness part.
limitations: Limitations are discussed except for the aspects listed in the weakness.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper tackled the noisy labels of image segmentation in a theoretical manner. The study is performed on accuracy and dice metrics in medical image segmentation. 
strengths_and_weaknesses: From a couple of years ago, dataset curation has been addressed in a wide range. However, most of the findings have not been shared with the community due to the private corporation policies. Thanks to the inconsistency of the  labels detected in the regular datasets used in the wide range of computer vision tasks, efficient solutions have been discovered and proposed. Therefore, this is one of the papers that can address not only segmentation but also edge detection, to name an example. Hence, the paper presents the first discoveries and solutions for the medical image segmentation; It also includes experiments on data from the Gold Atlas project.

Minor errors:
* Page 3 line 123 typo.

questions: No questions.
limitations: Yes
review_title: null

Meta-review: metareview: Three knowledgeable referees support accept and a fourth one does not oppose to accept so I recommend Accept.
confidence: Certain
Decision: Accept