Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new, updated segmentation uncertainty modeling objective based on the label style and ways to revise the segmentation uncertainty model architecture by including a discrete label style. 
strength_and_weaknesses: Strength
+ An interesting solution to reducing bias caused by aleatoric uncertainty and different label styles
+ Experiments show that the conditioned models can outperform standard models

Weaknesses
- number of image-annotation pairs seems not large
- only discrete label style is included, not the continuous

clarity,_quality,_novelty_and_reproducibility: This paper is well-written and easy to follow. The idea is interesting. Experiments are presented with details. 
summary_of_the_review: The paper presents an interesting idea and good experimental results. However, the evaluation needs improvement.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper suggests that datasets could be heterogeneous since the label generation process might vary systematically, which the paper calls 'label style'. Based on this factor, it further proposes to condition probabilistic segmentation models on label style and to train the models on datasets containing differing label styles. The paper then describes how it modifies Probabilistic U-net and Stochastic Segmentation Networks (SSN) to condition them on label style, and the results of experiments show that compared with Probabilistic U-net and SSN, c-prob. U-net and c-SSNs are able to utilize all data regardless of label style in a meaningful way while introducing less segmentation bias.
strength_and_weaknesses: Strength:
1. The paper presents a clear style of writing. 
2. The paper includes detailed experiments and the results are convincing enough to support the hypothesis. 
3. The proposed method is simple but surprisingly efficacious and of great potential in increasing the applicability of segmentation uncertainty models. 
Weakness:
1. The paper should present a more detailed description of the baseline methods in Sec.3.1 and Sec.3.2. 
2. Reasons or comparative experiments suggesting why choosing the modification methods might also be included. 
3. The sentence in Sec.1.1 ' We the proposed strategy’ should be ' We proposed strategy' instead.

clarity,_quality,_novelty_and_reproducibility: The paper is of great quality, novelty, and reproducibility. The clarity is good except in  Sec.3.1 and Sec.3.2 due to a lack of description of the baseline methods.
summary_of_the_review: The paper proposes a novel hypothesis about the annotation style, and gives a clear description of the hypothesis and method, except for the description of the baseline methods. The experiments are very detailed and the results are convincing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper explores conditioning on image annotation labelling style, to reduce aleatoric uncertainty relating to the segmentation of objects in medical images. In particular, the general problem of uncertainty estimation for segmentation assumes that the annotation is drawn from an unknown annotator distribution, p(a|x) where a represents the (unknown) annotator, and x is the image. The main contribution then is the proposal to also condition on (discrete) label style l, i.e. p(a|x,l), where the label style is known. This label conditioning is implemented into two segmentation models, the (conditioned) probabilistic U-Net (CPUN), and the conditioned stochastic segmentation network (CSSN), such that both models can condition their predictions on any label style used in training. Experiments on two datasets (ISIC, PhC-U373) suggest that such conditioning results in improved segmentation performance (as measured by IoU), whether compared to training using all available labels (regardless of label style, without conditioning), or using only annotations of the same label style (thus losing information possibly obtainable from the other label styles).

strength_and_weaknesses: Strengths:

1. The proposed label style conditioning method allows information on annotation style to be exploited, in segmentation tasks. Moreover, as coarse segmentations are (possibly greatly) less time-consuming for human graders to produce, the method may greatly reduce human effort, for large performance gains.

2. Two model architectures are implemented, and experiments are exhaustively performed over combinations of training and testing data subsets, each containing one or all label styles.

3. The distribution of overestimations/uncertainties are well-illustrated in suitable figures.

Possible Weaknesses/Considerations:

4. A major practical consideration about the proposed method, is the requirement for the label style to be known. For the experiments studied in this work, the datasets have annotations in prior-defined styles (e.g. Figure 2), which are then separately evaluated on as subsets each containing annotations of a single style. It might firstly be informative to evaluate performance where the annotation style is not a priori known.

5. Related to the above, coarse annotations (e.g. Label Styles 1 & 2 in Figure 2) appear to be automatically obtainable from detailed annotations (Label Style 0), using some basic image processing operations (e.g. dilation, then smoothing of the boundary). If so, this may suggest that the coarse annotations actually do not really provide any (significant) new information, and it is possible that with just the detailed annotations for each image, coarse annotations can be generated and used as the additional coarse annotations for the proposed conditional learning. This would however then fall into the (common) use of dynamic augmentation (e.g. minor rotation/scaling, elastic deformation etc.) preprocessing of the image & annotation labels, in training deep learning classification/segmentation models.
Given this, it might be appropriate to explore whether automatically-generated coarse segmentations are sufficient to replicate the performance gain currently observed with manually-annotated coarse segmentations. If so, this would further inform on the actual source of the performance gain, and further reduce human annotation burden.

6. Still related to the above, while area bias (and possibly other bias) is observed (Section 5.1), it is not clear the extent to which such bias has been attempted to be mitigated, particularly using the validation data. It is mentioned in Section 4.1 that the datasets have 20% of image-annotation pairs reserved as validation data, but it is not clear how this validation data has been used. It might be expected for area (and other) bias to be mitigated, by optimizing hyperparameters such as the pixel threshold (apparently a default threshold of 0.5 is mentioned in Section 5.3) used to separate the object from the background. This might be clarified.

7. The “pixel-wise entropy” referred to in Section 5.3 might be explicitly defined (possibly in the appendix), for completeness.

8. Theoretically, the formulation p(a|x,l) appears to still place emphasis on annotator distribution, while it might be supposed that inter-annotator label differences would be insignificant compared to inter-style label differences. This might be discussed.

clarity,_quality,_novelty_and_reproducibility: The background and theory is generally clearly written and easy to follow. The primary proposal of conditioning on style labels may not be entirely novel, but remains well-motivated and executed. The implementation of the proposed conditioning involves adding the label style variable to defined inputs of previously-described models, which should be replicable. However, some clarifications might remain on the exact training and hyperparameter optimization procedure (if any).

summary_of_the_review: A label style conditioning is proposed and implemented in two previously-described deep neural network-based image segmentation models, which exhibits improved performance on two datasets compared to the same models without conditioning. However, some issues may remain with whether the comparisons are fully fair and whether apparent bias has been attempted to be mitigated with more conventional and common techniques, as detailed in the Strength and Weaknesses section.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Hypothesizing that systematic differences in "labeling style" between different manual annotators of image segmentations might lead to bias in models trained on such datasets, the authors propose a method for adjusting the objective function used to train segmentation models, and adapt architectures to condition predictions on such label style differences. The method is shown to improve segmentation accuracy and reduces style bias.
justification_for_why_not_higher_score: The empirical evaluations were found to be limited in scope, and concerns were raised about the assumption that label style be known at training time.
justification_for_why_not_lower_score: Empirical evaluations of the method, while limited, still convincingly demonstrate improvements over SOTA.
Decision: Accept: poster