That Label's got Style: Handling Label Style Bias for Uncertain Image Segmentation
Authors: Kilian Zepf
Eike Petersen
Jes Frellsen
Aasa Feragen
Venue: ICLR 2023 poster (2023)
Keywords: Uncertainty Quantification
Segmentation

Abstract: Segmentation uncertainty models predict a distribution over plausible segmentations for a given input, which they learn from the annotator variation in the training set. However, in practice these annotations can differ systematically in the way they are generated, for example through the use of different labeling tools. This results in datasets that contain both data variability and differing label styles. In this paper, we demonstrate that applying state-of-the-art segmentation uncertainty models on such datasets can lead to model bias caused by the different label styles. We present an updated modelling objective conditioning on labeling style for aleatoric uncertainty estimation, and modify two state-of-the-art-architectures for segmentation uncertainty accordingly. We show with extensive experiments that this method reduces label style bias, while improving segmentation performance, increasing the applicability of segmentation uncertainty models in the wild. We curate two datasets, with annotations in different label styles, which we will make publicly available along with our code upon publication.

Published as a conference paper at ICLR 2023
THAT LABEL'S GOT STYLE: HANDLING LABEL STYLE BIAS FOR UNCERTAIN IMAGE SEGMENTATION
Kilian Zepf, Eike Petersen, Jes Frellsen, Aasa Feragen Technical University of Denmark (kmze, ewi pe, jefr, fhar @dtu.dd
ABSTRACT
Segmentation uncertainty models predict distribution over plausible segmenta- tions for a given input, which they learn from the annotator variation in the training set. However, in practice these annotations can differ systematically in the way they are generated, for example through the use of different labeling tools. This results in datasets that contain both data variability and differing label styles. In this paper, we demonstrate that applying state-of-the-an segmentation uncertainty models on such datasets can lead model bias caused by the different label styles. We present updated modelling objective conditioning on labeling style for aleatoric uncer- tainty estimation, and modify two state-of oottee-f-thee-eee ff-the-art--e archtechiisssures for segmentation uncertainty accordingly. We show with extensive experiments that this method reduces label style bias, while improving segmentation performance, increasing the applicability of segmentation uncertainty models in the wild. We curate two datasets, with annotations in different label styles, which we will make publicly available along with our code upon publication.
INTRODUCTION
Image segmentation is fundamental task in computer vision and biomedical image processing. As part of the effort to create safe and interpretable ML systems, the quantification of segmentation uncertainty has thus become crucial task as well. While different sources and therefore different types of uncertainties can be distinguished (Kiureghian & Ditlevsen, 2009; Gawlikowski et al., 2021), research has mainly focused on modelling two types: aleatoric and epistemic uncertainty. While epistemic uncertainty mainly refers to model uncertainty due to missing training data, aleatoric uncertainty arises through variability inherent to the data, caused for example by different opinions of the annotators about the presence, position and boundary of an object. Since aleatoric uncertainty estimates are directly inferred from the training data it important that the variability in the available ground-truth annotations represents the experts' disagreement. However, in practice the annotations might vary in systematic ways, caused by differing labeling tools or different labeling instructions. Especially in settings where opinions in form of annotations sourced from experts in different institutions, datasets can be heterogeneous in the way the labels are generated.
Even in the best of cases, in which manual annotators are given detailed instructions how to segment objects, they will still have to make choices on how to annotate ambiguous parts of the image. Moreover, annotators are always carefully trained, and may not have access to the same
Figure 1: Sample image the from PhC-U373 dataset with annotations (red). The first three annotators were instructed to delincate the boundary in detail, whereas the last three annotators were instructed to provide coarser and faster annotation.

---

Published as a conference paper at ICLR 2023
labeling tools. As a result, individual choices and external factors affect how annotations are made; we term this label style. Figure 1 shows an example of how annotations may vary in label style.
Label style can also depend on label cost: While detailed annotations are desirable, they also take more time, and one might desire to train models on cheaper, less detailed annotations. In the example of Fig. I, we have access to both detailed and coarse, or weak, annotations. It is not clear that adding the weaker annotations will necessarily improve performance; removing them to train on fewer but higher quality annotations could also be beneficial.
While weak annotations carry less precise information about the segmentation boundary, they do carry information about the annotator's beliefs concerning the presence and rough location of an object. Exploiting this information could improve the annotator distribution learned by the model, even tough the target might not be delineated in detailed way. In practice, however, neither datasets nor models distinguish between variations in label style and variations in the data. As result, current methods for segmentation uncertainty run the risk of being biased by this difference in label style.
1.1 CONTRIBUTION
In this paper, we demonstrate that applying tf-therrtt models on datasets that contain differing label styles can lead to systematic over-segmentation. We show how this bias can be reduced by stating an updated modelling objective for aleatoric uncertainty estimation conditioned style. We adjust two state-of-the-ar uncertainty segmentation architectures accordingly, presenting conditioned versions of the Probabilistic U-net (Kohl et al., 2018) and the Stochastic Segmentation Networks (Monteiro et al., 2020) that fit to the updated modelling objective and can be trained on datasets containing differing label styles. We compare the proposed method against the common strategy of removing the annotations of weaker label style from the dataset.
label
We curate two datasets, both with annotations in different label styles, ranging from detailed, close crops to over-segmented outlines. In series of experiments, we show that the conditioned models outperform standard models, trained on either all or a single label style. The conditioning reduces label style bias, improves overall segmentation accuracy and enables more precise flagging of probable segmentation errors. Our results stress that including all label styles using a conditioned model enables fully leveraging all labels in a dataset, opposed to naively excluding weaker label styles. As such, our model contributes to increasing the applicability of uncertainty segmentation models in practice. Our code and curated datasets will be made publicly available, to enable the community to further assess models for segmentation uncertainty in the scenario with differing label styles.
2 BACKGROUND AND RELATED WORK
Uncertainties in deep learning in general, and image segmentation in particular, can be studied under the Bayesian framework (Bishop, 2006; Kendall & Gal, 2017). Let D (X, AA) be dataset of N images n, E X with S pixels each, where each image I, is associated with & ground-truth annotations a A. drawn from the unknown annotator distribution p(a|zr.). Furthermore, let fxx, 0) denote a model of p(a[ær defined by parameters 0. Formulating the segmentation task in a Bayesian way, we seek model the probability distribution p(y]zz) over model predictions y given an image x to be as similar as possible to the annotator distribution p(a[z). This predictive distribution can be decomposed into the two types of uncertainty (Kiureghian & Ditlevsen, 2009) as follows:
/
plul,, D)
plula, 0)
d0.
aleatoric uncertuinty epistemic uncertuinty
After observing the data D during training, the posterior distribution p)) describes a density over the parameter space of the model, capturing epistemic uncertainty. The distribution Zz f). on the other hand, captures the variation in the data and possible model predictions, i.e., aleatoric uncertainty. Due to the typically intractable epistemic uncertainty distribution, the integral on the right hand side of equation 1 is usually not accessible. Therefore, it is of particular interest to develop suitable approximations of the predictive distribution or parts of the integral in 1, and various image segmentation approaches and models have been proposed for this purpose (Kohl et al., 2018; Monteiro al., 2020; Kohl et al., 2019; Baumgartner et al., 2019).
2

---

Published as a conference paper at ICLR 2023
In this context, the standard cross-entropy minimization approach pursued in most current deep learning research can be understood as approximating the posterior distribution p(0 by a Dirac distribution 5(0 - 81) and assuming that there is spatial correlation between the pixels in an image. Under these assumptions, one obtains
s
8
-log 1(a/z, D) = logp(a/z, = llgg II
p(a 01) =
Σ logp(a; Tz 01),
(2)
==1
which is precisely the standard negative log likelihood (or cross-entropy) loss.
Variational Bayesian methods approximate the intractable integrals arising in Bayesian inference directly through optimization. A special case that uses Bernoulli distributions to approximate the posterior distribution of the parameters as well the predictive distribution is the Monte Carlo Dropout method (Gal & Ghahramani, 2016a; 2015). Pixel-wise uncertainty values can be retrieved by averaging multiple forward passes, while applying dropout during inference time before each weight layer of neural network. The resulting approximation of the predictive distribution is always multi-modal and not necessarily expressive (Folgoc et al., 2021), and drawn samples can lack coherence (Czolbe et al., 2021; Gal & Ghahramani, 201
Ensemble methods (Lakshminnrayanan et al., 2017) also yield pixel-wise uncertainty estimates by averaging over the predictions of different models which are independently trained on the same dataset. Contrary to Monte Carlo Dropout, which implicitly averages multiple models during inference time, those models do not share the same weights but can be seen as a frequentist approach to estimating p D). On the other hand, ensembles and multi-head models (Rupprecht et al., 2017; Lee al., 2016) independently trained distribution over annotations, therefore also modelling the variability and disagreement between the annotators p (aaa (Czolbe et al., 2021).
expert's 's annotation each return in expectation
Models that mainly focus on modelling the annotator distribution p(a)x) have been introduced based on combinations of deterministic segmentation networks and generative modelling techniques such as conditional variational autoencoders and normalizing flows (Baumgartner et al., 2019; Kohl et al., 2019; Selvan et al., 2020). Moreover, Gaussian-Process based convolutional architecture was recently suggested to distinguish between annotator variability and estimator uncertainty (Popescu al., 2021). Other approaches model the distribution over annotations with probabilistic graphical models and combinations of those with neural networks (Batra et al., 2012; Kirillov et al., 201 5a;b; 2016; Arnab et al., 2018; Kamnitsas al., 2017). However, the computational expense of inference in those models usually prohibits more than maximum posteriori estimate of the targeted distribution.
In the following, we will describe detail two state-of-the-e methods for quantifying aleatoric segmentation uncertainty: the probabilistic U-net (Kohl al., 2018) and the Stochastic Segmentation Networks (Monteiro et al., 2020). We propose a new modelling objective for the annotator distribution p(a|x: including label styles, and show how both models can be modified to fit it, increasing their applicability to datasets with varying label styles.
3 SEGMENTATION UNCERTAINTY MODELS CONDITIONED ON LABEL STYLE
Segmentation uncertainty models fit a predictive distribution p(y|æ, 0) to the annotator distribution p(a)z)). We argue that, in practice, the targeted distribution of annotations should also be conditioned on label style. Therefore, we propose to model p(a|æ, Z) instead, where E {0,.. ....... .,... > 0} denotes discrete variable representing the label style, and i denotes the number of available label styles. In this setting, the segmentation model is to be learned from a dataset D = (X, A, L) containing tuples (z,, o (t)) of images I, E X, annotations at A, and corresponding label styles 1f e In the following, we present modified versions of the probabilistic U-net and Stochastic Segmentation Networks that incorporate label styles directly into training and inference by conditioning the models on discrete variable (Mirza & Osindero, 2014), therefore fitting the new modelling objective.
3.1 CONDITIONED PROBABILISTIC U-NET
The probabilistic U-net (Kohl et al., 2018) combines a U-net with a conditional variational auto- encoder. To encode plausible segmentation variants, encoder P parameterized by takes
3

---

Published as a conference paper at ICLR 2023
image as its input and estimates the mean (z) and variance ow(2) of a diagonal Gaussian (z), (2) in R During inference, a sample & from this distribution drawn and concate- nated with the deterministic output of the U-net go (z) and combined by 1 x convolutions fv. A prediction y for a given latent z can then be written as
y fgo (2),
The prior net is trained by minimizing the Kullback-Leibhe divergence between its predicted latent space distribution and a distribution given by the encoder Q, called the posterior net. Q has parameters and receives both the image and annotation of the distribution N (2, a), (z, a)). We adjust this model by adding a discrete label style variable / to the input of the prior net encoder, which is tiled and concatenated one-hot-encoded to the channel axis of the input image x. The latent variable 2 is then modelled as the normal distribution
inputs, estimating the parameters
2|2,,1 N (1 (2,, 1), (x,))))
where the covariance matrix aw is, again, assumed to be diagonal, and both aw and Mew are estimated by the prior net encoder. During training, the posterior net Q receives the style 1 in addition to the image Zn and annotation at. Like the original model, the architecture is trained by minimizing the variational lower bound as described Kohl et al. (2018) (see appendix A.5 for details).
3.2 CONDITIONED STOCHASTIC SEGMENTATION NETWORKS
Stochastic Segmentation Networks (SSN) (Monteiro et al., 2020) are recently suggested model class for quantifying aleatoric segmentation uncertainty. The method can be applied to any feature map received by deterministic segmentation network, in our case U-net go The feature map is passed through three separate convolutional layers, p(x), D(r), and P(x), that estimate the parameters of low-rank multivariate normal distribution over the logits 17. Since g is deterministic, we can write the layers as directly dependent on x. The covariance matrix is given by
[xx) = D(z) + P(z)
(5)
and one can then pass samples drawn from the estimated logit distribution
// N (u(x), E(x))
through softmax layer to receive predictions for given image æ. For details on the training procedure and loss function, we refer the reader to appendix A.5.
We adjust this model by passing the discrete label style variable 1 through an encoder and by concatenating the resulting feature map the feature map given by the U-net. The combined feature map is then passed through the respective convolutional layers that estimate the parameters of the logit distribution. The style variable is again tiled and concatenated (one-hot-encoded) to the channel axis of the feature maps. The distribution over the logits is then modelled
n/ 1 N(((x, 1), 1)r,
Given an image during test time, it now possible with both models to condition their predictions on a label style 1 E (O...., schematic overview of the baseline models and the conditioned versions during inference.
i > 0} that has been used for training. Figure 5 in appendix A.4 gives
EXPERIMENTS
To train and evaluate segmentation uncertainty models fitting the updated modelling objective p(a|z, æz I), we need datasets with multiple annotations per image in differing label styles. We consider two such datasets: subset of the ISIC19 dataset and a new version of the PhC-U373 dataset.
4.1 DATA
For first evaluation, consider subset of the ISIC19 skin lesion segmentation challenge (Combalia et al., 2019; Codella et al.. 2018: Tschandl et al., 2018), where each image has exactly three annotations available. A distinctive feature of this dataset is the clear difference between
4

---

Published as a conference paper at ICLR 2023
Label style 0
Label style
sabel-style 2
Label style 0
Label style
Label style 2
Figure 2: Two sample images from the ISIC dataset with annotations in different label styles (red); fine-grained delineations (style 0), smooth, close borders (style 1) and coarse crops (style 2). An image might have multiple annotations of a certain label style (as in the right example).
label styles, exemplified in Fig. 2: Some annotations follow the skin lesion boundary exactly as possible; this is important as the size and boundary features can be used to clinically classify lesions as malignant benign (Abbasi et al., 2004). Other annotations, conversely, consist of loosely defined regions containing the lesion. This type of weak labeling is more consistent with the task of detecting lesions than segmenting them.
The ISIC dataset contains ground-truth segmentations generated by three different methods, each of which we consider as one label style. Label style corresponds to tight annotations with detailed boundary, generated by a semi-automated flood-fill algorithm supervised by an expert. Label style I annotations stem from a fully-automated algorithm, reviewed and accepted by experts, while label style are polygonal annotations traced manually by specialists. Note that not every image has exactly one annotation of each label style, so different combinations can occur such as in Fig. 2 - we include such images purpose illustrate data when the choice annotations is beyond our control. We consider the fine-grained annotations of label style 0 the ground truth in the downstream analysis of annotator bias. For our experiments, the images and annotations are rescaled to 256 x 256 pixels.
models' ability to make the most of the available
Our second dataset is based PhC-U373 dataset of the ISBI cell tracing challenge (Ulrich al., 2009; Ulman al., 2017). We use the first video se- quence, containing 115 2D images of multiple cells, annotated with two classes (Cell, Background). To obtain images of single cells, we find the smallest bounding boxes around the ground-truth masks and extend those by 20px on all sites. We then crop these bounding boxes out of the original images and use only those patches where the expanded bounding box lies completely within the full-sized image. This results in a dataset containing 651 images of single cells, which are resized to 128 x 128 pixels. In addition to the ground-trute truth labels, all images were annotated by three researchers inde- pendently, of which two labeled the dataset twice (in different styles), resulting in five additional annotations. Labelers were instructed perform either detailed annotation (label style 0) or wider annotations (label style 1). In total, we end up with three annotations labeled in style and three annotations labeled in style 1 for each image in the dataset. Label style 0 is, again, considered the ground truth in the downstream task analysis of annotator bias later on.
cell-tracking video from the
Table 1: Datasets and number of image-annotation pairs contained in the different splits.
ISIC
Train
Val
Test
Total
Style Style
62
22
54
19
Style All
63
21
21
179
59
62
300
PhC-U373
Train
Val
Test
Total
Style Style All
1170
390
393
1170
390
393
2340
780
786
3906
Both datasets are divided into subsets containing only one label style and randomly split for training, validation and testing with ratio of 60%, 20% and 20% respectively. Table 1 gives an overview over the resulting datasets and splits. Since all models are trained on pairs of images and annotations, report this number for fair comparison of the dataset sizes.
4.2 MODELS
The following models are compared in our experiments: (1) The proposed conditioned models, namely the style-conditioned probabilistic U-net (c-prob. U-net) and the style-conditioned Stochastic Segmentation Network (c-SSN), which are trained on annotations of all label styles. During inference, we condition on the style that the model will be evaluated on. (2) Probabilistic U-nets (prob. U-net) (Kohl et al., 2018) and Stochastic Segmentation Networks (SSN) (Monteiro al., 2020) trained
5

---

Published as a conference paper at ICLR 2023
on subsets that only contain one specific label style. These models are indicated by (subset) tag. (3) Probabilistic U-nets and Stochastic Segmentation Networks trained on the complete dataset, containing annotations of all label styles, but not conditioned on label style. These models are indicated by an (all) tag. Note that this would be the most common way to use the data.
All models are implemented in PyTorch and share U-net backbone with four encoder and decoder blocks for comparability. Each block contains three convolution layers and bilinear interpolation was used for upsampling. Dropout (p 0.5) used in the lowest-level feature map of all architectures. For the probabilistic U-nets, we chose a latent space dimension of as in Kohl et al. (2018). Encoder networks in the probabilisticc U-net are identical the contraction path of the backbone U-net. For the stochastic segmentation networks, the output of the last decoder block of the backbone U-net is passed into three different 1 convolutional layers to estimate the low-rank approximation of the normal distribution. Refer to appendix A.1 for further details on the training procedure.
5 RESULTS
Both datasets considered in this paper contain annotations of high quality that closely outline the object of interest and ones of lower quality that generally over-segment the object of inter- est. In Figure 3 and Table 2, we show the distri- bution of area differences (measured in number of pixels) between the different models' predic- tions and the high-quality annotations a fixed label style 0 test set. Intuitively, training the prob. U-net and the SSNs on all annotations leads to predictions that are biased towards too large annotations compared to the high-quality segmentations of label style on both datasets. To evaluate whether it is possible to reduce this bias while maintaining the prediction quality as well as the ability of the those models to fit the annotator distribution, we compare the prob. U- net and SSNs in all experiments below against the two alternatives described above in section 4.2:
Table 2: Average area difference with standard deviation in pixels between model predictions and respective label style 0 ground-truth annotations achieved on both datasets.
Dataset
Model
ISIC
PhC-U373
Prob. U-net (all)
6019 (13648)
370 (402)
Prob. U-net (subset 0)
4879 (15106)
584 (534)
c-prob. U-net
311 (8332)
43 (417)
SSNs (all)
5176 (15629)
716 (1022)
SSNs (subset 0)
3142 (19438)
1534 (2131)
e-SSNs
3199 (12987)
262 (558)
Firstly, against both models trained on subsets and, secondly, against the conditioned versions that we proposed to fit the new objective for the annotator distribution p(a|æ,1) 1).
5.1 VANILLA MODELS HAVE AN AREA BIAS; THIS DECREASES UNDER THE UPDATED MODELLING OBJECTIVE
Figure and Table 2 show that the prob. U-net and SSNs, both trained on subsets of only label style 0 annotations, tend to over-segment the targets, despite removing the coarser label style annotations. Further, we find that the conditioned models show lower area bias compared to the baselines. For the ISIC dataset, the standard SSN model trained on the label style 0 subset shows a slightly lower area bias compared to the conditioned model. The same is true for the subset-trained prob. U-net in the PhC-U373 dataset, but in both cases, the model trained on the subset shows higher variance.
5.2 INCREASED PREDICTIVE PERFORMANCE AND FIT TO THE ANNOTATOR DISTRIBUTION
We evaluated predictive performance to assess how well the different models can predict in certain label style. To this end, we calculated the Intersection over Union (IoU) between the model's 's mean prediction against test sets that only contain annotations of the targeted label style, indicated by the column index in Table 3. Note that for column i, the prob. U-nets (subsets) or SSNs (subsets) were trained only on annotations of the targeted label style i. Across both datasets and all label styles, the best predictive performance in terms of IoU is achieved by the style-conditioned models, with the exception of the c-SSN conditioned on label style I on the ISIC dataset.
In addition, we computed the area under the receiver-operating characteristic curve (AUROC) with respect to pixel-wise model predictions compared the ground-truth segmentation mask. Table

---

Published as a conference paper at ICLR 2023
Overestimation lesion area (ISIC dataset)
Overestimation cell area (PhC-U373 dataset)
Differance from ground truth area
20000
Differencee from ground trulh area
3000
10000
2000
1000
-10000
-1000
-20000
oprob. U-net (/= 0)
prob. U-nel (sutsel 0)
prob. U-net (ail)
c-SSN (1 0) SSN (subset 0)
SSN (all)
oprab. U-net (/ 0)
prob. U-nel (sutsel 0)
prob. U-net (ail)
c-SSN (1 0)
SSN (subset 0)
SSN (all)
Figure 3: We assess bias lesion and cell area estimation for the two test datasets (ISIC, PhC-U373) by distribution of area difference (in pixels) between 100 model predictions per image and respective label style 0 ground truth annotations. Markers indicate the mean.
Table 3: Average IoU and standard deviation for the models' mean prediction on subsets of the ISIC and the PhC-U373 datasets containing only one label style each.
IoU wrt. to subset of label style ISIC
PhC-U373
Model
0
2
Prob. U-net (all)
0.65 (0.09)
0.65 (0.10)
0.69 (0.21)
0.89 (0.01)
0.90 (0.02)
Prob. U-net (subsets)
0.66 (0.11)
0.55 (0.17)
0.71 (0.15)
0.85 (0.03)
0.86 (0.03)
c-prob. U-net
0.77 (0.15)
0.78 (0.15)
0.76 (0.15)
0.92 (0.02)
0.92 (0.01)
SSNs (all)
0.72 (0.01)
0.77 (0.09)
0.71 (0.19)
0.89 (0.02)
0.93 (0.02)
SSNs (subsets)
0.61 (0.13)
0.75 (0.13)
0.61 (0.11)
0.85 (0.04)
0.92 (0.02)
e-SSNs
0.77 (0.10) 0.71 (0.20)
0.78 (0.19)
0.92 (0.01) 0.93 (0.01)
shows the values obtained by the different models; the conditioned models show higher AUROC values across all models and label styles.
Figure 8 and Figure 9 in Appendix A.7 show qualitative results of sample predictions on 5 images from the ISIC dataset for the different models as well as the respective annotations, illustrating that the conditioning on label style corrects for the overestimation bias.
The goal of quantifying aleatoric seg- mentation uncertainty is formulated fitting a model's predictive distribution distribution
Table 4: Pixel-wise AUROC on ISIC and PhC-U373 test sets that only contain annotations of the targeted label style, indicated by the column index.
p(y(x) to the unknown ()) represented by the available an- notations of ground -truth segmentations in the dataset. To assess this fit, we cal- culated the generalized energy distance (GED) (Székely & Rizzo, 2013; Kohl et al., 2018) between segmentations sam- pled from the predictive distribution of
ISIC
PhC-U373
Label style
0
2
Prob. U-net (all)
0.9289
0.9102
0.8153
0.9960
0.9959
Prob. U-net (subsets)
0.9171
0.9516
0.8819
0.9925
0.9903
e-prob. U-net
0.9444
0.9750
0.9407
0.9963
0.9968
SSNs (all) SSNs (subsets)
0.8873
0.9111
0.8633
0.9958
0.9950
0.7986
0.9351
0,8609
0.9893
0.9957
the models and the set of ground- -round-truth segmentations. In Table 5, show mean GED values and their standard deviation for different pairs of predictive and annotator distributions. Annotator distributions
c-SSNs
0.9249
0.9506
0.9255
0.9964
0.9963
7

---

Published as a conference paper at ICLR 2023
Prob. U-net
SSN
Entropy H
ISIC
0.0
TP
FP
TN
FN
TP
FP
TN
FN
PhC- U37-
Entropy
0.4
0.2
0.0
TP
FP
TN
FN
TP
FP
TN
FN
Error Type
Error Type
Conditioned model
Model trained on subset
Model trained full dataset
Figure 4: Distributions and medians of pixel-wise uncertainty (as quantified by the entropy of pixel- wise model predictions) for true positive (TP), false positive (FP), true negative (TN), and false negative (FN) predictions.
p(a|2, 1 = i) contain only the set of ground-truth segmentations of label style i of the test set, whereas p(afar) contains all available annotations across all label styles of the test set. Table rows contain the predictive distributions of the following models: the standard prob. U-nets and SSNs trained on all annotation styles; the standard prob. U-nets and SSNs trained subsets of only annotation style i for each column and the c-prob. U-net and c-SSN conditioned on label style i. We find that for the fine- graiirr predictions of style 0 on the ISIC dataset, the e-prob. U-net and the c-SSN exhibit lower GED values. indicating better model fit. This also holds for the other label styles on the ISIC dataset, with the exception of the c-SSN conditioned on label style On the PhC-U373 dataset, we find that the GED values are very similar across all models.
Table 5: Mean GED (with standard deviation) between models' predictive distributions and the targeted annotator distributions p(a|æ, Jr, 1) of the available label styles as well as the full annotator distribution p(a(x) of an ISIC and PhC-U373 test set.
ISIC
PhC-U373
Annotator distribation
p(a|a,1 = 0)
p(alar = 1)
p(aiz,/ = 2) >(ular)
plala, 0)
p(alx, = 1)
p(alar
U-net
0.58 (0.14)
(0.23)
0.54 (0.14)
0.57 (0.17)
(0.06)
0.60 (0.07)
(0.06)
Prob. U-net (subsets)
0.61 (0.14)
0.57 (0.23)
0.51 (0.17)
0.70 (0.06)
0.62 (0.07)
c-prob. U-net
0.57 (0.13)
0.55 (0.24)
0.49 (0.19)
0.55 (0.18)
0.68 (0.06)
0.61 (0.06)
0.61 (0.06)
(all)
0.59 (0.14)
(0.23)
0.51 (0.17)
0.55 (0.17)
0.69 (0.06)
0.60 (0.07)
(0.06)
SSNs (subsets)
0.61 (0.15)
0.55 (0.25)
0.55 (0.23)
0.70 (0.06)
0.62 (0.07)
c-SSNs
0.58 (0.13)
0.56 (0.28)
0.48 (0.19)
0.55 (0.20) 0.68 (0.06)
0.60 (0.06)
0.60 (0.06)
5.3
DOES HIGH SEGMENTATION UNCERTAINTY INDICATE PROBABLE SEGMENTATION ERROR?
To answer this question, we assessed the relationship between pixel-wise segmentation uncertainty and the likelihood of a segmentation error in that pixel compared the ground-truth segmentation mask) on the label style 0 test set. The distribution of the pixel-wise entropy (see appendix A.3) of the predictions (as a measure segmentation uncertainty) was assessed for the cases of true positive (TP), false positive (FP), true negative (TN), and false negative (FN) pixel predictions. A default threshold 0.5 was applied. The results are shown in Figure 4 for all models and both datasets. We find that the c-prob. U-net consistently assigns higher uncertainty to false positive and false negative predictions on both datasets compared to the standard prob. U-net variants. The c-SSN performs on par with the alternative models.
8

---

Published as a conference paper at ICLR 2023
6 DISCUSSION AND CONCLUSION
Our paper took starting point the hypothesis that probabilistic segmentation models could be biased by different label styles if these are not accounted for by the model. We find this supported by the results of Fig. 3, where see that the prob. U-net and SSNs trained in the standard way on all label styles tend to over-segment the objects of interest. In contrast, the c-prob. U-net conditioned on the close-cropped label style 0 can reduce this bias considerably on both datasets. Even compared to prob. U-nets and SSNs trained only on the label style O annotations, the conditioned models give lower or on par bias.
However, the bias reduction in practice is only of value if the prediction performance does not decrease. Indeed, we find that the c-prob. U-net and the c-SSNs outperform the compared models across label styles in terms of IoU, except for the c-SSN on the label style I test set. On average, the mean prediction of the conditioned models is more similar to the targeted label style compared to the standard models trained on all data or on respective subsets. This indicates that the proposed conditioning of the models allows the models to implicitly correct for confounding label styles while preserve the ability to segment in meaningful way. The strong predictive performance is supported by consistently higher AUROC values for the conditioned models in Table 4. Additionally, the conditioned models fit the annotator distributions better, as the results in Table 5 suggest. We observe that the advantage in predictive performance of the conditioned models measured by IoU is larger compared to the advantage in fitting the annotator distribution measured by GED. This might be due to the fact that the loU is calculated on the models' mean predictions while the GED is based on 100 samples drawn from the predictive distribution.
For any segmentation uncertainty model, it is of interest to which extent the uncertainty estimates can be used to flag high-probability segmentation error. In the context of label styles, a second highly relevant question is whether we really need fine- aineggriiedd annotations, or whether we can exploit coarse- coarse-grained annotations to obtain a more precise indication of potential segmentation errors. From our preliminary analysis of the relationship between uncertainty estimates and segmentation errors, it can be observed that, as desirable, entropy is higher in the case of segmentation errors across all models and datasets (Fig. 4). The c-prob. U-net gives consistent advantage across both datasets, while the c-SSN either improves on or performs on par with the alternative models across datasets.
To summarize, our results support that the proposed method of conditioning on label style provides an advantage over, firstly. the standard way of training a segmentation uncertainty model on all available data, ignoring label style, and, secondly, the strategy of removing confounding labels from the training set. We further demonstrate in appendix A.6 that dynamically augmenting coarse annotations from fine-graines ones does not outperform the strategy of including annotations of all label styles in a conditioned model. While we find that the standard models are biased by coarser annotations (styles or 2) and the models trained on subsets might suffer from the missing training data, our proposed method enables the segmentation uncertainty model to incorporate all annotations regardless of label style in a meaningful way. This leads to the conditioned models performing best in predicting fine-graine fine-grained label style O annotations. The modifications made to the architectures for conditioning based on the new modelling objective can be easily implemented and do not incur heavier models, while yielding better results. Finally, the ability to incorporate annotations of many different label styles into training allows for using real- weal-world datasets as they are, thus increasing the applicability of segmentation uncertainty models to datasets as they occur in the wild.
6.1 LIMITATIONS AND FURTHER RESEARCH
Due to limited availability of labeled data, the correctness of the segmentation mask distributions learned by the different models can only be assessed cursorily (as done by means of the GED in Table 5). To perform more comprehensive validation of the learned distribution, experiments with synthetic data could be performed, similar to the synthetic experiments reported by Kohl et al. (2018). More generally, an evaluation of the impact of different label styles, and the performance of the style-conditioned models, in more and larger datasets is of interest. While the conditioned models in this work require discrete label style variable, an interesting direction for further research could be the conditioning on continuous label styles as well as considering settings in which different label styles present but not labelled.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This research was supported by the Novo Nordisk Foundation through the Center for Basic Machine Learning Research in Life Science (NNF200C0062606) and the Pioneer Centre for AI, DNRF grant number Pl and Denmarks Frie Forskningsfond (9131-00097B). We want to thank Prof. Sanjay Kumar the Department of Bioengineering, University of California at Berkeley, Berkeley CA (USA) for his permission to use his dataset Glioblastoma ghioblastoma-astroyyooon astrootttoo U373 cells on a polyacrylamide substrate' the context of this work. We further thank the organizers of the Cell Tracking Challenge.
REFERENCES
Naheed R. Abbasi, Helen M. Shaw, Darrell S. Rigel, Robert Friedman, William McCarthy, Iman Osman, Alfred W. Kopf, and David Polsky. Early diagnosis of cutaneous melanoma revisiting the ABCD criteria. JAMA, 292122)2771-2776, 12 2004. ISSN 0098-7484.
Anurag Amab, Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes: Mâns Larsson, Alexander Kirillov, Bogdan Savchynskyy, Carsten Rother, Fredrik Kahl, and Philip H.S. Torr. Conditional random fields meet deep neural networks for semantic segmentation: Combining probabilistic graphical models with deep learning for structured prediction. IEEE Signal Processing Magazine, 35(1):37-52) 2018.
Dhruv Batra, Payman Yadollahpour, Abner Guzzann-Rivers and Gregory Shakhnarovich. Diverse M-Best solutions in Markov random fields. Computer Vision (ECCV), pp. 1-16, 2012.
Christian F. Baumgartner, Kerem C. Tezcan, Krishna Chaitanya, Andreas M. Hötker, Urs J. Muehlematter, Khoschy Schawkat, Anton S. Becker, Olivio Donati, and Ender Konukoglu. PHiSeg: Capturing uncertainty in medical image segmentation. Medical Image Computing and Computer Assisted Intervention (MICCAI), pp. 119-127, 2019.
Christopher M. Bishop. Puttern Recognition and Machine Learning (Information Science and Statistics), Springer-Verlag. Berlin, Heidelberg, 2006. ISBN 0387310738.
Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC). In IEEE 15th International Symposium on Biomedical Imaging, pp. 168-172 172. IEEE, 2018.
Mare Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halper,, Susana Puig, et al. BCN20000: Dermoscopic lesions the wild. arXiv preprint arXiv: /908: 022888, 2019.
Steffen Czolbe, Kasra Amavazz, Oswin Krause, and Aasa Feragen. Is segmentation uncertainty useful? In Information Processing in Medical Imaging, pp. 715-726, Cham, 2021. Springer International Publishing. ISBN 978-3-030-78191-0
Loic Le Folgoc, Vasileios Baltatzis, Sujal Desai, Anand Devaraj, Sam Ellis, Octavio E Martinez Manzanera, Arjun Nair, Huaqi Qiu, Julia Schnabel, and Ben Glocker. Is me dropout Bayesian? 2021.
arXiv preprint
Yarin Gal and Zoubin Ghahramani. Bayesian convolutional neural networks with Bernoulli approxi- mate variational inference. arXiv preprint 1 05558, 2015.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. International Conference on Machine Learning, pp. 1050-1059, 2016a.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Appendix, 2016b.
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. A survey of uncertainty in deep neural networks, arXiv preprint 10Xiv:2107.088 2021.
10

---

Published as a conference paper at ICLR 2023
Konstantinos Kamnitsas, Christian Ledig, Virginia F.J. Newcombe, Joanna P. Simpson, Andrew D. Kane, David K. Menon, Daniel Rueckert, and Ben Glocker. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical Image Analysis, 36:61-78. 2017. ISSN 1361-8415.
Alex Kendall and Yarin Gal. What uncertainties do we need in Bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 30, 2017.
Alexander Kirillov, Bogdan Savchynskyy, Dmitrij Schlesinger, Dmitry Vetrov, and Carsten Rother. Inferring M-Best diverse labelings in single one. International Conference Computer Vision (ICCV), pp. 1814-1822, 2015a.
Alexander Kirillov, Dmytro Shlezinger, Dmitry P Vetrov, Carsten Rother, and Bogdan Savchynskyy. M M-Best-Diverse labelings for submodular energies and beyond. Advances in Neural Information Processing Systems, 28, 2015b.
Alexander Kirillov, Alexander Shekhovtsov, Carsten Rother, and Bogdan Savchynskyy. Joint M- best-diverse labelings as a parametric submodular minimization. Advances in Neural Information Processing Systems, 29, 2016.
Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? Does it matter? Structural Safery, 31(2):105-113 2009. ISSN 0167-4730.
Simon Kohl, Bernardino Romera-Paredes, Clemens Meyer, Jeffrey De Fauw, Joseph R Ledsam, Klaus Maier-Hein, SM Eslami, Danilo Jimenez Rezende, and Olaf Ronneberger. probabilistic U-Net for segmentation of ambiguous images, Advances in Neural Information Processing Systems, 31, 2018.
Simon A. A. Kohl, Bernardino Pamerr--arreds, Klaus H. Maier-Hein, Danilo Jimenez Rezende, S. M. Ali Eslami, Pushmeet Kohli, Andrew Zisserman, and Olaf Ronneberger. A hierarchical probabilistic U-Net for modeling multi-scale ambiguities. arXiv preprint 1905. 2019.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
Stefan Lee, Senthil Purushwalkam Shiva Prakash, Michael Cogswell, Viresh Ranjan, David Crandall, and Dhruv Batra. Stochastic multiple choice learning for training diverse deep ensembles. Advances in Neural Information Processing Systems, 29, 2016.
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint
arXiv:1411.178 2014.
Miguel Monteiro, Loic Le Folgoe, Daniel Coelho de Castro, Nick Pawlowski, Bernardo Marques, Konstantinos Kamnitsas, Mark van der Wilk, and Ben Glocker. Stochastic segmentation networks: Modelling spatially correlated alcatoric uncertainty. Advances in Neural Information Processing Systems, 13:12756-12767 2020.
Sebastian G Popescu, David J Sharp. James Cole, Konstantinos Kamnitsas, and Ben Glocker. Distributional Gaussian process layers for outlier detection in image segmentation. In International Conference on Information Processing Medical Imaging, pp. 415-427 Springer, 2021.
Christian Rupprecht, Iro Laina, Robert DiPietro, Maximilian Baust, Federico Tombari, Nassir Navab, and Gregory D Hager. Learning in uncertain world: Representing ambiguity through multiple hypotheses. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3591- 3600, 2017.
Raghavendra Selvan, Frederik Faye, Jon Middleton, and Akshay Pai. Uncertainty quantification in medical image segmentation with normalizing flows. In Machine Learning in Medical Imaging, Lecture Notes in Computer Science, Switzerland, 2020. Springer.
Gábor J. Székely and Maria L. Rizzo. Energy statistics: A class of statistics based on distances. 143 (8):1249-1272, 2013. doi: 10. 1i..000/..sp..211..3.

---

Published as a conference paper at ICLR 2023
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5(1):1-9, 2018.
Vladimír Ulman, Martin Maška, Klas E G Magnusson, Olaf Ronneberger, Carsten Haubold, Nathalie Harder, Pavel Matula, Petr Matula, David Svoboda, Miroslav Radojevic, Ihor Smal, Karl Rohr, Joakim Jaldén, Helen M Blau, Oleh Dzyubachyk, Boudewijn Lelieveldt, Pengdong Xiao, Yuexiang Li, Siu-Yeung Cho. Alexandre C Dufour, /ean-Christophe Olivo-Marin, Constantino C Reyes- Aldasoro, Jose A Solis-Lemus, Robert Bensch, Thomas Brox, Johannes Stegmaier, Ralf Mikut, Steffen Wolf, Fred A Hamprecht, Tiago Esteves, Pedro Quelhas, Ömer Demirel, Lars Malmström, Florian Jug, Pavel Tomancak, Erik Meijering, Arrate Muñio-Barrutia, Michal Kozubek, and Carlos Ortiz de Solorzano. An objective comparison of cell-tracking algorithms. 14(12):1141- 1152, 2017. doi: 10.1038/nmeth.447..
Theresa A. Ulrich, Elena M. de Juan Pardo, and Sanjay Kumar. The mechanical rigidity of the extracellular matrix regulates the structure, motility, and proliferation of glioma cells. 69(10): 4167-4174, 2009, doi: 0155/00088-4422.aa-00------8-
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 TRAINING PROCEDURE FOR THE ISIC AND PHC-U373 DATASETS
All models are trained using the Adam optimizer. For the probabilistic U-net, as well the conditioned probabilistic U-net, we minimize the reconstruction term given by the binary cross- entropy loss, added with a weighted Kullback-Leilerr divergence between the estimated normal distributions, as described by Kohl et al. (2018). The Stochastic Segmentation Networks are trained on the loss function described in Monteiro al. (2020). Across all datasets we used a learning rate of 10-4 train the models. For the skin lesion datasets, we trained for 600 epochs with batch size 16; for the PhC-U373 dataset, we trained for 200 epochs with a batch size 32. The above hyperparameters were retrieved by grid search the validation set. All computations were performed on an internal GPU cluster.
A.2 GENERALIZED ENERGY DISTANCE
In section 5.2, we the Generalized Energy Distance (Székely & Rizzo, 2013; Kohl et al., 2018) as a distance measure between distributions. It is defined as
DOED () 2 (y 0)] Ey.y' v')l 1g (d( j')],
where d is set to 1 - (o)... We set d = 0 if both segmentations are empty. Low GED values indicate high similarity between distributions. The expectations are approximated with 100 sample predictions from each model as done by Monteiro et al. (2020).
Note that when calculating the GED between the full annotator distribution p(a[z) and the conditioned models' predictive distribution, we need to draw samples from p(y(ær, 1). To this end, we sample / from a categorical distribution with density p(1 = kc) Pk. where Pk: could be set to for different
label styles or be estimated form the training dataset. We choose to set pk = 3/ for the ISIC dataset and 3 for the PhC-U373 dataset and predict a sample prediction with the respective model, conditioned on the drawn label style.
A.3 PIXEL-WISE ENTROPY
In section 5.3, we use the pixel-wise entropy as a measure of segmentation uncertainty. It is given by
H (p(ym) jr, 1)) p(uml1. log pogm(uml (1 - -p((mm m x 1)) log(1 p(ym |x, 1)) (8)
for the conditioned models and
H (p((ml|z) (x)) -p(ym (z) log p(yml xx) - p(ym (x)) log(1 p(ym (x))
for the baseline models, respectively, where 1x 1) and p(um /z) are the probabilitic of being of the target class pixel m of the models' predictive distributions.
A.4 SCHEMATIC MODEL ARCHITECTURES
sample 2e So
a- N/.. 3)
E- D ppT
(a)
(b)
(c)
PM
Figure 5: Schematic model architectures (adapted from Kohl et al. (2018) and Monteiro et al. (2020)) of the prob, U-net (a) and SSN (b) during inference time. Our modifications to the models are shown in green. (b) illustrates normal distribution over the logit pixel space predicted by SSN.
13

---

Published as a conference paper at ICLR 2023
A.5 LOss FUNCTIONS
A.5.1 CONDITIONED PROBABILISTIC U-NET
For training the prob. U-net and the conditioned prob. U-net, we implemented the loss function as described by Kohl al. (2018). While in the following we formulate the loss function for the conditioned prob. U-net case, want to remark that the implementations of the loss functions for the conditioned and the standard model cases doe not differ. For formulation of the loss function of the standard prob. U-net refer the reader to Kohl et al. (2018). For training sample (zn, 0 4.) prior net distribution P and posterior net distribution Q, the loss function calculated as
C(a Inz (1) Qe an log (aaaall(aa =)) + B (L(Q(=/2n o (2)
M
Sxm log ym (I 2) +1 mm log(1 - Vm (In z))
m=1
Binary Cross-Entropy Loss
+ for one sample from the posterior net distribution Q
(L(Q(=/amn ( (=
(=2 a (t)
M denotes the number of pixels for the images r and annotations a. Since both Pnd Q are Gaussian, the Kullback-Leibhe divergence has closed form solution.
A.5.2 CONDITIONED STOCHASTIC SEGMENTATION NETWORKS
SSNs relax the spatial independence assumption made for deriving the standard cross- entropy loss (compare equation 2) and learn a low-rank normal distribution over logits. The resulting loss function as derived in Monteiro et al. (2020) is given by
C(a In. 1 logsumexp &
log p(a p(a + log(S)
==1
M
= -logsumexp
di (og((((())) + (1 a log(1 - σ (n)(a + log(SS)
1==1
Binary Crass-Entropy
where a denotes the softmax function. It is calculated by drawing S Monte-Carbe samples from the logit distribution
,|., [ N (u(In, ) ((xx))))))
(10)
The loss is backpropagated using the reparameterization trick. Again, formulate the loss function for the conditioned case while remarking that does not differ in implementation from the standard SSN case discussed in Monteiro et al. (2020).
A.6 DYNAMIC AUGMENTATION FOR FINE-GRAINED ANNOTATIONS
In this section, we test whether similar performance gains can be obtained by a dynamic augmentation strategy that only uses fine-gred grained annotations. For this experiment, we start with the ISIC training dataset that was used for the conditioned models and the baseline models trained on all label styles (not the subsets). For each image the training set, coarse annotations are substituted by augmented versions of the fine-grained annotations of that image. If there are more than two fine-grained annotations available, one selected randomly. The augmentation is done by applying a dilation operation followed by Gaussian filter that smooths out the boundary. We train the c-prob. U-net and the c-SSN on this dynamically augmented training set utilizing the style labels. For comparison, we trained the baselines (prob. U-net and SSN) on this training set not using the style labels as described in section 4.2. The four new models are indicated by an (aug) tag. The models are then evaluated in terms of loU, GED and area bias following the exact same experimental setup as in section 5.2.
14

---

Published as a conference paper at ICLR 2023
Table 6: Average IoU and standard deviation for the models' mean prediction on subsets of the ISIC
dataset containing only one label style each.
IoU wrt. to subset of label style
Model
Prob. U-net (all)
0,65 (0.09)
0.65 (0.10)
0.69 (0.21)
Prob. U-net (subsets)
0.66 (0.11)
0.55 (0.17)
0.71 (0.15)
Prob.
U-net (aug)
0.68 (0.16)
0.61 (0.14)
0.53 (0.23)
c-prob. U-net (aug)
0.73 (0.17)
0.70 (0.09)
0.69 (0.14)
e-prob. U-net
0.77 (0.15)
0.78 (0.15)
0.76 (0.15)
SSNs (all)
0.72 (0.01)
0.77 (0.09)
0.71 (0.19)
SSNs (subsets)
0.61 (0.13)
0.75 (0.13)
0.61 (0.11)
SSNs (aug)
0.69 (0.09)
0.62 (0.12)
0.59 (0.15)
c-SSNs (aug)
0.75 (0.11)
0.66 (0. 14)
0.68 (0.16)
c-SSNs
0.77 (0.10)
0.71 (0.20)
0.78 (0.19)
Table 7: Mean GED (with standard deviation) between models' predictive distributions and the targeted annotator distributions p(a|æ; 1) of the available label styles as well as the full annotator distribution p(a[rr) of an ISIC test set.
ISIC
Annotator distribution
p(a|x,1 0)
p(a|x11=1)
p(a|x, = 2)
p(a|z))
Prob. U-net (all)
0.58 (0.14)
0.57 (0.23)
0.54 (0.14)
0.57 (0.17)
Prob. U-net (subsets)
0.61 (0.14)
0.57 (0.23)
0.51 (0.17)
Prob. U-net (aug)
0.59 (0.14)
0.61 (0.20)
0.62 (0.13)
0.61 (0.15)
c-prob. U-net (aug)
0.59 (0.12)
0.57 (0.21)
0.53 (0.21)
0.58 (0.10)
c-prob. U-net
0.57 (0.13)
0.55 (0.24)
0.49 (0.19)
0.55 (0.18)
SSNs (all)
0.59 (0.14)
0.55 (0.23)
0.51 (0.17)
0.55 (0.17)
SSNs (subsets)
0.61 (0.15)
0.55 (0.25)
0.55 (0.23)
SSNs (aug)
0.60 (0.14)
0.58 (0.22)
0.55 (0.16)
0.57 (0.11)
c-SNNs (aug) c-SNNs
0.58 (0. (4.14)
0.57 (0.23)
0.53 (0.20)
0.57 (0.15)
0.58 (0.13)
0.56 (0.28)
0.48 (0.19)
0.55 (0.20)
As seen in Figure 6, the conditioned models still able correct for area bias compared to the baseline models in this setting. However, segmentation and uncertainty quantification performance are reduced. The IoU decreases relatively (see Table 6) while GED increases (see Table 7). For the baseline models, area bias and segmentation performance is worse compared to all other models.
These results support our reasoning that coarse annotations contain additional information about annotator variability, while at the same time adding area bias. Figure 7 shows examples from the ISIC dataset that illustrate exactly this: The coarse annotations do not always contain the fine-grain -rainnd annotations. It is, therefore, not possible to capture all the annotator variability when using only augmentations of fine-grained annotations.
A.7 QUALITATIVE RESULTS ON THE ISIC DATASET
Figure& 8 and 9 show sample images from the ISIC dataset with overlayed annotations. Fig, 8 shows predictions (threshold 0.5) of the probabilistic U-net trained on all data, the prob. U-nets trained on the label style subsets and the c-prob. U-net conditioned on the different label styles. Fig. 9 shows predictions (threshold 0.5) of the SSN trained on all data, the SSNs trained on the label style subsets and the c-SSN conditioned the different label styles. In both figures, the last column shows an
15

---

Published as a conference paper at ICLR 2023
Overestimation lesion area for the ISIC dataset
20000
Difference from ground truth area
15000
10000
5000
0
-5000
-10000
-15000
20000
c-prob. U-net (I=0)
prob. U-net (subset 0)
cprob. U-net (aug) (I=0)
prob. U-net (all)
prob, U-net (aug)
c-SSN (1=0)
SSN (subset 0)
C-SSN (aug) (1=0)
SSN (all)
SSN (aug)
Figure 6: We assess bias in lesion and cell area estimation for the ISIC test dataset by distribution of area difference (in pixels) between 100 model predictions per image and respective label style ground-truth annotations. Markers indicate the mean.
==1
i 1=0
[= 0
0
Figure 7: Three examples from the ISIC dataset with their respective annotations. The last column shows a heatmap the average all three annotation masks. The respective label style | written on each annotation.
16

---

Published as a conference paper at ICLR 2023
overlay of the conditioned models' prediction when conditioning each of the three label styles for the given image. In both figures, it visible that the conditioning on label style corrects for the overestimation bias.
17

---

Published as a conference paper at ICLR 2023
Figure 8: Mean predictions of the different prob. U-net models for 5 images from the ISIC dataset. From left to right: Image; Overlayed annotations, prob. U-net (all); prob. U-nel (subset 0): prob. U-net (subset 1); prob. U-net (subset 2); c-prob. U-net conditioned on style O: c-prob. U-nel conditioned on style I; c-prob. U-net conditioned on style 2; Overlayed predictions from c- e-prob. U-net for each label style.
18

---

Published as a conference paper at ICLR 2023
Figure 9: Mean predictions of the different SSN models for images from the ISIC dataset. From left to right: Image; Overlayed annotations: SSN (all); SSN (subset 0): SSN (subsct 1); SSN (subset 2); c-SSN conditioned on style 0; c-SSN conditioned on style 1; c-SSN comditioned on style 2: Overlayed predictions from c-SSN for each label style.
19


Reviews:
[{"review_id": "pUPCVSm_SRU", "reviewer": "Reviewer_f94i", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores conditioning on image annotation labelling style, to reduce aleatoric uncertainty relating to the segmentation of objects in medical images. In particular, the general problem of uncertainty estimation for segmentation assumes that the annotation is drawn from an unknown annotator distribution, p(a|x) where a represents the (unknown) annotator, and x is the image. The main contribution then is the proposal to also condition on (discrete) label style l, i.e. p(a|x,l), where the label style is known. This label conditioning is implemented into two segmentation models, the (conditioned) probabilistic U-Net (CPUN), and the conditioned stochastic segmentation network (CSSN), such that both models can condition their predictions on any label style used in training. Experiments on two datasets (ISIC, PhC-U373) suggest that such conditioning results in improved segmentation performance (as measured by IoU), whether compared to training using all available labels (regardless of label style, without conditioning), or using only annotations of the same label style (thus losing information possibly obtainable from the other label styles).\n", "summary_of_the_review": "A label style conditioning is proposed and implemented in two previously-described deep neural network-based image segmentation models, which exhibits improved performance on two datasets compared to the same models without conditioning. However, some issues may remain with whether the comparisons are fully fair and whether apparent bias has been attempted to be mitigated with more conventional and common techniques, as detailed in the Strength and Weaknesses section.\n", "strength_and_weaknesses": "Strengths:\n\n1. The proposed label style conditioning method allows information on annotation style to be exploited, in segmentation tasks. Moreover, as coarse segmentations are (possibly greatly) less time-consuming for human graders to produce, the method may greatly reduce human effort, for large performance gains.\n\n2. Two model architectures are implemented, and experiments are exhaustively performed over combinations of training and testing data subsets, each containing one or all label styles.\n\n3. The distribution of overestimations/uncertainties are well-illustrated in suitable figures.\n\nPossible Weaknesses/Considerations:\n\n4. A major practical consideration about the proposed method, is the requirement for the label style to be known. For the experiments studied in this work, the datasets have annotations in prior-defined styles (e.g. Figure 2), which are then separately evaluated on as subsets each containing annotations of a single style. It might firstly be informative to evaluate performance where the annotation style is not a priori known.\n\n5. Related to the above, coarse annotations (e.g. Label Styles 1 & 2 in Figure 2) appear to be automatically obtainable from detailed annotations (Label Style 0), using some basic image processing operations (e.g. dilation, then smoothing of the boundary). If so, this may suggest that the coarse annotations actually do not really provide any (significant) new information, and it is possible that with just the detailed annotations for each image, coarse annotations can be generated and used as the additional coarse annotations for the proposed conditional learning. This would however then fall into the (common) use of dynamic augmentation (e.g. minor rotation/scaling, elastic deformation etc.) preprocessing of the image & annotation labels, in training deep learning classification/segmentation models.\nGiven this, it might be appropriate to explore whether automatically-generated coarse segmentations are sufficient to replicate the performance gain currently observed with manually-annotated coarse segmentations. If so, this would further inform on the actual source of the performance gain, and further reduce human annotation burden.\n\n6. Still related to the above, while area bias (and possibly other bias) is observed (Section 5.1), it is not clear the extent to which such bias has been attempted to be mitigated, particularly using the validation data. It is mentioned in Section 4.1 that the datasets have 20% of image-annotation pairs reserved as validation data, but it is not clear how this validation data has been used. It might be expected for area (and other) bias to be mitigated, by optimizing hyperparameters such as the pixel threshold (apparently a default threshold of 0.5 is mentioned in Section 5.3) used to separate the object from the background. This might be clarified.\n\n7. The “pixel-wise entropy” referred to in Section 5.3 might be explicitly defined (possibly in the appendix), for completeness.\n\n8. Theoretically, the formulation p(a|x,l) appears to still place emphasis on annotator distribution, while it might be supposed that inter-annotator label differences would be insignificant compared to inter-style label differences. This might be discussed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MPzjj7_5yHX", "reviewer": "Reviewer_9x5M", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper suggests that datasets could be heterogeneous since the label generation process might vary systematically, which the paper calls 'label style'. Based on this factor, it further proposes to condition probabilistic segmentation models on label style and to train the models on datasets containing differing label styles. The paper then describes how it modifies Probabilistic U-net and Stochastic Segmentation Networks (SSN) to condition them on label style, and the results of experiments show that compared with Probabilistic U-net and SSN, c-prob. U-net and c-SSNs are able to utilize all data regardless of label style in a meaningful way while introducing less segmentation bias.", "summary_of_the_review": "The paper proposes a novel hypothesis about the annotation style, and gives a clear description of the hypothesis and method, except for the description of the baseline methods. The experiments are very detailed and the results are convincing.", "strength_and_weaknesses": "Strength:\n1. The paper presents a clear style of writing. \n2. The paper includes detailed experiments and the results are convincing enough to support the hypothesis. \n3. The proposed method is simple but surprisingly efficacious and of great potential in increasing the applicability of segmentation uncertainty models. \nWeakness:\n1. The paper should present a more detailed description of the baseline methods in Sec.3.1 and Sec.3.2. \n2. Reasons or comparative experiments suggesting why choosing the modification methods might also be included. \n3. The sentence in Sec.1.1 ' We the proposed strategy’ should be ' We proposed strategy' instead.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "1UMwZv0GoP", "reviewer": "Reviewer_Pt9b", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new, updated segmentation uncertainty modeling objective based on the label style and ways to revise the segmentation uncertainty model architecture by including a discrete label style. ", "summary_of_the_review": "The paper presents an interesting idea and good experimental results. However, the evaluation needs improvement.  ", "strength_and_weaknesses": "Strength\n+ An interesting solution to reducing bias caused by aleatoric uncertainty and different label styles\n+ Experiments show that the conditioned models can outperform standard models\n\nWeaknesses\n- number of image-annotation pairs seems not large\n- only discrete label style is included, not the continuous\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster