Calibrated Adversarial Refinement for Stochastic Semantic Segmentation
Authors: Elias Kassapis
Georgi Dikov
Deepak Gupta
Cedric Nugteren
Venue:  (2021)
Keywords: stochastic semantic segmentation
conditional generative models
adversarial training
calibration
uncertainty

Abstract: Ambiguities in images or unsystematic annotation can lead to multiple valid solutions in semantic segmentation. To learn a distribution over predictions, recent work has explored the use of probabilistic networks. However, these do not necessarily capture the empirical distribution accurately. In this work, we aim to learn a calibrated multimodal predictive distribution, where the empirical frequency of the sampled predictions closely reflects that of the corresponding labels in the training set. To this end, we propose a novel two-stage, cascaded strategy for calibrated adversarial refinement. In the first stage, we explicitly model the data with a categorical likelihood. In the second, we train an adversarial network to sample from it an arbitrary number of coherent predictions. The model can be used independently or integrated into any black-box segmentation framework to facilitate learning of calibrated stochastic mappings. We demonstrate the utility and versatility of the approach by attaining state-of-the-art results on the multigrader LIDC dataset and a modified Cityscapes dataset. In addition, we use a toy regression dataset to show that our framework is not confined to semantic segmentation, and the core design can be adapted to other tasks requiring learning a calibrated predictive distribution.

Under review as a conference paper at ICLR 2021
CALIBRATED ADVERSARIAL REFINEMENT STOCHASTIC SEMANTIC SEGMENTATION
FOR
Anonymous authors Paper under double-bling review
ABSTRACT
Ambiguities in images or unsystematic annotation can lead to multiple valid so- lutions in semantic segmentation. To learn a distribution over predictions, recent work has explored the use of probabilistic networks. However, these do not neces- sarily capture the empirical distribution accurately. In this work, we aim to learn a multimodal predictive distribution, where the empirical frequency of the sampled predictions closely reflects that the corresponding labels in the training set. To this end, propose novel two-stage, cascaded strategy for calibrated adversar- ial refinement. In the first stage, we explicitly model the data with a categorical likelihood. In the second, we train adversarial network to sample from it an arbitrary number of coherent predictions. The model can be used independently integrated into any black-box segmentation framework facilitate learning of calibrated stochastic mappings. We demonstrate the utility and versatility of the approach by attaining state-of-the-ar results on the multigrader LIDC dataset and a modified Cityscapes dataset. In addition, we use toy regression dataset to show that our framework not confined to semantic segmentation, and the core design can be adapted to other tasks requiring learning a calibrated predictive distribution.
INTRODUCTION
Real-world datasets are often riddled with ambiguities, allowing for multiple valid solutions for given input. These can emanate from an array of sources, such as ambiguous label space (Lee et al., 2016), sensor noise, occlusions, and inconsistencies or errors during manual data annotation. Despite this problem, the majority of the research encompassing semantic segmentation focuses on optimising models that assign single prediction to each input image (Ronneberger et al., 2015; Jégou et al., 2017; Takikawa et al., 2019; Chen et al., 2017a;b; 2016a;b; 2015). These are often incapable of capturing the entire empirical distribution of outputs. Moreover, since they optimise for a one-fits-all solution, noisy labels can lead to incoherent predictions and therefore compromise their reliability (Lee et al., 2016).
Ideally, in such situations one would use model that can sample multiple consistent hypotheses, capturing the different modalities of the ground truth distribution, and leverage uncertainty information to identify potential errors in each. Further, the sampled predictions should accurately reflect the occurrence frequencies of the labels the training set; that is, the predictive distribution should be calibrated (Guo et al., 2017: Kull et al., 2019). Such a system would be particularly useful for hypothesis-drive reasoning human-in-the-loop semi-automatic settings. For instance, large scale manual annotation of segmentation map is very labour- intouur-ntensiveeeee each label in the Cityscapes dataset takes on average 1.5 hours annotate (Cordts et al., 2016). Alternatively, having human operator manually selecting from set of automatically generated label proposals could accelerate this process dramatically. In addition, combining uncertainty estimates with sampling self-consistent labels, can be used focus the annotator's 's attention to ambiguous regions, where errors are likely to occur, thereby improving safety.
Several approaches have been proposed to capture label multimodality in image-to-image translation tasks (Huang et al., 2018; Lee et al., 2018; Zhu et al., 2017a; Bao et al., 2017; Zhang. 2018), with only stochastic semantic segmentation (Kohl et al., 2018; 2019; Baumgartner few of them applied et al.. 2019; Hu et al., 2019; Kamnitsas et al., 2017: Rupprecht et al., 2017; Bhattacharyya et al., 2018). These methods have the capacity to learn diverse set of labels for each input, however, they

---

Under review as a conference paper at ICLR 2021
either limited to fixed number of samples (Kamnitsas et al., 2017; Rupprecht al., 2017), return uncalibrated predictions, or do not account for uncertainty.
In this work, tackle all three challenges by introducing two-stage cascaded strategy. In the first stage we estimate pixelwise class probabilities and in the second, we sample confident predictions, calibrated relatively to the distribution predicted in the first stage. This allows to obtain both uncertainty estimates as well as self-consistent label proposals. The key contributions are
We propose a novel cascaded architecture that constructively combines explicit likelihood modelling with adversarial refinement to sample arbitrary number of confident, and self ceff-consistent predictions given an input image.
We introduce a novel loss term that facilitates learning of calibrated stochastic mappings when using adversarial neural networks. To our knowledge this is the first work to do
The proposed model can be trained independently or used to augment any pretrained black- box semantic segmentation model, endowing it with a multimodal predictive distribution.
2 RELATED WORK
Straightforward strategies towards learning multiple predictions include ensembling (Lakshmi- narayanan et al., 2017: Kamnitsas et al., 2017) or using multiple prediction heads (Rupprecht et al., 2017). Even though these approaches can capture a diverse set of sampled predictions, they are limited to only a fixed number of samples. Alternatively, probability distribution over the outputs can be induced by activating dropout during test time (Gal and Ghahramani, 2016 This method does offer useful uncertainty estimates over the pixel-space (Mukhoti and Gal, 2018), however, Isola et al. (2017) and Kohl et al. (2018) have demonstrated that it introduces only minor stochasticity in the output and returns incoherent samples.
Bhattacharyya et al. (2018) identify the maximum likelihood learning objective as the cause for this phenomenon in dropout Bayesian neural networks (Gal and Ghahramani, 2016b). They postulate that under cross entropy optimisation, all sampled models are forced to explain all the data, and thereby converge to the mean solution. To counter that, they propose to replace the cross entropy with and adversarial loss term parametrising synthetic likelihood (Rosca et al., 2017), thereby making it conducive to multimodality.
contrast to this method, our approach is simpler to implement as not cast in the framework of variational Bayes, which requires the specification of weight priors and variational distribution family.
Kohl et al. (2018) take an orthogonal approach in combining U-Net (Ronneberger et al., 2015) with a conditional variational autoencoder (cVAE) (Kingma and Welling, 2013) to learn a distribution over semantic labels. In Kohl et al. (2019) and Baumgartner et al. (2019) the authors build on Kohl et al. (2018) to improve the diversity of the samples by modelling the data on several scales of the image resolution. Nonetheless, these methods do not explicitly calibrate the predictive distribution in the pixel-space, and consequently do not provide reliable aleatoric uncertainty estimates (Kendall and Gal, 2017; Choi al., 2018; Gustafsson al., 2019). Hu et al. (2019) address this shortcoming by using the intergrader variability as additional supervision. A major limitation of this approach is the requirement of a-priori knowledge of all the modalities of the data distribution. For many real- world datasets, however, this information is not readily available.
In the more general domain of image-to-iman translation, alternative methods employ hybrid models that use adversarially trained cVAEs (Zhu et al., 201 Bao et al., 2017) to learn a distribution over a latent code, capturing multimodality, in order to sample diverse and coherent predictions. common hurdle in conditional generative adversarial network (cGAN) approaches is that simply incorporating a noise vector as an additional input often results in mode collapse. This occurs due to the lack of regularisation between the noise input and generator output, allowing the generator to learn to ignore the noise vector (Isola et al., 2017). This issue is commonly resolved by using supplementary cycle-consistency losses (Huang et al., 2018; Lee et al., 2018; Zhu et al., 2017a; Bao et al., 2017), as proposed by Zhu et al. (2017b) or with alternative regularisation losses on the generator (Yang et al., 2018). However, none of these methods explicitly address the challenge of calibrating the predictive distribution.
Code is publicly available <URI. OMITTED FOR ANONYMITY>
2

---

Under review as a conference paper at ICLR 2021
3 PRELIMINARIES
Given an imput image 2 E | semantic segmentation refers to the task of predicting a pixel- wise class label y {l...., K] H For dataset of N image and label pairs, D {Ii, ji the conditional distribution (y z) can be explicitly modelled through a likelihood go z), parametrised by a convolutional neural network F with weights 6, and activated by softmax func- tion (Ronneberger et al., 2015; Jégou et al., 2017). One simple, yet effective way to learn the class probabilities is to express y E [0, 1 [XXXXX XX as one-hot encoded label, and set go as a pixelwise factorised categorical distribution, given by:
go x)
IIIIIII Po(x)
The parameters are then optimised by minimising the cross entropy between pr and qo, defined as:
Lee (D, 8) = -Epo(z.)) llg go x)]]
(2)
When trained with Eq. (2), Fg learns an approximation of Epp (y æt] (Bishop. 2006) that captures the pixelwise class probabilities over the label corresponding to given input. This accommodates the quantification of the amount of noise inherent to the data, also referred to as aleatoric uncertainty, which can be obtained by computing the entropy of the output of Fe. (()(())) (Kendall and Gal, 2017). Further, the final segmentation map is typically obtained by applying the argmax function along the class dimension of the predicted probabilities. However, due to its deterministic nature, this approach is unable to produce multiple alternative predictions for the same input image. On the other hand, direct sampling from qe yields incoherent semantic maps, as for noisy datasets, maximising the factorised likelihood results in unconfident predictions in regions of inter-label inconsistencies, e. g. fuzzy object boundaries, exemplified later in the model overview diagram in Fig. 1.
The aforementioned limitations can be partially addressed by adapting the framework of generative adversarial networks (GANs) (Goodfellow et al., 2014) to the context of conditional semantic segmentation, as proposed by Lue et al. (2016). Formally, this involves training binary discriminator network D to optimally distinguish between ground truth and predictions, while concurrently training generative network G to maximise the probability that prediction samples G(x) perceived as real by D. Importantly, in contrast to explicit pixelwise likelihood maximisation, the adversarial setup learns an implicit sampler through G, capable of modelling the joint pixel configuration of the synthesised labels, and capturing both local and global consistencies present in the ground truth.
In practice, the generator loss is often complemented with the pixelwise loss from Eq. (2) to improve training stability and prediction quality (Luc et al., 2016; Ghafoorian et al., 2018; Samson et al., 2019). However, we argue that the two objective functions are not well aligned in the presence of noisy data. While the categorical cross entropy optimises for a single solution for each input thus encouraging high entropy in { x) within noisy regions of the data and calibrating the predictive distribution, the adversarial term allows multiple solutions while optimising for low entropy, label-like output. Therefore combining these losses in an additive manner, and enforcing them on the same set of param- eters can be suboptimal. This issue can be mitigated to some extent by scheduled downscaling of Lee, however, the residual conflict between the two losses adversely affects the optimisation process.
4 METHOD
4.1 CALIBRATED ADVERSARIAL REFINEMENT
In this work, we propose to avert potential conflict between the cross entropy and adversarial losses by decoupling them in a two-stage cascaded architecture consisting of a calibration network Fo optimised with Lce from Eq. (2), the output of which is fed to a refinement network GG. Go is then optimised with an adversarial loss term, parametrised by auxiliary discriminator D trained with a binary cross entropy loss. To account for the multimodality in the labels, we additionally condition the refinement network also condition the refinement network and the discriminator on input x. however, we do not show this explicitly for notational convenience. More formally, using the non-saturated version of the
extraneous noise variable
A/ 1). In practice,
3

---

Under review as a conference paper at ICLR 2021
Example labels y
Final predictions Veer Siets on Buef
Calibration network
Fee
Refinement network
Discriminator Dg
Input image at
Fg
G
Lp
LG
Alcatoric uncertainty (l(F(()))
Gaussian noise
Average final prediction
Bee
Figure 1: Model overview with an illustrative example where red and blue pixels are vertically segmented. A fuzzy boundary in the input image allows for multiple valid ground truth labels y First, the calibration network maps the input to a calibrated pixelwise distribution over the labels. This is then fed into the refinement network which samples an arbitrary number of diverse, crisp label proposals ......., - calibration target from the first stage through the Leat loss. Additionally, the aleatoric uncertainty can be readily extracted from the calibration target, e. g. by computing the entropy (H)())))).
.M To ensure calibration, the average of the final predictions is matched with the
adversarial loss (Goodfellow et al., 2014), the objectives for the refinement and discriminator networks respectively are given by:
Ladv (D,0, 0) E. (pp(a.)))(e) log (c((G)(F(()))))))
CD 0, d. v) -Bpo(xx))xtc)) [log (1 Dy e))) + log D (v))
(4)
To calibrate the predictive distribution, we impose diversity regularisation Go by introducing novel loss term that encourages the sample average (FF E (F) c)] to match the class probabilities predicted by Fx()). Here, G& (x)) serves as an approximation the implicit predictive distribution of the refinement network. To this end, define auxiliary fully-factorised categorical likelihood as:
(F
K 2o Fz())) G
where & is optimised by minimising the Leibler divergence KL(gs ge). Since both d and qo are categorical distributions, the divergence can be computed exactly. We coin this loss term as the calibration loss, defined as:
Leal (D, Φ) Epp [Tgg [log (1 Fo(x)) log go x)]]]
Since Leal optimises through (Fo rather than single sampled prediction, the model is not restricted to learning a single mode-ave boode-vvergging solution for each input x, and is therefore more compatible with Ladv- The total loss for the refinement network then becomes:
(D(00,,0) Ladv (D, 0, () ACeal (D,0.
where À > 0 is adjustable hyperparameter. Fig. 1 shows the interplay of Fe. Go and D& and the corresponding loss terms.
Intuitively, the calibration network Fa serves three main purposes. It accommodates the extraction of sample-fre free aleatoric uncertainty maps. It provides G with an augmented representation of x enclos- ing probabilistic information about y. Finally, it sets calibration target used by Leu to regularise the
The choice of divergence is heuristically motivated and can be changed to fit different use-case requirements. We delegate theoretical and experimental analysis of other divergences future work.
4

---

Under review as a conference paper at ICLR 2021
predictive distribution of Gg. in cycle- manner (Zhu al., 2017b). The refinement net- work can then be interpreted as a stochastic sampler, modelling the inter-pixel dependencies to draw self-consisten samples from the explicit likelihood provided by the calibration network. Thus both the pixelwise class probability and object coherency are preserved. This approach leads to improved mode coverage and training stability, and increased convergence speed, as demonstrated in Section 5.
4.2 PRACTICAL CONSIDERATIONS
An important consequence of the loss decomposition is that the weights of Fe be kept fixed, while the adversarial pair Go and D are being trained. This allows F9 to be pretrained isola- tion, consequently lowering the overall peak computational burden and improving training stability (see Algorithms and 2 in Appendix A. for an outline of the training and inference procedures). Further, computing Leal requires Monte Carlo estimation of G (x)).. where the quality of the loss feedback increases with the sample count. However, modern deep learning frameworks allow for the samples be subsumed in the batch dimension, and can therefore be efficiently computed on GPUs. We also note that on noisy data points, the trained model does not require to be provided with all solutions for given input. Instead, during training random input-label pair is sampled, and the model automatically learns to identify cases where the input is ambiguous. Finally, our method can augment any existing black-box model B for semantic segmentation, furnishing it with a multimodal predictive distribution. This can be done by conditioning Fg on the output of B, which we demonstrate Section 5.2.2.
EXPERIMENTS
5.1 ID BIMODAL REGRESSION
We give intuitive insight into the mechanics of the proposed calibration loss by designing and experimenting on a simple one-dimensional regression task. To create the dataset, an input æ E [0, mapped y as follows:
0.5 b+c (-1) + +
x E [0, 0.4)
=
E 00.4, 0.8)
(8)
[0.8,
where b Bernoulli(1 (T) and c A/0 o). We generate 9 different scenarios by varying the degree
of mode selection probability æ E {0.5, 0.6, 0.9) and the mode noise E {0.01, 0.02, 0.03).
Average log. log-likelihood
0.
mediun thaseline) mediam (with faal
-02-
-02
-18
(basclin))
0.4
-044
Gy(660).))
Gy(R
200 Training iteration (a)
0.2
0.4
0.6
02
0.4
0.6
(b)
(c)
Figure 2: (a) Median and interquartile range (iqr) over the data log-likelihood, averaged over all 9x5x2 experiments. (b) High bias and noise configuration (I 0.9, 0.03) with calibration loss. The ground truth target shown as black dots and the predicted samples as light blue dots. The predictions average in dark blue matches the calibration target in red. The discriminator output shown in the background in shades red (real) and blue (fake). (c) The same experiment configuration but without the proposed calibration loss, resulting in mode collapse.
5

---

Under review as a conference paper at ICLR 2021
For every data configuration, without calibration loss by setting the coefficient À in Eq. (7) to or O, respectively. All runs are trained with learning rate of 1e-4, and each experiment is repeated five times. Note that unlike the categorical likelihood used in semantic segmentation tasks, we employ a Gaussian likelihood with fixed scale of 1. This changes the formulation of both Eqs. (2) and (6) to mean squared error losses between ground truth labels y and predictions ÿ for Lee, and between the output of the calibration network (x:) and the average of multiple predictions (F(FF(()) for Leal (see Appendix A.2).
4-layer MLP for each of Fe, Go and De., and train with and
The results, depicted in Fig. 2, illustrate that when using calibration loss, the optimisation process shows improved stability, converges faster and results in better calibrated predictions, in comparison to the non-regularised baseline. The effect is more pronounced in data configurations with higher bias. Further plots of the individual experiments are presented in Appendix B.
5.2 STOCHASTIC SEMANTIC SEGMENTATION
In this section we examine the capacity of our model to learn shape and class multimodality in real-world segmentation datasets. We begin by sharing essential implementation details below.
Network architectures For the calibration network Fe. we use the encoder-decoder architecture from SegNet (Badrinarayanan et al., 2017). For Go, we designed a U-Net-style (Ronneberger et al., 2015) architecture with 4 down- and upsampling blocks, each consisting of convolutional layer, followed by a batch normalisation layer (Ioffe and Szegedy, 2015), a leaky ReLU activation, and dropout layer (Srivastava et al., 2014) with 0.1 dropout probability. We use base number of 32 channels, doubled or halved at every down- and upsampling transition. To propagate the sampled noise vector the output, we inject it into every upsampling block of the network in affine manner. To do so, we project the noise vector using two fully connected layers into scale and residual matrices, with the same number of channels as the feature maps the points of injection, and use these matrices to adjust the channel-wise mean and variance of the activations. This is similar to the mechanism used for adaptive instance normalisation (Huang and Belongie, 2017). We base the architecture for De on that used in DC-GAN (Radford et al., 2015) except that we remove batch normalisation. Any deviations from this setup are described in the corresponding sections.
Training details We utilise the Adam optimiser (Kingma and Ba, 2014) with an initial learning rate of 2c-4 for Fo and Go. and 1c-5 for Dz- The learning rates are linearly decayed over time and we perform scheduled updates to train the networks. Additionally, the discriminator loss is regularised by using the R2 zero-centered gradient penalty term (Mescheder et al., 2018). For a detailed list of hyperparameter values, see Appendix A.I.
Metrics Following Kohl et al. (2018; 2019), Huang et al. (2018) and Baumgartner et al. (2019), we use the Generalised Energy Distance (GED) (Szčkely and Rizzo, 2013) metric:
(9) where d(s, v) 1 - Lou 3). As additional metric, we follow Kohl et al. (2019) in using the Hungarian-matee matched IoU (HM-IoU) (Kuhn, 2005). In contrast to GED, which naively computes diversity as 1-loU between all possible pairs of ground truth sampled predictions, HM-IoU finds the optimal 1:1 matching between all labels and predictions, and therefore is more representative of how well the learnt predictive distribution fits the ground truth.
DeD 4a) = 2T 2 (d(s v)] Bg,sr~ag [d(s, 8 Ryss'pp [d(y. y/),
All experiments are performed in triplicate and we report results as mean and standard deviation. Further details regarding the exact implementation of the GED and HM-IoU metrics for each experiment can be found in Appendix A.3.
5.2.1 LEARNING SHAPE DIVERSITY ON THE LIDC DATASET
The Lung Image Database Consortium (LIDC) (Armato III et al., 2011) dataset consists of 1018 thoracic CT scans from 1010 lung cancer patients, graded independently by four expert annotators. We use the 180x180 crops from the preprocessed version of the LIDC dataset used and described in Kohl et al. (2018). The dataset is split in 8882, 1996 and 1992 images in the training, validation and test sets respectively. All models are trained on lesion-centered 128x128 crops where at least one of the four annotations indicates a lesion. The final evaluation is performed on the provided test set.

---

Under review as a conference paper at ICLR 2021
Table 1: Mean GED and HM-IoU scores on LIDC. Top section: approaches using the original data splits defined by Kohl et al. (2018), which we also adhere to; middle: approaches using random data splits; bottom: the regullair baseline and our leg-lagulrrseed eGAN, The three central columns show the GED score computed with 16, 50 and 100 samples, respectively, The rightmost column shows the HM-IoU score, computed with 16 samples. The arrows and & indicate if higher or lower score better.
Method
GED (16)
GED + (50)
GED | (100)
HM-loU 1 (16)
Kohl et al. (2018)
0.320 + 0.030
0.252 + N/A
0.500 + 0.030
Kohl al. (2019)
0.270 = 0.010
0.530 + 0.010
Hu et al. (2019)
0.267 0.012
Baumgartner et al. (2019)
0.224+N/A2
eGAN+Lcc CGAN+Ceal
0.639 0.002
0.477 + 0.004
0.264 = 0.002 0.248 0.004 0.243 + 0.004 0.592 + 0.005
This score is taken from Baumgartner et al. (2019).
Note that when following the data split methodology used in Baumgartner et al. (2019) and computing the GED (100) metric, we achieve a score of 0.228 instead of 0.243 (see Appendix A.3).
x
Fgt
Fg(x)
Fief
Vndd
V
Var
y
+
0.5
Figure 3: LIDC validation samples. From left right: input image x, followed by the four ground truth annotations .g. Ug. the mean of the labels Ugr the output of the calibration network Fo(x), the mean of the six refinement network samples Brer. shown in columns higt......
To assess the accuracy and diversity of samples generated by our model we examine how well the learnt conditional distribution captures the shape diversity of lesion segmentations in the dataset. We pretrain Fe (x) with L.e isolation, fix its weights, and then train G2 with LG, where we estimate Leal with 20 samples from Gg. Further experiments with varying sample size disclosed in Table in Appendix B.2. As a control experiment, we train using the same architecture but replace Leal in the refinement network loss function with
entropy loss Lee. used in Luc et al. (2016).
The -eau-regllrrised model performs on par with other state-of-the the-art art methods w. r. t. the GED score, and outperforms them the HM-IoU score (only available for Kohl et al. (2018; 2019)). Numerical results are summarised in Table The diversity and fidelity of sampled predictions are illustrated in Fig. 3. In contrast, the -ee-lagulrr baseline collapses the predictive distribution, showing perceptible diversity in the samples (see Fig. 10b in Appendix B.2.3), which results in a stark increase the mean GED score, and decrease in the HM-IoU score, as shown the bottom section of Table 1.
5.2.2 LEARNING CALIBRATED DISTRIBUTION ON MULTIMODAL CITYSCAPES DATASET
The Cityscapes dataset contains 1024x2048 RGB images of urban scenes, and their corresponding segmentation maps. It consists of 2975 training. 500 validation and 1525 test images. Following Kohl al. (2018), we use the version of the Cityscapes dataset with 19 semantic classes and downsampled images and segmentation maps at a spatial resolution of 256x512. They establish controlled multi- modality by augmenting the dataset with 5 new classes: sidewalk2, person2, car2, vegetation2 and road2, introduced by flipping their original counterparts with probabilities $/17, 7/17, 6/17, 5/17 and 4/17, respectively (see Fig. 4a). Following Kohl et al. (2018), we report results on the validation set.
7

---

Under review as a conference paper at ICLR 2021
mer
po
vegdatio
prnza
Ear
bays
out
(b)
(c)
Figure 4: (a) Input images overlaid with the corresponding labels. (b) Samples obtained from the refinement network. (c) Aleatoric uncertainty computed as the entropy the calibration output.
To demonstrate that our approach can be easily integrated on top of any existing black-box segmenta- tion model B, we employ the network from TensorFlow Deepl aa Model 7oo (2020), trained on the official Cityscapes dataset, which achieves mloU of 0.79 on the test set. We utilise its predictions as input to our calibration network, Fe, which consists of 5 convolutional blocks, each composed of a 3x3 convolutional layer, followed by a batch normalisation layer, a leaky ReLU activation, and a dropout layer with 0.1 dropout rate. We pretrain the calibration network in isolation, and subsequently apply it in inference mode while adversarially training the refinement network. We use a batch size of 16, and train with LG, estimating Leat with 7 samples from Gø- The same baseline in the LIDC experiment is employed, where we replace Leat with Loe- As second control experiment we completely omit the calibration network and instead condition the refinement network on the known ground truth pixelwise categorical distribution over the label. This allows us to directly evaluate the quality of sampling administered from the refinement network.
When training the refinement network with an additional cross entropy loss instead of the calibration loss Leal, the predictive distribution collapses, making the output deterministic. Conversely, when we train our refinement network with Leat, the learnt predictive distribution is well adjusted, with high diversity and reconstruction quality, significantly outperforming the current state-of-the-arr, as shown in Table 2. Fig. 4b displays representative sampled predictions from our model for three input images, and Fig. 4c illustrates the corresponding alcatoric uncertainty maps extracted from Fo(x). The learnt multimodality and noise in the dataset is reflected by regions of high uncertainty, where objects belonging to the different stochastic classes consistently display distinct shades of red, corresponding their respective flip probabilities. Finally, we show that when using the ground truth pixelwise distribution as the input to the refinement network, we attain an almost perfect GED score (0.038 0.00).
cilibrztion network
Table 2: Mean GED scores the modi- fied Cityscapes. Top section: competing model; middle: Leaulaguler cGAN and Lee-regularised baseline; bottom: ground truth calibrated cGAN The GED scores are computed using 16 samples.
Pixelwise probubility
seficemeni netwock network (with groun tuth)
Method
GED
sidewalk2
person2
car2
vegetation2
road2
Kohl et al. (2018)
0.206 + N/A
Figure 5: Calibration of the pixelwise probabil- ities of the five stochastic classes. Note that the calibration network (in orange) is conditioned on black-box predictions.
cGAN+ Ceal
0.164 + 0.01
cGAN+ L
0.632 + 0.07
cGAN+Ccal (gt)
0.038 + 0.00
8

---

Under review as a conference paper at ICLR 2021
directly Since manually set the flip probabilities for each stochastic class in this dataset, assess the calibration of our model by comparing the ground truth probabilities to the predicted probabilities from the calibration or refinement network. For Fa for each class, and for G. we obtain the empirical class probabilities via Go( (o())). computed from 16 samples (see Appendix A.4 for more details). The ensuing results shown qualitatively in Fig. which illustrates the calibration of our models on the stochastic classes, evaluated over the entire dataset. This demonstrates that our models are well calibrated, with the calibration offiset, computed as the absolute difference between the ground truth and predicted probabilities, being approximately 6% in the worst case (class "car2" for the calibration network). Note that the average calibration offset for (1( across the stochastic classes is 1.6%. Further, the ground truth conditioned baseline almost perfectly calibrated, reflecting the near-optimal GED score reported in Table Thus, we demonstrate that G learns calibrated refinement of the predictions from Fg. where the quality of the final predictive distribution depends on the quality of (z).
the mean confidence values
In order to further scrutinise the calibration qual- ity of (z), we construct a reliability diagram and compute the corresponding expected calibration error (ECE), as proposed by Guo et al. (2017). To create the diagram each pixel is considered independently and the associated class confidences are binned into
1.0
accuracy per bin
0.9
calibration
0.8
0.7-
Accuracy
0.6
10 equal intervals of size 0.1. We then compute the ac- curacy for all predictions in each bin. Fig. 6 shows the reliability diagram for the calibration network, where the orange bars depict the calibration gap, defined as the difference between the mean confidence for each interval and the corresponding accuracy. The corre- sponding ECE score amounts to 2. 2.1.. Note that this also considers the average calibration error puted for the stochastic classes, where we randomly sample the labels according the defined probabili- ties. Hence, we confirm that (x) is well calibrated.
0.5
0.4
0.3
0.2
0.1
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.7
0.8
0.9
1.0
Confidence
Figure 6: Reliability diagram for the calibra- tion network. ECE = 2.15%.
An important outstanding issue in approach is that the calibration network may not perfectly capture the class taass-prbbabiltties, e. g. for the car2 category in Fig. 5. The limitations of modern neural networks with respect calibration, as well possible solutions, Kull et al., 2019; Zhang et al., 2020). Miscalibration has been attributed to several factors, such as long-tailes data distributions, -ittt--ddistiittiinn inputs, specific network architectural elements or optimising procedures. However, since our approach seeks to calibrate relative to the calibration target, and none of the aforementioned issues directly interfere with this process, we consider the problem of absolute calibration to be beyond the scope of this work. Nevertheless we emphasise on its importance for further improving the sample quality.
well studied (Guo et al., 2017;
6 CONCLUSION
In this work, we developed novel framework for semantic segmentation capable of learning calibrated multimodal predictive distribution, closely matching an estimation of the ground truth distribution of labels. We attained improved results on modified Cityscapes dataset and competitive scores on the LIDC dataset, indicating the utility of our approach on real-world datasets. We also showed that our approach can be easily integrated into an off-the-shelf deterministic, black-box semantic segmentation model, enabling sampling arbitrary number of plausible segmentation maps. By providing multiple valid label proposals and highlighting regions of high data uncertainty, our approach can be used to identify and resolve ambiguities, diminishing risk in safety-critical systems. Therefore, we expect our approach to be particularly beneficial for applications such as map making for autonomous driving or computer-assisted medical diagnostics. Finally, even though the primary focus of this work is semantic segmentation, we demonstrated its versatility through an illustrative toy regression problem, alluding broader applicability beyond semantic image segmentation.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Stefan Lee, Senthil Purushwalkam Shiva Prakash, Michael Cogswell, Viresh Ranjan, David Crandall, and Dhruv Batra. Stochastic multiple choice learning for training diverse deep ensembles. Advances in Neural Information Processing Systems, pages 2119-2127, 2016.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234 234-241. Springer, 2015.
Simon Jégou, Michal Drozdzal, David Vazquez, Adriana Romero, and Yoshua Bengio. The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 11 11-19, 2017.
Towaki Takikawa, David Acuna, Varun Jampani, and Sanja Fidler. Gated-senn: Gated shape enns for semantic segmentation. In Proceedings of the IEEF International Conference on Computer Vision, pages 5229-5238,
2019.
Liang-Chich Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2017a.
Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arxiv:7706 1706.05887, 2017b.
Liang-Chieh Chen, Yang. Jiang Wang. Wei Xu, and Alan L Yuille. Attention to scale: Scale-aware semantic image segmentation. In CVPR, 2016a.
Liang-Chieh Chen, Jonathan T Barron, George Papandreou, Kevin Murphy, and Alan L Yuille. Semantic image segmentation with task-specific edge detection using cnns and discriminatively trained domain transform. In CVPR, 2016b.
Liang-Chise Chen, George Papandreou, Jasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Semantic image segmentation with deep convolutional nets and fully connected erfs. In ICL.R, 2015.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Muchine Learning- Volume 70, pages 1321-1330. JMLR. org, 2017.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In Advances in Neural Information Processing Systems, pages 1 2295-12305, 2019.
Marius Cordis, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213 --233, 2016.
Xun Huang, Ming-Yo Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 172-189, 2018.
Hsin- Ying Lee, Hung- Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Diverse image-to- image translation via disentangled representations. In Proceedings of the European conference on computer vision (ECCV), pages 35-51, 2018.
Jun-Yan Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros. Oliver Wang. and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in neural information processing systems, pages 465 476,46662266 2017a
Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. Cvae-gan: fine-grained image generation through asymmetric training. In Proceedings of the IEFE International Conference on Computer Vision, pages 2745-2754, 2017.
Yonggi Zhang. Xogan: One-to-many unsupervised image-to-image translation. arXiv preprint 18050077,, 2018.
Simon Kohl, Bernardino Camera-Paredes Clemens Meyer, Jeffrey De Fauw, Joseph R Ledsam, Klaus Maier- Hein, SM Ali Eslami, Danilo Jimenez Rezende, and Olaf Ronneberger. A probabilistic u-net for segmentation of ambiguous images. In Advances Neural Information Processing Systems, pages 6965-6975. 2018.
10

---

Under review as a conference paper at ICLR 2021
Simon AA Kohl, Bernardino Romera-Paredes, Klaus II Maier-Hlein, Danilo Jimenez Rezende, SM Eslami, Pushmeet Kohli, Andrew Zisserman, and Olaf Ronneberger. hierarchical probabilistic u-net for modeling multi-scale ambiguities. arXiv preprint arXiv: 2019.
Christian F Baumgartner, Kerem C Tezcan, Krishna Chaitanya, Andreas M Hötker, Urs J Muehlematter. Khoschy Schawkat, Anton S Becker, Olivio Donati, and Ender Konukoglu. Phiseg: Capturing uncertainty in medical image segmentation. In International Conference on Medical Image Computing and Computer-Asiisted Intervention, pages 119-127. Springer, 2019.
Shi Hu, Daniel Worrall, Stefan Knegt, Bas Veeling, Henkjan Huisman, and Max Welling, Supervised uncertainty quantification for segmentation with multiple annotations. In International Conference Medical Image Computing and Computer-Assisted Intervention, pages 137-145. Springer, 2019.
Konstantinos Kamnitsas, Wenjia Bai, Enzo Ferrante, Steven McDonagh, Matthew Sinclair, Nick Pawlowski, Martin Rajchl, Matthew Lee, Bernhard Kainz, Daniel Rueckert, et al. Ensembles of multiple models and architectures for robust brain tumour segmentation. International MICCAI Brainlesion Workshop, pages 450 462. Springer, 2017.
Christian Rupprecht, Iro Laina, Robert DiPietro, Maximilian Baust, Federico Tombari. Nassir Navab, and Gregory D Hager. Learning in an uncertain world: Representing ambiguity through multiple hypotheses. In Proceedings of the IEEE International Conference on Computer Vision, pages 3591 1-3600, 2017.
Apratim Bhattacharyya, Mario Fritz, and Bernt Schiele. Bayesian prediction of future street scenes using synthetic likelihoods. In International Conference on Learning Representations, 2018.
Balaji Lakshninarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles, In Advances in neural information processing systems, pages 6402 6413, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059, 2016a.
Jishnu Mukhoti and Yarin Gal. Evaluating bayesian deep learning methods for semantic segmentation. arXiv preprins arXiv: 181 1.811.122 12709, 2018.
Phillip Isola, Jun-Yan Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1 1225-1144 2017.
Yarin Gal and Zoubin Ghahramani. Bayesian convolutional neural networks with bernoulli approximate variational inference. ICLR workshop track, 2016b.
Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Variational approaches for auto-engoding generative adversarial networks. arXiv preprint arXiv:1706. 04987, 2017.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Advances in neural information processing systems, pages 5574-5584 2017.
Sungjoon Choi, Kyungjae Lee, Sunghin Lim, and Songhwai Oh. Uncertainty-awaa learning from demonstration using mixture density networks with sampling free variance modeling. In 2018 IEEE International Conference on Robotics and Automation (ICRA). pages 6915-6922. IEEE, 2018
Fredrik K Gustafsson, Martin Danelljan, and Thomas B Schön. Evaluating scalable bayesian deep learning methods for robust computer vision. arXiv preprint 1906v 2019.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexci A Efros. Unpaired image-to-ing -o-igge--- translation using cycle-consistent adversarial networks. In Proceedings of the IEFE international conference on computer vision, pages 2223-2232, 2017b.
Dingdong Yang. Seunghoon Hong, Yunscok Jang, Tianchen Zhao, and Honglak Lec. Diversity Diversity-sensitive sensitive conditional generative adversarial networks. In International Conference on Learning Representations, 2018.
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair. Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
11

---

Under review as a conference paper at ICLR 2021
Pauline Luc, Camille Couprie, Soumith Chintala, and Jakob Verbeek. Semantic segmentation using adversarial networks, arXiv preprint 1.10:400 2016.
Mohsen Chafoorian, Cedric Nugteren. Nóra Baka, Olaf Booij, and Michael Hofmann. El-gan: Embedding loss driven generative adversarial networks for lane detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 0-0, 2018.
Laurens Samson, Nanne van Noord, Olaf Booij, Michael Hofmann, Efstratios Gavves, and Mohsen Ghafoorian. I bet you are wrong: Gambling adversarial networks for structured semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0-0, 2019.
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: deep convolutional encoder-decoder architccture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 39())) 2481-2495, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 1502.03167, 2015.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929-1958, 2014.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision, pages 1501-1510, 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 1.5 1.06444, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 2014.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? arXiv preprint arXiv:1801.0 1801.04406. 2018.
Gábor Székely and Maria L Rizzo. Energy statisties: A class of statisties based distances. Journal of statistical planning and inference, 143(8):1249 1272 2013.
Harold W Kuhn. The hungarian method for the assignment problem. Naval Research Logistics (NRL), 52(1): 7-21, 2005.
Samuel G Armato III, Geoffrey FMcLennan, Luc Bidaut, Michael F McNitt-Gray, Charles R Meyer, Anthony P Reeves, Binsheng Zhao, Denise R Aberle, Claudia I Henschke, Eric A Hoffman, et al. The lung image database consortium (lide) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans. Medical physics, 38(2)9915-931, 2011.
TensorFlow DeepLab Model Zoo. Pretrained model: Sepptionn1_dpe_citeee citseccccssccsttttllie trainfine, 2020. URL / wwwpeepbeectscceeeeeeeeeeeeeeeee httpapess_xxppiioooeeeee
download. tensorflow fow. 1001__9_008__0
2018 09 00.0_000__000 tar. gz.
Jize Zhang, Bhavya Kailkhura, and T Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. arXiv preprint arXiv:2003.07329, 2020.
Alex Graves. Practical variational inference for neural networks. In Advances in neural information processing systems, pages 2348-2356, 201 1.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mo- hamed, and Alexander Lerchner. beta-VAE: Learning Basic Visual Concepts with Constrained Variational Framework. Iclr, 2(5):6, 2017.
Anders Boesen Lindbo Larsen, Sóteen Kaac Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. arXiv preprint a1Xiv:1512.09300, 2015.
Danijar Hafner, Dustin Tran, Timothy Lillicrap, Alex Irpan, and James Davidson. Noise contrastive priors for functional uncertainty. arXiv preprint 1807,09999 2018.
Christos Louizos, Xiahan Shi, Klamer Schutte, and Max Welling. The functional neural process, arXiv preprint arXiv: 08324, 2019.
12

---

Under review as a conference paper at ICLR 2021
Appendices
A IMPLEMENTATION DETAILS
In this section describe the overall training procedure and delve into the training and evaluation details for the stochastic segmentation experiments on the LIDC datasct and the modified Cityscapes dataset.
A.I TRAINING PROCEDURE
Algorithm outlines the practical procedure used to pretrain the calibration network, and the subsequent training of the refinement network. Even though the networks we use two-step training procedure to stabilise the training and reduce the memory consumption on the GPU. This way we are able to fit larger batches and/or more samples for the estimate of Leal Algorithm shows the inference procedure for obtaining M output samples.
trained end-to-end once, in our experiments
Algorithm 1 Model training with calibration network pretraining
require: training data 'D, number of samples M, learning rate 7, calibration loss scale A; 1: procedure TRAINING( .,, 1M,n,.. X)
2: while not converged
Pretraining of Fo
3: Sample batch {xz, y) E D
4: Update 0,+1 with V/ (oo((2
5: end while
6: while not converged do
Adversarial training of G and D
7: Sample batch {x, 1/ D
8: for i = 1,2, M do
9:
Sample Viel Go, (Fo- (1xt), where ~ ((011)
10:
end for
11:
Compute Go(Fo(T)) = M -i Viet
EM
12:
Compute Ceal(Te, 0r 00) Eino (Go(Fo(z))) (log Go(Fo(z)) - log Fe- (xi))) iig. Update des1 with -1 (La(0" Di, (x. + ( 0", d.))
13:
14:
Update Ve+1 with V LD( de the (2,9))
15:
end while
16:
end procedure
Algorithm 2 Inference procedure
require: data point æ, number of samples M;
procedure INFERENCH(2, M)
b Using 0* and Ø² from Algorithm
2: for i 1,2,.... M do
3: Sample Viet = Go- (x), where e; ~ 100,
4: end for
5: end procedure
Notice that any off-the-shelf optimisation algorithm can be used update the parameters 6, é and t.. For the segmentation experiments, utilise the Adam optimiser (Kingma and 2014) with 31 = 0.5, and weight decay 5e-4. Fo is trained with a learning rate of 2e-4 which is then lowered to le-4 after 30 epochs. Go and D. are updated according to a schedule, where Gg is updated at every iteration, and D trained in cycles of 50 iterations of weight updating. followed by 200 iterations with fixed weights. The refinement network is trained with initial learning rate of 2e-4, lowered le-4 after 30 epochs, whereas the discriminator has initial learning rate le-5, lowered to 5e-6 after 30 epochs. Additionally, we utilise the Hi zero-centered gradient penalty term (Mescheder et al.. 2018), to regularise the discriminator gradient on real data with weight of 10. Other hyperparameter specitics such as the batch-size and whether we inject stochasticity via random noise samples or latent code samples, depend on the experiment and are disclosed in the respective sections below in the main text.
0.99
13

---

Under review as a conference paper at ICLR 2021
A.2 ID BIMODAL REGRESSION
In the following we derive the mean squared error form of the cross entropy and calibration losses used in experiment Section 5.1 under the assumption that the likelihood model for qe and ds is a univariate Gaussian distribution with a fixed unit variance. Using the setup from Eq. (2) it then follows that:
Loe (D, 0) = (pD(s.)) Dog (yy Fo(a), 1)]
(10)
Poo [ly Fe (x)))² + const.
(11)
Based the definition of the calibration loss in Eq. (6) we show that:
Leal (D, 0 6) = Epp
[L Go(Fo(x)), 1) || No Fo(x), 1))] [
(12)
1) [logN (v (F((()))), 1) - log Fa(e), 1il]
Epp
(13)
1 [ (v))((((())) 2 + 2v 2 (u--B(()))) + const
Epp
(14)
B (x(olc((P(o)).)) (Gu(Fb(x)) Go(o(1))) - yFo(c) [
Epp
const
(15)
Ga(FF(t)))= Go(Fo(()) ((((())))_)G(((()) -u(tb())))G) =o(C)) + const
Epp
(16)
1 Epp (Co(( - -k(+))²] + const.
(17)
2
A.3 LIDC
Architectures For the calibration network, Fo. we use the encoder-decnder architecture from SegNet (Badri- narayanan et al., 2017), with a softmax activation on the output layer.
Training During training, we draw random 180x180 image- annotation pairs, and we apply random horizontal flips and crop the data to produce 128x128 lesion-centered image tiles. All of our models were implemented in Py and trained for 80k iterations on single 32GB Tesla V100 GPU.
We train all our models for the LIDC experiments using 8- d-dimensional noise vectors in the cGAN experiments, latent codes in the -VEE-AN experiments. This value empirically found to perform well, sufficiently capturing the shape diversity the dataset. Additionally, in the refinement networks loss, we set the weighting parameter À the total generator loss, defined in Eq. (7) in the main text, so as to establish ratio of Lo : Leal = : 0.5, where LG is the adversarial component of the loss, and Leal is the calibration loss component. In practice, the actual weights used are 10 for L.. and 5 for Leal-
Evaluation Following Kohl et al. (2018), Kohl al. (2019), Huang al. (2018), and Baumgartner et al. (2019) we use the Generalised Energy Distance (GED) (Székely and Rizzo, 2013) metric, given as:
Dewn (pD, 4+) 2Bmaas---pa [d(s, v)] Ee [d(s s')] Ey.y'rp [d(y. v v')
(18)
where d(s, v) 1 IoU y). Intuitively, the first term of Eq. (18) quantifies the disparity between sampled predictions and the ground truth labels, the second term-tho diversity between the predictions, and the third term the diversity between the ground truth labels. It is important to note that the GED is cample-based metric, and therefore the quality of the score scales with the number of samples. We approximate the expectations with all 4 ground truth labels (y pD) and 16, 50 or 100 samples from the model
qa) for each input image x.
As Kohl al. (2019) have pointed out, even though the GED metric good indicator for how well the learnt predictive distribution fits a multimodal ground truth distribution, can reward high diversity in sampled predictions even if the individual samples do not show high fidelity to the ground truth distribution. As an alternative metric that is less sensitive to such degenerate cases, Kohl et al. (2019) propose to use the Hungarian- matched IoU (HM-loU), which finds the optimal 1:1 IoU matching hetween ground truth samples and sampled predictions. Following Kohl et al. (2019), we duplicate the set of ground truth labels so that the number hoth ground truth and predicted samples are 16, and we report the HM-loU as the average of the best matched pairs for each input image.
In the main text show the evaluated performance with both GED and HM-IoU metrics over the entire test. set and compute the loU on only the foreground of the sampled labels and predictions. In the case where both
14

---

Under review as a conference paper at ICLR 2021
the matched up label and prediction do not show lesion, the IoU is set 1, that correct prediction of the absence of lesion is rewarded.
Note that the methods of Baumgartner et al. (2019) and Hu et al. (2019), whose GED scores we report in Table 1, use test sets that differ from the original splits defined in Kohl et al. (2018), which are used in Kohl et al. (2018; 2019) and our work. Baumgartner et al. (2019) uses random 60:20:20 split for the training, testing and validation sets, and 100 samples compute the GED score, whereas Hu et al. (2019) use random 70:15:15 split and 50 samples compute the GED score. Due to lack of reproducibility, this the conventional way of benchmarking. Therefore, in Table we only report the scores for our models evaluated on the original splits. Nevertheless. we also trained and tested our cGAN+Lea model on the split methodology defined by Baumgartner et al. (2019) to enable fairer comparison. This improved our GED score from 0.243 + 0.004 0.228 + 0.009, while the HM-IoU, evaluated using 16 samples, remained similar 0.590 + 0.007 (in the original splits we achieved an HM-IoU score of 0.592 + 0.005). This shows that different random splits can significantly affect the final performance w. r. GED score, while HM-IoU appears to be a more robust metric.
do not consider
A.4 CITYSCAPES
Architectures For the calibration network Fs, we design a small neural network with 5 convolutional blocks, each comprised of a 3x3 convolutional layer. followed batchnorm layer and leaky ReLU activation. The network activated with a softmax function.
Training During training, we apply random horizontal flips, scaling and crops of size 128x128 on the image-label pairs. All of our models were implemented in PyTorch and trained for 120k training iterations on a single 16GB Tesla V100 GPU.
We train all our models for the modified Cityscapes experiments using 32-dimensional noise vectors. Similarly the LIDC experiments, this value was empirically found perform well, however, it can be further tuned. As commonly practiced, we use the ignore-masks provided by the Cityscapes dataset to filter out the cross entropy, calibration and adversarial losses during training on the unlabelled pixels. Similarly to our LIDC experiment, we use weight of 10 for La. and 5 for Lea in the refinement networks loss.
The GED metric for Cityscapes is implemented as and evaluated across the entire validation set. In this dataset we have full knowledge of the ground truth class distribution and therefore we compute the GED metric by using the probabilities each mode directly, as follows:
Evaluation
described in the appendix of Kohl et al. (2018)
DOED (pv, q²) 2e [d(s, y)w(y)] Ea, [d(s, s')] Ey.y'<pp [dly, sf (w))) (19)
where w(-) is function mapping the mode of a given label y its corresponding probability mass. The distance d(s, y) is computed using the average IoU the switchable classes only, as done Kohl (2018), In the cases where none of the switchable classes are present in both the ground truth label and the prediction paired up in d(s, y), the distance score is not considered in the expectation. We use 16 samples to compute the GED score.
For the calibration results presented in Fig. 5, Section 5.2.2 in the main text, compute the calibration network class- probabirobaiiiites using the raw predictions of Fe (x:). We obtain class masks by computing the overlap between the ground truth labels and the black-box predictions for each class. Using these masks we then compute the average class-wise probabilities. The probabilities for the refinement network Ge were computed as the average over 16 samples. Here the class masks are obtained by finding the pixels that are specified as the class of interest in the ground truth labels.
15

---

Under review as a conference paper at ICLR 2021
B ADDITIONAL EXPERIMENT RESULTS
To reinforce the results reported in Section 5 present supplementary results for the bimodal regression experiment and the LIDC and Cityscapes segmentation experiments.
B.1 ID BIMODAL REGRESSION
Fig. shows the data log-likelihoods the 9 data configurations varying mode bias {0.5, 0.6, 0.9) and mode noise E {0.01, 0.02, 0.03) trained with and without the calibration loss Leat. Each experiment is repeated times and the individual likelihood curves are plotted in Fig. 7b and Fig. 7d respectively. The results show that high bias is harder to learn, reflected slowed down convergence, however, the Laarreullrried model shows greater robustness weight initialisation. contrast the non-regularised GAN exhibits mode oscillation expressed as fluctuation of higher likelihood (one mode is covered) and lower one (between modes).
Mode blas
Mode
modia ikkquathoongg
Mode spread
Mode spread
(a)
(b)
Mode blas 80
Mode
12
Mode spread
Mode spread
(c)
(d)
Figure 7: Log- likelihook curves for runs each of the data configurations. (a) No calibration loss (A = 0), averaged. (b) No calibration loss, individual runs. (c) With calibration loss (A = 1), averaged. (d) With calibration loss, individual runs.
16

---

Under review as a conference paper at ICLR 2021
B.2 LIDC
B.2.1
QUALITATIVE ANALYSIS
To further examine our regulaaald cGAN model trained on the LIDC dataset, we illustrate representative qualitative results in Fig. and Fig. For every input image provided by different expert annotators, overlaying the input image, in the four columns, sampled predictions .... Viet in the last six columns. From left right, three columns with dark blue background in the center of the figures show the average ground truth predictions Øg. the output of the calibration network (a()) and the average of 16 sampled predictions from the refinement network Ger. Our results show that even though there is significant variability between the refinement network samples for diversity regularisation
show the ground truth labels V ... B/a
randomly
given input image, yhar is almost identical to the calibration target Fo (1), enforced by the calibration loss Leat-
(x)
(x. (x.2g))
(A. (A.
Agt
Fa(x)
Ad
Fert
Feet
0.75
Figure 8: Qualitative results LIDC samples for the Leat regullarge cGAN model.
17

---

Under review as a conference paper at ICLR 2021
(x,1))
(x.sg)
(A.
Pgt
Fa(x)
Fné
Vnn
Seat
0.00
0.50
0.75
1.00
Figure 9: Qualitative results on LIDC samples for the regularule eGAN model.
From the qualitative results in Fig. and Fig. 9, it capture well the average of ground distribution Ve. affecting the fidelity the predictive distribution of the refinement network G This further highlights the importance future work on improving the calibration of Fo, e. implementing the approaches of Guo et al. (2017); Kull et al. (2019).
that the calibration target F2 does not always
18

---

Under review as a conference paper at ICLR 2021
B.2.2 TUNING THE NUMBER OF REFINEMENT NETWORK SAMPLES
Table 3: Mean GED and HM-IoU scores on LIDC for the Learregularised cGAN with 1, 5, 10, 15 and 20 samples. The number of samples used to compute the GED score is denoted in the parentheses in the header of each column. The arrows and 4 denote higher or lower score better.
Method
GED J (16)
GED (50)
GED | (100)
HM-IoU (16)
eGAN+Leal Leal
0.644 0.033
0.643 + 0.033
0.643 0.033
0.494 + 0.013
cGAN+Ceal (5)
0.278 = 0.000
0.252 = 0.001
0.585 = 0.003
cGAN+Leal (10)
0.277 = 0.003
0.257 = 0.003
0.250 = 0.003
0.589 + 0.007
Cea (15)
0.271 = 0.002
0.250 = 0.001
0.245 = 0.003
0.593 + 0.002
CGaN+Leal (20)
0.264 = 0.002
0.248 = 0.004
0.243 = 0.004
0.592 + 0.005
To investigate the effect of the number of samples used to compute Leal on the learnt predictive distribution, we experimented on the Leau-reularrised cGAN model using 5, 10, 15 or 20 samples from the refinement network Ge during training. As a control experiment. we also train the same model using one sample. Our results, reported in Table 3. show that increasing the number of samples improves the quality of the predictive distribution. whereas using only one sample collapses it. This is expected because increasing the number of samples reduces the variance of the sample mean Go and refines the approximation go of the implicit predictive distribution realised by G2 e). Since in our implementation we reuse the samples from G. in the adversarial component LG of the total refinement network loss LG. the discriminator D. interacts with larger set of diverse fake samples during each training iteration, thus also improving the quality of La.
is important note that the benefit of increasing the sample size on the quality Leat highly depends on the intrinsic multimodality in the data. In theory, if the number of samples used matches or exceeds the number of ground truth modes for a given input, it is sufficient to induce a calibrated predictive distribution. However, we usually do not have priori access this information. Conversely, if the sample size is too small, the Leat loss may introduce hias in the predictive distribution. This could lead to mode coupling mode collapse, as exemplified in our control experiment with one sample.
In the LIDC dataset, even though we have access to four labels per input image, we argue that the dataset exhibits distributed multimodality, where a given pattern in the input space, e. g. in patch of pixels, can be associated to many different local labels throughout the dataset. As a result, input image may correspond to solutions than the four annotations provided. Therefore increasing the number of samples more than four shows further improvement in performance. This however can come at the cost decreased training speed which can be regulated by tuming the sample count parameter while considering the system requirements.
B.2.3 INDUCING MULTIMODALITY IN LATENT VARIABLE MODELS ON THE LIDC DATASET
To examine whether conditioning the source of stochasticity in our model on the input is beneficial, we adapt framework in order to learn distribution over latent code z using variational inference (Graves, 201 I). Following most of the existing work on stochastic semantic segmentation (Kohl ct al., 2018; 2019; Hu et al., 2019; Baumgartner et al., 2019), we maximise a modified lower bound on the conditional marginal log log-likelihood, through a variational distribution g(= x. y). This is realised by minimising the loss function in Higgins et al. (2017), given by:
CELBO v) En [logply =x) 33 KL(g(> 22, || p(2)) 2 log ply xx)
(20)
where controls the amount of regularisation from a prior distribution p(=) on the approximate posterior q(> | x, Both q(> x, y) and p(z) are commonly taken as factorised Gaussian distributions.
To this end, we compare our cGAN model to two baselines where the refinement network Go is given as eVAE-GAN (Larsen et al., 2015). In the first one, we train G. by complementing the adversarial loss La with Eq. (20), using 3 E 10.1, 1, 10) and a fixed standard normal prior. In the second, we introduce a calibration network and train Go using Eq. (7). This does not necessitate specifying a prior.
For a fair comparison, we use the same core models for all of our experiments, introducing only minor modifications to the refinement network to convert the deterministic encoder into probabilistic one with Gaussian output. This is achieved by splitting the output head of the encoder so as predict the mean and standard deviation of the encoded distribution (Kingma and Welling, 2013; Kendall and Gal, 2017). Instead of using random noise sampled from a standard Gaussian as our source of stochasticity, the decoder of the refinement network is now injected with latent codes sampled from the Gaussian distribution encoded for each input image. To train the model, we pretrain the Fg in isolation, and subsequently apply it in inference mode while training G.g- We use a batch size of 32, an 8-dimensional latent code, and use 20 samples to compute Ceat-
19

---

Under review as a conference paper at ICLR 2021
Table 4: GED and H-IoU scores on LIDC. The top section shows the Ceerregulariee baseline and the reg-regulrr cGAN; the bottom section shows baseline and reguulagerr cVAE-GANS. All fea-regularieed models are trained using 20 samples. The three central columns show the GED score computed with 16, 50 and 100 samples, respectively. The last column shows the HM-loU score, computed with samples. The arrows f and y indicate whether higher or lower score is better.
Method
GED 4. (16)
GED (50)
GED (100)
HM-IoU (16)
cGAN+Loe
0.639 = 0.002
0.477 0.004
cGAN+Ccal
0.264 + 0.002
0.248 + 0.004
0.243 + 0.004
0.592 + 0.005
oVAE-GAN (3=0.1)
0.577 = 0.095
0,484 + 0.006
eVAE-GAN (3=1)
0.596 = 0.078
0.474 + 0.005
eVAE-GAN (3=10)
0.609 = 0.061
0.482 + 0.010
-VAE-GAN+Call (3=0)
0.272 = 0.006
0.252 = 0.006 0.246 + 0.006
0.593 0.003
We show that the eVAE-GAN model trained with Lest instead of the traditional complexity loss (L(g(2 x, y) p(z)) from Eq. (20)
to learn a distribution over segmentation maps, and performs similarly to our cGAN+Lea model. This is important because it abrogates the need for specifying latent-space prior, which is often selected for computational convenience, rather than task relevance (Hafner et al.. 2018; Louizos al., 2019). On the other hand, cVAE GAN models trained using the KL KL-divergence complexity term showed limited diversity, even large values. The results, shown quantitatively in the bottom part Table 4, and qualitatively Fig. 10a, demonstrate
interaction between LG and Leal can sufficiently induce a multimodal predictive distribution in latent variable models, and indicate that for the purpose of stochastic semantic segmentation the use of probabilistic encoder is not strictly required.
Vig
Mig
yet
Fa(x)
Fee
Feef
Fref
Vid
Vief
y
y
O
0.0
0.5
1.0
1ight
Fa(x)
Fie
ya
0.5
(b)
Figure 10: LIDC validation samples for the (a) cVAE- -AAN and (b) cGAN+Log baseline model.
20

---

Under review as a conference paper at ICLR 2021
B.3 CITYSCAPES
tmmh
regitation
pers
Figure 11: 10 input images, the corresponding aleatoric maps from the calibration network and 16 samples from the refinement network. For visualisation purposes, the samples are split into 8 per row.
B.3.1 QUALITATIVE ANALYSIS
In this section provide additional qualitative results for the Lea-regularised cGAN model trained the modified Cityscapes dataset (Kohl al., 2018). In Fig. 11, show 16 randomly sampled predictions for representative input images z, their corresponding aleatoric uncertainty maps, obtained by computing entropy of the output of the calibration network. H(+a(z)). as done in Kendall and Gal (2017). The predicted samples are of high quality, evident by object coherence and crisp outlines, and high diversity, where all classes are well represented. Our model effectively learns the entropy of the ground truth distribution in the stochastic classes (sidewalk, person, car, vegetation and road), their distinct entropy levels captured different
21

---

Under review as a conference paper at ICLR 2021
shades of red in the entropy maps, corresponding the different flip probabilities (*/17, 7/17, 9/17, f/17 and 4/17 respectively). Additionally, it can be seen that edges or object boundaries are also highlighted in the alcatoric uncertainty maps, which reflects inconsistency during manual annotation, which often occurs on input pixels that are difficult to segment.
Fig. 12d shows the entropy of the predictive distribution of the refinement network Co. (IF(()))))) where Go (z)( is computed as the average of 16 samples from Gg. Our results demonstrate that (F similar to Hi( [=())), depicted in Fig. 12c, encouraged by the Leal regularisation. Notice that object boundaries are also highlighted in IH (F))))) indicating that our model captures shape ambiguity as well as class ambiguity. However, some uncertainty information from H())( is not present H (F) g. the entropy of the different stochastic classes are not always consistent across images, as evident from the different shades of red seen for the road class in Fig. 12d. We expect that increasing the number of samples from the refinement network will improve the alcatoric uncertainty estimates. Nevertheless, the sample-free estimate extracted from Fz( is cheaper to obtain and more reliable than the sample-based average from Gø. highlighting an important henefit of our cascaded approach.
Finally, we illustrate in Fig. 12e the high confidence of the predictions from the refinement network Go (FF (x)). reflected by their low entropy, ((((((FF(xz)) This is attributed to the adversarial component in the refinement loss functionn, which encourages the predictions annotations. Even though each prediction the refinement network is highly confident, the average the predictions GF(Fs Fe (x)) is calibrated, as shown in Fig. 12d. This is a clear illustration of the advantage of complementing the adversarial loss term CG with the calibration loss term Ces in the training objective for the refinement network.
one-hot representation, matching the ground truth
22

---

Under review as a conference paper at ICLR 2021
(a)
(b)
t 4
Entropy
Entrggy
(d)
Entepy
Figure 12: (a) Three input images overlaid with the corresponding labels; (b) Incoherent samples from the predictive distribution of the calibration network; (c) The aleatoric maps from the calibration network: (d) Aleatoric maps computed as the entropy of the average of 16 predictions of the refinement network; (e) The entropy of one sample of the refinement network output for each input image.
23


Reviews:
[{"review_id": "qB6WaDHpv6V", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Summary: \nThis work addresses the context of semantic segmentation where a single input image could be associated with multiple valid labels, as a result of natural ambiguities. Starting from a pretrained deterministic segmentation network F, this work proposes to use an additional conditional generative model G, named as *refinement network*, to generate multiple segmentation predictions; the model G is conditioned on the segmentation probabilistic output of F and the input image. G is trained with adversarial loss and the proposed *calibration loss*, essentially the KL-divergence between the probabilistic output of F and the sample average of G. At runtime, the unified pipeline of F and G can produce multiple segmentation predictions. On one toy example and two real benchmarks, the proposed method show improvements over addressed baselines, in terms of generalized energy distance (GED) and Hungarian-matched IoU (HM-IoU).\n\n** Strengths:\n- The idea of using conditional GANs to produce multiple predictions is interesting.\n- The proposed framework and learning scheme are simple. I think it's easy to reimplement and reproduce results.\n\n** Weakness:\n- Going through the paper I had trouble understanding how the refinement network G can guarantee to produce calibrated probabilities of segmentation modes. The calibration network F, to my understanding, is a deterministic segmentation model trained in a conventional fashion using only the cross-entropy loss. I believe numerous works proved that a model trained that way will end up with over-confident predictions, which are uncalibrated (actually shown in Figure 5). It actually seems misleading to name F with calibration.\n\n- Outputs of pretrained F is then used to regularize the training of the cGAN G via the KL-divergence \"calibration loss\" (which is more like a reconstruction loss to me). Can the authors explain how the refinement network, trained to match sample average with uncalibrated probabilistic targets, can successfully produce calibrated probabilistic outcomes? Also, I would love to see results with calibration metrics like NLL and ECE.\n\n- On the Cityscapes experiments, the segmentation network B is finetuned with or without class-flipping labels? It's quite confusing when sometimes F is a full segmentation network as in Sec 3, Sec 4 and Sec 5.2.1, sometimes F is an ad-hoc network like in 5.2.2. Also F's architecture is detailed in the beginning of 5.2 as SegNet, but only used in 5.2.1. \n\n- Can the authors please provide experimental evidence of how the cross-entropy loss and adversarial loss are not well aligned in the presence of noisy data?\n\n- Minor typos: \n\t+ In Table 2, shouldn't the GED of Kohl et al be 0.206?\n\t+ It may look obvious but should the notations like H,W,C,K be introduced? I thought C is the number of classes at first.\n\n** Preliminary evaluation: this work targets an interesting task of stochastic semantic segmentation. The architecture design and learning scheme seems reasonable to me. The major problem is the lack of evidence to support the claim on output calibration. In terms of writing, I find the paper hard to follow with lots of confusions. Due to those limitations, I give an initial rating of 5.\n\n-- Post-rebuttal -----------------------------------------------------------------------------------------------\n\nGiven the improvement of the last revision, I increase my rating to 6. The revised version has been very much improved, especially in the abstract and introduction Sections. Still I think it's important to additionally have one or two sentences to make very clear on the meaning of calibration, as to not confuse readers.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XtO7NsY2YdT", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work the authors propose a two-stage, adversarial training technique to calibrate a semantic segmentation model when faced with conflicting labels in the train set.  Their approach is to first train a segmentation model. Then, it is used to feed a GAN model which itself is trained against a discriminator to produce diverse segmentations that reflect the diversity in the train set.  The authors compare their approach to similar methods on a synthetic data set and two semantic segmentation data sets. \n\nPros:\n1) The technique is conceptually simple as training a segmentation model with cross entropy loss as well as training a GAN are both well-understood.\n2) The technique can really be used to calibrate any differentiable semantic segmentation model, regardless of who trained it or how.\n3) The experimental details are described in enough detail to likely be able to replicate most of the results\n4) Minus some organizational issues, the writing is good overall.\n\n\nCons:\n1) I found the motivation in the introduction to be slightly difficult to follow.  More specifically, the concepts ambiguity in data labels leading to a multi-modal data distribution and the need to model uncertainty need to be more tightly discussed.  As it is written now, the authors first argue for modeling a noisy empirical distribution that captures the ambiguities, which seems counter intuitive since there assumedly exists a single true segmentation of the image then later discuss uncertainty.  I think discussing uncertainty modeling, and specifically calibration, first would alleviate this issue.\n2) \"Isola et al. (2017) have demonstrated that it introduces only minor stochasticity in the output and returns inconsistent samples.\" (page 2) - In Isola et al. (2017)  dropout is used for image to image translation in a GAN.  I do not think this provides sufficient evidence that using dropout BNNs  such as in Kendal and Gal (2017) has these properties.\n3) The authors argue that combining the GAN loss and the pixel-wise cross entropy is not well-suited for noisy data because they are often at odds.  It's not clear to me that this is a problem in practice, as the cross entropy will spread probability mass across different conflicting labeled instances.  Assume a pixel in the data set appears twice with two different labels.  The sum of the cross entropy over these two examples is minimized by a model that puts probability 0.5 on each of the two classes.  This seems like the multi-modal behavior the authors argue for.  Perhaps there is something more subtle going on, but the lack of formal analysis makes it difficult to understand how much of an issue it is.  Further, the proposed method is adding a KL Divergence term to the generator loss that is at odds with the GAN loss, which was what the authors argue against.  In short, the argument against the most similar method and the proposed method is weak in my opinion.\n4) I think the name calibration network is a bit misleading. It seems the calibration network is the base segmentation model and the refinement network is calibrating the model.\n5) The paper would benefit from an algorithm sketch to explicitly show the two stages of training.\n6) In the experiments the authors switch to mean squared loss after arguing against cross-entropy.  It's not clear why this is done or why this is a proper baseline.\n7) Looking at figure 3, it would seem the refinement network does not generate outputs that closely match any of the ground truth annotations, but rather some combinations of them.  In practice, I would assume that someone would use the mean and standard deviation to understand the predictions and not samples, so this is less of an issue, but it highlights an issue with considering each pixel independently.\n8) While I think the GED metric make sense here, I think calibration (like expected calibration error) or uncertainty focused (like those proposed in (Mukhoti and Gal; 2019)) would be useful to tell a more complete story of the evaluation. \n9) Figure 5 is unclear.  The text seems to imply that this shows that their technique does not calibrate their model well.  To me this is a strong argument against their approach.  I am not sure the value of a diverse set of segmentations if the model cannot accurately convey uncertainty in predictions.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "RJXYo2XiIME", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-----------------------------------------------------------------------------------------------------------------------------------------------------------------\nPOST REBUTTAL\n-----------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nThe rebuttal has addressed most of my concerns and I am happy to increase the score.\n\n-----------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nThe main strengths of the work are -\n* The approach is relatively novel and addresses an important problem.\n* The paper clearly highlights issues with prior work e.g. generator loss is often complemented with pixelwise loss, however, these two objective functions are not well aligned in the presence of noisy data.\n* The paper includes experiments on toy data -- which highlight the contributions of the paper.\n* The proposed approach outperforms prior work -- Hu et al. (2019), Baumgartner et al. (2019) on the LIDC dataset.\n* The proposed approach outperforms Kohl et al. (2018) on CityScapes in case of the GED metric.\n\nThe main weaknesses are,\n* No comparison with closely related approach [1] -- which also proposes a Bayesian approach to capture a calibrated multimodal predictive distributions. In fact, the experiments on 1d bimodal toy data are similar to the experiment in Figure 1 in [1]. A detailed comparison is necessary.\n* Experiments on 1D bimodal data -- the baseline $G_\\phi(F_\\theta(x), \\epsilon)$ is a typical conditional GAN? This seems to be a weak baseline, as recent works [3] address the model collapse issues of conditional GANs. Strong baselines e.g. [1,3], conditional VAEs etc should be considered. \n* Experiments on CityScapes -- the paper does not show results using metrics used by prior work [2] -- in particular Precision-Recall curves and calibration plots which shows the frequency of correctly predicted labels for each bin of predicted probability values. These metrics are also used in [1]. These metrics would better illustrate the calibration of predictions of the proposed approach.\n* Several unclarities -- What is the contribution of the two components - Calibration network and Refinement network on the calibration of the final output? Does the Refinement network aid in improving calibration? \n\n[1] Bayesian Prediction of Future Street Scenes using Synthetic Likelihoods, ICLR 2019.\n[2] What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?, NeurIPS 2017.\n[3] Diversity-Sensitive Conditional Generative Adversarial Networks, ICLR 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7XjARDysAwq", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper presents an approach to stochastic semantic segmentation. The proposed strategy involves a simple extension of neural network architectures for semantic segmentation. In particular, the probabilistic output of a semantic segmentation network is fed into a GAN, which generates a final segmentation. In addition to the classic loss, the GAN is trained such that the average of its prediction matches the input distribution, hence calibrating the distribution. The experiments on a toy dataset and two segmentation datasets demonstrate the superiority of the proposed approach compared to using standard segmentation loss and a few related works.\n\nPros:\n- The writing is good, and the method is explained intuitively\n- The proposed approach is simple yet seems effective \n- The proposed approach is modular and can be applied to pre-trained network architectures \n\nCons:\n- My major concern is that the experimental evaluation does not sufficiently demonstrate that the proposed module is indeed necessary. In the preliminaries section, the authors discuss several simpler alternatives that are not tested in the experiments. E.g., that direct sampling from q_theta yields incoherent segmentation maps or that combining the generator loss with the pixel-wise loss in Eq.2 is not sufficient. I think these settings would serve as good additional baselines in the experimental evaluation.\n- A minor additional criticism is that the authors do not compare to other relevant work such as Kendall and Gal 2017. Moreover, on the Cityscapes dataset, they only compare to one single baseline.\n\n--------------------------------------------------------------------------------------------------------------------------\nPost rebuttal\n\nI thank the authors for providing detailed answers to my concerns. Considering the concerns of the other reviewers and the authors' answers and additional experiments, I think that the paper provides a sufficient contribution to an important research topic. Therefore, I retain my initial rating.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses stochastic semantic segmentation with a two-step approach: a standard segmentation network learned with cross-entropy serves as a guide to calibrate a second refinement network to generate diverse predictions while their expectation matches the calibration model. 

The reviewers acknowledge the paper merits', e.g. the decoupling between the segmentation and generation networks. However, they also highlight serious concerns on the the clarity of the presentation, and the need for a consolidated evaluation. 

The AC carefully reads the paper and the discussion among authors and reviewers. Despite improvements in paper presentation, the AC still considers that the paper would benefit from clarifications, e.g. the fact that the paper does not address calibration, and that stronger baselines as those mentioned by reviewers are needed for fully validating the approach.  
Therefore, the AC recommends rejection. 
Decision: Reject