Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this work the authors propose a two-stage, adversarial training technique to calibrate a semantic segmentation model when faced with conflicting labels in the train set.  Their approach is to first train a segmentation model. Then, it is used to feed a GAN model which itself is trained against a discriminator to produce diverse segmentations that reflect the diversity in the train set.  The authors compare their approach to similar methods on a synthetic data set and two semantic segmentation data sets. 

Pros:
1) The technique is conceptually simple as training a segmentation model with cross entropy loss as well as training a GAN are both well-understood.
2) The technique can really be used to calibrate any differentiable semantic segmentation model, regardless of who trained it or how.
3) The experimental details are described in enough detail to likely be able to replicate most of the results
4) Minus some organizational issues, the writing is good overall.


Cons:
1) I found the motivation in the introduction to be slightly difficult to follow.  More specifically, the concepts ambiguity in data labels leading to a multi-modal data distribution and the need to model uncertainty need to be more tightly discussed.  As it is written now, the authors first argue for modeling a noisy empirical distribution that captures the ambiguities, which seems counter intuitive since there assumedly exists a single true segmentation of the image then later discuss uncertainty.  I think discussing uncertainty modeling, and specifically calibration, first would alleviate this issue.
2) "Isola et al. (2017) have demonstrated that it introduces only minor stochasticity in the output and returns inconsistent samples." (page 2) - In Isola et al. (2017)  dropout is used for image to image translation in a GAN.  I do not think this provides sufficient evidence that using dropout BNNs  such as in Kendal and Gal (2017) has these properties.
3) The authors argue that combining the GAN loss and the pixel-wise cross entropy is not well-suited for noisy data because they are often at odds.  It's not clear to me that this is a problem in practice, as the cross entropy will spread probability mass across different conflicting labeled instances.  Assume a pixel in the data set appears twice with two different labels.  The sum of the cross entropy over these two examples is minimized by a model that puts probability 0.5 on each of the two classes.  This seems like the multi-modal behavior the authors argue for.  Perhaps there is something more subtle going on, but the lack of formal analysis makes it difficult to understand how much of an issue it is.  Further, the proposed method is adding a KL Divergence term to the generator loss that is at odds with the GAN loss, which was what the authors argue against.  In short, the argument against the most similar method and the proposed method is weak in my opinion.
4) I think the name calibration network is a bit misleading. It seems the calibration network is the base segmentation model and the refinement network is calibrating the model.
5) The paper would benefit from an algorithm sketch to explicitly show the two stages of training.
6) In the experiments the authors switch to mean squared loss after arguing against cross-entropy.  It's not clear why this is done or why this is a proper baseline.
7) Looking at figure 3, it would seem the refinement network does not generate outputs that closely match any of the ground truth annotations, but rather some combinations of them.  In practice, I would assume that someone would use the mean and standard deviation to understand the predictions and not samples, so this is less of an issue, but it highlights an issue with considering each pixel independently.
8) While I think the GED metric make sense here, I think calibration (like expected calibration error) or uncertainty focused (like those proposed in (Mukhoti and Gal; 2019)) would be useful to tell a more complete story of the evaluation. 
9) Figure 5 is unclear.  The text seems to imply that this shows that their technique does not calibrate their model well.  To me this is a strong argument against their approach.  I am not sure the value of a diverse set of segmentations if the model cannot accurately convey uncertainty in predictions.

review_title: The authors propose a two-stage calibration technique for semantic segmentation.  The approach is conceptually simple and can be applied to a number of base segmentation models.  However, there is a lack of strong justification for their approach, seemingly poor pixel-wise calibration results, and the main novel contribution is a simple additive KL Divergence term in the generator's objective. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: -----------------------------------------------------------------------------------------------------------------------------------------------------------------
POST REBUTTAL
-----------------------------------------------------------------------------------------------------------------------------------------------------------------

The rebuttal has addressed most of my concerns and I am happy to increase the score.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------

The main strengths of the work are -
* The approach is relatively novel and addresses an important problem.
* The paper clearly highlights issues with prior work e.g. generator loss is often complemented with pixelwise loss, however, these two objective functions are not well aligned in the presence of noisy data.
* The paper includes experiments on toy data -- which highlight the contributions of the paper.
* The proposed approach outperforms prior work -- Hu et al. (2019), Baumgartner et al. (2019) on the LIDC dataset.
* The proposed approach outperforms Kohl et al. (2018) on CityScapes in case of the GED metric.

The main weaknesses are,
* No comparison with closely related approach [1] -- which also proposes a Bayesian approach to capture a calibrated multimodal predictive distributions. In fact, the experiments on 1d bimodal toy data are similar to the experiment in Figure 1 in [1]. A detailed comparison is necessary.
* Experiments on 1D bimodal data -- the baseline $G_\phi(F_\theta(x), \epsilon)$ is a typical conditional GAN? This seems to be a weak baseline, as recent works [3] address the model collapse issues of conditional GANs. Strong baselines e.g. [1,3], conditional VAEs etc should be considered. 
* Experiments on CityScapes -- the paper does not show results using metrics used by prior work [2] -- in particular Precision-Recall curves and calibration plots which shows the frequency of correctly predicted labels for each bin of predicted probability values. These metrics are also used in [1]. These metrics would better illustrate the calibration of predictions of the proposed approach.
* Several unclarities -- What is the contribution of the two components - Calibration network and Refinement network on the calibration of the final output? Does the Refinement network aid in improving calibration? 

[1] Bayesian Prediction of Future Street Scenes using Synthetic Likelihoods, ICLR 2019.
[2] What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?, NeurIPS 2017.
[3] Diversity-Sensitive Conditional Generative Adversarial Networks, ICLR 2019.
review_title: Interesting method but with missing comparisons and metrics

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper presents an approach to stochastic semantic segmentation. The proposed strategy involves a simple extension of neural network architectures for semantic segmentation. In particular, the probabilistic output of a semantic segmentation network is fed into a GAN, which generates a final segmentation. In addition to the classic loss, the GAN is trained such that the average of its prediction matches the input distribution, hence calibrating the distribution. The experiments on a toy dataset and two segmentation datasets demonstrate the superiority of the proposed approach compared to using standard segmentation loss and a few related works.

Pros:
- The writing is good, and the method is explained intuitively
- The proposed approach is simple yet seems effective 
- The proposed approach is modular and can be applied to pre-trained network architectures 

Cons:
- My major concern is that the experimental evaluation does not sufficiently demonstrate that the proposed module is indeed necessary. In the preliminaries section, the authors discuss several simpler alternatives that are not tested in the experiments. E.g., that direct sampling from q_theta yields incoherent segmentation maps or that combining the generator loss with the pixel-wise loss in Eq.2 is not sufficient. I think these settings would serve as good additional baselines in the experimental evaluation.
- A minor additional criticism is that the authors do not compare to other relevant work such as Kendall and Gal 2017. Moreover, on the Cityscapes dataset, they only compare to one single baseline.

--------------------------------------------------------------------------------------------------------------------------
Post rebuttal

I thank the authors for providing detailed answers to my concerns. Considering the concerns of the other reviewers and the authors' answers and additional experiments, I think that the paper provides a sufficient contribution to an important research topic. Therefore, I retain my initial rating.

review_title: Simple yet effective strategy for stochastic segmentation with limitations in the experimental evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: ** Summary: 
This work addresses the context of semantic segmentation where a single input image could be associated with multiple valid labels, as a result of natural ambiguities. Starting from a pretrained deterministic segmentation network F, this work proposes to use an additional conditional generative model G, named as *refinement network*, to generate multiple segmentation predictions; the model G is conditioned on the segmentation probabilistic output of F and the input image. G is trained with adversarial loss and the proposed *calibration loss*, essentially the KL-divergence between the probabilistic output of F and the sample average of G. At runtime, the unified pipeline of F and G can produce multiple segmentation predictions. On one toy example and two real benchmarks, the proposed method show improvements over addressed baselines, in terms of generalized energy distance (GED) and Hungarian-matched IoU (HM-IoU).

** Strengths:
- The idea of using conditional GANs to produce multiple predictions is interesting.
- The proposed framework and learning scheme are simple. I think it's easy to reimplement and reproduce results.

** Weakness:
- Going through the paper I had trouble understanding how the refinement network G can guarantee to produce calibrated probabilities of segmentation modes. The calibration network F, to my understanding, is a deterministic segmentation model trained in a conventional fashion using only the cross-entropy loss. I believe numerous works proved that a model trained that way will end up with over-confident predictions, which are uncalibrated (actually shown in Figure 5). It actually seems misleading to name F with calibration.

- Outputs of pretrained F is then used to regularize the training of the cGAN G via the KL-divergence "calibration loss" (which is more like a reconstruction loss to me). Can the authors explain how the refinement network, trained to match sample average with uncalibrated probabilistic targets, can successfully produce calibrated probabilistic outcomes? Also, I would love to see results with calibration metrics like NLL and ECE.

- On the Cityscapes experiments, the segmentation network B is finetuned with or without class-flipping labels? It's quite confusing when sometimes F is a full segmentation network as in Sec 3, Sec 4 and Sec 5.2.1, sometimes F is an ad-hoc network like in 5.2.2. Also F's architecture is detailed in the beginning of 5.2 as SegNet, but only used in 5.2.1. 

- Can the authors please provide experimental evidence of how the cross-entropy loss and adversarial loss are not well aligned in the presence of noisy data?

- Minor typos: 
	+ In Table 2, shouldn't the GED of Kohl et al be 0.206?
	+ It may look obvious but should the notations like H,W,C,K be introduced? I thought C is the number of classes at first.

** Preliminary evaluation: this work targets an interesting task of stochastic semantic segmentation. The architecture design and learning scheme seems reasonable to me. The major problem is the lack of evidence to support the claim on output calibration. In terms of writing, I find the paper hard to follow with lots of confusions. Due to those limitations, I give an initial rating of 5.

-- Post-rebuttal -----------------------------------------------------------------------------------------------

Given the improvement of the last revision, I increase my rating to 6. The revised version has been very much improved, especially in the abstract and introduction Sections. Still I think it's important to additionally have one or two sentences to make very clear on the meaning of calibration, as to not confuse readers.
review_title: Interesting idea, but paper and experiments need revision 

Meta-review: This paper addresses stochastic semantic segmentation with a two-step approach: a standard segmentation network learned with cross-entropy serves as a guide to calibrate a second refinement network to generate diverse predictions while their expectation matches the calibration model. 

The reviewers acknowledge the paper merits', e.g. the decoupling between the segmentation and generation networks. However, they also highlight serious concerns on the the clarity of the presentation, and the need for a consolidated evaluation. 

The AC carefully reads the paper and the discussion among authors and reviewers. Despite improvements in paper presentation, the AC still considers that the paper would benefit from clarifications, e.g. the fact that the paper does not address calibration, and that stronger baselines as those mentioned by reviewers are needed for fully validating the approach.  
Therefore, the AC recommends rejection. 
Decision: Reject