Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies semantic segmentation. The authors developed a Gaussian Mixture-based Generative Semantic Segmentation Model. The experimental results on serval datasets demonstrate the effectiveness of the proposed method.
strengths_and_weaknesses: [Strengths]
+ Compared to the baselines, the proposed methods could bring constant improvements 
+ The abundant experiments are introduced to prove the effectiveness of the proposed method.

[Weaknesses]
- some important references are missing.
[a] Top-down Learning for Structured Labeling with Convolutional Pseudoprior, ECCV 2016.
[b] Exploring Cross-Image Pixel Contrast for Semantic Segmentation, ICCV 2021.

- Both [a] and [b] introduce data distribution p(pixel feature|class), they all need to be included in the discussion. [b] also learn the class-related feature and bring constant improvements. It could be better to add it into comparison.

questions: What's the effect of the proposed method on the model inference speed?
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a novel paradigm for semantic segmentation. Specifically, the authors propose to frame the segmentation task as a generative model by modeling the probability of p(x|c) as a mixture of Gaussians. This is rather different from the popular discriminative model that adopts Softmax.   
strengths_and_weaknesses: + interesting idea
+ novel paradigm, as compared to a large number of semantic segmentation models. 

- the assumption of p(c) = 1/C might not be very reasonable. Normally, the prior distribution of different classes is not a uniform distribution. 
 
questions: - what about using a different p(c) ? 
limitations: - the assumption of uniform distribution of p(c) is not very reasonable.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes GMMSeg, a generative model based on GMM for image segmentation that models the joint distribution of pixel features and classes. For each class, GMMSeg builds GMMs via EM so as to capture class-conditional densities. A deep network is trained end-to-end in a discriminative manner to extract features for the GMMs. This endows GMMSeg with the strengths of both generative and discriminative models. With a variety of segmentation architectures and backbones, GMMSeg outperforms the discriminative counterparts on three closed-set datasets, ADE20K, Cityscapes and COCO. Without any modification, GMMSeg even performs well on open-world datasets. 
strengths_and_weaknesses: Strength:
1. It is reasonable to use generative classification models with discriminative feature learning to combine the strengths of both generative and discriminative models. 
2. The properties of GMMs make GMMSeg well adapt to multimodal data densities and allows GMMSeg to naturally reject abnormal inputs.
3. GMMSeg is fully compatible with different network architectures. 
4. Extensive experiments are conducted on multiple benchmark datasets. It shows that the proposed method improves the performances of multiple baselines. 

Weakness:
1. The proposed generative framework is complex. The motivation of using GMMs on top of a deep network is not very clear. Why not directly model the joint distribution of image and label maps using deep networks?
2. The meaning of the components of GMMs in the proposed framework is not very clear. It looks like all classes have the same number of components and every component is expected to have the same amount of pixels. However, the modality of different classes and the pixels of different components are not always the same. 
3. It would be more convincing if there are some discussion on what each components represent and a visualization of the learned components. 
4. Compared with the baseline, GMMSeg brings more overheads at both inference and training time. There should be a comparison in terms of training/inference speed. it will be more convincing if the proposed method is compared with baseline+CRF post processing, as both GMMSeg and CRF improve performance at the cost of computation overhead. 
5. It is oversimplified to model segmentation as pixel classification using GMMs as it does not handle the correlations among pixels in an image. It might be more reasonable to use MRF, which has been explored in previous work, e.g. [a] 
[a] Liu, Ziwei, et al. "Deep learning markov random field for semantic segmentation." IEEE transactions on pattern analysis and machine intelligence 40.8 (2017): 1814-1828.


questions: Please refer to the Strengths And Weaknesses section
limitations: The authors addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes using a Gaussian Mixture Model (GMM) at the end of semantic segmentation models. The EM component maps the representations produced by a fully-convolutional neural network to the class labels and their probabilities. The GMM is learned simultaneously with the neural network; its parameters are optimized with Sinkhorn EM to fit the features from a running window over multiple training iterations. The authors show that this performs well both on classical semantic segmentation benchmarks and on "open world" tasks where anomalous/out-of-distribution objects are introduced in the test set.
strengths_and_weaknesses: ## Strengths

### i) Produces a generative, probabilistic model of the CNNs representations and the output classes.

A key part of the segmentation model, the final mapping from neural representations to class probabilities and labels, is done with an interpretable and well-calibrated model. A lot of value comes from the model producing a probability distribution, for interpretability, reliability, calibration, etc. Some reasonable estimation of uncertainty is important when using neural networks within a larger system (e.g. in a commercial product) where one might need to handle uncertain predictions differently, as opposed to the neural network being used in isolation (e.g. to maximize scores on a benchmark). This work contributes significantly to literature focused on achieving good uncertainty estimates in neural-net-based-models [a,b,c], and this is only one aspect of the contributions.

Using a GMM in this way produces a well-calibrated uncertainty estimate. The authors allude to this with the 3rd limitation of softmax described in the introduction. Reliability diagrams are given in the supplement (Fig S1).

### ii) Good choice of techniques for jointly training a generative model with a CNNs, well-validated by ablation studies.

Adding new types of layers, including ones inspired by non-deep-learning models, is a common subject of a paper. This work does a more principled treatment of this than usual. First, the GMM model is learned with a choice of algorithm (Sinkhorn EM) that will produce consistent results in few iterations, which seems especially important when this model fit is repeated throughout the overall SGD training. Use of a memory bank is also a good improvement to this.

The value of Sinkhorn EM is validated by ablation study. The impact of the memory bank is also studied in the supplement.

### iii) Good performance on anomaly detection.

The scores reported in the submission would not put it at the top of Fishyscapes leaderboard (https://fishyscapes.com/results). For example, as of writing the best AP score on the Fishyscapes lost & found is 63.43, vs 43.47 reported by the submission in Tables 2 & 3. However, this seems quote good for a method that isn't directly trained for anomaly detections. like the top published results in Fishyscapes. For instance, the current Fishyscapes leader [d] introduces synthetic anomalies when training a network. Instead, in the submission good anomaly detection can be extracted from a GMMSeg model trained on ordinary segmentation tasks.

This seem like a good demonstration of interpretability. As much as "interpretability" is an ambiguous term, the ability to infer new information from the model to use in other contexts seems like a useful example.

## Weaknesses

### iv) Less clear that it's an improvement over using CRF-based models in the same place, or why.

The authors contrast their against using softmax directly on the CNN features. They provide good reasons why GMMSeg would better model this problem. It seems like CRFs share a lot of the same advantages, and the contrast between GMMSeg and those methods is less obvious.

The original V1 of DeepLab [2], and several follow-up works [e,61] use CRFs to map the NN representation to class probabilities. These are cited in the related works, but the probabilistic modelling aspect of these works is not really discussed. Experimental comparisons against DeepLab are done with V3+ [47], a more recent version that dropped the CRF component.

GMMSeg also lacks the "context" priors that CRFs give, due to the pairwise (or higher-order) potentials included in those models.

### v) Not a fully combined optimization of EM and feature extractor.

In Section 3.2, the authors clarify:

> the extractor’s parameters θ are only updated by the gradient back-propagated from the discriminative loss, while the GMM’s parameters are only optimized by EM

I'd interpret this to mean that the feature extractor, which remains an important part of the model in GMMSeg, is still trained in a discriminative way that doesn't differ from previous methods. And the NN features are not optimized for input to the GMM component.

Given that this is the case, it's unclear why the EM needs to be learned at the same time as the feature extractor. Would there be any change to the training of the feature extraction NN if the GMM was simply ignored? And if so, could the GMM be learned *after* the NN is fully trained, without any change in results?

The authors make good design choices for an end-to-end joint optimization (iii), but stop short of doing so. However, empirical results do seem to show that the GMM still describes a better mapping from the NN features to the class labels than the original softmax that the NN was trained with.

### vi) Only considers segmentation.

All of the authors' justifications for using a GMM to map representations to class labels seems like they'd apply equally well to classification, or any deep network trained with softmax. I wonder why the paper focuses on image segmentation exclusively, and it does make it a less widely-impactful work.

### vii) Unclear presentation of feature extraction component.

Adjacent pixels are usually important in segmentation, the feature in the classification problem is not really the $x_n \in R^3$ as described in Section 3.1. The same is true for the formulation of the "dense feature extractor." I'd emphasize that the feature extractor really maps $R^{H \times W \times 3}$ to $R^{H \times W \times D}$.

[a] Oh et. al. "Modeling uncertainty with hedged instance embedding" ICLR 2019. \
[b] Caesar et. al. "Joint Calibration for Semantic Segmentation" BMVC 2015. \
[c] Nado et. al. "Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning" \
[d] Grcić et. al. "Dense anomaly detection by robust learning on synthetic negative data" \
[e] Vemulapalli et. al. "Gaussian Conditional Random Field Network for Semantic Segmentation" CVPR 2016.
questions: - After training, and in inference are the EM parameters simply those from the last training iteration? This'll still be based on a smaller memory bank, how far off from "ideal" EM parameters could these be? The supplement says that a large enough memory bank represents the overall distribution, is this assertions based only on the fact that the performance has saturated (up to 3 digits) at 32K?

### Minor comments/typos:

  * 68: Forth -> Fourth
  * 86: Suggest "modify [the] FCN architecture"
  * 88: "latter" or "later"?
  * 94: "modeling [the] underlying data distribution"
  * 181: Is "hard" the right adjective here? Maybe the grammar would be better with "...why it is hard for existing segmentation models to..."

limitations: The limitations seem to be discussed reasonably well. The submission focuses on a problem that is well-described by benchmarks, and the it demonstrates that the method is likely to perform better than competitors on out-of-distribution or anomalous examples.
review_title: null

Meta-review: metareview: This paper proposes to learn generative model (mixture of Gaussian) on the discriminative features. The proposed method achieves strong performance on semantic segmentation and it is capable of anomaly detection. 

The paper was reviewed by 4 reviewers. 

Reviewer o8w9 (rating: 5) pointed out 2 missing references and asked about speed. The authors clarified the difference between their work and the two references, and showed that the inference speed is negligible. 

Reviewer 5AWr (rating: 6) asked about assumption of uniform prior on classes. The authors explained that this is a common assumption, and added results on learned non-uniform class prior. 

Reviewer TvcJ (rating: 5) asked about the motivation for the Gaussian mixture model, the meaning of the mixture components, computational overhead, comparison with MRF prior. The authors addressed the questions in detail and added new results. The reviewer read the authors' rebuttal and raised the rating to the current level 5. 

Reviewer 1Te2 (rating: 7) wrote a very detailed review and was mostly satisfied with the authors' rebuttal, although this reviewer was still not entirely sure about joint training. 

---------------

Overall, for the two reviewers with ratings 5, their concerns did not suggest serious flaws of the paper, and the authors addressed their concerns satisfactorily. 

I thus lean toward accepting this paper. 

confidence: Less certain
Decision: Accept