GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models
Authors: Chen Liang
Wenguan Wang
Jiaxu Miao
Yi Yang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Semantic Segmentation
Generative Classifier
Anomaly Segmentation

Abstract: Prevalent semantic segmentation solutions are, in essence, a dense discriminative classifier of p(class|pixel feature). Though straightforward, this de facto paradigm neglects the underlying data distribution p(pixel feature|class), and struggles to identify out-of-distribution data. Going beyond this, we propose GMMSeg, a new family of segmentation models that rely on a dense generative classifier for the joint distribution p(pixel feature,class). For each class, GMMSeg builds Gaussian Mixture Models (GMMs) via Expectation-Maximization (EM), so as to capture class-conditional densities. Meanwhile, the deep dense representation is end-to-end trained in a discriminative manner, i.e., maximizing p(class|pixel feature). This endows GMMSeg with the strengths of both generative and discriminative models. With a variety of segmentation architectures and backbones, GMMSeg outperforms the discriminative counterparts on three closed-set datasets. More impressively, without any modification, GMMSeg even performs well on open-world datasets. We believe this work brings fundamental insights into the related fields.

GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models
Chen L.angl.3.t,
1.3+t, Wenguan Wang"* Jiaxu Miao Yi Yang'
CCAAL Zhejiang University
ReLER, AAlI, University of Technology Sydney BBaidu Research
https://githui com/ wttlluu...oommeeooonoooooao hwtp///eoonnnnnnneeee
Abstract
Prevalent semantic segmentation solutions are, in essence, dense discriminative classifier of p( class feature) Though straightforward, this de facto paradigm neglects the underlying data distribution pl pixel class), and struggles to identify out-of cut-of-distribution data. Going beyond this, propose GMMSeg, new family of segmentation models that rely on a dense generative classifier for the joint distribution p( pizel class). For each class, GMMSeg builds Gaussian Mixture Models (GMMs) via Exxpectation-Mxxiiizzae class-conditional densities. Meanwhile, the deep dense representation is end-to-end trained in discriminative manner, ie., maximizing pl class pixel feature). This endows GMMSeg with the strengths of both generative and discriminative models. With variety of segmentation architectures and backbones, GMMSeg outperforms the discriminative counterparts on three closed-set datasets. More impressively, without any modification, GMMSeg even performs well on open- world datasets. We believe this work brings fundamental insights into the related fields.
(EM), so as to capture
open-world
Introduction
Semantic segmentation aims explain visual semantics at the pixel level. It typically considered as a problem of pixel-wise classification, Le., assigning a class label cE (1,...,CC C} to each pixel data a. Under this regime, deep-neural solutions are naturally built as a combination two parts (Fig. I(a)): an encoder-decoder, dense feature extractor that maps a to high-dimensional feature representation æ, and dense classifier that conducts C-way classification given input pixel feature æ. Starting from the first end-to-end segmentation solution fully convolutional networks (FCN) [1], researchers leave the classifier as parametric softmax, and fully devote to improving the dense feature extractor for learning better representation. As result, a huge amount of FCN-based solutions [2-5] emerged and their of-te-of-thh-aeee was further pushed forward by recent Transformer [6]-style algorithms [7-10].
From a probabilistic perspective, the softmax classifier, supervised by the cross-entropy loss together with the feature extractor, directly models the class probability given an input, i.e., posterior p(c|æz). This is known as a discriminative classifier, as the conditional probability distribution discriminates directly between the different values of [[ As discriminative classifiers directly find the classifica- tion rule with the smallest етror rate, they often give excellent performance in downstream tasks, and hence become the de facto paradigm in segmentation. Yet. due the discriminative nature, softmax- based segmentation models suffer from several limitations: First, they only learn the decision boundary between classes, without modeling the underlying data distribution [11]. Second, as only one weight vector is learned per class, they assume unimodality for each class [12, 13], bearing no within-class variation. Third, they learn prediction space where the model accuracy deteriorates rapidly away
Equaa contributions.
I Work partly done during an internship at Baidu Research.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

pixel embedding space
feature
dense extmeter
extractor
tmiining
training
pixel
pixel
samplex
samples
oullier from wnseen class
decision boundaries learned by diveriminative suftmau
(a) (b)
data densitics modeledd by generative
Figure I: (a) Existing softmax based discriminative regime only learns decision boundaries on the pixel embedding space. (b) Our GMMSeg models pixel feature densities via generative GMMs.
from the decision boundaries [14] and thus yield poorly calibrated predictions [15], struggling to recognize
data [16]. The first two limitations may hinder the expressive power of segmentation models, and the last challenges the adoption of segmentation models in decision- critical tasks (e.g., autonomous driving) and motivates the development of anomaly segmentation methods [17-19] (which, however, rely on pre-trained discriminative segmentation models).
As an alternative of discriminative classifiers, generative classifiers first find the joint probability p(æ,, c), and use p(z, c) to evaluate the class-conditionss densities p(z[c). Then classification is con- ducted using Bayes rule. Numerous theoretical and empirical comparisons [20,21] between these two approaches have been initiated even before the deep learning revolution. They reach the agreement that generative classifiers have potential to overcome shortcomings of their discriminative counterparts, as they able to model the input data itself. This stimulates the recent investigation of generative (and hybrid [22,23] classifiers in trustworthy Al [24-27] and semi superiupe learning [22, 23], while the discriminative classifiers are still dominant in most downstream tasks.
In light of this background, we propose a GMM based segmentation framework GMMSeg - that addresses the limitations of current discriminative solutions from generative perspective (Fig. (b)). Our work not only represents novel effort to advocate generative classifiers for end-to-end segmen- tation, but also evidences the merits of generative approaches in a challenging, dense classification task setting. particular, we adopt a separate mixture of Gaussians for modeling the data distribution of each class in the feature space, i.e., class-conditional feature densities p cc). During training, GMM classifier is online optimized by a momentum version of (Sinkhorn) EM [28] on large-scale, so to ensure its generative nature and synchronization with the evolving feature space. Mcan- while, the feature extractor is end-to-end trained with the discriminative (cross-entropy) loss, i.e., maximizing the conditional likelihood p(clæz) derived with the generative GMM, so as enable expressive representation learning. In this way, GMMSeg smartly learns generative classification with end-to-end discriminative representation in a compact and collaborative manner, exploiting the benefit of both generative and discriminative approaches. This also greatly distinguishes GMMSeg from most existing GMM based neural classifiers, which are either discriminatively trained [12, 29-31 ]] or trivially estimate a GMM in the feature space of a pre-trained discriminative classifier [19,32,33], 33].
GMMSeg has several appcaling facets: First, with the hybrid training strategy online EM based classifier optimization and end-to-end discriminative representation learning, GMMSeg can precisely approximate the data distribution over robust feature space. Second, the mixture components make GMMSeg a structured model that well adapts to multimodal data densities. Third, the distribution- preserving property allows GMMSeg to naturally reject abnormal inputs, without neither architectural change (like [34-37]) nor re- re-training (like (38-40]) nor post- calibbaaaat (like [17, 18, 8,41-46]). Fourth, GMMSeg is a principled framework, fully compatible with modern segmentation network architectures.
For thorough examination, $4.1, we approach GMMSeg on several representative segmentation architectures (i.e., Laeps.abbs [47], OCRNet [48], UperNet [49], SegFomer (7]). with diverse back- bones (i.e., ResNet [50], HRNet[51],, Swin [52], [7] Experimental results demonstrate GMMScg outperforms the softmax- based discriminative counterparts, e.g., 0.6% 1.5%. 0.5% % 0.8%, and 0.7% 1.7% 71% mIoU gains over 2DE [53], Cityscapes [54], and [Suff respectively. Furthermore, in $4.2, we validate our approach on anomaly segmentation. Without any modification, our Cityscapes-traine GMMSeg model is directly tested on Fishyscapes Lost&Found [56] and Road Anomaly [36] datasets, and outperforms all haid-tailored discriminative competitors.
To our best knowledge, GMMSeg is the first semantic segmentation method that reports promising results on both closed-set and open-world scenarios by using a single model instance. More notably, our impressive results manifest the advantages of generative classifiers in a large-scale real-world setting. We feel this work opens avenue for research in this field.

---

2 Related Work
Semantic Segmentation. Since the seminal work of FCN [1], deep-net segmentation solutions are typically built in a dense classification fashion, i.e. learning dense representation and categorization end-to-end. By directly adopting discriminative softmax for categorization, FCN-style solutions put focus on learning expressive dense representation; they modify the FCN architecture from various aspects, such as enlarging the receptive field (2,3,47,57-6
modeling multi-scale context [48, 59, 61-77], investigating non-local operations [4,5, 78-84J, and exploring hierarchical information [85- 87]. With similar goal of sharpening representation, later Transformer-style solutions [7-9, 17-9,88, 89] empower attentive networks with, for instance, local contiguity [7] and multi-level feature aggregation [9, 10]. Two very recent attentive models [90,91] formulate the task in an alternative form of mask classification, however, still relying on discriminative softmax.
From the discussion above, we can find that current prevalent segmentation solutions are in essence a pixel-wise, discriminative classifier, which only learns decision boundaries between classes in the pixel feature space [14, 92], without modeling the underlying data distribution. In contrast, our GMMSeg tackles the task from a generative viewpoint. GMMSeg deeply embeds generative opti- mization of GMMs into end-to-end dense representation learning, so as to comprehensively describe the class-aware knowledge [93, 94] in a discriminative feature space. GMMSeg is partly inspired by [13,95], that also probe data structures via intra-class clustering. However, the dense classification in the two works are achieved via non-parametric, nearest centroid retrieving still discriminative model. In [96], though data density is estimated (as mixture of vMF distributions [97]. it is only used as a supervisory signal for dense embedding learning, and the final prediction is still made by a discriminative classifier k--N. Our work represents the first step towards formulating (closed-set) semantic segmentation within a generative neural classification framework.
Discriminative vs Generative Classifiers. Generative classifiers and discriminative classifiers represent two contrasting ways of solving classification tasks [24]. Basically, the generative classifiers (such as Linear Discriminant Analysis and naive Bayes) learn the class densities p(æ|c), while the discriminative classifiers (such as softmax) learn the class boundaries p(c|æ) without regard to the underlying class densities. In practical classification tasks, softmax discriminative classifier is used exclusively [24]. due to its simplicity and excellent discriminative performance. Nonetheless, genera- tive classifiers are widely agreed to have several advantages over their discriminative counterparts [21, 98], e.g., accurately modeling the input distribution, and explicitly identifying unlikely inputs in a natural way. Driven by this common belief, surge of deep learning literature [14,99 99-101] investi- gated the potential (and the limitation) of generative classifiers in adversarial defense [25,26, (02-104), explainable AI [24],
detection [27, 105], and semi semi-supervised learning [22, 23, 99].
As GMMs can express (almost) arbitrary continuous distributions, has been adopted in many neural classifiers [23, 106]. However, most of these GMM classifiers are discriminative models [12, 29,30] that are trained 'discriminatieel"" (i.e., maximizing posteriors p(c|æ)). In GMMSeg, the GMM is purely optimized via EM (i.e., estimating class densities p(z(c)) while the deep representation is trained via gradient backpropagation of the discriminative loss. Thus the whole GMMSeg is a hybrid of genera- tive GMM and discriminative representation, getting the best of two worlds. Although bearing the general idea of trading-off between generative and discriminative classifiers [21-23, 98, 107, 108], none of the previous hybrid algorithms demonstrate their utility in challenging segmentation tasks.
Anomaly Segmentation. Anomaly segmentation strives identify unknown object regions, typically in road-driving scenarios [56]. Existing solutions can be generally categorized into three classes: i) Uncertainty estimation based algorithms [17, 18,41-46) usually approximate the uncertainty from simple statistics of the classification probability or logits of pre-trained segmentation models [17, 18, 41-43), or adopt Bayesian ncural networks with Monte-Carlo dropout to capture pixel uncertainty [44- 46]. ii Outlier exposure based algorithms make use of auxiliary datasets training samples of unexpected objects [38-40]. Therefore, this type of algorithms requires re- re-training the segmentation network, resulting in performance degradation. iiii Image resynthesis based algorithms reconstruct the input image and discriminate the anomaly instances according to the reconstruction error [34-37].
With a generative classifier, our GMMSeg handles anomaly segmentation naturally. without neither external datasets of outliers, nor additional image resynthesis models. also greatly differs from most uncertainty estimation scores of softmax- based segmentation networks [17, 18,41-433 The rosevvan ones are maybe few density estimation-ben based models [56, 1000,,1111,, which directly measure the likelihood of samples w.n.t.
methods that are post-processing techniques adjusting the prediction
3

---

the data distribution. However, they are either limited to pre-trained representation [56] or specialized for anomaly detection with simple data [109, 110]. To our best knowledge, this is the first time to report promising results on both closed-set and open-world large-scale settings, through a single model instance without any change of network architecture as well as training and inference protocols.
3
Methodology
In this section, we first formalize modern semantic segmentation models within a dense discriminative classification framework and discuss defects of such discriminative regime from probabilistic view- point ($3.1). Then we describe our new segmentation framework GMMSeg - that brings a paradigm shift from the discriminative to generative ($3.2). Finally, in $3.3, we provide implementation details.
3.1 Existing Segmentation Solutions: Dense Discriminative Classifier
In the standard semantic segmentation setting, we are given a training dataset D= (2 ) of N pairs of pixel samples 2n, E R and corresponding semantic labels C, E {1..... C}. The goal is to use 'D to learn a classification rule which can predict the label c' E {1,,.. C} of an unseen pixel x'.
Recent mainstream solutions employ a deep neural network for pixel representation learning and softmax for semantic label prediction. Hence they are usually built as a composition of fog:
A dense feature extractor fe: R which is typically an encoder-decoder network that maps the input pixel to a D D-dimensional feature representation æ. ie.. z= z= fo()) ER and A dense classifier 9w: P C which is achieved by parametric softmax that maps each pixel representation a E RD to C real-valued numbers (y.E ] termed as logits, i.e., {ye) I -- (a)).
and uses the logits to compute the posterior probability:
exp(ye)
exp(w.. x be)
exp(w. fo(x) + be) (x) exp( (x)) + b
p(c/ w. 0) expx((y))
(1)
exp(ww__mm be)
where W.E D and b. IR are the weight and bias for class c, respectively; and w = (wic b.c)..
The final prediction is the class with the highest predicted probability: arg max p[c|a:: w, 0).
The feature extractor f and softmax- -based classifier I are jointly trained end-to-end. Their corre- sponding parameters {0, w) are optimized by minimizing the so-called cross-entropy loss on D:
0- w = arg min -E log p(cp:::,..)
(2)
which is equivalent maximizing conditional likelihood, i.e., (((p)))ppp((l)). In some literature [11, 111], such learning strategy is called discriminative training. As softmax directly models the conditional probability distribution p(c|æ)) with no concern for modeling the input distribution p(æ,, c), existing softmax-based segmentation models are in essence a dense discriminative classifier.
Discriminative softmax typically gives good predictive performance, the pixel classification rule depends only on the conditional distribution p(c|x) in the sense of minimum error rate and softmax optimizes the quantity of interest in a concise manner, i.e., learning direct map from inputs the class labels c. In spite of its prevalence and effectiveness, this dense discriminative regime has some drawbacks that still poorly understood: First, it attends only to learning the decision boundaries between the C' classes on the pixel embedding space, i.e., splitting the D-dimensional feature space using C different (D 1D-11 -immenesiol hyperplanes. It achieves simplified approach that eliminates extra parameters for modeling the data (representation) distribution [112]. However, from another perspective, it fails to capture the intrinsic class characteristies and is hard achieve good generaliza- tion on unseen data. Second, in softmax, each class corresponds to only a single weight (we, be). That means existing segmentation models rely on an implicit assumption of unimodality of data of each class in the feature space [12, 113, 1 114]. However, this unimodality assumption is rarely the case in real- world scenarios and makes the model less tolerant of intra-class variances [13], especially when the multimodality remains in the feature space [12]. Third, softmax is not capable of inferring the data distribution it notorious with inflating the probability of the predicted class as a result of the exponent employed on the network outputs [1 15). Thus the prediction score of a class is useless besides its comparative value against other classes. This is the root cause of why existing segmentation models
Sttrctlyy speaking, the dense feature extractor fe typically maps pixel samples with image context, iee. fe: 1t ghxux - > xw xD where l and w (h' and w') denote the spatial resolution of the image (feature map). Here simplify the notations, ie.., fo: 122 +R? to keep straightforward formulation.

---

are hard to identify pixel samples x' of an unseen class (out-of- distributiinn data), i.e., (1. Accordingly, argue that the time might be right rethink the current de facto, discriminative segmentation regime, where the softmax classifier may actually cause more harm than good.
3.2 GMMSeg: Dense GMM Generative Classification
Our GMMSeg reformulates the task from a dense generative classification point of view. Instead of building posterior p(c(ar) directly, generative classifiers predict labels using Bayes rule. Specifically, generative classifiers model the joint distribution p(æ, c). by estimating the class-conditional distribu- tion p(z]e) along with the class prior p(c). Then, following Bayes rule, the posterior is derived as:
p(c)p(x)e) p(c[2) P(e (pppp()))))
(3)
Since the class probabilities p(c) are typically set as uniform prior (also in our case), estimating the class-conditional distributions (i.e., data densities) p(æ: c) is the core and most difficult part of building a generative classifier. is also worth noting that generative classifiers are optimized by approximating the data distribution (((x))))D((()))) which is called generative training [1]].
Although discriminative classifiers demonstrate impressive performance in many application tasks, there are several crucial reasons for using generative rather than discriminative classifiers, which can be succinctly articulated by Feynman's mantra "What cannot create, I do not understand". Surprisingly, generative classifiers have been rarely investigated in modern segmentation models.
Driven by the belief that generative classifiers are the right way remove the shortcomings of diseri- minative approaches, we revisit GMM - one the most classic generative probabilistic classifiers. We couple the generative EM optimization of GMMs with the discriminative learning of the dense feature extractor f f the most successful part of modern segmentation models, leading to a powerful, principled, and dense generative classification based segmentation framework GMMSeg (Fig. 2).
Specifically, GMMSeg adopts a weighted mixture of M multivariate Gaussians for modeling the pixel data distribution of each class in the D-dimensional embedding space:
p(æle; )b.) = Σ M
m=1 p(m]e; n. (.)p( m:; Me. Ex)
Σ
M m=1
Tem (xx; peom Eem.
(4)
Here m e~ Multinomial (T:) is the prior probability, i.e., Σm Tem 1;; Mem R2 and Dem DxD are the mean vector and covariance matrix for component in class G, and Φ. { (T Me. E.}. The mixture nature allows GMMSeg to accurately approximate the data densities and to be superior over softmax assuming unimodality for each class. Each Gaussian component has an independent co- variance structure, enabling flexible local measure of importance along different feature dimensions.
To find the optimal parameters of the GMM classifier, i.e., { standard approach is EM [1 16), ie., maximizing the log likelihood over the feature-label pairs [(r,, Cn 1 in the training dataset D: ]
O arg max
Σ
un. log p( (.) = arg max
Σ
log > m=1 p(æ., m/c; d..,, (5)
dc
EM starts with some initial guess at the maximum likelihood parameters Φ and then proceeds to it- eratively create successive estimates Φ for 1,,,,, 2,,..,,b by repeatedly optimizing F function [117]:
E-Step: d arg max, F(ge, " M-Step: $5 arg max F(g. (.)).
(6)
qel [=|m]=p(m|z;c o.) gives the probability that data is assigned to component m. Fis defined as: F(ge, Φb.) = Eg [log p(æ, mlc; db)] (qc),
(7)
where E. [t] gives the expectation
the distribution over the M components given by yos and H(g.) = -.gg [log ge [m]] defines the entropy of ge- Based on 4-7. for Væ, Ca C, we have:
(t-1) N((, (p (t-1))
E-Step:
2 [m]
Nem
ΣM =1 cm"
(1-1) (111) (( (1-1) (1-11)
cm!
(8)
((())
M-Step: (0)
(4) [m]
(0)
Σ
(8) [m]] p (xn p
N.
(6
[ and
5

---

EM optimization of generative GMM
{p(ca) Nem
{ Y
Fiu Frumd
dense feature
M-step
(Eq.
{ 42.
cross-
eatropy
training
pixel
Estee (Fq.9)
{g), 9s
C Ba
(q
samples
data Rlw
paramette optimization
stockastic gradient descent of discriminative representation learning
Figure 2: Through inneerativeeddscrmmmaaty hybrid training, GMMSeg gains the best the two worlds.
where is the number of training samples labeled as and Nom=-nic.= 9en [m).
In E-step, compute the posterior de over the M components given the old parameters (b(t-1) In M-step, with the soft cluster assignment q the parameters are updated as o! such that the F function is maximized.
In practice, we find standard EM suffers from slow convergence and delivers unsatisfactory re- sults (ef. $4.3). A potential reason the parameter sensitivity of EM convergent parameters may change vastly even with slightly different initialization [118]. Drawing inspiration from recent optimal transport (OT) based clustering algorithms [119, 120], we introduce uniform prior on the mixture weights Te, i.., Vc, m::eem NI Recalling [m] pmm// c), we can derive a constraint
Q.-(qe La,.,,. [m] -x Then E-step in Eq. 6 is performed by restricting the optimi-
M
zation of 9. over the set Q::
E-Step:
( = arg max R. F(ge: &
(9)
This can be intuitively viewed as an equipartition constraint guided clustering process: inside each class c, we expect the No pixel samples to be evenly assigned M components. As indicated by [28]. Eq. 9 is analogous to retropy-regularized OT:
min Σ Q.E0%
n,m Qe(n, (m) e'= (Q. Q.1"=1%M (Q.) Mc No
(10)
where the transport matrix Qe (i.e., target solution) can be viewed as the posterior distribution ge of N₂ samples over the M components (i.e., Qn yon qmm [m]). the cost matrix O2E R. given as the negative log- ligelikli,ood, i.e., Oz(n, m)= - log p p(=,,ll.ee (c, m), and the entropy H(.) is penalized by c. The set O. encapsulates all the desired constraints over Qc, where M is a --ddimnniinnal all-ones vector. Intuitively, the more plausible a pixel sample a,, is with respect to component m. the less it costs to transport the underlying mass. Eq. 10 can be efficiently solved via Sinkhorn Knopp Iteration [120], where is set as the default (i.e., 0.05). This optimization scheme, called Sinkhorn EM, proved to have the same global optimum with the EM in Eq. yet is less prone to getting stuck in local optima [28], which is in line with our empirical results (cf. $4.3).
Our GMMSeg adopts a hybrid training strategy that is partly generative and partly discriminative:
Generative Optimization (Sinkhorn EM) of GMM Classifier: {
M
(arg max &e R
Σ log pog.l (1) = {arg max de
Σ log
Σ
Tem (ax Mem, Eem)
(11)
Discriminative Learning (Cross-Entropy Loss) of Dense Representation: 0
VM Tem ((o(T)) Mom: Eem)
arg min Σ log ( 0) aag min
-- log (x,c)ED
C D Tem ((f(x)) phems Eem)
9
(T.C)ED
In GMMSeg, GMM classifier (has C' M components in total) is purely optimized in a generative fashion, i.e., applying Sinkhorn EM to model the data densities p(ælcc) within each class e in the fea- ture space fe- 'Te feature extractor/space fe. in contrast, is end-to-end trained in a discriminative manner, i.e., minimizing the cross-entropy loss over the posteriors output by the GMM. During each training iteration, the extractor's parameters 0 are only updated by the gradient backpropagated from the discriminative loss, while the GMM's parameters { e are only optimized by EM. To accurately estimate the GMM distributions, an external memory is adopted to store a large set of pixel representations, sampled from several preceding training batches, enabling large-scale EMM Moreover, since the feature space fe gradually evolves during training, we opt for momentum EM: we directly use the GMM's parameters } estimated in the latest iteration the initial guess in the current

---

&T + where the momentum coefficient set as 009.99.. This makes our training more stable and accelerates the convergence of EM we empirically find even one EM loop per training iteration is good enough.
iteration ( Je: and adopt momentum update in the M-Step, ie., + (1-r)<b
This hybrid training scheme brings several advantages: First, GMMSeg achieves the merits of both generative and discriminative learning. The online EM based generative optimization enables the GMM to best fit the data distribution even on the evolving feature space. On the other hand, the feature space is discriminatively end-to-end trained under the guidance of the GMM classifier, so as to maximize the pixel-wise predictive performance. Second, as the generative EM optimization and discriminative stochastic training work in an independent yet closely collaborative manner, GMMSeg fully compatible with modern segmentation network architectures and existing discriminative training objectives. It can be further advanced with the development of network architectures of the discriminative counterparts. Third, as GMMSeg explicitly models class-conditional data distribution p(æ]e), it can naturally handle off-manifold examples, i.e., directly giving meaningful likelihood of the example fitting each class GMM distribution (see $4.2 for experiments on anomaly segmentation).
3.3 Implementation Details
Network Architecture. GMMSeg is general framework that can be built upon any modern segmen- tation network by replacing softmax with the GMM classifier. In our experiments (cf. $4.1), we approach GMMSeg on a variety of segmentation models [7.47-49] and backbones [50,51]. J. In the GMM classifier, a Ix] 1 conv is used to compress each pixel feature to a 64-dimensional vector, i.e D D= 64, and the covariance matrices Σ DDD are constrained to be diagonal, for computational efficiency. In our implementation, each class e is represented by a mixture of M = 5 Gaussians (there are a total of 5C Gaussian components for a segmentation task with C semantic classes). Furthermore, we adopt the winner-take-all assumption [121, 122], i.e., the class-wise responsibility (Eq. 4) is dominated by the largest term, for better performance.
Training In each training iteration, we conduct one loop of momentum (Sinkhorn) EM (i.e., t=1) on current training batch as well as the external memory for the generative optimization of GMM, and backpropagate the gradient of the cross-entropy loss on current batch for the discriminative training of the feature extractor. The external memory maintains queue for each component in each class; each queue gathers 32K pixel features from previous training batches in firsi in, first oul manner. To improve the diversity of the stored pixel features, we sample sparse set of 100 pixels per class from each image, instead of directly storing the whole images into the memory. Note that the memory is discarded after training, and does not introduce extra overheads in inference.
Inference. GMMSeg only brings negligible delay in the inference speed compared to the discrimina- tive counterparts (see experiments in $4.3). For standard (closed-set) semantic segmentation, pixel prediction is made using Bayes rule (cf. Eq. 3): arg max, p(c/æ), where p(c|æ) x p(æ|c) with the uniform class distribution prior: p(c) 1/C. For anomaly segmentation, the pixel-wise uncertain- ty/anomaly score can be naturally raised as: -max, p( c), i.e., the outlier input should reside in low-probability regions [123].
4 Experiments
We respectively examine the efficacy and robustness of GMMSeg on semantic segmentation ($4.1) and anomaly segmentation ($4.2). In $4.3, we provide diagnostic analysis on our core model design.
4.1 Experiments on Semantic Segmentation
Datasets. We conduct experiments on three widely used semantic segmentation datasets:
ADE [53] has 20K/2K/3K images in train/val/test set, with 150 stuff/objjet categories in total.
Cityscapes [54] has 2,975/500/1, 10244/00001200440000 fine-labeled images for train/val/test set with 19 classes.
COCO- [55] has 10K images (9K/IK for train/test), pixel-wise labeled with 171 classes.
Base Segmentation Architectures and Backbones. For thorough evaluation we apply GMMSeg to four famous segmentation architectures (i.e., DeepLabva. OCRNet [48]. UPenNet [49], Segfor- mer [7]), with various backbones (i.e., ResNet [50], HRNet Swin [52], MiT [7). For fairness, we re-implement these models using the standardized hyper-parameter setting in MMSegmentation [124].
Training Details. GMMSeg is implemented on MMSegmentation [ [124] and follows the standard training setting for each datasct. All models are initialized with ImageNet- [1[55]] pretrained back-
7

---

ADEZUM
Glyyscape
COCO Shiff
SegFormer [7]
GMMSeg
SegFormer [7]
GMMSeg
SegFormer [7]
GMMSeg
Figure 3: Qualitative results ($4.1) on ADE2OK [53], Cityscapes [54], and COCO-Stuff [55].
bones and trained with commonly used data augmentations including resizing, flipping, color jittering images are cropped
and cropping. For 20E20K 768 and models are trained for 160K/80K/80K iterations with 16/16/8 batch size, using 8/16 NVIDIA Tesla A00 GPUs. Other training hyper-parameee (i.e., optimizers, learning rates, weight decays, schedulers) are set as the default in MMSegmentation and can be found in the supplementary.
Inference Details. For ADE2ok and COCO-Stufii we keep the aspect ratio of test images and rescale the short side to 512. For Cityscapes, sliding window inference is used with 768x768 window size. Note that for fairness, all our results are reported without any test-time data augmentation.
Quantitative Results. Table demonstrates our quantitative results. Although mainly focusing on the comparison with the four base segmentation models [7, 17447,49, we further include five widely recognized methods [1,3,8, 9, 90] for completeness. As can be seen, our GMMSeg outperforms all its discriminative counterparts across various datasets, backbones, and network architectures (FCN-style and Transformer-like likke:
Table I: Quantitative results ($4.1) 1) on ADE2oK [53] val, City- scapes [54] val, and COCO-Stur [55] test with mean IoU.
ADE [53] val. With FCN- style segmentation neural ar- chitectures, i.e., DeepLabys+ and OCR, GMMSeg provides 1.2%/1.5% mloU gains over corresponding discriminative models. Similar performance improvements, i.e., 1.0% and 0.6%. are also obtained with attentive neural architectures, ie.., Swinnnnn-eeeNeee and SegFor- mer, manifesting the universal- ity and efficacy of GMMSeg. Cityscapes AgaiinnAAgain our GMMSeg surpasses all its dis- criminative counterparts by large margins, e.g., 0.5% over DeepLabv3+. 0.8% over OCRNet, 0.7%
Method
Backbone
2DE
Citys
COCO.
FCN [CVIRSs]
[]]
ResNetgo
39.9
75.5
32.6
PSPNet [CVPR17] []
ResNetjo
44.4
79.8
37.8
SETR (CVP821)
9
La
48.2
79.2
Segmenter (осv1)) [8]
$51.8
79.1
MaskFormer (Ncari1S21 [90]
twwinnas
$52.7
DeepLaby3, [BCCV18] [47]
45.5
80.6
33.8
ResNetjo
GMMSeg
46.7
81.1 +0.5
35.5 11.
OCRNel [BC
[48]
"43.3"
"80.4""
37.6
HRNely2w4s
GMMSeg
44.8
81.2
39.2 1.6
UertNe (BCCVIS)
[49]
48.0
81.1
43.4
Swingase
GMMSeg
49.0
81.8 10.7
44.3 10.9
FegFormer [NeutPS21]
[7]
50.0
822.0
44.0
MiT
GMMSeg
50.6 -0.6
82.6 10.6
44.7 10.7
t: pretrained on ImageNet22:: 1: using larger crop-size, te., 640x640
over Swin- UperNe,, and 0.6% over SegFormer, suggesting its wide utility in this field.
COCO-Stuff [55] test. Our GMMSeg also demonstrates promising results. This is particularly impressive considering these results are achieved by a dense generative classifier, while the semantic segmentation task is commonly considered as battlefield for discriminative approaches.
Qualitative Results. In Fig. 3, we illustrate the qualitative comparisons of our GMMSeg against SegFormer [7]. It is evident that, among the representative samples in the three datasets, our method yields more accurate predictions when facing challenging scenarios, e.g., unconspicuous objects.
4.2 Experiments on Anomaly Segmentation
Datasets. To fully reveal the merits of our generative method, we next test its robustness for abnormal data, i.e., identifying test samples of unseen classes, using two popular anomaly segmentation datasets: Fishyscapes Lost& [56], built upon [126], has 100/275 val/test images. It collected under the same setup as Cityscapes [54] but with real obstacles on the road. Pixels are labeled as either back- ground (i.e., dre-de Cityscapes classes) or anomaly (i.e., other unexpected classes like crate).
Road Anomaly [36] has 60 images containing anomalous objects in unusual road conditions.
Evaluation Metrics. The area under receiver operating characteristics (AUROC), average precision (AP), and false positive rate (FPRg5) at true positive rate of 95%, are adopted following 35,65.
8

---

Table 2: Quantitative results ($4.2) on Fishyscapes Foun& [56] val and Road Anomaly [36].
Extra
OOD
Fishyscapes
Lost&Found
Road Anomaly
Method
mloU
Resyn.
Data
AUROC1
APt
FPRg +
AUROC1
APT
FPR +
SynthCP (BOCV20) [35]
80.3
88.34
6.54
45.95
76.08
24.86
64.69
SynBoost (CVPR21) [34]
/
/
96.21
60.58
31,02
81.91
38.21
64.75
MSP poiam [7]]
80.3
86.99
6.02
45.63
73.76
20.59
68.44
Entropy (Cuam [17]
80.3
88.32
13.91
44.85
75.12
22.38
68.15
SMI (oeess) [18]
80.3
96.88
36.55
14.53
81.96
25.82
49.74
"Mahalanobis [Nwotts1s] [19]
80.3
92.51
27.83
30.17
76.73
22.85
59.20
81.1
97.34
43.47
13.11
84.71
34,42
47.90
"GMMSeg-PCN
76.7
96.28
32.94
16.07
78.99
24.51
56.95
WWMMSeg-SegFormeg
X
82.6
97.83
50.03
12.55
89.37
57.65
44.34
"I confidence derived with generative formulation
MSP [17)
GMMSeg
MSP
GMMSey
Figure 4: Qualitative results ($4.2) of anomaly heatmaps on Fishyscapes Fost&Found [56] val.
Experiment Protocol. As in [17, 18, 127], we adopt ResNetro: -DeepLabys- architecture. For com- pleteness, we also report the results of our GMMSeg based on ResNettor- and MiT Forgorrr. All our models are the same ones in Table 1, i.e., trained on Cityscapes train only. As GMMSeg esti- mates class densities p(zr(c), it can naturally reject unlikely inputs (ef. $3.3), iee, directly thresholding -max, p(a]cc for computing the anomaly segmentation metrics, without any poxt-processingg
Quantitative Results. As shown in Table 2. based on DeepLabv3+ architecture. GMMSeg outper- forms all the competitors under the same setting, i.e neither using external out-of-distribun data nor extra resynthesis module. Note that, [17-19] rely on pre-trained discriminative segmentation models and thus have to make post-calibration. However, GMMSeg directly derives meaningfuul confidence scores from likelihood p(z|c). Mahalanobis [19] also models data density, yet, merely on pre-trained feature space with single Gaussian per class. In contrast, GMMSeg performs much better, proving the superiority of mixture modeling and hybrid training. Even with a weaker architecture, i.e., FCN, GMMSeg still performs robustly. When adopting SegFormer, better performance is achieved.
Qualitative Results. In Fig. 4, we visualize the anomaly score heatmaps generated by MSP (17]- Deep [47] and MaMMeg--Deppl with overconfident predictions; contrast, GMMSeg naturally rejects them (red colored regions).
The softmax based counterpart ignores the anomalies
4.3 Diagnostic Experiments
For in-depth analysis, we conduct ablative studies using DeepLabya [47]-Reseet-- [50] segmentation architecture. Due to limited space, we put some diagnostic experiments in our supplementary material.
Online Hybrid Training. We first investigate our hybrid training strategy (cf. Eq. 11), where the discriminative feature extractor and generative GMM classifier are online optimized iteratively. Owe to this ingenious design, both components are gradually up- dated, aligned with and adaptive each other, making GMMSeg a compact model. To fully demonstrate the effectiveness, we study a variant, DeepLabys- + GMM,
Table 3: Online hybrid training
($4.3), evaluated on ADE [53].
Method
mloU (%)
DecpLabys, GMM
31.6
GMMSeg-DeeLLabss
46.0
where a GMM classifier is directly fitted onto the feature space trained with the softmax elassifier beforehand. As shown in Table 3, a clear performance drop is observed, i.e., mloU: 46.0%- 31.6%, revealing the appealing efficacy of our end-to-end hybrid training strategy.
Discriminative GMMSeg Generative GMMSeg.
Table 4: Discriminative GMMSeg vs. generative GMMSeg ($4.3),
Cityscapes mloU?
Fishyscapes Found AUROCT API FPR
GMMSeg
Training Objective
Our GMMSeg learns gen- erative GMM via EM, i..,, max p(x|p Φ). with discri- Generative
Discriminative
max plc 0) max p(arle maxxplller
8L.0
89.77
17.68
51.81
81.1
97.34
43.47
13.11
minative representation learning, iee., max p(c ac;; 0). A discriminative counterpart can be achieved by end-to-end learning all the parameters, i.e., {Ø, 0}. with cross-entropy loss, i.e., p(ckæ; &, 0). Discriminative GMMSeg sacrifices data characterization for more flexiblility in discrimination, and
yields poor performance in open-world setting. While inapparent effect on closed-set Cityscapes is observed, which in turn verifies the accurate specification of data distribution in generative GMMSeg.

---

Standard EM vs. Sinkhorn EM. In our GMMSeg, we leverage the entropic OT
Table 5: Ablative studies ($4.3) on ADE [53] val. The adopted settings are marked in red.
based Sinkhorn EM [28] (cf. Eq. 10) in-
EM algorithm
Loop
mloU (%)
Component
mloU (%)
stead of the classic one (ef. Eq. 8) for
the generative optimization of the GMM.
vanilla EM
10
42.7
44.8
A = 1
= 3
44,2
45.3
In Table Sa, we investigate the impacts of these two different EM algorithms and show that Sinkhorn EM is more fa- vored. More specifically, during the E- step, rather than the vanilla EM assigning data samples to Gaussian components independently, Sinkhorn EM restricts the assignment with equipartition constraint. As pointed out in [28], in- corporating such prior information about the mixing weights of GMM components leads to higher curvature around the global optimum. Our empirical results confirm this theoretical finding.
46.0
46.0
Sinkhom EM
46.0
M 10
46.0
10
46.0
M 15
45.7
(a) EM optimization
(b) # Component per class
Number of EM Loop per Training Iteration. EM algorithm alternates between E-step and M-step for maximum- -ikellhooo inference (cf. Eq. 6). In GMMSeg, in order to blend EM with stochastic gradient descent, we adopt an online version of (Sinkhorn) EM based on momentum update. In Table 5a, we also study the influence of looping EM different times per training iteration. We can find that one loop per iteration is enough to catch the drift of the gradually updated feature space.
Number of Gaussian Components per Class. In GMMSeg, data distribution of each class is modeled by a mixture of M Gaussian components (cf. Eq. 4). Table 5b shows the results with different values of M. When M = 1, each class corresponds to a single Gaussian, which is directly estimated via Gaussian Discriminant Analysis, without EM. This baseline achieves 44.2% mloU. After adopting the mixture model, i.e., M:1 + -5,, the performance is greatly improved, i.e., mloU: 44.2%-> 45.3%-46.00 This verifies our hypothesis of class multimodality. Yet, further increasing compo- nent number (i.e., :5 + 15) only brings marginal even negative gains, due to overparameteriizatio.
Confidence Calibration. We further study the model calibration of GMM- Seg and the discriminative counterpart, i.e., DeepLabv3, [49] with the softmax classifier. In Fig. 5, we illustrate the Expected Calibration Error (ECE) [15] along with reliability diagrams, which plot the expected pixel accuracy as
10
1.0
Accuracy
Accuracy
0.8
Gap
0.8
Gap
Accuracy
0.6
0.6
0.4
0.4
0.2
0.2
ECE:
0.1065
ECE: 0.0766
function of confidence [15]. As seen, GMMSeg yields better calibrated prefic- tions, i..., smaller gaps between the ex- pected accuracy and confidence. On the other hand, the discriminative softmax produces confidences that deviate more from the true probabil- ities, and suffers higher calibration error accordingly. This again verifies the better reliability and interpretability of GMMSeg compared to its discriminative counterparts.
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
10
Confidence
Figure 5: Reliability diagrams for DeepLabys- [49] (left) and GMMSeg DeeSee--eeppabys. (right) on Cityscapes val.
Runtime Analysis. The inference speed of GMMSeg is 13.37 fps, which only yields negligible overhead w.r.t. its discriminative softmax counterpart, i.e., 13.37 vs. 14.16 fps. We measure the fps on single NVIDIA GeForce RTX 3090 GPU with a batch size of one.
Conclusion
We presented GMMSeg, the first generative neural framework for semantic segmentation. By explicitly modeling data distribution as GMMs, GMMSeg shows promise to solve the intrinsic limitations of current softmax based discriminative regime. It successfully optimizes generative GMM with end- to-end discriminative representation learning in a compact and collaborative manner. This makes GMMSeg principled and well applicable in both closed-set and open-world settings. We believe this work provides fundamental insights and can benefit a broad range of application tasks. As a part of future work, we will explore algorithm in image classification and trustworthy Al related tasks.
Acknowledgement. This work was partially supported by the Fundamental Research Funds for the Central Universities (No. 226-2022-00087 and by the National Key R&D Program of China (No. 2020AAA0108000), Wenguan Wang acknowledges partial support from Australian Research Council (ARC), DECRA DE220101390.
10

---

References
[1] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmen- tation. In CVPR, 2015. 1, 8
[2] Liang-Chieen -hich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.
IEEE TPAMI, 2017. 1,3
[3] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing
network. In CVPR, 2017. 1, 3,
[4] Jie Hu, Li Shen, and Cang Sun. Squeeze-and -ndieeettndexxitiioonnn networks. In CVPR, 2018. 1.3
[5] Jun Fu, Jing Liu, Hajjie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu, Dual attention network for seene segmentation. In CVPR, 2019. 1,3
[6] Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[7] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurlPS, 2021. 1, 3, 8,
16
[8] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, 2021. 3, 8, 16
[9] Sixiao Zheng, Jiachen Lu. Hengshuang Zhao. Xiatian Zhu. Zekun Luo. Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequene perspective with transformers. In CVPR, 2021. 3,
[10] Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang. Hrformer:
High-resolution transformer for dense prediction. In NeurIPS, 2021. 3, 16
[11] JM Bernardo, MJ Bayarri, JO Berger, AP Dawid, D Heckerman, AFM Smith, and M West. Generative or discriminative? getting the best of both worlds. Bayesian statistics, 2007. 1, 4,5
[12] Hidcaki Hayashi and Seiichi Uchida. A discriminative gaussian mixture model with sparsity. In ICLR, 2021. 1, 2, 3, 4
[13] Tianfei Zhou, Wenguan Wang. Ender Konukoglu, and Luc Van Gool. Rethinking semantic segmentation:
A prototype view. In CVPR, 2022. 1, 3, 4,
[14] Lynton Ardizzone, Radek Mackowiak, Carsten Rother. and Ullrich Köthe. Training normalizing flows
with the information bottleneck for competitive generative classification. In NewrlPS, 2020. 2, 3
[15) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML, 2017. 2, 10
[16] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. NeurlPS, 2019. 2
[17] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of--istiiiutiio examples neural networks. In ICLR, 2017. 2, 3,
[18] Sanghun Jung. Jungsoo Lee. Daehoon Gwak, Sungha Choi, and Jaegul Choo. Standardized max logits: A simple yet effective approach for identifying unexpected road obstacles in urban-scene segmentation. In ICCV, 2021. 2, 3,
[19] Kimin Lec, Kibok Lec, Honglak Loe, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. NewrIPS, 2018. 2,
[20] Bradley Efron. The efficiency of logistic regression compared normal discriminant analysis. Journal of the American Statistical Association, 1975. 2
[21] Andrew Ng and Michael Jordan. On discriminative vs. generative classifiers: comparison of logistic
regression and naive bayes. In NewrIPS, 2001. 2,
[22) Eric Nalisnick, Akihiro Matsukawa, Yee Whye Tch, Dilan Gorur, and Balaji Lakshminarayanan. Hybrid models with deep and invertible features. In /CML., 2019. 2,3
[23] Pavel Izmailov, Polina Kirichenko, Mare Finzi, and Andrew Gordon Wilson. Semi-supervised leamning with normalizing flows. In JCML, 2020. 2.
[24] Radek Mackowiak, Lynton Ardizzone, Ullrich Kothe, and Carsten Rother. Generative classifiers as basis for trustworthy image classification. In CVPR, 2021. 2,3
[25] Lukas Schott, Jonas Rauber, Matthis Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on mnist. In NewrIPS, 2018. 2,

---

[26] Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Lever- aging generative models understand and defend against adversarial examples. In ICLR. 2018. 2.
3
[27] Joan Serrå, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F Núñezz and Jordi Luque. Input complexity and ditto--distributinn detection with likelihood-based generative models. In ICLR, 2019. 2,
[28] Gonzalo Mena. Amin Nejatbakhsh, Erdem Varol, and Jonathan Niles-Weed, Sinkhorn em: an expectation- maximization algorithm based entropic optimal transport. arXiv preprint (XXiv:2006.16444 2020.
10
[29] Ehsan Variani, Erik McDermott, and Georg Heigold. A gaussian mixture model layer jointly optimized
with discriminative features within a deep neural network architecture. In ICASSP, 2015. 2,3
[30] Zoltán Tüske, Muhammad Ali Tahir, Ralf Schlüter, and Hermann Ney. Integrating gaussian mixtures into
deep neural networks: Softmax layer with hidden variables. In ICASSP, 2015. 2,
[31] Aldebaro Klautau, Nikola Jevtic, and Alon Orlitsky. Discriminative gaussian mixture models:
comparison with kernel classifiers. In ICML, 2003.
[32] Zhihao Zheng and Pengyu Hong. Robust detection of adversarial attacks modeling the intrinsic properties of deep neural networks. NeurlPS, 2018. 2
[33] Kimin Lee. Sukmin Yun. Kibok Lee, Honglak Lee. Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In ICML, 2019.
[34] Giancarlo Di Biase, Hermann Blum, Roland Siegwart, and Cesar Cadena. Pixel-wise anomaly detection in complex driving scenes. In CVPR, 2021. 2, 2,
[35] Yingda Xia, Yi Zhang, Fengze Liu, Wei Shen, and Alan Yuille. Synthesize then compare: Detecting failures and anomalies for semantic segmentation. In ECCV, 2020. 2, 3, 8,
[36] Krzysztof Lis. Krishna Nakka, Pascal Fua, and Mathieu Salzmann. Detecting the unexpected via image resynthesis. In ICCV, 2019. 2, 2, 8,
[37] Tomas Vojir, Tomáš Šipka, Rahaf Aljundi, Nikolay Chumerin, Daniel Olmeda Reino, and Jiri Matas. Road anomaly detection by partial image reconstruction with segmentation coupling. In ICCV, 2021. 2.3
[38] Petra Bevandić, Ivan Krešo, Marin Oršić, and Siniša Segvić. Simultancous semantic segmentation and outlier detection in presence of domain shift. GCPR, 2019. 2,
[39] Robin Chan, Matthias Rottmann, and Hanno Gottschalk. Entropy maximization and meta classification for -iittrirbbuu detection in semantic segmentation. In ICCV, 2021. 2,
[40] Dan Hendrycks. Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv: 0 2.04606, 2018. 2,
[41] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of duitr--disttibution image
detection in neural networks. arXiv preprint arXiv: 1706. 02690, 2017. 2,
[42] KIMIN LEE, Kibok Lee, Honglak Lee, and Jinwoo Shin. Training coniiennee-calireeg classifiers for detecting diittr--iitiiittionn samples. In ICLR, 2018. 2,
[43] Matthias Rottmann, Pascal Colling. Thomas Paul Hack, Robin Chan, Fabian Hüger, Peter Schlicht, and Hanno Gottschalk. Prediction error meta classification in semantic segmentation: Detcction via aggregated dispersion measures of softmax probabilitics. IJCNN, 2020. 2,
[44] Alex Kendall and Yarin Gal. What uncertainties do need in bayesian deep learning for computer vision? In NeurIPS, 2017. 2,
[45] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurlPS, 2017. 2,3
[46] Jishnu Mukhoti and Yarin Gal. Evaluating bayesian deep learning methods for semantic segmentation. arXiv preprint 1rXiv:1811.122 127099 2018. 2,
[47] Liang-Chich Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018. 2, 7, 8,
[48] Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmenta- tion. In ECCV, 2020. 2, 3, 7,8
[49] Tete Xiao, Yingeheng Liu, Bolci Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding, In ECCV, 2018. 2.7.8, 10
[50] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 2, 7.9
12

---

[51] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang. Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE TPAMI, 2020. 2.7
[52] Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In ICCV, 2021. 2,7
[53] Bolei Zhou, Hang Zhao, Xavier Puig. Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing
through ade20k dataset. In CVPR, 2017. 2. 8, 9, 10
[54] Marius Cordts, Mohamed Omran, Schastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Be- nenson, Uwe Franke. Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding, In CVPR, 2016. 2, 8
[55] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018. 2, 7,8
[56] Hermann Blum, Paul-Edde aduu-Edo Sarlin, Juan Nieto, Roland Siegwart, and Cesar Cadena. The fishyscapes benchmark: Measuring blind spots in semantic segmentation. ICV, 2021. 2, 3, 4, 8,9
[57] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In CVPR, 2017. 3
[58] Maoke Yang, Kun Yu, Chi Zhang. Zhiwei Li, and Kuiyuan Yang. Denseaspp for semantic segmentation in street scenes. In CVPR, 2018. 3
[59] Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR, 2016, [60] Guolei Sun. Wenguan Wang, Jifeng Dai, and Luc Van Gool. Mining cross cross-image semantics for weakly supervised semantic segmentation. In ECCV, 2020. 3
[61] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical
image segmentation. In MICCAI, 2015. 3
[62] Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paree Paredes, Vibhav Vinect, Zhizhong Su, Dalong Du, Chang Huang, and Philip HS Torr. Conditional random fields as recurrent neural networks. In ICCV, 2015. 3
[63] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE TPAMI. 2017. 3
[64] Ziwei Liu, Xiaoxiao Li, Ping Luo, Chen Change Loy, and Xiaoou Tang. Deep learning markov random field for semantic segmentation. IEEE TPAMI, 2017. 3
[65] Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid, Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In CVPR, 2017. 3
[66] Sachin Mehta, Mohammad Rastegari, Anat Caspi, Linda Shapiro, and Hannaneh Hajishirzi. Espnet: Efficient spatial pyramid of dilated convolutions for semantic segmentation. ECCV, 2018.
[67] Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi. and Amit Agrawal Context encoding for semantic segmentation. In CVPR, 2018.
[68] Junjun He, Zhongying Deng, Lei Zhou, Yali Wang, and Yu Qiao. Adaptive pyramid context network for semantic segmentation. In CVPR, 2019.
[69] Hanzhe Hu, Deyi Ji, Weihao Gan, Shuai Bai, Wei Wu, and Junjie Yan. Class-wise dynamic graph convolution for semantic segmentation. In ECCV, 2020. 3
[70] Changqian Yu, Jingbo Wang, Changxin Gao, Gang Yu, Chunhua Shen, and Nong Sang. Context prior for scene segmentation. In CVPR, 2020. 3
[71] Mingyuan Liu, Dan Schonfeld, and Wei Tang. Exploit visual dependency relations for semantic segmen- tation. In CVPR, 2021. 3
[72] Chi- Wei Hsiao, Cheng Sun, Hwann-Tzong Chen, and Min Sun. Specialize and fuse: Pyramidal output representation for semantic segmentation. In ICCV, 2021. 3
[73] Zhenchao Jin, Bin Liu, Qi Chu, and Nenghai Yu. Isnet: Integrate image-level and semantic-level context for semantic segmentation. In ICCV, 2021. 3
[74] Zhenchao Jin, Tao Gong, Dongdong Yu, Qi Chu, Jian Wang. Changhu Wang. and Jie Shao. Mining contextual information beyond image for semantic segmentation. In ICCV, 2021. 3
[75] Wenguan Wang. Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool. Exploring cross-image pixel contrast for semantic segmentation. In ICCV, 2021.
[76] Jiaxu Miao, Yunchao Wei, Yu Wu, Chen Liang, Guangrui Li, and Yi Yang. Vspw: A sarge-scale dataset for video scene parsing in the wild. In CVPR, 2021. 3
13

---

[77] Zongxin Yang, Yunchao Wei, and Yi Yang. Collaborative video object segmentation by multi-scale boreground-backgrouudd integration. IEEE TPAMI, 2021. 3
[78] Adam W Harley, Konstantinos G Derpanis, and lasonas Kokkinos. Segmentation-awenr convolutional
networks using local attention masks. In ICCV, 2017. 3
[79] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR,
2018. 3
[80] Hanchao Li, Pengfei Xiong, Jie An, and Lingxue Wang. Pyramid attention network for semantic segmentation. arXiv preprint a805 10180, 2018. 3
[81] Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia. Psanet:
Point wise spatial attention network for scene parsing. In ECCV, 2018.
[82] Junjun He, Zhongying Deng, and Yu Qiao. Dynamic multi-scale filters for semantic segmentation. In
ICCV, 2019. 3
[83] Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang. Zhouchen Lin, and Hong Liu. Expectation- maximization attention networks for semantic segmentation. In ICCV, 2019. 3
[84] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Conet: Criss-cross atlention for semantic segmentation. In ICCV, 2019.
[85] Liulei Li, Tianfci Zhou, Wenguan Wang. Jianwu Li, and Yi Yang. Deep hierarchical semantic segmenta- tion. In CVPR, 2022. 3
[86] Wenguan Wang. Hailong Zhu, Jifeng Dai, Yanwei Pang. Jianbing Shen, and Ling Shao. Hierarchical human parsing with typed part-relation reasoning. In CVPR, 2020. 3
[87] Wenguan Wang, Zhijic Zhang, Siyuan Qi, Jianbing Shen, Yanwei Pang, and Ling Shao. Learning compositional neural information fusion for human parsing. In ICCV, 2019. 3
[88] Zongxin Yang. Yunchao Wei, and Yi Yang. Associating objects with transformers for video object segmentation. NewrIPS, 2021. 3
[89] Zongxin Yang. Jiaxu Miao, Xiaohan Wang, Yunchao Wei, and Yi Yang. Associating objects with scalable transformers for video object segmentation. arXiv preprint arXiv:2203. 1442, 2022. 3
[90] Bowen Cheng. Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification not all you need for semantic segmentation. In NeurlPS, 2021. 3,8
[91] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. CVPR, 2022.
[92] Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. In ICMI, 2016. 3
[93] Yi Yang. Yueting Zhuang, and Yunhe Pan. Multiple knowledge representation for big data artificial intelligence: framework, applications, and case studies. Frontiers of Information Technology & Electronic
Engineering, 2021. 3
[94] Wenguan Wang and Yi Yang. Towards data-and knowledge-driven artificial intelligence: A survey on neuro-symbolic computing. arXiv preprint wxXiv:2210.15883 2022. 3
[95] Wenguan Wang, Cheng Han, Tianfei Zhou, and Dongfang Liu. Visual recognition with deep nearest centroids. arXiv preprint wxXiv2220.073888 2022. 3
[96] Jyh-Jing Hwang, Stella X Yu, Jianbo Shi, Maxwell D Collins, Tien-Ju Yang. Xiao Zhang, and Liang-Chich Chen. Segsort: Segmentation by discriminative sorting of segments. In ICCV, 2019. 3
[97] Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, Suvrit Sra, and Greg Ridgeway. Clustering on the unit hypersphere using von mises-fisher distributions. Journal of Machine Learning Research, 2005.
[98] Rajat Raina, Yirong Shen, Andrew Mccallum, and Andrew Ng. Classification with hybrid generative/dis- criminative models. NewrlPS, 2003. 3
[99] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurlPS, 2016. 3
[100] Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Hearik Jacobsen. Residual flows for invertible generative modeling. In NewrIPS, 2019. 3
[101] Will Grathwohl, Kuan-Chich Wang, Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In ICLR, 2019.
[102] Yingzhen Li. John Bradshaw, and Yash Sharma. Are generative classifiers more robust to adversarial attacks? In ICML, 2019.
14

---

[103] Xinshuai Dong, Hong Liu, Rongrong Ji, Liujuan Cao, Qixiang Ye. Jianzhuang Liu, and Qi Tian. Api-net: Robust generative classifier via single discriminator. In ECCV, 2020. 3
[104] Ethan Fetaya, Körn-Henrik Jacobsen, Will Grathwohl, and Richard Zemel. Understanding the limitations of conditional generative models. In ICLR, 2020. 3
[105] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? In ICLR, 2019. 3
[106] Florian Wenzel, Théo Galy-Fajou, Christan Donner, Marius Kloft, and Manfred Opper. Efficient gaussian process classification using polya-gamma data augmentation. In AAAI, 2019. 3
[107] Tommi Jaakkola and David Haussler. Exploiting generative models in discriminative classifiers. In NeurlPS, 1998. 3
[108] Julia A Lasserre, Christopher M Bishop, and Thomas P Minka. Principled hybrids of generative and discriminative models. In CVPR, 2006.
[109] Hyunsun Choi, Frie lang, and Alexander Alemi. Waic, but why? generative ensembles for robust anomaly detection, arXiv preprint arXiv:/800.01. 1810.01392 2018. 3,4
[110] Jie Ren, Peter Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios
distrribbttonnoonnnnn detection. NeurIPS, 2019. 3,4
[111] Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi-supervised learning with deep generative models. In NeurlPS, 2014. 4
[112] Guillaume Bouchard and Bill Triggs. The tradcoff between generative and discriminative classifiers. In IASC International Symposium on Computational Statistics, 2004, 4
[113] Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Distance-based image classification: Generalizing to new classes at near-zero cost. IEEE TPAMI, 2013. 4
[114] Kateryna Chumachenko, Alexandros Iosifidis, and Moncef Gabbouj. Feedforward neural networks initialization based on discriminant lcarning. Neural Networks, 146, 2022. 4
[115] Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning quantify classification uncertainty. In NeurlPS, 2020.
[116] Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via
the em algorithm. Journal of the Royal Statisticul Society: Series B (Methodological), 1977. 5
[117) Radford M Neal and Geoffrey Hinton, A view the em algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models. 1998. 5
[118] Naonori Ueda and Ryohei Nakano. Deterministic annealing variant of the em algorithm. NeurlPS, 1994.
[119] Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning. In ICLR, 2020. 6
[120] Marco Cuturi. Sinkhom distances: Lightspeed computation of optimal transport. In NeurlPS, 2013. 6
[121] Steven Nowlan. Maximum likelihood competitive learning. NeurIPS, 1989.
[122] Nanda Kambhatla and Todd Leen. Classifying with gaussian mixtures and clusters. NeurlPS, 1994. 7
[123] Xuefeng Du, Xin Wang, Gabriel Gozum, and Yixuan Li. Unknown-aware object detection: Learning what you don't 't know from videos in the wild. CVPR, 2022.
[124]
MMSogmentation Contributors. MMScgmentation: Openmmlab semantic segmentation toolbox and benchmark. https: //tgtttb../..t//////////.. com/ wwwnopplanneaaeeeeeeee.eaaaaaaam mmwab/ /mw///mmtttatiinnnm.mmmmo 2020. 7
[125] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet
large scale visual recognition challenge. IJCV, 2015. 7
[126] Peter Pinggera, Sebastian Ramos, Stefan Gehrig, Uwe Franke, Carsten Rother, and Rudolf Mester. Lost and found: detecting small road hazards for self-driving vehicles. In IROS, 2016. 8
[127] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- Ensrgy-based out-of distiiibubtto detection. 2020.
NewrIPS, 2020. 9
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the pa- per's contributions and scope? [Yes] Our main claims accurately reflect the paper's contributions and scope.
15

---

(b) Did you describe the limitations of your work? [Yes] We provide detailed discussions on the limitations in the supplemental material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We provide detailed discussions on potential negative societal impacts in the supplemental material. (d) Have you read the ethies review guidelines and ensured that your paper conforms to them? [Yes] We have read the full ethics review guidelines and ensure that our paper conforms them.
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Not applicable. No theoretical contribution is claimed.
(b) Did you include complete proofs of all theoretical results? [N/A] Not applicable. No theoretical contribution is claimed.
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as a URL)? [No] We promise code and instructions shall be made publicly available right after acceptance. And we believe we have revealed sufficient details of our pipeline ($3.2), implementation details on network architecture/raannnnnneeee data and commonly adopted evaluation protocols ($4.1, 1, $4.2).
($3.3, $4.1. $4.2), publicly available
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] The training details reported in $4.1. Also see $4.3 and the supplemental material for diagnostic experiments.
(c) Did you report error bars (c.g., with respect to the random seed after running ex- periments multiple times)? [No] We follow the standard evaluation protocol in this field [7,8 8, 10, 13] to report and compare the results. It is unaffordable in terms of both time and money to run all experiments multiple times in our paper.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] Eight sixteen NVIDIA Tesla A100 GPUs are used for all experiments. Also see $4.1.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the used assets in $4.1 and $4.2.
(b) Did you mention the license of the assets? [Ye]] We list the license of the used assets in the supplemental material.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A] We do not claim newly released assets.
(d) Did you discuss whether and how consent was obtained from people whose data you' rre're using/curating? [Yes] We provide detailed discussions about data using consent in the supplemental material.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] The data used in our paper neither contain any personally identifiable information nor offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] Not applicable. Neither crowdsourcing nor human subjective research is conducted.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] Not applicable. Neither crowdsourcing nor human subjective research conducted.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] Not applicable. Neither crowdsourcing human subjective research is conducted.
16


Reviews:
[{"review_id": "hUSxQb3KhMj", "reviewer": "Reviewer_1Te2", "summary": "This paper proposes using a Gaussian Mixture Model (GMM) at the end of semantic segmentation models. The EM component maps the representations produced by a fully-convolutional neural network to the class labels and their probabilities. The GMM is learned simultaneously with the neural network; its parameters are optimized with Sinkhorn EM to fit the features from a running window over multiple training iterations. The authors show that this performs well both on classical semantic segmentation benchmarks and on \"open world\" tasks where anomalous/out-of-distribution objects are introduced in the test set.", "questions": "- After training, and in inference are the EM parameters simply those from the last training iteration? This'll still be based on a smaller memory bank, how far off from \"ideal\" EM parameters could these be? The supplement says that a large enough memory bank represents the overall distribution, is this assertions based only on the fact that the performance has saturated (up to 3 digits) at 32K?\n\n### Minor comments/typos:\n\n  * 68: Forth -> Fourth\n  * 86: Suggest \"modify [the] FCN architecture\"\n  * 88: \"latter\" or \"later\"?\n  * 94: \"modeling [the] underlying data distribution\"\n  * 181: Is \"hard\" the right adjective here? Maybe the grammar would be better with \"...why it is hard for existing segmentation models to...\"\n", "limitations": "The limitations seem to be discussed reasonably well. The submission focuses on a problem that is well-described by benchmarks, and the it demonstrates that the method is likely to perform better than competitors on out-of-distribution or anomalous examples.\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n### i) Produces a generative, probabilistic model of the CNNs representations and the output classes.\n\nA key part of the segmentation model, the final mapping from neural representations to class probabilities and labels, is done with an interpretable and well-calibrated model. A lot of value comes from the model producing a probability distribution, for interpretability, reliability, calibration, etc. Some reasonable estimation of uncertainty is important when using neural networks within a larger system (e.g. in a commercial product) where one might need to handle uncertain predictions differently, as opposed to the neural network being used in isolation (e.g. to maximize scores on a benchmark). This work contributes significantly to literature focused on achieving good uncertainty estimates in neural-net-based-models [a,b,c], and this is only one aspect of the contributions.\n\nUsing a GMM in this way produces a well-calibrated uncertainty estimate. The authors allude to this with the 3rd limitation of softmax described in the introduction. Reliability diagrams are given in the supplement (Fig S1).\n\n### ii) Good choice of techniques for jointly training a generative model with a CNNs, well-validated by ablation studies.\n\nAdding new types of layers, including ones inspired by non-deep-learning models, is a common subject of a paper. This work does a more principled treatment of this than usual. First, the GMM model is learned with a choice of algorithm (Sinkhorn EM) that will produce consistent results in few iterations, which seems especially important when this model fit is repeated throughout the overall SGD training. Use of a memory bank is also a good improvement to this.\n\nThe value of Sinkhorn EM is validated by ablation study. The impact of the memory bank is also studied in the supplement.\n\n### iii) Good performance on anomaly detection.\n\nThe scores reported in the submission would not put it at the top of Fishyscapes leaderboard (https://fishyscapes.com/results). For example, as of writing the best AP score on the Fishyscapes lost & found is 63.43, vs 43.47 reported by the submission in Tables 2 & 3. However, this seems quote good for a method that isn't directly trained for anomaly detections. like the top published results in Fishyscapes. For instance, the current Fishyscapes leader [d] introduces synthetic anomalies when training a network. Instead, in the submission good anomaly detection can be extracted from a GMMSeg model trained on ordinary segmentation tasks.\n\nThis seem like a good demonstration of interpretability. As much as \"interpretability\" is an ambiguous term, the ability to infer new information from the model to use in other contexts seems like a useful example.\n\n## Weaknesses\n\n### iv) Less clear that it's an improvement over using CRF-based models in the same place, or why.\n\nThe authors contrast their against using softmax directly on the CNN features. They provide good reasons why GMMSeg would better model this problem. It seems like CRFs share a lot of the same advantages, and the contrast between GMMSeg and those methods is less obvious.\n\nThe original V1 of DeepLab [2], and several follow-up works [e,61] use CRFs to map the NN representation to class probabilities. These are cited in the related works, but the probabilistic modelling aspect of these works is not really discussed. Experimental comparisons against DeepLab are done with V3+ [47], a more recent version that dropped the CRF component.\n\nGMMSeg also lacks the \"context\" priors that CRFs give, due to the pairwise (or higher-order) potentials included in those models.\n\n### v) Not a fully combined optimization of EM and feature extractor.\n\nIn Section 3.2, the authors clarify:\n\n> the extractor’s parameters θ are only updated by the gradient back-propagated from the discriminative loss, while the GMM’s parameters are only optimized by EM\n\nI'd interpret this to mean that the feature extractor, which remains an important part of the model in GMMSeg, is still trained in a discriminative way that doesn't differ from previous methods. And the NN features are not optimized for input to the GMM component.\n\nGiven that this is the case, it's unclear why the EM needs to be learned at the same time as the feature extractor. Would there be any change to the training of the feature extraction NN if the GMM was simply ignored? And if so, could the GMM be learned *after* the NN is fully trained, without any change in results?\n\nThe authors make good design choices for an end-to-end joint optimization (iii), but stop short of doing so. However, empirical results do seem to show that the GMM still describes a better mapping from the NN features to the class labels than the original softmax that the NN was trained with.\n\n### vi) Only considers segmentation.\n\nAll of the authors' justifications for using a GMM to map representations to class labels seems like they'd apply equally well to classification, or any deep network trained with softmax. I wonder why the paper focuses on image segmentation exclusively, and it does make it a less widely-impactful work.\n\n### vii) Unclear presentation of feature extraction component.\n\nAdjacent pixels are usually important in segmentation, the feature in the classification problem is not really the $x_n \\in R^3$ as described in Section 3.1. The same is true for the formulation of the \"dense feature extractor.\" I'd emphasize that the feature extractor really maps $R^{H \\times W \\times 3}$ to $R^{H \\times W \\times D}$.\n\n[a] Oh et. al. \"Modeling uncertainty with hedged instance embedding\" ICLR 2019. \\\n[b] Caesar et. al. \"Joint Calibration for Semantic Segmentation\" BMVC 2015. \\\n[c] Nado et. al. \"Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning\" \\\n[d] Grcić et. al. \"Dense anomaly detection by robust learning on synthetic negative data\" \\\n[e] Vemulapalli et. al. \"Gaussian Conditional Random Field Network for Semantic Segmentation\" CVPR 2016.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XJBy0FOo9xl", "reviewer": "Reviewer_o8w9", "summary": "This paper studies semantic segmentation. The authors developed a Gaussian Mixture-based Generative Semantic Segmentation Model. The experimental results on serval datasets demonstrate the effectiveness of the proposed method.", "questions": "What's the effect of the proposed method on the model inference speed?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n+ Compared to the baselines, the proposed methods could bring constant improvements \n+ The abundant experiments are introduced to prove the effectiveness of the proposed method.\n\n[Weaknesses]\n- some important references are missing.\n[a] Top-down Learning for Structured Labeling with Convolutional Pseudoprior, ECCV 2016.\n[b] Exploring Cross-Image Pixel Contrast for Semantic Segmentation, ICCV 2021.\n\n- Both [a] and [b] introduce data distribution p(pixel feature|class), they all need to be included in the discussion. [b] also learn the class-related feature and bring constant improvements. It could be better to add it into comparison.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VgN_x5Jk193", "reviewer": "Reviewer_TvcJ", "summary": "This paper proposes GMMSeg, a generative model based on GMM for image segmentation that models the joint distribution of pixel features and classes. For each class, GMMSeg builds GMMs via EM so as to capture class-conditional densities. A deep network is trained end-to-end in a discriminative manner to extract features for the GMMs. This endows GMMSeg with the strengths of both generative and discriminative models. With a variety of segmentation architectures and backbones, GMMSeg outperforms the discriminative counterparts on three closed-set datasets, ADE20K, Cityscapes and COCO. Without any modification, GMMSeg even performs well on open-world datasets. ", "questions": "Please refer to the Strengths And Weaknesses section", "limitations": "The authors addressed the limitations and potential negative societal impact of their work", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. It is reasonable to use generative classification models with discriminative feature learning to combine the strengths of both generative and discriminative models. \n2. The properties of GMMs make GMMSeg well adapt to multimodal data densities and allows GMMSeg to naturally reject abnormal inputs.\n3. GMMSeg is fully compatible with different network architectures. \n4. Extensive experiments are conducted on multiple benchmark datasets. It shows that the proposed method improves the performances of multiple baselines. \n\nWeakness:\n1. The proposed generative framework is complex. The motivation of using GMMs on top of a deep network is not very clear. Why not directly model the joint distribution of image and label maps using deep networks?\n2. The meaning of the components of GMMs in the proposed framework is not very clear. It looks like all classes have the same number of components and every component is expected to have the same amount of pixels. However, the modality of different classes and the pixels of different components are not always the same. \n3. It would be more convincing if there are some discussion on what each components represent and a visualization of the learned components. \n4. Compared with the baseline, GMMSeg brings more overheads at both inference and training time. There should be a comparison in terms of training/inference speed. it will be more convincing if the proposed method is compared with baseline+CRF post processing, as both GMMSeg and CRF improve performance at the cost of computation overhead. \n5. It is oversimplified to model segmentation as pixel classification using GMMs as it does not handle the correlations among pixels in an image. It might be more reasonable to use MRF, which has been explored in previous work, e.g. [a] \n[a] Liu, Ziwei, et al. \"Deep learning markov random field for semantic segmentation.\" IEEE transactions on pattern analysis and machine intelligence 40.8 (2017): 1814-1828.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BZ_iHsk8wS1", "reviewer": "Reviewer_5AWr", "summary": "This paper presents a novel paradigm for semantic segmentation. Specifically, the authors propose to frame the segmentation task as a generative model by modeling the probability of p(x|c) as a mixture of Gaussians. This is rather different from the popular discriminative model that adopts Softmax.   ", "questions": "- what about using a different p(c) ? ", "limitations": "- the assumption of uniform distribution of p(c) is not very reasonable. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ interesting idea\n+ novel paradigm, as compared to a large number of semantic segmentation models. \n\n- the assumption of p(c) = 1/C might not be very reasonable. Normally, the prior distribution of different classes is not a uniform distribution. \n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept