Image Modeling with Deep Convolutional Gaussian Mixture Models
Authors: Alexander Gepperth
Benedikt Pfülb
Venue:  (2021)
Keywords: Gaussian Mixture Model
Deep Learning
Unsupervised Representation Learning
Sampling

Abstract: In this conceptual work, we present DCGMM, a deep hierarchical Gaussian Mixture Model (GMM) that is particularly suited for describing and generating images.
Vanilla (i.e., "flat") GMMs require a very large number of components to well describe images, leading to long training times and memory issues. 
DCGMMs avoid this by a stacked architecture of multiple GMM layers, linked by convolution and pooling operations. 
This allows to exploit the compositionality of images in a similar way as deep CNNs do.
This sets them apart from vanilla GMMs which are trained by EM, requiring a prior k-means initialization which is infeasible in a layered structure.
For generating sharp images with DCGMM, we introduce a new gradient-based technique for sampling through non-invertible operations like convolution and pooling.
Based on the MNIST and FashionMNIST datasets, we validate the DCGMM model by demonstrating its superiority over "flat" GMMs for clustering, sampling and outlier detection.
We additionally demonstrate the applicability of DCGMM to variant generation, in-painting and class-conditional sampling. 

Under review conference paper ICLR 2021
IMAGE MODELING WITH DEEP CONVOLUTIONAL GAUSSIAN MIXTURE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
In this conceptual work. we present Deep Convolutional Gaussian Mixture Models (DCGMMs), deep hierarchical Gaussian Mixture Model (GMM) that particu- larly suited for describing and generating images. Vanilla (i.c., flat) GMMs require a very large number of components to well describe images, leading to long train- ing times and memory issues. DCGMMs avoid this by stacked architecture of multiple GMM layers, linked by convolution and pooling operations. This allows exploit the compositionality of images in similar way deep CNNs do. DCG- MMs can be trained end-to-end by Stochastic Gradient Descent. This sets them apart from vanilla GMMs which are trained by Expectation Expectation-Maximizzttioo re- quiring prior k- memass initialization which is infeasible in layered structure. For generating sharp images with DCGMMs, introduce new gradient-hased tech- nique for sampling through non-invertible operations like convolution and pool- ing. Based on the MNIST and FashionMNIST datasets, we validate the DCGMM model by demonstrating its superiority flat GMMs for clustering, sampling and outlier detection.
INTRODUCTION
This conceptual work is in the context of probabilistic image modeling, whose main objectives density estimation and image generation (sampling). Since images usually do not precisely follow a Gaussian mixture distribution, such a treatment is inherently approximative in nature. This implies that clustering, even though it is possible and has long history in the context of Gaussian Mixture Models (GMMs), is not main objective. Sampling is an active research topic mainly relying on Generative Adverserial Networks (GANs) discussed in [Section 1.2] Similar techniques are being investigated for generating videos (Ghazvinian Zanjani et al. 2018 Piergiovanni & Ryoo 2019).
An issue with GANs is that their probabilistic interpretation remains unclear. This is outlined by the fact that there no pasy-to-compute probabilistic measure of the current fit-to-data that is optimized by GAN training. Recent evidence seems to indicate that GANs may not model the full image distribution given by training data (Richardson & Weiss 2018). Besides, images generated by GANs appear extremely realistic and diverse, and the GAN model has been adapted to perform a wide range of visually impressive functionalities.
In contrast, GMMs explicitly describe the distribution p(X), given by set of training data X [En) J. as a weighted mixture of Gaussian component densities A(x; Mk. Σ&) = (x) p(x) = ΣA (x). GMMs require the mixture weights be normalized: Σx ग. and the covariance matrices be positive definite: xx EE 0 Va.. The quality of the current fit-to-data is expressed by the logelikel
((X)
En log Σ (iN)
which is what GMM training optimizes, usually by variants of Mnxetitton-Maaimmzzatiin (EM) (Dempster et al. 1977). It can be shown that arbitrary distributions can, given enough components, be approximated by mixtures of Gaussians (Goodfellow et al. (016). Thus, GMMs are guaranteed to model the complete data distribution, but only to the extent allowed by the number of components K.

---

Under review conference paper ICLR 2021
this respect, GMMs are similar flat neural networks with single hidden layer: although, by the universal approximation theorem of Pinkus (1999) and Homiik et al. (1989), they can approximate arbitrary functions (from certain rather broad function classes), they fail To do so in practice. The reason for this that the number of required hidden layer elements is unknown, and usually beyond the reach of any reasonable computational capacity. For images, this problem was largely solved by introducing deep Convolutional Neural Networks (CNNs). CNNs model the statistical structure of images (hierarchical organization and translation invariance) by chaining multiple convolution and pooling layers. Thus the number of parameters without compromising accuracy can be reduced.
1.1 OBJECTIVE, CONTRIBUTION AND NOVELTY
The objectives of this article are to introduce a GMM architecture which exploits the same principles that led to the performance explosion of CNNs. In particular, the genuinely novel characteristies are:
formulation of GMMs as a deep hierarchy, including convolution and pooling layers,
end-to-end training by SGD from random initial conditions (no k- k-means initialization).
generation of realistic samples by new sharpening procedure.
better empirical performance than vanilla GMMs for sampling, clustering and outlier detection.
In addition, we provide publicly available TensorFlow implementation which supports a Keras-like flexible construction of Deep Convolutional Gaussian Mixture Models instances.
1.2 RELATED WORK
Generative Adverserial Networks The currently most widely used models of image modeling and generation are Generative Adverserial Networks (Arjovsky et al., 2017; Mirza Osindero 2014 Goodfellow et al., 2014). Variational Autoencoders (VAEs) follow the classic autoencoder principle (Kingma & Welling, 2013). trying reconstruct their inputs through a bottleneck layer, whose activities are additionally constrained to have Gaussian distribution. GANs are trained ad- versarially, mapping Gaussian noise to image instances, while trying to fool an additional discrim- inator network, which in turn aims to distinguish real from generated samples. They are capable of generating photo-realistic images (Richardson Weiss 2018), although their probabilistic inter- pretation remains unclear since they do not possess a differentiable loss function that is minimized by training. They may suffer from what is termed mode collapse, which is hard to detect automati- cally due to the absence a loss function. Due to their ability to generate realistic images, they are prominently used in models of continual learning (Shin el al. 2018).
Hierarchical GMMs Ghahramani & Hinton 1997) can be considered as hierarchical GMMs because they are formu- lated in terms of a dimen-iina latent-variable representation, which is mapped to higher- dimensional space. The use of MFAs for describing natural images is discussed in detail in Richard- son & Weiss (0188) showing that the MFA model alone, without further hierarchical structure, compares quite favorably to GANs when considering image generation. A straightforward hierar- chical extension of GMMs is presented Li et al. (2002) with the goal of unsupervised clustering: subsequent GMM, together with an adaptive
Mixture of Factor Analyzers (MFAs) models (McLachlan & Peel 2005;
responsibilities of one GMM are treated as inputs mechanism that determines the depth of the hierarchy. [Garcia et al.) (2010) present a comparable, more information- -ooommation--hooettii approach. A hierarchy of MFA layers with sampling in mind is pre- sented by Viroli & McLachlan (2019), where each layer is sampling values for the latent variables of the previous one, although transformations between layers are exclusively linear. Van Den Oord & Schrauwen (2014) and (Tang et al. 2012) pursue similar approach. All described approaches use (quite complex) ) extensions of the EM algorithm initialized by k-means for training hierarchical GMMs, except Richardson & Weiss (0118) use Stochastic Gradient Descent (SGD), although with a k-means initialization. None of these models consider convolutional or max-pooling operations which have been proven be important for modeling the statistical structure of images.
Convolutional GMMs The only work we could identify proposing to estimate hicrarchical con- volutional GMMs is Ghazvinian Zanjani et al. (2018), although the article described a hybrid model where CNN and GMM are combined.

---

Under review conference paper ICLR 2021
SGD and End-to-End GMM Training Training GMMs by SGD challenging due to local op- tima and the need to enforce model constraints, most notably the constraint of positive-defintte co- variance matrices. This has recently been discussed in Hosseiii & (ra20) although the proposed solution requires parameter initialization by k-means and introduces several new ayper-parameters and is, thus, unlikely to work as-is in a hierarchical structure. An SGD approach that achieves robust convergence even without k-means-based parameter initialization is presented by [epperth & Pîlbj (2020) Undesirable local optima caused random parameter initialization circumvented by an adaptive annealing strategy.
2 DATA
the evaluation we use the following image data sets:
MNIST (LeCun et al. 1998) is the common benchmark for computer vision systems and classifica- tion problems. It consists of 60 000 28 28 gray scale images of handwritten digits (0-9). MNSHionMNIST (Xiao like the MNIST dataset.
2017) consists of images of clothes 10 categories and is structured
Although these datasets are not particularly challenging for classification, their dimensionality of 784 is at least magnitude higher than datascts used for validating other hierarchical GMM ap- proaches in the literature.
3 DCGMM: MODEL OVERVIEW
The Deep Convolutional Gaussian Mixture Model is hierarchical model consisting of layers analogy CNNs Each layer with index L expects an input tensor (LL-1) E of di- mensions N, (LL-1), (iI1-1),((--1 and produces output tensor A 1 of dimensions N, H W ((.), Layers can have internal variables ((T) that are adapted during SGD training.
An DCGMM layer I, has two basic operating modes (see Figure for (density) estimation, an input tensor A (1.-1) from layer L - is transformed into an output tensor A (r), For sampling, the direction is reversed: each layer receives a control signal ((++1) from layer L+1 (same dimensions
A (1)), which is transformed into control signal (!*) to layer L-1 (same dimensions as A (I. (I.-1),
estimation
Input
Folding
GMM
Pooling
Folding
GMM
Filter: 5x5
K: 49
Kernel: 5x5
Filter: 2x2
K: 25
Stride: 1x]
Stride: 2x2
Stride: 1x]
32<32<3
28x28H75
28*28*49
14x14x49
13x13x196
13x13H25
sampling
Figure 1: Illustration of a sample DCGMM instance containing all four layer types, with exemplary dimensionalities and parameters for each layer.
3.1
LAYER TYPES
We define three layer types: Folding (F), Pooling (P) and convolutional GMM (G). Each layer implements distinct operations for each of the two modes, i.e., estimation and sampling.
FonsorFlow code available under httpt://tggggtt ottpa//iaaaiiiitt////////oeeoeaa r2021 22tp22222111mmmmmmmnmmm

---

Under review conference paper ICLR 2021
Folding Layer For density estimation, this layer performs a part of the well-known con- volution operation known from CNNs. Based on the filter sizes f& N² as well as the filter strides Ax), AY, all entries of the input tensor inside the range of the slid- ing filter window are dumped into the channel dimension of the output tensor. We thus obtain an output tensor of dimensions N, = 14 A.
W(() =1 w(b-1)--
AX
and C = =00 fx H8) whose entries are computed as A w- w=w// /xxx and c' ((h' ) + w. ( C(L-)) c'. When sampling, it per- forms the inverse mapping which is not a one-to-one correspondence: imput tensor elements which receive several contributions are simply averaged over all contributions.
nhwc = A 1L--1)
nh'w'd with h h=hl/ff
Pooling Layer For density estimation, pooling layers perform the same operations as standard (2)) and strides X A/H). When (max-)pooling layers in CNNs based on the kernel sizes ky =xx sampling, pooling layers perform a simple nearest-neighbor up-sampling by factor indicated by the kernel sizes and strides.
GMM Layer This layer type contains GMM components, each of which is associated with trainable parameters Tye f4g and Σ₁. h: = 1...K, representing the GMM weights, centroids and covariances. What makes GMM layers convolutional is that they do model single input vectors, but the channel content all positions This is analog to the way CNN layer models image content at all sliding window positions using the same filters. A GMM layer thus maps the input tensor A (Z--1) E N.8 12 P,w'e 1 c_ 1) A E ( gp(6-1), KK. each GMM component k: {O..... K) contributing the likelihood Ankark of having generated the channel content at position h, w (for sample n in the mini-batch). This likelihood is often referred to as responsibility and is computed as
of the input A A.a.b... using a shared set of parameters.
(1-1) b.:
Pulwok (A (1)--1) = (A pki Σx)
A (L)
Probook
(2)
nluck
Ee Puhone
For training the GMM layer, we optimize the GMM log- log-likelihood ((4) for each layer L:
Σ log Σ F ( (1--1))
(3)
(1))
han Jaa
H LL--))
Training is performed by SGD according the technique, and with the recommended parameters. presented by Gepperth & Pflb (220)). which uses a max-component approximation to ((4), In sam- pling mode, a control signal (( is produced by standard GMM sampling, performed separately for all positions h, w. GMM sampling position h, first selects component by drawing from a multinomial distribution. If the GMM layer the last layer a DCGMM instance, the multino- mial's parameters are the mixing weights т. for each position h, vc Otherwise, the control signal ((++1)
viow: received from layer L+1 1 is used, It is consistent to use the control signal for component se- lection in layer 4, since it sampled by layer /. 1, which was in turn trained the component responsibilities of layer L. The selected component (still at position w) then samples T(A) It often beneficial for sampling to restrict component selection to the S components with the highest control signal (top-S sampling).
3.2 ARCHITECTURE- LEVEL FUNCTIONALITIES
The DCGMM approach proposes several functionalities different architectural levels.

---

Under review conference paper ICLR 2021
3.2.1 END-TO-END TRAINING
To train DCGMM instance, we optimize ((L) for each GMM layer L by vanilla SGD using learning rates ((,) This is different from standard CNN classifier, where only single loss function minimized, usually cross-entropy loss computed from the last layer's outputs. Learning is not conducted layer-wise
end-to-end. Parameter initialization for GMM layers selects the initial values for the mixing weights as T K-1, centroid elements sampled from μal 0000.01...01 and diagonal covariances initialized to unit entries. To ensure good convergence, training is conducted in two phases, in the first one of which only centroids are adapted.
3.2.2 DENSITY ESTIMATION AND HIERARCHICAL OUTLIER DETECTION
Outlier detection requires the computation of long-term averages in all layers and positions, E, nhw and variances Var, (Chite) over the training set, preferably during later, stable part of training. Thus, for every layer and position h, w, inliers are characterized by
(L)
(L how
L har
= E, ov Var, (C
(4)
Larger values of imply less restrictive characterization of inliers.
Assuming that the topmost GMM layer is global (h= w= 1), Equation 4 reducess to a single con- dition that determines whether the sample, as whole, is an inlier. However, we can also localize inlier/outlier image parts by evaluating [Equation 4jin lower GMM layers.
3.2.3 SAMPLING AND SHARPENING
Sampling starts the highest layer, assumed to be an GMM layer, and propagates control signals downwards (see Figure 1 and Section 3.1), with control signal ((1) constituting the sampling result. Sampling suffers from information loss due to the not-invertible mappings effected by Pooling and Folding layers. To counteract this, each Folding layer / performs sharpening on the generated con- (T(())
trol signal ((), This involves computing L(L") (T (1)) of the next-highest GMM layer at level L > 4, T(L)
and performing G' gradient ascent steps nivere nhwe + (000 nlwce The reason for sharp- ening is that filters Folding layers usually overlap, and neighboring filter results are correlated. This correlation is captured by all higher GMM layers, and most prominently by the next- highest one. Therefore, modifying T(4) by gradient ascent will recover some of the information lost by pooling and folding. After sharpening, the tensor T(L) is passed as signal to 1-
EXPERIMENTS
We define various DCGMM instances (with 2 or 3 GMM layers) for evaluation, see Table 1 plus layee-layer DCGMM baseline which is nothing but vanilla GMM. A DCGMM instance is defined by the parameters its layers: Folding(fy, fx, Av, Ax), (Max))Pooling(ky: lix, Ay, Ax) and GMM(K). Unless stated otherwise, training is always conducted for 25 epochs, using the recommended parameters from (Gepperth & Pílb 2020). Sharpening is always performed for 000 iterations with a step size of 0.I.
4.1 SAMPLING, SPARSITY AND INTERPRETABILITY
We show that trained DCGMM parameters are sparse and have intuitive interpretation in terms of sampling. To this effect, we train DCGMM instance 21-a (see Table T). After training (see Section 3.2.Tp. centroids of layer (left of (Figure 2) are easily interpretable and reflect the patterns that can occur in any of the 2 x 2 input patches to layer G2 of size 20 x 20. The 36 6 x 6 centroids of G4 (right of Figure 2) express typical responsibility patterns computed from each of the 2 x 2 input patches to G2 and can observed to be very sparsely populated. Auother interpretation of G4
plot and interpret the centroids of the GMM layers 2 (G2) and 4 (G4). The
2 Advanced SGD strategies like RMSProp HHinonn al. 2012) Adam Kingma Ba 2015 seem in- compatible with GMM optimization.

---

Under review conference paper ICLR 2021
Table Configurations and parameters of different DCGMM architectures.
DD
layer
1/
21.-a
21.-b
2/-cc
21.dd
21-e
31-aa
31-b
F(28.28,1,1)
8(20.20.8.8)
F(7.77.77)
P07.7,333
F(28,28,1.1)
P(7.7.333
1(3.31.11)
1(28,281.1)
2
G(25)
G(25)
G(25)
G(25)
G(25)
G(25)
G(25)
G(25)
3
F(2.2.1.1)
1(4,4.1.1)
F(8.8.1,1)
F(1.1.111)
F(8,8.1.1)
P(2.2)
F(1,1.111
4
G(36)
G(36)
G(36)
G(36)
G(36)
1(4,4,1,1)
G(25)
5
G(25)
F
6
P(2,2)
G(25)
7
1(6,6,1,1)
8
G(49)
comment
vanilla GMM
cony.
conv.
conv.
conv.
cony.
n0
layer
layer
layer
convolutions
layer
layers
convolutions
centroids can be found in of sampling (see Section 3.2.3 which would first select a random G4 component to generate a sample of dimensions H, W, C' 1, 2 x x from it, and pass it on as a control signal to G2 Traversing Folding layer 3 only reshapes the control signal to dimension H, W, C
5 5, depicted in the middle of Figure This signal controls component selection each the x 2 positions in G2: due to their sparsity, we can directly read off the components likely to be selected for sampling at each position. G2 thus generates control signal whose x positions of dimensions H. W, C 20, 20, overlap in the input plane (this is resolved by sharpening Folding layer 1). In this case, it is easy to see that sampling produces a particular representation of the digit zero.
O
1.0
3,4
0.1
3
3
3.2
1,0
/
3
Figure 2: Sampling from DCGMM instance 2L-a, see Table Shown are learned GMM centroids (left: G2, right: G4, see text) and an illustration of sampling. having initially selected the layer 4 component highlighted in red. In the middle, the selected G4 centroid is shown in more detail.
4.2 OUTLIER DETECTION
For outlier detection, we compare DCGMM architectures from Table the highest layer a criterion as detailed in (Section 3.2.2] We first train DCGMM instance classes 0-4, and subsequently use trained classes inlier- class 5-9 outlier-dtection. We vary e in the range [-2, 2, 2), resulting in different outlier and inlier percentages. Figure shows the ROC-like curves thus clearly indicate that the deep convolutional DCGMM instances perform best, whereas deep but non-convolutional instances like 2L-d and 3L-b consistently perform badly.
using the log-likelihood of
4.3 CLUSTERING
We compare DCGMMs to vanilla GMMs using established clustering metrics, namely the Dunn index (Dunn 1973) and the -ouidin score (Davies & Bouldin 1979). The DCGMM in- stances from Table
tested both image datasets. We observe that mainly the deep but non- convolutional DCGMM instances perform well clustering, whereas convolutional instances, even if they are deep. are compromised. Please note that these metrics do not measure the classification accuracy obtained clustering but intrinsic clustering-related properties.

---

Under review conference paper ICLR 2021
2Ld
100
1-1L
- LL-
outlees loopt (1)
+- 21-b a 3-3 75
=-2 &2
75
2.21-n
4-22.b
2-21
+ 3L-a
oustiere loopt
+ 21.0
31.h
50
00
23
inliers kegt (%)
inliers keps
Figure 3: Visualization of different DCGMM architectures and its outlier detection capabilities for MNIST (left) and FashionMNIST (right).
Table 2:
Two metrics, Dunn index (higher is better) and Davies-Bouldin (DB) score (smaller is better), evaluated for all tested DCGMM architectures MNIST and FaShi Best results are marked in bold. The given numbers are worst cases over independent runs.
DCGMM
Dataset
1L 2L-a 2L-b 2 2L-d 2L-c 3L-a 3L-b
Metric
Dunn index
0.14 0.14 0.13 0.12
0.19
0.15
0.15
MNIST
DB score
2.59 2.73 3.06
2.62
2.57
2.65
2.53
Fashion-
Dunn index
0.14 0.15
0.16
0.15
0.11
0.11
0.096
0.13
MNIST
DB score
2.37 2.77 2.62
2.7 2.40
2.92
3.2
2.35
4.4 SAMPLING AND SHARPENING
The results presented here were obtained by training on classes 0-4 of both datasets, and have to be confirmed by visual inspection of generated samples.
To assess this, compare the samples generated by the non-convolutional DCGMM instances L, 2L-d to the samples generated by the convolu- tional instances 2L-d, 2 for top-1-sampling The results of Figuree 4 clearly indicate that con- volutional DCGMMs generate diverse samples, whereas non-convolutional ones basically replicate their centroids with added white noise. FashionMNIST results are given Appendix A.2
Effects of Convolution Sampling Diversity
2 444333
3 3000300 03
4 4 4
2 3
0 0
O
0 134000
24 3/333333
2 00
0
V
03 03400 2
3 11/3302 302
2 2230030
4
2
4 400 1
133230/0
0000 0000310
13 3
302 3 2.2.3
0/0232300
4031
0
3 2 4 /./.0
4/103035
1
0 0/4000
0 040
0/100015
200 2
2
3 4 1400300 033
O3 342
0 3 0 0 0
3 3
1
3
Figure 4: Sampling diversity: top-1 sampling shown MNIST, from left to right, for DCGMM
architectures LT (vanilla GMM), 2 (non-convolutional 2-layer), 2 and 2 (convolutional 2-layer). Please observe duplicated samples in the non-convolutional architectures, marked in red.
Generating Sharp Images Figure shows effect sharpening DCGMM instance 2L-c using top-l-sampling. We can observe that the overall shape of a sample is not changed but that the outlines are crisper, effect visible especially for FashionMNIST. Thus, sharpening does no harm and rather improves the visual quality of generated samples. See Appendix A.3 for FashionMNIST results.
Controlling Diversity by SSmpligg Using instance 2L-c, Figure F demonstrates how sam- ple diversity is related to S: a higher value yields more diverse samples, but increases the risk gen- erating corrupted samples or outliers. As the corresponding FashionMNIST results in Appendix A.I show, good value of is clearly problem-dependent.

---

Under review conference paper ICLR 2021
4
4
4
/
4
4
2
4 2
/
2
0
0
2
4
2
/
I
2
0
0
2
2
3
0
0
3
2
O
0
0
0
0
4
0
3
4 0
3
4
4
1
4 2 2
///0/
4
2
4
2 2
/
4
2
4 2
2
0
0
2
2
O
2
O
0
2
0
0
3
3
0
0
0 3
3
0
0
0
Figure 5: Impact of sharpening on top-S- with S=1, see Section 3.2.3, shown for DCGMM instance 2L-c MNIST. Shown unsharpened samples (left), sharpened samples (middle) and differences (right). Samples at the same position were generated by the same top-level prototype.
5 SUMMARY, DISCUSSION AND CONCLUSION
The Objective the article was establish the conceptual foundations of deep GMM hierarchies (see also Appendix A) that leverage important mechanisms from the CNN domain. Convolution and pooling layers make it possible to apply the model to high-dimensional image data with off-the-shelf hardware (typical training runs take approximately minutes GeForce GTX 1080).
Important results show the illustration of important functionalities such as outlier detection, clus- tering and sampling, which no other work on hicrarchical GMMs can present related to such high- dimensional image datasets. We also propose method to generated sharp images with GMMs, which has been a problem perimental results is that non-convolutional DCGMMs seem to perform better at clustering, whereas convolutional ones better at outlier detection and sampling.
past (Richardson Weiss, 2018). interesting facet of our ex-
Key point of the article is the compositionality in natural images. This property is at the root of DCGMM's ability to produce realistic samples with relatively few parameters. When considering top-S- s-aplia in layer with H (L)) p(L) positions, the number of distinct control signals generated layer L Spi)) DCGMM instance with multiple GMM layers {L.} can thus sample IIr SP different patterns, which grows with the depth of the hierarchy and the number of distinct positions in layer, making strong argument in favor of deep convolutional hierarchies such as DCGMM. This is argument similar to the about different paths through hierarchical MFA model in (Viroli & McLachlan 2019). although their number grows more strongly for DCGMM.
Differences to other hierarchical models such as (Viroli & McLachlan 2019 Van Den Oord & Schrauwen 2014 Tang et al. 2012) are most notably the introduction of convolution and pooling layers. Our experimental validation can therefore be performed images, with moderate computational cost, instead of fow-dimensional problems such as the artifi- cial Smiley task the Ecoli and related problems. Our experimental validation does not exclusively focus on clustering performance (problematic with images) but on demonstrating the capacity for realistic sampling and outlier detection. Lastly, training DCGMMs by SGD facilitates efficient par- allelizable implementations, demonstrated with the provided TensorFlow implementation.
sigh-dimensiona data, such as
Next steps will consist of exploring the layered DCGMM architecture, mainly top-S- lop-S-sampling, for generating natural images.

---

Under review conference paper ICLR 2021
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. 2017. URL kttp:
//axxiv.ar//abb 007855
David I.. Davies and Donald Bouldin. Cluster Separation Measure. IEFE Transactions on
Pattern Analysis and Machine Intelligence, PAMI-1(2)224 227, 1979. ISSN 01628828. doi:
10.1109/TPAM. 19w.M 1979.4766909.
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum Likelihood from Incomplete Data Via the EM Algorithm. Journal of the Royal Statistical Society: Series B (Methodological,, 39(1):1-22,
1977. doi: 111111//5577666111777.60000
C. Dunn. fuzzy relative of the ISODATA process and its use in detecting compact separated clusters. Journal of Cybernetics, 3(3)322-57, 1973. ISSN 00220280. doi: 10. 1080/ 01969727305546046.
Vincent Garcia. Frank Nielsen, and Richard Nock. Hierarchical Gaussian mixture model. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 4070-4073. IEER, 2010. ISBN 978- -478--4-4444495-
doi: 10.1 10.1000CCASSPP220000500777
URL http://pp.aaaaaaaaoooooooo......
Alexander Gepperth and Benedikt Pflb. Gradient-based training of gaussian mixture models in high-dimensional spaces, 2020.
Zoubin Ghahramani and Geoffrey Hinton. The EM Algorithm for Mixtures of Factor Analyzers. Compute, pp. 1-8, 1997. ISSN 0003-0023, doi: 173068.
Farhad Ghazvinian Zanjani, Svitlana Zinger, and Peter H. N. de With. Deep convolutional gaus- sian mixture model for stain-color normalization of histopathological images. In Alejandro Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola-Lóóp López, and Gabor Fichtinger (eds.), Medical Image Computing and Computer Assisted Intervention MICCAI 2018, pp. 274- 282, Cham, 2018. Springer International Publishing. ISBN 978-3-033--0033--..
Farhad Ghazvinian Zanjani, Svitlana Zinger, and Peter H. N. de With. Deep Con- volutional Gaussian Mixture Model for Cain- Normalization of Histopatholog- Miccai, volume I, pp. 274-282. Springer International Publish-
ical Images.
ing. 2018. ISBN 9783030009342. doi: 10. (0007997-3-00000092--00
URL
http://dx doi /ttp:///.o.....000..000000011-00
21th444/22222tttt////a/ 122http://link.
spr springess www...grr... wwtgger.co//000 1007/ 000-000334-20111
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning, MIT Press, 2016. http:
/ //www. www..eeeppleannnngooc.oeooo
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Orair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances Neu- ral Information Processing Systems, 3(anuar)::672-26690 2014. ISSN 10495258. URL
http:://axx.v.orr//a//////
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Lecture 6.5 RmsProp: Divide the gradient by rumning average of its recent magnitude, 2012.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are
universal approximators. Neural Networks, 2(5):359-366, 1989. ISSN 08936080. doi: 10. 0011016/00933-60008889990022
Reshad Hosseini and Suvrit Sra. An alternative EM for Gaussian mixture models: batch and stochastic Riemannian optimization. Mathematical Programming, 181(1):17-223,, 2020. ISSN 14364646. doi: 10. 1011007//10007:099-11388/// URL https::::///// :ttp:s://ddo..o org 1007/ 100007-00 01381-4

---

Under review conference paper ICLR 2021
Diederik Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. 3rd Interna- lional Conference on Learning Representations, ICLR 2015 Conference Track Proceedings, pp. 1-15, 2015.
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding
variational
bayes. arXiv preprint
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2323, 1998. ISSN 00189219. doi: 10.1 109/5.72679.
feee wtt....rg//ccccceettt//mm/ /227oc
Ming Liu, Eric Chang, and Bei Qian Dai. Hierarchcial Gaussian mixture model for speaker verifica- tion. 7th International Conference on Spoken Language Processing, ICSLP 2002, pp. 1353-1356, 2002.
Geoffrey McLachlan and David Peel. Mixtures of Factor Analyzers. pp. 238-256, jan 2005. doi: 10. 1002/047172 1721 182.ch8. URL http:///ww..llley.oooo.......
ch8
Mehdi Mirza and Simon Osindero. Conditional Generative Adversarial Nets. pp- 1-7, 2014. URL
http://arxiv org/ abs/ /141l.111.... 11411111880 1784
Aj Piergiovanni and Michael Ryoo. Temporal Gaussian mixture layer for videos. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5152-5161, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL.tttp://rroceddii
press/ v97 http://////eeeiiaarrr..mmmm 9a .tml
Allan Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica, 8:
143-195, 1999. ISSN 14740508. doi: 10. (L.0017//006622229000999
Eitan Richardson and Yair Weiss. On GANs and GMMs. Advances in Neural Information Process- ing Systems. 2018-Decem December(NeutfPS:555777 5858, 2018. ISSN 10495258.
Hoo Chang Shin, Neil A. Tenenholtz, Jameson K. Rogers, Christopher G. Schwarz, Matthew L. Senjem, Jeffrey Gunter, Katherine P. Andriole, and Mark Michalski. Medical image synthesis for data augmentation and anonymization using generative adversarial networks. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes Bioinformatics). 11037 LNCS:1 2018. ISSN 16113349. doi: 00001/99788300-00555553 1.
Yichuan Tang, Ruslan Salakhutdinov, and Geoffrey Hinton. Deep mixtures of factor analysers. Proceedings of the 29th International Conference on Machine Learning, ICML 2012, 1:505-512. 2012.
Aäron Van Den Oord and Benjamin Schrauwen. Factoring variations in natural images with deep Gaussian mixture models. Advances in Neural Information Processing Systems, 4(January):3518 3526, 2014. ISSN 10495258.
Cinzia Viroli and Geoffrey J. McLachlan. Deep Gaussian mixture models. Statistics and Computing, 29(1):43-,,, 2019. ISSN 15731375. doi: 10.11077s11222-017-999
Han Xiao, Kashif Rasul, and Roland Vollgraf. Rashion-MNISTI a Novel Image Dataset for Bench- marking Machine Learning Algorithms. pp. 1-6, 2017. URL http:: org/bbs/
abs/
1 7088. 07747
10

---

Under review conference paper ICLR 2021
A PROBABILISTIC INTERPRETATION OF DCGMMS
probabilistic interpretation of the DCGMM model possible despite its complex structure. The simple reason is that DCGMM instances produce outputs which are inherently normalizable, mean- ing that the integral over infinite domain (e.g., data space) remains finite. Thus, DCGMM outputs can be interpreted as a probability which not the case for DNN/CNNs due to the use of scalar prod- ucts.
Here, we prove that GMMs are normalizable in the sense that the integral of the log- log-probability [(x) log Ex TT)px() finite. This holds for any GMM layer hierarchy regardless of its imput, provided that the input is finite (which is assured because Pooling and Folding layers cannot introduce infinities). For simplicity. we integrate over the whole d-dimensional space R Since the component probabilities are Gaussian and thus strictly positive, and since furthermore the mixing weights are normalized and 2 0, the sum is strictly positive. Thus, is sufficient to show that the integral over the inner sum (the argument of the logarithm) is finite. We thus have
Rd Σ (
Σ TK Px(a))
Σ mh det (.>>2)
(5)
which trivially finite because Gaussians are normalized.
A.I TOP-S SAMPLING RESULTS FOR FASHIONMNIST
Figure 6: Impact of higher values of in top-S sampling, shown for DCGMM instance 21-c. From
left right: 2=,,5,10.
A.2 EFFECTS OF CONVOLUTION ON SAMPLING DIVERSITY FOR FASHIONMNIST
Convolutional architecture is helpful for sampling: top-1 sampling shown on Fashion- MNIST, from left to right, for DCGMM architectures 1L (vanilla GMM), 2L-d (non-convolutional 2-layer DCGMM), 2L-c and 2L-e (both convolutional 2-layer DCGMMs). Please observe dupli- cated samples in the non-convolutionall architectures.
Figure 7:

---

Under review conference paper ICLR 2021
A.3 EFFECTS OF SHARPENING ON FASHIONMNIST
Figure 8: Impact of sharpening on tsa-S-snnpling with S=1, see Section 3.2.3 shown for DCGMM instance 21-c on FashionMNIST. Shown are unsharpened samples (left), sharpened samples (mid- dle) and differences (right). Samples at the same position were generated by the same top-level prototype and are thus directly comparable.
12


Reviews:
[{"review_id": "h_utj1jmjWz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------------\nThis paper defines encoding and decoding procedures which use transformations inspired by Gaussian mixture models (GMMs). The decoding procedure further involves \"sharpening\" steps. A heuristic for training the parameters shared by the encoder and decoder is proposed which optimizes the likelihoods of GMMs defined on various outputs of the encoder. The decoder is evaluated in terms of clustering performance, sample quality, and outlier detection.\n\nQuality (1/5)\n-----------------\nIt seems like a stretch to call the proposed model a \"deep Gaussian mixture model\" or a probabilistic model at all. For a probabilistic model we should be able to assign a probability to any (measurable) set of the input space. (For some models such as GANs this probability is intractable but it is still easily defined.) However, it is not clear from the model's description (Section 3) what this probability should be. This lack of a well defined density (or measure) is surprising given the authors' emphasis on \"density estimation\" as a \"main objective\" of probabilistic image modeling and a potential application of their model.\n\nOne could view the entire decoding process as a complicated generative model which involves an iterative sharpening procedure, but this is not how the model is presented. In particular, the training procedure does not seem to be optimizing any divergence of this model and it is not clear how the encoder relates to the posterior of this model.\n\nThe output of the GMM layer (\"responsibilities\") live on a simplex (Eq. 2). If we stack two GMM layers, doesn't the likelihood of the second GMM explode (since the differential entropy of the inputs is negative infinity)?  Is suspect the reason that the training loss doesn't explode may be an artefact of SGD and/or the pooling layers.\n\nClarity (3/5)\n----------------\nI appreciated that the encoding and decoding procedure as well as the training objective were clearly described.\n\nOn the other hand, already in the abstract and the first two paragraphs the authors make confusing claims such as the following:\n\n(1) The authors claim in the abstract that \"DCGMMs can be trained end-to-end by SGD\" and that this \"sets them apart from vanilla GMMs which are trained by EM, requiring a prior k-means initialization\". But vanilla GMMs may very well be trained with SGD as the authors note themselves in the related work section. While k-means may speed up training, it is not \"required\" by GMMs.\n\n(2) The authors claim that since \"images usually do not precisely follow a GMM distribution [...] clustering is not a main objective [of image modeling].\" This seems like a non-sequitur.\n\n(3) \"An issue with GANs is that their probabilistic interpretation remains unclear. This is outlined by the fact that there is no easy-to-compute probabilistic measure of the current fit-to-data that is optimized by GAN training.\" I would argue that the divergence(s) (approximately) optimized by GANs as well as their probabilistic interpretation are much better understood than the proposed model, as discussed above.\n\nThe authors point out that \"training GMMs by SGD is challenging\" because the covariance matrices are constraint to be positive definite. Isn't reparametrization relatively easy (C = AA')? And don't you have the same issue in your model (Eq. 2)? How do you enforce positive definiteness in your model?\n\nOriginality (2/5)\n---------------------\nI would have expected a more thorough comparison with deep GMMs (van den Oord & Schrauwen, 2014) which appears to be the most closely related model.\n\nAnother line of research not being discussed are autoregressive Gaussian mixture models (e.g., Domke, 2008; Hosseini et al., 2011; Theis et al., 2012). These models generalize Gaussian mixture models and are able to efficiently model images of arbitrary size by (like convolutions) making a stationarity assumption. Deep extensions exist as well (Theis & Bethge, 2015).\n\nSignificance (1/5)\n-----------------------\nA lack of conceptual insights or a principled motivation would be fine if the empirical results made up for it. Unfortunately the empirical evaluation seems rather limited as well. No comparisons were made to previously published baselines. Instead, all results are only compared to other DCGMM results provided by the authors.\n\nThe chosen tasks and datasets (MNIST, FashionMNIST) are rather limited as well. In particular, unconditional image generation is not a well defined task (and certainly not a \"main objective\" of image modeling) but rather a (poor) proxy for evaluating generative models (see Theis et al., 2016).", "extra_scores": {}, "extra_text": {}}, {"review_id": "FjCNg1L6Tr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this manuscript the authors present a variant of stacked Gaussian mixture models they propose for modeling images called Deep Convolutional Gaussian Mixture Model. This model may contain analogues of convolutional layers and nonlinearities between the stacked Gaussian mixture models. This model can then be trained using stochastic gradient decent on the gradients propagated through the model. Finally the authors show some experimental evaluation on FashionMNIST and MNIST.\n\nOverall I vote for rejection. While the model the authors present seems to work in principle on images I do not think the authors present a good argument why their model should be used for modeling images and there are definitely other models the authors should compare their model to. Also I have doubts whether the model as presented is a proper probabilistic model.\n\nPros:\n1) This is a new Gaussian mixture based model.\n2) It is stackable and inherits some of the benefits of DNNs.\n3) It is trainable with (stochastic) gradient descent\n\nCons:\n1) I disagree with the authors in the introduction. While GANs as they discuss are not fully probabilistic models and thus have limitations to their applicability, other models do have very clear probabilistic interpretations and apply to all the tasks discussed here. Examples of such networks are the numerous variations of the variational autoencoder, the invertable network based variations like FLOW or GLOW and diverse others. As these are ignored by the authors, I don’t think they place their work well into the literature and do not see a particularly strong argument here that gaussian mixture models would be a great addition to the modeling of images.\n\n2) The authors do not compare their method against any competing methods outside the Gaussian mixture model framework. I think they would have to present some comparisons to state of the art methods for the tasks they test their model on. At very least compare to some basic models which generate some intuition where Gaussian mixture models overall lie in terms of performance. Without that I am completely lost whether the performance in these tasks is any good.\n\n3) If the authors instead want to focus on the conceptual level or advancing our understanding of the presented kind of model I still think a lot more could and should be done: For example, the authors decide to perform outlier detection based solely on the last GMM layer. While this might be somewhat sensible for the decision between different numbers or categories in MNIST, in general, this is not the probability of the observed data under the model. Why is this used? Similarly: What is the structure of the representations produced by the model?\n\n4) Given that the main claimed advantage of the model is its probabilistic interpretation, I find the probabilistic description and analysis of the model somewhat lacking:\n    - How exactly is the probability of a given sample computed in this model?\n    - How can convolution and pooling layers be interpreted as probabilistically given that they are not invertible. It seems to me, that for both types of layers, the input may even have zero probability to be produced by the described sampling processes. I.e. either some inputs have 0 probability under the model or the sampling methods do not actually sample from the model.\n    - just ignoring pooling and convolution steps in the calculation of the probabilities under the model as I guess the authors do here seems wrong.\n\n5) Each Gaussian mixture model layer in the proposed model converts a continuous input space into  a probability distribution over Group assignments. These assignment probabilities are then linearly mapped and pooled before another Gaussian mixture model layer again interprets the input as point in a R^n to be modeled by a mixture of Gaussians. This is technically possible to some degree if we ignore the restrictions on the support to achieve a valid distribution, but I do not get the intuition how this may well represent the composition of an image. It is stated as fact that the presented model is good for that, but I think this part requires some justification. Wouldn’t we expect operations which further work on distributions over discrete spaces instead?", "extra_scores": {}, "extra_text": {}}, {"review_id": "ErBHuHwEaE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThe authors propose a convolutional neural network architecture where some layers realize a Gaussian mixture model (GMM) that performs density estimation over the embeddings of the previous layers it receives as inputs. \nThese embeddings collect probability values. \nAs such, their whole architecture, DCGMM, can be viewed as stacking density estimators, each of which is fit by maximum likelihood over the output embeddings of the preceding ones, i.e. it estimates a density on a series of latent spaces.\nAs such, when applied to images, DCGMM does not provide an explicit likelihood over the pixel space -- or better only its first layer, comprising a shallow GMM, does.\nThe authors use DCGMM as a simulator, i.e., to sample images and devise a sharpening scheme that tries to work around the non-invertibility of pooling layers (but also of the GMM layer?)\n\n## Presentation\nThe paper is overall well-written and readable.\nThe biggest concern I have regarding presentation is that the proposed DCGMM is not a density estimator per se nor it is used for desity estimation in the experiments, while it is presented as such.\nI would suggest authors a rewriting that includes a deep discussion that also properly contrasts DCGMM with other hierarchical models listed in Section 1.2. It seems to me that the latter define a proper density $p(X)$ over the observables $X$ while DCGMM leaves the task to the shallow GMM of its first layer.\nIn particular, the way to evaluate (and then train) the model, can be named 'forward evaluation' mode more than density estimation.\n\n## Contributions\nThe main contribution, i.e., to perform stacked density estimation, seems novel to me.\nHowever, I would advice authors to directly present it in this way and not as a normal density estimator over oversables.\nThis implies properly contrasting DCGMM w.r.t. the existing literature (see also comments above).\nDeep (Gaussian) mixture models have been research in the literature [1,2,3]\nDIfferently from DCGMM, they retain an explicit and tractable likelihoods (but also marginalization and conditioning) as well as the ability to exactly sampling from them.\nThese deep mixtures also show how it is possible to train a discrete latent variable model with thousands of latent variables effectively with EM [4]\n\nI feel that the proposed sampling scheme needs more space for discussion and clarifications.\nFirst, it is not clear to me if the proposed sampling scheme is consistently and correctly sampling from the space $X, Z_1, Z_2, ..., Z_D$ where the latter are the $D$ latent variable spaces associated to the GMM layers in DCGMM.\nSecond, I wonder if the GMM layer is invertible per se: two different incoming inputs can get associated the same likelihood by a single Gaussian due to the symmetry of its density. The same might happen when the GMM retains some symmetries as well.\n\nConcerning the usage of DCGMM as an outlier detector, it is not clear why only the likelihoods on the top-most latent space ($Z_D$) are employed. I wonder what happens when all the layers are utilized (singularly or collectively).\n\n\n## Experiments\nThe experiments focus on using DCGMM as a simulator and as an outlier detector (or clustering) for MNIST and FashionMNIST.\nThese count as interestnig preliminary results but go against the original motivation that DCGMM overcomes the limitations of other deep mixture models that are not able to scale to larger datasets.\nFurthermore, it would be insightful to compare DCGMM -- even on MNIST and FashionMNIST alone -- against the deep mixture models referenced in Section 1.2. \n\nFor the sampling experiment, the effect of sharpening and/ top-S sampling could be measured in a more rigorous way.\nFor instance, the quality of samples can be assesed via FID scores or any other analogous metric from the GAN/VAE literature.\nMoreover,  are 'duplicates' -- possibly signaling mode collapse -- exact replicas or slight variations? This can be further inspected by reporting for each generated sample its top-k nearest neighbor samples in the training set and some metric (even in pixel-space) of divergence between them.\n\nWhy are authors training only on classes \"0-4\"? Is there an issue in scalability?\n\n\n## References\n\n[1] Sharir, Or, et al. \"Tensorial mixture models.\" AISTATS (2018).\n[2] Jaini, Priyank, Pascal Poupart, and Yaoliang Yu. \"Deep homogeneous mixture models: representation, separation, and approximation.\" Advances in Neural Information Processing Systems. 2018.\n[3] Butz, Cory J., et al. \"Deep convolutional sum-product networks.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.\n[4] Peharz, Robert, et al. \"Einsum Networks: Fast and Scalable Learning of Tractable Probabilistic Circuits.\" ICML (2020).", "extra_scores": {}, "extra_text": {}}, {"review_id": "A8gO9I_VYPj", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "==== Summary ====\n\nThe paper proposes a model that combines hierarchical Gaussian Mixture Models with a convolutional architecture, supporting both estimation and sampling. The model is trained end-to-end via SGD and is composed of 3 types of layers: standard convolutional and max-pooling layers and a newly proposed GMM layer. The latter operates by modeling the stack of channels at each spatial location as vectors sampled from a GMM. The outputs of the layer are the component probabilities at each location, followed by channel-wise normalization. The loss function is the average log-likelihood of every location at every GMM layer. The paper argues for using it as an alternative to other, less interpretable, probabilistic models of images and demonstrates its capacity to model the MNIST and FashionMNIST datasets.\n\n==== Detailed Review ====\n\nMain strengths:\n* A novel architecture inspired by both ConvNets and hierarchical GMMs, allowing for more interpretable representation of images.\n* Demonstrates that the model can handle simple image datasets and provides the code to reproduce the results.\n\nMain weaknesses:\n* There are no obvious theoretical advantages over other probabilistic models.\n* Experiments do not compare to other methods beyond GMM, so it is hard to determine if there are any significant practical benefits. Additionally, the experiments are limited to only MNIST and FashionMNIST, which raises the question of whether this method is applicable to more complex datasets.\n* The model does not represent a proper probability distribution. There is also a mismatch between the training objective and the sampling process.\n* Missing references to other relevant probabilistic models of images that combine ConvNets and GMM.\n\nI do not recommend acceptance due to the lack of theoretical or practical benefit of the proposed method and the lack of appropriate comparisons to prior approaches. In more detail:\n1. The paper does not argue for any advantage to the proposed method over the alternatives beyond a general claim of interpretability. The experiments merely demonstrate that the method can model very simple image datasets and has a basic ability to detect outliers. Many models can accomplish the same, and yet they are not compared. The authors should explain why someone would prefer using this model over the alternatives (GAN, VAE, autoregressive models like PixelCNN, or even proper hierarchical graphical models).\n2. The model itself is not a proper distribution, as opposed to GAN, VAE, and autoregressive models, which do represent distributions. There is a lack of theoretical justification for the proposed loss function and its connection to the generative process (I believe you might be able to show your objective is a lower bound on the true log-likelihood). Regardless, if the only measure of success is image representation, then there are other non-probabilistic methods the authors could have compared to (e.g., plain AE or Generative Latent Optimization). \n3. It is claimed to be the first method to combine ConvNets and GMMs in an end-to-end manner, but prior works have already done this, though using different constructions. Specifically, Sum-Product Networks with Gaussian leaves [1,2,3,4] have been trained with convolutional architectures [2, 4] and SGD end-to-end [2, 3, 4] on several image datasets, including MNIST, FashionMNISt. These models are proper distributions, equally interpretable, and their samples are comparable to those produced in this paper. They also do not scale well to more complex image datasets (to the best of my knowledge), which is why it is so essential to show experiments on other datasets beyond these basic ones.\n\n[1] Sum-Product Networks: A New Deep Architecture. Pool et al., 2012.\n[2] Tensorial Mixture Models. Sharir et al., 2016.\n[3] Deep Convolutional Sum-Product Networks. Butz et al., 2019.\n[4] Random Sum-Product Networks: A Simple and Effective Approach to Probabilistic Deep Learning. Peharz et al., 2020.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a clear reject. None of the reviewers supports publication of this work. The concerns of the reviewers are largely valid.
Decision: Reject