Image Modeling with Deep Convolutional Gaussian Mixture Models

1 INTRODUCTION . This conceptual work is in the context of probabilistic image modeling , whose main objectives are density estimation and image generation ( sampling ) . Since images usually do not precisely follow a Gaussian mixture distribution , such a treatment is inherently approximative in nature . This implies that clustering , even though it is possible and has a long history in the context of Gaussian Mixture Models ( GMMs ) , is not a main objective . Sampling is an active research topic mainly relying on Generative Adverserial Networks ( GANs ) discussed in Section 1.2 . Similar techniques are being investigated for generating videos ( Ghazvinian Zanjani et al. , 2018 ; Piergiovanni & Ryoo , 2019 ) . An issue with GANs is that their probabilistic interpretation remains unclear . This is outlined by the fact that there is no easy-to-compute probabilistic measure of the current fit-to-data that is optimized by GAN training . Recent evidence seems to indicate that GANs may not model the full image distribution as given by training data ( Richardson & Weiss , 2018 ) . Besides , images generated by GANs appear extremely realistic and diverse , and the GAN model has been adapted to perform a wide range of visually impressive functionalities . In contrast , GMMs explicitly describe the distribution p ( X ) , given by a set of training data X = { xn } , as a weighted mixture of K Gaussian component densities N ( x ; µk , Σk ) ≡Nk ( x ) : p ( x ) = ∑K k πkNk ( x ) . GMMs require the mixture weights to be normalized : ∑ k πk = 1 and the covariance matrices to be positive definite : xTΣkx > 0 ∀x . The quality of the current fit-to-data is expressed by the log-likelihood L ( X ) = En [ log ∑ k πkNk ( xn ) ] , ( 1 ) which is what GMM training optimizes , usually by variants of Expectation-Maximization ( EM ) ( Dempster et al. , 1977 ) . It can be shown that arbitrary distributions can , given enough components , be approximated by mixtures of Gaussians ( Goodfellow et al. , 2016 ) . Thus , GMMs are guaranteed to model the complete data distribution , but only to the extent allowed by the number of componentsK . In this respect , GMMs are similar to flat neural networks with a single hidden layer : although , by the universal approximation theorem of Pinkus ( 1999 ) and Hornik et al . ( 1989 ) , they can approximate arbitrary functions ( from certain rather broad function classes ) , they fail to do so in practice . The reason for this is that the number of required hidden layer elements is unknown , and usually beyond the reach of any reasonable computational capacity . For images , this problem was largely solved by introducing deep Convolutional Neural Networks ( CNNs ) . CNNs model the statistical structure of images ( hierarchical organization and translation invariance ) by chaining multiple convolution and pooling layers . Thus the number of parameters without compromising accuracy can be reduced . 1.1 OBJECTIVE , CONTRIBUTION AND NOVELTY . The objectives of this article are to introduce a GMM architecture which exploits the same principles that led to the performance explosion of CNNs . In particular , the genuinely novel characteristics are : • formulation of GMMs as a deep hierarchy , including convolution and pooling layers , • end-to-end training by SGD from random initial conditions ( no k-means initialization ) , • generation of realistic samples by a new sharpening procedure , • better empirical performance than vanilla GMMs for sampling , clustering and outlier detection . In addition , we provide a publicly available TensorFlow implementation which supports a Keras-like flexible construction of Deep Convolutional Gaussian Mixture Models instances . 1.2 RELATED WORK . Generative Adverserial Networks The currently most widely used models of image modeling and generation are Generative Adverserial Networks ( Arjovsky et al. , 2017 ; Mirza & Osindero , 2014 ; Goodfellow et al. , 2014 ) . Variational Autoencoders ( VAEs ) follow the classic autoencoder principle ( Kingma & Welling , 2013 ) , trying to reconstruct their inputs through a bottleneck layer , whose activities are additionally constrained to have a Gaussian distribution . GANs are trained adversarially , mapping Gaussian noise to image instances , while trying to fool an additional discriminator network , which in turn aims to distinguish real from generated samples . They are capable of generating photo-realistic images ( Richardson & Weiss , 2018 ) , although their probabilistic interpretation remains unclear since they do not possess a differentiable loss function that is minimized by training . They may suffer from what is termed mode collapse , which is hard to detect automatically due to the absence of a loss function . Due to their ability to generate realistic images , they are prominently used in models of continual learning ( Shin et al. , 2018 ) . Hierarchical GMMs Mixture of Factor Analyzers ( MFAs ) models ( McLachlan & Peel , 2005 ; Ghahramani & Hinton , 1997 ) can be considered as hierarchical GMMs because they are formulated in terms of a lower-dimensional latent-variable representation , which is mapped to a higherdimensional space . The use of MFAs for describing natural images is discussed in detail in Richardson & Weiss ( 2018 ) , showing that the MFA model alone , without further hierarchical structure , compares quite favorably to GANs when considering image generation . A straightforward hierarchical extension of GMMs is presented by Liu et al . ( 2002 ) with the goal of unsupervised clustering : responsibilities of one GMM are treated as inputs to a subsequent GMM , together with an adaptive mechanism that determines the depth of the hierarchy . Garcia et al . ( 2010 ) present a comparable , more information-theoretic approach . A hierarchy of MFA layers with sampling in mind is presented by Viroli & McLachlan ( 2019 ) , where each layer is sampling values for the latent variables of the previous one , although transformations between layers are exclusively linear . Van Den Oord & Schrauwen ( 2014 ) and ( Tang et al. , 2012 ) pursue a similar approach . All described approaches use ( quite complex ) extensions of the EM algorithm initialized by k-means for training hierarchical GMMs , except Richardson & Weiss ( 2018 ) use Stochastic Gradient Descent ( SGD ) , although with a k-means initialization . None of these models consider convolutional or max-pooling operations which have been proven to be important for modeling the statistical structure of images . Convolutional GMMs The only work we could identify proposing to estimate hierarchical convolutional GMMs is Ghazvinian Zanjani et al . ( 2018 ) , although the article described a hybrid model where a CNN and a GMM are combined . SGD and End-to-End GMM Training Training GMMs by SGD is challenging due to local optima and the need to enforce model constraints , most notably the constraint of positive-definite covariance matrices . This has recently been discussed in Hosseini & Sra ( 2020 ) , although the proposed solution requires parameter initialization by k-means and introduces several new hyper-parameters and is , thus , unlikely to work as-is in a hierarchical structure . An SGD approach that achieves robust convergence even without k-means-based parameter initialization is presented by Gepperth & Pflb ( 2020 ) . Undesirable local optima caused by random parameter initialization are circumvented by an adaptive annealing strategy . 2 DATA . For the evaluation we use the following image data sets : MNIST ( LeCun et al. , 1998 ) is the common benchmark for computer vision systems and classification problems . It consists of 60 000 28× 28 gray scale images of handwritten digits ( 0-9 ) . FashionMNIST ( Xiao et al. , 2017 ) consists of images of clothes in 10 categories and is structured like the MNIST dataset . Although these datasets are not particularly challenging for classification , their dimensionality of 784 is at least one magnitude higher than datasets used for validating other hierarchical GMM approaches in the literature . 3 DCGMM : MODEL OVERVIEW . The Deep Convolutional Gaussian Mixture Model is a hierarchical model consisting of layers in analogy to CNNs.1 Each layer with index L expects an input tensor A ( L−1 ) ∈R4 of dimensions N , H ( L−1 ) , W ( L−1 ) , C ( L−1 ) and produces an output tensor A ( L ) ∈R4 of dimensions N , H ( L ) , W ( L ) , C ( L ) . Layers can have internal variables θ ( L ) that are adapted during SGD training . An DCGMM layer L has two basic operating modes ( see Figure 1 ) : for ( density ) estimation , an input tensor A ( L−1 ) from layer L− 1 is transformed into an output tensor A ( L ) . For sampling , the direction is reversed : each layer receives a control signal T ( L+1 ) from layer L+1 ( same dimensions as A ( L ) ) , which is transformed into a control signal T ( L ) to layer L-1 ( same dimensions as A ( L−1 ) ) . 3.1 LAYER TYPES . We define three layer types : Folding ( F ) , Pooling ( P ) and convolutional GMM ( G ) . Each layer implements distinct operations for each of the two modes , i.e. , estimation and sampling . 1TensorFlow code is available under https : //github.com/iclr2021-dcgmm/dcgmm Folding Layer For density estimation , this layer performs a part of the well-known convolution operation known from CNNs . Based on the filter sizes f ( L ) X , f ( L ) Y as well as the filter strides ∆ ( L ) X , ∆ ( L ) Y , all entries of the input tensor inside the range of the sliding filter window are dumped into the channel dimension of the output tensor . We thus obtain an output tensor of dimensions N , H ( L ) = 1 + H ( L−1 ) −f ( L ) Y ∆ ( L ) Y , W ( L ) = 1 + W ( L−1 ) −f ( L ) X ∆ ( L ) X and C ( L ) =C ( L ) f ( L ) X f ( L ) Y , whose entries are computed as A ( L ) nhwc = A ( L−1 ) nh′w′c′ with h=h ′/f ( L ) Y , w=w′/f ( L ) X and c= c ′+ ( ( h′−h∆ ( L ) Y ) f ( L ) X +w ′−w∆ ( L ) X ) C ( L−1 ) + c′ . When sampling , it performs the inverse mapping which is not a one-to-one correspondence : input tensor elements which receive several contributions are simply averaged over all contributions . Pooling Layer For density estimation , pooling layers perform the same operations as standard ( max- ) pooling layers in CNNs based on the kernel sizes k ( L ) Y , k ( L ) X and strides ∆ ( L ) X , ∆ ( L ) Y . When sampling , pooling layers perform a simple nearest-neighbor up-sampling by a factor indicated by the kernel sizes and strides . GMM Layer This layer type contains K GMM components , each of which is associated with trainable parameters πk , µk and Σk , k = 1 . . .K , representing the GMM weights , centroids and covariances . What makes GMM layers convolutional is that they do not model single input vectors , but the channel content at all positions h , w of the input A ( L−1 ) n , w , h , : , using a shared set of parameters . This is analog to the way a CNN layer models image content at all sliding window positions using the same filters . A GMM layer thus maps the input tensor A ( L−1 ) ∈RN , H ( L−1 ) , W ( L−1 ) , C ( L−1 ) to A ( L ) ∈RN , H ( L−1 ) , W ( L−1 ) , K , each GMM component k ∈ { 0 , . . . , K } contributing the likelihood A ( L ) nhwk of having generated the channel content at position h , w ( for sample n in the mini-batch ) . This likelihood is often referred to as responsibility and is computed as pnhwk ( A ( L−1 ) ) = Nk ( A ( L−1 ) nhw : ; µk , Σk ) A ( L ) nhwk ≡ pnhwk∑ c′ pnhwc′ . ( 2 ) For training the GMM layer , we optimize the GMM log-likelihood L ( L ) for each layer L : L ( L ) hw = ∑ n log ∑ k πkpnhwk ( A ( L−1 ) ) L ( L ) = ∑ hw L ( L ) hw H ( L−1 ) W ( L−1 ) ( 3 ) Training is performed by SGD according to the technique , and with the recommended parameters , presented by Gepperth & Pflb ( 2020 ) , which uses a max-component approximation to L ( L ) . In sampling mode , a control signal T ( L ) is produced by standard GMM sampling , performed separately for all positions h , w . GMM sampling at position h , w first selects a component by drawing from a multinomial distribution . If the GMM layer is the last layer of a DCGMM instance , the multinomial ’ s parameters are the mixing weights π : for each position h , w . Otherwise , the control signal T ( L+1 ) nhw : received from layer L+ 1 is used . It is consistent to use the control signal for component selection in layer L , since it was sampled by layer L+ 1 , which was in turn trained on the component responsibilities of layer L. The selected component ( still at position h , w ) then samples T ( L ) nhw : . It is often beneficial for sampling to restrict component selection to the S components with the highest control signal ( top-S sampling ) .

Summary: The paper proposes a model that combines hierarchical Gaussian Mixture Models with a convolutional architecture, supporting both estimation and sampling. The model is trained end-to-end via SGD and is composed of 3 types of layers: standard convolutional and max-pooling layers and a newly proposed GMM layer. The latter operates by modeling the stack of channels at each spatial location as vectors sampled from a GMM. The outputs of the layer are the component probabilities at each location, followed by channel-wise normalization. The loss function is the average log-likelihood of every location at every GMM layer. The paper argues for using it as an alternative to other, less interpretable, probabilistic models of images and demonstrates its capacity to model the MNIST and FashionMNIST datasets.