Gradient-based training of Gaussian Mixture Models for High-Dimensional Streaming Data

1 INTRODUCTION . This contribution focuses Gaussian Mixture Models ( GMMs ) , which represent a probabilistic unsupervised model for clustering and density estimation and allowing sampling and outlier detection . GMMs have been used in a wide range of scenarios , e.g. , Melnykov & Maitra ( 2010 ) . Commonly , free parameters of a GMM are estimated by using the Expectation-Maximizations ( EMs ) algorithm ( Dempster et al. , 1977 ) , as it does not require learning rates and automatically enforces all GMM constraints . A popular online variant is stochastic Expectation Maximization ( sEM ) ( Cappé & Moulines , 2009 ) , which can be trained mini-batch wise and is , thus , more suited for large datasets or streaming data . 1.1 MOTIVATION . Intrinsically , EM is a batch-type algorithm . Memory requirements can therefore become excessive for large datasets . In addition , streaming-data scenarios require data samples to be processed one by one , which is impossible for a batch-type algorithm . Moreover , data statistics may be subject to changes over time ( concept drift/shift ) , to which the GMM should adapt . In such scenarios , an online , mini-batch type of optimization such as SGD is attractive , as it can process samples one by one , has modest , fixed memory requirements and can adapt to changing data statistics . 1.2 RELATED WORK . Online EM is a technique for performing EM mini-batch wise , allowing to process large datasets . One branch of previous research Newton et al . ( 1986 ) ; Lange ( 1995 ) ; Chen et al . ( 2018 ) has been devoted to the development of stochastic Expectation Maximization ( sEM ) algorithms that reduce the original EM method in the limit of large batch sizes . The variant of Cappé & Moulines ( 2009 ) is widely used due to its simplicity and efficiency for large datasets . These approaches come at the price of additional hyper-parameters ( e.g. , learning rate or mini-batch size ) , thus , removing a key advantage of EM over SGD . Another common approach is to modify the EM algorithm itself by , e.g. , including heuristics for adding , splitting and merging centroids Vlassis & Likas ( 2002 ) ; Engel & Heinen ( 2010 ) ; Pinto & Engel ( 2015 ) ; Cederborg et al . ( 2010 ) ; Song & Wang ( 2005 ) ; Kristan et al . ( 2008 ) ; Vijayakumar et al . ( 2005 ) . This allows GMM-like models to be trained by presenting one sample after another . The models work well in several application scenarios , but their learning dynamics are impossible to analyze mathematically . They also introduce a high number of parameters . Apart from these works , some authors avoid the issue of extensive datasets by determining smaller “ core sets ” of representative samples and performing vanilla EM Feldman et al . ( 2011 ) . SGD for training GMMs has , as far as we know , been recently treated only by Hosseini & Sra ( 2015 ; 2019 ) . In this body of work , GMM constraint enforcement is ensured by using manifold optimization techniques and re-parameterization/regularization . Thereby , additional hyper-parameters are introduced . The issue of local optima is sidestepped by a k-means type centroid initialization , and the used image datasets are low-dimensional ( 36 dimensions ) . Additionally , enforcing positive definiteness constraints by Cholesky decomposition is discussed . Annealing and Approximation approaches for GMMs were proposed by Verbeek et al . ( 2005 ) ; Pinheiro & Bates ( 1995 ) ; Ormoneit & Tresp ( 1998 ) ; Dognin et al . ( 2009 ) . However , the regularizers proposed by Verbeek et al . ( 2005 ) ; Ormoneit & Tresp ( 1998 ) significantly differ from our scheme . GMM log-likelihood approximations , similar to the one used here , are discussed in , e.g. , Pinheiro & Bates ( 1995 ) and Dognin et al . ( 2009 ) , but only in combination with EM training . GMM Training in High-Dimensional Spaces is discussed in several publications : A conceptually very interesting procedure is proposed by Ge et al . ( 2015 ) . It exploits the properties of highdimensional spaces in order to achieve learning with a number of samples that is polynomial in the number of Gaussian components . This is difficult to apply in streaming settings , since higher-order moments need to be estimated beforehand , and also because the number of samples usually can not be controlled in practice . Training GMM-like lower-dimensional factor analysis models by SGD on high-dimensional image data is successfully demonstrated in Richardson & Weiss ( 2018 ) , avoiding numerical issues , but , again , sidestepping the local optima issue by using k-means initialization . The numerical issues associated with log-likelihood computation in high-dimensional spaces are generally mitigated by using the “ logsumexp ” trick Nielsen & Sun ( 2016 ) , which is , however , insufficient for ensuring numerical stability for particularly high-dimensional data , such as images . 1.3 GOALS AND CONTRIBUTIONS . The goals of this article are to establish GMM training by SGD as a simple and scalable alternative to sEM in streaming scenarios with potentially high-dimensional data . The main novel contributions are : • a proposal for numerically stable GMM training by SGD that outperforms sEM for high data dimensionalities • an automatic annealing procedure that ensures SGD convergence from a wide range of initial conditions without prior knowledge of the data ( e.g. , no k-means initialization ) which is especially beneficial for streaming data • a computationally efficient method for enforcing all GMM constraints in SGD Apart from these contents , we provide a publicly available TensorFlow implementation.1 2 DATASETS . We use a variety of different image-based datasets as well as a non-image dataset for evaluation purposes . All datasets are normalized to the [ 0 , 1 ] range . MNIST ( LeCun et al. , 1998 ) contains gray scale images , which depict handwritten digits from 0 to 9 in a resolution of 28×28 pixels – the common benchmark for computer vision systems . SVHN ( Wang et al. , 2012 ) contains color images of house numbers ( 0-9 , resolution 32× 32 ) . FashionMNIST ( Xiao et al. , 2017 ) contains gray scale images of 10 cloth categories and is considered as more challenging classification task compared to MNIST . Fruits 360 ( Murean & Oltean , 2018 ) consists of colored pictures showing different types of fruits ( 100× 100× 3 pixels ) . The ten best-represented classes are selected from this dataset . Devanagari ( Acharya et al. , 2016 ) includes gray scale images of handwritten Devanagari letters with a resolution of 32×32 pixels – the first 10 classes are selected . NotMNIST ( Yaroslav Bulatov , 2011 ) is a gray scale image dataset ( resolution 28× 28 pixels ) of 1https : //github.com/gmm-iclr21/sgd-gmm letters from A to J extracted from different public available fonts . ISOLET ( Cole & Fanty , 1990 ) is a non-image dataset containing 7 797 samples of spoken letters recorded from 150 subjects . Each sample was encoded and is represented by 617 float values . 3 GAUSSIAN MIXTURE MODELS . GMMs are probabilistic models that intend to explain the observed data X = { xn } by expressing their density as a weighted mixture of K Gaussian component densities N ( x ; µk , Pk ) ≡Nk ( x ) : p ( x ) = ∑K k πkNk ( x ) . We work with precision matrices P = Σ−1 instead of covariances Σ . This is realized by optimizing the ( incomplete ) log-likelihood L = En [ log ∑ k πkNk ( xn ) ] . ( 1 ) 3.1 GMMS AND SGD . GMMs require the mixture weights to be normalized : ∑ k πk = 1 and the precision matrices to be positive definite : xTPkx≥ 0 ∀x . These constraints must be explicitly enforced after each SGD step : Weights πk are adapted according to Hosseini & Sra ( 2015 ) , which replaces them by other free parameters ξk from which the πk are computed so that normalization is ensured : πk = exp ( ξk ) ∑ j exp ( ξj ) . ( 2 ) Precision Matrices need to be positive-definite , so we re-parameterize these as Pk = ( DTkDk ) , where the upper-diagonal matrices Dk result from a Cholesky decomposition . Consequently , detΣk = detP −1 k = ( det ( DTkDk ) ) −1 = ( tr ( Dk ) ) −2 can be computed efficiently . To avoid recomputing the costly Cholesky decomposition of the Pk at every iteration , we perform it on the initial precision matrices and just erase the elements below the diagonal in theDk after each gradient step . 3.2 MAX-COMPONENT APPROXIMATION FOR GMMS . The log-likelihood Eq . ( 1 ) is difficult to optimize by SGD ( see Sec . 3.3 ) . This is why we intend to find a lower bound that we can optimize instead . A simple scheme is given by L=En [ log ∑ k πkNk ( xn ) ] ≥En [ log maxk ( πkNk ( xn ) ) ] = L̂=En [ log ( πk∗Nk∗ ( xn ) ) ] ( 3 ) where k∗= arg maxk πkNk ( xn ) . This is what we call the max-component approximation of Eq . ( 3 ) . In contrast to the lower bound that is constructed for EM-type algorithms , this bound is usually not tight . The advantages of L̂ are the avoidance of local optima in SGD , and the elimination of exponentials causing numerical instabilities for high data dimensions . The “ logsumexp ” trick is normally employed with GMMs to rectify this by factoring out the largest component probability Nk∗ . This mitigates , but does not avoid numerical problems when distances are high . To give an example : we normalize the component probability Nk = e−101 ( using 32-bit floats ) by the highest probability Nk∗ = e3 , and we obtain NkNk∗ = e −104 , which is numerically problematic . 3.3 UNDESIRABLE LOCAL OPTIMA IN SGD TRAINING . An issue when performing SGD without k-means initialization concerns undesirable local optima . Degenerate Solutions occur when naively optimizing L by SGD ( see Fig . 1a ) . All components have the same weight , centroid and covariance matrix : πk ≈ 1K , µk =E [ X ] , Σk = Cov ( X ) ∀ k , in which case all gradients vanish ( see App . A.3 for a proof ) . These solutions are avoided by L̂ , since only a subset of components is updated by SGD , thereby breaking the symmetry between components . Single/Sparse-Component Solutions occur when optimizing L̂ by SGD ( see Fig . 1b ) . They are characterized by one or several components { ki } that have large weights with centroid and precision matrices given by the mean and covariance of a significant subsetXki ⊂X of the dataX : πki 0 , µki =E [ Xki ] , Σki = Cov ( Xki ) , whereas the remaining components k are characterized by πk ≈ 0 , µk =µ ( t= 0 ) , Pk =P ( t = 0 ) . Thus , these unconverged components are almost never best-matching components k∗ . The max-operation in L̂ causes gradients like ∂L̂∂µk to contain δkk∗ ( see App . A.3 ) . This implies that they are non-zero only for the best-matching component k∗ . Thus the gradients of unconverged components vanish , implying that they remain in their unconverged state .

Summary: A major concern about the paper is related to the unsupported claims and contribution throughout the paper. For example, the way the training copes with distribution shift or alleviate forgetting is not clear or elaborated on. Beyond the abstract and before the empirical validation no theory or justification is provided to substantiate this claim. The idea of the paper and the motivation are very interesting. The experiments look convincing. Writing and presentation are a good start point for improving the paper.