Gradient-based training of Gaussian Mixture Models for High-Dimensional Streaming Data

1 INTRODUCTION . This contribution focuses Gaussian Mixture Models ( GMMs ) , which represent a probabilistic unsupervised model for clustering and density estimation and allowing sampling and outlier detection . GMMs have been used in a wide range of scenarios , e.g. , Melnykov & Maitra ( 2010 ) . Commonly , free parameters of a GMM are estimated by using the Expectation-Maximizations ( EMs ) algorithm ( Dempster et al. , 1977 ) , as it does not require learning rates and automatically enforces all GMM constraints . A popular online variant is stochastic Expectation Maximization ( sEM ) ( Cappé & Moulines , 2009 ) , which can be trained mini-batch wise and is , thus , more suited for large datasets or streaming data . 1.1 MOTIVATION . Intrinsically , EM is a batch-type algorithm . Memory requirements can therefore become excessive for large datasets . In addition , streaming-data scenarios require data samples to be processed one by one , which is impossible for a batch-type algorithm . Moreover , data statistics may be subject to changes over time ( concept drift/shift ) , to which the GMM should adapt . In such scenarios , an online , mini-batch type of optimization such as SGD is attractive , as it can process samples one by one , has modest , fixed memory requirements and can adapt to changing data statistics . 1.2 RELATED WORK . Online EM is a technique for performing EM mini-batch wise , allowing to process large datasets . One branch of previous research Newton et al . ( 1986 ) ; Lange ( 1995 ) ; Chen et al . ( 2018 ) has been devoted to the development of stochastic Expectation Maximization ( sEM ) algorithms that reduce the original EM method in the limit of large batch sizes . The variant of Cappé & Moulines ( 2009 ) is widely used due to its simplicity and efficiency for large datasets . These approaches come at the price of additional hyper-parameters ( e.g. , learning rate or mini-batch size ) , thus , removing a key advantage of EM over SGD . Another common approach is to modify the EM algorithm itself by , e.g. , including heuristics for adding , splitting and merging centroids Vlassis & Likas ( 2002 ) ; Engel & Heinen ( 2010 ) ; Pinto & Engel ( 2015 ) ; Cederborg et al . ( 2010 ) ; Song & Wang ( 2005 ) ; Kristan et al . ( 2008 ) ; Vijayakumar et al . ( 2005 ) . This allows GMM-like models to be trained by presenting one sample after another . The models work well in several application scenarios , but their learning dynamics are impossible to analyze mathematically . They also introduce a high number of parameters . Apart from these works , some authors avoid the issue of extensive datasets by determining smaller “ core sets ” of representative samples and performing vanilla EM Feldman et al . ( 2011 ) . SGD for training GMMs has , as far as we know , been recently treated only by Hosseini & Sra ( 2015 ; 2019 ) . In this body of work , GMM constraint enforcement is ensured by using manifold optimization techniques and re-parameterization/regularization . Thereby , additional hyper-parameters are introduced . The issue of local optima is sidestepped by a k-means type centroid initialization , and the used image datasets are low-dimensional ( 36 dimensions ) . Additionally , enforcing positive definiteness constraints by Cholesky decomposition is discussed . Annealing and Approximation approaches for GMMs were proposed by Verbeek et al . ( 2005 ) ; Pinheiro & Bates ( 1995 ) ; Ormoneit & Tresp ( 1998 ) ; Dognin et al . ( 2009 ) . However , the regularizers proposed by Verbeek et al . ( 2005 ) ; Ormoneit & Tresp ( 1998 ) significantly differ from our scheme . GMM log-likelihood approximations , similar to the one used here , are discussed in , e.g. , Pinheiro & Bates ( 1995 ) and Dognin et al . ( 2009 ) , but only in combination with EM training . GMM Training in High-Dimensional Spaces is discussed in several publications : A conceptually very interesting procedure is proposed by Ge et al . ( 2015 ) . It exploits the properties of highdimensional spaces in order to achieve learning with a number of samples that is polynomial in the number of Gaussian components . This is difficult to apply in streaming settings , since higher-order moments need to be estimated beforehand , and also because the number of samples usually can not be controlled in practice . Training GMM-like lower-dimensional factor analysis models by SGD on high-dimensional image data is successfully demonstrated in Richardson & Weiss ( 2018 ) , avoiding numerical issues , but , again , sidestepping the local optima issue by using k-means initialization . The numerical issues associated with log-likelihood computation in high-dimensional spaces are generally mitigated by using the “ logsumexp ” trick Nielsen & Sun ( 2016 ) , which is , however , insufficient for ensuring numerical stability for particularly high-dimensional data , such as images . 1.3 GOALS AND CONTRIBUTIONS . The goals of this article are to establish GMM training by SGD as a simple and scalable alternative to sEM in streaming scenarios with potentially high-dimensional data . The main novel contributions are : • a proposal for numerically stable GMM training by SGD that outperforms sEM for high data dimensionalities • an automatic annealing procedure that ensures SGD convergence from a wide range of initial conditions without prior knowledge of the data ( e.g. , no k-means initialization ) which is especially beneficial for streaming data • a computationally efficient method for enforcing all GMM constraints in SGD Apart from these contents , we provide a publicly available TensorFlow implementation.1 2 DATASETS . We use a variety of different image-based datasets as well as a non-image dataset for evaluation purposes . All datasets are normalized to the [ 0 , 1 ] range . MNIST ( LeCun et al. , 1998 ) contains gray scale images , which depict handwritten digits from 0 to 9 in a resolution of 28×28 pixels – the common benchmark for computer vision systems . SVHN ( Wang et al. , 2012 ) contains color images of house numbers ( 0-9 , resolution 32× 32 ) . FashionMNIST ( Xiao et al. , 2017 ) contains gray scale images of 10 cloth categories and is considered as more challenging classification task compared to MNIST . Fruits 360 ( Murean & Oltean , 2018 ) consists of colored pictures showing different types of fruits ( 100× 100× 3 pixels ) . The ten best-represented classes are selected from this dataset . Devanagari ( Acharya et al. , 2016 ) includes gray scale images of handwritten Devanagari letters with a resolution of 32×32 pixels – the first 10 classes are selected . NotMNIST ( Yaroslav Bulatov , 2011 ) is a gray scale image dataset ( resolution 28× 28 pixels ) of 1https : //github.com/gmm-iclr21/sgd-gmm letters from A to J extracted from different public available fonts . ISOLET ( Cole & Fanty , 1990 ) is a non-image dataset containing 7 797 samples of spoken letters recorded from 150 subjects . Each sample was encoded and is represented by 617 float values . 3 GAUSSIAN MIXTURE MODELS . GMMs are probabilistic models that intend to explain the observed data X = { xn } by expressing their density as a weighted mixture of K Gaussian component densities N ( x ; µk , Pk ) ≡Nk ( x ) : p ( x ) = ∑K k πkNk ( x ) . We work with precision matrices P = Σ−1 instead of covariances Σ . This is realized by optimizing the ( incomplete ) log-likelihood L = En [ log ∑ k πkNk ( xn ) ] . ( 1 ) 3.1 GMMS AND SGD . GMMs require the mixture weights to be normalized : ∑ k πk = 1 and the precision matrices to be positive definite : xTPkx≥ 0 ∀x . These constraints must be explicitly enforced after each SGD step : Weights πk are adapted according to Hosseini & Sra ( 2015 ) , which replaces them by other free parameters ξk from which the πk are computed so that normalization is ensured : πk = exp ( ξk ) ∑ j exp ( ξj ) . ( 2 ) Precision Matrices need to be positive-definite , so we re-parameterize these as Pk = ( DTkDk ) , where the upper-diagonal matrices Dk result from a Cholesky decomposition . Consequently , detΣk = detP −1 k = ( det ( DTkDk ) ) −1 = ( tr ( Dk ) ) −2 can be computed efficiently . To avoid recomputing the costly Cholesky decomposition of the Pk at every iteration , we perform it on the initial precision matrices and just erase the elements below the diagonal in theDk after each gradient step . 3.2 MAX-COMPONENT APPROXIMATION FOR GMMS . The log-likelihood Eq . ( 1 ) is difficult to optimize by SGD ( see Sec . 3.3 ) . This is why we intend to find a lower bound that we can optimize instead . A simple scheme is given by L=En [ log ∑ k πkNk ( xn ) ] ≥En [ log maxk ( πkNk ( xn ) ) ] = L̂=En [ log ( πk∗Nk∗ ( xn ) ) ] ( 3 ) where k∗= arg maxk πkNk ( xn ) . This is what we call the max-component approximation of Eq . ( 3 ) . In contrast to the lower bound that is constructed for EM-type algorithms , this bound is usually not tight . The advantages of L̂ are the avoidance of local optima in SGD , and the elimination of exponentials causing numerical instabilities for high data dimensions . The “ logsumexp ” trick is normally employed with GMMs to rectify this by factoring out the largest component probability Nk∗ . This mitigates , but does not avoid numerical problems when distances are high . To give an example : we normalize the component probability Nk = e−101 ( using 32-bit floats ) by the highest probability Nk∗ = e3 , and we obtain NkNk∗ = e −104 , which is numerically problematic . 3.3 UNDESIRABLE LOCAL OPTIMA IN SGD TRAINING . An issue when performing SGD without k-means initialization concerns undesirable local optima . Degenerate Solutions occur when naively optimizing L by SGD ( see Fig . 1a ) . All components have the same weight , centroid and covariance matrix : πk ≈ 1K , µk =E [ X ] , Σk = Cov ( X ) ∀ k , in which case all gradients vanish ( see App . A.3 for a proof ) . These solutions are avoided by L̂ , since only a subset of components is updated by SGD , thereby breaking the symmetry between components . Single/Sparse-Component Solutions occur when optimizing L̂ by SGD ( see Fig . 1b ) . They are characterized by one or several components { ki } that have large weights with centroid and precision matrices given by the mean and covariance of a significant subsetXki ⊂X of the dataX : πki 0 , µki =E [ Xki ] , Σki = Cov ( Xki ) , whereas the remaining components k are characterized by πk ≈ 0 , µk =µ ( t= 0 ) , Pk =P ( t = 0 ) . Thus , these unconverged components are almost never best-matching components k∗ . The max-operation in L̂ causes gradients like ∂L̂∂µk to contain δkk∗ ( see App . A.3 ) . This implies that they are non-zero only for the best-matching component k∗ . Thus the gradients of unconverged components vanish , implying that they remain in their unconverged state .

Summary: This paper presented a stochastic gradient descent approach to learn a non-stationary high-dimensional Gaussian mixture model from online data. The authors identified 3 challenges - local optima, numerical instability, and catastrophic forgetting, and proposed to address these challenges respective with adaptive annealing, exponential-free approximation, and adaptive SGD learning rate. The proposed approach is demonstrated with several vision/non-vision tasks.