Quantum Expectation-Maximization for Gaussian Mixture Models

1 INTRODUCTION . Over the last few years , the effort to find real world applications of quantum computers has greatly intensified . Along with chemistry , material sciences , finance , one of the fields where quantum computers are expected to be most beneficial is machine learning . A number of different algorithms have been proposed for quantum machine learning ( Biamonte et al. , 2017 ; Wiebe et al. , 2017 ; Kerenidis & Prakash , 2018 ; Harrow et al. , 2009 ; Subaşı et al. , 2019 ; Farhi & Neven , 2018 ) , both for the supervised and unsupervised setting , and despite the lack of large-scale quantum computers and quantum memory devises , some quantum algorithms have been demonstrated in proof-of-principle experiments ( Li et al. , 2015 ; Otterbach et al. , 2017 ; Jiang et al. , 2019 ) . Here , we look at Expectation-Maximization ( EM ) , a fundamental algorithm in unsupervised learning , that can be used to fit different mixture models and give maximum likelihood estimates with the so-called latent variable models . Such generative models are one of the most promising approaches for unsupervised problems . The goal of a generative model is to learn a probability distribution that is most likely to have generated the data collected in a training set V ∈ Rn×d of n vectors of d features . Fitting the model consists in learning the parameters of a probability distribution p in a certain parameterized family that best describes our vectors vi . We will see that , thanks to this formulation , we can reduce a statistical problem into an optimization problem using maximum likelihood estimation ( ML ) estimation . The likelihood is the function that we use to measure how good a model is for explaining a given dataset . For a given machine learning model with parameters γ , the likelihood of our data set V is the probability that the data have been generated by the model with parameters γ , assuming each point is independent and identically distributed . We think the likelihood as a function of γ , holding the dataset V fixed . For p ( vi|γ ) the probability that a point vi comes from model γ , the likelihood is defined as L ( γ ; V ) : = ∏n i=1 p ( vi|γ ) . From this formula , we can see that in order to find the best parameters γ∗ of our model we need to solve an optimization problem . For numerical and analytical reasons , instead of maximizing the likelihood L , it is common practice to find the best model by maximizing the log-likelihood function ` ( γ ; V ) = logL ( γ ; V ) = ∑n i=1 log p ( vi|γ ) . In this context , we want to find the model that maximizes the log-likelihood : γ∗ML : = arg maxγ ∑n i=1 log p ( vi|γ ) . The procedure to calculate the log-likelihood depends on the specific model under consideration . A possible solution would be to use a gradient based optimization algorithm on ` . Unfortunately , due to the indented landscape of the function , gradient based techniques often do not perform well . Therefore , it is common to solve the maximum likelihood estimation ( or maximum a priori ) problem using the Expectation-Maximization ( EM ) algorithm . EM is an iterative algorithm which is guaranteed to converge to a ( local ) optimum of the likelihood . This algorithm has a striking variety of applications , and has been successfully used for medical imaging ( Balafar et al. , 2010 ) , image restoration ( Lagendijk et al. , 1990 ) , problems in computational biology ( Fan et al. , 2010 ) , and so on . EM has been proposed in different works by different authors , but has been formalized as we know it only in 1977 ( Dempster et al. , 1977 ) . For more details , we refer to ( Lindsay , 1995 ; Bilmes et al. , 1998 ) . In this work , we introduce Quantum Expectation-Maximization ( QEM ) , a new algorithm for fitting mixture models . We detail its usage in the context of Gaussian Mixture Models , and we extend the result to other distributions in the exponential family . We also generalize the result by showing how to compute the MAP : the Maximum A Posteriori estimate of a mixture model . MAP estimates can be seen as the Bayesian version of maximum likelihood estimation problems . MAP estimates are often preferred over ML estimates , due to a reduced propensity to overfit . Our main result can be stated as : Result ( Quantum Expectation-Maximization ) . ( see Theorem 3.9 ) For a data matrix V ∈ Rn×d stored in an appropriate QRAM data structure and for parameters δθ , δµ > 0 , Quantum Expectation-Maximization ( QEM ) fits a Maximum Likelihood ( or a Maximum A Posteriori ) estimate of a Gaussian Mixture Model with k components , in running time per iteration which is dominated by : Õ ( d2k4.5η3κ2 ( V ) κ2 ( Σ ) µ ( Σ ) δ3µ ) , ( 1 ) where Σ is a covariance matrix of a Gaussian distribution , η is a parameter of the dataset related to the maximum norm of the vectors , δθ , δµ are error parameters in the QEM algorithm , µ ( < √ d ) is a factor appearing in quantum linear algebra and κ is the condition number of a matrix . Here we only kept the term in the running time that dominates for the range of parameters of interest . In Theorem 3.9 we explicate the running time of each step of the algorithm . The QEM algorithm runs for a number of iterations until a stopping condition is met ( defined by a parameter τ > 0 ) which implies a convergence to a ( local ) optimum . Let ’ s have a first high-level comparison of this result with the standard classical algorithms . The runtime of a single iteration in the standard implementation of the EM algorithm is at least O ( knd2 ) ( Pedregosa et al. , 2011 ; Murphy , 2012 ) . The advantage of the quantum algorithm is an exponential improvement with respect to the number of elements in the training set , albeit with a worsening on other parameters . It is crucial to find datasets where such a quantum algorithm can offer a speedup . For a reasonable range of parameters ( d = 40 , k = 10 , η = 10 , δ = 0.5 , κ ( V ) = 25 , κ ( Σ ) = 5 , µ ( Σ ) = 4 ) which is motivated by some experimental evidence reported in Section 4 , datasets where the number of samples in the order of O ( 1012 ) might be processed faster on a quantum computer . One should expect that some of the parameters of the quantum algorithm can be improved , especially the dependence on the condition numbers and the errors , which can make enlarge the type of datasets where QEM can offer an advantage . Note that we expect the number of iterations of the quantum algorithm to be proportional to the number of iteration of the classical case . This is to be expected since the convergence rate does not change , and it is corroborated by previous experimental evidence in a similar scenario : the number of iterations needed by q-means algorithm for convergence , is proportional to the number of iterations of the classical k-means algorithm ( Kerenidis et al. , 2018 ) . Expectation-Maximization is widely used for fitting mixture models in machine learning ( Murphy , 2012 ) . Most mixture models use a base distribution in the exponential family : Poisson ( Church & Gale , 1995 ) , Binomial , Multinomial , log-normal ( Dexter & Tanner , 1972 ) , exponential ( Ghitany et al. , 1994 ) , Dirichlet multinomial ( Yin & Wang , 2014 ) , and others . EM is also used to fit mixtures of experts , mixtures of the student T distribution ( which does not belong to the exponential family , and can be fitted with EM using ( Liu & Rubin , 1995 ) ) and for factor analysis , probit regression , and learning Hidden Markov Models ( Murphy , 2012 ) . 1.1 PREVIOUS WORK . There is a fair amount of quantum algorithms that have been proposed in the context of unsupervised learning . ( Aı̈meur et al. , 2013 ; Lloyd et al. , 2013b ; Otterbach et al. , 2017 ) . Recently , classical machine learning algorithms were obtained by “ dequantizing ” quantum machine learning algorithms ( Tang , 2018b ; a ; c ; Gilyén et al. , 2018a ; Chia et al. , 2018 ) . The runtime of these classical algorithm is poly-logarithmic in the dimensions of the dataset . However , the polynomial dependence on the rank , the error , and the condition number , make these new algorithms impractical on interesting datasets , as shown experimentally by ( Arrazola et al. , 2019 ) . Fast classical algorithm for GMM exists , albeit assuming only one shared covariance matrix ( Dasgupta , 1999 ) , and without a polylogarithmic dependence in the number of elements in the training set . Independently of us , Miyahara , Aihara , and Lechner extended the q-means algorithm ( Kerenidis et al. , 2018 ) for Gaussian Mixture Models ( Miyahara et al. , 2019 ) , using similar techniques . The main difference is that in their work the update step is performed using a hard-clustering approach ( as in the k-means algorithm ) , that is for updating the centroid and the covariance matrices of a cluster j , only the data points for which cluster j is nearest are taken into account . In our work , we use the soft clustering approach ( as in the classical EM algorithm ) , that is for updating the centroid and the covariance matrices of cluster j , all the data points weighted by their responsibility for cluster j are taken into account . Both approaches have merits and can offer advantages ( Kearns et al. , 1998 ) . 2 EXPECTATION-MAXIMIZATION AND GAUSSIAN MIXTURE MODELS . As is common in machine learning literature , we introduce the Expectation-Maximization algorithm by using it to fit Gaussian Mixture Models ( GMM ) . Mixture models are a popular generative model in machine learning . The intuition behind mixture models is to model complicated distributions by using a group of simpler ( usually uni-modal ) distributions . In this setting , the purpose of the learner is to model the data by fitting the joint probability distribution which most likely have generated our samples . In this section we describe GMM : probably the most used mixture model used to solve unsupervised classification problems . In fact , given a sufficiently large number of mixture components , it is possible to approximate any density defined in Rd ( Murphy , 2012 ) . In unsupervised settings , we are given a training set of unlabeled vectors v1 · · · vn ∈ Rd which we represent as rows of a matrix V ∈ Rn×d . Let yi ∈ [ k ] one of the k possible labels for a point vi . We posit that the joint probability distribution of the data p ( vi , yi ) = p ( vi|yi ) p ( yi ) , is defined as follow : yi ∼ Multinomial ( θ ) for θ ∈ Rk , and p ( vi|yi = j ) ∼ N ( µj , Σj ) . The θj are the mixing weights , i.e . the probabilities that yi = j , and N ( µj , Σj ) is the Gaussian distribution centered in µj ∈ Rd with covariance matrix Σj ∈ Rd×d . Note that the variables yi are unobserved , and thus are called latent variables . There is a simple interpretation for this model . We assume the data is created by first selecting an index j ∈ [ k ] by sampling according to Multinomial ( θ ) , and then a vector vi is sampled from N ( µj , Σj ) . Fitting a GMM to a dataset reduces to finding an assignment for the parameters : γ = ( θ , µ , Σ ) = ( θ , µ1 , · · · , µk , Σ1 , · · · , Σk ) that best maximize the log-likelihood for a given dataset . Note that the algorithm used to fit GMM can return a local minimum which might be different than γ∗ : the model that represents the global optimum of the likelihood function . We use the letter φ to represent our base distribution , which in this case is the probability density function of a multivariate Gaussian distribution N ( µ , Σ ) . Using this formulation , a GMM is expressed as : p ( v ) = ∑k j=1 θjφ ( v ; µj , Σj ) where θj are the mixing weights of the multinomial distribution such that ∑k j=1 θj = 1 . The probability for an observation vi to be assigned to the component j is given by : rij = θjφ ( vi ; µj , Σj ) ∑k l=1 θlφ ( vi ; µl , Σl ) . This value is called responsibility , and corresponds to the posterior probability of the sample i being assigned label j by the current model . As anticipated , to find the best parameters of our generative model , we maximize the log-likelihood of the data . For GMM , the likelihood is given by the following formula ( Ng , 2012 ) : ` ( γ ; V ) = ` ( θ , µ , Σ ; V ) = ∑n i=1 log p ( vi ; θ , µ , Σ ) = . Alas , it is seldom possible to solve maximum likelihood estimation analytically ( i.e . by finding the zeroes of the derivatives of the log-like function , and this is one of those cases . Fortunately , Expectation-Maximization is an iterative algorithm that solves numerically the optimization problem of ML estimation . To complicate things , the likelihood function for GMM is not convex , and thus we might find some local minima ( Hastie et al. , 2009 ) . If we were to know the latent variable yi , then the log-likelihood for GMM would be : f ` ( γ ; V ) = ∑n i=1 log p ( vi | yi ; µ , Σ ) + log p ( yi ; θ ) This formula can be easily maximized with respect to the parameters θ , µ , and Σ . In the Expectation step we calculate the missing variables yi ’ s , given a guess of the parameters ( θ , µ , Σ ) of the model . Then , in the Maximization step , we use the estimate of the latent variables obtained in the Expectation step to update the estimate of the parameters . While in the Expectation step we calculate a lower bound on the likelihood , in the Maximization step we maximize it . Since at each iteration the likelihood can only increase , the algorithm is guaranteed to converge , albeit possibly to a local optimum ( see ( Hastie et al. , 2009 ) for the proof ) . During the Expectation step all the responsibilities are calculated , while in the Maximization step we update our estimate on the parameters γt+1 = ( θt+1 , µt+1 , Σt+1 ) . The stopping criterion for GMM is usually a threshold on the increment of the log-likelihood : if the log-likelihood changes less than a threshold between two iterations , then the algorithm stops . Notice that , since the value of the log-likelihood significantly depends on the amount of data points in the training sets , it is often preferable to adopt a scale-free stopping criterion , which does not depend on the number of samples . For instance , in the toolkit scikit-learn ( Pedregosa et al. , 2011 ) the stopping criterion is given by a tolerance on the average increment of the log-probability , which is chosen to be smaller than a certain τ , say 10−3 . More precisely , the stopping criterion is |E [ log p ( vi ; γt ) ] − E [ log p ( vi ; γt+1 ) ] | < τ which we can estimate as | 1n ∑n i=1 log p ( vi ; γ t ) − 1n ∑n i=1 log p ( vi ; γ t+1 ) | < τ . Dataset assumptions in GMM As in q-means ( Kerenidis et al. , 2018 ) , we have an assumption on the dataset that all elements of the mixture contribute proportionally to the total responsibility : i , e∑n i=1 rij∑n i=1 ril = Θ ( 1 ) ∀j , l ∈ [ k ] This is equivalent to requiring that clusters share a comparable amount of points in the “ well-clusterability ” assumption in q-means ( Kerenidis et al. , 2018 ) . It is also equivalent to assuming that θj/θl = Θ ( 1 ) ∀j , l ∈ [ k ] . For convenience , in this work , we also assume that the dataset is normalized such that the shortest vector has norm 1 and define η : = maxi ‖vi‖2 to be the maximum norm squared of a vector in the dataset . This is not a necessary requirement for our dataset , but it will simplify the analysis of our algorithm , allowing us to give strict bounds on the runtime . Preliminaries We assume a basic understanding of quantum computing , we recommend Nielsen and Chuang ( Nielsen & Chuang , 2002 ) for an introduction to the subject . A vector state |v〉 for v ∈ Rd is defined as |v〉 = 1‖v‖ ∑ j∈ [ d ] vj |j〉 , where |j〉 represents ej , the jth vector in the standard basis . The dataset is represented by a matrix V ∈ Rn×d , i.e . each row is a vector vi ∈ Rd for i ∈ [ n ] that represents a single data point . The cluster centers , called centroids , at time t are stored in the matrix Ct ∈ Rk×d , such that the jth row ctj for j ∈ [ k ] represents the centroid of the cluster Ctj . We denote as V≥τ the matrix ∑ ` i=0 σiuiv T i where σ ` is the smallest singular value which is greater than τ . With nnz ( V ) we mean the number of non-zero elements of the rows of V . When we say κ ( V ) we mean the condition number of the matrix V , that is the ratio between the biggest and the smallest ( non-zero ) singular value . All the tools used in this work , like quantum algorithms for computing distances and linear algebraic operations , are reported in the Supplementary Material section .

Summary: The authors present a quantum algorithm for expectation maximization on a quantum machine, which is poly-logarithmic in the size of the dataset (and polynomial in other parameters such as the dimension of the feature space, and number of mixture components, condition number of the data and covariance matrices, some precision/error parameters etc) per iteration. So, compared to the . regular EM algorithm, this yields exponential speedup in the number of data points, but is worse in other factors (such as a k^4.5 dependence on the number of mixture components). So, the quantum system could be superior to a conventional computer for some settings of parameters. They run some (simulation) experiments on a dataset (VoxForge) to report accuracies (though there is no comparison of the accuracy of the classical algorithm). Also, there does not seem to be any experimental results to confirm the theoretical analysis of the scaling characteristics (i.e., to show that the scaling is as predicted by the theory).