ML: GMM & EM Algorithm
Tags: Machine Learning, Math, Em Algorithm, Data Science, Gaussian Mixture Model

GMM is a really popular clustering method you should know as a data scientist. K-means clustering is also a part of GMM. GMM can overcome the limitation of k-means clustering. In this post, I will explain how GMM works and how it will be trained.

Mixture Model

Gaussian Mixture Model equation

What is the mixture model? We can think of just the combination of the models. If you combine the Gaussian Distribution with weights, then it will be the Gaussian Mixture Model. The pi in the equations is the weight, mixing coefficient, of the model, the sum of every weight should be 1 because the probability cannot over 1. It also should be between 0 and 1.

Normal distribution

Each of the normal distribution in the mixture has its own mean parameter and covariance parameter.

Old Faithful dataset from Bishop’s book

Why do we need the mixture model? It is clear if you watch the above picture. The Gaussian should be dense around the mean but it is not. If we want to fit this properly, we need two Gaussian models, clustering.

We can also overlap the Gaussian to fit the data. The numbers next to the gaussian in the left graph are weight. The right picture shows the fitted density with a 3D image.

Generative model

We can use GMM as a Generative Model. If you select the components and you know the parameter of the corresponding gaussian, then you can sample from the Gaussian. Actually, now we can change our minds. It is somehow intuitive and tricky.

Can we think the pi, weight, as a probability? Yes, we can. What does it mean? It means how likely we choose a specific model, it can be interpreted as how likely we choose a specific parameter set of the Gaussian distribution. Then,

This will be a likelihood. Zn is a categorical variable.

We can calculate the probability of the specific data like above. This is the equivalent generative formulation with an explicit latent variable Zn. We can illustrate this.

The shaded circle is the observed data. If we know pi, we can generate Zn. If we know Zn, we can generate new data. In our case, it is Gaussian distribution but it can be any distribution you want.

Let’s look at this with bayesian eyes. If you don’t familiar with it, you can check here.

I already define the likelihood above and we define other things here. We want to calculate the posterior, it means what is the probability of the parameter given data from component k, specific gaussian distribution. The posterior is also called responsibility. We can understand this as the specific data group how likely to be assigned in the specific class. However, Gaussians contribute to every data point with weight, we need to compare this with every other Gaussians. We can calculate this with Bayes’s rule.

Posterior is calculated by multiplying likelihood and prior and evidence is constant.

Bishop’s book

We assign the data point into the class where it has the biggest responsibility and color it. It will look like this. However, you should check the last picture has a blurred area where we cannot decide which cluster is the right cluster.

Parameter Estimation

We need to find out the covariance matrix, mixing coefficient, and mean vector. First of all, we assume that we know the K. I will explain how to choose k later. It is difficult to calculate because of the covariance matrix, it is p by p size. We assume i.i.d and we can conclude the likelihood is:

The log-likelihood will be:

If you are familiar with other log-likelihoods, you can feel weird in this equation because it is hard to differentiate the function. This is a problematic part of the parameter estimation. If you just go for it, you can get this equation, we set the derivative to zero:

If you differentiate it by yourself, you can get the responsibility term. The equation above contains responsibility terms. We still can not calculate the mean because we don’t know the responsibility, it contains every parameter. We can do something for covariance.

Same result for pi.

How do we solve this? We can do it by setting the initial parameter and calculate the log-likelihood and update the parameters with respect to responsibility, we can get it because we initialized the parameters. Repeat this until convergence. This is called the EM method.

Expectation Maximization

This is one way to do maximum likelihood estimation when you have missing data. EM is an iterative algorithm.

0. Initialize the parameters and evaluate the log-likelihood.

Compute the Posterior distribution of Z gives current estimator of parameters:

2. Compute Complete likelihood under this distribution of Z (Expectation):

3. We maximize this Q functions(Maximization):

What the algorithm does is simple. We don’t know the Z. We built the distribution, in GMM, the responsibility based on previous guesses, about Z, hidden variables, based on the previous guess. We calculate the expected value of complete log-likelihood under the distribution of the missing data using the previous guess, Expectation Step, and we maximize it with respect to the parameters for the next parameter, Maximization Step.

Expectation-Maximization for GMM

We assume that hidden variables and latent variables are equal. What we are finding now is covariance, mixing coefficient, and mean.

Again, you have to remember the important point in the E step, we calculate the expected value of complete log-likelihood under the Z distribution using the previous guess.

Now, we still got the responsibility. However, it is not about the true parameter. It is the responsibility based on guessed parameters. The equation is also easy to differentiate for M step.

Responsibility becomes constant.

Same for other parameters.

This is the whole picture of EM for GMM.

Theoretical Guarantee

EM monotonically increases observed data likelihood until the local maximum or saddle point. Let’s look at why it is the case. The context you should know is Jansen’s inequality. If we have convex function f, then:

The function of the linear combination of x will less than a linear combination of functions of x. What we need is when f is a log, concave function. It will be the opposite.

We calculate the log-likelihood but now we set q as an arbitrary distribution over the latent variables. We got lower bound with Jansen’s Inequality. Finally, we get the Q function, we got the expected value of log-likelihood minus entropy. We need to choose q to maximize the Q function. To sum up, we maximize the lower bound to touch the original likelihood.

I changed the lower bound formula into -KL divergence + Log-likelihood. Our goal is to make KL divergence zero because we want to consider log-likelihood. It turns out EM did the same process, making KL divergence zero.

This picture tells the process.

BIC & AIC

These are the popular information criterion, you can use both in general. We will use this to choose K in GMM. AIC consists of log-likelihood, which we maximized for a good model and the number of parameters, k in our case. We penalized the loglikelihood with the number of parameters because many parameters can lead to the perfect model but it is definitely overfitting. BIC is almost the same but it has weight for the number of parameters, a log of the number of observed data. More data, more penalty. This is not good to build a complex model even if you have enough data.

In this graph, I would like to say 3, 5 are a nice choice for k because the model is not that complex and nice AIC and BIC score.

Covariance Types

We can have different types of Gaussian distribution depending on the covariance matrices. If the covariance is always the same in every direction, then it will be spherical. It is a diagonal matrix but the same variances. You should know they don’t share the same covariance. The diagonal means the covariance matrix is diagonal but it is not the same. Tied means they share the covariance together.

This post is published on 9/24/2020.