EM Algorithm. Mathematical Background and Example | by Mengsay Loem | Towards Data Science

Working with a stochastic approach based-machine learning, we consider the information origin as a type of probability distribution. Once we estimate the distribution, it is straightforward to classifier unknown data as well as to predict future generated data. There are two phases to estimate a probability distribution. 1) Decide a model to define the distribution, for example, the form of probability density function (Gaussian distribution, Multinomial distribution...). Here, consider the Gaussian Mixture Model (GMM) as an example. The form of probability density function can be defined by
where w_k is the ratio data generated from the k-th Gaussian distribution.
2) After deciding a form of probability density function, we estimate its parameters from observed data. For example, in the case of Gaussian distribution, mean and variance are parameters to estimate.
Our current known knowledge is observed data set D and the form of generative distribution (unknown parameter Gaussian distributions). However, to solve 2) we need the information on which Gaussian distribution each observed data is generated from, and this information is not directly shown in the observed data.
In this case, the variables which represent the information that cannot be obtained directly from the observed data set are called Latent Variables. In the above example, w_k is a latent variable.
Therefore, in GMM, it is necessary to estimate the latent variable first. And next, we use the estimated latent variable to estimate the parameters of each Gaussian distribution.
An effective method to estimate parameters in a model with latent variables is the Expectation and Maximization algorithm (EM algorithm).
Let’s prepare the symbols used in this part.
D={x_i | i=1,2,3,...,N} : Observed data set of stochastic variable x : where x_i is a d-dimension vector.
z : Latent variable. z_i corresponds with x_i.
theta: Set of parameters to be estimated
Our purpose is to estimate theta from the observed data set D with EM algorithm.
In the case that observed data is i.i.d, the log-likehood function is
We can rewrite our purpose in the following form.
The probability shown in log-likelihood function p(x,z|theta) can be represented with the probability of latent variable z as the following form. The third relation is the result of marginal distribution on the latent variable z
Now, our goal is to determine the parameter theta which maximizes the log-likelihood function log p(x|theta). However, it is not possible to directly maximize this value from the above relation. Therefore, we decide a process to update the parameter theta while maximizing the log p(x|theta). We consider theta be the optimal parameter to be defined, theta(t) be the t-th step value of parameter theta. In the following process, we tend to define an update rule to increase log p(x|theta(t)) compare to log p(x|theta).
Here, we represent q(z) by conditional probability given recent parameter theta and observed data.
Hence, log p(x|theta) can be written as
We start by focusing on the change of log p(x|theta)-log p(x|theta(t)) when update theta(t). We try to define rule which lead to decrease the amount of log p(x|theta)-log p(x|theta(t)).
Rewrite this relation, we get the following form. Equation (1):
Now, we need to evaluate the right-hand side to find a rule in updating parameter theta. To do this, consider a well-known mathematical relationlog x ≤ x-1. Using this relation, we can obtain the following inequality.
Therefore, the 3rd term of Equation(1) is
The first and second term of Equation(1) is non-negative. It is true because, when we replace theta by theta(t), term1-term2=0 then by maximizing the first term, term1-term2 becomes larger or equal to 0.
Consider this relation, log p(x|theta)-log p(x|theta(t))≥0. Therefore, to maximize the left-hand side of Equation(1), we just update theta(t) with a value of theta(t) which maximizes the first term of the right-hand side of Equation(1).
Let the subject of argmax of the above update rule be function Q(theta).
We can translate this relation as an expectation value of log p(x,z|theta) when theta=theta(t).
From this update, we can summary the process of EM algorithm as the following E step and M step.
Let’s take a 2-dimension Gaussian Mixture Model as an example.
Random variable: x_n (d-dimension vector) Latent variable: z_mMixture ratio: w_kMean : mu_k (d-dimension vector)Variance-covariance matrix: Sigma_k (dxd matrix)
Here, if an observed data x is generated from m-th Gaussian distribution, then z_m = 1, else z_m = 0. The distribution of latent variable z, therefore can be written as
The probability density function of m-th Gaussian distribution is given by
Therefore, the probability which data x belongs to m-th distribution is p(z_m=1|x) which is calculated by
Randomly initialize mu, Sigma and w. t = 1.
Calculate p(z|x,theta(t))
Using theta(t) to calculate the expectation value of latent variable z.
Therefore, if z_nm is the latent variable of x_n, N_m is the number of observed data in m-th distribution, the following relation is true.
Update parameter
Our goal in this step is to define w_m, mu_m, Sigma_m which maximize Q(theta|theta(t)).
Solve this equation, the update of mu is
Solve this equation, the update of Sigma is
To derivate the update relation of w, we use Lagrange method to maximize Q(theta|theta(t)) subject to w_1+w_2+...+w_M=1.
Solving this equation for lambda and use the restraint relation, the update rule for w_m is
All Parameters Updates:
n1=150; n2=250; n3=300n = n1+n2+n3x = []x.append(np.dot(np.random.randn(n1,2),np.sqrt(np.diag([1,4]))))x.append(np.random.randn(n2,2)+np.dot(np.asarray([[1.]*n2]).T,np.asarray([[8.0,4.0]])))x.append(np.dot(np.random.randn(n3,2),np.sqrt(np.asarray([[2.45,1],[1,4]])))+16.0)X = np.concatenate((x[0],x[1],x[2]),axis=0)
def genSigma():    A = np.random.rand(2,2)+np.eye(2,2)    #A = np.eye(2,2)*np.random.rand(1)    S = np.dot(A.T,A)    #S = np.diag(np.diag(S))    return Sdef genMu(m):    mu = np.zeros((2,m))    mu[0] = np.linspace(np.min(X,axis=0)[0],np.max(X,axis=0)[0],m)    mu[1] = np.linspace(np.min(X,axis=0)[1],np.max(X,axis=0)[1],m)    return mu.TSigma = np.zeros((m,2,2))    for i in range(m):        Sigma[i] = (genSigma())Mu = genMu(m)        W = np.asarray([1.]*m)    W = W/sum(W)    L = -float('inf')iteration = 0
while True:        if iteration >=100:            break        iteration+=1# E step        wN = np.zeros((n,m))        for j in range(m):            for i in range(n):                wN[i,j]=W[j]*gauss(X[i],Sigma[j],Mu[j])sum_wN = np.sum(wN,axis=1)        Lnew = np.sum(np.log(sum_wN))        if (Lnew-L) < 0.00001:            print("Model Converged")            break        L = Lnewprint("iter#",iteration," Lnew:",Lnew)        iters.append(iteration)        Log.append(Lnew)# M stepeta = np.zeros((n,m))        for i in range(n):            eta[i,:]=wN[i]/sum_wN[i]sum_eta = np.sum(eta,axis=0)        W = sum_eta/nfor j in range(m):            tmp = np.zeros((2,))            for i in range(n):                tmp += eta[i,j]*X[i]            Mu[j]=tmp/sum_eta[j]for j in range(m):            tmp = np.zeros((2,2))            for i in range(n):                xmu = (X[i]-Mu[j]).reshape(2,1)                tmp1 = np.dot(xmu,xmu.T)*eta[i][j]                tmp += tmp1            Sigma[j]=tmp/sum_eta[j]
import numpy as npimport randomfrom scipy.interpolate import griddatafrom scipy.stats import multivariate_normalimport numpy.ma as mafrom numpy.random import uniform, seedfrom matplotlib import cmfrom scipy.special import softmaximport matplotlib.pyplot as plt# Gaussian pdf def gauss(x,sigma,mu):    n = multivariate_normal(mu,sigma)    return n.pdf(x)# Visualizedef plot_countour(sigma,mu):    xy_range = np.zeros((2,2))    xy_range[0] = np.linspace(mu[0]-2*sigma[0,0],mu[0]+2*sigma[0,0],2)    xy_range[1] = np.linspace(mu[1]-2*sigma[1,1],mu[1]+2*sigma[1,1],2)    x, y = np.mgrid[xy_range[0][0]:xy_range[0][1]:.01, xy_range[1][0]:xy_range[1][1]:.01]    pos = np.dstack((x, y))    rv = multivariate_normal(mu, sigma)    z = rv.pdf(pos)    levels = np.linspace(np.min(z),np.max(z),5)    plt.contour(x, y, z,linewidths=1.0,colors='black',levels=levels)    #plt.contourf(x,y,z,15,cmap=plt.cm.jet)    plt.xlim([-5,25])
From the result, with the EM algorithm, the log-likelihood function always converged after repeat the update rules on parameters. However, since the EM algorithm is an iterative calculation, it easily falls into local optimal state. As saw in the result(1),(2) differences in M value(number of mixture model) and initializations offer different changes in Log-likelihood convergence and estimate distribution. To solve this problem, a simple method is to repeat the algorithm with several initialization states and choose the best state from those works.