Group similar Image by using the Gaussian mixture model (EM algorithm) | by AKASH KUMAR | Towards Data Science

Clustering one of the most popular unsupervised machine learning problem. We are already familiar with k-means clustering algorithm but wait there is a problem in this:
Highly dependent on the initial value of centroids: If we change it ’s initialization then cluster position then it is highly likely that the final cluster will change its position.Clusters are the varying size and density: cluster which having difference shape (spread f data points) and size (no. of data points) can’t be addressed by this simple technique.Outliers Points: It may happen a data point that is very far from any of the clusters it can act as a separate cluster. To avoid such problems we normally removed those points.
Highly dependent on the initial value of centroids: If we change it ’s initialization then cluster position then it is highly likely that the final cluster will change its position.
Clusters are the varying size and density: cluster which having difference shape (spread f data points) and size (no. of data points) can’t be addressed by this simple technique.
Outliers Points: It may happen a data point that is very far from any of the clusters it can act as a separate cluster. To avoid such problems we normally removed those points.
If you don’t want to understand this nasty looking math then please scroll down to go to Implementation Of GMM section where I talk about implementation in python.
Density estimation: probability density estimation is the construction of an approximate based on observed data of an unobservable underlying probability density function. This step involves the selection of probability distribution function and the parameter of that function which describes the best joint probability distribution of the observed data.Maximum Likelihood Estimation (MLE): It is a method of estimating the parameters of probability distribution by maximizing a log-likelihood function.Latent variables: It is opposite to observed variables, i.e. not directly observed but rather inferred from the mathematical model. We will talk about it later.
Density estimation: probability density estimation is the construction of an approximate based on observed data of an unobservable underlying probability density function. This step involves the selection of probability distribution function and the parameter of that function which describes the best joint probability distribution of the observed data.
Maximum Likelihood Estimation (MLE): It is a method of estimating the parameters of probability distribution by maximizing a log-likelihood function.
Latent variables: It is opposite to observed variables, i.e. not directly observed but rather inferred from the mathematical model. We will talk about it later.
Now maximum likelihood estimation did not work in the presence of latent variables very well. The expectation-maximization algorithm is one method by which can find appropriate model parameters in the presence of latent variables.
EM is an iterative method to find maximum likelihood or maximum a posteriori estimates of parameters, where the model depends on unobserved latent variables.
Let simplify it a little bit.
Given N observation
and a probabilistic model p(X |θ)
Our goal is to find the value of θ which maximizes p(X |θ this is called maximum likelihood estimation.
IF the model is simple Gaussian distribution then parameter be like
where
If this is the case most common approach to solve this problem in ML is gradient descent by minimizing the log-likelihood function which is –log (p(X |θ)) as loss function (remember we actually maximizing the probability function at parameter
is a simple hack for numerical stability since it changes the product into sum and the minimization of negative of this function is the same as maximizing this function).
But Gradient descent is a method for solving nonlinear problems, typically for the minimum of some multi-dimensional distribution. Computing the mean (or other moments) of a Gaussian mixture is not a nonlinear problem and so does not require methods designed for nonlinear problems. So is there is any better way to solve this problem.
There comes the EM algorithm. Let see how the EM algorithm is used in the Gaussian mixture model.
Maximum Likelihood Estimation (MLE) can be simplified by introducing the Latent variable. A latent variable model makes the assumption that an observation xi is caused by some underlying latent variable. Hmmm... still not clear ok consider this image
This type of probabilistic model is called the Gaussian mixture model (GMM), a weighted sum of C Gaussian components in this example C = 3 is
πc and Σc are the weight, mean vector and covariance matrix of mixture component Cc, respectively. The weights are non-negative and sum up to 1 i.e.
Parameter vector
denotes the set of all model parameters. If we introduce a discrete latent variable ‘t’ that determines the assignment of observations to mixture components we can define a joint distribution over observed and latent variables p(x, t| θ) in terms of a conditional distribution p(x| t, θ) and a prior distribution p( t | θ)
Where
The values of t are one-hot encoded. For example, t2=1 refers to the second mixture component which means t = (0, 1, 0) if there are C=3 components in total. The marginal distribution p (x | θ) is obtained by summing over all possible states of t.
For each observation xi we have one latent variable ti, which is also called responsibility.
Considering X be all sets of observation, set of all latent variable T then we can easily maximize complete data log-likelihood p(X, T | θ). Since after finding log-likelihood we then get to know cluster assignments of points that is why we will find marginal log-likelihood or incomplete data log-likelihood p (X | θ).so mathematically,
Let see step by step how Our Image gets clustered by using a Gaussian Mixture Model. I am using python here for implementing GMM model:
External Python library required:
imageio: For fetching RGB features from Imagepandas: For handling datasetnumpy: For mathematical operations
imageio: For fetching RGB features from Image
pandas: For handling dataset
numpy: For mathematical operations
Step 1:
Let start from our dataset we have given a folder path where all the image resides. We will create a method from which we can extract the average value of R, G B vector for each Image. For single Image the function should be:
######## loading external package dependency ####################Import pandas as pdImport numpy as npfrom scipy.stats import multivariate_normalImport imageioFrom functools import reducedef get_image_feature(path):    Im = imageio.imread(os.path.join(path), pilmode=’RGB’)    temp = Im/255. # divide by 255 to get in fraction    mn = temp.sum(axis=0).sum(axis=0)/(temp.shape[0]*temp.shape[1])    return mn/np.linalg.norm(mn, ord=None) # taking 2nd norm to scale vector
Let provided image path is ‘(some_path)/Image_SunSet.jpg’ this function return [Red, Green, Blue] image pixels intensity be like [0.9144867979, 0.3184891297, 0.2495567485] we can clearly see Red color has more intensity.
Similarly, we can do this for all of our images and create a pandas Dataframe which contains Image details:
file_name: name of the file
path: the path of a file
extension: Image Extension like .jpg, .png e.t.c.
ImageBase64: Base64 encode of Image
red: Red color Average Intensity
green: Green color Average Intensity
blue: Blue color Average Intensity
Now we have the dataset contain features (i.e. [Red, Green, blue] column of the dataset, not all of the rest of the data use for View only). If you want to go for a higher number of k (number of clusters) you can create your own feature extracted from an image but represent a wider range of color code not just r, g, and b. but for sake of simplicity, I took these 3 primary color vector as features.
Step 2:
Model Generation (Implementing the EM algorithm for Gaussian mixture models):
Since in GMM we assume all the data points collectively coming from k Gaussian distribution where k is nothing but a number of clusters. Hmmm... bit tricky let simplify it a bit.
Consider this 1D probability distribution. If we assume all the data points coming from 2 Gaussian distribution that means we assume k=2(number of clusters) then if we want to know which data came from which Gaussian distribution, all we need to do is to compute the mean and the variance for both the Gaussian distribution. But wait we don’t know yet. But for a second if we assume we know these parameters (i.e. mean and variance of k Gaussian distribution) then we can know the likelihood of each data point from which Gaussian it is coming (this likelihood refers to the responsibilities). Wow, but we don’t know this either. So our conclusion is
If we know the parameter (i.e. mean and variance of Gaussian) we can calculate the responsibilities and if we know the responsibility then we can calculate the parameters.
That’s where EM algorithm comes where it starts by placing ‘k’ Gaussians randomly (generate random mean and variance for the Gaussians) into the picture where it iterates between E and M step until it converges where:
E- step: assign cluster responsibilities, given current parameters
M-step: Update parameters, given current cluster responsibilities
Let’s talk about its implementation in more detail.
Let’s remember why GMM is used for. Unlike k- mean it does not do hard assignments to the cluster, say an example if an image
If we are working with k=4 i.e. 4 clusters model i.e. Sunset, forest, cloud, and sky then In k mean algorithm this picture belongs only to cloud this is called a hard assignment but in GMM it belongs to 97% cloud and 3% forest this is called a soft assignment.
E step:
In this step, we compute cluster responsibilities. Let rik represent responsibility of cluster k for data point i. You can assume responsibility as the probability of that data point ‘i’ belonging to the ‘k’ cluster. Since it’s a probability that means
To know how much a cluster is responsible for a given data point, we compute the likelihood of the data point under the particular cluster assignment, multiplied by the weight of the cluster. For data point i and cluster k, mathematically:
Where N (xi | μk, Σk) is the Gaussian distribution for cluster k (with mean μk and covariance Σk).
We used ∝ because the quantity N (xi | μk, Σk) is not yet the responsibility we want. To ensure that all responsibilities over each data point add up to 1, we add the normalization constant in the denominator:
But our data is 3 dimension vector i.e. [red, green, blue]. Scipy provides a convenient function to calculate multivariate normal distribution. check this
which is multivariate_normal.pdf([data point], mean=[mean vector], cov = [covariance matrix])
We will talk about later how to calculate the mean vector and covariance matrix in the M step.
# data (numpy array) : array of observations# weights (numpy array) : numpy array of weight of each clusters of size (1, n_clusters)#means (numpy array) : numpy array of means of each clusters of size (n_cluster, dimension)#covariances(numpy array) : numpy array of covariance metrix of size (n_clusters, dimension, dimension)def get_responsibilities( data, weights, means, covariances):    n_data = len(data)    n_clusters = len(means)    resp = np.zeros((n_data, n_clusters))    for i in range(n_data):       for k in range(n_clusters):          resp[i, k] = weights[k]*   multivariate_normal.pdf(data[i],means[k],covariances[k],allow_singular=True)        # Add up responsibilities over each data point and normalize    row_sums = resp.sum(axis=1)[:, np.newaxis]    resp = resp / row_sums    return resp
Remember this responsibility is our latent variable.
M-Step:
Now cluster responsibility is calculated, we have to update our cluster parameters i.e. (weights (πk), means (μk), and covariances (Σk ) ) associated with each cluster.
Update Weights:
The cluster weight gives how much each cluster represents all data points. Mathematically it is defined as:
python code
# resp(numpy array) : responsibility numpy array size (n_sample, n_clusters)def get_soft_counts(resp):    return np.sum(resp, axis=0)# counts (numpy array) : count list of sum of soft counts for all clusters of size (n_cluster)def get_weights(counts):    n_clusters = len(counts)    sum_count = np.sum(counts)    weights = np.array(list(map(lambda k : counts[k]/sum_count, range(n_clusters))))    return weights
Updates means:
The mean of each cluster is equal to the weighted average of all data points, weighted by the cluster responsibilities. Mathematically for each data point xi and responsibilities for kth cluster rik is Mean of k-th cluster can be defined as:
python code
# data (numpy array): array of observation points# resp (numpy array) : responsibility numpy array size (n_sample, n_clusters)# counts (numpy array) : count list of sum of soft counts for all clusters of size (n_cluster) def _get_means( data, resp, counts):    n_clusters = len(counts)    n_data = len(data)    means = np.zeros((n_clusters, len(data[0])))        for k in range(n_clusters):        weighted_sum = reduce(lambda x,i : x + resp[i,k]*data[i],  range(n_data), 0.0)        means[k] = weighted_sum/counts[k]    return means
update covariance:
The covariance of each cluster is equal to the weighted average of all outer products weighted by the cluster responsibilities. Mathematically it is defined as:
Where
is the outer product. Let take a simple example of what outer product looks like.
Let
be the two vector then the outer product of a and b is defined as:
Transpose is for just to make the column vector to row vector. Since by convention vector represented as a column vector in machine learning.
python code
# data (numpy array) : array of observation points# resp(numpy array) : responsibility numpy array size (n_sample, n_clusters)# counts (numpy array) : count list of sum of soft counts for all clusters of size (n_cluster)# means (numpy array) : numpy array of means of each clusters of size (n_cluster, dimension)def _get_covariances( data, resp, counts, means):    n_clusters = len(counts)    dimension = len(data[0]) # to get dimention of data    n_data = len(data)    covariances = np.zeros((n_clusters, dimension, dimension))        for k in range(n_clusters):       weighted_sum = reduce (lambda x, i :x + resp[i,k] *  np.outer((data[i]-means[k]), (data[i]- means[k]).T), range(n_data), np.zeros((dimension, dimension)))              covariances[k] = weighted_sum /counts[k] # normalize by total sum of counts    return covariances
But how we will measure our mixture of Gaussian model. For this, we will calculate the log-likelihood of the Gaussian mixture. It Quantifies the probability of observing a given set of data under particular parameters i.e. means, covariance, and weights. We will keep iterating EM algorithms with different sets of parameters and check for its convergence i.e. log-likelihood doesn’t change too much or till fix number of iterations.
Python code
# Z (numpy array) : numpy array of size (1, n_clusters)def _ln_sum_exp( Z):    return np.log(np.sum(np.exp(Z)))# data (numpy array) : array of observation points# weights (numpy array) : numpy array of weight of each clusters ofsize (1, n_clusters)# means (numpy array) : numpy array of means of each clusters of size (n_cluster, dimension)# covs (numpy array) : numpy array of covariance metrix of size (n_clusters, dimension, dimension)def get_log_likelihood( data, weights, means, covs):    n_clusters = len(means)    dimension = len(data[0])    sum_ln_exp = 0        for d in data:            Z = np.zeros(n_clusters)      for k in range(n_clusters):        # compute exponential term in multivariate_normal        delta = np.array(d) — means[k]                inv = np.linalg.inv(covs[k])        exponent_term = np.dot (delta.T, np.dot (inv, delta))        # Compute loglikelihood contribution for this data point and this cluster        Z[k] += np.log (weights[k])        det = np.linalg.det(covs[k])                Z[k] -= 1/2. * (dimension * np.log (2*np.pi) + np.log (det) + exponent_term)        # Increment loglikelihood contribution of this data point across all clusters            sum_ln_exp += _ln_sum_exp(Z)    return sum_ln_exp
Now It’s time for the actual main method of Implementation of EM algorithms
Python code
# data (numpy array) : array of observation points# init_weights (numpy array) : numpy array of initial weight of each clusters ofsize (1, n_clusters)# init_means (numpy array) : numpy array of initial means of each clusters of size (n_cluster, dimension)# init_covariances (numpy array) : numpy array of initial covariance metrix of size (n_clusters, dimension, dimension)# maxiter (int) : maximum iteration to rum EM (optional)# threshold (float) : maximum threshold to stop EM (optional)def em_from_parameter(data, init_means, init_covariances, init_weights, maxiter=1000, thresh=1e-4):    # Make copies of initial parameters    means = init_means[:]    covariances = init_covariances[:]    weights = init_weights[:]        # Infer length of dataset    n_data = len(data)        # Infer number of cluster    n_clusters = len(means)        # Initialize some useful variables    resp = np.zeros((n_data, n_clusters), dtype=np.float64)        l1 = get_log_likelihood(data, weights, means, covariances)    l1_list = [l1]        for it in range(maxiter):            # E-step: calculate responsibilities            resp = get_responsibilities(data, weights, means, covariances)            # M-step calculate cluster parameter      counts = get_soft_counts(resp)      weights = _get_weights(counts)      means = _get_means(data, resp, counts)      covariances = _get_covariances(data, resp, counts, means)            l1_new = get_log_likelihood(data, weights, means, covariances)            l1_list.append(l1_new)            if abs(l1_new – l1) < thresh :        break            l1 = l1_new        param = {'weights': weights, 'means': means, 'covariances': covariances, 'loglikelihood': l1_list, 'responsibility': resp, 'Iterations': it}    return param
So, all set now but what will be our initial parameter. There are many approaches we can go with.
One way is to initialize all clusters weights equally i.e. (1/k) for each cluster where k is the total number of clusters.
Initialization of covariance matrix for 1 cluster is the square diagonal matrix of dimension equal to the dimension of data * dimension of data (in this case dimension of data is 3 i.e. [r, g, b]) mathematically
Where σxy = is the covariance between x and y
For mean initialization, it’s a bit tricky. Remember the k-mean algorithm. We use the output of these algorithms to initialize our EM mean parameter. Since by using the k-mean algorithm we push mean co-ordinate to reach the near point of convergence. So if we start from this point EM algorithm will take less time to converge. But there is a problem with how we will initialize initial centroids for k-mean.
Hmmm. Well for this problem we will go with the k mean++ initialization strategy in which it selects k centroids farthest from each other. Now we have all the three-parameter to start with.
NOTE: For Full solution look my git-hub link PYTHONAPI you will find full Flask code for this clustering problem End to End and can be consumed by any client-side technology. I select Angular 8 as client-side which code is available on ML_algorithmUI.
For those who want to check only EM related code, the link is this one which contains two folders Expectation_Maximization and K_Mean which contain the relative code separately. Please go through the code and feel free to ask me if you face any problems.
Now coming back to code so, driver code of this program
Python code
# k: number of clusters# data: observations coming from distribution# return final weights, means, covariances, list of loglikelihood till convergence and final responsibility of each observation and number of Iterations it takes till convergencedef em(k, data) :    dimension = len(data[0])    # mean co-ordinate of all clusters not recommended at all since it is a very bad method to initialized the mean points alternatively use the k-mean initializing method as discussed early    means = np.random.rand(k, dimension)    #### alternate better method to initialize mean points by kmean ####    ## from KM import KM ## self created KM class available in my github repos    ## check K_Mean folder in listed github repos path    km = KM()    means, dont_require = km.kmeans(data, k) # return centroids and cluster assignments        cov = np.diag(np.var(data, axis=0)) # get covariance of data initialize by diagonal metrix of covariance of data        covariances = np.array([cov] * k) # initiate covariance metrix with diagonal element is covariance of data    weights = np.array([1/ k] * k) # initialize equal weight to all clusters    return em_from_parameter(data, means, covariances, weights)
images_path = [‘../image2.jpg’, ‘../image2.jpg’, ... ] # list of image path .................................................................................................(1)
########### Data generation ################data = []For path in images_path:data.append(get_image_feature(path))data = np.array(data)
Now our data is ready and the value of k i.e. number of the cluster is all personal choice. For simplicity I take k = 4 means we are assuming the distribution of data is coming from 4 Gaussian. Since our data and value of k are ready then we are ready to create a model.
####### create model #######parameters = em(k, data) # store returned parameters of model along with responsibilities
. . . . . . .....................................................................................(2)
It will take time don’t worry but in the end, you got what you want. Now if you have new observation data you can find out the soft assignments since we already got the correct cluster parameters.
Python code
# data (numpy array) : array of observation points# weights (numpy array) : numpy array of weight of each clusters of size (1, n_clusters)# means (numpy array) : numpy array of means of each clusters of size (n_cluster, dimension)# covs (numpy array) : numpy array of covariance metrix of size (n_clusters, dimension, dimension)def get_responsibilities(self, data, weights, means, covariances):    n_data = len(data)    n_clusters = len(means)        resp = np.zeros((n_data, n_clusters))for i in range(n_data):for k in range(n_clusters):resp[i, k] = weights[k]* multivariate_normal.pdf(data[i],means[k],covariances[k])# Add up responsibilities over each data point and normalizerow_sums = resp.sum(axis=1)[:, np.newaxis]resp = resp / row_sumsreturn resp
create your numpy array of observations and call this method with returned parameter by em as shown in (2) and pass it to this function you will get the soft assignment of each cluster. For example, let you have an image with path “../../test_example.jpg”
first fetch [r, g, b ] vector of the image as shown in (1) you will get something like [0.9144867979, 0.3184891297, 0.2495567485 ] pass this data to get_responsibilities function along with the returned parameters of clusters. You will get this in parameters variable i.e
note: remember this function takes an array of observation and in this case, we have only one image i.e. one observation so convert this single observation to an array first then call the function.
data = np.array([ np.array([0.9144867979, 0.3184891297, 0.2495567485 ])])mean = parameters[‘means’]cov = parameters[‘covariances’]weights = parameters[‘weights’]resp = get_responsibilities(data, weights, mean, cov)
Hurray, you got your cluster assignments. I made a web-based result visualization of some of the screenshots.
Let’s talk about how it removes all disadvantages of k mean:
Better Initialization strategy: Since we are using K mean++ initialization strategy so it is very less likely that we will end up getting wrong cluster formations.Varying cluster density: Unlike k mean it has varying cluster density which gives us probability wise estimation of observations with their respective cluster assignments.Outlier points: outliers are taken care of consideration as it contains very fewer responsibilities to its closest cluster.
Better Initialization strategy: Since we are using K mean++ initialization strategy so it is very less likely that we will end up getting wrong cluster formations.
Varying cluster density: Unlike k mean it has varying cluster density which gives us probability wise estimation of observations with their respective cluster assignments.
Outlier points: outliers are taken care of consideration as it contains very fewer responsibilities to its closest cluster.
conclusion: So coming to end we can say that any image with the primary set of colors can be cluster with this technique. We can use this approach to cluster Image as many numbers as we want. Here I just took a simple example with primary color but according to need feature generation may be altered with multiple shades of colors. GMM is best suited for those cases where you want the probability of image assignment to a particular cluster.
Now it’s time to close my blog post If you are really interested in this code please visit my git-hub link and feel free to contribute and give suggestions for improvements. Happy learning....
Links and Reference:
Martin Krasser Latent variable Model Part-1 Gaussian Mixture model and EM algorithm (November 21, 2019): https://krasserm.github.io/2019/11/21/latent-variable-models-part-1/
ethen8181: http://ethen8181.github.io/machine-learning/clustering/GMM/GMM.html
Wiki reference: https://en.wikipedia.org/wiki/Expectation–maximization_algorithm
Google Developers resources: https://developers.google.com/machine-learning/clustering/algorithm/advantages-disadvantages