## 5 Gaussian Mixture Model Modeling Using Data-Parallel Primitives  
The Gaussian Mixture Model is a parametric distribution representation. It can represent a complicated distribution using a few parameters to provide a compact and accurate distribution representation. Therefore, GMM has been widely used to facilitate scientific data reduction and visualization [2; 8; 17; 18; 19; 37; 38]. GMM is an extension of a single Gaussian distribution; it is a statistical model that decomposes the distribution of samples into the weighted sum of K Gaussian distributions. GMM expresses the weighted sum of these K Gaussian components by Equation (2). \[p(x|\Theta)=\sum_{i=1}^{K}w_{i}*g(x|\mu_{i},\Sigma_{i}) \tag{2}\]  
where \(x\) is the D-variant sample, \(w_{i}\) is the weight of the \(i^{th}\) Gaussian component, and \(\sum_{i=1}^{K}w_{i}=1\), \(g(x|\mu_{i},\Sigma_{i})\) is the probability density of \(x\) on the D-variate Gaussian distribution with the mean vector, \(\mu_{i}\), and the covariance matrix, \(\Sigma_{i}\). The definition of \(\Theta\) is represented by Equation (3), which is called the parameter set of the Gaussian Mixture Model. \[\Theta=\{w_{i},\mu_{i},\Sigma_{i}\},i=1,2,\ldots,K. \tag{3}\]The expectation-maximization (EM) algorithm is generally used to estimate parameters of a GMM of the given samples. The EM algorithm estimates the parameters by maximizing the following likelihood function through an iterative process:  
\[\operatorname*{arg\,min}_{\Theta}L(\Theta)=\sum_{i=1}^{N}\sum_{k=1}^{K}w_{k}*g( x_{i}|\mu_{k},\Sigma_{k}) \tag{4}\]  
where \(\Theta\) represents the parameters of GMM and \(x_{i}\) is the \(i^{th}\) input sample. \(K\) is the number of Gaussian components. \(w_{k}\), \(\mu_{k}\), and \(\Sigma_{k}\) represent the weight, mean vector, and covariance matrix of the \(k^{th}\) Gaussian component, respectively. Algorithm 3 is the pseudo code of the serial version of the multi-set EM algorithm for GMM modeling. Each set of input samples will be modeled by a GMM. The input array, \(Samples\), contains samples from all sets. As the input samples could be multi-variant, an input sample, \(Sample[i][j]\), is a vector whose length is determined by the number of variables of the input dataset. \(Samples[i]\) is all input samples of \(i^{th}\) set and \(size(Samples[i])\) is the total number of input samples of the \(i^{th}\) set. The loop at Line 5 iterates through each set to fit the samples to a GMM with \(K\) Gaussian components. ```
1:Input : \(Samples[TotalSets]\)
2:Output : \(Weight[TotalSets][K]\)
3:Output : \(Mean[TotalSets][K]\)
4:Output : \(ConMatrix[TotalSets][K]\)
5:for\(s=1,2,...,TotalSets\)do
6:/*/



In the initialization step, we set the responsibilities between all input samples and Gaussian components. The responsibilities are stored in the \(Resp\) array. \(Resp[i][k]\) can be interpreted as the probability that the \(Sample[i]\) is generated by the \(k^{th}\) Gaussian component. Therefore, responsibilities between \(i^{th}\) sample and all Gaussian components must satisfy \(\sum_{j=1}^{K}Resp[i][j]=1\). \(Resp\) will be updated through the EM algorithm and used to update the parameters of GMMs. The EM algorithm will iterate the following three steps to update \(Resp\), and thereby estimate the parameters \(\Theta\) of GMM of each set:

* M-step: use the current \(Resp\) to estimate the parameter \(\Theta\) (Line 10). * E-step: compute the maximum likelihood of current GMM, and update responsibilities between all samples and current Gaussian components of the corresponding GMM, and store responsibilities in \(Resp\) (Line 31). * Check whether the maximum likelihood has converged (Line 48). When the algorithm is finished, the estimated weights, mean vectors, and covariance matrices of all GMMs are stored in \(Weight\), \(Mean\), and \(CovMatrix\), respectively. ### Input and Output Arrays

In this section, we introduce the proposed Data-Parallel Primitive (DPP) version of the multi-set EM algorithm.