# Gaussian Mixture Models Based on Principal Components and Applications

Authors: Nada A. Alqahtani; Zakiah I. Kalantan
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

Data scientists use various machine learning algorithms to discover patterns in large data that can lead to actionable insights. In general, high-dimensional data are reduced by obtaining a set of principal components so as to highlight similarities and differences. In this work, we deal with the reduced data using a bivariate mixture model and learning with a bivariate Gaussian mixture model. We discuss a heuristic for detecting important components by choosing the initial values of location parameters using two different techniques: cluster means,k-means and hierarchical clustering, and default values in the “mixtools” R package. The parameters of the model are obtained via an expectation maximization algorithm. The criteria from Bayesian point are evaluated for both techniques, demonstrating that both techniques are efficient with respect to computation capacity. The effectiveness of the discussed techniques is demonstrated through a simulation study and using real data sets from different fields.


---

## Body

## 1. Introduction

In real data such as engineering data, efficient dimension reduction is required to reveal underlying patterns of information. Dimension reduction can be used to convert data sets containing millions of functions into manageable spaces for efficient processing and analysis. Unsupervised learning is the main approach to reducing dimensionality. Conventional dimensional reduction approaches can be combined with statistical analysis to improve the performance of big data systems [1]. Many dimension reduction techniques have been developed by statistical and artificial intelligence researchers. Principal component analysis (PCA), introduced in 1901 by Pearson [2], is one of the most popular of these methods. The main purpose of PCA is to reduce the dimensionality of a data set consisting of a large number of interrelated variables, while retaining as much as possible of the variation present in the data set. Among the many PCA methods, singular value decomposition is used in numerical analysis and Karhunen–Loève expansion in electrical engineering. Eigenvector analysis and characteristic vector analysis are often used in the physical sciences. In image analysis, the Hotelling transformation is often used for principal component projection.In recent years, there has been increasing interest in PCA mixture models. Mixture models provide a useful framework for the modelling of complex data with a weighted component distribution. Owing to their high flexibility and efficiency, they are used widely in many fields, including machine learning, image processing, and data mining. However, because the component distributions in a mixture model are commonly formalized as probability density functions, implementations in high-dimensional spaces are constrained by practical considerations.PCA mixture models are based on a mixture-of-experts technique, which models a nonlinear distribution through a combination of local linear submodels, each with a fairly simple distribution [3]. For the selection of the model, a PCA mixture model was proposed by Kim, Kim, and Bang [4], which has a more straightforward expectation maximization (EM) calculation, does not require a Gaussian error term for each mixture component, and uses an efficient technique for model order selection. The researchers applied the proposed model to the classification of synthetic data and eye detection [4].For multimode processes, the Gaussian mixture model (GMM) was developed to estimate the probability density function of the process data under normal operating conditions. However, in the case of high and collinear process variables, learning from process data with GMM can be difficult or impossible. A novel multimode monitoring approach based on the PCA mixture model was proposed by Xu, Xie, and Wang [5] to address this issue. In this method, first, the PCA technique is applied directly to each Gaussian component’s covariance matrix to reduce the dimension of process variables and to obtain nonsingular covariance matrices. Then, an EM algorithm is used to automatically optimize the number of mixture components. A novel process monitoring scheme for the detection of multimode processes was developed using the resulting PCA mixture model. The monitoring performance of the proposed approach has been evaluated through case studies [5].In recent years, hyperspectral imaging has become an important research subject in the field of remote sensing. An important application of hyperspectral imaging is the identification of land cover areas. The rich content of hyperspectral data enables forests, urban areas, crop species, and water supplies to be recognized and classified. In 2016, Kutluk, Kayabol, and Akan [6] proposed a supervised classification and dimensionality reduction method for hyperspectral images, using a mixture of probability PCA (PPCA) models. The proposed mixture model simultaneously allows the reduction of dimensionality and spectral classification of the hyperspectral image. Experimental findings obtained using real hyperspectral data indicate that the proposed approach results in better classification than the state-of-the-art methods [6].In the field of face recognition, Ahmadkhani and Adibi [7] proposed a supervised version of the PPCA mixture model. This model provides a number of local linear underlying data samples. The underlying manifolds are used for face recognition applications to achieve dimensionality reduction without loss of information.In this work, we reduce the dimensions of data by applying a PCA technique and then deal with the reduced data or principal components scores using one GMM. Then, we obtain estimates of the parameters using an EM algorithm. Finally, we compare the selection of initial values for location parameters in the mixture model using three different techniques:k-means, hierarchical clustering, and default values in the “mixtools” package.The rest of this paper is organized as follows. Section2 briefly defines the concept of PCA. In Section 3, the mixture densities are discussed. Section 4 provides the probability density function of the Gaussian mixture distribution and the EM algorithm that is used to estimate the mixture’s parameters. Section 5 shows a PCA mixture model based on the proposed scenario. In Section 6, the experimental results are presented in three subsections, and the main conclusions are listed Section 7.


## 2. Principal Component Analysis

Suppose we havep-dimensional vectors and need to reduce them to a q-dimensional subspace. The reduction can be achieved by projecting the original vectors on to q dimensions, the principal components, which span the subspace. Suppose that X is a vector of p random variables. To find the principal components, we compute several linear functions of X with maximum variance; most of the variation in X will be accounted for by m principal components, where m≪p. The PCA method determines the correlation between PCA components and data variables; a high correlation indicates important variables. Let Σ be the known covariance matrix for the random variable X. For k=1,2,…,p, the kth PC is given by zk=αkx‵, where αk is an eigenvector corresponding to the kth largest eigenvalue λk. If αk is chosen to have unit length, αkαk‵=1 and varzk=λk. Then, the technique of Lagrange multipliers can be used, by maximizing(1)α1‵Σα1−λα1‵α1−1,(2)α1‵Σα1=α1‵λα1=λα1‵α1=λ,and choosing λ to be as large as possible. Here, α1 is the eigenvector corresponding to the largest eigenvalue of Σ, that is, the first principal component of x. In general, the kth principal component of x is αkx‵, and varαkx‵=λk. The second principal component, α2‵x, maximizes α2‵Σα2, subject to being uncorrelated with α1x`. The uncorrelation constraint can be expressed using any of the following equations:(3)covα1x‵,α2x‵=α1‵Σα2=α1‵Σα1=α2‵λ1α1‵=λ1α2‵α1=λ1α1‵α2=0.If we choose equation (3), we can write a Lagrange to maximize α2 as follows:(4)α2‵Σα2−λα2‵α2−1−ϕα2‵α1.Differentiation of this quantity with respect toα2 gives us(5)Σα2−λα2−ϕα1=0.Next, left multiplyingα1‵ into this expression, we have(6)α1‵Σα2−λα1‵α2−ϕα1‵α1=0,where, as mentioned above, the first two terms equal zero and α1α1‵=1, resulting in ϕ=0. Therefore, Σα2−λα2=0, or Σ−λIpα2=0, is another eigenvalue equation, and we use the same strategy of choosing α2 to be the eigenvector associated with the second largest eigenvalue that yields the second principal component of x, namely, α2‵x [8, 9].


## 3. Mixture Densities

A mixture density is defined as a weighted sum ofk component densities [9, 10]. Denote the jth component density by px;θj, where θj indicates the component parameters. We use πj to denote the weighting factor or “mixing proportions” of the jth component in the combination, with the constraints that πj≥0 and ∑j=1Kπj=1, and pj represents the probability that a data sample belongs to the jth mixture component. A K component mixture density is then defined as(7)px=∑j=1Kπjpx;θj,j=1,…,K.The mixture model has a vector of parameters,θ¯=θ1,…,θk,π1,…πk.We consider a mixture density to model a process by selecting a “source” according to the multinomial distributionπ1,…πk and then drawing a sample from the corresponding component density px;θj. Therefore, the probability of selecting source j and datum x is πjpx;θj. Equation (7) gives the marginal probability of selecting datum x. We can think of the source that generated a data vector x as “missing information”; that is, given a data point x, we want to infer which source it is likely to belong to. Section 4 presents the EM algorithm, which is used to iteratively estimate this missing information [11, 12].In mixture models, we deal with hidden variables as a latent variable, denoted byZ. It takes values 1,…,K as a discrete set satisfying zkϵ0,1 and ∑zzk=1. We define the joint distribution px,z in terms of a marginal distribution pz and a conditional distribution pxz. Generally, in the mixture model, we first choose a sample z from a multinomial distribution and then draw observations for sample x from a distribution that depends on z, i.e.,(8)pz,x=pzpxz.The marginal distribution overz is specified in terms of the mixing coefficients πk, such that,(9)pzk=1=πk.


## 4. Mixtures of Gaussians

The probability density function ofX is defined as(10)pxμk,Σk=12πΣ−1exp−12x−μxΣx−1x−μxT,where μx is a vector of means, μx1,…,μxN and Σx is an N×N covariance matrix.The Gaussian mixture distribution can be written as a linear superposition of Gaussians in the form(11)px=∑k=1Kπkpxμk,Σk.Now, the conditional distribution ofx given a particular value of z is a Gaussian:(12)pxzk=1=pxμk,Σk,(13)pxz=∏k=1Kpxμk,Σkzk.The marginal distribution ofx is obtained by summing the joint distribution over all possible states of z to give(14)px=∑zpzpxz=∑k=1Kπkpxμk,Σk.An important derived quantity is the “posterior probability” on a mixture component for a given data vector [11]:(15)γzk≡pzk=1x=pzk=1pxzk=1∑j=1Kpzj=1pxzj=1,=πkpxμk,Σk∑j=1Kπjpxμj,Σj.In the example shown in Figure1, the resulting distribution is bimodal, suggesting that the data come from two different sources. In the figure, the red and green lines indicate two components of the Gaussian mixture distribution.Figure 1
Example of density modelled by a mixture of two Gaussian probability density functions.


### 4.1. EM for Gaussian Mixtures

The EM algorithm is an estimation method used to find the estimators of maximum likelihood when a data set has missing values or latent variables. In this work, we assume a GMM with a fixed number ofk and that are known a priori.The EM is obtained as follows:(1)
Initialize the meansμk, covariances Σk, and mixing coefficients πk and evaluate the initial value of the log-likelihood.(2)
E step: evaluate the responsibilities using the current parameter values:(16)γznk=πkNxnμk,Σk∑j=1KπjNxnμj,Σj.(3)
M step: re-estimate the parameters using the current responsibilities:(17)μknew=1Nk∑n=1Nγznkxn,(18)Σknew=1Nk∑n=1Nγznkxn−μknewxk−μknewT,(19)πknew=NkN,where(20)Nk=∑n=1Nγznk.Evaluate the log-likelihood:(21)lnpXμ,Σ,π=∑n=1Nln∑k=1KπkNxnμk,Σk,and check for convergence of either the parameters or the log-likelihood. If the convergence criterion is not satisfied, return to step 2 [13].


## 4.1. EM for Gaussian Mixtures

The EM algorithm is an estimation method used to find the estimators of maximum likelihood when a data set has missing values or latent variables. In this work, we assume a GMM with a fixed number ofk and that are known a priori.The EM is obtained as follows:(1)
Initialize the meansμk, covariances Σk, and mixing coefficients πk and evaluate the initial value of the log-likelihood.(2)
E step: evaluate the responsibilities using the current parameter values:(16)γznk=πkNxnμk,Σk∑j=1KπjNxnμj,Σj.(3)
M step: re-estimate the parameters using the current responsibilities:(17)μknew=1Nk∑n=1Nγznkxn,(18)Σknew=1Nk∑n=1Nγznkxn−μknewxk−μknewT,(19)πknew=NkN,where(20)Nk=∑n=1Nγznk.Evaluate the log-likelihood:(21)lnpXμ,Σ,π=∑n=1Nln∑k=1KπkNxnμk,Σk,and check for convergence of either the parameters or the log-likelihood. If the convergence criterion is not satisfied, return to step 2 [13].


## 5. PCA of Gaussian Mixture Model

In this section, we present the steps of the proposed method. These steps are also illustrated in Figure2.(1)
Use the PCA technique to reduce the dimensionality of ap-dimensional data set. To find the principal components, we first obtain the eigenvalues and eigenvectors of the covariance matrix. The eigenvalues are the principal components. The total number of principal components corresponds to the total number of variables in the data set.(2)
Choose them eigenvectors that correspond to the m largest eigenvalues, where m is the number of dimensions of the new feature subspace m≤p.(3)
Transform the original data set to the lower-dimensional version.(4)
Use thek-means clustering method to partition the data set into clusters. When using k-means, it is important to determine the correct number of clusters for the data.(5)
The new data are modelled by a mixture of Gaussians with parameters set to assumed initial values.(6)
Use the EM algorithm to estimate the unknown parameters representing the mixing proportions between the Gaussians and the means and covariances of each.(7)
Use the Bayesian information criterion (BIC) test to assess the fit of the model; BIC is a model selection among a finite set of models, where the model with the lowest BIC is preferred [14].Figure 2
PCA of Gaussian mixture model.


## 6. Experimental Results

To study the effectiveness of the proposed method, we consider two scenarios. In the first scenario, the mixture model is fitted to the reduced data produced by the PCA method. In the second scenario, the clustering method is applied to the reduced data, and then the mixture model is fitted to the new data by taking the cluster means as initial values for the means in the mixture model. We use different types of data sets. The method is implemented on a scaled data set, and the results are illustrated in the following sections. We use the “stats”, “mclust”, and “mixtools” R packages to implement this method [15–17].


### 6.1. Simulation Case

We implement the proposed method on simulation data with different sample sizes: 50, 100, and 500. Consider a data set of four variables defined as follows:(22)X1∼N10,1.52,(23)X2∼N0,32,(24)X3=X1∗X2,(25)X4=0.5Z1+0.3Z2+0.7Z3+N0,1,where Z1,Z2, and Z3 are scaled variables of X1,X2, and X3 , respectively. The simulated data consist of four variables, X1,X2,X3,andX4. The mean values for each variable in the simulated data are μ1=10.003, μ2=−0.1651, μ3=−1.287, and μ4=0.032.The implementation includes a graphical plot of the data set as displayed in Figure3, which shows a pairs plot of the simulation data and its three-dimensional surface. As a first step, we applied the PCA method; the results are summarized in Table 1, which presents the total variance of components. Practically, PCA describes the data in few variables without loss of information. As shown in Figure 4(a) and Table 1, two components explained 93% of the total variance. We considered those two components to comprise a new data set denoted RD data, which contained 93% of the information of the original data. The empirical distribution of the RD data is presented in Figure 4(b).Figure 3
Simulation case: (a) pairs plot of data and (b) three-dimensional surface of data.
(a)(b)Table 1
The results of PCA implementation on the simulation study.
Importance of components:PC1PC2PC3PC4Standard deviation1.61051.06580.510730.09656Proportion of variance0.64840.28400.065210.00233Cumulative proportion0.64840.93250.997671.00000Figure 4
Simulation case: (a) scree plot of PCA and (b) plot of reduced data.
(a)(b)Hence, we fitted a two-component bivariate mixture model to the RD data. Figure5(a) displays the fitting of the two components GMM on the new data (with 500 data points); the plot specifies each component’s mean and sigma values. In order to estimate the density parameters for each component, we used an EM algorithm for mixtures of bivariate data. Table 2 presents the estimates of model parameters for each component; it also displays the estimates for different sample sizes. We computed the BIC of the model for the three cases presented in Table 2 and observed that the BIC value became large with increasing sample size. Figure 5(b) shows a plot of log-likelihood versus number of iterations; it is clear that the log-likelihood remained low when the number of iterations increased and that the EM method reached convergence.Figure 5
Simulation case (first scenario): (a) fitting of two-component Gaussian mixture and (b) log-likelihood versus iteration.
(a)(b)Table 2
Simulation case: parameter estimates of the bivariate GMM (first technique) (the log-likelihood, BIC, and number of iterations are provided).
Sample sizeModel parametersFirst componentSecond componentπ1μ1μ2σ12σ22π2μ1μ2σ12σ22500.53806−0.197440.461462.681851.356520.461940.22997−0.537502.353900.30614covσ1,σ2=1.3565245covσ1,σ2=−0.30614414Log-likelihood = −331.7415, BIC = 704.9295Number of iterations: 701000.380401.07288−0.68292.330421.047020.61959−0.65870.419271.460890.73289covσ1,σ2=0.945838covσ1,σ2=0.1452915Log-likelihood = −332.917, BIC = 707.2805Number of iterations: 965000. 65985−0.48985−0.20382.108791.007320.340150.95030.395322.292430.96703covσ1,σ2=−0.2169786covσ1,σ2=−0.1483861Log-likelihood = −1678.241, BIC = 3379.508Number of iterations: 77The second scenario involves the selection of the initial values for the means in the mixture model; this was done by applying thek-means method to the reduced data. Then, the centres (or the means) of the clusters were taken as initial values. Thus, the RD data were partitioned into two clusters using the k-means method, denoted as PC1 and PC2, as shown in Figure 6(a). The cluster means for PC1 were 1.343257 and −1.210463, whereas those of PC2 were −0.02877005 and 0.02592586. In the next step, the bivariate GMM was fitted to the RD data using the cluster means as initial values. A visualization of the fitted bivariate GMM is presented in Figure 6(b). A summary of the results is given in Table 3. The resulting BIC value was 3389.031, which was the same as the BIC for GMM, as displayed in Table 2.Figure 6
Reduced data (simulation data; second scenario): (a) estimated number of observations through PC1 and PC1 after applying thek-means method and (b) fitted GMM.
(a)(b)Table 3
Simulation case: parameter estimates of the bivariate GMM (second technique), taking cluster means as initial values of model means.
Sample sizeModel parametersFirst componentSecond componentπ1μ1μ2σ12σ22π2μ1μ2σ12σ225000.835860.083080.069592.996320.939380.16414−0.4231−0.354380.591441.61035covσ1,σ2=0.0173803covσ1,σ2=1.6103530Log-likelihood = –1673.792, BIC = 3389.031Number of iterations: 734Parameter estimates were compared for the mixture and clustering methods. Thek-means method computes the conventional Euclidean distance of given data, whereas GMM computes the weighted distance by considering the variance in its measurement calculations. One can think of mixture models as generalizing k-means clustering to incorporate information about the covariance structure of the data as well as the centres of the latent Gaussians.


### 6.2. Forensic Glass Fragment Data

In this section, the proposed method is implemented using forensic glass fragment (FG) data (available in an R package). The FG data include 214 observations and 10 variables. Consider the four selected variables, manganese (Mg), aluminium (Al), silicon (Si), and potassium (K), and the types of fragments (WinF, WinNF, Veh, Con, Tabl, and Head). Note that the variable type is used only to classify the data. Figure7(a) shows a pairwise plot for four measurements of the scaled data set.Figure 7
(a) Pairs plot of data and (b) plot of PCA.
(a)(b)As a first step, we applied the PCA method to the data to obtain a linear estimate of dimensionality. The results are summarized in Table4. As shown in the scree plot in Figure 7(b), three components explained 89% of the total variance and two components explained 70% of it. We therefore consider the first two components in the following.Table 4
Forensic glass fragments data: summary of PCA implementation on the data.
Importance of components:PC1PC2PC3PC4Standard deviation1.2571.12050.86610.6439Proportion of variance0.3950.31390.18750.1037Cumulative proportion0.3950.70880.89631.00000Hence, the two-component bivariate mixture model was fitted to the FG data, as shown in Figure8(a). The mean and sigma values for each component are also shown in Figure 8(a). In order to estimate the density parameters for each component, we used the EM algorithm for mixtures of bivariate data. Note that the initial values were chosen using the “mixtools” package. Table 5 presents estimates of the model parameters for each component, as well as the BIC. Figure 8(b) shows a plot of log-likelihood versus number of iterations; it is clear that the log-likelihood remained low when the number of iterations increased and that the EM method reached convergence.Figure 8
Forensic glass fragments data: (a) fitting two-component Gaussian mixture nd (b) log-likelihood versus iteration.
(a)(b)Table 5
Forensic glass fragments data: parameter estimates and BIC of the bivariate GMM (first technique).
ValuesModel parametersFirst componentSecond componentπ1μ1μ2σ12σ22π2μ1μ2σ12σ22Parameter estimation0.243111.58580−0.629821.977953.972840.75689−0.509350.202290.375070.20665covσ1,σ2=1.63918covσ1,σ2=−0.10266Log-likelihood = −531.5947, BIC = 1111.483Number of iterations: 27Next, we studied the selection of initial values for the location parameters in the mixture model based on the centres of clusters obtained with different clustering methods:k-means and hierarchical clustering. First, the k-means method was applied to the reduced FG data, and then the centres (or the means) of clusters were taken as initial values. A visualization of the resulting data for PC1 and PC2 is given in Figure 9(a). The cluster centres (means) for PC1 were −0.36697 and 1.45934 and those for PC2 are 0.35596 and −1.41558. Then, the bivariate GMM was fitted to the reduced FG data using the cluster means as initial values. The results are summarized in Table 6, and a plot of the fitted mixture model is shown in Figure 9(b). The resulting BIC was 1174.485.Figure 9
Forensic glass fragments data: (a) cluster criterion and centres and (b) fitted GMM.
(a)(b)Table 6
Forensic glass fragments data: parameter estimates and BIC of the bivariate GMM (second technique) by taking cluster means as initial values of model means, and the initial values for PC1 were −0.36697 and 1.45934 and those for PC2 are 0.35596 and −1.41558.
ValuesModel parametersFirst componentSecond componentπ1μ1μ2σ12σ22π2μ1μ2σ12σ22Parameter estimation0.96102–0.10113−0.120631.105420.726450.038982.493072.973256.617894.94843covσ1,σ2=0.54182covσ1,σ2=5.64324Log-likelihood = –563.0958, BIC = 1174.485Number of iterations: 28Second, the initial values for the location parameters were selected using hierarchical clustering with the “mclust” package. As a first step, the hierarchical clustering was applied to the reduced FG data. A visualization of the resulting data for PC1 and PC2 is given in Figure10(a). The resulting centres were 1.49372 and −0.51297 for PC1, while those for PC2 were −0.58542 and 0.20104. In the next step, the bivariate GMM was fitted to the reduced FG data using the cluster means as initial values, as shown in Figure 10(b). Table 7 shows the results for the mixture model, and the resulting BIC was 1111.483.Figure 10
(a) Cluster criterion and centres (using “mclust”), and (b) fitted Gaussian mixture model.
(a)(b)Table 7
Forensic glass fragments data: parameter estimates and BIC of the bivariate GMM (second technique) by taking cluster means as initial values of model means, and the initial values for 1.49372 and −0.51297 for PC1, while those for PC2 were −0.58542 and 0.20104.
ValuesModel parametersFirst componentSecond componentπ1μ1μ2σ12σ22π2μ1μ2σ12σ22Parameter estimation0.242931.58683−0.630171.977513.974910.75707−0.509190.202220.375190.20682covσ1,σ2=1.64088covσ1,σ2=−0.10269Log-likelihood = −531.5947, BIC = 1111.483Number of iterations: 7We observe that the selection of initial values for location parameters using clustering methods provided good results, similar to those obtained by selection of initial values with the “mixtools” package.


### 6.3. Applications to Real Data

In this section, the proposed method is implemented on a real data set obtained from the “Knoema” website [18]. Knoema is one of the most comprehensive sources of global decision-making data. The data set for cancer incidence in 100 countries in a specific year (2016) includes 3168 observations and 32 variables and covers different cancer types. In the first step, we used the PCA approach to approximate the dimensionality of the data in a linear manner. The results of the first five main components are summarized in Table 8. As shown in the scree plot in Figure 11, 93% of the overall variance was described by the first two components. We thus consider the first two components in the following.Table 8
Summary of PCA implementation on cancer incidence data.
Importance of componentsPC1PC2PC3PC4PC5Standard deviation5.15881.79821.016420.571510.47353Proportion of variance0.83170.10110.032280.010210.00701Cumulative proportion0.83170.93270.964990.975200.98220Figure 11
Cancer incidence data: plot of PCA.The data were thus fitted by the two-component bivariate mixture model, as shown in Figure12(a), which gives the mean and sigma values for each component. We used the EM algorithm for bivariate data mixtures to determine density parameters for each component. The initial values were as suggested by the “mixtools” package. The estimates of model parameters are provided in Table 9 for each component, as well as the BIC. Figure 12(b) shows a plot of log-likelihood versus number of iterations; the log-likelihood remained low as the number of iterations increased, and the EM method reached convergence.Figure 12
Cancer incidence data: (a) Fitting of the two-components Gaussian mixture, (b) log-likelihood versus iteration.
(a)(b)Table 9
Cancer incidence data: parameter estimates and BIC of the bivariate GMM (first technique).
ValuesModel parametersFirst componentSecond componentπ1μ1μ2σ12σ22π2μ1μ2σ12σ22Parameter estimation0.237545.47758−1.2667370.7293211.13280.76245−1.706510.394640.256540.07405covσ1,σ2=9.3649covσ1,σ2=−0.0824Log-likelihood = −242.2074, BIC = 556.9625Number of iterations: 30Then, we used the centres of clusters with different clustering techniques (k-means and hierarchical clustering) to analyse the choice of initial parameters for the location parameters of the mixture model. First, the k-means method was applied to the reduced data, and then the centres (or the means) of clusters were taken as initial values. A visualization of the resulting data for PC1 and PC2 is given in Figure 13(a). The cluster centres (means) for PC1 were 14.19303 and −1.0799, while those of PC2 were −3.22915 and 0.24569. Then, the bivariate GMM was fitted to the reduced data using the cluster means as initial values. The results are summarized in Table 10, and a plot of the fitted mixture model is shown in Figure 13(b). The resulting BIC was 556.9627.Figure 13
Cancer incidence data: (a) cluster criterion and centres and (b) fitted GMM.
(a)(b)Table 10
Cancer incidence data: parameter estimates and BIC of the bivariate GMM (second technique) by taking cluster means as initial values of model means.
ValuesModel parametersFirst componentSecond componentπ1μ1μ2σ12σ22π2μ1μ2σ12σ22Parameter estimation0.237845.47040−1.2649670.6793211.12140.76216−1.70710.394750.255570.07398covσ1,σ2=9.34287covσ1,σ2=−0.0824Log-likelihood = −242.2075, BIC = 556.9627Number of iterations: 30Second, a hierarchical clustering procedure was used to specify the initial values of the location parameters; this was achieved using the “mclust” package. The hierarchical clustering of the reduced data was implemented as a first step. Figure14(a) provides a visualization of the data obtained for PC1 and PC2: the resulting centres were 9.39997 and −0.58877 for PC1, while those for PC2 were 2.25038 and −0.14095. In the next step, the bivariate GMM was fitted on the reduced data using the cluster means as initial values; the fitted data are presented in Figure 14(b). Table 11 shows the results of the mixture model and the BIC of 556.9629.Figure 14
Cancer incidence data: (a) cluster criterion and centres (using “mclust”) and (b) fitted GMM.
(a)(b)Table 11
Cancer incidence data: the estimate results of fitting of the bivariate mixture model by taking cluster means as initial values of model means.
ValuesModel parametersFirst componentSecond componentπ1μ1μ2σ12σ22π2μ1μ2σ12σ22Parameters estimation0.237195.48573−1.2687770.786211.14590.76281−1.705790.394530.257650.07416covσ1,σ2=9.39014covσ1,σ2=−0.08263Log-likelihood = −242.2076, BIC = 556.9629Number of iterations: 13We observe that the selection of initial values for location parameters using clustering methods provided good results, similar to those obtained when the initial values were selected using the “mixtools” package.


## 6.1. Simulation Case

We implement the proposed method on simulation data with different sample sizes: 50, 100, and 500. Consider a data set of four variables defined as follows:(22)X1∼N10,1.52,(23)X2∼N0,32,(24)X3=X1∗X2,(25)X4=0.5Z1+0.3Z2+0.7Z3+N0,1,where Z1,Z2, and Z3 are scaled variables of X1,X2, and X3 , respectively. The simulated data consist of four variables, X1,X2,X3,andX4. The mean values for each variable in the simulated data are μ1=10.003, μ2=−0.1651, μ3=−1.287, and μ4=0.032.The implementation includes a graphical plot of the data set as displayed in Figure3, which shows a pairs plot of the simulation data and its three-dimensional surface. As a first step, we applied the PCA method; the results are summarized in Table 1, which presents the total variance of components. Practically, PCA describes the data in few variables without loss of information. As shown in Figure 4(a) and Table 1, two components explained 93% of the total variance. We considered those two components to comprise a new data set denoted RD data, which contained 93% of the information of the original data. The empirical distribution of the RD data is presented in Figure 4(b).Figure 3
Simulation case: (a) pairs plot of data and (b) three-dimensional surface of data.
(a)(b)Table 1
The results of PCA implementation on the simulation study.
Importance of components:PC1PC2PC3PC4Standard deviation1.61051.06580.510730.09656Proportion of variance0.64840.28400.065210.00233Cumulative proportion0.64840.93250.997671.00000Figure 4
Simulation case: (a) scree plot of PCA and (b) plot of reduced data.
(a)(b)Hence, we fitted a two-component bivariate mixture model to the RD data. Figure5(a) displays the fitting of the two components GMM on the new data (with 500 data points); the plot specifies each component’s mean and sigma values. In order to estimate the density parameters for each component, we used an EM algorithm for mixtures of bivariate data. Table 2 presents the estimates of model parameters for each component; it also displays the estimates for different sample sizes. We computed the BIC of the model for the three cases presented in Table 2 and observed that the BIC value became large with increasing sample size. Figure 5(b) shows a plot of log-likelihood versus number of iterations; it is clear that the log-likelihood remained low when the number of iterations increased and that the EM method reached convergence.Figure 5
Simulation case (first scenario): (a) fitting of two-component Gaussian mixture and (b) log-likelihood versus iteration.
(a)(b)Table 2
Simulation case: parameter estimates of the bivariate GMM (first technique) (the log-likelihood, BIC, and number of iterations are provided).
Sample sizeModel parametersFirst componentSecond componentπ1μ1μ2σ12σ22π2μ1μ2σ12σ22500.53806−0.197440.461462.681851.356520.461940.22997−0.537502.353900.30614covσ1,σ2=1.3565245covσ1,σ2=−0.30614414Log-likelihood = −331.7415, BIC = 704.9295Number of iterations: 701000.380401.07288−0.68292.330421.047020.61959−0.65870.419271.460890.73289covσ1,σ2=0.945838covσ1,σ2=0.1452915Log-likelihood = −332.917, BIC = 707.2805Number of iterations: 965000. 65985−0.48985−0.20382.108791.007320.340150.95030.395322.292430.96703covσ1,σ2=−0.2169786covσ1,σ2=−0.1483861Log-likelihood = −1678.241, BIC = 3379.508Number of iterations: 77The second scenario involves the selection of the initial values for the means in the mixture model; this was done by applying thek-means method to the reduced data. Then, the centres (or the means) of the clusters were taken as initial values. Thus, the RD data were partitioned into two clusters using the k-means method, denoted as PC1 and PC2, as shown in Figure 6(a). The cluster means for PC1 were 1.343257 and −1.210463, whereas those of PC2 were −0.02877005 and 0.02592586. In the next step, the bivariate GMM was fitted to the RD data using the cluster means as initial values. A visualization of the fitted bivariate GMM is presented in Figure 6(b). A summary of the results is given in Table 3. The resulting BIC value was 3389.031, which was the same as the BIC for GMM, as displayed in Table 2.Figure 6
Reduced data (simulation data; second scenario): (a) estimated number of observations through PC1 and PC1 after applying thek-means method and (b) fitted GMM.
(a)(b)Table 3
Simulation case: parameter estimates of the bivariate GMM (second technique), taking cluster means as initial values of model means.
Sample sizeModel parametersFirst componentSecond componentπ1μ1μ2σ12σ22π2μ1μ2σ12σ225000.835860.083080.069592.996320.939380.16414−0.4231−0.354380.591441.61035covσ1,σ2=0.0173803covσ1,σ2=1.6103530Log-likelihood = –1673.792, BIC = 3389.031Number of iterations: 734Parameter estimates were compared for the mixture and clustering methods. Thek-means method computes the conventional Euclidean distance of given data, whereas GMM computes the weighted distance by considering the variance in its measurement calculations. One can think of mixture models as generalizing k-means clustering to incorporate information about the covariance structure of the data as well as the centres of the latent Gaussians.


## 6.2. Forensic Glass Fragment Data

In this section, the proposed method is implemented using forensic glass fragment (FG) data (available in an R package). The FG data include 214 observations and 10 variables. Consider the four selected variables, manganese (Mg), aluminium (Al), silicon (Si), and potassium (K), and the types of fragments (WinF, WinNF, Veh, Con, Tabl, and Head). Note that the variable type is used only to classify the data. Figure7(a) shows a pairwise plot for four measurements of the scaled data set.Figure 7
(a) Pairs plot of data and (b) plot of PCA.
(a)(b)As a first step, we applied the PCA method to the data to obtain a linear estimate of dimensionality. The results are summarized in Table4. As shown in the scree plot in Figure 7(b), three components explained 89% of the total variance and two components explained 70% of it. We therefore consider the first two components in the following.Table 4
Forensic glass fragments data: summary of PCA implementation on the data.
Importance of components:PC1PC2PC3PC4Standard deviation1.2571.12050.86610.6439Proportion of variance0.3950.31390.18750.1037Cumulative proportion0.3950.70880.89631.00000Hence, the two-component bivariate mixture model was fitted to the FG data, as shown in Figure8(a). The mean and sigma values for each component are also shown in Figure 8(a). In order to estimate the density parameters for each component, we used the EM algorithm for mixtures of bivariate data. Note that the initial values were chosen using the “mixtools” package. Table 5 presents estimates of the model parameters for each component, as well as the BIC. Figure 8(b) shows a plot of log-likelihood versus number of iterations; it is clear that the log-likelihood remained low when the number of iterations increased and that the EM method reached convergence.Figure 8
Forensic glass fragments data: (a) fitting two-component Gaussian mixture nd (b) log-likelihood versus iteration.
(a)(b)Table 5
Forensic glass fragments data: parameter estimates and BIC of the bivariate GMM (first technique).
ValuesModel parametersFirst componentSecond componentπ1μ1μ2σ12σ22π2μ1μ2σ12σ22Parameter estimation0.243111.58580−0.629821.977953.972840.75689−0.509350.202290.375070.20665covσ1,σ2=1.63918covσ1,σ2=−0.10266Log-likelihood = −531.5947, BIC = 1111.483Number of iterations: 27Next, we studied the selection of initial values for the location parameters in the mixture model based on the centres of clusters obtained with different clustering methods:k-means and hierarchical clustering. First, the k-means method was applied to the reduced FG data, and then the centres (or the means) of clusters were taken as initial values. A visualization of the resulting data for PC1 and PC2 is given in Figure 9(a). The cluster centres (means) for PC1 were −0.36697 and 1.45934 and those for PC2 are 0.35596 and −1.41558. Then, the bivariate GMM was fitted to the reduced FG data using the cluster means as initial values. The results are summarized in Table 6, and a plot of the fitted mixture model is shown in Figure 9(b). The resulting BIC was 1174.485.Figure 9
Forensic glass fragments data: (a) cluster criterion and centres and (b) fitted GMM.
(a)(b)Table 6
Forensic glass fragments data: parameter estimates and BIC of the bivariate GMM (second technique) by taking cluster means as initial values of model means, and the initial values for PC1 were −0.36697 and 1.45934 and those for PC2 are 0.35596 and −1.41558.
ValuesModel parametersFirst componentSecond componentπ1μ1μ2σ12σ22π2μ1μ2σ12σ22Parameter estimation0.96102–0.10113−0.120631.105420.726450.038982.493072.973256.617894.94843covσ1,σ2=0.54182covσ1,σ2=5.64324Log-likelihood = –563.0958, BIC = 1174.485Number of iterations: 28Second, the initial values for the location parameters were selected using hierarchical clustering with the “mclust” package. As a first step, the hierarchical clustering was applied to the reduced FG data. A visualization of the resulting data for PC1 and PC2 is given in Figure10(a). The resulting centres were 1.49372 and −0.51297 for PC1, while those for PC2 were −0.58542 and 0.20104. In the next step, the bivariate GMM was fitted to the reduced FG data using the cluster means as initial values, as shown in Figure 10(b). Table 7 shows the results for the mixture model, and the resulting BIC was 1111.483.Figure 10
(a) Cluster criterion and centres (using “mclust”), and (b) fitted Gaussian mixture model.
(a)(b)Table 7
Forensic glass fragments data: parameter estimates and BIC of the bivariate GMM (second technique) by taking cluster means as initial values of model means, and the initial values for 1.49372 and −0.51297 for PC1, while those for PC2 were −0.58542 and 0.20104.
ValuesModel parametersFirst componentSecond componentπ1μ1μ2σ12σ22π2μ1μ2σ12σ22Parameter estimation0.242931.58683−0.630171.977513.974910.75707−0.509190.202220.375190.20682covσ1,σ2=1.64088covσ1,σ2=−0.10269Log-likelihood = −531.5947, BIC = 1111.483Number of iterations: 7We observe that the selection of initial values for location parameters using clustering methods provided good results, similar to those obtained by selection of initial values with the “mixtools” package.


## 6.3. Applications to Real Data

In this section, the proposed method is implemented on a real data set obtained from the “Knoema” website [18]. Knoema is one of the most comprehensive sources of global decision-making data. The data set for cancer incidence in 100 countries in a specific year (2016) includes 3168 observations and 32 variables and covers different cancer types. In the first step, we used the PCA approach to approximate the dimensionality of the data in a linear manner. The results of the first five main components are summarized in Table 8. As shown in the scree plot in Figure 11, 93% of the overall variance was described by the first two components. We thus consider the first two components in the following.Table 8
Summary of PCA implementation on cancer incidence data.
Importance of componentsPC1PC2PC3PC4PC5Standard deviation5.15881.79821.016420.571510.47353Proportion of variance0.83170.10110.032280.010210.00701Cumulative proportion0.83170.93270.964990.975200.98220Figure 11
Cancer incidence data: plot of PCA.The data were thus fitted by the two-component bivariate mixture model, as shown in Figure12(a), which gives the mean and sigma values for each component. We used the EM algorithm for bivariate data mixtures to determine density parameters for each component. The initial values were as suggested by the “mixtools” package. The estimates of model parameters are provided in Table 9 for each component, as well as the BIC. Figure 12(b) shows a plot of log-likelihood versus number of iterations; the log-likelihood remained low as the number of iterations increased, and the EM method reached convergence.Figure 12
Cancer incidence data: (a) Fitting of the two-components Gaussian mixture, (b) log-likelihood versus iteration.
(a)(b)Table 9
Cancer incidence data: parameter estimates and BIC of the bivariate GMM (first technique).
ValuesModel parametersFirst componentSecond componentπ1μ1μ2σ12σ22π2μ1μ2σ12σ22Parameter estimation0.237545.47758−1.2667370.7293211.13280.76245−1.706510.394640.256540.07405covσ1,σ2=9.3649covσ1,σ2=−0.0824Log-likelihood = −242.2074, BIC = 556.9625Number of iterations: 30Then, we used the centres of clusters with different clustering techniques (k-means and hierarchical clustering) to analyse the choice of initial parameters for the location parameters of the mixture model. First, the k-means method was applied to the reduced data, and then the centres (or the means) of clusters were taken as initial values. A visualization of the resulting data for PC1 and PC2 is given in Figure 13(a). The cluster centres (means) for PC1 were 14.19303 and −1.0799, while those of PC2 were −3.22915 and 0.24569. Then, the bivariate GMM was fitted to the reduced data using the cluster means as initial values. The results are summarized in Table 10, and a plot of the fitted mixture model is shown in Figure 13(b). The resulting BIC was 556.9627.Figure 13
Cancer incidence data: (a) cluster criterion and centres and (b) fitted GMM.
(a)(b)Table 10
Cancer incidence data: parameter estimates and BIC of the bivariate GMM (second technique) by taking cluster means as initial values of model means.
ValuesModel parametersFirst componentSecond componentπ1μ1μ2σ12σ22π2μ1μ2σ12σ22Parameter estimation0.237845.47040−1.2649670.6793211.12140.76216−1.70710.394750.255570.07398covσ1,σ2=9.34287covσ1,σ2=−0.0824Log-likelihood = −242.2075, BIC = 556.9627Number of iterations: 30Second, a hierarchical clustering procedure was used to specify the initial values of the location parameters; this was achieved using the “mclust” package. The hierarchical clustering of the reduced data was implemented as a first step. Figure14(a) provides a visualization of the data obtained for PC1 and PC2: the resulting centres were 9.39997 and −0.58877 for PC1, while those for PC2 were 2.25038 and −0.14095. In the next step, the bivariate GMM was fitted on the reduced data using the cluster means as initial values; the fitted data are presented in Figure 14(b). Table 11 shows the results of the mixture model and the BIC of 556.9629.Figure 14
Cancer incidence data: (a) cluster criterion and centres (using “mclust”) and (b) fitted GMM.
(a)(b)Table 11
Cancer incidence data: the estimate results of fitting of the bivariate mixture model by taking cluster means as initial values of model means.
ValuesModel parametersFirst componentSecond componentπ1μ1μ2σ12σ22π2μ1μ2σ12σ22Parameters estimation0.237195.48573−1.2687770.786211.14590.76281−1.705790.394530.257650.07416covσ1,σ2=9.39014covσ1,σ2=−0.08263Log-likelihood = −242.2076, BIC = 556.9629Number of iterations: 13We observe that the selection of initial values for location parameters using clustering methods provided good results, similar to those obtained when the initial values were selected using the “mixtools” package.


## 7. Conclusions

This work aimed to study the applications of PCA in mixture models. First, we discussed the use of the well-known PCA technique for dimension reduction and applied it to high-dimensional data sets. Then, in the reduced data (which contained only two variables), we dealt with the two variables together and fitted a two-component bivariate GMM to the data. We used an EM algorithm to estimate the model parameters. This approach is suitable for large data sets with high dimension and can solve the problem of overfitting. We compared three different techniques for the selection of initial values of location parameters in the mixture model: two clustering methods,k-means and hierarchical clustering, and default values from the “mixtools” package. With all three techniques, EM convergence was reached and similar BIC values were obtained.



---

*Source: 1202307-2020-07-31.xml*