# Entropy-Based Anomaly Detection for Gaussian Mixture Modeling
## 2 Materials and Methods  
### Gaussian Mixtures in Model-Based Clustering and Density Estimation  
Model-based clustering assumes that the observed data are generated from a mixture of \(G\) components, each representing the probability distribution for a different group or cluster [1; 15]. For continuous data, the density of each mixture component is often described by the multivariate Gaussian distribution. Thus, the general form of a Gaussian mixture model (GMM) is  
\[f(\mathbf{x})=\sum_{k=1}^{G}\pi_{k}\phi(\mathbf{x}|\mathbf{\mu}_{k},\mathbf{\Sigma}_{k}), \tag{1}\]  
where \(\pi_{k}\) represents the mixing probabilities, so that \(\pi_{k}>0\) and \(\sum_{k=1}^{G}\pi_{k}=1\), \(\phi(\cdot)\) is the multivariate Gaussian density with parameters \((\mathbf{\mu}_{k},\mathbf{\Sigma}_{k})\) (\(k=1,\ldots,G\)). Clusters described by a GMM are ellipsoidal, centered at the means \(\mathbf{\mu}_{k}\), and with other geometric characteristics (such as volume, shape, and orientation) determined by the covariance matrices \(\mathbf{\Sigma}_{k}\). Parsimonious parameterization of covariance matrices can be controlled by introducing some constraints on the covariance matrices through the following eigen-decomposition [16; 17]:  
\[\mathbf{\Sigma}_{k}=\lambda_{k}\mathbf{U}_{k}\mathbf{\Delta}_{k}\mathbf{U}_{k}^{\top}, \tag{2}\]where \(\lambda_{k}=|\mathbf{\Sigma}_{k}|^{1/d}\) is a scalar, which controls the volume, \(\mathbf{\Delta}_{k}\) is a diagonal matrix, such that \(|\mathbf{\Delta}_{k}|=1\) and with the normalized eigenvalues of \(\mathbf{\Sigma}_{k}\) in decreasing order, which controls the shape, \(\mathbf{U}_{k}\) is an orthogonal matrix of eigenvectors of \(\mathbf{\Sigma}_{k}\), which controls the orientation. A list of 14 parameterizations available in the R [18] package mclust[19] is included in [PERSON] et al. ([20], Table 3). Given a random sample of observations \(\{\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{n}\}\) in \(d\) dimensions, the log-likelihood of a GMM with \(G\) components is given by  
\[\ell(\mathbf{\theta})=\sum_{i=1}^{n}\log\Biggl{\{}\sum_{k=1}^{G}\pi_{k}\mathbf{\phi}( \mathbf{x}_{i},\mathbf{\mu}_{k},\mathbf{\Sigma}_{k})\Biggr{\}}. \tag{3}\]  
where \(\mathbf{\theta}=(\pi_{1},\ldots,\pi_{G-1},\mathbf{\mu}_{1},\ldots,\mathbf{\mu}_{G},\mathbf{ \Sigma}_{1},\ldots,\mathbf{\Sigma}_{G})\) are the parameters to be estimated. Maximizing the log-likelihood function (3) directly is often complicated, so maximum likelihood estimation (MLE) of \(\mathbf{\theta}\) is usually performed using the EM algorithm [21] by including component membership as a latent variable. The EM algorithm consists of two steps: the E-step (Expectation step) and the M-step (Maximization step). In the E-step, the algorithm calculates the expected membership probabilities of each data point to each of the mixture components based on the current estimates of the model parameters. In the M-step, the algorithm updates the model parameters by maximizing the likelihood of the observed data given the estimated membership probabilities. These two steps are repeated until convergence or a maximum number of iterations is reached. Details on the use of the EM algorithm in finite mixture modeling is provided by [PERSON] and [PERSON] [1], while a thorough treatment and further extensions can be found in [PERSON] and [PERSON] [22]. Following the fitting of a GMM and the determination of the MLEs of parameters, the maximum a posteriori (MAP) procedure can be used to classify the observations into the most likely cluster.