# Entropy-Based Anomaly Detection for Gaussian Mixture Modeling
## 2 Materials and Methods
### Initial Noise Detection Using Entropy Contribution of Data Points  
Entropy is a measure of average uncertainty or information content in a random variable that plays a central role in information theory [28]. For a multivariate continuous random variable \(X\in\mathbb{R}^{d}\) with probability density function \(f(x)\), the entropy is defined as  
\[H(X)=-\int_{\mathcal{X}}f(\mathbf{x})\log f(\mathbf{x})\ \mathrm{d}\mathbf{x}=-\mathbb{E}[ \log f(\mathbf{x})], \tag{6}\]  
where \(\mathcal{X}=\{\mathbf{x}:f(\mathbf{x})>0\}\) is the support of the random variable [29]. A mixture-based estimate of the entropy has been recently proposed by [PERSON] and [PERSON] [30]. Assuming that the distribution of the multivariate random variable \(X\) can be expressed as a finite mixture of Gaussian components, the proposed estimate is easily obtained in practice using  
\[\widehat{H}(X)=-\frac{1}{n}\sum_{i=1}^{n}\log\widehat{f}(\mathbf{x}_{i};\widehat{ \mathbf{\theta}})=\sum_{i=1}^{n}h_{i}. \tag{7}\]  
where \(\widehat{f}(\mathbf{x}_{i};\widehat{\mathbf{\theta}})\) is the mixture-based estimate of the density from Equation (1) with \(\widehat{\mathbf{\theta}}\) the MLE of the mixture parameters. Thus, an estimate of the entropy is obtained by summing over the contribution  
\[h_{i}=-\frac{1}{n}\log\widehat{f}(\mathbf{x}_{i};\widehat{\mathbf{\theta}})\]  
from each data point. This can also be interpreted as a measure of the degree of anomaly or outlierness of each observation. A data-driven automatic procedure for selecting the initial outlying observations can be based on a comparison of the entropy contributions \(h_{i}\), for \(i=1,\ldots,n\), with those arising from a uniform distribution over the hyper-rectangle enclosing the data. Recalling that the entropy of a multivariate continuous uniform distribution is given by  
\[H(U)=-\int_{\mathcal{U}_{1}}\int_{\mathcal{U}_{2}}\cdots\int_{\mathcal{U}_{d} }V^{-1}\log(V^{-1})\;\mathrm{d}u_{1}\mathrm{d}u_{2}\ldots\mathrm{d}u_{d}=\log (V).\]  
Then, the contribution of each data point under the uniform distribution model can be computed as  
\[u_{i}=\log(V)/n\qquad\text{for all }i=1,\ldots,n.\]  
A preliminary noise assignment is made by defining the set \(\widetilde{\mathcal{C}}_{0}=\{\mathbf{x}_{i}:h_{i}>u_{i}\}\), with \(\widetilde{n}_{0}=\#\{\widetilde{\mathcal{C}}_{0}\}\) giving the number of initial noisy data. The remaining \(n-\widetilde{n}_{0}\) observations are then partitioned using, for instance, model-based agglomerative clustering [31], to obtain \(\widetilde{\mathcal{C}}_{k}=\{\mathbf{x}_{i}:h_{i}\leq u_{i}\wedge\mathbf{x}_{i}\in \mathcal{P}_{k}\}\), where \(\mathcal{P}_{k}\) is the \(k\)th part of the partition with size \(\widetilde{n}_{k}=\#\{\widetilde{\mathcal{C}}_{k}\}\) for \(k=1,\ldots,G\), so that \(n=\widetilde{n}_{0}+\widetilde{n}_{1}+\cdots+\widetilde{n}_{G}\). With this initial partition, the log-likelihood in (5) can be maximized using the EM algorithm. As a final result, the clusters \(\mathcal{C}_{1},\ldots,\mathcal{C}_{G}\) and the group of anomalies \(\mathcal{C}_{0}\) are estimated, such that \(\{\mathcal{C}_{0}\cup\mathcal{C}_{1}\cup\cdots\cup\mathcal{C}_{G}\}=\{\mathbf{x}_ {1},\mathbf{x}_{2},\ldots,\mathbf{x}_{n}\}\) and \(\mathcal{C}_{j}\cap\mathcal{C}_{k}=\emptyset\) for \(j\
eq k\). The general approach to anomaly detection proposed in this paper is summarized in Algorithm 1. ```
Input: \(\bullet\) Data matrix \(\mathbf{X}\) of \(n\) observations on \(d\) variables or features. Steps: 1. Fit a GMM by maximizing the log-likelihood in (3) via the EM algorithm. 2. Compute the contribution of each data point \(h_{i}\) to the entropy. 3. Select the initial noisy data by comparing the \(h_{i}\) values with the reference values \(u_{i}\) derived assuming a uniform distribution for the anomalies. 4. Fit a GMM with the noise component by maximizing the log-likelihood in (5) via the EM algorithm.