# Decoding Urban Intelligence: Clustering and Feature Importance in Smart Cities
## 3 Background
### Gaussian Mixture Models  
Gaussian mixture models (GMM, [22]) are probabilistic models that assume the data are generated from a mixture of several Gaussian distributions with unknown parameters. GMMs are used for clustering, density estimation, and as generative models for data. A GMM is defined as a weighted sum of \(K\) Gaussian component densities. The probability density function of a GMM is given by the following:  
\[p(\mathbf{x}|\boldsymbol{\theta})=\sum_{k=1}^{K}\pi_{k}\mathcal{N}(\mathbf{x}| \boldsymbol{\mu}_{k},\boldsymbol{\Sigma}_{k})\]  
where \(\pi_{k}\) are the mixture weights, \(\boldsymbol{\mu}_{k}\) are the mean vectors, \(\boldsymbol{\Sigma}_{k}\) are the covariance matrices of the Gaussian components, and \(\boldsymbol{\theta}\) represents all the parameters of the model. The parameters of a GMM can be estimated using the expectation-maximization (EM) algorithm, which iteratively improves the estimates of the parameters. The EM algorithm consists of two steps: the expectation (E) step and the maximization (M) step. In the E step, the algorithm calculates the posterior probabilities (responsibilities) for each data point to belong to each Gaussian component:  
\[\gamma_{ik}=\frac{\pi_{k}\mathcal{N}(\mathbf{x}_{i}|\boldsymbol{\mu}_{k}, \boldsymbol{\Sigma}_{k})}{\sum_{j=1}^{K}\pi_{j}\mathcal{N}(\mathbf{x}_{i}| \boldsymbol{\mu}_{j},\boldsymbol{\Sigma}_{j})}\]  
In the M step, the parameters are updated using the current responsibilities:  
\[\pi_{k}=\frac{N_{k}}{N},\quad\boldsymbol{\mu}_{k}=\frac{1}{N_{k}}\sum_{j=1}^{ N}\gamma_{ik}\mathbf{x}_{i},\quad\boldsymbol{\Sigma}_{k}=\frac{1}{N_{k}}\sum_{j=1}^ {N}\gamma_{ik}(\mathbf{x}_{i}-\boldsymbol{\mu}_{k})(\mathbf{x}_{i}-\boldsymbol {\mu}_{k})^{\top}\]where \(N_{k}=\sum_{i=1}^{N}\gamma_{ik}\) and \(N\) is the total number of data points. GMMs are used for clustering because they can model complex data distributions with multiple peaks, unlike k-means which assumes spherical clusters. GMMs provide a soft clustering approach where each data point is assigned a probability of belonging to each cluster, which is more informative than hard clustering. Also, GMMs are suitable for density estimation and can generate new samples from the learned distribution. GMMs are flexible in representing data distributions and can model clusters of different shapes and sizes. They provide probabilistic cluster assignments, which can be useful for uncertainty quantification. However, GMMs have limitations, including sensitivity to initialization and the assumption that each cluster follows a Gaussian distribution. They can also be computationally expensive, especially for high-dimensional data and many components. For clustering tasks, GMMs can provide more accurate and flexible cluster assignments compared to k-means, especially when the data have non-spherical clusters. GMMs can also be used for anomaly detection by identifying data points with low probability under the learned model. Furthermore, GMMs can be integrated into more complex models, such as hidden Markov models (HMMs), to model sequential data. \[L(\mathbf{x},\mathbf{\hat{x}})=\|\mathbf{x}-\mathbf{\hat{x}}\|^{2}.\]  
Models Evaluation  
In addition to the aforementioned elbow and silhouette methods, this work considers further model performance evaluation techniques. In particular, the Davies-Bouldin index is used to evaluate the quality of clustering. It is defined as follows:  
\[\text{Davies-Bouldin Index}=\frac{1}{n}\sum_{i=1}^{n}\max_{i\
eq j}\left(\frac{S _{i}+S_{j}}{M_{ij}}\right)\]  
where:  
* \(n\) is the number of clusters. * \(S_{i}\) is the average distance between each point in cluster \(i\) and the centroid of cluster \(i\) (intra-cluster distance). * \(M_{ij}\) is the distance between the centroids of clusters \(i\) and \(j\) (inter-cluster distance). * The formula computes the similarity between each cluster \(i\) and the cluster most similar to it (the maximum similarity between \(i\) and any other cluster \(j\)).