# Determining the Quality of a Dataset in Clustering Terms
## 2 Materials and Methods
### Methodology
#### 2.2.11 Proposed Methodology  
Clustering comparisons were made on two datasets: resembers and customers. First, embeddings for categorical variables were generated with the use of an autoencoder. An autoencoder is a neural network designed to encode input data in the form of the so-called hidden representation and then decode them in such a way that the reconstructed data are as similar to the original ones as possible. It is worth noting that most embedding generation methods require knowledge of the target variable. Using an autoencoder removes that need, since the loss function is calculated by comparing the reconstruction with the input data. A diagram of how an autoencoder combined with an embedding layer, known as AutoEmbedder, works is shown in Figure 1. A similar usage of autoencoders can be seen in [6; 32]. In the case of continuous variables, a transformation to a categorical variable was performed before their inclusion in the model by determining classes (ranges) of variable  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
\begin{tabular}{c} The cluster \\ centroids \\ \end{tabular} & & \\ \hline \hline \end{tabular}
\end{table}
Table 1: Summary of main notations in the paper.  
values. Embeddings were then generated for the categories using the neural network's embedding layer. The embeddings were the input data for the autoencoder.  
AutoEmbedder was trained on the training set. Once it had been trained, the embedding layer and encoder were used to generate the embeddings. A principal component analysis (PCA) [33] was then performed on the coded data, assuming that 95% of the variance remained explained. After that procedure, each observation was in the form of a real number vector. For the observations coded as described above, with the selection of an appropriate similarity measure, we generated a similarity matrix. Further on, we proceeded to select clustering algorithms, including graph algorithms. The methodology is graphically illustrated by Figure 2.  
As a similarity metric, a cosine similarity scaled to a range from 0 to 1 was chosen.  
The following algorithms were used to find groups of similar customers: \(k\)-means, DBSCAN, Louvain community, greedy modularity, label propagation. The last three are graph methods. For their implementation, a graph needed to be created to represent the relationship between customers in the set. When building a graph representing the relationship between the most similar customers, it was assumed that the graph was consistent. The graph was constructed based on the similarity matrix. Using the cut-off parameter, the creation of edges between similar clients was controlled. An edge was created between clients for which the similarity between them was greater than or equal to the set cut-off parameter. The cut-off parameters we tested were 0.25, 0.5 and 0.75.  
The resulting clusters were compared using metrics available for unsupervised clustering. Calinski-Harabasz and Davies-Bouldin indices were calculated for the clusterings, which helped to select the most optimal clustering. In order to compare how similar the two clustering data were, the NMI parameter (normalized mutual information) and the Fowlkes-Mallows index could be used. Moreover, for graph methods, one could compare their modularity. When determining the optimal cluster number in the \(k\)-means algorithm, we used the results of the elbow and silhouette methods and the Calinski-Harabasz and Davies-Bouldin index values. In DBSCAN clustering, the parameters were selected based on the silhouette criterion.  
Figure 1: Architecture of AutoEmbedder.