# Determining the Quality of a Dataset in Clustering Terms
## 4 Discussion  
The presented methodology and research indicated how the set quality could be determined by considering the observation segmentation. Because of the data's homogeneity or their excessive heterogeneity, not every set could be divided into groups in a meaningful way, even if advanced preprocessing (categorization, autoencoder, PCA) was used. The indices that determined the clustering quality were the modularity parameter, the Calinski-Harabasz index and the Davies-Bouldin index. In the future, selecting the optimal cut-off parameter could be automated by choosing the highest value (in the range \((0,1)\)) at which the graph remains consistent.  
Most clustering algorithms can work only for categorical data or only for numerical data. Before using such algorithms, it is necessary to preprocess the data, which includes the discretization of continuous variables or one-hot encoding of categorical variables. The discretization process always generates some loss of information. In that context, our methodology potentially could be improved by some other treatment of continuous variables.  
Another potential problem is the presence of missing values. We did not encounter this problem in the two chosen datasets; however, it may be a common occurrence in real-life application of our framework. To overcome the problems of missing data, imputation can be used or the observations with missing data might be deleted. However, simply deleting data is not a good practice, as it can lead to the loss of important information. In addition, deleting missing observations can result in leaving too few objects in the dataset, especially when a large number of variables contain missing data. A novel approach to imputation of missing data in clustering tasks was shown in [39]. There, an innovative algorithm called \(k\)-CMM was presented. That algorithm combined imputing the missing data with the clustering algorithm. It was significantly better than many other algorithms especially in the case when the number of missing data increased in the dataset.  
In our research, we encountered a problem with high-dimensional data, because of the embeddings produced by AutoEmbedder. It is known that a higher dimensionality in the context of clustering may have an effect on distance or similarity measures. In particular, most clustering techniques depend critically on the measure of distance or similarity, and require that the objects within clusters are, in general, closer to each other than to objects in other clusters [5]. To minimize the problems connected to high dimensionality, we used PCA to reduce the number of features.  
However, one would like to highlight that despite the use of sophisticated methods, the set of resellers did not tend to cluster. It is important to note that not all markets were clearly segmented. Customised tools were also provided to verify this issue. The set of retail customers was homogeneous due to some characteristics. It was important to choose a cut-off parameter that would ensure the consistency of the graph and at the same time allow us to extract the most heterogeneous groups of customers. The best clustering was provided by the Louvain method with a cut-off parameter of 0.75. With a cut-off parameter of 0.75, [PERSON]'s method divided the set of customers into three heterogeneous groups.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & **Cut-Off** & \multirow{2}{*}{**Group 0**} & \multirow{2}{*}{**Group 1**} & \multirow{2}{*}{**Group 2**} & **Total** \\  & **Parameter** & & & & **Group 2** \\ \hline Louvain & 0.5 & 151 & 215 & 335 & 3 \\ Greedy & 0.5 & 351 & 350 & - & 2 \\ Label & 0.5 & 350 & 351 & - & 2 \\ Louvain & 0.25 & 1048 & 1192 & - & 2 \\ Greedy & 0.25 & 1131 & 1109 & - & 2 \\ Label & 0.25 & 356 & 345 & - & 2 \\ \(k\)-means & - & 382 & 319 & - & 2 \\ DBSCAN & - & 119 & - & - & 1 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Total number of groups and group sizes for each method (wholesale customers).