# Topic Detection Based on Sentence Embeddings and Agglomerative Clustering with Markov Moment
## 3 Comparison of the Proposed Approach with the Silhouette Score and the Elbow Method
If we a priori assume that, according to the Euclidean metric, similar texts have similar vector representations, they should form spherical clusters. At the same time, the problem of fuzzy boundaries between different types of texts is well-known. According to [PERSON] (2016), there are no clear boundaries between types of texts, but only a certain center for each of the types. Therefore, we can assume that the representation of textual data by normalized vectors in the Euclidean space most closely matches the one of the dataset 3 at Figure 1. Additionally, this is exactly where our method works best. Second, for the datasets with one or two clusters, our method detects a big number of clusters (see visualizations for the datasets 1, 2, and 6 at Figure 1). This proves that use of the \"e-2\" hypothesis allows for catching the moment when the clustering process begins, before the chain effect shows up. Again, this proves that our method is suitable for datasets with large number of clusters, like the real-world ones used for topic detection. Figure 1: Results of the proposed approach made on datasets from scikit-learn. Now, let us see how the \"e-2\" hypothesis works in comparison with more traditional ways of stopping to determine the preferred number of clusters, namely the silhouette metric and the elbow method. Our approach differs from both of them. The elbow method is a heuristic used to determine the number of clusters in a dataset. It consists of finding the \"elbow\" of the graph of change as a function of the number of clusters. The most common variation that is used in the elbow method is total variance within the cluster. The proposed method for determining the number of clusters using approximation-estimation tests is also based on finding the \"elbow\" of the graph of some values that characterize the clustering process. The difference is that the definition number of clusters with help of \"elbow\" is based upon heuristic visual assessment, but the definition number of clusters with help of approximation-estimation tests and \"e-2\" hypothesis is based upon sequential statistical analysis. Unlike the elbow method, the silhouette coefficient is non-heuristic. It is based on two scores: \(a(x)\) is the average distance from current point \(x\) to objects of its own cluster \(A\), and \(b(x)\) is the average distance from the current point \(x\) to the objects of cluster \(B\), which is the closest one to \(x\). The silhouette coefficient of point \(x\) is the number \(s(x)\):  
\[s(x)=\frac{b(x)-a(x)}{\max\{a(x),b(x)\}}. \tag{19}\]  
It is easy to see that \(-1\leq s(x)\leq 1\). The value of s(x) being close to 1 implies that the point is clustered well; the value being close to \(-1\) implies that the point is clustered incorrect. If the value is close to 0, it it means that the distances from \(x\) to both clusters \(A\) and \(B\) are the same, and the point x should be assigned to either \(A\) or \(B\). This case can be considered as \"intermediate\". The silhouette coefficient of a dataset is defined as the average of the silhouette coefficients for all of the points of the sample. The number of clusters with the highest silhouette coefficient for the fixed clustering method can be determined as the optimal number of clusters for this method applied to the dataset under scrutiny [27]. However, the silhouette coefficient has its shortcomings. It takes much time to compute the silhouette coefficients for every data unit in large datasets for every possible clustering result. Additionally, the silhouette coefficient cannot detect well the original number of clusters if the clusters overlap and have fuzzy boundaries, while our method helps to eliminate this problem.