# Topic Detection Based on Sentence Embeddings and Agglomerative Clustering with Markov Moment
## 3 Comparison of the Proposed Approach with the Silhouette Score and the Elbow Method  
As stated above, to define the preferred number of clusters, we use the \"e-2\" hypothesis. In detail, the \"e-2\" hypothesis works the following way. The coefficient of trend \(q\) increases monotonically, with some discreteness step from 0 to \(+\infty\). The first stable clustering interval is \(Q_{1}=[0,a_{1}]\). For any \(q\) from \([0,a_{1}]\), the same clustering result is obtained, and the sample is divided into the largest number of clusters. These clusters are the smallest. The next stable clustering interval, \(Q_{2}=[a_{1},a_{2}]\), for any \(q\) from \([a_{1},a_{2}]\), the same result is obtained again. Nevertheless, the number of clusters decreases, and the clusters become larger, etc. When the \(q\) trend coefficient increases, the preferred number of clusters is obtained. We denote this stable clustering interval \(Q_{e-2}=[a_{e-3},a_{e-2}]\). It is clear that, for any \(q\) from \([a_{e-3},a_{e-2}]\), the preferred number of clusters is obtained. On the interval of stable clustering \(Q_{e-1}=[a_{e-2},a_{e-1}]\), a \"chain effect\" occurs, described in the literature on cluster analysis, for example [19]. In this interval of stable clustering, some of the already formed clusters merge. The coefficient \(q\) increases and reaches such a value \(a_{e}\) that all sampling elements are combined into one cluster. We denote this ray of stable clustering by \(Q_{e}=[a_{e},+\infty)\). From the text stated above, it is obvious that if the marked dataset contains less than three clusters, then the interval of stable clustering \(Q_{e-1}=[a_{e-2},a_{e-1}]\) disappears. Therefore, for such datasets, the \"e-2\" hypothesis gives a bad result. Thus, it is clear that, for two-cluster datasets, use of the \"e-2\" hypothesis is not recommended, as the \"e-1\" step will be eliminated; however, for data with the presumed number of clusters being three or more, our method is capable of providing refined results. It needs to be underlined that theoverwhelming majority of real-world datasets contain far more than two clusters: even in the simplest tasks of sentiment analysis, there are three clusters (negative, positive, and neutral), while, in topic detection, the number of clusters in large corpora is expected to be much bigger. Thus, our method is suitable for such datasets. Below, we support this claim by applying our method first to the simplest standard model datasets from scikit-learn ([[https://scikit-learn.org/stable/modules/clustering.htmllibrary](https://scikit-learn.org/stable/modules/clustering.htmllibrary)]([https://scikit-learn.org/stable/modules/clustering.htmllibrary](https://scikit-learn.org/stable/modules/clustering.htmllibrary))). The scikit-learn visualization of runs of clustering methods on six datasets of varying nature demonstrates that various clustering methods work differently on datasets with varying number of clusters (one, two, and three) and varying spreads of data units within the Euclidean space. The pictures at scikit-learn show that some methods work better on two-cluster datasets, and some do best on three-cluster ones. Figure 1 shows the results of application of our method to the six model datasets. Figure 1 allows for two conclusions. First, for three-cluster datasets, our method works well, the results are close to the Ward method, but work even better with borderline units (see visualizations for the datasets 3, 4, and 5, in comparison with the respective visualizations at scikit-learn for the Ward method). The visualization confirms that our method suits well for the datasets with three or more clusters. In this respect, the dataset 3 at Figure 1 deserves particular attention. This dataset contains three spherical clusters with fuzzy boundaries (compare to the dataset 5). Such distribution of points in a Euclidean space is typical for text representations by normalized vectors (e.g., embeddings-based representations).