# AutoElbow: An Automatic Elbow Detection Method for Estimating the Number of Clusters in a Dataset  
[PERSON]  
1 Next Generation Enterprises and Institutions, Council for Scientific and Industrial Research, Pretoria 0001, South Africa; [EMAIL_ADDRESS] (D.N.M. ); [EMAIL_ADDRESS] (S.J.I. ); [EMAIL_ADDRESS] (A.M.A.-M.)  
[PERSON]  
1 Next Generation Enterprises and Institutions, Council for Scientific and Industrial Research, Pretoria 0001, South Africa; [EMAIL_ADDRESS] (D.N.M. ); [EMAIL_ADDRESS] (S.J.I. ); [EMAIL_ADDRESS] (A.M.A.-M.)  
[PERSON]  
1 Next Generation Enterprises and Institutions, Council for Scientific and Industrial Research, Pretoria 0001, South Africa; [EMAIL_ADDRESS] (D.N.M. ); [EMAIL_ADDRESS] (S.J.I. ); [EMAIL_ADDRESS] (A.M.A.-M.)  
2  
Footnotethe silhouette method produces a coefficient that is typically greater for convex clusters than other cluster concepts, thus, making it unreliable in such instances. In contrast, the gap statistic compares the total within-cluster variation for different values of \(k\) to their anticipated values under a null reference distribution of the data (normally the uniform distribution) [9]. The ideal cluster estimate will, thus, be the value that maximizes the gap statistic. This indicates that the clustering structure is significantly distinct from the uniform random distribution of points. The fact that the gap statistic is not always a convex or monotonic function makes it more difficult to determine the best point. In addition, as a \"relative\" statistic, it may not determine whether or not there are meaningful clusters in the data. The elbow, or \"knee of a curve,\" approach is the most common and simplest means of determining the appropriate cluster number prior to running clustering algorithms, such has the K-means algorithm. The elbow method entails running the clustering algorithm (often the K-means algorithm) on the dataset repeatedly across a range of \(k\) values, i.e., \(k=1,2,\ldots,K\), where \(K\) is the total number of clusters to be iterated. For each value of \(k\), the average score for all clusters is calculated. This score, called the within-cluster dispersion (WCD) is typically calculated by adding the squared distance between each point and its related centroid. We note that, apart from the WCD, there are other metrics that may be used to assess the number of clusters in a dataset, such as the Calinski-Harabasz score, Davies-Bouldin, and Dunn value [10]. The best \(k\) value, i.e., \(k^{*}\), may then be discovered by graphing any of these scores against each cluster number; \(k^{*}\) is often located where the graph resembles an arm (i.e., the \"elbow\"--the point of inflection of the curve) as will be shown in Section 3. The primary problem with the elbow approach is that it is interpretation is dependent on the visual analysis of an elbow-based graph, which necessitates manual human inspection in order to determine the elbow point and, as a result, can produce very subjective results. Furthermore, when the approach produces a graph with a smooth curve, then estimating the value of \(k^{*}\) becomes difficult since the actual elbow point becomes harder to distinguish. Most notably, the outcome of the elbow graph is typically a function of the number of \(K\) iterations. Consequently, the elbow point would often vary dependent on the value of \(K\), a circumstance for which many approaches are not readily adaptable. Nonetheless, there are a number of notable methods for automating the identification of the elbow point in the pursuit of building automated K-means or other K-dependent clustering algorithms, such as the elbow strength approach (ESA) [11], knee-point algorithm (KPA) [12], Kneedle algorithm [13], slope changing ratio method (SCRM) [14], and the angle-based discriminant method (ADM) [15]. A closer examination reveals that both the KPA and the SCRM work upon an identical concept of computing the intersection point that minimizes the error point of both best lines of fit.