Systematic Analysis of Cluster Similarity Indices: How to Validate Validation Measures

1 INTRODUCTION . Clustering is an unsupervised machine learning problem , where the task is to group objects that are similar to each other . In network analysis , a related problem is called community detection , where grouping is based on relations between items ( links ) , and the obtained clusters are expected to be densely interconnected . Clustering is used across various applications , including text mining , online advertisement , anomaly detection , and many others ( Allahyari et al. , 2017 ; Xu & Tian , 2015 ) . To measure the quality of a clustering algorithm , one can use either internal or external measures . Internal measures evaluate the consistency of the clustering result with the data being clustered , e.g. , Silhouette , Hubert-Gamma , Dunn , and many other indices . Unfortunately , it is unclear whether optimizing any of these measures would translate into improved quality in practical applications . External ( cluster similarity ) measures compare the candidate partition with a reference one ( obtained , e.g. , by human assessors ) . A comparison with such a gold standard partition , when it is available , is more reliable . There are many tasks where external evaluation is applicable : text clustering ( Amigó et al. , 2009 ) , topic modeling ( Virtanen & Girolami , 2019 ) , Web categorization ( Wibowo & Williams , 2002 ) , face clustering ( Wang et al. , 2019 ) , news aggregation ( see Section 3 ) , and others . Often , when there is no reference partition available , it is possible to let a group of experts annotate a subset of items and compare the algorithms on this subset . Dozens of cluster similarity measures exist and which one should be used is a subject of debate ( Lei et al. , 2017 ) . In this paper , we systematically analyze the problem of choosing the best cluster similarity index . We start with a series of experiments demonstrating the importance of the problem ( Section 3 ) . First , we construct simple examples showing the inconsistency of all pairs of different similarity indices . Then , we demonstrate that such disagreements often occur in practice when well-known clustering algorithms are applied to real datasets . Finally , we illustrate how an improper choice of a similarity index can affect the performance of production systems . So , the question is : how to compare cluster similarity indices and choose the best one for a particular application ? Ideally , we would want to choose an index for which good similarity scores translate to good real-world performance . However , opportunities to experimentally perform such a validation of validation indices are rare , typically expensive , and do not generalize to other applications . In contrast , we suggest a theoretical approach : we formally define properties that are desirable across various applications , discuss their importance , and formally analyze which similarity indices satisfy them ( Section 4 ) . This theoretical framework would allow practitioners to choose the best index based on relevant properties for their applications . In Section 5 , we advocate two indices that are expected to be suitable across various applications . While many ideas discussed in the paper can be applied to all similarity indices , we also provide a more in-depth theoretical characterization of pair-counting ones ( e.g. , Rand and Jaccard ) , which gives an analytical background for further studies of pair-counting indices . We formally prove that among dozens of known indices , only two have all the properties except for being a distance : Correlation Coefficient and Sokal & Sneath ’ s first index ( Lei et al. , 2017 ) . Surprisingly , both indices are rarely used for cluster evaluation . The correlation coefficient has an additional advantage of being easily convertible to a distance measure via the arccosine function . The obtained index has all the properties except constant baseline , which is still satisfied asymptotically . Constant baseline is a particular focus of the current research : this is one of the most important and non-trivial properties . Informally , a sensible index should not prefer one candidate partition over another just because it has too large or too small clusters . To the best of our knowledge , we are the first to develop a rigorous theoretical framework for analyzing this property . In this respect , our work improves over the previous ( mostly empirical ) research on constant baseline of particular indices ( Albatineh et al. , 2006 ; Lei et al. , 2017 ; Strehl , 2002 ; Vinh et al. , 2009 ; 2010 ) , we refer to Appendix A for a detailed comparison to related research . 2 CLUSTER SIMILARITY INDICES . We assume that there is a set of elements V with size n = |V | . A clustering is a partition of V into disjoint subsets . Capital letters A , B , C will be used to name the clusterings , and we will represent them as A = { A1 , . . . , AkA } , where Ai is the set of elements belonging to i-th cluster . If a pair of elements v , w ∈ V lie in the same cluster in A , we refer to them as an intra-cluster pair of A , while inter-cluster pair will be used otherwise . The total number of pairs is denoted by N = ( n 2 ) . The value that an index I assigns to the similarity between partitions A and B will be denoted by I ( A , B ) . Let us now define some of the indices used throughout the paper , while a more comprehensive list , together with formal definitions , is given in Appendix B.1 and B.2 . Pair-counting indices consider clusterings to be similar if they agree on many pairs . Formally , let ~A be the N -dimensional vector indexed by the set of element-pairs , where the entry corresponding to ( v , w ) equals 1 if ( v , w ) is an intra-cluster pair and 0 otherwise . Further , let MAB be the N × 2 matrix that results from concatenating the two ( column- ) vectors ~A and ~B . Each row of MAB is either 11 , 10 , 01 , or 00 . Let the pair-counts N11 , N10 , N01 , N00 denote the number of occurrences for each of these rows in MAB . Definition 1 . A pair-counting index is a similarity index that can be expressed as a function of the pair-counts N11 , N10 , N01 , N00 . Some popular pair-counting indices are Rand and Jaccard : R = N11 +N00 N11 +N10 +N01 +N00 , J = N11 N11 +N10 +N01 . Adjusted Rand ( AR ) is a linear transformation of Rand ensuring that for a random B we have AR ( A , B ) = 0 in expectation . A less widely used index is the Pearson Correlation Coefficient ( CC ) between the binary incidence vectors ~A and ~B.1 Another index , which we discuss further in more details , is the Correlation Distance CD ( A , B ) : = 1π arccosCC ( A , B ) . In Table 4 , we formally define 27 known pair-counting indices and only mention ones of particular interest throughout the main text . Information-theoretic indices consider clusterings similar if they share a lot of information , i.e. , if little information is needed to transform one clustering into the other . Formally , let H ( A ) : = H ( |A1|/n , . . . , |AkA |/n ) be the Shannon entropy of the cluster-label distribution of A . Similarly , the joint entropyH ( A , B ) is defined as the entropy of the distribution with probabilities ( pij ) i∈ [ kA ] , j∈ [ kB ] , where pij = |Ai ∩ Bj |/n . Then , the mutual information of two clusterings can be defined as 1Note that Spearman and Pearson correlation are equal when comparing binary vectors . Kendall rank correlation for binary vectors coincides with the Hubert index , which is linearly equivalent to Rand . M ( A , B ) = H ( A ) + H ( B ) − H ( A , B ) . There are multiple ways of normalizing the mutual information , the most widely used ones are : NMI ( A , B ) = M ( A , B ) ( H ( A ) +H ( B ) ) /2 , NMImax ( A , B ) = M ( A , B ) max { H ( A ) , H ( B ) } . NMI is known to be biased towards smaller clusters , and several modifications try to mitigate this bias : Adjusted Mutual Information ( AMI ) and Standardized Mutual Information ( SMI ) subtract the expected mutual information from M ( A , B ) and normalize the obtained value ( Vinh et al. , 2009 ) , while Fair NMI ( FNMI ) multiplies NMI by a penalty factor e−|kA−kB |/kA ( Amelio & Pizzuti , 2015 ) . 3 MOTIVATING EXPERIMENTS . As discussed in Section 2 , many cluster similarity indices are used by researchers and practitioners . A natural question is : how to choose the best one ? Before trying to answer this question , it is important to understand whether the problem is relevant . Indeed , if the indices are very similar to each other and agree in most practical applications , then one can safely take any index . In this section , we demonstrate that this is not the case , and the proper choice matters . First , we illustrate the inconsistency of all indices . We say that two indices I1 and I2 are inconsistent for a triplet of partitions ( A , B1 , B2 ) if I1 ( A , B1 ) > I1 ( A , B2 ) but I2 ( A , B1 ) < I2 ( A , B2 ) . We took 15 popular cluster similarity measures and constructed just four triplets such that each pair of indices is inconsistent for at least one triplet . One example is shown in Figure 1 : for this simple example , about half of the indices prefer the left candidate , while the others prefer the right one . Other examples can be found in Appendix F.1 . NMI VI AR S & S1 CC NMI – 40.3 15.7 20.1 18.5 VI – 37.6 36.0 37.2 AR – 11.7 8.3 S & S1 – 3.6 CC – tition and B1 , B2 are provided by two algorithms . For a given pair of indices and all such triplets , we look at whether the indices are consistent . Table 1 shows the relative inconsistency for several indices ( the extended table together with a detailed description of the experimental setup and more analysis is given in Appendix F.2 ) . The inconsistency rate is significant : e.g. , popular measures Adjusted Rand and Variation if Information disagree in almost 40 % of the cases , which is huge . Interestingly , the best agreeing indices are S & S and CC , which satisfy most of our properties , as shown in the next section . In contrast , the Variation of Information very often disagrees with other indices . To show that the choice of similarity index may affect the final performance in a real production scenario , we conducted an experiment within a major news aggregator system . The system groups news articles to events and shows the list of most important events to users . For grouping , a clustering algorithm is used , and the quality of this algorithm affects the user experience : merging different clusters may lead to not showing an important event , while too much splitting may cause duplicate events . When comparing several candidate clustering algorithms , it is important to determine which one is the best for the system . Online experiments are expensive and can be used only for the best candidates . Thus , we need a tool for an offline comparison . For this purpose , we manually created a reference partition on a small fraction of news articles . We can use this partition to evaluate the candidates . We performed such an offline comparison for two candidate algorithms and observed that different indices preferred different algorithms . Then , we launched an online user experiment and verified that one of the candidates is better for the system according to user preferences . Hence , it is important to be careful when choosing a similarity index for the offline comparison . See Appendix F.3 for more detailed description of this experiment and quantitative analysis .

Summary: Cluster Similarity Indices (CSIs) take as input two clusterings A, B and assign a similarity score for the given pair of clusterings. The index calculates a score based on the number of pairs of elements that clustered together on both clustering (N++), those that are not clustered together in non of A,B (N--), those that are clustered together in A but not in B (N+-), and vice versa (N-+). CSIs can be used to evaluate clusterings produced by different algorithms with respect to some reliable reference clustering on a single instance, and choose the one that is the closest to the reference clustering (indicated by the CSIs). The selected clustering algorithm can be then applied to different instances of the same kind where we do not have a reference clustering.