Pairwise Adjusted Mutual Information

A well-known metric for quantifying the similarity between two clusterings is1 the adjusted mutual information . Compared to mutual information , a corrective2 term based on random permutations of the labels is introduced , preventing two3 clusterings being similar by chance . Unfortunately , this adjustment makes the4 metric computationally expensive . In this paper , we propose a novel adjustment5 based on pairwise label permutations instead of full label permutations . Specifically,6 we consider permutations where only two samples , selected uniformly at random,7 exchange their labels . We show that the corresponding adjusted metric , which8 can be expressed explicitly , behaves similarly to the standard adjusted mutual9 information for assessing the quality of a clustering , while having a much lower10 time complexity . Both metrics are compared in terms of quality and performance11 on experiments based on synthetic and real data.12 1 Introduction13 A well-known metric for quantifying the similarity between two clusterings of the same data is14 the adjusted mutual information [ Nguyen et al. , 2009 ; Vinh et al. , 2010 ] . Compared to mutual15 information , this metric is adjusted against chance , meaning that the similarity can not be due to16 randomness but only to the structure of the dataset , appearing in both clusterings . This is the reason17 why this metric is widely used in unsupervised learning , see [ Zhang et al. , 2013 ; Thirion et al. , 2014 ; 18 Taha and Hanbury , 2015 ; Yang et al. , 2016 ; Wang et al. , 2017 ] for various applications.19 The standard way of adjusting mutual information against chance is through random label permuta-20 tions of one of the clusterings [ Vinh et al. , 2010 ] . Unfortunately , this adjustment makes the metric21 computationally expensive . Specifically , the time complexity of the metric is in O ( max ( k , l ) n ) ,22 where k , l are the numbers of clusters in each clustering and n is the number of samples [ Romano et23 al. , 2014 ] . As a comparison , the time complexity of mutual information is equal to O ( kl ) given the24 contingency matrix of the clusterings , i.e. , the matrix counting the number of samples in each pair of25 clusters , one per clustering . The additional computational effort required by adjustment is significant26 as the number of samples n is typically much larger than the numbers of clusters k , l.27 In this paper , we propose a novel adjustment based on pairwise permutations . That is , we consider28 permutations where only two samples , selected uniformly at random , exchange their labels . We29 show that the corresponding adjusted metric , we refer to as pairwise adjusted mutual information,30 is as efficient as adjusted mutual information for assessing the quality of a clustering , with a much31 lower time complexity . In particular , the time complexity is the same as that of mutual information.32 The gain in complexity is significant , as the computation time is now independent of the number of33 samples n , given the contingency matrix.34 The rest of the paper is organized as follows . We first provide the definition and key properties of35 adjusted mutual information in the general setting of information theory . We then introduce mutual36 information with pairwise adjustement and explain why the exact same properties are satisfied by37 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . this new notion of adjusted mutual information . The application of both notions of adjustment to38 clustering , including the explicit expressions of the corresponding metrics , is presented in section 4.39 Experiments on both synthetic and real data are presented in section 5 . Section 6 concludes the paper.40 2 Adjusted mutual information41 Let P be the uniform probability measure on Ω = { 1 , . . . , n } , for some positive integer n. Let X , Y42 be random variables on the probability space ( Ω , P ) . Without any loss of generality , we assume that43 X and Y are mapping from Ω to sets consisting of consecutive integers , starting from 1 . Denoting by44 H the entropy , the mutual information between X and Y is defined by [ Cover and Thomas , 1991 ] :45 I ( X , Y ) = H ( X ) +H ( Y ) −H ( X , Y ) . ( 1 ) This is the information shared by X and Y , which is equal to 0 if X and Y are independent . A distance between X and Y can then be defined by : d ( X , Y ) = H ( X , Y ) − I ( X , Y ) = H ( X|Y ) +H ( Y |X ) . This distance , known as the variation of information , is a metric in the quotient space of random46 variables under the equivalence relation X ∼ Y if and only if there is some bijection ϕ such that47 X = ϕ ( Y ) [ Meilă , 2003 ] .48 Adjusted mutual information . The adjusted mutual information betweenX and Y , corresponding49 to the mutual information between X and Y adjusted against chance , is defined by:50 ∆I ( X , Y ) = I ( X , Y ) − E ( I ( X , Yσ ) ) , ( 2 ) where Yσ is the random variable Y ◦ σ , for any permutation σ of { 1 , . . . , n } , and the expectation is51 taken over all permutations σ , chosen uniformly at random.52 Remark 1 ( Normalization ) . It is frequent to also normalize adjusted mutual information , so as to53 get a score between 0 and 1 [ Vinh et al. , 2010 ; Romano et al. , 2014 ] . In this paper , we only focus on54 the adjustment step . Note that normalization can be equally applied to both considered notions of55 adjustment and thus be studied separately.56 We have the equivalent definition:57 ∆I ( X , Y ) = E ( H ( X , Yσ ) ) −H ( X , Y ) , = 1 2 ( E ( d ( X , Yσ ) ) − d ( X , Y ) ) . ( 3 ) This equivalence follows from Proposition 1 and the fact that the definition is symmetric in X and Y .58 All proofs are available in the supplementary material.59 Proposition 1 . We have for any random variables X and Y :60 H ( X ) = E ( H ( Xσ ) ) , E ( H ( X , Yσ ) ) = E ( H ( Xσ , Y ) ) , E ( I ( X , Yσ ) ) = E ( I ( Xσ , Y ) ) . In view of ( 3 ) , we expect ∆I ( X , Y ) to be positive if X and Y share information , as X is expected to61 be closer to Y ( for the distance d ) than to Yσ , a randomized version of Y . There are specific cases62 where ∆I ( X , Y ) = 0 , as stated in Proposition 2 ; these cases will be interpreted in terms of clustering63 in section 4.64 Proposition 2 . We have ∆I ( X , Y ) = 0 whenever Y ( or X , by symmetry ) is constant or equal to65 some permutation of { 1 , . . . , n } .66 Adjusted entropy . Observing that H ( X ) = I ( X , X ) , we define similarly the adjusted entropy of X by : ∆H ( X ) = ∆I ( X , X ) = H ( X ) − E ( I ( X , Xσ ) ) . By ( 1 ) , we get:67 ∆H ( X ) = E ( H ( X , Xσ ) ) −H ( X ) = 1 2 E ( d ( X , Xσ ) ) . ( 4 ) Since d is a metric , this shows that the adjusted entropy of X is non-negative.68 Proposition 3 . We have ∆H ( X ) = 0 if and only if X is constant or equal to some permutation of69 { 1 , . . . , n } .70 Proposition 3 characterizes random variables with zero adjusted entropy . Again , this result will be71 interpreted in terms of clustering in section 4.72 3 Pairwise adjustment73 In this section , we introduce pairwise adjusted mutual information . The definition is the same as74 adjusted mutual information , except that the permutation σ is now restricted to the set of pairwise75 permutations . Specifically , we consider permutations σ for which there exists i , j ∈ { 1 , . . . , n } 76 such that σ ( i ) = j and σ ( j ) = i , whereas σ ( t ) = t for all t 6= i , j . We consider the set of such77 permutations σ where the samples i , j are drawn uniformly at random in the set { 1 , . . . , n } . We78 denote by σp such a random permutation . Observe that σp is the identity with probability 1/n ( the79 probability that i = j ) .80 Pairwise adjusted mutual information . We define the pairwise adjusted mutual information as : ∆pI ( X , Y ) = I ( X , Y ) − E ( I ( X , Yσp ) ) . This is exactly the same definition as the adjusted mutual information , except for the considered81 permutations σp . It can be readily verified that the same properties apply , with the exact same proofs,82 a key property being that the random permutations σp and σp−1 have the same distributions . In83 particular , we have the analogue of ( 3 ) :84 ∆pI ( X , Y ) = E ( H ( X , Yσp ) ) −H ( X , Y ) , = 1 2 ( E ( d ( X , Yσp ) ) − d ( X , Y ) ) . ( 5 ) Moreover , ∆pI ( X , Y ) = 0 whenever X or Y is constant or equal to some permutation of { 1 , . . . , n } .85 Pairwise adjusted entropy . We also define the pairwise adjusted entropy as : ∆pH ( X ) = ∆pI ( X , X ) = H ( X ) − E ( I ( X , Xσp ) ) . We have ∆pH ( X ) ≥ 0 , with equality if and only if X is constant or equal to some permutation of86 { 1 , . . . , n } .87 4 Application to clustering88 Let A = { A1 , . . . , Ak } and B = { B1 , . . . , Bl } be two partitions of some finite set { 1 , . . . , n } into k89 and l clusters , respectively . Let Ω = { 1 , . . . , n } and P be the uniform probability measure over Ω.90 Consider the random variables X and Y defined on ( Ω , P ) by X−1 ( i ) = Ai for all i = 1 , . . . , k and91 Y −1 ( j ) = Bj for all j = 1 , . . . , l. Note that X ( ω ) and Y ( ω ) can be interpreted as the labels i and j92 of sample ω in clusterings A and B , for each ω ∈ { 1 , . . . , n } .93 We denote by ai = |Ai| the size of cluster Ai , by bj = |Bj | the size of cluster Bj , and by94 nij = |Ai ∩ Bj | the number of samples both in cluster Ai and cluster Bj , for all i = 1 , . . . , k and95 j = 1 , . . . , l. The matrix ( nij ) 1≤i≤k,1≤j≤l is known as the contingency matrix . Note that ai and bj96 are the sums of row i and column j of the contingency matrix , respectively.97 Adjusted mutual information . A well-known metric for assessing the similarity s ( A , B ) between98 clusterings A and B is the adjusted mutual information1 ∆I ( X , Y ) between the corresponding99 random variables X and Y . In words , this is the common information shared by clusterings A and B100 not due to randomness.101 By Proposition 2 , we have s ( A , B ) = 0 whenever clustering A ( or B , by symmetry ) is trivial , that is,102 it consists of a single cluster or of n clusters ( one per sample ) . This is a key property , showing the103 interest of the adjustment.104 1Recall that we don ’ t normalize the metric , see Remark 1 . It is known that [ Vinh et al. , 2010 ] :105 s ( A , B ) = − k∑ i=1 l∑ j=1 nij n log nij n + k∑ i=1 l∑ j=1 min ( ai , bj ) ∑ c= ( ai+bj−n ) + ai ! bj ! ( n− ai ) ! ( n− bj ) ! n ! c ! ( ai − c ) ! ( bj − c ) ! ( n− ai − bj + c ) ! c n log c n , ( 6 ) with the notation ( · ) + = max ( · , 0 ) . The time complexity of this formula , which is dominated by the106 second term , is in O ( max ( k , l ) n ) [ Romano et al. , 2014 ] . In particular , it is linear in the number of107 samples n.108 Interestingly , we can similarly assess the quantity of information q ( A ) contained in clustering109 A through the adjusted entropy ∆H ( X ) of the corresponding random variable X . This is the110 information contained in A not due to randomness . We have q ( A ) ≥ 0 and , by Proposition 3,111 q ( A ) = 0 if and only if clustering A is trivial , that is , it consists of a single cluster or of n clusters112 ( one per sample ) .113 Since q ( A ) = s ( A , A ) , it follows from ( 6 ) that:114 q ( A ) = − k∑ i=1 ai n log ai n + K∑ i , j=1 min ( ai , aj ) ∑ c= ( ai+aj−n ) + + ai ! aj ! ( n− ai ) ! ( n− aj ) ! n ! c ! ( ai − c ) ! ( aj − c ) ! ( n− ai − aj + k ) ! c n log c n . The time complexity of this formula , also dominated by the second term , is in O ( kn ) . Again , this115 complexity is linear in the number of samples n.116 Pairwise adjusted mutual information . The main contribution of the paper is the following new117 measure of similarity sp ( A , B ) between clusterings A and B , based on the pairwise adjusted mutual118 information ∆pI ( X , Y ) between the corresponding random variables X and Y . We have an explicit119 expression for this similarity:120 Theorem 1 . We have for any clusterings A , B:121 sp ( A , B ) = 2 k∑ i=1 l∑ j=1 nij ( n− ai − bj + nij ) n2 ( nij n log nij n − nij − 1 n log nij − 1 n ) + 2 k∑ i=1 l∑ j=1 ( ai − nij ) ( bj − nij ) n2 ( nij n log nij n − nij + 1 n log nij + 1 n ) . The time complexity of this formula is in O ( kl ) , like mutual information . It is independent of the122 number of samples n , given the contingency matrix . Corollary 1 shows that the time complexity123 reduces to O ( m ) , where m is the number of non-zero entries of the contingency matrix , provided the124 latter is stored in sparse format.125 Corollary 1 . We have for any clusterings A , B:126 sp ( A , B ) = 2 ∑ i , j : nij > 0 nij ( n− ai − bj + nij ) n2 ( nij n log nij n − nij − 1 n log nij − 1 n ) + 2 ∑ i , j : nij > 0 ( ai − nij ) ( bj − nij ) n2 ( nij n log nij n − nij + 1 n log nij + 1 n + 1 n log 1 n ) − 2 n2 − k∑ i=1 a2i − l∑ j=1 b2i + ∑ i , j : nij > 0 n2ij 1 n log 1 n . Similarly , we can define the quantity of information qp ( A ) in clustering A through the pairwise127 adjusted entropy ∆pH ( X ) of the corresponding random variable X . Again , qp ( A ) ≥ 0 , with128 qp ( A ) = 0 if and only if clustering A is trivial.129 Corollary 2 . We have for any clustering A:130 qp ( A ) = 2 k∑ i=1 ai ( n− ai ) n2 ( ai n log ai n − ai − 1 n log ai − 1 n − 1 n log 1 n ) . Note that the time complexity of this formula in O ( k ) . It only depends on the number of clusters k,131 and not on the number of samples n.132 5 Experiments133 In this section , we compare both notions of adjusted mutual information through experiments134 involving synthetic and real data . The experiments are run on a computer equipped with an AMD135 Ryzen Threadripper 1950X 16-Core Processor and 32 GB of RAM , with a a Debian 10 OS . All codes136 and datasets used in the experiments are available in the supplementary material.137 Synthetic data . We start with the simple case of n = 100 samples with clusters of even sizes,138 consisting of consecutive samples . Specifically , we consider the set of clusterings A ( s ) , consisting of139 clusters of size s ( except possibly the last one ) , for s = 1 , 2 , . . . , 100 . In particular , both A ( 1 ) and140 A ( 100 ) are trivial clusterings while A ( 5 ) consists of 20 clusters of size 5.141 Figure 1 gives the similarity between clusterings A ( 10 ) and A ( s ) with respect to s in terms of adjusted142 mutual information , for both notions of adjustment , i.e. , s ( A ( 10 ) , A ( s ) ) and sp ( A ( 10 ) , A ( s ) ) . We143 observe very close behaviors , suggesting that both notions of adjustment tend to capture the same144 patterns in the clusterings . Note that the maximum similarity is attained for s = 10 in both cases , as145 expected . The similarity is equal to 0 for s ∈ { 1 , 100 } for both cases , in agreement with Proposition146 2 . We also observe local peaks at s = 20 , 30 , . . . , 90 , which can be interpreted by the fact that147 clustering A ( 10 ) is a refinement of clustering A ( s ) for these values of s ; similarly , the local peak at148 s = 5 may be interpreted by the fact that clustering A ( 5 ) is a refinement of clustering A ( 10 ) . The149 Spearman correlation between both metrics over all values of s is equal to 0.99 . 150 We now consider random clusterings . Specifically , we assign n samples to k clusters independently151 at random , according to some probability distribution p = ( p1 , . . . , pk ) , which is itself drawn at152 random2 . Consider three such random clusterings A , B , C ( with the same parameters n and k , but153 different probability distributions p ) . We would like to know whether A is “ closer '' to B or to C. In154 particular , we are interested in testing whether both notions of adjusted mutual information give the155 same ordering in the sense that:156 ( s ( A , B ) − s ( A , C ) ) ( sp ( A , B ) − sp ( A , C ) ) ≥ 0 . ( 7 ) 2Namely , p ∝ U where U = ( U1 , . . . , Uk ) is a vector of k i.i.d . random variables uniformly distributed over [ 0 , 1 ] . We compute the average precision score ( fraction of triplets A , B , C for which ( 7 ) is true ) over 1 000157 independent samples of A , B , C , for different values of n and k. We repeat the experiment 100 times158 to get the mean and standard deviation . The results are given in Table 1 . We observe a very high159 precision score , always higher than 93 % , showing that both notions of adjusted mutual information160 tend to give the same ordering of these random clusterings.161 For the performance gain , we compare the computation times of both versions of adjusted mutual162 information for the similarity between clusterings A and B , where A consists of k = 10 clusters163 of same size and B is a random clustering , drawn as in the previous experiment . Both versions of164 adjusted mutual information are coded in Python , with the standard version imported from scikit-learn.165 Figure 2 shows the computation time when the number of samples n grows from 102 to 107 . The166 performance gain brought by pairwise adjustement is significant . In particular , the computation time167 becomes independent of the number of samples.168 Real data . We first consider the 79 datasets of the benchmark suite [ Gagolewski , 2020 ] 3 . We apply169 to each dataset each of the following clustering algorithms:170 • k-means171 • Affinity propagation172 • Mean shift173 • Spectral clustering174 • Ward175 • Agglomerative clustering176 • DBSCAN177 • OPTICS178 • Birch179 • Gaussian Mixture180 3See https : //github.com/gagolews/clustering_benchmarks_v1 We use the scikit-learn4 implementation of these algorithms , with the corresponding default param-181 eters5 . We get 10 clusterings per dataset . The quality of each clustering is assessed through the182 similarity with the available ground-truth labels , using adjusted mutual information with either full183 adjustment or pairwise adjustment . We then compute the Spearman correlation of the corresponding184 similarities , a value of 1 meaning the exact same ordering of the 10 clusterings with full adjustment185 and pairwise adjustment . The results are shown in Figure 3 , together with the speed-up in computation186 time due to pairwise adjustment . In both cases , the 79 datasets are ordered by the number of samples,187 ranging from 105 to 105 600 [ Gagolewski , 2020 ] .188 We first observe that the correlation is very high , suggesting again that both notions of adjusted mutual189 information tend to provide the same results . For 65 datasets among 79 , the Spearman correlation is190 higher than 95 % . As for the computation time , we observe a significant performance gain , by one191 order of magnitude for the largest datasets.192 We have conducted the same experiments with OpenML [ Vanschoren et al. , 2013 ] 6 . We selected all193 datasets with at least 1,000 but no more than 50,000 samples , at most 100 features ( all numerical ) , no194 missing data and ground-truth labels forming clusters of at least 5 samples on average . The results195 are shown in Figure for the resulting 34 datasets . Again , the datasets are ordered by the number of196 samples , here ranging from 1,188 to 45,918 . The conclusions are similar . In particular , the Spearman197 correlation is higher than 95 % for 30 datasets among 34 , and the performance gain exceeds 25 for the198 largest datasets.199 4https : //scikit-learn.org/ 5Dimension reduction is applied to the MNIST datasets , consisting of 70 000 images of size 28× 28 each , see the supplementary material for details . 6https : //www.openml.org 6 Conclusion200 We have proposed another way of adjusting mutual information against chance , through pairwise201 label permutations . The novel metric , whose explicit expression is given in Theorem 1 , has a much202 lower complexity than the usual adjusted mutual information . Interestingly , both metrics can also be203 used to assess the quantity of information contained in a clustering , which the common property of204 being equal to 0 if and only if the clustering is trivial , as stated in Proposition 3 ; again , the pairwise205 adjusted entropy , given in Corollary 2 , has a much lower complexity . Experiments on synthetic and206 real data show that pairwise adjusted mutual information tends to provide the same results as the usual207 adjusted mutual information for comparing clusterings , while involving much less computations.208 For future work , we plan to extend this idea to other similarity metrics . While the practical interest209 is less obvious for the Adjusted Rand Index [ Hubert and Arabie , 1985 ] , due to the fact that the210 time complexity of this metric is already independent of the number of samples , it would be worth211 considering other versions of information theoretic measures , as those studied in [ Romano et al.,212 2016 ] .213 References214 Thomas M Cover and Joy A Thomas . Elements of Information Theory . Wiley , 1991.215 Marek Gagolewski . Benchmark suite for clustering algorithms – version 1 , 2020.216 Lawrence Hubert and Phipps Arabie . Comparing partitions . Journal of classification , 2 ( 1 ) :193–218,217 1985.218 Marina Meilă . Comparing clusterings by the variation of information . In Learning theory and kernel219 machines , pages 173–187 . Springer , 2003.220 Xuan Vinh Nguyen , Julien Epps , and James Bailey . Information theoretic measures for clusterings221 comparison : is a correction for chance necessary ? In ICML , 2009.222 Simone Romano , James Bailey , Vinh Nguyen , and Karin Verspoor . Standardized mutual information223 for clustering comparisons : one step further in adjustment for chance . In International Conference224 on Machine Learning , pages 1143–1151 , 2014.225 Simone Romano , Nguyen Xuan Vinh , James Bailey , and Karin Verspoor . Adjusting for chance226 clustering comparison measures . The Journal of Machine Learning Research , 17 ( 1 ) :4635–4666,227 2016.228 Abdel Aziz Taha and Allan Hanbury . Metrics for evaluating 3d medical image segmentation : analysis,229 selection , and tool . BMC medical imaging , 15 ( 1 ) :29 , 2015.230 Bertrand Thirion , Gaël Varoquaux , Elvis Dohmatob , and Jean-Baptiste Poline . Which fmri clustering231 gives good brain parcellations ? Frontiers in neuroscience , 8:167 , 2014.232 Joaquin Vanschoren , Jan N. van Rijn , Bernd Bischl , and Luis Torgo . Openml : Networked science in233 machine learning . SIGKDD Explorations , 15 ( 2 ) :49–60 , 2013.234 Nguyen Xuan Vinh , Julien Epps , and James Bailey . Information theoretic measures for clusterings235 comparison : Variants , properties , normalization and correction for chance . The Journal of Machine236 Learning Research , 11:2837–2854 , 2010.237 Bo Wang , Junjie Zhu , Emma Pierson , Daniele Ramazzotti , and Serafim Batzoglou . Visualization238 and analysis of single-cell rna-seq data by kernel-based similarity learning . Nature methods,239 14 ( 4 ) :414–416 , 2017.240 Zhao Yang , René Algesheimer , and Claudio J Tessone . A comparative analysis of community241 detection algorithms on artificial networks . Scientific reports , 6:30750 , 2016.242 Jiajie Zhang , Paschalia Kapli , Pavlos Pavlidis , and Alexandros Stamatakis . A general species243 delimitation method with applications to phylogenetic placements . Bioinformatics , 29 ( 22 ) :2869–244 2876 , 2013.245 Checklist246 1 . For all authors ... 247 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s248 contributions and scope ? [ Yes ] A variant of adjusted mutual information.249 ( b ) Did you describe the limitations of your work ? [ Yes ] Pairwise adjustement only applied250 to mutual information in the present work , see Section 6.251 ( c ) Did you discuss any potential negative societal impacts of your work ? [ N/A ] 252 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to253 them ? [ Yes ] 254 2 . If you are including theoretical results ... 255 ( a ) Did you state the full set of assumptions of all theoretical results ? [ Yes ] No specific256 assumption is required . Theorem 1 , Corollary 2 and 3 give explicit expressions using257 notations defined at the beginning of Section 4.258 ( b ) Did you include complete proofs of all theoretical results ? [ Yes ] See the supplementary259 material.260 3 . If you ran experiments ... 261 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-262 mental results ( either in the supplemental material or as a URL ) ? [ Yes ] See the Jupyter263 notebooks in the supplementary material.264 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they265 were chosen ) ? [ N/A ] This is a metric for unsupervised learning . No data split required,266 no hyperparameter.267 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running ex-268 periments multiple times ) ? [ Yes ] See Table 1 and Figure 2 ( not applicable to other269 experiments ) .270 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type271 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] See the running times provided in272 the Figures ; the resources used are detailed at the beginning of section 5.273 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 274 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] See the references for275 the datasets.276 ( b ) Did you mention the license of the assets ? [ N/A ] 277 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ N/A ] 278 279 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re280 using/curating ? [ N/A ] 281 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable282 information or offensive content ? [ N/A ] 283 5 . If you used crowdsourcing or conducted research with human subjects ... 284 ( a ) Did you include the full text of instructions given to participants and screenshots , if285 applicable ? [ N/A ] 286 ( b ) Did you describe any potential participant risks , with links to Institutional Review287 Board ( IRB ) approvals , if applicable ? [ N/A ] 288 ( c ) Did you include the estimated hourly wage paid to participants and the total amount289 spent on participant compensation ? [ N/A ] 290

Summary: In this paper authors propose an efficient version of adjusted mutual information, termed pairwise adjusted mutual information, for cluster similarity evaluation. The topic is attractive and the results on both synthetic and real-world datasets demonstrate the effectiveness of the algorithm. However, the expression of the paper needs to be finely improved. Also, the depth of the theoretical analysis needs to be further improved. More analysis should be given.