Title: $p$-value Adjustment for Monotonous, Unbiased, and Fast Clustering Comparison
Venue: NeurIPS 2023
Accepted: yes
Keywords: Clustering

(Other) Machine Learning Topics
Abstract: Popular metrics for clustering comparison, like the Adjusted Rand Index and the Adjusted Mutual Information, are type II biased. The Standardized Mutual Information removes this bias but suffers from counterintuitive non-monotonicity and poor computational efficiency. We introduce the $p$-value adjusted Rand Index ($\operatorname{PMI}_2$), the first cluster comparison method that is type II unbiased and provably monotonous. The $\operatorname{PMI}_2$ has fast approximations that outperform the Standardized Mutual information. We demonstrate its unbiased clustering selection, approximation quality, and runtime efficiency on synthetic benchmarks. In experiments on image and social network datasets, we show how the $\operatorname{PMI}_2$ can help practitioners choose better clustering and community detection algorithms.

p -value Adjustment for Monotonous, Unbiased, and Fast Clustering Comparison

Abstract

Popular metrics for clustering comparison, like the Adjusted Rand Index and the Adjusted Mutual Information, are type II biased. The Standardized Mutual Information removes this bias but suffers from counterintuitive non-monotonicity and poor computational efficiency. We introduce the p -value adjusted Rand Index ( PMI 2 ), the first cluster comparison method that is type II unbiased and provably monotonous. The PMI 2 has fast approximations that outperform the Standardized Mutual information. We demonstrate its unbiased clustering selection, approximation quality, and runtime efficiency on synthetic benchmarks. In experiments on image and social network datasets, we show how the PMI 2 can help practitioners choose better clustering and community detection algorithms.

1 Introduction

Clustering is fundamental to unsupervised learning, and practitioners can choose from many algorithms to partition a dataset into homogeneous clusters. Therefore it is common to annotate parts of an otherwise unlabeled dataset and select the clustering algorithm that best reproduces the annotations [1]. A good selection crucially depends on the clustering comparison method, like the Mutual Information (MI) [25] or the Rand Index (RI) [39]. The importance of the comparison method further increases with the advent of deep learning methods for applications such as community detection, in which they serve as components of loss functions during network training [10, 12, 33]. Some use cases admit multiple clustering solutions [23, 38], and clustering comparison can help identify qualitatively different clustering solutions for a single dataset. Other applications of clustering comparison include categorical feature selection or consensus clustering [5, 17].

The MI and RI are biased towards clusterings with particular cluster size distributions [19] (type I bias). For example, the MI favors larger clusters for any reference clustering [24]. The Adjusted Rand Index (ARI) [11] and Adjusted Mutual Information (AMI) [25] achieve a constant baseline value by subtracting the expected value under random permutation of the cluster labels. However, they still exhibit a bias when multiple clusterings are compared via a fixed ground truth (type II bias) [29], as opposed to comparing two random clusterings with each other. This type II scenario typically arises when selecting the best algorithm for a given task on a labeled subset of the data.

Romano et al. [30] showed via the Tsallis entropy, that the AMI and ARI are special cases of generalized information-theoretic clustering comparison measures AMI q and proposed standardization to resolve both types of biases. However, the runtime for standardization generally exhibits a substantial time complexity of O ( N 3 k A max( k A , k B )) [30], where N represents the number of data points and k A and k B denote the respective number of clusters. This complexity is prohibitive for many applications [15]. Gösgens et al. [9] found that the Standardized Mutual Information (SMI) does not

increase monotonically as one clustering is adjusted to match a reference and therefore reject the SMI entirely.

This work presents the p -value of the MI q (denoted PMI q ) as a provably monotonous type II bias correction. We formally define type II bias and prove that the PMI q does not suffer from it. For the SMI q , there is only empirical evidence [29, 30]. We show that the PMI q is monotonous for q ≥ 2 . This includes the p -value of the RI, but for the MI, the p -value is not monotonous. When normalized with the normal CDF, the SMI q approximates the PMI q , which we confirm via Monte Carlo simulation. We reduce the runtime of the SMI 2 from O ( N 3 k A max( k A , k B )) to a much more practical O ( k A k B ) by a reformulation of the variance term. We demonstrate the impact of type II unbiased algorithm selection for community detection on a social network dataset and clustering on images of handwritten digits and human faces.

2 Generalized information theoretic clustering comparison measures

A clustering A of N data points is a partition of the set { 1 , . . . , N } into disjoint subsets A = { A 1 , . . . , A k A } . A i denotes the set of points in the i -th cluster of size a i := | A i | and k A is the number of clusters in A . Clustering comparison measures quantify the similarity between two clusterings A and B , and can be expressed as a function of the contingency table (Table 1).

Table 1: Contingency table for clusterings A,B with k A and k B clusters respectively. Lower case a i and b j denote the cluster size of the i -th and j -th clusters in A and B , while n ij represents the size of their overlap. Clustering comparison measures can be expressed in terms of the elements of this contingency table.

· · ·

·

· · ·

While many clustering comparison methods exist in the literature [2, 9], many well-known methods like the Variation of Information, Mirkin Index, or Rand Index belong to the family of generalized information-theoretic clustering comparison measures [30, 31]. When adjusted for chance, these measures reduce to the mutual information with Tsallis q -entropy [35], which will be the focus of this work.

Definition 2.1 (Tsallis q -entropy) . Let q ∈ R + , and A be a clustering. Then, the Tsallis q -entropy is

̸

with the q -logarithm log q ( y ) := ( y 1 -q -1) / (1 -q ) if q = 1 and the natural logarithm for q = 1 , where x log 1 ( x ) = 0 for x = 0 .

The generalized mutual information is defined in analogy to the mutual information but with Tsallis q -entropy replacing the Shannon entropy.

Definition 2.2 (Generalized mutual information) . Let A,B be two clusterings of the set { 1 , . . . , N } and q ∈ R + , then the generalized mutual information is

Here H q ( A,B ) denotes the joint q -entropy H q ( { A i ∩ B j | A i ∈ A ∧ B j ∈ B } ) .

3 Adjustment for chance

The bare MI q has limited value as a clustering comparison measure. When comparing two clusterings directly with one another, it is biased towards larger or smaller clusters, depending on the value of

q (type I bias) [19, 25]. But even after adjusting for type I bias, there is another, more subtle bias when multiple clusterings are compared via a single ground truth [29] (Figure 1). In Section 3.2, we introduce the p -value as an adjustment to the latter type II bias.

3.1 Type I bias

It is well known throughout the literature that the MI 1 is biased towards smaller clusters in direct clustering comparisons [19, 25]. To make this precise, Gösgens et al. [9] defined a family of clustering distributions for which the expected similarity to a reference clustering should be constant:

Definition 3.1 (Element-symmetric distribution) . A distribution over clusterings B is elementsymmetric if every two clusterings B and B ′ with the same cluster sizes have the same probability.

An example of an element-symmetric distribution is the uniform distribution over all clusterings of N elements into k clusters. If the clustering is random, a comparison measure should not favor one particular k over another. If it does, we call it type I biased [9].

Definition 3.2 (Type I unbiased) . A clustering measure V is type I unbiased if there is a constant c , such that for any clustering A with 1 < k A < N and every element-symmetric distribution B the expected value E B ∼B [ V ( A,B )] = c is constant.

In other words, type I unbiased means that when comparing a fixed clustering A to all permutations of any clustering B , the average metric value is the same for all A . As the MI q has this type I bias, it is commonly adjusted by subtracting its expected value under random permutation, yielding a type I unbiased measure [30].

S N denotes the symmetric group and 1 2 ( H q ( A ) + H q ( B )) is an upper bound to the MI q such that the AMI q is normalized to c = 0 for random clusterings and upper bounded by 1 [25, 30]. Adjustments with respect to other random models are possible [8, 18]. However, the random permutation model remains the most popular and is the focus of this work. Due to the generalization using Tsallis entropy, the AMI 2 corresponds to the Adjusted Rand Index [11, 30].

3.2 Type II bias

However, in a typical external validation scenario, a single absolute value of the AMI q is of little help. While it is easy to understand that an AMI q of zero means a clustering algorithm is no better than random and a value of one means optimal agreement, the scale of the range in between is unclear. Therefore, the AMI q values of multiple candidate solutions with a reference are typically compared against each other to find the best algorithm for a given dataset.

As a toy model for this scenario, we uniformly generate 5000 clusterings of N = 500 elements for each number of clusters k B ∈ { 2 , 6 , 10 , 14 , 18 , 22 } [29]. We compare them to a fixed clustering A with k A = 10 evenly sized clusters and plot the selection probabilities for the RI , MI , and AMI q for q ∈ { 1 , 2 } (Figures 1a, b, d and e). The bare MI and RI and their adjusted variants AMI q are biased towards certain values of k B .

We generalize and formalize this observation by demanding a clustering comparison measure to favor no element-symmetric distribution over another.

Definition 3.3 (Type II unbiased) . Let V be a clustering comparison measure and B , B ′ be elementsymmetric clustering distributions. V is type II unbiased if

for any clustering A with 1 < k A < N , where θ denotes the Heaviside step function with θ (0) = 1 / 2 .

Intuitively, Type I bias means that certain cluster sizes receive higher metric values. Type II bias, on the other hand, gives a higher relative rank to certain cluster sizes when multiple clusterings are compared with a ground truth.

0

.

00

0

.

25

0

.

50

0

.

75

MI

a)

0

.

0

0

.

1

0

.

2

AMI

1

b)

0

.

0

0

.

1

0

.

2

PMI

1

c)

2

6

10

14

18

22

0

.

00

0

.

25

0

.

50

0

.

75

RI

d)

2

6

10

14

18

22

0

.

0

0

.

1

0

.

2

AMI

2

e)

2

6

10

14

18

22

0

.

0

0

.

1

0

.

2

PMI

2

f)

Selection probability

Selected number of clusters

Romano et al. [29] introduced standardization to correct for type II bias

They observed in numerical simulations that in the toy model above (Figure 1), the SMI q selects each k B with approximately equal probability [30].

However, the MI q is not normally distributed under random permutation (Figure 2a), and standardization is only an approximation to the true p -value (Figure 2b). The p -value quantifies what percentage of all permutations of the data would have led to higher mutual information, and we propose to use it for clustering comparison.

Definition 3.4 ( p -value adjusted, generalized mutual information) . Let A,B be two partitions of the set { 1 , . . . , N } , q ∈ R + . Assuming the random permutation model, the p -value adjusted, generalized mutual information is

̸

Note that as the marginal entropies are independent of the permutation, PMI q = E σ ∈ S N [ θ ( H q ( σ ( A ) , B ) -H q ( A,B ))] . For q = 1 , this is the p -value of the mutual information and the variation of information by definition. For q = 1 the PMI q further simplifies to E σ ∈ S N [ θ ( n ′ q ij -n q ij )] for q > 1 and E σ ∈ S N [ θ ( n q ij -n ′ q ij )] for q < 1 with n ′ ij being the elements of the contingency table for σ ( A ) , B . In a sense, the details of the generalized mutual information don't matter under p -value adjustment, except the exponent q of the contingency matrix elements. In fact, the p -value of the Rand Index is equivalent to the PMI 2 by a very similar argument.

In the experiment in Figure 1, we observe that the PMI 1 and PMI 2 select each k B with approximately equal probability.

Proposition 3.1. The PMI q is type I and type II unbiased.

We formally prove this result in Appendix A.

MI

2

probability

PMI

2

a)

MI

2

probability density

Φ(SMI

2

)

b)

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

01

0

.

02

0

.

03

0

.

04

min

PMI

2

,

1

PMI

2

|

PMI

2

-

Φ(SMI

2

)

|

c)

N

50

100

200

500

1000

{

-

}

4 Monotonicity

When one clustering is changed to resemble another more, any clustering similarity measure should increase. A major drawback of the SMI 1 ( A,B ) is its non-monotonous behavior as the number of pairs of elements that agree in A and B increases [9]. We show that the PMI q , on the other hand, is monotonous for q ≥ 2 .

4.1 Definition of monotonicity

The atomic operations that add new pairs of agreeing elements in two clusterings are the perfect split and the perfect merge [9].

Definition 4.1 (Perfect split) . B ′ is a perfect split of B with respect to A if it splits a single cluster B 1 ∈ B into B ′ 1 , B ′ 2 ∈ B ′ such that for all i either A i ∩ B 1 ⊂ B ′ 1 or A i ∩ B 1 ⊂ B ′ 2 .

Definition 4.2 (Perfect merge) . B ′ is a perfect merge of B with respect to A if B ′ is obtained by merging two clusters B 1 , B 2 ∈ B with B 1 , B 2 ⊂ A i for some i .

Gösgens et al. [9] require that any clustering similarity measure increases monotonically for any combination of perfect splits and perfect merges.

Definition 4.3 ( A -consistent improvement) . B ′ is an A -consistent improvement of B iff there exists a series of perfect splits and perfect merges that change B into B ′ .

Definition 4.4 (Monotonicity) . A symmetric clustering comparison measure V is monotonous if for every A,B with 1 < k A < N and any A -consistent improvement B ′ of B , V ( A,B ′ ) > V ( A,B ) .

We show that the PMI q is monotonous for q ≥ 2 , making it the first known clustering comparison measure to be both type II unbiased and monotonous. The case q = 2 is particularly interesting as it corresponds to the well-known Rand Index.

4.2 Proof of monotonicity for PMI q with q ≥ 2

The proof can be broken down into monotonicity under perfect splits and perfect merges. We first show that the joint q -entropy increases under any split that is not perfect.

Lemma 4.1. Let A,B be clusterings with 1 < k A < n and B ′ be obtained by splitting a cluster B j ∈ B into non-empty clusters B ′ j 1 , B ′ j 2 . Then H q ( A,B ′ ) = H q ( A,B ) + ∆ H q with ∆ H q ≥ 0 and equality iff the split is perfect with respect to A .

This statement is a direct consequence of the subadditivity of the q -entropy [7]. In particular h q : p ↦→ p q log q p is a strictly convex function with h q (0) = 0 and hence strictly superadditive, i.e. h q ( p 1 + p 2 ) ≥ h q ( p 1 ) + h q ( p 2 ) for any p 1 , p 2 ≥ 0 with equality iff p 1 = 0 ∨ p 2 = 0 .

Proof of Lemma 4.1. We express ∆ H q as

From n ij = n ′ ij 1 + n ′ ij 2 and the strict superadditivity of h q follows ∆ H q ≥ 0 with equality iff n ′ ij 1 = 0 ∨ n ′ ij 2 = 0 , i.e. when the split is perfect.

Conversely, a perfect merge maximizes the difference in joint entropy.

Lemma 4.2. Let A,B,B ′ and ∆ H q be as in Lemma 4.1, then for q ≥ 2 , ∆ H q is maximal iff B is a perfect merge of B ′ with respect to A .

Proof of Lemma 4.2. When B is a perfect merge of B ′ with respect to A , then

To show that ∆ H perfect q is superadditive as a function of b j , we take its second derivative

For q > 2 , it is strictly convex and thus strictly superadditive. For q = 2 and b j = 0 , the difference ∆ H perfect q = -b 2 j 1 is negative and thus ∆ H perfect 2 is also strictly superadditive. Now consider ˜ A such that B is not a perfect merge with respect to ˜ A . Then at least two ˜ A i 1 , ˜ A i 2 have non-vanishing overlap ˜ n i 1 j , ˜ n i 2 j > 0 with B j such that

With the superadditivity of h q follows H perfect q > H not perfect q (Compare Eq. 7).

Now that we know how the joint entropy behaves under perfect splits and perfect merges, we can put together the proof of the monotonicity of the PMI q .

Theorem 4.3. Let A,B be clusterings with 1 < k A < n and B ′ an A -consistent improvement of B . Then PMI q ( A,B ′ ) > PMI q ( A,B ) for q ≥ 2 .

Proof of Theorem 4.3. It suffices to show monotonicity for B ′ a perfect split or perfect merge since any A -consistent improvement of B can be obtained by a sequence of perfect splits and perfect merges.

Case 1. B ′ is a perfect split.

Since A is not a singleton cluster, a permutation σ exists such that B ′ is not a perfect split with respect to σ ( A ) and with Lemma 4.1 it follows

10

2

10

3

10

4

10

5

number of datapoints

N

10

-

4

10

-

3

10

-

2

10

-

1

10

0

10

1

10

2

runtime [s]

k

A

=

k

B

= 10

a)

SMI

1

SMI

2

PMI

2

O

(

N

3

)

O

(1)

O

(log

N

)

10

-

3

10

-

2

approximation

a

10

-

3

10

-

2

10

-

1

10

0

10

1

runtime [s]

N

= 1000

b)

Φ(SMI

2

)

PMI

2

O

(

a

-

2

)

error

However, B ′ is a perfect split of B with respect to A and equality holds in Lemma 4.1

Case 2. B ′ is a perfect merge.

Let b 1 , b 2 ∈ B denote the merged clusters that form b ′ 1 ∈ B ′ . Using Lemma 4.2, we find as there is at least one permutation σ for which the merge is not perfect.

5 Approximations and runtime

A limitation of the PMI q is its computational complexity. Its exact calculation is intractable even for small datasets, as it requires a sum over all contingency tables with given marginals. To mitigate this limitation, we propose two approximation schemes:

Standardized approximation ( q = 2 ): We approximate the true, discrete distribution of MI q with a continuous normal distribution that matches its first and second statistical moments (See Figure 2a and b). While this approximation is particularly fast for q = 2 , it does not preserve the theoretical guarantees of the PMI q .

Monte Carlo approximation: Given two clusterings A and B , we sample contingency tables with the same cluster sizes. The fraction of tables with MI q lower than MI q ( A,B ) approximates the true p -value. In this approach, the theoretical guarantees hold up to a tunable approximation error at the cost of higher runtime.

5.1 The standardized Rand Index

We approximate the PMI q with the SMI q , normalized with the normal CDF Φ (Figure 2b). This can be seen as a truncated, second-order Gram Charlier A series of the PMI q , and while this could be continued for higher statistical moments, it is difficult to find an exact error term [37]. A more cautious

normalization permits the lower bound PMI q ( A,B ) ≥ 1 -1 1+(SMI q ( A,B ))) 2 for SMI q ( A,B ) > 0 , but has little practical significance in the context of this work [30]. Therefore, we evaluate the approximation quality experimentally 1 on 1000 pairs of clusterings drawn uniformly from the set of all clusterings with N ∈ { 50 , 100 , 200 , 500 , 1000 } using a method described in [16]. We compare Φ(SMI 2 ) := (1 + erf (SMI 2 / √ 2)) / 2 with a Monte Carlo estimate of the PMI 2 with approximation error 0 . 001 in Figure 2c. The values are highly correlated ( r Pearson = 0 . 9983 for N = 50 ), and the approximation improves with larger values of N ( r Pearson = 0 . 9995 for N = 1000 ). So although Φ(SMI q ) itself is not monotonous, it closely matches the PMI q , which is monotonous for q ≥ 2 .

While Φ(SMI q ) is a simplification over the PMI q , its computational complexity O ( N 3 k A max( k A , k B )) for general q is far from practical [15, 30]. In this work, we contribute a novel algorithm for the special case q = 2 that improves the computational complexity.

Proposition 5.1. The computational complexity of SMI 2 is O ( k A k B ) .

The proof is in Appendix C. This special case q = 2 is of particular interest because of the correspondence with the well-known Rand Index and the monotonicity of PMI 2 . Our improved algorithm for the SMI 2 allows comparisons of moderately sized clusterings N ≈ 10 , 000 that are computationally out of reach for, e.g., the SMI 1 (Figure 3a).

5.2 Monte Carlo approximation

The standardized approximation has two limitations:

It is computationally inefficient only for q = 2 .

There is no guarantee that it preserves the desirable theoretical properties of the PMI 2 .

̸

We address both of these limitations by introducing a Monte Carlo approximation at the cost of increased runtime. For two clusterings A,B , the method samples contingency tables uniformly from all tables with their respective cluster sizes a 1 , . . . , a k A ; b 1 , . . . , b k B (Compare Table 1), using the algorithms proposed in [4, 26]. The fraction of samples with MI q lower than MI q ( A,B ) is an unbiased estimator of the PMI q . The sampling procedure terminates when a given approximation error a is reached. This way, the theoretical properties of the PMI q are preserved up to the tunable approximation error.

However, lower approximation errors require more samples:

Proposition 5.2. The computational complexity of the Monte Carlo PMI q is O (min( N,k A k B log N ) /a 2 ) , with the desired approximation error a .

̸

The proof is in Appendix B, and Figure 3 shows an experimental study of the runtime compared to the standardized approximation. The Monte Carlo approach is computationally more expensive, especially for larger datasets. Therefore, the standardized approach is for choice when q = 2 and moderate approximation quality is acceptable. The Monte Carlo method should be used if q = 2 or theoretical guarantees are required.

6 Algorithm selection on real-world datasets

6.1 k -means clustering on image datasets

As a first example, we mimic the synthetic experiment in Figure 1. For several numbers of clusters k , we apply k -means clustering [21] with 1000 different random seeds. We select the clustering with the highest RI , AMI 2 , and PMI 2 , approximated by Φ(SMI 2 ) and denote the corresponding number of clusters k selected. Figure 4a shows the selection probabilities of k selected when compared with a ground truth with k true clusters for a handwritten digit dataset [6] and Figure 4b for a dataset of human faces [27]. Naturally, all measures favor solutions where k selected > k true, as a higher number of clusters increases the chances of k -means matching the reference decision boundaries. However, the RI and AMI 2 additionally suffer from type II bias, leading to a higher overestimation of k selected

25

50

75

Leiden

175

200

k

selected

selection probability

c)

40

44

48

52

56

60

k

selected

b)

E

[

k

selected

]

RI

AMI

2

PMI

2

10 11 12 13 14 15

k

selected

0

.

0

0

.

2

0

.

4

0

.

6

selection probability

a)

P

(

k

selected

)

RI

AMI

2

PMI

2

(compare with Figure 1). The difference between the AMI 2 and the PMI 2 is subtle, but this is expected. Type II bias correction is just one step forward in clustering comparison and does not turn existing assessments based on metrics like the AMI 2 on its head. In practice, much wider ranges of k selected can arise from different clustering algorithms which could potentially amplify the effect. Two additional experiments with spectral clustering instead of k -means can be found in Appendix E.

6.2 Community detection in social networks

In social networks, detecting communities is ubiquitous and can help to detect fraud, deliver personalized content, or target ads [13]. As a result, many community detection algorithms are known in the literature [3, 28, 34]. However, community detection is inherently unsupervised, and it is a priori unclear which algorithm with which parameters will perform best for a given application. In practice, human experts often annotate a subset of the dataset, and an unsupervised algorithm is selected via a clustering comparison measure on that subset.

We simulate this procedure on a network of email conversations between European research institutions, where each institution is a ground truth cluster [20]. We select a connected subset with k true = 30 institutions and detect communities using Degree Ordered Label Propagation, Label Propagation, Leiden, Louvain, and Louvain Map Equation [32] with 22 parameter configurations (Appendix D). We then select the most similar algorithm to the ground truth using RI , AMI 2 , and PMI 2 ≈ Φ(SMI 2 ) . This process is repeated for 100 subsets per dataset, and the resulting probabilities are shown in Figure 4c. Label propagation is a fast but inaccurate method [28] and overestimates k true by almost an order of magnitude in our experiment. During algorithm selection, RI was the only metric to choose Label Propagation in some cases. The PMI 2 differs from the AMI 2 in that it selected Leiden more frequently over the Louvain Map Equation, both of which are improvements over the original Louvain method [14, 34]. However, Leiden comes closer to the true number of clusters k true, and in that sense, PMI 2 led to a better choice of algorithm.

NMI

NMI

Fair NMI

VI

FMeasure

BCubed

Dice

Sokal&Sneath

Rand Index

AMI

1

AMI

2

SMI

1

SMI

2

PMI

1

PMI

2

Type I unbiased

✗

✗

✗

✗

✗

✗

✗

✗

✗

✓

✓

✗

✗

✓

✓

✓

✓

✓

✓

Type II unbiased

✗

✗

✗

✗

✗

✗

✗

✗

✗

✗

✗

✗

✗

✗

✗

/circle_blank

/circle_blank

✓

✓

Monotonicity

✓

✗

✗

✓

✗

✓

✓

✗

✓

✓

✓

✓

✓

✓

✓

✗

✗

✗

✓

Comp. tractable

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✗

✓

/circle_blank

/circle_blank

7 Conclusion and outlook

Table 2 summarizes our findings for the PMI q and compares its theoretical properties to 17 clustering comparison measures from the systematic review by Gösgens et al. [9]. We introduce the first type II unbiased and monotonous cluster comparison method, the p -value adjusted Rand Index ( PMI 2 ). Existing methods that addressed type II bias, namely the Standardized Mutual Information ( SMI 1 ) and the Standardized Rand Index ( SMI 2 ) are not monotonous, meaning clusterings closer to the ground truth can score worse. In addition, the SMI 1 has high computational complexity, making it unsuitable in practice. For the SMI 2 we showed that an efficient algorithm exists and we leverage this algorithm for an efficient approximation of the proposed PMI 2 . However, our analysis of the errors in this standardized approximation is limited to experimental observations, leaving a theoretical analysis for future work. We devised a Monte Carlo approximation for the PMI 2 with tunable approximation error, for when theoretical guarantees are required. To validate our theoretical findings, synthetic experiments confirm that the presented PMI 2 selects different cluster sizes with equal probability and is not subject to type II bias. In practice, the PMI 2 chooses better clustering algorithms from a set of candidates when a ground truth reference is available. Thanks to its monotonicity and computational efficiency, the PMI 2 is a practical candidate for evaluating cluster similarity without type II bias. While we investigated p -value adjustment for the family of generalized information-theoretic clustering comparison measures, further research is required to understand if other comparison measures, like the Jaccard Index, could benefit from a similar adjustment.

Acknowledgments and Disclosure of Funding

This work was funded by the Digital Europe Grant Testing and Experimentation Facility for Health AI and Robotics (TEF-Health) , Project number 101100700 and the Bayerischen Verbundförderprogramm (BayVFP) - Förderlinie Digitalisierung - Förderbereich Informations- und Kommunikationstechnik of the Bavarian Ministry of Economic Affairs, Regional Development and Energy and supported by Bayern Innovativ - Bayerische Gesellschaft für Innovation und Wissenstransfer mbH .

References

C. C. Aggarwal and C. K. Reddy, editors. Data Clustering: Algorithms and Applications . CRC Press, 2014. ISBN 978-1-4665-5821-2.

N. Arinik, V. Labatut, and R. Figueiredo. Characterizing and comparing external measures for the assessment of cluster analysis and community detection. IEEE Access , 9:20255-20276, 2021. doi: 10.1109/ACCESS.2021.3054621.

V. D. Blondel, J.-L. Guillaume, R. Lambiotte, and E. Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment , 2008(10):P10008, 2008.

J. M. Boyett. Algorithm AS 144: Random R × C tables with given row and column totals. Journal of the Royal Statistical Society. Series C (Applied Statistics) , 28(3):329-332, 1979. ISSN 00359254, 14679876.

C. Carpineto and G. Romano. Consensus clustering based on a new probabilistic rand index with application to subtopic retrieval. IEEE Trans. Pattern Anal. Mach. Intell. , 34(12):2315-2326, 2012. doi: 10.1109/TPAMI.2012.80.

D. Dua and C. Graff. UCI machine learning repository, 2019.

S. Furuichi. Information theoretical properties of Tsallis entropies. Journal of Mathematical Physics , 47(2):023302, Feb. 2006. ISSN 0022-2488. doi: 10.1063/1.2165744.

A. J. Gates and Y.-Y. Ahn. The impact of random models on clustering similarity. Journal of Machine Learning Research , 18:87:1-87:28, 2017.

M. Gösgens, A. Tikhonov, and L. Prokhorenkova. Systematic analysis of cluster similarity indices: How to validate validation measures. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 3799-3808. PMLR, 2021.

D. He, L. Zhai, Z. Li, D. Jin, L. Yang, Y . Huang, and P. S. Yu. Adversarial mutual information learning for network embedding. In C. Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020 , pages 3321-3327. ijcai.org, 2020. doi: 10.24963/ijcai.2020/459.

L. Hubert and P. Arabie. Comparing partitions. Journal of Classification , 2(1):193-218, Dec. 1985. ISSN 1432-1343. doi: 10.1007/BF01908075.

D. Jin, Z. Yu, P. Jiao, S. Pan, D. He, J. Wu, P. S. Yu, and W. Zhang. A survey of community detection approaches: From statistical modeling to deep learning. IEEE Trans. Knowl. Data Eng. , 35(2):1149-1170, 2023. doi: 10.1109/TKDE.2021.3104155.

A. Karata¸ s and S. ¸ Sahin. Application areas of community detection: A review. In 2018 International Congress on Big Data, Deep Learning and Fighting Cyber Terrorism (IBIGDELFT) , pages 65-70, 2018. doi: 10.1109/IBIGDELFT.2018.8625349.

Y. Kim and H. Jeong. Map equation for link communities. Physical Review E , 84(2):026110, 2011.

K. Klede, L. Schwinn, D. Zanca, and B. M. Eskofier. Fastami - a monte carlo approach to the adjustment for chance in clustering comparison metrics. In B. Williams, Y. Chen, and J. Neville, editors, Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, ThirtyFifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023 , pages 8317-8324. AAAI Press, 2023. doi: 10.1609/aaai.v37i7. 26003.

D. Knuth. The complexity of nonuniform random number generation. Algorithms and Complexity, New Directions and Results , pages 357-428, 1976.

A. Lancichinetti and S. Fortunato. Consensus clustering in complex networks. Scientific Reports , 2(1):336, Mar. 2012. ISSN 2045-2322. doi: 10.1038/srep00336.

D. Lazarenko and T. Bonald. Pairwise adjusted mutual information. CoRR , abs/2103.12641, 2021.

Y. Lei, J. C. Bezdek, S. Romano, X. V. Nguyen, J. Chan, and J. Bailey. Ground truth bias in external cluster validity indices. Pattern Recognition , 65:58-70, 2017. doi: 10.1016/j.patcog. 2016.12.003.

J. Leskovec, J. M. Kleinberg, and C. Faloutsos. Graph evolution: Densification and shrinking diameters. ACM Trans. Knowl. Discov. Data , 1(1):2, 2007. doi: 10.1145/1217299.1217301.

J. MacQueen. Classification and analysis of multivariate observations. In 5th Berkeley Symp. Math. Statist. Probability , pages 281-297. University of California Los Angeles LA USA, 1967.

T. Mansour, M. Schork, and M. Shattuck. The generalized Stirling and Bell numbers revisited. J. Integer Seq , 15(8):47, 2012.

E. Müller, S. Günnemann, I. Färber, and T. Seidl. Discovering multiple clustering solutions: Grouping objects in different views of the data. In G. I. Webb, B. Liu, C. Zhang, D. Gunopulos, and X. Wu, editors, ICDM 2010, the 10th IEEE International Conference on Data Mining, Sydney, Australia, 14-17 December 2010 , page 1220. IEEE Computer Society, 2010. doi: 10.1109/ICDM.2010.85.

X. V. Nguyen, J. Epps, and J. Bailey. Information theoretic measures for clusterings comparison: Is a correction for chance necessary? In A. P. Danyluk, L. Bottou, and M. L. Littman, editors, Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, Montreal, Quebec, Canada, June 14-18, 2009 , volume 382 of ACM International Conference Proceeding Series , pages 1073-1080. ACM, 2009. doi: 10.1145/1553374.1553511.

X. V. Nguyen, J. Epps, and J. Bailey. Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance. Journal of Machine Learning Research , 11:2837-2854, 2010.

W. M. Patefield. Algorithm AS 159: An efficient method of generating random R × C tables with given row and column totals. Journal of the Royal Statistical Society. Series C (Applied Statistics) , 30(1):91-97, 1981. ISSN 00359254, 14679876.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

U. N. Raghavan, R. Albert, and S. Kumara. Near linear time algorithm to detect community structures in large-scale networks. Physical review E , 76(3):036106, 2007.

S. Romano, J. Bailey, X. V. Nguyen, and K. Verspoor. Standardized mutual information for clustering comparisons: One step further in adjustment for chance. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014 , volume 32 of JMLR Workshop and Conference Proceedings , pages 1143-1151, 2014.

S. Romano, X. V. Nguyen, J. Bailey, and K. Verspoor. Adjusting for chance clustering comparison measures. Journal of Machine Learning Research , 17:134:1-134:32, 2016.

D. A. Simovici. On generalized entropy and entropic metrics. J. Multiple Valued Log. Soft Comput. , 13(4-6):295-320, 2007.

C. L. Staudt, A. Sazonovs, and H. Meyerhenke. NetworKit: A tool suite for large-scale complex network analysis. Network Science , 4(4):508-530, 2016. doi: 10.1017/nws.2016.20.

X. Su, S. Xue, F. Liu, J. Wu, J. Yang, C. Zhou, W. Hu, C. Paris, S. Nepal, D. Jin, Q. Z. Sheng, and P. S. Yu. A comprehensive survey on community detection with deep learning. CoRR , abs/2105.12584, 2021.

V. A. Traag, L. Waltman, and N. J. Van Eck. From Louvain to Leiden: Guaranteeing wellconnected communities. Scientific reports , 9(1):5233, 2019.

C. Tsallis. Possible generalization of Boltzmann-Gibbs statistics. Journal of Statistical Physics , 52(1):479-487, July 1988. ISSN 1572-9613. doi: 10.1007/BF01016429.

J. Vanschoren, J. N. van Rijn, B. Bischl, and L. Torgo. Openml: networked science in machine learning. SIGKDD Explorations , 15(2):49-60, 2013. doi: 10.1145/2641190.2641198.

D. L. Wallace. Asymptotic Approximations to Distributions. The Annals of Mathematical Statistics , 29(3):635-654, Sept. 1958. ISSN 0003-4851, 2168-8990. doi: 10.1214/aoms/ 1177706528.

S. Wei, G. Han, R. Wang, Y. Yang, H. Zhang, and S. Li. Inductive multi-view multiple clusterings. In 2021 7th International Conference on Big Data and Information Analytics (BigDIA) , pages 308-315, 2021. doi: 10.1109/BigDIA53151.2021.9619704.

William M. Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association , 66(336):846-850, 1971. doi: 10.1080/01621459.1971. 10482356.

Appendix

A Type II Bias

We show that the PMI q is both type I and type II unbiased:

Proof of Proposition 3.1. The choice of θ (0) = allows us to use θ ( x ) = 1 θ ( x ) such that

with ˜ B = σ -1 ( B ) and ˜ σ = σ -1 . Now in the definition of type I unbiasedness, B is elementsymmetric, such that ˜ B ∈ B and

Hence, PMI q is type I unbiased.

For type II unbiasedness, B , B ′ are element symmetric and with Eq. 14 we get

For all other clustering comparison measures in Table 2, the proof or a counterexample to type I unbiasedness are given in [9]. Here, we provide an example for their type II bias. For N = 4 data points, we compare the clustering A = {{ 1 , 4 } , { 2 , 3 }} with all clusterings with 2 clusters, distributed uniformly B . We also compare A to all clusterings from the uniform distribution B ′ of N = 4 data points into 3 clusters. All comparison measures except the PMI 2 prefer either 2 or 3 clusters in the sense that their expectation value in Eq. 4 is not equal to 1 / 2 , and are thus type II biased (See Table 3).

̸

Table 3: Example for type II bias of other clustering comparison measures. For A = {{ 1 , 4 } , { 2 , 3 }} and B , B ′ the uniform distribution of all clusterings with 4 elements into 2 , 3 clusters respectively, the expected value E B ∼B ,B ′ ∼B ′ [ θ ( V ( A,B ) -V ( A,B ′ ))] = 1 2 and hence these measures are type II biased.

NMI

NMI

Fair NMI

VI

FMeasure

BCubed

Dice

Sokal&Sneath

Rand Index

AMI

1

AMI

2

SMI

1

SMI

2

1

7

1

7

11

21

10

21

11

21

13

21

13

21

5

7

13

21

11

21

11

21

10

21

1

3

11

21

11

21

25

42

25

42

B Monte Carlo PMI q

Proof of proposition 5.2. Random contingency tables with fixed marginals can be generated in O (min( N,k A k B log N )) [4, 26]. The error a = √ PMI q (1 -PMI q ) /N samples decreases with the inverse square root of the number of Monte Carlo samples N samples. Hence we need N samples ≤ a -2 / 4 samples to reach the approximation error a .

C The standardized Rand Index

For the proof of the runtime complexity of the SMI 2 , we use the fact that it is equivalent to the SRI (Corollary 2 in [30])

Proof of Proposition 5.1. Hubert and Arabie [11] derived the expected value under random permutation as

For the variance, we also need the second moment. While Romano et al. [30] give a general formula in Eq. (13), it is impractical in the given form as the authors themselves note the runtime complexity is O ( N 3 k A max( k A , k B )) . We observe that in the special case of the SRI, the higher moments can be simplified by leveraging the identity [39]

with the generalized Stirling Number S 2 , 2 ( m,k ) [22] and the hypergeometric distribution Hyp . Note that the right-hand side is completely independent of n ij . Hence the expected values under the hypergeometric distributions in Eq. (13) in [30] can be calculated in O (max { k A , k B } ) time, giving

for the most general case N ≥ 4 and no cluster larger than N -2 , with

The other cases can be treated analogously using Eq. (19). Hence the dominant factor for the runtime complexity of the SRI is the RI itself with O ( k A k B ) for the sum over all contingency matrix elements.

D Community detection parameter configurations

For the experiment in Figure 4c, we compared five community detection algorithms implemented in networkit [32]. Label Propagation and Degree Ordered Label Propagation do not have any parameters. The parameter choices for the other algorithms is listed in Table 4.

∈ {

E Spectral clustering on image datasets

We conducted two additional experiments similar to the ones in Section 6.1, but using spectral clustering instead of k -means. We apply spectral clustering with varying number of clusters k to the UCI image segmentation dataset [6] and a texture classification dataset from OpenML [36]. We compare each clustering solution with the ground truth labels and select the clustering with the highest RI , AMI 2 and PMI 2 . Figure 5 shows the selection frequency of each number of clusters k selected after 1000 trials with different random seeds. In the case of the image segmentation dataset, different subsets of 1000 samples were chosen for each trial, due to the steep runtime requirements of spectral clustering. The results align with the k -means experiment in Section 6.1: The PMI 2 selects clusterings in a less biased way, in the sense that the selected number of clusters k selected is closer to the true number of clusters k true than for RI and AMI 2 .

4

5

6

7

8

9

10

k

selected

0

.

0

0

.

2

0

.

4

0

.

6

a)

E

[

k

selected

]

P

(

k

selected

)

RI

AMI

2

PMI

2

k

true

RI

AMI

2

PMI

2

7

8

9

10 11 12 13 14 15 16

k

selected

0

.

0

0

.

1

0

.

2

0

.

3

b)

}

Peer reviews:
comments: SUMMARY:
p-value Adjustment. The p-value adjusted Rand Index is unbiased. The authors claim that its approximations outperform STD Mutual Information. 
First, generalized MI relies on the TSALLIS entropy (same family of RENYI). AMIq comes from subtracting the expectation under random permutations. PMIq is derived and it is monotonic for q>=2 where q=2 is the Rand Index. Monotonicity is due to the properties of the bypass TSALLIS entropy.

STRENGTHS:
This is a general tool for testing clustering algorithms. The properties of bypass entropies such as Tsallis are leveraged. 
Computational complexity is quadratic.

WEAKNESSES:
It is quite theoretical and general. Poor experiments. Seems more a statistical paper. Please, motivate the use of bypass entropy estimators. It is more than a matter of efficiency. Other formal properties are also key (e.g. monotonicity).

QUESTIONS:
How critical is the choice of the Tsallis entropy? What about the  Kozackenko-Leonenko/Kraskow et al.’s approach? In ML there is an intererest in Rényi entropies and KNN graphs. Can you say anything about the properties of the graph associated with the clustering table.

LIMITATIONS:
The paper is general but few baselines are explored. Should be nice to test with segmented images or outputs of spectral clustering where there is an implicit bias. In other words, in my opinion the paper needs to tackle more recent datasets used in deep learning and image analysis to approach this result to the NEURIPS community.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article introduces a new measurement called $\\text{PMI}_q$ for comparing clustering methods. The name \"p-value adjustment\" comes from $\\text{PMI}_1$, which represents the p-value of the mutual information's variation. This new metric has desirable properties, including type II unbiasedness and monotonicity. The paper claims that $\\text{PMI}_q, q\\ge 2$ is the first clustering comparison method to satisfy both of these properties.\n\nThe definition of $\\text{PMI}_q$ involves taking an expectation over all permutations in $S_N$, which can be computationally challenging. To address this challenge, the author proposes a Monte Carlo estimation method for $\\text{PMI}_2$ and applies it to empirical experiments.\n\nThe paper's contribution lies in proposing a new unbiased and provably monotonic method for addressing the existing bias problem in current clustering comparison metrics. The author provides detailed and sufficient proofs to support the proposed method.\n\nThe paper's limitations include its disorganized structure and lack of clarity in some sections. The article's figures and tables need improvement, and the author does not explicitly address any limitations or potential future research directions.\n\nSTRENGTHS:\n1. This paper provides valuable ideas and techniques for improving cluster comparison, particularly a new unbiased and provably monotonic method for addressing the existing bias problem, which may be useful to practitioners and researchers in the field of cluster comparison.\n\n2.Detailed and sufficient proofs give relatively clear theoretical and technical support on new methods.\n\nWEAKNESSES:\n1. The article's structure is somewhat disorganized and can be challenging to read. To improve clarity, the authors could provide more examples and intuitive explanations for Definitions 3.1 to 3.4 and 4.1 to 4.4.\n\n2. It would be beneficial to compare the proposed clustering comparison metric with other commonly used metrics in the field. By doing so, the authors could inform readers about whether these metrics possess desirable properties such as type II unbiasedness and monotonicity.\n\n3. The figures and tables in the article need improvement, as some elements are unclear or difficult to read. For example, Figure 3a has too a large legend. \n\n4. The results in Figure 2(c) could be more effectively presented to allow for easier differentiation across different N values. The authors may want to explore alternative visual representations or labeling techniques to better convey this information.\n\nQUESTIONS:\n1. Is $\\text{PMI}_q$ type I unbiased? \n\n2. Section 6 is unclear on how to calculate $k_{pred}$. Although $\\text{PMI}$ can compare two clusterings, it is unclear which clusterings are being compared in the experiments discussed in Section 6. As such, further clarification is needed to understand how $k_{pred}$ is computed in this section.\n\nLIMITATIONS:\n1. The paper does not explicitly address any limitations or potential avenues for future research.\n\n2. While the author asserts that $\\text{PMI}_2$ is the first clustering comparison method to satisfy both type II unbiasedness and monotonicity, it is challenging to evaluate the significance of this claim without further comparison.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper presents a performance measurement method for cluster analysis. The method can avoid Type II bias that exists in previous approaches. A tractable approximation is given. The proposed method demonstrates advantages in both synthetic and real-world data sets.

STRENGTHS:
The work consists of solid theoretical contributions and satisfactory empirical results. The finding is an important step forward in clustering research.

WEAKNESSES:
More empirical results would make the work more convincing.

QUESTIONS:
* It is better to elaborate the importance of Type II unbiasedness. Specifically, explaining Definition 3.3 in common words help readers understand the concepts and your contribution.

* There could be a mistake in Proposition 5.2. In the current formula, a higher accuracy leads to lower complexity, while a zero accuracy gives infinity.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new method called the p-value adjusted Rand Index (PMI2) for comparing clustering and community detection algorithms. The paper highlights the limitations of existing metrics, such as the Rand Index and Adjusted Rand Index, which suffer from bias and non-monotonicity issues. The PMI2 method addresses these issues by providing a type II unbiased and provably monotonic metric that has fast approximations. The paper also provides experimental results on image and social network datasets to demonstrate the effectiveness of the PMI2 method.\n\nSTRENGTHS:\nOriginality: The paper introduces a new clustering comparison metric, the p-value adjusted Rand Index (PMI2), which addresses the limitations of existing metrics. The PMI2 method is the first to be type II unbiased and provably monotonic, and it has fast approximations.\n\nQuality: The paper provides a thorough analysis of the limitations of existing clustering comparison metrics and demonstrates the effectiveness of the PMI2 method through experiments on synthetic benchmarks, image, and social network datasets. The paper also provides theoretical proofs of the PMI2 method's properties.\n\nClarity: The paper is well-organized and clearly presents the motivation, background, methodology, and experimental results of the PMI2 method. The paper also provides detailed explanations of the theoretical proofs and the approximations used in the method.\n\nSignificance: The paper's contributions are significant as they provide a more reliable and accurate way to evaluate clustering and community detection algorithms. The PMI2 method's properties make it a valuable tool for practitioners to choose better algorithms for their datasets. The paper's theoretical proofs and experimental results also provide a deeper understanding of clustering comparison metrics and their limitations.\n\nWEAKNESSES:\nOne potential weakness of the paper is that it does not compare the PMI2 method with other recently proposed clustering comparison metrics. While the paper provides a thorough analysis of the limitations of existing metrics, it would be valuable to compare the PMI2 method with other state-of-the-art methods to demonstrate its superiority. \n\nAnother weakness is that the paper does not provide a detailed explanation of the approximations used in the PMI2 method. While the paper mentions that the method has fast approximations, it would be helpful to provide more information on how these approximations work and how they affect the accuracy of the method.\n\nQUESTIONS:\nA more extensive comparison of the PMI2 method with other recently proposed clustering comparison metrics to demonstrate its superiority would help to further validate the effectiveness of the PMI2 method.\n\nA more detailed and intuitive explanation of the approximations used in the PMI2 method and how they affect its accuracy would help readers to better understand the method and its limitations.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an improved clustering comparison metric. While many current metrics fix a \"type I\" bias (being biased towards certain cluster size distributions), many still suffer from a \"type II\" bias (a bias towards certain clusterings when they are compared to a ground truth clustering). Building on top of the Adjusted Rand Index, the type II bias is analyzed and a fast  and a p-value adjusted variant is introduced which is type II unbiased. A previous approach also solves this issue (SMI), however, it suffers from being non-monotonous, while the proposed metric is monotonous, i.e. when modifying a clustering in such a way that it objectively becomes better, the metric reflects this. While the proposed PMI metric relies on Monte Carlo approximation and thus is fairly costly, the SMI_2 metric is fast to compute and with a small modification seems to be a good approximation to the PMI_2. Results on three different datasets show that the best selected clusterings according to the PMI metric seem more reasonable when compared to two baselines.\n\nSTRENGTHS:\n- The paper is fairly easy to follow and seems to be well written. Even though I am not an expert in this field, I appreciate that the paper guided me through the most important considerations and different steps to get to the proposed metric.\n- Monotonicity is a reasonable aspect to consider from a clustering comparison metric, as also highlighted by Gösgens et al. [9]. Properly defining this concept and showing that the proposed PMI_q metric is monotonous for q>=2 is a valuable contribution.\n- The fast approximation of PMI_2 makes the proposed metric more practically applicable.\n- Code is provided.\n\nWEAKNESSES:\nMy main concern is the overall practical significance of the proposed metric. While SMI is not monotonous according to Gösgens et al. [9], there seems to be empirical evidence for SMI_q. A proper proof for PMI_q is very nice, but the real-world experiments are based on the SMI_2 based approximation of PMI_2. In my mind the obvious baseline to compare to, would be SMI_2, but this is omitted for some reason and I really wonder why. Even though PMI_2 seems to select the more reasonable clusterings in all experiments, the margin to AMI_2 is not very large in two of the three setups and I would not be surprised if it is even smaller for the SMI_2 score. Adding such a comparison would give a more complete picture of how the different metrics compare.\n\nQUESTIONS:\nGiven that you only theoretically prove that PMI_2 is type II unbiased, but you then in practice approximate it with SMI_2, is there some guarantee that this property still holds?\n\nLIMITATIONS:\nLimitations or potential negative impact have not been discussed in the paper. I would not be able to come up with a direct negative societal impact myself, but a discussion of potential limitations of the approach (if any are clear) would have been valuable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}