SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We approximate the  $\mathrm{PMI}_q$  with the  $\mathrm{SMI}_q$ , normalized with the normal CDF  $\Phi$  (Figure 2b). This can be seen as a truncated, second-order Gram Charlier A series of the  $\mathrm{PMI}_q$ , and while this could be continued for higher statistical moments, it is difficult to find an exact error term [37]. A more cautious
normalization permits the lower bound  $\mathrm{PMI}_q(A,B)\geq 1 - \frac{1}{1 + (\mathrm{SMI}_q(A,B))^2}$  for  $\mathrm{SMI}_q(A,B) > 0$  but has little practical significance in the context of this work [30]. Therefore, we evaluate the approximation quality experimentally on 1000 pairs of clusterings drawn uniformly from the set of all clusterings with  $N\in \{50,100,200,500,1000\}$  using a method described in [16]. We compare  $\Phi (\mathrm{SMI}_2)\coloneqq (1 + \mathrm{erf}(\mathrm{SMI}_2 / \sqrt{2})) / 2$  with a Monte Carlo estimate of the  $\mathrm{PMI}_2$  with approximation error 0.001 in Figure 2c. The values are highly correlated  $(r_{\mathrm{Pearson}} = 0.9983$  for  $N = 50$ ), and the approximation improves with larger values of  $N$ $(r_{\mathrm{Pearson}} = 0.9995$  for  $N = 1000$ ). So although  $\Phi (\mathrm{SMI}_q)$  itself is not monotonous, it closely matches the  $\mathrm{PMI}_q$ , which is monotonous for  $q\geq 2$ .
While  $\Phi (\mathrm{SMI}_q)$  is a simplification over the  $\mathrm{PMI}_q$ , its computational complexity  $\mathcal{O}(N^3 k_A\max (k_A,k_B))$  for general  $q$  is far from practical [15, 30]. In this work, we contribute a novel algorithm for the special case  $q = 2$  that improves the computational complexity.
Proposition 5.1. The computational complexity of  $\mathrm{SMI}_2$  is  $\mathcal{O}(k_A k_B)$ .
The proof is in Appendix C. This special case  $q = 2$  is of particular interest because of the correspondence with the well-known Rand Index and the monotonicity of  $\mathrm{PMI}_2$ . Our improved algorithm for the  $\mathrm{SMI}_2$  allows comparisons of moderately sized clusterings  $N \approx 10,000$  that are computationally out of reach for, e.g., the  $\mathrm{SMI}_1$  (Figure 3a).
5.2 Monte Carlo approximation
The standardized approximation has two limitations:
- It is computationally inefficient only for  $q \neq 2$ .
- There is no guarantee that it preserves the desirable theoretical properties of the  $\mathrm{PMI}_2$ .
We address both of these limitations by introducing a Monte Carlo approximation at the cost of increased runtime. For two clusterings  $A, B$ , the method samples contingency tables uniformly from all tables with their respective cluster sizes  $a_1, \ldots, a_{k_A}$ ;  $b_1, \ldots, b_{k_B}$  (Compare Table 1), using the algorithms proposed in [4, 26]. The fraction of samples with  $\mathrm{MI}_q$  lower than  $\mathrm{MI}_q(A, B)$  is an unbiased estimator of the  $\mathrm{PMI}_q$ . The sampling procedure terminates when a given approximation error  $a$  is reached. This way, the theoretical properties of the  $\mathrm{PMI}_q$  are preserved up to the tunable approximation error.
However, lower approximation errors require more samples:
Proposition 5.2. The computational complexity of the Monte Carlo  $\mathrm{PMI}_q$  is  $\mathcal{O}(\min (N,k_Ak_B\log N) / a^2)$ , with the desired approximation error  $a$ .
The proof is in Appendix B, and Figure 3 shows an experimental study of the runtime compared to the standardized approximation. The Monte Carlo approach is computationally more expensive, especially for larger datasets. Therefore, the standardized approach is for choice when  $q = 2$  and moderate approximation quality is acceptable. The Monte Carlo method should be used if  $q \neq 2$  or theoretical guarantees are required.
6 Algorithm selection on real-world datasets
6.1  $k$ -means clustering on image datasets
As a first example, we mimic the synthetic experiment in Figure 1. For several numbers of clusters  $k$ , we apply  $k$ -means clustering [21] with 1000 different random seeds. We select the clustering with the highest RI,  $\mathrm{AMI}_2$ , and  $\mathrm{PMI}_2$ , approximated by  $\Phi(\mathrm{SMI}_2)$  and denote the corresponding number of clusters  $k_{\text{selected}}$ . Figure 4a shows the selection probabilities of  $k_{\text{selected}}$  when compared with a ground truth with  $k_{\text{true}}$  clusters for a handwritten digit dataset [6] and Figure 4b for a dataset of human faces [27]. Naturally, all measures favor solutions where  $k_{\text{selected}} > k_{\text{true}}$ , as a higher number of clusters increases the chances of  $k$ -means matching the reference decision boundaries. However, the RI and  $\mathrm{AMI}_2$  additionally suffer from type II bias, leading to a higher overestimation of  $k_{\text{selected}}$
<sup>1</sup>The experiments were executed on an AMD Ryzen 9 5950X. The code for all experiments is available at https://github.com/mad-lab-fau/pmi-experiments.
Figure 4: We apply  $k$ -means clustering [21] with varying  $k$  to a) the UCI handwritten digit dataset [6] and b) the Olivetti faces dataset [27] and select the solution with the highest similarity to the ground truth. We repeat this experiment 1000 times with different random seeds and plot the selection probability under the RI,  $\mathrm{AMI}_2$ , and normal approximation of the  $\mathrm{PMI}_2$ . The  $\mathrm{PMI}_2$  selects candidates where  $k_{\text{selected}}$  is closer to the true number of clusters  $k_{\text{true}}$  on average (dashed lines) compared to the RI and  $\mathrm{AMI}_2$ . In c), we select a connected subset of  $k_{\text{true}} = 30$  communities from the email EU core dataset [20] and detect communities using five algorithms with 22 parameter configurations. The RI,  $\mathrm{AMI}_2$ , and  $\mathrm{PMI}_2$  select the best solution, and we plot the selection probability for  $k_{\text{selected}}$  after 100 repetitions. The  $\mathrm{PMI}_2$  prefers the Leiden algorithm, which produces  $k_{\text{selected}}$  on the order of  $k_{\text{true}}$ . The  $\mathrm{AMI}_2$  gives a higher probability for Louvain Map Equation, and the RI sometimes selects low-quality Label Propagation results.
(compare with Figure 1). The difference between the  $\mathrm{AMI}_2$  and the  $\mathrm{PMI}_2$  is subtle, but this is expected. Type II bias correction is just one step forward in clustering comparison and does not turn existing assessments based on metrics like the  $\mathrm{AMI}_2$  on its head. In practice, much wider ranges of  $k_{\mathrm{selected}}$  can arise from different clustering algorithms which could potentially amplify the effect. Two additional experiments with spectral clustering instead of  $k$ -means can be found in Appendix E.
6.2 Community detection in social networks
In social networks, detecting communities is ubiquitous and can help to detect fraud, deliver personalized content, or target ads [13]. As a result, many community detection algorithms are known in the literature [3, 28, 34]. However, community detection is inherently unsupervised, and it is a priori unclear which algorithm with which parameters will perform best for a given application. In practice, human experts often annotate a subset of the dataset, and an unsupervised algorithm is selected via a clustering comparison measure on that subset.
We simulate this procedure on a network of email conversations between European research institutions, where each institution is a ground truth cluster [20]. We select a connected subset with  $k_{\mathrm{true}} = 30$  institutions and detect communities using Degree Ordered Label Propagation, Label Propagation, Leiden, Louvain, and Louvain Map Equation [32] with 22 parameter configurations (Appendix D). We then select the most similar algorithm to the ground truth using RI,  $\mathrm{AMI}_2$ , and  $\mathrm{PMI}_2 \approx \Phi(\mathrm{SMI}_2)$ . This process is repeated for 100 subsets per dataset, and the resulting probabilities are shown in Figure 4c. Label propagation is a fast but inaccurate method [28] and overestimates  $k_{\mathrm{true}}$  by almost an order of magnitude in our experiment. During algorithm selection, RI was the only metric to choose Label Propagation in some cases. The  $\mathrm{PMI}_2$  differs from the  $\mathrm{AMI}_2$  in that it selected Leiden more frequently over the Louvain Map Equation, both of which are improvements over the original Louvain method [14, 34]. However, Leiden comes closer to the true number of clusters  $k_{\mathrm{true}}$ , and in that sense,  $\mathrm{PMI}_2$  led to a better choice of algorithm.
Table 2: Comparison of the  $\mathrm{PMI}_q$  to the clustering comparison metrics in the systematic review by Gósgens et al. [9]. Examples for type II biasedness can be found in Appendix A. We consider a metric computationally tractable if its asymptotic complexity is linear in the number of data points  $N$  but not necessarily in the numbers of clusters  $k_A, k_B$ . The rationale is that in many cases, the number of clusters is much lower than the number of data points and metrics like the  $\mathrm{AMI}_1$  with  $\mathcal{O}(N \max\{k_A, k_B\})$  are widely used in practice [27, 29]. The  $\mathrm{PMI}_2$  is the first metric to be Type II unbiased and monotonous and, while computationally demanding, has efficient approximations.
NMI	NMI max	Fair NMI	VI	FMeasure	BCubed	Jaccard	Wallace	Dice	Corr. Coeff.	Sokal&Sneath	Corr. Dist.	Rand Index	AM1	AM2	SM1	SM2	PML1	PM2
Type I unbiased	X	X	X	X	X	X	X	X	X	✓	✓	X	X	✓	✓	✓	✓	✓	✓
Type II unbiased	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	○	○	✓	✓
Monotonicity	✓	X	X	✓	X	✓	✓	X	✓	✓	✓	✓	✓	✓	✓	X	X	X	✓
Comp. tractable	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	X	✓	○	○
7 Conclusion and outlook
Table 2 summarizes our findings for the  $\mathrm{PMI}_q$  and compares its theoretical properties to 17 clustering comparison measures from the systematic review by Gösgens et al. [9]. We introduce the first type II unbiased and monotonous cluster comparison method, the  $p$ -value adjusted Rand Index  $(\mathrm{PMI}_2)$ . Existing methods that addressed type II bias, namely the Standardized Mutual Information  $(\mathrm{SMI}_1)$  and the Standardized Rand Index  $(\mathrm{SMI}_2)$  are not monotonous, meaning clusterings closer to the ground truth can score worse. In addition, the  $\mathrm{SMI}_1$  has high computational complexity, making it unsuitable in practice. For the  $\mathrm{SMI}_2$  we showed that an efficient algorithm exists and we leverage this algorithm for an efficient approximation of the proposed  $\mathrm{PMI}_2$ . However, our analysis of the errors in this standardized approximation is limited to experimental observations, leaving a theoretical analysis for future work. We devised a Monte Carlo approximation for the  $\mathrm{PMI}_2$  with tunable approximation error, for when theoretical guarantees are required. To validate our theoretical findings, synthetic experiments confirm that the presented  $\mathrm{PMI}_2$  selects different cluster sizes with equal probability and is not subject to type II bias. In practice, the  $\mathrm{PMI}_2$  chooses better clustering algorithms from a set of candidates when a ground truth reference is available. Thanks to its monotonicity and computational efficiency, the  $\mathrm{PMI}_2$  is a practical candidate for evaluating cluster similarity without type II bias. While we investigated  $p$ -value adjustment for the family of generalized information-theoretic clustering comparison measures, further research is required to understand if other comparison measures, like the Jaccard Index, could benefit from a similar adjustment.
This work was funded by the Digital Europe Grant Testing and Experimentation Facility for Health AI and Robotics (TEF-Health), Project number 101100700 and the Bayerischen Verbundforderprogramm (BayVFP) - Forderlinie Digitalisierung - Forderbereich Informations- und Kommunikationstechnik of the Bavarian Ministry of Economic Affairs, Regional Development and Energy and supported by Bayern Innovativ - Bayerische Gesellschaft für Innovation und Wissentransfer mbH.
[1] C. C. Aggarwal and C. K. Reddy, editors. Data Clustering: Algorithms and Applications. CRC Press, 2014. ISBN 978-1-4665-5821-2.
[2] N. Arinik, V. Labatut, and R. Figueiredo. Characterizing and comparing external measures for the assessment of cluster analysis and community detection. IEEE Access, 9:20255-20276, 2021. doi: 10.1109/ACCESS.2021.3054621.
[37] D. L. Wallace. Asymptotic Approximations to Distributions. The Annals of Mathematical Statistics, 29(3):635-654, Sept. 1958. ISSN 0003-4851, 2168-8990. doi: 10.1214/aoms/1177706528.
[38] S. Wei, G. Han, R. Wang, Y. Yang, H. Zhang, and S. Li. Inductive multi-view multiple clusterings. In 2021 7th International Conference on Big Data and Information Analytics (BigDIA), pages 308-315, 2021. doi: 10.1109/BigDIA53151.2021.9619704.
[39] William M. Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 66(336):846-850, 1971. doi: 10.1080/01621459.1971.10482356.
A Type II Bias
We show that the  $\mathrm{PMI}_q$  is both type I and type II unbiased:
Proof of Proposition 3.1. The choice of  $\theta(0) = \frac{1}{2}$  allows us to use  $\theta(x) = 1 - \theta(-x)$  such that
$$
\begin{array}{l} \mathrm {P M I} _ {q} (A, B) = \quad \mathbb {E} _ {\sigma \in S _ {N}} \left[ \theta \left(H _ {q} (A, B) - H _ {q} (\sigma (A), B)\right) \right] = \\ \mathbb {E} _ {\tilde {\sigma} \in S _ {N}} \left[ \theta \left(H _ {q} (\tilde {\sigma} (A), \tilde {B}) - H _ {q} (A, \tilde {B})\right) \right] = \tag {14} \\ 1 - \mathbb {E} _ {\tilde {\sigma} \in S _ {N}} \left[ \theta \left(H _ {q} (A, \tilde {B}) - H _ {q} (\tilde {\sigma} (A), \tilde {B})) \right], \right. \\ \end{array}
$$
with  $\tilde{B} = \sigma^{-1}(B)$  and  $\tilde{\sigma} = \sigma^{-1}$ . Now in the definition of type I unbiasedness,  $\mathcal{B}$  is element-symmetric, such that  $\tilde{B} \in \mathcal{B}$  and
$$
\mathbb {E} _ {B \sim \mathcal {B}} [ \mathrm {P M I} _ {q} (A, B) ] = 1 - \mathbb {E} _ {\tilde {B} \sim \mathcal {B}} [ \mathrm {P M I} _ {q} (A, \tilde {B}) ] = \frac {1}{2}. \tag {15}
$$
Hence,  $\mathrm{PMI}_q$  is type I unbiased.
For type II unbiasedness,  $\mathcal{B},\mathcal{B}^{\prime}$  are element symmetric and with Eq. 14 we get
$$
\begin{array}{l} \mathbb {E} _ {B \sim B, B ^ {\prime} \sim B ^ {\prime}} [ \theta (\mathrm {P M I} _ {q} (A, B) - \mathrm {P M I} _ {q} (A, B ^ {\prime})) ] = \\ 1 - \mathbb {E} _ {B \sim \mathcal {B}, B ^ {\prime} \sim \mathcal {B} ^ {\prime}} \left[ \theta \left(\operatorname {P M I} _ {q} (A, B ^ {\prime}) - \operatorname {P M I} _ {q} (A, B)\right) \right] = \tag {16} \\ 1 - \mathbb {E} _ {\tilde {B} \sim \mathcal {B}, \tilde {B} ^ {\prime} \sim \mathcal {B} ^ {\prime}} [ \theta (\mathrm {P M I} _ {q} (A, \tilde {B}) - \mathrm {P M I} _ {q} (A, \tilde {B} ^ {\prime})) ] = \frac {1}{2}. \\ \end{array}
$$
For all other clustering comparison measures in Table 2, the proof or a counterexample to type I unbiasedness are given in [9]. Here, we provide an example for their type II bias. For  $N = 4$  data points, we compare the clustering  $A = \{\{1,4\}, \{2,3\}\}$  with all clusterings with 2 clusters, distributed uniformly  $\mathcal{B}$ . We also compare  $A$  to all clusterings from the uniform distribution  $\mathcal{B}'$  of  $N = 4$  data points into 3 clusters. All comparison measures except the  $\mathrm{PMI}_2$  prefer either 2 or 3 clusters in the sense that their expectation value in Eq. 4 is not equal to  $1/2$ , and are thus type II biased (See Table 3).
Table 3: Example for type II bias of other clustering comparison measures. For  $A = \{\{1,4\}, \{2,3\}\}$  and  $\mathcal{B}, \mathcal{B}'$  the uniform distribution of all clusterings with 4 elements into 2, 3 clusters respectively, the expected value  $\mathbb{E}_{B \sim \mathcal{B}, B' \sim \mathcal{B}'}[\theta(V(A, B) - V(A, B'))] \neq \frac{1}{2}$  and hence these measures are type II biased.
NMI	NMI max	Fair NMI	VI	FMeasure	BCuhed	Jaccard	Wallace	Dice	Corr. Coeff.	Sokal&Sneath	Corr. Dist.	Rand Index	AM1	AM2	SM1	SM2
1/7	1/7	11/21	10/21	11/21	13/21	13/21	5/7	13/21	11/21	11/21	10/21	1/3	11/21	11/21	25/42	25/42
B Monte Carlo  $\mathrm{PMI}_q$
Proof of proposition 5.2. Random contingency tables with fixed marginals can be generated in  $\mathcal{O}(\min (N,k_Ak_B\log N))$  [4, 26]. The error  $a = \sqrt{\mathrm{PMI}_q(1 - \mathrm{PMI}_q) / N_{\mathrm{samples}}}$  decreases with the inverse square root of the number of Monte Carlo samples  $N_{\mathrm{samples}}$ . Hence we need  $N_{\mathrm{samples}}\leq a^{-2} / 4$  samples to reach the approximation error  $a$ .
C The standardized Rand Index
For the proof of the runtime complexity of the  $\mathrm{SMI}_2$ , we use the fact that it is equivalent to the SRI (Corollary 2 in [30])
$$
\mathrm {S R I} := \frac {\mathrm {R I} - \mathbb {E} [ \mathrm {R I} ]}{\sqrt {\operatorname {V a r} [ \mathrm {R I} ]}} = \frac {\sum_ {i j} \binom {n _ {i j}} {2} - \mathbb {E} \left[ \sum_ {i j} \binom {n _ {i j}} {2} \right]}{\mathbb {E} \left[ \left(\sum_ {i j} \binom {n _ {i j}} {2}\right) ^ {2} \right] - \mathbb {E} \left[ \sum_ {i j} \binom {n _ {i j}} {2} \right] ^ {2}} = \mathrm {S M I} _ {2}. \tag {17}
$$
Proof of Proposition 5.1. Hubert and Arabie [11] derived the expected value under random permutation as
$$
\mathbb {E} \left[ \sum_ {i j} \binom {n _ {i j}} {2} \right] = \sum_ {i j} \frac {\binom {a _ {i}} {2} \binom {b _ {j}} {2}}{\binom {N} {2}}. \tag {18}
$$
For the variance, we also need the second moment. While Romano et al. [30] give a general formula in Eq. (13), it is impractical in the given form as the authors themselves note the runtime complexity is  $\mathcal{O}(N^3 k_A\max (k_A,k_B))$ . We observe that in the special case of the SRI, the higher moments can be simplified by leveraging the identity [39]
$$
\mathbb {E} \left[ \binom {n _ {i j}} {2} ^ {m} \mid n _ {i j} \sim \operatorname {H y p} \left(a _ {i}, b _ {j}, N\right) \right] = \sum_ {l = 2} ^ {\min  (N, 2 m)} \frac {S _ {2 , 2} (m , l)}{2 ^ {m}} \prod_ {p = 0} ^ {l - 1} \frac {(a _ {i} - p) (b _ {j} - p)}{N - p}, \tag {19}
$$
with the generalized Stirling Number  $S_{2,2}(m,k)$  [22] and the hypergeometric distribution Hyp. Note that the right-hand side is completely independent of  $n_{ij}$ . Hence the expected values under the hypergeometric distributions in Eq. (13) in [30] can be calculated in  $\mathcal{O}\big(\max \{k_A,k_B\} \big)$  time, giving
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {i j} {\binom {n _ {i j}} {2}} ^ {2} \right] = \left(2 \gamma_ {a} \sum_ {j = 1} ^ {k _ {B}} (N - b _ {j}) (N - 3 (b _ {j} - 1)) (b _ {j} - 1) b _ {j} \right. \\ + \sum_ {i = 1} ^ {k _ {A}} a _ {i} ^ {2} (a _ {i} - 1) \sum_ {j = 1} ^ {k _ {B}} (4 N - 5 b _ {j} + 3) (b _ {j} - 2) (b _ {j} - 1) b _ {j} \\ + \sum_ {i = 1} ^ {k _ {A}} a _ {i} ^ {3} (a _ {i} - 1) \sum_ {j = 1} ^ {k _ {B}} (b _ {j} - 3) (b _ {j} - 2) (b _ {j} - 1) b _ {j} \\ + \left(\gamma_ {a} ^ {2} - \gamma_ {a, 2}\right) \sum_ {j = 1} ^ {k _ {B}} b _ {j} (b _ {j} - 1) (b _ {j} - 2) (b _ {j} - 3) \tag {20} \\ + \left(\gamma_ {b} ^ {2} - \gamma_ {b, 2}\right) \sum_ {i = 1} ^ {k _ {A}} a _ {i} (a _ {i} - 1) (a _ {i} - 2) (a _ {i} - 3) \\ \left. + \left(\gamma_ {a} ^ {2} - \gamma_ {a, 2}\right) \left(\gamma_ {b} ^ {2} - \gamma_ {b, 2}\right)\right) \\ \frac {1}{4 \left(N (N - 1) (N - 2) (N - 3)\right)}, \\ \end{array}
$$
for the most general case  $N \geq 4$  and no cluster larger than  $N - 2$ , with
$$
\gamma_ {a} = \sum_ {i = 1} ^ {N} a _ {i} (a _ {i} - 1) \quad \gamma_ {a, 2} = \sum_ {i = 1} ^ {N} a _ {i} ^ {2} (a _ {i} - 1) ^ {2} \quad \gamma_ {b} = \sum_ {j = 1} ^ {N} b _ {j} (b _ {j} - 1) \quad \gamma_ {b, 2} = \sum_ {j = 1} ^ {N} b _ {j} ^ {2} (b _ {j} - 1) ^ {2}. \tag {21}
$$
The other cases can be treated analogously using Eq. (19). Hence the dominant factor for the runtime complexity of the SRI is the RI itself with  $\mathcal{O}(k_A k_B)$  for the sum over all contingency matrix elements.
D Community detection parameter configurations
For the experiment in Figure 4c, we compared five community detection algorithms implemented in networkkit [32]. Label Propagation and Degree Ordered Label Propagation do not have any parameters. The parameter choices for the other algorithms is listed in Table 4.
Table 4: Parameter configurations for the experiment on the email EU core dataset. For Leiden, we used all combinations of  $\gamma$  and randomize.
Algorithm	Parameters
Louvain	γ ∈ {0.001, 0.01, 0.1, 1.0}
Louvain Map Equation	hierarchical ∈ {True, False}
Leiden	γ ∈ {1 × 10-6, 1 × 10-5, 0.0001, 0.001, 0.01, 0.1, 1.0}
randomize ∈ {True, False}
E Spectral clustering on image datasets
We conducted two additional experiments similar to the ones in Section 6.1, but using spectral clustering instead of  $k$ -means. We apply spectral clustering with varying number of clusters  $k$  to the UCI image segmentation dataset [6] and a texture classification dataset from OpenML [36]. We compare each clustering solution with the ground truth labels and select the clustering with the highest RI,  $\mathrm{AMI}_2$  and  $\mathrm{PMI}_2$ . Figure 5 shows the selection frequency of each number of clusters  $k_{\mathrm{selected}}$  after 1000 trials with different random seeds. In the case of the image segmentation dataset, different subsets of 1000 samples were chosen for each trial, due to the steep runtime requirements of spectral clustering. The results align with the  $k$ -means experiment in Section 6.1: The  $\mathrm{PMI}_2$  selects clusterings in a less biased way, in the sense that the selected number of clusters  $k_{\mathrm{selected}}$  is closer to the true number of clusters  $k_{\mathrm{true}}$  than for RI and  $\mathrm{AMI}_2$ .
Figure 5: We apply spectral clustering to a) the UCI image segmentation dataset and b) a texture classification dataset. The number of clusters parameter  $k$  is set to eleven values between  $k_{\mathrm{true}} / 2$  and  $3k_{\mathrm{true}} / 2$ . We compare the resulting clusterings with the ground truth via RI,  $\mathrm{AMI}_2$ , and  $\mathrm{PMI}_2$  and select the best clustering  $k_{\mathrm{selected}}$  according to each metric. We repeat the experiment with 1000 different random seeds and plot the selection probabilities of  $k_{\mathrm{selected}}$  for each metric. The  $\mathrm{PMI}_2$  selects candidates where the number of clusters is closer to the true number of clusters  $k_{\mathrm{true}}$  on average (dashed lines) compared to the RI and  $\mathrm{AMI}_2$ .