SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$p$ -value Adjustment for Monotonous, Unbiased, and Fast Clustering Comparison
Kai Klede<sup>1</sup> Thomas Altstidl<sup>1</sup> Dario Zanca<sup>1</sup> Björn Eskofier<sup>1,2</sup>
<sup>1</sup>Machine Learning and Data Analytics (MaD) Lab
Friedrich-Alexander Universität Erlangen-Nürnberg
2Translational Digital Health Group
Institute of AI for Health, Helmholtz Zentrum München
{kai.klede; thomas.r.altstidl; dario.zanca; bjoern. eskofier}@fau.de
Popular metrics for clustering comparison, like the Adjusted Rand Index and the Adjusted Mutual Information, are type II biased. The Standardized Mutual Information removes this bias but suffers from counterintuitive non-monotonicity and poor computational efficiency. We introduce the  $p$ -value adjusted Rand Index (PMI $_2$ ), the first cluster comparison method that is type II unbiased and provably monotonous. The PMI $_2$  has fast approximations that outperform the Standardized Mutual information. We demonstrate its unbiased clustering selection, approximation quality, and runtime efficiency on synthetic benchmarks. In experiments on image and social network datasets, we show how the PMI $_2$  can help practitioners choose better clustering and community detection algorithms.
Clustering is fundamental to unsupervised learning, and practitioners can choose from many algorithms to partition a dataset into homogeneous clusters. Therefore it is common to annotate parts of an otherwise unlabeled dataset and select the clustering algorithm that best reproduces the annotations [1]. A good selection crucially depends on the clustering comparison method, like the Mutual Information (MI) [25] or the Rand Index (RI) [39]. The importance of the comparison method further increases with the advent of deep learning methods for applications such as community detection, in which they serve as components of loss functions during network training [10, 12, 33]. Some use cases admit multiple clustering solutions [23, 38], and clustering comparison can help identify qualitatively different clustering solutions for a single dataset. Other applications of clustering comparison include categorical feature selection or consensus clustering [5, 17].
The MI and RI are biased towards clusterings with particular cluster size distributions [19] (type I bias). For example, the MI favors larger clusters for any reference clustering [24]. The Adjusted Rand Index (ARI) [11] and Adjusted Mutual Information (AMI) [25] achieve a constant baseline value by subtracting the expected value under random permutation of the cluster labels. However, they still exhibit a bias when multiple clusterings are compared via a fixed ground truth (type II bias) [29], as opposed to comparing two random clusterings with each other. This type II scenario typically arises when selecting the best algorithm for a given task on a labeled subset of the data.
Romano et al. [30] showed via the Tsallis entropy, that the AMI and ARI are special cases of generalized information-theoretic clustering comparison measures  $\mathrm{AMI}_q$  and proposed standardization to resolve both types of biases. However, the runtime for standardization generally exhibits a substantial time complexity of  $\mathcal{O}(N^3 k_A\max (k_A,k_B))$  [30], where  $N$  represents the number of data points and  $k_{A}$  and  $k_{B}$  denote the respective number of clusters. This complexity is prohibitive for many applications [15]. Gösgens et al. [9] found that the Standardized Mutual Information (SMI) does not
increase monotonically as one clustering is adjusted to match a reference and therefore reject the SMI entirely.
This work presents the  $p$ -value of the  $\mathrm{MI}_q$  (denoted  $\mathrm{PMI}_q$ ) as a provably monotonous type II bias correction. We formally define type II bias and prove that the  $\mathrm{PMI}_q$  does not suffer from it. For the  $\mathrm{SMI}_q$ , there is only empirical evidence [29, 30]. We show that the  $\mathrm{PMI}_q$  is monotonous for  $q \geq 2$ . This includes the  $p$ -value of the RI, but for the MI, the  $p$ -value is not monotonous. When normalized with the normal CDF, the  $\mathrm{SMI}_q$  approximates the  $\mathrm{PMI}_q$ , which we confirm via Monte Carlo simulation. We reduce the runtime of the  $\mathrm{SMI}_2$  from  $\mathcal{O}(N^3 k_A\max (k_A,k_B))$  to a much more practical  $\mathcal{O}(k_Ak_B)$  by a reformulation of the variance term. We demonstrate the impact of type II unbiased algorithm selection for community detection on a social network dataset and clustering on images of handwritten digits and human faces.
2 Generalized information theoretic clustering comparison measures
A clustering  $A$  of  $N$  data points is a partition of the set  $\{1, \dots, N\}$  into disjoint subsets  $A = \{A_1, \dots, A_{k_A}\}$ .  $A_i$  denotes the set of points in the  $i$ -th cluster of size  $a_i := |A_i|$  and  $k_A$  is the number of clusters in  $A$ . Clustering comparison measures quantify the similarity between two clusterings  $A$  and  $B$ , and can be expressed as a function of the contingency table (Table 1).
Table 1: Contingency table for clusterings  $A$ ,  $B$  with  $k_{A}$  and  $k_{B}$  clusters respectively. Lower case  $a_{i}$  and  $b_{j}$  denote the cluster size of the  $i$ -th and  $j$ -th clusters in  $A$  and  $B$ , while  $n_{ij}$  represents the size of their overlap. Clustering comparison measures can be expressed in terms of the elements of this contingency table.
While many clustering comparison methods exist in the literature [2, 9], many well-known methods like the Variation of Information, Mirkin Index, or Rand Index belong to the family of generalized information-theoretic clustering comparison measures [30, 31]. When adjusted for chance, these measures reduce to the mutual information with Tsallis  $q$ -entropy [35], which will be the focus of this work.
Definition 2.1 (Tsallis  $q$ -entropy). Let  $q \in \mathbb{R}_+$ , and  $A$  be a clustering. Then, the Tsallis  $q$ -entropy is
$$
H _ {q} (A) = - \sum_ {i = 1} ^ {k _ {A}} \left(\frac {a _ {i}}{N}\right) ^ {q} \log_ {q} \frac {a _ {i}}{N}, \tag {1}
$$
with the  $q$ -logarithm  $\log_q(y) \coloneqq (y^{1 - q} - 1) / (1 - q)$  if  $q \neq 1$  and the natural logarithm for  $q = 1$ , where  $x \log_1(x) = 0$  for  $x = 0$ .
The generalized mutual information is defined in analogy to the mutual information but with Tsallis  $q$ -entropy replacing the Shannon entropy.
Definition 2.2 (Generalized mutual information). Let  $A, B$  be two clusterings of the set  $\{1, \dots, N\}$  and  $q \in \mathbb{R}_+$ , then the generalized mutual information is
$$
\mathrm {M I} _ {q} (A, B) = H _ {q} (A) + H _ {q} (B) - H _ {q} (A, B). \tag {2}
$$
Here  $H_{q}(A,B)$  denotes the joint  $q$ -entropy  $H_{q}(\{A_{i}\cap B_{j}\mid A_{i}\in A\land B_{j}\in B\})$
3 Adjustment for chance
The bare  $\mathrm{MI}_q$  has limited value as a clustering comparison measure. When comparing two clusterings directly with one another, it is biased towards larger or smaller clusters, depending on the value of
$q$  (type I bias) [19, 25]. But even after adjusting for type I bias, there is another, more subtle bias when multiple clusterings are compared via a single ground truth [29] (Figure 1). In Section 3.2, we introduce the  $p$ -value as an adjustment to the latter type II bias.
3.1 Type I bias
It is well known throughout the literature that the  $\mathrm{MI}_1$  is biased towards smaller clusters in direct clustering comparisons [19, 25]. To make this precise, Gösgens et al. [9] defined a family of clustering distributions for which the expected similarity to a reference clustering should be constant:
Definition 3.1 (Element-symmetric distribution). A distribution over clusterings  $\mathcal{B}$  is element-symmetric if every two clusterings  $B$  and  $B'$  with the same cluster sizes have the same probability.
An example of an element-symmetric distribution is the uniform distribution over all clusterings of  $N$  elements into  $k$  clusters. If the clustering is random, a comparison measure should not favor one particular  $k$  over another. If it does, we call it type I biased [9].
Definition 3.2 (Type I unbiased). A clustering measure  $V$  is type I unbiased if there is a constant  $c$ , such that for any clustering  $A$  with  $1 < k_{A} < N$  and every element-symmetric distribution  $\mathcal{B}$  the expected value  $\mathbb{E}_{B \sim \mathcal{B}}[V(A, B)] = c$  is constant.
In other words, type I unbiased means that when comparing a fixed clustering  $A$  to all permutations of any clustering  $B$ , the average metric value is the same for all  $A$ . As the  $\mathrm{MI}_q$  has this type I bias, it is commonly adjusted by subtracting its expected value under random permutation, yielding a type I unbiased measure [30].
$$
\mathrm {A M I} _ {q} (A, B) := \frac {\mathrm {M I} _ {q} (A , B) - \mathbb {E} _ {\sigma \in S _ {N}} [ \mathrm {M I} _ {q} (A , \sigma (B)) ]}{\frac {1}{2} \left(H _ {q} (A) + H _ {q} (B)\right) - \mathbb {E} _ {\sigma \in S _ {N}} [ \mathrm {M I} _ {q} (A , \sigma (B)) ]}. \tag {3}
$$
$S_{N}$  denotes the symmetric group and  $\frac{1}{2} (H_q(A) + H_q(B))$  is an upper bound to the  $\mathrm{MI}_q$  such that the  $\mathrm{AMI}_q$  is normalized to  $c = 0$  for random clusterings and upper bounded by 1 [25, 30]. Adjustments with respect to other random models are possible [8, 18]. However, the random permutation model remains the most popular and is the focus of this work. Due to the generalization using Tsallis entropy, the  $\mathrm{AMI}_2$  corresponds to the Adjusted Rand Index [11, 30].
3.2 Type II bias
However, in a typical external validation scenario, a single absolute value of the  $\mathrm{AMI}_q$  is of little help. While it is easy to understand that an  $\mathrm{AMI}_q$  of zero means a clustering algorithm is no better than random and a value of one means optimal agreement, the scale of the range in between is unclear. Therefore, the  $\mathrm{AMI}_q$  values of multiple candidate solutions with a reference are typically compared against each other to find the best algorithm for a given dataset.
As a toy model for this scenario, we uniformly generate 5000 clusterings of  $N = 500$  elements for each number of clusters  $k_{B} \in \{2,6,10,14,18,22\}$  [29]. We compare them to a fixed clustering  $A$  with  $k_{A} = 10$  evenly sized clusters and plot the selection probabilities for the RI, MI, and  $\mathrm{AMI}_q$  for  $q \in \{1,2\}$  (Figures 1a, b, d and e). The bare MI and RI and their adjusted variants  $\mathrm{AMI}_q$  are biased towards certain values of  $k_{B}$ .
We generalize and formalize this observation by demanding a clustering comparison measure to favor no element-symmetric distribution over another.
Definition 3.3 (Type II unbiased). Let  $V$  be a clustering comparison measure and  $\mathcal{B},\mathcal{B}^{\prime}$  be element-symmetric clustering distributions.  $V$  is type II unbiased if
$$
\mathbb {E} _ {B \sim \mathcal {B}, B ^ {\prime} \sim \mathcal {B} ^ {\prime}} [ \theta (V (A, B) - V (A, B ^ {\prime})) ] = \frac {1}{2} \tag {4}
$$
for any clustering  $A$  with  $1 < k_{A} < N$ , where  $\theta$  denotes the Heaviside step function with  $\theta(0) = 1/2$ .
Intuitively, Type I bias means that certain cluster sizes receive higher metric values. Type II bias, on the other hand, gives a higher relative rank to certain cluster sizes when multiple clusterings are compared with a ground truth.
Selected number of clusters
Figure 1: We compare a fixed reference clustering with  $k_{A} = 10$  even clusters, to random clusterings with  $k_{B} \in \{2,6,10,14,18,22\}$  clusters. The plot shows the selection probabilities of each  $k_{B}$  for the MI and RI and its adjusted  $(\mathrm{AMI}_q)$  and our  $p$ -value adjusted  $(\mathrm{PMI}_q)$  variants after 5000 repetitions. The RI, MI, and  $\mathrm{AMI}_q$  are type II biased, while our  $\mathrm{PMI}_q$  selects each cluster size with equal probability.
Romano et al. [29] introduced standardization to correct for type II bias
$$
\operatorname {S M I} _ {q} (A, B) := \frac {\operatorname {M I} _ {q} (A , B) - \mathbb {E} _ {\sigma \in S _ {N}} \left[ \operatorname {M I} _ {q} (A , \sigma (B)) \right]}{\sqrt {\mathbb {E} _ {\sigma \in S _ {N}} \left[ \operatorname {M I} _ {q} (A , \sigma (B)) \right] ^ {2} - \mathbb {E} _ {\sigma \in S _ {N}} \left[ \operatorname {M I} _ {q} (A , \sigma (B)) ^ {2} \right]}}. \tag {5}
$$
They observed in numerical simulations that in the toy model above (Figure 1), the  $\mathrm{SMI}_q$  selects each  $k_B$  with approximately equal probability [30].
However, the  $\mathrm{MI}_q$  is not normally distributed under random permutation (Figure 2a), and standardization is only an approximation to the true  $p$ -value (Figure 2b). The  $p$ -value quantifies what percentage of all permutations of the data would have led to higher mutual information, and we propose to use it for clustering comparison.
Definition 3.4 ( $p$ -value adjusted, generalized mutual information). Let  $A, B$  be two partitions of the set  $\{1, \ldots, N\}$ ,  $q \in \mathbb{R}_+$ . Assuming the random permutation model, the  $p$ -value adjusted, generalized mutual information is
$$
\mathrm {P M I} _ {q} := \mathbb {E} _ {\sigma \in S _ {N}} [ \theta (\mathrm {M I} _ {q} (A, B) - \mathrm {M I} _ {q} (\sigma (A), B)) ]. \tag {6}
$$
Note that as the marginal entropies are independent of the permutation,  $\mathrm{PMI}_q = \mathbb{E}_{\sigma \in S_N}[\theta (H_q(\sigma (A),B) - H_q(A,B))]$ . For  $q = 1$ , this is the  $p$ -value of the mutual information and the variation of information by definition. For  $q \neq 1$  the  $\mathrm{PMI}_q$  further simplifies to  $\mathbb{E}_{\sigma \in S_N}[\theta (n_{ij}' - n_{ij}^q)]$  for  $q > 1$  and  $\mathbb{E}_{\sigma \in S_N}[\theta (n_{ij}' - n_{ij}'^q)]$  for  $q < 1$  with  $n_{ij}'$  being the elements of the contingency table for  $\sigma (A), B$ . In a sense, the details of the generalized mutual information don't matter under  $p$ -value adjustment, except the exponent  $q$  of the contingency matrix elements. In fact, the  $p$ -value of the Rand Index is equivalent to the  $\mathrm{PMI}_2$  by a very similar argument.
In the experiment in Figure 1, we observe that the  $\mathrm{PMI}_1$  and  $\mathrm{PMI}_2$  select each  $k_B$  with approximately equal probability.
Proposition 3.1. The  $\mathrm{PMI}_q$  is type I and type II unbiased.
We formally prove this result in Appendix A.
Figure 2: The probability of obtaining a particular  $\mathrm{MI}_2$  under random permutation for two fixed clusterings  $A, B$  each of 100 elements. Our  $\mathrm{PMI}_2$  (blue bars in a)) takes the true distribution of the  $\mathrm{MI}_2$  into account, whereas the  $\mathrm{SMI}_2$  (shaded blue region in b)) is based on a continuous normal approximation. However, when normalized with the normal CDF  $\Phi$ , the  $\mathrm{SMI}_2$  is a good approximation of the  $\mathrm{PMI}_2$  as shown in Figure c). Here, we sampled 1000 pairs of clusterings uniformly at random for different numbers of elements  $N$ . We plot the absolute difference between Monte Carlo estimates of the  $\mathrm{PMI}_2$  and normalized  $\mathrm{SMI}_2$  values as a function of the two-sided  $p$ -value. The larger the dataset size  $N$ , the better  $\Phi(\mathrm{SMI}_2)$  approximates the true  $\mathrm{PMI}_2$ .
4 Monotonicity
When one clustering is changed to resemble another more, any clustering similarity measure should increase. A major drawback of the  $\mathrm{SMI}_1(A,B)$  is its non-monotonous behavior as the number of pairs of elements that agree in  $A$  and  $B$  increases [9]. We show that the  $\mathrm{PMI}_q$ , on the other hand, is monotonous for  $q \geq 2$ .
4.1 Definition of monotonicity
The atomic operations that add new pairs of agreeing elements in two clusterings are the perfect split and the perfect merge [9].
Definition 4.1 (Perfect split).  $B'$  is a perfect split of  $B$  with respect to  $A$  if it splits a single cluster  $B_1 \in B$  into  $B_1', B_2' \in B'$  such that for all  $i$  either  $A_i \cap B_1 \subset B_1'$  or  $A_i \cap B_1 \subset B_2'$ .
Definition 4.2 (Perfect merge).  $B'$  is a perfect merge of  $B$  with respect to  $A$  if  $B'$  is obtained by merging two clusters  $B_1, B_2 \in B$  with  $B_1, B_2 \subset A_i$  for some  $i$ .
Gösgens et al. [9] require that any clustering similarity measure increases monotonically for any combination of perfect splits and perfect merges.
Definition 4.3 (A-consistent improvement).  $B'$  is an  $A$ -consistent improvement of  $B$  iff there exists a series of perfect splits and perfect merges that change  $B$  into  $B'$ .
Definition 4.4 (Monotonicity). A symmetric clustering comparison measure  $V$  is monotonous if for every  $A, B$  with  $1 < k_A < N$  and any  $A$ -consistent improvement  $B'$  of  $B$ ,  $V(A, B') > V(A, B)$ .
We show that the  $\mathrm{PMI}_q$  is monotonous for  $q\geq 2$ , making it the first known clustering comparison measure to be both type II unbiased and monotonous. The case  $q = 2$  is particularly interesting as it corresponds to the well-known Rand Index.
4.2 Proof of monotonicity for  $\mathrm{PMI}_q$  with  $q\geq 2$
The proof can be broken down into monotonicity under perfect splits and perfect merges. We first show that the joint  $q$ -entropy increases under any split that is not perfect.
Lemma 4.1. Let  $A, B$  be clusterings with  $1 < k_A < n$  and  $B'$  be obtained by splitting a cluster  $B_j \in B$  into non-empty clusters  $B_{j_1}', B_{j_2}'$ . Then  $H_q(A, B') = H_q(A, B) + \Delta H_q$  with  $\Delta H_q \geq 0$  and equality iff the split is perfect with respect to  $A$ .
This statement is a direct consequence of the subadditivity of the  $q$ -entropy [7]. In particular  $h_q:p\mapsto p^q\log_qp$  is a strictly convex function with  $h_q(0) = 0$  and hence strictly superadditive, i.e.  $h_q(p_1 + p_2)\geq h_q(p_1) + h_q(p_2)$  for any  $p_1,p_2\geq 0$  with equality iff  $p_1 = 0\lor p_2 = 0$
Proof of Lemma 4.1. We express  $\Delta H_{q}$  as
$$
\begin{array}{l} \Delta H _ {q} = \sum_ {i} \left[ \left(\frac {n _ {i j}}{N}\right) ^ {q} \log_ {q} \left(\frac {n _ {i j}}{N}\right) - \left(\frac {n _ {i j _ {1}} ^ {\prime}}{N}\right) ^ {q} \log_ {q} \left(\frac {n _ {i j _ {1}} ^ {\prime}}{N}\right) - \left(\frac {n _ {i j _ {2}} ^ {\prime}}{N}\right) ^ {q} \log_ {q} \left(\frac {n _ {i j _ {2}} ^ {\prime}}{N}\right) \right] \\ = \sum_ {i} h _ {q} \left(\frac {n _ {i j}}{N}\right) - h _ {q} \left(\frac {n _ {i j 1} ^ {\prime}}{N}\right) - h _ {q} \left(\frac {n _ {i j 2} ^ {\prime}}{N}\right). \tag {7} \\ \end{array}
$$
From  $n_{ij} = n_{ij_1}' + n_{ij_2}'$  and the strict superadditivity of  $h_q$  follows  $\Delta H_q \geq 0$  with equality iff  $n_{ij_1}' = 0 \lor n_{ij_2}' = 0$ , i.e. when the split is perfect.
Conversely, a perfect merge maximizes the difference in joint entropy.
Lemma 4.2. Let  $A, B, B'$  and  $\Delta H_q$  be as in Lemma 4.1, then for  $q \geq 2$ ,  $\Delta H_q$  is maximal iff  $B$  is a perfect merge of  $B'$  with respect to  $A$ .
Proof of Lemma 4.2. When  $B$  is a perfect merge of  $B'$  with respect to  $A$ , then
$$
\Delta H _ {q} ^ {\text {p e r f e c t}} = h _ {q} \left(\frac {b _ {j}}{N}\right) - h _ {q} \left(\frac {b _ {j _ {1}} ^ {\prime}}{N}\right) - h _ {q} \left(\frac {b _ {j} - b _ {j _ {1}} ^ {\prime}}{N}\right). \tag {8}
$$
To show that  $\Delta H_q^{\mathrm{perf}}$  is superadditive as a function of  $b_{j}$ , we take its second derivative
$$
\frac {\mathrm {d} ^ {2}}{\mathrm {d} b _ {j} ^ {2}} \Delta H _ {q} ^ {\text {p e r f e c t}} = \frac {q}{N ^ {q}} \left(b _ {j} ^ {q - 2} - \left(b _ {j} - b _ {j _ {1}} ^ {\prime}\right) ^ {q - 2}\right) \geq 0 \text {f o r} q \geq 2 \text {a n d} b _ {j} > b _ {j _ {1}} ^ {\prime} > 0. \tag {9}
$$
For  $q > 2$ , it is strictly convex and thus strictly superadditive. For  $q = 2$  and  $b_{j} = 0$ , the difference  $\Delta H_{q}^{\mathrm{perfect}} = -b_{j1}^{2}$  is negative and thus  $\Delta H_{2}^{\mathrm{perfect}}$  is also strictly superadditive. Now consider  $\tilde{A}$  such that  $B$  is not a perfect merge with respect to  $\tilde{A}$ . Then at least two  $\tilde{A}_{i_1}, \tilde{A}_{i_2}$  have non-vanishing overlap  $\tilde{n}_{i_1j}, \tilde{n}_{i_2j} > 0$  with  $B_{j}$  such that
$$
\Delta H _ {q} ^ {\text {p e r f e c t}} > \sum_ {i} h _ {q} \left(\frac {\tilde {n} _ {i j}}{N}\right) - h _ {q} \left(\frac {b _ {j _ {1}} ^ {\prime}}{N}\right) - h _ {q} \left(\frac {\tilde {n} _ {i j} - b _ {j _ {1}} ^ {\prime}}{N}\right) \text {f o r} q \geq 2. \tag {10}
$$
With the superadditivity of  $h_q$  follows  $H_q^{\mathrm{perfect}} > H_q^{\mathrm{not~perfect}}$  (Compare Eq. 7).
Now that we know how the joint entropy behaves under perfect splits and perfect merges, we can put together the proof of the monotonicity of the  $\mathrm{PMI}_q$ .
Theorem 4.3. Let  $A, B$  be clusterings with  $1 < k_A < n$  and  $B'$  an  $A$ -consistent improvement of  $B$ . Then  $\mathrm{PMI}_q(A, B') > \mathrm{PMI}_q(A, B)$  for  $q \geq 2$ .
Proof of Theorem 4.3. It suffices to show monotonicity for  $B'$  a perfect split or perfect merge since any  $A$ -consistent improvement of  $B$  can be obtained by a sequence of perfect splits and perfect merges.
Case 1.  $B^{\prime}$  is a perfect split.
Since  $A$  is not a singleton cluster, a permutation  $\sigma$  exists such that  $B'$  is not a perfect split with respect to  $\sigma(A)$  and with Lemma 4.1 it follows
$$
\operatorname {P M I} _ {q} (A, B ^ {\prime}) > \mathbb {E} _ {\sigma \in S _ {N}} \left[ \theta \left(H _ {q} (\sigma (A), B) - H _ {q} (A, B ^ {\prime})\right) \right]. \tag {11}
$$
Figure 3: Runtime of the Monte Carlo  $\mathrm{PMI}_2$  and the  $\mathrm{SMI}_2$  for random clusterings of a)  $N$  elements into  $k_{A} = k_{B} = 10$  clusters and of b) random clusterings with  $N = 1000$  and varying approximation error  $a$ . The  $\mathrm{SMI}_1$  calculation, as proposed in [29], is prohibitively expensive for medium-sized datasets. Our exact reformulation of the  $\mathrm{SMI}_2$  and the Monte Carlo  $\mathrm{PMI}_2$  maintain practical runtimes for high  $N$ . The  $\Phi (\mathrm{SMI}_2)$  is faster, while the Monte Carlo  $\mathrm{PMI}_2$  allows for higher accuracy.
However,  $B'$  is a perfect split of  $B$  with respect to  $A$  and equality holds in Lemma 4.1
$$
\operatorname {P M I} _ {q} (A, B ^ {\prime}) > \mathbb {E} _ {\sigma \in S _ {N}} \left[ \theta \left(H _ {q} (\sigma (A), B) - H _ {q} (A, B)\right) \right] = \operatorname {P M I} _ {q} (A, B). \tag {12}
$$
Case 2.  $B^{\prime}$  is a perfect merge.
Let  $b_{1}, b_{2} \in B$  denote the merged clusters that form  $b_{1}' \in B'$ . Using Lemma 4.2, we find
$$
\operatorname {P M I} _ {q} (A, B ^ {\prime}) = \mathbb {E} _ {\sigma \in S _ {N}} \left[ \theta \left(H _ {q} (\sigma (A), B) - H _ {q} (A, B) + \Delta H _ {q} ^ {\sigma (A)} - \Delta H _ {q} ^ {A}\right) \right] > \operatorname {P M I} _ {q} (A, B), \tag {13}
$$
as there is at least one permutation  $\sigma$  for which the merge is not perfect.
5 Approximations and runtime
A limitation of the  $\mathrm{PMI}_q$  is its computational complexity. Its exact calculation is intractable even for small datasets, as it requires a sum over all contingency tables with given marginals. To mitigate this limitation, we propose two approximation schemes:
1. Standardized approximation  $(\mathbf{q} = 2)$ : We approximate the true, discrete distribution of  $\mathrm{MI}_q$  with a continuous normal distribution that matches its first and second statistical moments (See Figure 2a and b). While this approximation is particularly fast for  $q = 2$ , it does not preserve the theoretical guarantees of the  $\mathrm{PMI}_q$ .
2. Monte Carlo approximation: Given two clusterings  $A$  and  $B$ , we sample contingency tables with the same cluster sizes. The fraction of tables with  $\mathrm{MI}_q$  lower than  $\mathrm{MI}_q(A, B)$  approximates the true  $p$ -value. In this approach, the theoretical guarantees hold up to a tunable approximation error at the cost of higher runtime.
5.1 The standardized Rand Index