SlowGuess/NeurIPS_2023_OCR huggingface dataset:
An exponential family  $\mathcal{E}_{\psi}$  is a parametric class of probability distributions whose densities can be canonically written as  $p_{\theta, \psi}(x) = e^{<\theta, x> - \psi(\theta)}$ , where the density is taken with respect to an appropriate measure,  $\theta \in \Theta$  is a function of the parameters of the distribution that must belong to an open convex space  $\Theta$ , and  $\psi$  is a convex function.
We consider the model defined in (3.1), such that  $f_{ab}$  are zero-inflated distributions and are given by
$$
f _ {a b} (x) = (1 - p _ {a b}) \delta_ {0} (x) + p _ {a b} \tilde {f} _ {a b} (x), \tag {4.1}
$$
where  $p_{ab} \in [0,1]$  is the interaction probability between blocks  $a$  and  $b$ ,  $\delta_0(x)$  is the Dirac delta at zero, and  $\tilde{f}_{ab}$  is a probability density with no mass at zero. Note that this model can represent sparse weighted networks, as edges between nodes in blocks  $a$  and  $b$  are absent with probability  $1 - p_{ab}$ .
Finally, suppose that the distributions  $\{\tilde{f}_{ab}\}$  and  $\{h_a\}$  belong to exponential families. More precisely,
$$
\tilde {f} _ {a b} (x) = e ^ {<   \theta_ {a b}, x > - \psi (\theta_ {a b})} \quad \text {a n d} \quad h _ {a} (y) = e ^ {<   \eta_ {a}, y > - \phi (\eta_ {a})}, \tag {4.2}
$$
for some parameters  $\theta_{ab},\eta_a$  and functions  $\psi ,\phi$  . The following lemma provides the expression of the Chernoff divergence of this model.
Lemma 1. Let  $f_{ab}$  and  $h_a$  be defined as in (4.1)-(4.2). Suppose that  $p_{ab} = \alpha_{ab}\delta$  where  $\alpha_{ab}$  is constant and  $\delta \ll 1$ . We have
$$
I = (1 + o(1))\min_{a\neq b}\sup_{t\in (0,1)}\left\{\sum_{c\in [K]}\pi_{c}\left[tp_{ac} + (1 - t)p_{bc} - p_{ac}^{t}p_{bc}^{1 - t}e^{-J_{\psi}(\theta_{ac}\| \theta_{bc})}\right] + J_{\phi}(\eta_{a}\| \eta_{b})\right\} ,
$$
where  $J_{\psi}(\theta_{ab}\| \theta_{bc}) = t\psi (\theta_{ac}) + (1 - t)\psi (\theta_{bc}) - \psi (t\theta_{ac} + (1 - t)\theta_{bc})$
In combination with Theorem 1, Lemma 1 provides the expression for the exact recovery threshold for node-attributed networks with interaction and attribute distributions given by (4.1)-(4.2).
4.2 Log-likelihood
Given a convex function  $\psi$ , the Bregman divergence  $d_{\psi} \colon \mathbb{R}^m \times \mathbb{R}^m \to \mathbb{R}_+$  is defined by
$$
d _ {\psi} (x, y) = \psi (x) - \psi (y) - <   x - y, \nabla \psi (y) >.
$$
The log-likelihood of the density  $p_{\psi, \theta}$  of an exponential family distribution is linked to the Bregman divergence by the following relationship (see for example [6, Equation 13])
$$
\log p _ {\psi , \theta} (x) = - d _ {\psi^ {*}} (x, \mu) + \psi^ {*} (x), \tag {4.3}
$$
where  $\mu = \mathbb{E}_{p_{\psi, \theta}}(X)$  is the mean of the distribution, and  $\psi^*$  denotes the Legendre transform of  $\psi$ , defined by  $\psi^*(t) = \sup_{\theta} \{<\theta, t> - \psi(\theta)\}$ . The following Lemma provides an expression for the log-likelihood of  $f_{ab}$  when  $f_{ab}$  is a distribution belonging to a zero-inflated exponential family.
Lemma 2. Let  $f_{ab}$  be a probability density as defined in (4.1)-(4.2). For  $x, y \in (0,1)$ , let  $\mathrm{d}_{\mathrm{KL}}(x,y)$  be the Kullback-Leibler divergences between  $\operatorname{Ber}(x)$  and  $\operatorname{Ber}(y)$ , and let  $H(x) = x\log x + (1 - x)\log (1 - x)$ , with the usual convention  $0\log 0 = 1$ . Then,
$$
- \log f _ {a b} (x) = d _ {\mathrm {K L}} \left(x \| p _ {a b}\right) + s d _ {\psi^ {*}} \left(x, \mu_ {a b}\right) - s \psi^ {*} (x) - H (s),
$$
where  $s = 1 (x \neq 0)$ .
Proof. To express  $\log f_{ab}(x)$ , we first note that
$$
\log f _ {a b} (x) = (1 - s) \log (1 - p _ {a b}) + s \log p _ {a b} + s \log (\tilde {f} _ {a b} (x)),
$$
where  $c = 1(x\neq 0)$ . The result follows by adding and subtracting  $H(b)$  in the previous expression and expressing  $\log \tilde{f}_{ab}(x)$  with a Bregman divergence as in (4.3).
Suppose that  $X, Y$  follow the model (3.1) with probability distributions given by (4.1)-(4.2). Let  $A$  be a binary matrix such that  $A_{ij} = 1(X_{ij} \neq 0)$ . We have
$$
- \log \mathbb {P} (X, Y \mid z) = \sum_ {i} \left\{\frac {1}{2} \sum_ {j \neq i} \left[ d _ {\mathrm {K L}} \left(A _ {i j}, p _ {z _ {i} z _ {j}}\right) + A _ {i j} d _ {\psi^ {*}} \left(X _ {i j}, \mu_ {z _ {i} z _ {j}}\right) \right] + d _ {\phi^ {*}} \left(Y _ {i}, \nu_ {z _ {i}}\right) \right\} + c,
$$
where the additional term  $c$  is a function of  $X, Y$  but does not depend on  $z$ . Denoting  $Z \in \{0, 1\}^{n \times K}$  the one-hot membership matrix such that  $Z_{ik} = 1 (z_i = k)$ , observe that  $p_{z_i z_j} = (ZpZ^T)_{ij}$  where  $p$  is a symmetric matrix with the interaction probabilities between different blocks,  $\mu_{z_i z_j} = (Z\mu Z^T)_{ij}$  where  $\mu$  is a symmetric matrix with the expected value of the interaction between different blocks (edge weights), and  $\nu_{z_i} = (Z^T\nu)_i$  where  $\nu$  is a vector with the expected value of the attribute for different blocks. Thus, up to some additional constants, the negative log-likelihood  $-\log \mathbb{P}(X,Y|Z)$  is equal to
$$
\sum_ {i} \left\{\frac {1}{2} \mathrm {d} _ {\mathrm {K L}} \left(A _ {i \cdot}, \left(Z p Z ^ {T}\right) _ {i \cdot}\right) + \frac {1}{2} d _ {\psi^ {*}} ^ {\prime} \left(X _ {i \cdot}, \left(Z \mu Z ^ {T}\right) _ {i \cdot}\right) + d _ {\phi^ {*}} \left(Y _ {i}, \left(Z ^ {T} \nu\right) _ {i}\right) \right\} + c, \tag {4.4}
$$
where  $d_{\psi^{*}}^{\prime}(B,C) = \sum_{j = 1}^{n}1(B_{j}\neq 0)d_{\psi^{*}}(B_{j},C_{j})$  for two vectors  $B,C\in \mathbb{R}^n$
4.3 Clustering by iterative likelihood maximisation
Following the log-likelihood expression derived in (4.4), we propose an iterative clustering algorithm that places each node in the block maximising  $\mathbb{P}\left(X,Y\mid z_{-i},z_i = a\right)$  for  $1\leq a\leq K$ , the likelihood that node  $i$  is in community  $a$  given the community labels of the other nodes,  $z_{-i}$ . Let  $Z^{(ia)}$  denote the membership matrix obtained from  $Z$  by placing node  $i$  in block  $a$ , and let  $L_{ia}(Z^{(ia)})$  denote the contribution of node  $i$  to the negative log-likelihood when node  $i$  is placed in block  $a$ . Equation (4.4) shows that
$$
L _ {i a} (Z) = \frac {1}{2} \mathrm {d} _ {\mathrm {K L}} \left(A _ {i \cdot}, \left(Z p Z ^ {T}\right) _ {i \cdot}\right) + \frac {1}{2} d _ {\psi^ {*}} ^ {\prime} \left(X _ {i \cdot}, \left(Z \mu Z ^ {T}\right) _ {i \cdot}\right) + d _ {\phi^ {*}} \left(Y _ {i}, \left(Z ^ {T} \nu\right) _ {i}\right), \tag {4.5}
$$
where the  $p$ ,  $\mu$  and  $\nu$  in the equation above must be estimated from  $X$ ,  $Y$ , and the community membership matrix  $Z$ . Let  $\hat{p} = \hat{p}(A,Z)$ ,  $\hat{\mu} = \hat{\mu}(X,Z)$ , and  $\hat{\nu} = \hat{\nu}(Y,Z)$  denote the estimators for  $p$ ,  $\mu$  and  $\nu$ , respectively. Their values can be computed as follows:
$$
\hat {p} (A, Z) = \left(Z ^ {T} Z\right) ^ {- 1} Z ^ {T} A Z \left(Z ^ {T} Z\right) ^ {- 1},
$$
$$
\hat {\mu} (X, Z) = \left(Z ^ {T} A Z\right) ^ {- 1} Z ^ {T} X Z, \tag {4.6}
$$
$$
\hat {\nu} (Y, Z) = \left(Z ^ {T} Z\right) ^ {- 1} Z ^ {T} Y.
$$
Note that the matrix inverse  $\left(Z^T Z\right)^{-1}$  can be easily computed since  $Z^T Z$  is a  $K$ -by- $K$  diagonal matrix. This approach is described in Algorithm 1.
Algorithm 1: Bregman hard clustering for node-attributed SBM.
Input: Interactions  $X\in \mathcal{X}^{n\times n}$  , attributes  $Y\in \mathcal{V}^n$  , convex functions  $\psi^{*},\phi^{*}$  , clustering  $Z_{0}$    
1 Let  $Z = Z_{0}$    
2 repeat   
3 Compute  $\hat{p},\hat{\mu},\hat{\nu}$  according to (4.6)   
4 Let  $Z^{\mathrm{new}} = 0_{n\times K}$    
5 for  $i = 1,\dots ,n$  do   
6 Let  $Z^{(ia)}$  be the membership matrix obtained from  $Z$  by placing node  $i$  in community a   
7 Find  $k^{*} = \arg \max L_{ia}\left(Z^{(ia)}\right)$  , where  $L_{ia}\left(Z^{(ia)}\right)$  is defined in (4.5);   
8 Let  $Z_{ik}^{\mathrm{new}} = 1(k = k^{*})$  for all  $k = 1,\ldots ,K$    
9 Let  $Z = Z^{\mathrm{new}}$    
10 until convergence;   
Return: Node-membership matrix  $Z$
A fundamental aspect of many likelihood maximization iterative algorithms such as Algorithm 1 is the initial membership assignment,  $Z_{0}$ . This initial assignment often has a profound influence on the final membership assignment, and thus, it is important to have an adequate initialization. In the numerical section, we proceed as follows. We construct the matrix  $W \in \mathbb{R}^{n \times 2K}$  such that the first  $K$  columns of  $W$  are the first  $K$  eigenvectors of the graph normalised Laplacian, while the last  $K$  columns of  $W$  are the first  $K$  eigenvectors of the Gram matrix  $YY^{T}$ .
5.1 Performance of Algorithm 1
We first compare in Figure 1 the performance of Algorithm 1 in terms of exact recovery (fraction of times the algorithm correctly recovers the community of all nodes) with the theoretical threshold for exact recovery proved in the paper (red curve in the plots) in two settings: Figure 1a shows binary weight with Gaussian attributes, and Figure 1b shows zero-inflated Gaussian weights with Gaussian attributes. A solid black (resp., white) square means that over 50 trials, the algorithms failed 50 times (resp., succeeded 50 times) at exactly recovering the block structure.
5.2 Comparison with other algorithms
In this section, we compare Algorithm 1 with other algorithms presented in the literature. We used the Adjusted Rand Index (ARI) [21] between the predicted clusters and the ground truth ones to evaluate the performance of each algorithm.
In Figure 2, we compare Algorithm 1 with the variational-EM algorithm of [24] and the algorithm of [23] (which is also based on Bregman divergences, but tailored for dense networks). Because both of these algorithms are designed for dense networks, we observe that Algorithm 1 has overall better performance on sparse networks.
We also compare Algorithm 1 with the  $IR\_sLs$  algorithm from [8]. This is one of the most recent algorithms for node-attributed SBM and it comes with theoretical guarantees (for binary networks
(a) Binary weights with Gaussian attributes
(b) zero-inflated Gaussian weights with Gaussian attributes.
Figure 1: Phase transition of exact recovery. Each pixel represents the empirical probability that Algorithm 1 succeeds at exactly recovering the clusters (over 50 runs), and the red curve shows the theoretical threshold.
(a)  $n = 500$ ,  $K = 2$ ,  $f_{\mathrm{in}} = \mathrm{Ber}(\alpha n^{-1}\log n)$ ,  $f_{\mathrm{out}} = \mathrm{Ber}(n^{-1}\log n)$ . The attributes are 2d-spherical Gaussian with radius  $(\pm r\sqrt{\log n},0)$  and identity covariance matrix.
(b)  $n = 600$ ,  $K = 3$ ,  $f_{\mathrm{in}} = (1 - \rho)\delta_0 + \rho \mathrm{Nor}(\mu, 1)$ ,  $f_{\mathrm{out}} = (1 - \rho)\delta_0 + \rho \mathrm{Nor}(0, 1)$  with  $\rho = 5n^{-1}\log n$ . The attributes are 2d-spherical Gaussian whose means are the vertices of a regular polygon on the circle of radius  $r\sqrt{\log n}$ .
Figure 2: Comparison of Algorithm 1 with algorithms of [23] and [24]. Error bars show the standard deviations over 25 realisations. Attributes are 2d-spherical Gaussian attributes with radius  $(\pm 1,0)$ .
(a)  $n = 100$ ,  $K = 2$ ,  $f_{\mathrm{in}} = (1 - p_{\mathrm{in}})\delta_0 + p_{\mathrm{in}} \operatorname{Poi}(\tilde{5})$ ,  $f_{\mathrm{out}} = (1 - 0.03)\delta_0 + 0.03 \operatorname{Poi}(1)$ .
(b)  $n = 100$ ,  $K = 2$ ,  $f_{\mathrm{in}} = (1 - 0.07)\delta_0 + 0.07 \, \mathrm{Poi}(\mu_{\mathrm{in}})$ ,  $f_{\mathrm{out}} = (1 - 0.04)\delta_0 + 0.04 \, \mathrm{Poi}(1)$ .
with Gaussian attributes). We also compare with the EM algorithm of [34], attSBM, which is designed for binary networks with Gaussian attributes. Finally, we compare with baseline methods for clustering using the network or the attributes alone. EM-GMM refers to fitting a Gaussian Mixture Model via EM on attribute data  $Y$ , and sc refers to spectral clustering on network data  $X$ .
Figure 3 shows the results for binary networks with Gaussian attributes. Algorithm 1 successfully learns from both the signal coming from the network and the attributes, even in scenarios where one of them is non-informative. Moreover, Algorithm 1 has better performance than the two other node-attributed clustering algorithms, and those algorithms also show a large variance<sup>3</sup>. We also note that  $IR\_sLs$  and  $attSBM$  are both tailor-made for binary edges and Gaussian attributes. Even in such a setting, Algorithm 1 outperforms these two algorithms. We show in the supplement material that when the network is weighted and the attributes non-Gaussian,  $IR\_sLs$  and  $attSBM$  perform poorly.
5.3 Evaluation using real datasets
The following three benchmark datasets were used to evaluate and compare the proposed algorithm: CiteSeer ( $n = 3279$ ,  $m = 9104$ ,  $K = 6$ ,  $d = 3703$ ), Cora ( $n = 2708$ ,  $m = 10556$ ,  $K = 7$ ,
<sup>3</sup>As the large variance makes the figures less readable, we provide all results in the supplemental material.
(a) Varying edge distribution.
(b) Varying attribute distributions.
Figure 3: Performance on binary networks with Gaussian attributes. We take  $n = 600$ ,  $K = 2$ ,  $f_{\mathrm{in}} = \mathrm{Ber}(an^{-1}\log n)$ ,  $f_{\mathrm{out}} = \mathrm{Ber}(5n^{-1}\log n)$ , and 2-dimensional Gaussian attributes with unit variance and mean ( $\pm r, 0$ ). Results are averaged over 60 runs.
$d = 1433$ ), and Cornell ( $n = 183$ ,  $m = 298$ ,  $K = 5$ ,  $d = 1703$ ) (all available in Pytorch Geometric). For each network, the original node attribute vector was reduced to have dimension  $d = 10$  by selecting the 10 best features according to the chi-square test. Algorithm 1 assumed a multivariate Gaussian distribution with  $d = 10$  for node attributes and Bernoulli edges (these networks have no edge weights). The initialization for Algorithm 1 and attSBM used spectral clustering of both the node similarity matrix (using node attributes) and network edges.
Dataset	CiteSeer	Cora	Cornell
Algorithm 1	0.20	0.12	0.49
attSBM	0.17	0.09	0.46
EM-GMM	0.13	0.06	0.37
sc	0.00	0.00	0.02
Table 1: Average ARI results (over independent runs) for the three benchmark datasets.
Table 1 shows that Algorithm 1 outperformed the other three algorithms. Spectral clustering  $(sc)$  has near zero performance, indicating that the network structure in these data sets has little information concerning the clusters of the nodes. Moreover, both Algorithm 1 and attSBM (that leverage network and node attributes) outperform EM-GMM that use only node attributes. These preliminary results indicate that Algorithm 1 is promising even in real data sets with little pre-processing.
This work made the following contributions to community detection in node-attributed networks: i) extended the known thresholds for exact recovery in binary SBM to non-binary (weighted) networks with node attributes, providing a clean expression for a new information-theoretic quantity, known in the binary setting as the Chernoff-Hellinger divergence; ii) proposed an iterative algorithm based on the likelihood function that can infer community memberships from a problem instance. The algorithm leverages the framework of Bregman divergences and is simple and computationally efficient. Numerical experiments indicate the superiority of this algorithm when compared to recent state-of-the-art approaches.
The first author would like to thank Lasse Leskelä for helpful discussions and comments.
This work has been partially funded by the Brazilian-Swiss Joint Research Program (grant IZBRZ2_186313), the Brazilian National Council for Scientific and Technological Development (CNPq), and the Carlos Chagas Filho Research Foundation of the State of Rio de Janeiro (FAPERJ).
[23] Bo Long, Zhongfei Mark Zhang, and Philip S Yu. A probabilistic framework for relational clustering. In ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD), pages 470-479, 2007.
[24] Mahendra Mariadassou, Stéphane Robin, and Corinne Vacher. Uncovering latent structure in valued graphs: A variational approach. The Annals of Applied Statistics, 4(2):715-742, 2010.
[25] Angelo Mele, Lingxin Hao, Joshua Cape, and Carey E Priebe. Spectral inference for large stochastic blockmodels with nodal covariates. arXiv preprint arXiv:1908.06438, 2019.
[26] Mohamed Ndaoud. Sharp optimal recovery in the two component Gaussian mixture model. The Annals of Statistics, 50(4):2096-2126, 2022.
[27] Mark EJ Newman and Aaron Clauset. Structure and inference in annotated networks. Nature communications, 7(1):11863, 2016.
[28] Frank Nielsen. Chernoff information of exponential families. arXiv preprint arXiv:1102.2684, 2011.
[29] Subhadeep Paul and Yuguo Chen. Consistent community detection in multi-relational data through restricted multi-layer stochastic blockmodel. Electronic Journal of Statistics, 10(2):3807 - 3870, 2016.
[30] Detlef Plachky and Joseph Steinebach. A theorem about probabilities of large deviations with an application to queuing theory. Periodica Mathematica Hungarica, 6(4):343-345, 1975.
[31] Hussein Saad and Aria Nosratinia. Community detection with side information: Exact recovery under the stochastic block model. IEEE Journal of Selected Topics in Signal Processing, 12(5):944-958, 2018.
[32] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI magazine, 29(3):93-93, 2008.
[33] Laura M Smith, Linhong Zhu, Kristina Lerman, and Allon G Percus. Partitioning networks with node attributes by compressing information flow. ACM Transactions on Knowledge Discovery from Data (TKDD), 11(2):1-26, 2016.
[34] Natalie Stanley, Thomas Bonacci, Roland Kwitt, Marc Niethammer, and Peter J Mucha. Stochastic block models with multiple continuous attributes. Applied Network Science, 4(1):1-22, 2019.
[35] Tim Van Erven and Peter Harremos. Rényi divergence and Kullback-Leibler divergence. IEEE Transactions on Information Theory, 60(7):3797-3820, 2014.
[36] Min Xu, Varun Jog, and Po-Ling Loh. Optimal rates for community estimation in the weighted stochastic block model. The Annals of Statistics, 48(1):183 - 204, 2020.
[37] Bowei Yan and Purnamrita Sarkar. Covariate regularized community detection in sparse graphs. Journal of the American Statistical Association, 116(534):734-745, 2021.
[38] Se-Young Yun and Alexandre Proutiere. Optimal cluster recovery in the labeled stochastic block model. Advances in Neural Information Processing Systems, 29, 2016.
[39] Anderson Y. Zhang and Harrison H. Zhou. Minimax rates of community detection in stochastic block models. The Annals of Statistics, 44(5):2252 - 2280, 2016.
A Establishing the exact recovery threshold
The proof of Theorem 1 is structured as follows. We start by establishing some concentration results on the block sizes in Section A.1. In Section A.2, we delve into the asymptotic analysis of log-likelihood ratios, establishing fundamental results. Building upon these findings, we prove the converse statement, demonstrating the impossibility of exact recovery below the threshold, in Section A.3. Conversely, in Section A.4, we establish the positive statement, demonstrating the possibility of exact recovery above the threshold. To complement the proof, Section A.5 presents a set of technical combinatorial lemmas.
Notations In the following, we denote by  $z \in [K]^n$  the true block-labelling vector, and by  $z' \in [K]^n$  another block-labelling vector, and the conditional probabilities are denoted by  $\mathbb{P}_z(\cdot) = \mathbb{P}(\cdot | z)$ .
A.1 Preliminaries on the block sizes
For any  $z' \in [K]^n$ , we for all  $c \in [K]$  define  $\hat{\pi}_c(z')$ , the empirical size of block  $c$ , as
$$
\hat {\pi} _ {c} (z ^ {\prime}) = \frac {| \{i \in [ n ] : z _ {i} ^ {\prime} = c \} |}{n}.
$$
For any  $0 < \xi < 1$ , we define
$$
\mathcal {Z} _ {\xi} = \left\{z \in [ K ] ^ {n} \colon (1 - \xi) \pi_ {c} \leq \hat {\pi} _ {c} (z) \leq (1 + \xi) \pi_ {c} \quad \forall c \in [ K ] \right\}. \tag {A.1}
$$
Recalling that the true block-labelling vector  $z$  is sampled from  $\pi^{\otimes n}$ , we have by concentration of multinomial distributions (see for example [5, Lemma A1])
$$
\mathbb {P} \left(\mathcal {Z} _ {\xi}\right) \geq 1 - 2 K \exp \left(- \frac {\xi^ {2}}{3} n \min  _ {c \in [ K ]} \pi_ {c}\right).
$$
In the following, we chose  $\xi = \frac{\log\log n}{\sqrt{n}}$ , and hence  $\mathbb{P}(\mathcal{Z}_{\xi}) = 1 - o(1)$ .
A.2 Asymptotic study of log-likelihood ratios
This section studies the asymptotic behaviour of  $\mathbb{P}_z(L(z) \geq L(z'))$ , where  $L(z') = \mathbb{P}_{z'}(X, Y)$  the likelihood of the block-labelling vector  $z'$  given the observed data  $X, Y$ .
A.2.1 Upper-bound
The following lemma provides an upper bound on  $\mathbb{P}_z(L(z) \geq L(z'))$ .
Lemma 3. Let  $z, z' \in \mathcal{Z}$  be two block labelling vectors such that  $\mathrm{Ham}(z, z') = m \geq 1$ , and let  $L(z) = \mathbb{P}(X, Y \mid z)$  be the likelihood of labelling  $z$ . We have
$$
\mathbb {P} _ {z} \left(L (z ^ {\prime}) \geq L (z)\right) \leq e ^ {- (1 + o (1)) \left(1 - \frac {m}{n \pi_ {\operatorname* {m i n}}}\right) m n I}.
$$
Moreover, let  $d = \sup_{t\in (0,1)}\min_{\substack{a\neq b\in [K]\\ c\in [K]}}(1 - t)\mathrm{D}_t(f_{ac}\| f_{bc})$ . Then we also have
$$
\mathbb {P} _ {z} \left(L \left(z ^ {\prime}\right) \geq L (z)\right) \leq e ^ {- m (n - m) d}.
$$
Proof. Using Chernoff's bound, we have for all  $t \geq 0$ ,
$$
\mathbb {P} _ {z} \left(\log \frac {\mathbb {P} _ {z ^ {\prime}} (X , Y)}{\mathbb {P} _ {z} (X , Y)} > \epsilon\right) \leq e ^ {- t \epsilon} \mathbb {E} _ {z} \left[ e ^ {t \log \frac {\mathbb {P} _ {z ^ {\prime}} (X , Y)}{\mathbb {P} _ {z} (X , Y)}} \right] = e ^ {- t \epsilon - (1 - t) \mathrm {D} _ {t} \left(P _ {z}\right) \left\| P _ {z ^ {\prime}}\right)}, \tag {A.2}
$$
where the probability measure  $\mathbb{P}_z$  is defined on  $\mathcal{X}\times \mathcal{Y}$  by (3.1) in the main text. The linearity of the Renyi divergence with respect to product distributions implies
$$
\begin{array}{l} \mathrm {D} _ {t} \left(P _ {z} \| P _ {z ^ {\prime}}\right) = \frac {1}{2} \sum_ {i \neq j} \mathrm {D} _ {t} \left(f _ {z _ {i} z _ {j}} \| f _ {z _ {i} ^ {\prime} z _ {j} ^ {\prime}}\right) + \sum_ {1 \leq i \leq n} \mathrm {D} _ {t} \left(h _ {z _ {i}} \| h _ {z _ {i} ^ {\prime}}\right) \\ = \frac {1}{2} \sum_ {1 \leq a, b, c, d \leq K} M _ {(a b) (c d)} \mathrm {D} _ {t} \left(f _ {a c} \| f _ {b d}\right) + \sum_ {1 \leq a, b \leq K} N _ {a b} \mathrm {D} _ {t} \left(h _ {a} \| h _ {b}\right), \tag {A.3} \\ \end{array}
$$
$$
N _ {a b} = | \{i \in [ n ] \colon (z _ {i}, z _ {i} ^ {\prime}) = (a, b) \} |,
$$
$$
M _ {(a b) (c d)} = \left| \left\{i \neq j: \left(z _ {i}, z _ {i} ^ {\prime}\right) = (a, b) \text {a n d} \left(z _ {j}, z _ {j} ^ {\prime}\right) = (c, d) \right\} \right|.
$$
Moreover, Lemma 6 ensures that
$$
M _ {(a b) (c d)} = N _ {a b} \left(N _ {c d} - 1 _ {(a b) = (c d)}\right) = N _ {c d} \left(N _ {a b} - 1 _ {(a b) = (c d)}\right).
$$
$$
\begin{array}{l} \sum_ {1 \leq a, b, c, d \leq K} M _ {(a b) (c d)} \mathrm {D} _ {t} \left(f _ {a c} \| f _ {b d}\right) \geq \sum_ {a \neq b} \sum_ {c} M _ {(a b) (c c)} \mathrm {D} _ {t} \left(f _ {a c} \| f _ {b c}\right) + \sum_ {a} \sum_ {c \neq d} M _ {(a a) (c d)} \mathrm {D} _ {t} \left(f _ {a c} \| f _ {a d}\right) \\ = 2 \sum_ {a \neq b} N _ {a b} \sum_ {c} N _ {c c} \mathrm {D} _ {t} \left(f _ {a c} \| f _ {b c}\right). \tag {A.4} \\ \end{array}
$$
Using  $N_{cc} = \left|z^{-1}(c)\right| - \sum_{d\neq c}N_{cd}$  and that  $\sum_{c}\sum_{d\neq c}N_{cd} = m$ , we obtain that
$$
\begin{array}{l} \frac {1}{2} \sum_ {1 \leq a, b, c, d \leq K} M _ {(a b) (c d)} \mathrm {D} _ {t} \left(f _ {a c} \| f _ {b d}\right) \geq n \sum_ {a \neq b} N _ {a b} \sum_ {c} \hat {\pi} _ {c} (z) \left(1 - \frac {\sum_ {d \neq c} N _ {c d}}{n \hat {\pi} _ {c} (z)}\right) \mathrm {D} _ {t} \left(f _ {a c} \| f _ {b c}\right) \\ \geq n \left(1 - \frac {m}{n \operatorname* {m i n} _ {c} \hat {\pi} _ {c} (z)}\right) \sum_ {a \neq b} N _ {a b} \sum_ {c} \hat {\pi} _ {c} (z) \mathrm {D} _ {t} \left(f _ {a c} \| f _ {b c}\right). \\ \end{array}
$$