SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Parts i. and ii. can be easily verified numerically. To show Part iii., take the derivative with respect to  $x$ . This derivative, i.e.,  $\log \left(\frac{2 - x - x'}{x + x'}\right) - \log \left(\frac{1 - x}{x}\right)$ , is always non-negative for  $1 > x > x'$ . For Part iv, the second partial derivative of  $h_D(x,x')$  with respect to  $x$  is equal to  $\frac{1}{x(1 - x)} - \frac{2}{(x + x')(2 - x - x')}$  which is always positive for  $0 \leqslant x,x' \leqslant 1$ . Finally, we show the convexity with respect to both variables  $x$  and  $x'$  simultaneously. The Hessian of the function  $h_D(x,x')$  equals
$$
H = \left[ \begin{array}{c c} \frac {1}{x (1 - x)} - \frac {2}{(x + x ^ {\prime}) (2 - x - x ^ {\prime})} & - \frac {2}{(x + x ^ {\prime}) (2 - x - x ^ {\prime})} \\ - \frac {2}{(x + x ^ {\prime}) (2 - x - x ^ {\prime})} & \frac {1}{x ^ {\prime} (1 - x ^ {\prime})} - \frac {2}{(x + x ^ {\prime}) (2 - x - x ^ {\prime})} \end{array} \right]. \tag {24}
$$
We show the eigenvalues of this symmetric matrix is always non-negative, and hence  $H$  is positive semi-definite. This completes the proof.
$$
\left| \lambda I _ {2} - H \right| = \left| \left[ \begin{array}{c c} \lambda - \frac {1}{x (1 - x)} + \frac {2}{(x + x ^ {\prime}) (2 - x - x ^ {\prime})} & \frac {2}{(x + x ^ {\prime}) (2 - x - x ^ {\prime})} \\ \frac {2}{(x + x ^ {\prime}) (2 - x - x ^ {\prime})} & \lambda - \frac {1}{x ^ {\prime} (1 - x ^ {\prime})} + \frac {2}{(x + x ^ {\prime}) (2 - x - x ^ {\prime})} \end{array} \right] \right| = 0, \tag {25}
$$
reduces to solving
$$
\left(\lambda + a\right) ^ {2} - \left(\frac {1}{x (1 - x)} + \frac {1}{x ^ {\prime} \left(1 - x ^ {\prime}\right)}\right) (\lambda + a) + \frac {1}{x (1 - x) x ^ {\prime} \left(1 - x ^ {\prime}\right)} - a ^ {2} = 0, \tag {26}
$$
where  $a \coloneqq \frac{2}{(x + x') (2 - x - x')}$ . The roots of this equation are
$$
\lambda = \frac {\left(\frac {1}{x (1 - x)} + \frac {1}{x ^ {\prime} \left(1 - x ^ {\prime}\right)}\right) \pm \sqrt {\left(\frac {1}{x (1 - x)} - \frac {1}{x ^ {\prime} \left(1 - x ^ {\prime}\right)}\right) ^ {2} + 4 a ^ {2}}}{2} - a. \tag {27}
$$
It is straightforward to verify that both roots are always non-negative, which completes the proof.
E.4 Proof of Theorem 3
$$
\inf  _ {\mathbf {Q} \in \mathcal {Q} _ {i i}} \mathbb {E} _ {\mathbf {Y}, \mathbf {Y} ^ {\prime}} \left[ D _ {K L} \left(\mathbb {E} _ {\mathbf {X} ^ {\prime}, \mathbf {X}, W} \left[ P _ {\hat {Y} | X, W} ^ {\otimes 2 n} \left(\hat {\mathbf {Y}}, \hat {\mathbf {Y}} ^ {\prime} \mid \mathbf {X}, \mathbf {X} ^ {\prime}, W\right) \right] \| \mathbf {Q}\right) \right] = I (\mathbf {T}; \hat {\mathbf {Y}} ^ {2 n} | Y ^ {2 n}). \tag {28}
$$
Let  $\mathcal{Q}_{ii}^{\prime}$ , be the set of conditional priors  $\mathbf{Q}'$  that can be written as
$$
\mathbf {Q} ^ {\prime} \left(\hat {\mathbf {Y}}, \hat {\mathbf {Y}} ^ {\prime} \mid \mathbf {Y}, \mathbf {Y} ^ {\prime}\right) = \mathbb {E} _ {\mathbf {T}} \left[ \mathbf {Q} _ {1} ^ {\prime} \left(\hat {Y} _ {\mathbf {T}} ^ {2 n}, \hat {Y} _ {\mathbf {T} ^ {c}} ^ {2 n} \mid Y _ {\mathbf {T}} ^ {2 n}, Y _ {\mathbf {T} ^ {c}} ^ {2 n}\right) \right], \tag {29}
$$
for some arbitrary (and not necessarily symmetric distribution  $\mathbf{Q}_1'$ . Here  $Y^{2n}$  and  $\hat{Y}^{2n}$  are the concatenations of the vectors  $(\mathbf{Y},\mathbf{Y}')$  and  $(\hat{\mathbf{Y}},\hat{\mathbf{Y}}')$ , respectively. It is easy to verify that  $\mathcal{Q}_{ii} = \mathcal{Q}_{ii}'$ . Hence, by denoting
$P(\hat{\mathbf{Y}},\hat{\mathbf{Y}}^{\prime}|\mathbf{Y},\mathbf{Y}^{\prime}):=\mathbb{E}_{\mathbf{X}^{\prime},\mathbf{X},W}\left[P_{\hat{Y}|X,W}^{\otimes 2n}(\hat{\mathbf{Y}},\hat{\mathbf{Y}}^{\prime}|\mathbf{X},\mathbf{X}^{\prime},W)\right]$  we can write
$$
\begin{array}{l} \mathrm {L H S} = \inf  _ {\mathbf {Q} \in \mathcal {Q} _ {i i}} \mathbb {E} _ {\mathbf {Y}, \mathbf {Y} ^ {\prime}} \left[ D _ {K L} \left(P (\hat {\mathbf {Y}}, \hat {\mathbf {Y}} ^ {\prime} | \mathbf {Y}, \mathbf {Y} ^ {\prime}) \| \mathbf {Q}\right) \right] \\ = \inf  _ {\mathbf {Q} ^ {\prime} \in \mathcal {Q} _ {i i} ^ {\prime}} \mathbb {E} _ {\mathbf {Y}, \mathbf {Y} ^ {\prime}} \left[ D _ {K L} \left(P (\hat {\mathbf {Y}}, \hat {\mathbf {Y}} ^ {\prime} | \mathbf {Y}, \mathbf {Y} ^ {\prime}) \| \mathbf {Q} ^ {\prime}\right) \right] \\ = \inf  _ {\mathbf {Q} ^ {\prime} \in \mathcal {Q} _ {i i} ^ {\prime}} \mathbb {E} _ {\mathbf {Y} ^ {2 n}} \mathbb {E} _ {\mathbf {T}} \left[ \right. D _ {K L} \left( \right.P \left(\hat {Y} _ {\mathbf {T}} ^ {2 n}, \hat {Y} _ {\mathbf {T} ^ {c}} ^ {2 n} \mid Y _ {\mathbf {T}} ^ {2 n}, Y _ {\mathbf {T} ^ {c}} ^ {2 n}\right)\left\| \right. \mathbf {Q} ^ {\prime}\left. \right)\left. \right] \\ = \inf  _ {\mathbf {Q} _ {1} ^ {\prime}} \mathbb {E} _ {Y ^ {2 n}} \mathbb {E} _ {\mathbf {T}} \left[ D _ {K L} \left(P \left(\hat {Y} _ {\mathbf {T}} ^ {2 n}, \hat {Y} _ {\mathbf {T} ^ {c}} ^ {2 n} \mid Y _ {\mathbf {T}} ^ {2 n}, Y _ {\mathbf {T} ^ {c}} ^ {2 n}\right) \Big \| \mathbb {E} _ {\mathbf {T}} \left[ \mathbf {Q} _ {1} ^ {\prime} \left(\hat {Y} _ {\mathbf {T}} ^ {2 n}, \hat {Y} _ {\mathbf {T} ^ {c}} ^ {2 n} \mid Y _ {\mathbf {T}} ^ {2 n}, Y _ {\mathbf {T} ^ {c}} ^ {2 n}\right) \right]\right) \right] \\ = I \left(\mathbf {T}; \hat {Y} ^ {2 n} \mid Y ^ {2 n}\right). \\ \end{array}
$$
This completes the proof of showing the equality in (28).
Now we proceed to show the upper bound on  $nh_{D}\Big(\mathbb{E}_{W}[\mathcal{L}(W)],\mathbb{E}_{S,W}\Big[\hat{\mathcal{L}} (S,W)\Big]\Big)$ . Let  $\mathbf{Q}$  be any fixed type-II symmetric conditional prior on  $(\hat{Y}^n,\hat{Y}^{\prime n})$  given  $(Y^n,Y^{\prime n})$ . We show that for  $n\geqslant 10$
$$
\begin{array}{l} \left. n h _ {D} \left(\mathbb {E} _ {W} [ \mathcal {L} (W) ], \mathbb {E} _ {S, W} \left[ \hat {\mathcal {L}} (S, W) \right]\right)\right) \\ \leqslant \mathbb {E} _ {Y ^ {n}, Y ^ {\prime m}} \left[ D _ {K L} \left(\mathbb {E} _ {S ^ {\prime}, S, W} \left[ P _ {\hat {Y} | X, W} ^ {\otimes 2 n} \left(\hat {Y} ^ {n}, \hat {Y} ^ {\prime m} \mid X ^ {n}, X ^ {\prime m}, W\right) \right] \| \mathbf {Q}\right) \right] + \log (n), \\ \end{array}
$$
where  $Y^n, Y'^n \sim \mu_Y^{\otimes 2n}$  and  $S', S, W \sim P_{S'|Y'^n}P_{S,W|Y^n}$ .
For ease of notation, denote
$$
P _ {\hat {Y} ^ {n}, \hat {Y} ^ {\prime n} | Y ^ {n}, Y ^ {\prime n}} := \mathbb {E} _ {S ^ {\prime}, S, W} \Big [ P _ {\hat {Y} | X, W} ^ {\otimes 2 n} (\hat {Y} ^ {n}, \hat {Y} ^ {\prime n} | X ^ {n}, X ^ {\prime n}, W) \Big ],
$$
where  $S^{\prime}, S, W \sim P_{S^{\prime}|Y^{\prime n}}P_{S,W|Y^{n}}$ . We have
$$
\begin{array}{l} \left. n h _ {D} \left(\mathbb {E} _ {W} [ \mathcal {L} (W) ], \mathbb {E} _ {S, W} \left[ \hat {\mathcal {L}} (S, W) \right]\right)\right) \\ \leqslant n \mathbb {E} _ {P _ {S ^ {\prime}} P _ {S, W}} \left[ h _ {D} \left(\hat {\mathcal {L}} \left(S ^ {\prime}, W\right), \hat {\mathcal {L}} (S, W)\right) \right] \\ = n \mathbb {E} _ {S ^ {\prime}, S, W} \left[ h _ {D} \left(\mathbb {E} _ {P _ {\hat {Y} | X, W} ^ {\otimes n} (\hat {Y} ^ {\prime n} | X ^ {\prime n}, W)} \left[ \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {1} _ {\{Y _ {i} ^ {\prime} \neq \hat {Y} _ {i} ^ {\prime} \}} \right], \mathbb {E} _ {P _ {\hat {Y} | X, W} ^ {\otimes n} (\hat {Y} ^ {n} | X ^ {n}, W)} \left[ \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {1} _ {\{Y _ {i} \neq \hat {Y} _ {i} \}} \right]\right) \right] \\ \leqslant n \mathbb {E} _ {P _ {S ^ {\prime}} P _ {S, W} P _ {\hat {Y} | X, W} ^ {\otimes 2 ^ {n}} (\hat {Y} ^ {n}, \hat {Y} ^ {\prime m} | X ^ {n}, X ^ {\prime m}, W)} \left[ h _ {D} \left(\frac {1}{n} \sum_ {i \in [ n ]} \mathbb {1} _ {\{Y _ {i} ^ {\prime} \neq \hat {Y} _ {i} ^ {\prime} \}}, \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {1} _ {\{Y _ {i} \neq \hat {Y} _ {i} \}}\right) \right] \\ \leqslant D _ {K L} \left(\mu_ {Y} ^ {\otimes 2 n} P _ {\hat {Y} ^ {n}, \hat {Y} ^ {\prime n} | Y ^ {n}, Y ^ {\prime n}} \left\| \mu_ {Y} ^ {\otimes 2 n} \mathbf {Q} \left(\hat {Y} ^ {n}, \hat {Y} ^ {\prime n} | Y ^ {n}, Y ^ {\prime n}\right)\right)\right) \\ + \log \mathbb {E} _ {\mathbf {Y} ^ {n}, Y ^ {\prime n}, \hat {Y} ^ {n}, \hat {Y} ^ {\prime n} \sim \mu_ {Y} ^ {\otimes 2 n}} \mathbf {Q} \left(\hat {Y} ^ {n}, \hat {Y} _ {i} ^ {\prime n} \mid Y ^ {n}, Y ^ {\prime n}\right) \left[ e ^ {n h _ {D} \left(\frac {1}{n} \sum_ {i \in [ n ]} \mathbb {1} _ {\{Y _ {i} ^ {\prime} \neq \hat {Y} _ {i} ^ {\prime} \}}, \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {1} _ {\{Y _ {i} \neq \hat {Y} _ {i} \}}\right)} \right]. \tag {30} \\ \end{array}
$$
Now, we compute the last term, which does not depend on  $W$  anymore. Suppose that  $\mathrm{Unif}(2n)$  is a distribution that picks uniformly  $n$  indices among indices  $2n$  indices, i.e., the probability of each one is  $\frac{1}{\binom{2n}{n}}$ . We denote such indices by  $\mathbf{T} = (T_1,\ldots ,T_n)$ , and the corresponding distribution by  $\mathrm{Unif}(2n)$ . For a vector  $Y^{2n}$  of length  $2n$ , we denote the elements corresponding to  $n$  indices picked by  $\mathbf{T}$  as  $Y_{\mathbf{T}}^{2n} = (Y_{T_1}^{2n},\dots,Y_{T_n}^{2n})$ . We denote by  $\mathbf{T}^c = (T_1^c,\dots,T_n^c)$  the other remaining  $n$  elements in  $1,\dots,2n$  that are not picked by  $\mathbf{T}$ . We denote  $\mathfrak{V}^{T^{n,c}} = (\mathfrak{V}_{T_1^c},\dots,\mathfrak{V}_{T_n^c})$ . Then,
$$
\begin{array}{l} \log \mathbb {E} _ {Y ^ {n}, Y ^ {\prime n}, \hat {Y} ^ {n}, \hat {Y} ^ {\prime n} \sim \mu_ {Y} ^ {\otimes 2 n}} \mathbf {Q} (\hat {Y} ^ {n}, \hat {Y} _ {i} ^ {\prime n} | Y ^ {n}, Y ^ {\prime n}) \left[ e ^ {n h _ {D} \left(\frac {1}{n} \sum_ {i \in [ n ]} \mathbb {1} _ {\{Y _ {i} ^ {\prime} \neq \hat {Y} _ {i} ^ {\prime} \}}, \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {1} _ {\{Y _ {i} \neq \hat {Y} _ {i} \}}\right)} \right] \\ = \log \mathbb {E} _ {\mathfrak {Y} ^ {2 n}, \mathfrak {Y} ^ {2 n}, \mathbf {T}, \sim \mu^ {\otimes 2 n} \mathbf {Q} (\mathfrak {Y} ^ {2 n} | \mathfrak {Y} ^ {2 n})} \text {U n i f} (2 n) \left[ e ^ {n h _ {D} \left(\frac {1}{n} \sum_ {i \in [ n ]} \mathbb {1} _ {\{\mathfrak {V} _ {T _ {i} ^ {c}} \neq \hat {\mathfrak {V}} _ {T _ {i} ^ {c}} \}}, \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {1} _ {\{\mathfrak {V} _ {T _ {i}} \neq \hat {\mathfrak {V}} _ {T _ {i}} \}}\right)} \right]. \tag {31} \\ \end{array}
$$
Let  $V$  be a random variable indicating  $V \coloneqq \sum_{i \in [2n]} \mathbb{1}_{\{\mathfrak{Y}_i \neq \hat{\mathfrak{Y}}_i\}}$  in the sequence  $\mathfrak{Y}^{2n}$ . Then, we consider different cases for  $V$  and show that
$$
\mathbb {E} _ {\mathbf {T} \sim \operatorname {U n i f} (2 n)} \left[ e ^ {n h _ {D} \left(\frac {1}{n} \sum_ {i \in [ n ]} \mathbb {1} _ {\{\mathfrak {Y} _ {T _ {i} ^ {c}} \neq \mathfrak {Y} _ {T _ {i} ^ {c}} \}}, \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {1} _ {\{\mathfrak {Y} _ {T _ {i}} \neq \mathfrak {Y} _ {T _ {i}} \}}\right)} \right] \leqslant n, \tag {32}
$$
for  $n\geqslant 10$  . This completes the proof.
We use the following lemma repeatedly in the rest of the proof.
Lemma 2 ([Gal68, Exercise 5.8.a]). For  $j \geqslant 1$  and  $n - j \geqslant 1$ , where  $j, n \in \mathbb{N}$ ,
$$
\sqrt {\frac {n}{8 j (n - j)}} \leqslant \binom {n} {j} e ^ {- n h _ {b} (j / n)} \leqslant \sqrt {\frac {n}{2 \pi j (n - j)}}. \tag {33}
$$
i. If  $V\in [1,n]$  ..
$$
\begin{array}{l} \sum_ {j = 0} ^ {V} e ^ {n h _ {D} (j / n, (V - j) / n)} \frac {\binom {n} {j} \binom {n} {V - j}}{\binom {2 n} {V}} = \frac {2}{\binom {2 n} {V}} + \sum_ {j = 1} ^ {V - 1} e ^ {n h _ {D} (j / n, (V - j) / n)} \frac {\binom {n} {j} \binom {n} {V - j}}{\binom {2 n} {V}} \\ \stackrel {(a)} {\leqslant} \frac {2}{\binom {2 n} {V}} + \sum_ {j = 1} ^ {V - 1} \frac {\sqrt {n V (2 n - V)}}{\pi \sqrt {j (n - j) (V - j) (n - V + j)}} \\ \stackrel {(b)} {\leqslant} \frac {2}{\binom {2 n} {V}} + \frac {1}{2 \pi} \sqrt {n V (2 n - V)} \sum_ {j = 1} ^ {V - 1} \left(\frac {1}{j (n - j)} + \frac {1}{(V - j) (n - V + j)}\right) \\ = \frac {2}{\binom {2 n} {V}} + \frac {1}{\pi} \sqrt {n V (2 n - V)} \sum_ {j = 1} ^ {V - 1} \frac {1}{j (n - j)} \\ = \frac {2}{\binom {2 n} {V}} + \frac {1}{n \pi} \sqrt {n V (2 n - V)} \sum_ {j = 1} ^ {V - 1} \left(\frac {1}{j} + \frac {1}{n - j}\right) \\ \stackrel {(c)} {\leqslant} \frac {2}{\binom {2 n} {V}} + \frac {2}{n \pi} \sqrt {n ^ {3}} \sum_ {j = 1} ^ {n - 1} \frac {1}{j} \\ \leqslant \frac {2}{2 n} + \frac {2 \sqrt {n}}{\pi} (\log (n - 1) + 0. 5 8 + 1 / (2 n - 2)) \\ \begin{array}{c} (d) \\ \leqslant n, \end{array} \\ \end{array}
$$
where  $(a)$  is deduced using Lemma 2,  $(b)$  is due to inequality  $\frac{1}{\sqrt{xy}} \leqslant \frac{1}{2x} + \frac{1}{2y}$  for any  $x, y > 0$ ,  $(c)$  by the upper bound on the Harmonic series, and  $(d)$  holds for  $n \geqslant 2$ .
ii. If  $V \in [n + 1, 2n - 1]$ :
$$
\begin{array}{l} \sum_ {j = V - n} ^ {n} e ^ {n h _ {D} (j / n, (V - j) / n)} \frac {\binom {n} {j} \binom {n} {V - j}}{\binom {2 n} {V}} \\ = 2 e ^ {n h _ {D} (n / n, (V - n) / n)} \frac {\binom {n} {V - n}}{\binom {2 n} {V}} + \sum_ {j = V - n + 1} ^ {n - 1} e ^ {n h _ {D} (j / n, (V - j) / n)} \frac {\binom {n} {j} \binom {n} {V - j}}{\binom {2 n} {V}} \\ \stackrel {(a)} {\leqslant} 2 \max  \left(\max  _ {V \in [ n + 2, 2 n - 1 ]} \frac {\sqrt {2 V}}{\sqrt {\pi (V - n)}}, e ^ {n h _ {D} (1, 1 / n)} \frac {n}{\binom {2 n} {n + 1}}\right) \\ + \sum_ {j = 1} ^ {V - 1} \frac {\sqrt {n V (2 n - V)}}{\pi \sqrt {j (n - j) (V - j) (n - V + j)}} \\ \stackrel {(b)} {\leqslant} 2 \sqrt {\frac {(n + 2)}{\pi}} + \frac {1}{2 \pi} \sqrt {n V (2 n - V)} \sum_ {j = V - n + 1} ^ {n - 1} \left(\frac {1}{j (n - j)} + \frac {1}{(V - j) (n - V + j)}\right) \\ \end{array}
$$
$$
\begin{array}{l} = 2 \sqrt {\frac {(n + 2)}{\pi}} + \frac {1}{\pi} \sqrt {n V (2 n - V)} \sum_ {j = V - n + 1} ^ {n - 1} \frac {1}{j (n - j)} \\ = 2 \sqrt {\frac {(n + 2)}{\pi}} + \frac {1}{n \pi} \sqrt {n V (2 n - V)} \sum_ {j = V - n + 1} ^ {n - 1} \left(\frac {1}{j} + \frac {1}{n - j}\right) \\ \stackrel {(c)} {\leqslant} 2 \sqrt {\frac {(n + 2)}{\pi}} + \frac {2}{n \pi} \sqrt {n ^ {3}} \sum_ {j = 1} ^ {n - 1} \frac {1}{j} \\ \leqslant 2 \sqrt {\frac {(n + 2)}{\pi}} + \frac {2 \sqrt {n}}{\pi} (\log (n - 1) + 0. 5 8 + 1 / (2 n - 2)) \\ \begin{array}{c} (d) \\ \leqslant n, \end{array} \\ \end{array}
$$
where  $(a)$  is deduced using Lemma 2,  $(b)$  is due to inequality  $\frac{1}{\sqrt{xy}} \leqslant \frac{1}{2x} + \frac{1}{2y}$  for any  $x, y > 0$  and by verifying the first term for  $V = n + 1$  numerically,  $(c)$  by the upper bound on the Harmonic series, and  $(d)$  holds for  $n \geqslant 10$ .
iii. If  $V = 2n$ : In this case  $\sum_{j = V - n}^{n}e^{nh_D(j / n,(V - j) / n)}\frac{\binom{n}{j}\binom{n}{V - j}}{\binom{2n}{V}} = 1.$
E.5 Proof of Theorem 4
Let  $\mathbf{Q}$  be a conditional prior over  $U^{2n}$  given  $X^{2n}, Y^{2n}$  and  $\hat{W}_e \in \mathcal{W}_e$ , which is symmetric in the following sense:  $\mathbf{Q}\Big((u_{\pi(1)}, \ldots, u_{\pi(2n)})|(x_1, \ldots, x_{2n}),(y_1, \ldots, y_{2n}), \hat{w}_e\Big)$  remains the same for all permutations  $\pi \colon [2n] \mapsto [2n]$  that preserves the label, i.e.,  $y_{\pi(i)} = y_i$ . Then, we show that for the  $K$ -classification learning task,
$$
\mathbb {E} _ {S, W} [ \mathrm {g e n} (S, W) ] \leqslant 2 \sqrt {\frac {2 B + K + 2}{n}} + \epsilon ,
$$
$$
B := \inf  _ {S, S ^ {\prime}, \hat {W} _ {e} \sim P _ {S, \hat {W} _ {e}} P _ {S ^ {\prime}}} \left[ D _ {K L} \left(P _ {U | X, \hat {W} _ {e}} ^ {\otimes 2 n} \left(U ^ {n}, U ^ {\prime n} | X ^ {n}, X ^ {\prime n}, \hat {W} _ {e}\right) \| \mathbf {Q}\right) \right],
$$
and the infimum is over all  $P_{\hat{W}_e|S}$  such that for  $\hat{W} = (\hat{W}_e, W_d)$ ,
$$
\mathbb {E} _ {P _ {S, W} P _ {\hat {W} _ {e} | S}} \left[ \operatorname {g e n} (S, W) - \operatorname {g e n} (S, \hat {W}) \right] \leqslant \epsilon .
$$
Proof. We prove the theorem for  $\hat{W} = W$  and  $\epsilon = 0$ . The general result follows by the distortion criterion and applying the theorem on  $\operatorname{gen}(S, \hat{W})$ . We start the proof similar to the proof of Theorem 1; but with a different expansion of the probability distribution.
$$
\begin{array}{l} \lambda \mathbb {E} _ {S, W} [ \operatorname {g e n} (S, W) ] \\ = \mathbb {E} _ {S, S ^ {\prime}, W, U ^ {n}, U ^ {\prime n}, \hat {Y} ^ {n}, \hat {Y} ^ {\prime n} \sim P _ {S ^ {\prime}} P _ {S, W} \nu_ {1}} \left[ \frac {\lambda}{n} \sum_ {i \in [ n ]} \left(\mathbb {1} _ {\{Y _ {i} ^ {\prime} \neq \hat {Y} _ {i} ^ {\prime} \}} - \mathbb {1} _ {\{Y _ {i} \neq \hat {Y} _ {i} \}}\right) \right] \\ \leqslant D _ {K L} \left(P _ {S ^ {\prime}} P _ {S, W} P _ {U | X, W _ {e}} ^ {\otimes 2 n} \left(U ^ {n}, U ^ {\prime n} \mid X ^ {n}, X ^ {\prime n}, W _ {e}\right) \Big \| P _ {S ^ {\prime}} P _ {S, W} \mathbf {Q} \left(U ^ {n}, U ^ {\prime n} \mid S, S ^ {\prime}, W\right)\right) \\ + \log \mathbb {E} _ {S, S ^ {\prime}, W, U ^ {n}, U ^ {\prime n}, \hat {Y} ^ {n}, \hat {Y} ^ {\prime n} \sim P _ {S ^ {\prime}} P _ {S, W} \nu_ {2}} \left[ e ^ {\frac {\lambda}{n} \sum_ {i \in [ n ]} \left(\mathbb {1} _ {\{Y _ {i} ^ {\prime} \neq \hat {Y} _ {i} ^ {\prime} \}} - \mathbb {1} _ {\{Y _ {i} \neq \hat {Y} _ {i} \}}\right)} \right], \tag {34} \\ \end{array}
$$
$$
\begin{array}{l} \nu_ {1} := P _ {U | X, W _ {e}} ^ {\otimes 2 n} (U ^ {n}, U ^ {\prime n} | X ^ {n}, X ^ {\prime n}, W _ {e}) P _ {\hat {Y} | U, W _ {d}} ^ {\otimes 2 n} (\hat {Y} ^ {n}, \hat {Y} ^ {\prime n} | U ^ {n}, U ^ {\prime n}, W _ {d}), \\ \nu_ {2} := \mathbf {Q} (U ^ {n}, U ^ {\prime m} | S, S ^ {\prime}, W) P _ {\hat {Y} | U, W _ {d}} ^ {\otimes 2 n} (\hat {Y} ^ {n}, \hat {Y} ^ {\prime n} | U ^ {n}, U ^ {\prime n}, W _ {d}). \\ \end{array}
$$
Now, we bound the last term. Let
$$
P _ {\mathbf {Q}} \left(\hat {Y} ^ {n}, \hat {Y} ^ {\prime n} | S, S ^ {\prime}, W\right) := \mathbb {E} _ {\left(U ^ {n}, U ^ {\prime n}\right) \sim \mathbf {Q} \left(U ^ {n}, U ^ {\prime n} \mid S, S ^ {\prime}, W\right)} \left[ P _ {\hat {Y} | U, W _ {d}} ^ {\otimes 2 n} \left(\hat {Y} ^ {n}, \hat {Y} ^ {\prime n} \mid U ^ {n}, U ^ {\prime n}, W _ {d}\right) \right]. \tag {35}
$$
Then, the last term in (34) can be written as
$$
\log \mathbb {E} _ {S, S ^ {\prime}, W, \hat {Y} ^ {n}, \hat {Y} ^ {\prime n} \sim P _ {S ^ {\prime}} P _ {S, W} P _ {\mathbf {Q}} (\hat {Y} ^ {n}, \hat {Y} ^ {\prime n} | S, S ^ {\prime}, W)} \left[ e ^ {\frac {\Delta}{n} \sum_ {i \in [ n ]} \left(\mathbb {1} _ {\{Y _ {i} ^ {\prime} \neq \hat {Y} _ {1, i} \}} - \mathbb {1} _ {\{Y _ {i} \neq \hat {Y} _ {2, i} \}}\right)} \right].
$$
Note that since we have  $P_{\hat{Y} |U,W_d,X,Y} = P_{\hat{Y} |U,W_d}$ , it can be easily verified that the distribution  $P_{\mathbf{Q}}\left(\hat{Y}^n,\hat{Y}^m |S,S',W\right)$  is symmetric with respect to all permutations  $\pi$  that preserve the labels of  $Y$ .
For each  $s, s'$ , let  $f\colon [n] \to [n], i \in [n]$  and  $k\colon [n] \to [n], i \in [n]$  be permutations of indices of  $s$  and  $s'$ , respectively, where  $Y_{f_i} = Y_{k_i}'$  for  $i \leqslant T$  and  $Y_{f_i} \neq Y_{k_i}'$  for  $i > T$ , and where  $T$  equals  $n - \frac{n}{2}\|\hat{p}_s - \hat{p}_{s'}\|_1$ . Here,  $\hat{p}_s$  and  $\hat{p}_s$  are empirical distributions of  $Y$  in  $s$  and  $s'$ , respectively. We denote  $(\hat{y}^n, \hat{y}'^n)$  by  $\hat{y}^{2 \times n}$  where  $\forall i \in [n], \hat{y}_{1,i} = \hat{y}_i'$  and  $\hat{y}_{2,i} = \hat{y}_i$ .
Consider the binary random variable  $K_{i}$  taking value as  $(2, f_{i})$  or  $(1, k_{i})$  with probability  $1/2$  (independent of other  $j \neq i$ ). Denote the complementary choice as  $K_{i}^{c}$ , i.e.,  $K_{i}^{c} = (2, f_{i})$  iff  $K_{i} = (1, k_{i})$ .
Then, due to the particular symmetry of  $\mathbf{Q}$ , and by using shorthand notation
$$
\mathsf {P} := P _ {S ^ {\prime}} P _ {S, W} P _ {\mathbf {Q}} \left(\hat {Y} ^ {n}, \hat {Y} ^ {\prime n} | S, S ^ {\prime}, W\right),
$$
$$
\begin{array}{l} \mathbb {E} _ {S, S ^ {\prime}, W, \hat {Y} ^ {2 \times n} \sim \mathsf {P}} \left[ e ^ {\frac {\lambda}{n} \sum_ {i \in [ n ]} \left(\mathbb {1} _ {\{Y _ {k _ {i}} ^ {\prime} \neq \hat {Y} _ {1, k _ {i}} \}} - \mathbb {1} _ {\{Y _ {f _ {i}} \neq \hat {Y} _ {2, f _ {i}} \}}\right)} \right] \\ = \mathbb {E} _ {S, S ^ {\prime}, W, \hat {Y} ^ {2 \times n} \sim \mathsf {P}} \left[ \left\{\prod_ {i = 1} ^ {T} e ^ {\frac {\lambda}{n} \left(\mathbb {1} _ {\{Y _ {k _ {i}} ^ {\prime} \neq \hat {Y} _ {1, k _ {i}} \}} - \mathbb {1} _ {\{Y _ {f _ {i}} \neq \hat {Y} _ {2, f _ {i}} \}}\right)} \right\} \left\{\prod_ {i = T + 1} ^ {n} e ^ {\frac {\lambda}{n} \left(\mathbb {1} _ {\{Y _ {k _ {i}} ^ {\prime} \neq \hat {Y} _ {1, k _ {i}} \}} - \mathbb {1} _ {\{Y _ {f _ {i}} \neq \hat {Y} _ {2, f _ {i}} \}}\right)} \right\} \right] \\ \leqslant \mathbb {E} _ {S, S ^ {\prime}, W, \hat {Y} ^ {2 \times n} \sim \mathsf {P}} \left[ \left\{\prod_ {i = 1} ^ {T} e ^ {\frac {\lambda^ {n}}{n} \left(\mathbb {1} _ {\{Y _ {i} \neq \hat {Y} _ {1, k _ {i}} \}} - \mathbb {1} _ {\{Y _ {f _ {i}} \neq \hat {Y} _ {2, f _ {i}} \}}\right)} \right\} e ^ {\frac {\lambda (n - T)}{n}} \right] \\ = \mathbb {E} _ {S, S ^ {\prime}, W, \hat {Y} ^ {2 \times n}, K ^ {n} \sim \mathsf {P U n i f} ((2, f _ {i}), (1, k _ {i})) ^ {\otimes n}} \left[ \left\{\prod_ {i = 1} ^ {T} e ^ {\frac {\lambda}{n} \left(\mathbb {1} _ {\{Y _ {i} \neq \hat {Y} _ {K _ {i}} \}} - \mathbb {1} _ {\{Y _ {f _ {i}} \neq \hat {Y} _ {K _ {i}} ^ {c} \}}\right)} \right\} e ^ {\frac {\lambda (n - T)}{n}} \right] \\ \leqslant \mathbb {E} _ {S, S ^ {\prime}, W} \sim P _ {S ^ {\prime}} P _ {S, W} \left[ \left(\frac {e ^ {\lambda / n} + e ^ {- \lambda / n}}{2}\right) ^ {T} e ^ {\frac {\lambda (n - T)}{n}} \right] \\ \leqslant \mathbb {E} _ {S, S ^ {\prime}, W} \sim P _ {S ^ {\prime}} P _ {S, W} \left[ e ^ {\frac {\lambda^ {2} T}{2 n ^ {2}}} e ^ {\frac {\lambda (n - T)}{n}} \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a)} {\leqslant} e ^ {\frac {\lambda^ {2}}{2 n}} \times \mathbb {E} _ {S, S ^ {\prime}, \sim P _ {S ^ {\prime}} P _ {S, W}} \left[ e ^ {\frac {\lambda}{2} \| \hat {p} _ {S} - \hat {p} _ {S ^ {\prime}} \| _ {1}} \right] \\ \leqslant \exp \left(\frac {\lambda^ {2}}{2 n} + \frac {K + 2}{2} + \frac {3 \lambda^ {2}}{2 n}\right) \\ \leqslant \exp \left(\frac {2 \lambda^ {2}}{n} + \frac {K + 2}{2}\right), \\ \end{array}
$$
where  $(a)$  is deduced from Lemma 3, conditioned that  $\lambda /n < 1.36$
Now, combining this with (34), and letting  $\lambda = \lambda^{*} = \sqrt{n(2B + K + 2) / 4}$ , the expectation of generalization error is upper bounded by
$$
\begin{array}{l} \mathbb {E} _ {S, W} [ \operatorname {g e n} (S, W) ] = \frac {2 B + K + 2}{2 \lambda} + \frac {2 \lambda}{n} \\ \leqslant 2 \sqrt {\frac {(2 B + K + 2)}{n}}, \\ \end{array}
$$
if  $\lambda^{*} / n < 1.36$ . Note that if  $1.36 < \lambda^{*} / n = \sqrt{\frac{(2B + K + 2)}{4n}}$ , then
$$
2 \sqrt {\frac {(2 B + K + 2)}{n}} = 4 (\lambda^ {*} / n) > 1.
$$
Since generalization error is always bounded by 1, hence, this bound always holds.
Lemma 3. Suppose that  $Y^n$  and  $Y'^n$  are  $2n$  i.i.d. instances of  $Y \sim p \in [K]$ . Then, if  $\lambda / n < 0.68$ , then,
$$
\log \mathbb {E} _ {Y ^ {n}, Y ^ {\prime n}} \left[ e ^ {\lambda \| \hat {p} _ {Y ^ {n}} - \hat {p} _ {Y ^ {\prime n}} \| _ {1}} \right] \leqslant \frac {K + 2}{2} + \frac {6 \lambda^ {2}}{n}.
$$
The lemma is proved in Appendix E.9 using results of [Dev83].
E.6 Proof of Theorem 5
Let  $\mathbf{Q}$  be any fixed symmetric prior on  $(\hat{Y}^n,\hat{Y}^{\prime n})$  that could depend on  $(X^n,Y^n,X^{\prime n},Y^{\prime n})$ . We show that for any  $\epsilon \in \mathbb{R}$  and  $\delta \in \mathbb{R}^{+}$  with probability at least  $(1 - \delta)$  over choices of  $S$  and  $S^{\prime}$ , we have that  $\mathbb{E}_{W\sim Q}[\mathrm{gen}(S,W)]$  is upper bounded by
$$
\sqrt {\frac {\log (2 / \delta)}{2 n}} + \inf  \sqrt {\frac {\mathbb {E} _ {\hat {W} \sim P _ {\hat {W} | S}} \left[ D _ {K L} \left(P _ {\hat {Y} | X , \hat {W}} ^ {\otimes 2 n} \left(\hat {Y} ^ {n} , \hat {Y} ^ {\prime n} \mid X ^ {n} , X ^ {\prime n} , \hat {W}\right) \| \mathbf {Q}\right) \right] + \log (\sqrt {8 n} / \delta)}{(2 n - 1) / 4} + \epsilon}, \tag {36}
$$
where the infimum is over all  $P_{\hat{W}|S}$  that satisfy