SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {k \in [ K ]} \left\| \mathbf {V} _ {K} ^ {\top} \mathbf {P} _ {\mathbf {U}} ^ {\perp} \mathbf {y} _ {k} \right\| _ {2} ^ {2} \leq \sum_ {k \in [ K ]} \| \mathbf {y} _ {k} \| _ {2} ^ {2} \left\| \mathbf {P} _ {\mathbf {U}} ^ {\perp} \mathbf {V} _ {K} \right\| _ {F} ^ {2} = \left\| \mathbf {P} _ {\mathbf {U}} ^ {\perp} \mathbf {V} _ {K} \right\| _ {F} ^ {2} \sum_ {k \in [ K ]} P (\mathcal {X} _ {k}) = \left\| \mathbf {P} _ {\mathbf {U}} ^ {\perp} \mathbf {V} _ {K} \right\| _ {F} ^ {2}.
$$
$$
P \left(M \left(f _ {\left| \mathbf {X} ^ {u}\right)}\right) \leq 2 \max  \left(\frac {\beta^ {2}}{\gamma^ {2}}, 1\right) \left(\frac {\alpha}{\lambda_ {K + 1}} + \left\| \mathbf {P} _ {\mathbf {U}} ^ {\perp} \mathbf {V} _ {K} \right\| _ {F} ^ {2}\right). \right. \tag {16}
$$
Upper bounding  $\left\| \mathbf{P}_{\mathbf{U}}^{\perp}\mathbf{V}_{K}\right\|_{F}^{2}$ . We first recall from Equation (13) that  $\overline{\mathbf{W}}(\mathcal{X}) = \sum_{i=1}^{|\mathcal{X}|} (1 - \lambda_i)\mathbf{v}_i\mathbf{v}_i^\top$ . For  $f_{|\mathcal{X}} \in \mathcal{F}_{\mathbf{L}(\mathcal{X})}$ , Eckart-Young-Mirsky theorem [Eckart and Young, 1936] implies that  $\mathbf{F}_{\mathcal{X}}\mathbf{F}_{\mathcal{X}}^\top = \sum_{i=1}^{K} (1 - \lambda_i)\mathbf{v}_i\mathbf{v}_i^\top$ , and
$$
R \left(f _ {| \mathcal {X}}\right) = \left\| \overline {{\mathbf {W}}} (\mathcal {X}) - \mathbf {F} _ {\mathcal {X}} \mathbf {F} _ {\mathcal {X}} ^ {\top} \right\| _ {F} ^ {2} = \left\| \sum_ {i > K} \left(1 - \lambda_ {i}\right) \mathbf {v} _ {i} \mathbf {v} _ {i} ^ {\top} \right\| _ {F} ^ {2} = \sum_ {i > K} \left(1 - \lambda_ {i}\right) ^ {2}.
$$
Therefore,  $R\left(f_{|\mathbf{X}^u}\right)\leq R\left(f_{|\mathcal{X}}\right) + \Delta$  implies that
$$
\begin{array}{l} \sum_ {i > K} \left(1 - \lambda_ {i}\right) ^ {2} + \Delta \geq \left\| \overline {{\mathbf {W}}} (\mathcal {X}) - \mathbf {U} _ {\mathcal {X}} \mathbf {U} _ {\mathcal {X}} ^ {\top} \right\| _ {F} ^ {2} = \left\| \left(\overline {{\mathbf {W}}} (\mathcal {X}) - \mathbf {P} _ {\mathbf {U}} \overline {{\mathbf {W}}} (\mathcal {X})\right) + \left(\mathbf {P} _ {\mathbf {U}} \overline {{\mathbf {W}}} (\mathcal {X}) - \mathbf {U} _ {\mathcal {X}} \mathbf {U} _ {\mathcal {X}} ^ {\top}\right) \right\| _ {F} ^ {2} \\ = \left\| \mathbf {P} _ {\mathbf {U}} ^ {\perp} \overline {{\mathbf {W}}} (\mathcal {X}) + \mathbf {P} _ {\mathbf {U}} \left(\overline {{\mathbf {W}}} (\mathcal {X}) - \mathbf {U} _ {\mathcal {X}} \mathbf {U} _ {\mathcal {X}} ^ {\top}\right) \right\| _ {F} ^ {2} \quad \left(\text {s i n c e} \mathbf {P} _ {\mathbf {U}} \text {a n d} \mathbf {P} _ {\mathbf {U}} ^ {\perp} \text {a r e o r t h o g o n a l}\right) \\ = \left\| \mathbf {P} _ {\mathbf {U}} ^ {\perp} \overline {{\mathbf {W}}} (\mathcal {X}) \right\| _ {F} ^ {2} + \left\| \mathbf {P} _ {\mathbf {U}} \left(\overline {{\mathbf {W}}} (\mathcal {X}) - \mathbf {U} _ {\mathcal {X}} \mathbf {U} _ {\mathcal {X}} ^ {\top}\right) \right\| _ {F} ^ {2} \\ \geq \left\| \mathbf {P} _ {\mathbf {U}} ^ {\perp} \overline {{\mathbf {W}}} (\mathcal {X}) \right\| _ {F} ^ {2} = \left\| \sum_ {i = 1} ^ {| \mathcal {X} |} \left(1 - \lambda_ {i}\right) \left(\mathbf {P} _ {\mathbf {U}} ^ {\perp} \mathbf {v} _ {i}\right) \mathbf {v} _ {i} ^ {\top} \right\| _ {F} ^ {2} \\ = \operatorname {t r} \left(\sum_ {i = 1} ^ {| \mathcal {X} |} \sum_ {j = 1} ^ {| \mathcal {X} |} \left(1 - \lambda_ {i}\right) \left(1 - \lambda_ {j}\right) \left(\mathbf {P} _ {\mathbf {U}} ^ {\perp} \mathbf {v} _ {i}\right) \mathbf {v} _ {i} ^ {\top} \mathbf {v} _ {j} \left(\mathbf {P} _ {\mathbf {U}} ^ {\perp} \mathbf {v} _ {j}\right) ^ {\top}\right) \\ = \sum_ {i = 1} ^ {| \mathcal {X} |} \left(1 - \lambda_ {i}\right) ^ {2} \left\| \mathbf {P} _ {\mathbf {U}} ^ {\perp} \mathbf {v} _ {i} \right\| _ {2} ^ {2}. \tag {17} \\ \end{array}
$$
Meanwhile, the assumption  $\Delta < (1 - \lambda_K)^2$  implies that  $\mathrm{rank}\left(\mathbf{P}_{\mathbf{U}}^{\perp}\right) = |\mathcal{X}| - K$ , and thus
$$
\sum_ {i = 1} ^ {| \mathcal {X} |} \left\| \mathbf {P} _ {\mathbf {U}} ^ {\perp} \mathbf {v} _ {i} \right\| _ {2} ^ {2} = \left\| \mathbf {P} _ {\mathbf {U}} ^ {\perp} \right\| _ {F} ^ {2} = \operatorname {t r} \left(\mathbf {P} _ {\mathbf {U}} ^ {\perp}\right) = \operatorname {r a n k} \left(\mathbf {P} _ {\mathbf {U}} ^ {\perp}\right) = | \mathcal {X} | - K. \tag {18}
$$
Otherwise by contradiction, suppose  $\mathrm{rank}(\mathbf{P}_{\mathbf{U}}) = \mathrm{rank}(\mathbf{U}_{\mathcal{X}}) < K$ , then Eckart-Young-Mirsky theorem [Eckart and Young, 1936] implies that
$$
\min _ {\mathrm {r a n k} (\mathbf {U} _ {\mathcal {X}}) <   K} \left\| \overline {{\mathbf {W}}} (\mathcal {X}) - \mathbf {U} _ {\mathcal {X}} \mathbf {U} _ {\mathcal {X}} ^ {\top} \right\| _ {F} ^ {2} = \sum_ {i = K} ^ {| \mathcal {X} |} (1 - \lambda_ {i}) ^ {2} > \sum_ {i > K} (1 - \lambda_ {i}) ^ {2} + \Delta = R (f _ {| \mathcal {X}}) + \Delta ,
$$
which contradicts  $R\left(f_{|\mathbf{X}^u}\right)\leq R\left(f_{|\mathcal{X}}\right) + \Delta$
Furthermore, since  $\| \mathbf{v}_i\| _2 = 1$  and  $\mathbf{P}_{\mathbf{U}}^{\perp}$  is an orthogonal projection, for all  $i\in [|X|]$
$$
0 \leq \left\| \mathbf {P} _ {\mathbf {U}} ^ {\perp} \mathbf {v} _ {i} \right\| _ {2} ^ {2} \leq 1. \tag {19}
$$
For every  $i \in [|\mathcal{X}|]$ , we denote  $\xi_i \triangleq \left\| \mathbf{P}_{\mathbf{U}}^\perp \mathbf{v}_i \right\|_2^2$ . Then, upper bounding  $\left\| \mathbf{P}_{\mathbf{U}}^\perp \mathbf{V}_K \right\|_F^2 = \sum_{i=1}^{K} \xi_i$  can be recast as a linear programming problem:
(P):  $\max_{\xi \in \mathbb{R}^{|\mathcal{X}|}}\sum_{i = 1}^{K}\xi_{i}$
$$
\begin{array}{l l} \text {s . t .} & 0 \leq \xi_ {i} \leq 1, \end{array}
$$
$$
\sum_ {i = 1} ^ {| \mathcal {X} |} \xi_ {i} = | \mathcal {X} | - K, \tag {20}
$$
$$
\sum_ {i = 1} ^ {| \mathcal {X} |} \left(1 - \lambda_ {i}\right) ^ {2} \xi_ {i} \leq \sum_ {i > K} \left(1 - \lambda_ {i}\right) ^ {2} + \Delta ,
$$
whose dual can be expressed as
(D):  $\min_{\left(\omega_1,\dots ,\omega_{|\mathcal{X}|},\omega_s,\omega_\Delta\right)}\sum_{i = 1}^{|\mathcal{X}|}\omega_i - (|\mathcal{X}| - K)\omega_s + \left(\Delta +\sum_{i > K}\left(1 - \lambda_i\right)^2\right)\omega_\Delta$
$$
\text {s . t .} \quad \omega_ {1}, \dots , \omega_ {| \mathcal {X} |}, \omega_ {s}, \omega_ {\Delta} \geq 0, \tag {21}
$$
$$
\omega_ {i} - \omega_ {s} + (1 - \lambda_ {i}) ^ {2} \omega_ {\Delta} \geq 1 \quad \forall i \leq K,
$$
$$
\omega_ {i} - \omega_ {s} + (1 - \lambda_ {i}) ^ {2} \omega_ {\Delta} \geq 0 \quad \forall i > K.
$$
By taking
$$
\omega_ {i} = 0 \quad \forall i \leq K _ {0},
$$
$$
\omega_ {i} = \frac {(1 - \lambda_ {K _ {0}}) ^ {2} - (1 - \lambda_ {i}) ^ {2}}{(1 - \lambda_ {K _ {0}}) ^ {2} - (1 - \lambda_ {K + 1}) ^ {2}} \quad \forall K _ {0} <   i \leq K,
$$
$$
\omega_ {i} = \frac {\left(1 - \lambda_ {K + 1}\right) ^ {2} - \left(1 - \lambda_ {i}\right) ^ {2}}{\left(1 - \lambda_ {K _ {0}}\right) ^ {2} - \left(1 - \lambda_ {K + 1}\right) ^ {2}} \quad \forall K + 1 \leq i \leq | \mathcal {X} |,
$$
$$
\omega_ {s} = \frac {\left(1 - \lambda_ {K + 1}\right) ^ {2}}{\left(1 - \lambda_ {K _ {0}}\right) ^ {2} - \left(1 - \lambda_ {K + 1}\right) ^ {2}},
$$
$$
\omega_ {\Delta} = \frac {1}{(1 - \lambda_ {K _ {0}}) ^ {2} - (1 - \lambda_ {K + 1}) ^ {2}},
$$
the dual optimum can be upper bounded by
(D) =  $\sum_{i=1}^{K} \omega_i + \Delta \cdot \omega_\Delta$
$$
\begin{array}{l} = \frac {\Delta}{(1 - \lambda_ {K _ {0}}) ^ {2} - (1 - \lambda_ {K + 1}) ^ {2}} + \sum_ {i = K _ {0} + 1} ^ {K} \frac {(1 - \lambda_ {K _ {0}}) ^ {2} - (1 - \lambda_ {i}) ^ {2}}{(1 - \lambda_ {K _ {0}}) ^ {2} - (1 - \lambda_ {K + 1}) ^ {2}} \\ \leq \frac {\left(1 + \left(K - K _ {0}\right) C _ {K _ {0}}\right) \Delta}{\left(1 - \lambda_ {K _ {0}}\right) ^ {2} - \left(1 - \lambda_ {K + 1}\right) ^ {2}}, \\ \end{array}
$$
given  $C_{K_0} \triangleq \frac{(1 - \lambda_{K_0})^2 - (1 - \lambda_K)^2}{\Delta} = O(1)$ .
With both the primal Equation (20) and the dual Equation (21) being feasible and bounded, the strong duality then implies that
$$
\left\| \mathbf {P} _ {\mathbf {U}} ^ {\perp} \mathbf {V} _ {K} \right\| _ {F} ^ {2} = \sum_ {i = 1} ^ {K} \xi_ {i} = (\mathrm {P}) = (\mathrm {D}) \leq \frac {\left(1 + \left(K - K _ {0}\right) C _ {K _ {0}}\right) \Delta}{\left(1 - \lambda_ {K _ {0}}\right) ^ {2} - \left(1 - \lambda_ {K + 1}\right) ^ {2}},
$$
which, when plugging back into Equation (16), completes the proof.
E Proofs and Discussions for Section 6
E.1 Expansion-based Data Augmentation
For the detailed analysis, we further recall the following notion of constant expansion and its relation with the concept of  $c$ -expansion in Definition 6.1 from Wei et al. [2021].
Definition E.1  $((q,\xi)$ -constant expansion [Wei et al., 2021]). We say the expansion-based data augmentation (Definition 6.1) satisfies  $(q,\xi)$ -constant expansion if for any  $S\subseteq \mathcal{X}$  such that  $P(S)\geq q$  and  $P(S\cap \mathcal{X}_k)\leq P(\mathcal{X}_k) / 2$  for all  $k\in [K]$ ,
$$
P \left(\operatorname {N B} (S)\right) > \min  \left\{P (S), \xi \right\} + P (S).
$$
Lemma E.1 (c-expansion implies constant expansion [Wei et al., 2021]). The c-expansion property (Definition 6.1) implies  $\left(\frac{\xi}{c - 1},\xi\right)$ -constant expansion (Definition E.1) for any  $\xi >0$
Proof of Lemma E.1. Consider a subset  $S \subseteq \mathcal{X}$  such that  $P(S) \geq q$  and  $P\left(S \cap \mathcal{X}_k\right) \leq P\left(\mathcal{X}_k\right) / 2$ . Let  $S_k \triangleq S \cap \mathcal{X}_k$  for every  $k \in [K]$ .
(i) If  $c\in (1,2)$ , since  $P(\mathcal{X}_k) - P(S_k)\geq P(S_k)\geq (c - 1)P(S_k)$
$$
\begin{array}{l} P \left(\mathrm {N B} (S) \backslash S\right) \geq \sum_ {k = 1} ^ {K} P \left(\mathrm {N B} \left(S _ {k}\right)\right) - P \left(S _ {k}\right) \\ > \sum_ {k = 1} ^ {K} \min  \left\{\left(c - 1\right) P \left(S _ {k}\right), P \left(\mathcal {X} _ {k}\right) - P \left(S _ {k}\right) \right\} \\ \geq \sum_ {k = 1} ^ {K} (c - 1) P \left(S _ {k}\right) = (c - 1) P (S) \geq (c - 1) q = \xi \\ \end{array}
$$
(ii) If  $c\geq 2$  , since  $c - 1\geq 1$  and  $P(\mathcal{X}_k) - P(S_k)\geq P(S_k)$  , we have
$$
P \left(\mathrm {N B} (S) \backslash S\right) > \sum_ {k = 1} ^ {K} \min  \left\{\left(c - 1\right) P \left(S _ {k}\right), P \left(\mathcal {X} _ {k}\right) - P \left(S _ {k}\right) \right\} \geq \sum_ {k = 1} ^ {K} P \left(S _ {k}\right) = P (S)
$$
$$
P \left(\operatorname {N B} (S)\right) = P \left(\operatorname {N B} (S) \setminus S\right) + P (S) > \min  \left\{P (S), \xi \right\} + P (S).
$$
E.2 Data Augmentation Consistency Regularization
To bridge Equation (8) with common DAC regularization algorithms in practice, in Example E.1, we instantiate FixMatch [Sohn et al., 2020] - a state-of-the-art semi-supervised learning algorithm that leverages DAC regularization by encouraging similar predictions of proper weak and strong data augmentations  $\mathbf{x}^w$ ,  $\mathbf{x}^s \in \mathcal{A}(\mathbf{x})$  of the same sample  $\mathbf{x}$ .
Example E.1 (FixMatch [Sohn et al., 2020]). FixMatch minimizes the loss between the outputs of the strong augmentation  $f(\mathbf{x}^s)$  and the pseudo-label of the weak augmentation  $y_{f}(\mathbf{x}^{w})$  when the confidence of such pseudo-labeling is high (i.e.,  $\max_{k\in [K]}f(\mathbf{x}^w)_k$  is larger than a threshold).
To connect FixMatch with Equation (8), we consider a fixed (unknown) margin-robust neighborhood  $\mathcal{F}' \supseteq \mathcal{F}_{\mathbf{X}^u}^\tau \ni f_*$  where the strong augmentation characterizes the worst-case (minimum) robust margin  $\mathbf{x}^s \in \bigcap_{f \in \mathcal{F}'} \operatorname{argmin}_{\mathbf{x}' \in \mathcal{A}(\mathbf{x})} m(f, \mathbf{x}', y_f(\mathbf{x}))$ ; while the weak augmentation preserves the classification  $y_f(\mathbf{x}^w) = y_f(\mathbf{x})$ .
Then, for any  $f \in \mathcal{F}'$  during learning, minimizing the cross-entropy loss between softmax  $(f(\mathbf{x}^s))$  and the one-hot label  $\overrightarrow{y_f}(\mathbf{x}^w)$  is equivalent to enforcing a large enough margin  $m(f, \mathbf{x}^s, y_f(\mathbf{x})) \geq \tau$ , whereas  $m(f, \mathbf{x}', y_f(\mathbf{x})) \geq m(f, \mathbf{x}^s, y_f(\mathbf{x})) \geq \tau$  for all  $\mathbf{x}' \in \mathcal{A}(\mathbf{x})$  by construction implies  $m_{\mathcal{A}}(f, \mathbf{x}) \geq \tau$  as required by the constraints in Equation (8).
E.3 Proof of Theorem 6.2
Proof of Theorem 6.2. We first recall from Lemma E.1 that  $c$ -expansion implies  $\left(\frac{2\nu(f)}{c - 1}, 2\nu(f)\right)$ -constant expansion (Definition E.1) for any  $f \in \mathcal{F}$ . Therefore, it is sufficient to show that with  $(q, 2\nu(f))$ -constant expansion (i.e.,  $q = \frac{2\nu(f)}{c - 1}$ ),
$$
P (M (f)) \leq \max  \{q, 2 \nu (f) \} \quad \forall f \in \mathcal {F} _ {\mathbf {X} ^ {u}} ^ {\tau}. \tag {22}
$$
For any  $f \in \mathcal{F}_{\mathbf{X}^u}^\tau$ , we notice that Equation (22) is automatically satisfied when  $P(M(f)) < q$ . Otherwise, when  $P(M(f)) \geq q$ , then along with  $P(S \cap \mathcal{X}_k) \leq P(\mathcal{X}_k) / 2$  for all  $k \in [K]$ ,  $(q, 2\nu(f))$ -constant expansion implies
$$
P (\mathrm {N B} (M (f))) > \min  \left\{P (M (f)), 2 \nu (f) \right\} + P (M (f)). \tag {23}
$$
Meanwhile, we observe that by union bound,
$$
P (\text {N B} (M (f)) \backslash M (f)) \leq 2 \mathbb {P} [ \exists \mathbf {x} ^ {\prime} \in \mathcal {A} (\mathbf {x}) s. t. y _ {f} (\mathbf {x}) \neq y _ {f} \left(\mathbf {x} ^ {\prime}\right) ] = 2 \nu (f) \tag {24}
$$
because for any  $\mathbf{x} \in \mathrm{NB}(M(f)) \setminus M(f)$ , there exists  $\mathbf{x}' \in M(f)$  such that one can find some  $\mathbf{x}'' \in \mathcal{A}(\mathbf{x}) \cap \mathcal{A}(\mathbf{x}')$ ; and therefore exactly one of the follows must hold
(i) if  $\mathbf{x}'' \in M(f)$ , then  $\widetilde{y}_f(\mathbf{x}'') \neq y_*(\mathbf{x}) = \widetilde{y}_f(\mathbf{x})$  implies  $y_f(\mathbf{x}'') \neq y_f(\mathbf{x})$  for  $\mathbf{x}'' \in \mathcal{A}(\mathbf{x})$ ;
(ii) if  $\mathbf{x}'' \notin M(f)$ , then  $\widetilde{y_f}(\mathbf{x}^{\prime\prime}) = y_*(\mathbf{x}) \neq \widetilde{y_f}(\mathbf{x}')$  implies  $\dot{y}_f(\mathbf{x}^{\prime\prime}) \neq \dot{y}_f(\mathbf{x}')$  for  $\mathbf{x}^{\prime\prime} \in \mathcal{A}(\mathbf{x}')$ .
Leveraging Equation (23) and Equation (24), we have
$$
\begin{array}{l} P (M (f)) \geq P (\mathrm {N B} (M (f))) - P (\mathrm {N B} (M (f)) \backslash M (f)) \\ > \min  \left\{P (M (f)), 2 \nu (f) \right\} + P (M (f)) - 2 \nu (f), \\ \end{array}
$$
which leads to  $\min \left\{P(M(f)), 2\nu(f)\right\} < 2\nu(f)$  and implies  $P(M(f)) < 2\nu(f)$ .
F Technical Lemmas
Here, we briefly review the standard Rademacher-complexity-based generalization analysis based on McDiarmid's inequality and a classical symmetrization argument Wainwright [2019], Bartlett and Mendelson [2003].
Lemma F.1 (Generalization guarantee with Rademacher complexity). Given a distribution over a set  $\mathcal{Z}^{13}$ , we consider a set of i.i.d. samples  $\{\mathbf{z}_i\}_{i\in [n]}\sim \mathcal{Z}^n$  with  $\mathbf{Z} = [\mathbf{z}_1,\dots ,\mathbf{z}_n]$  and a  $B$ -bounded function class  $\mathcal{H} = \{h:\mathcal{Z}\to \mathbb{R}\mid |h(\mathbf{z}) - h(\mathbf{z}^{\prime})|\leq B\forall \mathbf{z},\mathbf{z}^{\prime}\in \mathcal{Z}\}$ . By denoting
$$
H (h) \triangleq \mathbb {E} _ {\mathbf {z} \sim \mathcal {Z}} [ h (\mathbf {z}) ] \quad a n d \quad \widehat {H} _ {\mathbf {Z}} (h) \triangleq \frac {1}{n} \sum_ {i = 1} ^ {n} h (\mathbf {z} _ {i}) \quad s. t. \quad \mathbb {E} _ {\mathbf {Z} \sim \mathcal {Z} ^ {n}} \left[ \widehat {H} _ {\mathbf {Z}} (h) \right] = H (h),
$$
<sup>13</sup>Without ambiguity, we overload the notation  $\mathcal{Z}$  as the set as well as the distribution over the set.
then for  $h^* \in \operatorname{argmin}_{h \in \mathcal{H}} H(h)$  and  $\widehat{h} \in \operatorname{argmin}_{h \in \mathcal{H}} \widehat{H}_{\mathbf{Z}}(h)$ , we have that with probability at least  $1 - \delta / 2$  over  $\mathbf{Z}$ ,
$$
H (\widehat {h}) - H (h ^ {*}) \leq 4 \Re_ {n} (\mathcal {H}) + \sqrt {\frac {2 B ^ {2} \log (4 / \delta)}{n}}.
$$
where  $\Re_n(\mathcal{H})$  is the Rademacher complexity of  $\mathcal{H}$
$$
\mathfrak{R}_{n}\left(\mathcal{H}\right)\triangleq \mathbb{E}_{\substack{\mathbf{Z}\sim \mathcal{Z}^{n}\\ \rho \sim \mathrm{Rad}^{n}}}\left[\sup_{h\in \mathcal{H}}\frac{1}{n}\sum_{i = 1}^{n}\rho_{i}\cdot h(\mathbf{z}_{i})\right],\quad \mathrm{Rad}(\rho) = \left\{ \begin{array}{ll}1 / 2, & \rho = -1\\ 1 / 2, & \rho = 1\\ 0, & \rho \neq \pm 1 \end{array} \right..
$$
Proof of Lemma F.1. Let  $g(\mathbf{z}_1, \ldots, \mathbf{z}_n) \triangleq g(\mathbf{Z}) \triangleq \sup_{h \in \mathcal{H}} H(h) - \widehat{H}_{\mathbf{Z}}(h)$ . Then  $h \in \mathcal{H}$  being  $B$ -bounded implies that for any  $\mathbf{z} \sim \mathcal{Z}$  and  $i \in [n]$ ,
$$
\left| g \left(\mathbf {z} _ {1}, \dots , \mathbf {z} _ {i}, \dots , \mathbf {z} _ {n}\right) - g \left(\mathbf {z} _ {1}, \dots , \mathbf {z}, \dots , \mathbf {z} _ {n}\right) \right| \leq \frac {B}{n}.
$$
Therefore, by Diarmid's inequality [Bartlett and Mendelson, 2003],
$$
\mathbb {P} \left[ g (\mathbf {Z}) \geq \mathbb {E} [ g (\mathbf {Z}) ] + t \right] \leq \exp \left(- \frac {2 n t ^ {2}}{B ^ {2}}\right),
$$
where  $\exp \left(-\frac{2nt^2}{B^2}\right) \leq \frac{\delta}{4}$  when  $t \geq \sqrt{\frac{B^2 \log(4 / \delta)}{2n}}$ .
Meanwhile, by a classical symmetrization argument (e.g., proof of Wainwright [2019] Theorem 4.10), we can bound the expectation  $\mathbb{E}[g(\mathbf{Z})]$ : for an independent sample set  $\mathbf{Z}' \sim \mathcal{Z}^n$  that is also independent of  $\mathbf{Z}$ ,
$$
\begin{array}{l} \mathbb {E} \left[ g (\mathbf {Z}) \right] = \mathbb {E} _ {\mathbf {Z}} \left[ \sup  _ {h \in \mathcal {H}} H (h) - \widehat {H} _ {\mathbf {Z}} (h) \right] \\ = \mathbb {E} _ {\mathbf {Z}} \left[ \sup  _ {h \in \mathcal {H}} \mathbb {E} _ {\mathbf {Z} ^ {\prime}} \left[ \widehat {H} _ {\mathbf {Z} ^ {\prime}} (h) - \widehat {H} _ {\mathbf {Z}} (h) \mid \mathbf {Z} \right] \right] \\ \leq \mathbb {E} _ {\mathbf {Z}} \left[ \mathbb {E} _ {\mathbf {Z} ^ {\prime}} \left[ \sup _ {h \in \mathcal {H}} \widehat {H} _ {\mathbf {Z} ^ {\prime}} (h) - \widehat {H} _ {\mathbf {Z}} (h) \Big | \mathbf {Z} \right] \right] \\ = \mathbb {E} _ {\left(\mathbf {Z}, \mathbf {Z} ^ {\prime}\right)} \left[ \sup  _ {h \in \mathcal {H}} \widehat {H} _ {\mathbf {Z} ^ {\prime}} (h) - \widehat {H} _ {\mathbf {Z}} (h) \right], \\ \end{array}
$$
where in the last line we have used the law of iterated conditional expectation. Then, with Rademacher random variables  $\pmb {\rho} = (\rho_{1},\dots ,\rho_{n})\sim \mathrm{Rad}^{n}$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ g (\mathbf {Z}) \right] \leq \mathbb {E} _ {\left(\mathbf {Z}, \mathbf {Z} ^ {\prime}, \boldsymbol {\rho}\right)} \left[ \sup  _ {h \in \mathcal {H}} \frac {1}{n} \sum_ {i = 1} ^ {n} \rho_ {i} \cdot \left(h \left(\mathbf {z} _ {i} ^ {\prime}\right) - h \left(\mathbf {z} _ {i}\right)\right) \right] \\ \leq 2 \mathbb {E} _ {\left(\mathbf {Z}, \boldsymbol {\rho}\right)} \left[ \sup  _ {h \in \mathcal {H}} \frac {1}{n} \sum_ {i = 1} ^ {n} \rho_ {i} \cdot h \left(\mathbf {z} _ {i}\right) \right] \\ \leq 2 \Re_ {n} (\mathcal {H}). \\ \end{array}
$$
Therefore, with probability at least  $1 - \delta / 4$  over  $\mathbf{Z}$ , we have
$$
\sup _ {h \in \mathcal {H}} H (h) - \widehat {H} _ {\mathbf {Z}} (h) \leq 2 \Re_ {n} (\mathcal {H}) + \sqrt {\frac {B ^ {2} \log (4 / \delta)}{2 n}},
$$
while the same tail bound holds for  $\sup_{h\in \mathcal{H}} - H(h) + \widehat{H}_{\mathbf{Z}}(h)$  via an analogous argument.
By recalling the definition of  $\widehat{h}$  and  $h^*$ , we can decompose the excess risk  $H(\widehat{h}) - H(h^*)$  as
$$
\begin{array}{l} H (\widehat {h}) - H (h ^ {*}) = \left(H (\widehat {h}) - \widehat {H} _ {\mathbf {Z}} (\widehat {h})\right) + \left(\widehat {H} _ {\mathbf {Z}} (\widehat {h}) - \widehat {H} _ {\mathbf {Z}} (h ^ {*})\right) + \left(\widehat {H} _ {\mathbf {Z}} (h ^ {*}) - H (h ^ {*})\right) \\ \leq \left(H (\widehat {h}) - \widehat {H} _ {\mathbf {Z}} (\widehat {h})\right) + \left(\widehat {H} _ {\mathbf {Z}} (h ^ {*}) - H (h ^ {*})\right) \\ \end{array}
$$
where  $\widehat{H}_{\mathbf{Z}}(\widehat{h}) - \widehat{H}_{\mathbf{Z}}(h^{*})\leq 0$  by the basic inequality. With both  $\widehat{h},h^{*}\in \mathcal{H}$  , we have
$$
H (\widehat {h}) - H (h ^ {*}) \leq 2 \sup  _ {h \in \mathcal {H}} \left| H (h) - \widehat {H} _ {\mathbf {Z}} (h) \right|,
$$
$$
\mathbb {P} \left[ H (\widehat {h}) - H (h ^ {*}) \geq t \right] \leq \mathbb {P} \left[ \sup  _ {h \in \mathcal {H}} H (h) - \widehat {H} _ {\mathbf {Z}} (h) \geq \frac {t}{2} \right] + \mathbb {P} \left[ \sup  _ {h \in \mathcal {H}} - H (h) + \widehat {H} _ {\mathbf {Z}} (h) \geq \frac {t}{2} \right].
$$
Overall, with probability at least  $1 - \delta /2$  over  $\mathbf{Z}$