SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 3.4. Consider the setting in Example 1 and suppose that  $\pmb{\mu}$  lies in the 2-dimensional subspace  $\Re$  in Theorem A.4 (see Appendix A.1 for details about  $\Re$  and see Figure 1 for intuitive illustration). Let  $\ell_{01}(x,y) = \mathbb{1}[x\neq y]$ , where  $\mathbb{1}[\cdot ]$  is the indicator function and  $B_p^r (\pmb {x}) = \{\pmb{x}':\| \pmb {x} - \pmb{x}'\| _p\leq r\}$ , consider training with hypothesis class  $\mathcal{H} = \{h_{\pmb{w}}:h_{\pmb{w}}(x) = sign(\pmb{w}^T x),\pmb {w}\in \mathbb{R}^d\}$ , and denote  $\mathcal{D}^{(ij)} = \frac{1}{2}\mathcal{D}^{(i)} + \frac{1}{2}\mathcal{D}^{(j)}, i,j\in \{0,1,2\}, i\neq j$ . Consider the  $\ell_2$ -norm adversarial attack with radius  $\epsilon$ , for notation convenience, let  $\widetilde{\mathcal{R}}_{ij}(\boldsymbol {w}) = \mathcal{R}_{\mathcal{D}^{(ij)}}^{B_2^\epsilon}(\ell_{01},h_{\boldsymbol{w}})$  and  $\widetilde{\mathcal{R}}_i(\pmb {w}) = \mathcal{R}_{\mathcal{D}^{(i)}}^{B_2^\epsilon}(\ell_{01},h_{\boldsymbol{w}})$ , let  $\Phi (\cdot)$  be the distribution function of the standard normal distribution, let  $\alpha$  denote the angle between  $\pmb{\mu}$  and  $Q\pmb{\mu}$ , which is the rotation angle of  $Q$  in subspace  $\Re$ , suppose that  $0 < \alpha \leq \arccos_{\| \pmb{\mu}_1\|_2}^{\epsilon}$ , then we have:
1. If we train with  $\mathcal{D}^{(0)}$  and  $\mathcal{D}^{(1)}$  under  $\mathcal{H}$ , let  $\pmb{w}_{(01)} = \pmb{\mu} + \underline{Q}\pmb{\mu}$ , which achieves the minimum of  $\widetilde{\mathcal{R}}_{01}(\pmb{w})$ , then the robust accuracy of  $\pmb{w}_{(01)}$  on the test distribution  $\mathcal{D}^{(2)}$  is:
$$
\widetilde {\mathcal {R}} _ {2} (\boldsymbol {w} _ {(0 1)}) = \Phi \left(\frac {\epsilon}{\sigma} - \frac {(\boldsymbol {\mu} + \underline {{Q}} \boldsymbol {\mu}) ^ {T} \underline {{Q}} ^ {2} \boldsymbol {\mu}}{\sigma \| \boldsymbol {\mu} + \underline {{Q}} \boldsymbol {\mu} \| _ {2}}\right).
$$
2. If we train with  $\mathcal{D}^{(0)}$  and  $\mathcal{D}^{(2)}$  under  $\mathcal{H}$ , let  $\pmb{w}_{(02)} = \pmb{\mu} + \underline{Q}^2\pmb{\mu}$ , which achieves the minimum of  $\widetilde{\mathcal{R}}_{02}(\pmb{w})$ , then the robust accuracy of  $\pmb{w}_{(02)}$  on the test distribution  $\mathcal{D}^{(1)}$  is:
$$
\widetilde {\mathcal {R}} _ {1} \left(\boldsymbol {w} _ {(0 2)}\right) = \Phi \left(\frac {\epsilon}{\sigma} - \frac {\left(\boldsymbol {\mu} + \underline {{Q}} ^ {2} \boldsymbol {\mu}\right) ^ {T} \underline {{Q}} \boldsymbol {\mu}}{\sigma \| \boldsymbol {\mu} + \underline {{Q}} ^ {2} \boldsymbol {\mu} \| _ {2}}\right).
$$
3. If we train with  $\mathcal{D}^{(1)}$  and  $\mathcal{D}^{(2)}$  under  $\mathcal{H}$ , then  $\boldsymbol{w}_{(12)} = \underline{Q}\boldsymbol{\mu} + \underline{Q}^2\boldsymbol{\mu}$ , which achieves the minimum of  $\widetilde{\mathcal{R}}_{12}(\boldsymbol{w})$ , then the robust accuracy of  $\boldsymbol{w}_{(12)}$  on the test distribution  $\mathcal{D}^{(0)}$  is:
$$
\widetilde {\mathcal {R}} _ {0} \left(\boldsymbol {w} _ {(1 2)}\right) = \Phi \left(\frac {\epsilon}{\sigma} - \frac {\left(Q \boldsymbol {\mu} + \underline {{Q}} ^ {2} \boldsymbol {\mu}\right) ^ {T} \boldsymbol {\mu}}{\sigma \| Q \boldsymbol {\mu} + \underline {{Q}} ^ {2} \boldsymbol {\mu} \| _ {2}}\right).
$$
What's more, we have  $\widetilde{\mathcal{R}}_1(\pmb{w}_{(02)}) < \widetilde{\mathcal{R}}_2(\pmb{w}_{(01)}) = \widetilde{\mathcal{R}}_0(\pmb{w}_{(12)})$ .
Proof of Theorem 3.4. The values of  $\widetilde{\mathcal{R}}_0(\pmb{w}_{(12)})$ ,  $\widetilde{\mathcal{R}}_1(\pmb{w}_{(02)})$ ,  $\widetilde{\mathcal{R}}_2(\pmb{w}_{(01)})$  is a direct result of Lemma A.2 and Lemma A.3. Next we compare the three adversarial risks.
Let  $C_{(01)} = \frac{(\pmb{\mu} + Q\pmb{\mu})^TQ^2\pmb{\mu}}{\sigma\|\pmb{\mu} + Q\pmb{\mu}\|_2}, C_{(02)} = \frac{(\pmb{\mu} + Q^2\pmb{\mu})^TQ\pmb{\mu}}{\sigma\|\pmb{\mu} + Q^2\pmb{\mu}\|_2}, C_{(12)} = \frac{(Q\pmb{\mu} + Q^2\pmb{\mu})^T\pmb{\mu}}{\sigma\|\pmb{\mu} + Q^2\pmb{\mu}\|_2}$ , i.e., the subtractor in  $\widetilde{\mathcal{R}}_2(\pmb{w}_{(01)}), \widetilde{\mathcal{R}}_1(\pmb{w}_{(02)}), \widetilde{\mathcal{R}}_0(\pmb{w}_{(12)})$  respectively. Since  $\Phi(u) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{u} e^{-\frac{y^2}{2}} du$  is monotonely increasing, to judge which of the three robust errors in Theorem 3.4 is the smallest, we need only to judge which of the corresponding  $C_{(ij)}$  is the largest.
Firstly, let  $Q_{e}$  be the matrix of  $\underline{Q}$  under the orthonormal basis  $\pmb{e}_1, \dots, \pmb{e}_d$  where  $(\pmb{e}_i)_j = \delta_{ij}$  and  $\delta_{ij} = 1$  if  $i = j$ ,  $\delta_{ij} = 0$  otherwise. Then we have  $Q_{\mu} = Q_{e}\pmb{\mu}$  and we can see that  $C_{(12)} = \frac{(Q_e\pmb{\mu} + Q_e^2\pmb{\mu})^T\pmb{\mu}}{\sigma\|\mathcal{Q}_e\pmb{\mu} + Q_e^2\pmb{\mu}\|_2} = \frac{\pmb{\mu}^T Q_e^T\pmb{\mu} + \pmb{\mu}^T Q_e^T\pmb{\mu}}{\sigma\|\pmb{\mu} + Q_e\pmb{\mu}\|_2} = \frac{\pmb{\mu}^T Q_e^T Q_e Q_e\pmb{\mu} + \pmb{\mu}^T Q_e^2\pmb{\mu}}{\sigma\|\pmb{\mu} + Q_e\pmb{\mu}\|_2} = \frac{(\pmb{\mu} + Q_e\pmb{\mu})^T Q_e^2\pmb{\mu}}{\sigma\|\pmb{\mu} + Q_e\pmb{\mu}\|_2} = C_{(01)}$  (where  $a$  is from the fact that  $\| Q_{e}\pmb{x}\|_{2} = \| \pmb{x}\|_{2}, \forall \pmb{x}$  and  $\pmb{x}^T A\pmb{x} = \pmb{x}^T A^T\pmb{x}, \forall \pmb{x}, A$ ). So  $\widetilde{\mathcal{R}}_2(w_{(01)}) = \widetilde{\mathcal{R}}_0(w_{(12)})$ .
To compare which one of  $\widetilde{\mathcal{R}}_1(\pmb{w}_{(02)})$  and  $\widetilde{\mathcal{R}}_2(\pmb{w}_{(01)})$  is smaller, we note that  $\pmb{\mu}$  lies in the 2-dimensional subspace  $\Re \in \mathbb{R}^d$  in Theorem A.4 as specified in Theorem 3.4, so there exists a scalar  $k \in \mathbb{R}$  such that  $\pmb{\mu} + \underline{Q}^2\pmb{\mu} = k\underline{Q}\pmb{\mu}$ , and we know that  $\underline{Q}\pmb{\mu} \in \operatorname*{arg min}_{\pmb{w}:h_{\pmb{w}} \in \mathcal{H}} \widetilde{\mathcal{R}}_{02}(\pmb{w})$  (by Lemma A.2), so we know that  $\widetilde{\mathcal{R}}_1(\pmb{w}_{(02)}) \leq \widetilde{\mathcal{R}}_2(\pmb{w}_{(01)}) = \widetilde{\mathcal{R}}_0(\pmb{w}_{(12)})$  since  $\inf_{\pmb{w}:h_{\pmb{w}} \in \mathcal{H}} \widetilde{\mathcal{R}}_0(\pmb{w}) = \inf_{\pmb{w}:h_{\pmb{w}} \in \mathcal{H}} \widetilde{\mathcal{R}}_1(\pmb{w}) = \inf_{\pmb{w}:h_{\pmb{w}} \in \mathcal{H}} \widetilde{\mathcal{R}}_2(\pmb{w})$  (which is obvious according to Lemma A.2). And following the same direction of the proof of Theorem A.4, it's easy to see that  $\pmb{w}_{(01)} = \pmb{\mu} + \underline{Q}\pmb{\mu}$  is not in the same direction of  $\underline{Q}^2\pmb{\mu}$  (the weight attains smallest robust error on  $\mathcal{D}^{(2)}$ ), so  $\widetilde{\mathcal{R}}_2(\pmb{w}_{(01)}) > \inf_{\pmb{w}:h_{\pmb{w}} \in \mathcal{H}} \widetilde{\mathcal{R}}_2(\pmb{w})$ , so we have  $\widetilde{\mathcal{R}}_1(\pmb{w}_{(02)}) < \widetilde{\mathcal{R}}_2(\pmb{w}_{(01)}) = \widetilde{\mathcal{R}}_0(\pmb{w}_{(12)})$ .
Theorem A.4. Let  $\underline{Q}$  be a rotation transformation on  $\mathbb{R}^d$  and  $\underline{Q} \neq \underline{I}$ , then there exists at least one 2-dimensional subspace  $\Re \in \mathbb{R}^d$  such that there exists a scalar  $k$  s.t.  $\boldsymbol{r} + \underline{Q}^2\boldsymbol{r} = k\boldsymbol{r}$  for any  $\boldsymbol{r} \in \Re$ .
Proof of Theorem A.4. By the results of matrix theory, we know that since  $Q$  is an orthonormal transformation, then there exists an orthonormal basis  $\pmb{\eta}_1,\dots \pmb{\eta}_d$  under which the corresponding matrix of  $Q,Q$ , has the form:
$$
Q = \operatorname {d i a g} \left\{R _ {1}, \dots , R _ {m}, \lambda_ {1}, \dots , \lambda_ {r} \right\},
$$
where  $\lambda_{i} = 1$  or  $-1$ ,  $i = 1,2,\dots,r,0\leq r\leq d$ ;  $R_{j} = \begin{bmatrix} \cos \theta_{j} & -\sin \theta_{j}\\ \sin \theta_{j} & \cos \theta_{j} \end{bmatrix}$ ,  $0 < \theta_{j} < \pi$ ,  $j = 1,2,\dots,m,0\leq m\leq \frac{d}{2}$ .
Since  $\underline{Q}$  is a rotation transformation,  $Q$  is a rotation matrix, so  $\lambda_1 = \dots = \lambda_r = 1$ . Once given the basis, linear transformations on  $\mathbb{R}^d$  and matrices in  $\mathbb{R}^{d\times d}$  has a one-to-one correspondence, so  $\underline{Q}\neq \underline{I}$  tells us that  $Q\neq I$ , where  $I$  is the matrix of  $\underline{I}$  under the orthonormal basis  $\pmb{\eta}_1,\ldots ,\pmb{\eta}_d$ . So we have  $m\geq 1$  in our case, without loss of generality we consider the subspace  $\Re = \operatorname {span}\langle \pmb {\eta}_1,\pmb {\eta}_2\rangle$  be the subspace spanned by the vectors  $\{\pmb {\eta}_1,\pmb {\eta}_2\}$ .
For any  $\pmb{\alpha} \in \Re$ , the coordinates of  $\pmb{\alpha}$  under the basis  $\pmb{\eta}_1, \dots, \pmb{\eta}_d$  is then  $[\alpha_1, \alpha_2, 0, \dots, 0]^T$  and the coordinates of  $\underline{Q}\pmb{\alpha}, \underline{Q}^2\pmb{\alpha}$  is  $Q[\alpha_1, \alpha_2, 0, \dots, 0]^T$ ,  $Q^2[\alpha_1, \alpha_2, 0, \dots, 0]^T$  respectively. We now prove that there exists  $\overline{k} = 2\cos \theta_1$  such that  $[\alpha_1, \alpha_2, 0, \dots, 0]^T + Q^2[\alpha_1, \alpha_2, 0, \dots, 0]^T = kQ[\alpha_1, \alpha_2, 0, \dots, 0]^T$ . We have:
$$
Q \left[ \begin{array}{c} \alpha_ {1} \\ \alpha_ {2} \\ 0 \\ \vdots \\ 0 \end{array} \right] = \left[ \begin{array}{c} R _ {1} \left( \begin{array}{c} \alpha_ {1} \\ \alpha_ {2} \end{array} \right) \\ 0 \\ \vdots \\ 0 \end{array} \right] = \left[ \begin{array}{c} \alpha_ {1} \cos \theta_ {1} - \alpha_ {2} \sin \theta_ {1} \\ \alpha_ {1} \sin \theta_ {1} + \alpha_ {2} \cos \theta_ {1} \\ 0 \\ \vdots \\ 0 \end{array} \right], \text {a n d}
$$
$$
Q ^ {2} \left[ \begin{array}{c} \alpha_ {1} \\ \alpha_ {2} \\ 0 \\ \vdots \\ 0 \end{array} \right] = \left[ \begin{array}{c} R _ {1} ^ {2} \left( \begin{array}{c} \alpha_ {1} \\ \alpha_ {2} \end{array} \right) \\ 0 \\ \vdots \\ 0 \end{array} \right] = \left[ \begin{array}{c} \alpha_ {1} \cos 2 \theta_ {1} - \alpha_ {2} \sin 2 \theta_ {1} \\ \alpha_ {1} \sin 2 \theta_ {1} + \alpha_ {2} \cos 2 \theta_ {1} \\ 0 \\ \vdots \\ 0 \end{array} \right].
$$
So the coordinates of  $\pmb{\alpha} + Q^2\pmb{\alpha}$  is:
$$
\begin{array}{l} \left[ \begin{array}{c} \alpha_ {1} \\ \alpha_ {2} \\ 0 \\ \vdots \\ 0 \end{array} \right] + Q ^ {2} \left[ \begin{array}{c} \alpha_ {1} \\ \alpha_ {2} \\ 0 \\ \vdots \\ 0 \end{array} \right] = \left[ \begin{array}{c} \alpha_ {1} (\cos 2 \theta_ {1} + 1) - \alpha_ {2} \sin 2 \theta_ {1} \\ \alpha_ {1} \sin 2 \theta_ {1} + \alpha_ {2} (\cos 2 \theta_ {1} + 1) \\ 0 \\ \vdots \\ 0 \end{array} \right] = \left[ \begin{array}{c} 2 \alpha_ {1} \cos^ {2} \theta_ {1} - 2 \alpha_ {2} \sin \theta_ {1} \cos \theta_ {1} \\ 2 \alpha_ {2} \sin \theta_ {1} \cos \theta_ {1} + 2 \alpha_ {1} \cos^ {2} \theta_ {1} \\ 0 \\ \vdots \\ 0 \end{array} \right] \\ = 2 \cos \theta_ {1} \left[ \begin{array}{c} \alpha_ {1} \cos \theta_ {1} - \alpha_ {2} \sin \theta_ {1} \\ \alpha_ {1} \sin \theta_ {1} + \alpha_ {2} \cos \theta_ {1} \\ 0 \\ \vdots \\ 0 \end{array} \right] = k \left[ \begin{array}{c} \alpha_ {1} \cos \theta_ {1} - \alpha_ {2} \sin \theta_ {1} \\ \alpha_ {1} \sin \theta_ {1} + \alpha_ {2} \cos \theta_ {1} \\ 0 \\ \vdots \\ 0 \end{array} \right], \\ \end{array}
$$
which is just  $k$  times of the coordinates of  $\underline{Q}\alpha$ . So we get  $\alpha + \underline{Q}^2\alpha = k\underline{Q}\alpha$  for any  $\alpha \in \Re$ .
A.2 Proofs of the Average Case
Theorem 3.1. Suppose the loss function is bounded, i.e.,  $\ell \in [0,U]$ , then with probability at least  $1 - \delta$  over the sampling of  $\widehat{\mathcal{D}}_1,\dots ,\widehat{\mathcal{D}}_t$ , for all  $h\in \mathcal{H}$ , we have:
Proof of Theorem 3.1. Our proof consists of 2 steps, our 2-step uniform convergence analysis is different from (but based on) the standard Rademacher complexity generalization error bounds.
Step 1, upper bound the difference between  $\mathcal{L}_p^{\mathcal{B}}(\ell,h)$  and  $\mathcal{L}_{\hat{p}}^{\mathcal{B}}(\ell,h)$  for any  $h\in \mathcal{H}$ .
Let  $\mathfrak{D} = \{\mathcal{D}_1,\dots ,\mathcal{D}_t\}$ , we define:
$$
\Phi^ {\mathcal {B}} (\mathfrak {D}) = \sup  _ {h \in \mathcal {H}} \left[ \mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , h) - \mathcal {L} _ {\hat {p}} ^ {\mathcal {B}} (\ell , h) \right].
$$
Since  $\ell \in [0,U]$ , changing one of the  $\mathcal{D}_i$  in  $\mathfrak{D}$  will lead to at most  $\frac{U}{t}$  change in  $\Phi^{\mathcal{B}}(\mathfrak{D})$ . So by McDiarmid's inequality we have:
$$
\mathbb {P} \left[ \Phi^ {\mathcal {B}} (\mathfrak {D}) - \mathbb {E} \left(\Phi^ {\mathcal {B}} (\mathfrak {D})\right) \right] \leq \exp \left\{- \frac {2 t \epsilon^ {2}}{U ^ {2}} \right\},
$$
so with probability at least  $1 - \frac{\delta}{4}$  over the choice of  $\mathfrak{D}$ , we have:
$$
\Phi^ {\mathcal {B}} (\mathfrak {D}) \leq \mathbb {E} \left(\Phi^ {\mathcal {B}} (\mathfrak {D})\right) + U \sqrt {\frac {\ln 4 / \delta}{2 t}}. \tag {A.1}
$$
Then we give an upper bound for  $\mathbb{E}(\Phi^{\mathcal{B}}(\mathfrak{D}))$ , by the symmetric technique, let  $\mathfrak{S} = \{S_1,\dots ,S_t\}$  be a set of independent copy of  $\mathfrak{D}$ , we have:
$$
\begin{array}{l} \mathbb {E} \left(\Phi^ {\mathcal {B}} (\mathfrak {D})\right) = \underset {\mathfrak {D} \sim p ^ {t}} {\mathbb {E}} \left[ \sup  _ {h \in \mathcal {H}} \left(\mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , h) - \mathcal {L} _ {\hat {p}} ^ {\mathcal {B}} (\ell , h)\right) \right] \\ = \underset {\mathfrak {D} \sim p ^ {t}} {\mathbb {E}} \left[ \sup  _ {h \in \mathcal {H}} \left(\underset {\mathcal {S} \sim p} {\mathbb {E}} \mathcal {R} _ {\mathcal {S}} ^ {\mathcal {B}} (\ell , h) - \frac {1}{t} \sum_ {i = 1} ^ {t} \mathcal {R} _ {\mathcal {D} _ {i}} ^ {\mathcal {B}} (\ell , h)\right) \right] \\ = \underset {\mathfrak {D} \sim p ^ {t}} {\mathbb {E}} \left[ \sup  _ {h \in \mathcal {H}} \frac {1}{t} \sum_ {i = 1} ^ {t} \underset {\mathcal {S} _ {i} \sim p} {\mathbb {E}} \left(\mathcal {R} _ {\mathcal {S} _ {i}} ^ {\mathcal {B}} (\ell , h) - \mathcal {R} _ {\mathcal {D} _ {i}} ^ {\mathcal {B}} (\ell , h)\right) \right] \\ \stackrel {a} {\leq} \underset {\mathfrak {D} \sim p ^ {t}} {\mathbb {E}} \left[ \underset {\mathfrak {S} \sim p ^ {t}} {\mathbb {E}} \sup  _ {h \in \mathcal {H}} \frac {1}{t} \sum_ {i = 1} ^ {t} \left(\mathcal {R} _ {\mathcal {S} _ {i}} ^ {\mathcal {B}} (\ell , h) - \mathcal {R} _ {\mathcal {D} _ {i}} ^ {\mathcal {B}} (\ell , h)\right) \right] \\ \stackrel {b} {=} \underset {\mathfrak {D} \sim p ^ {t}, \mathfrak {S} \sim p ^ {t}} {\mathbb {E}} \underset {\boldsymbol {\sigma}} {\mathbb {E}} \left[ \sup  _ {h \in \mathcal {H}} \frac {1}{t} \sum_ {i = 1} ^ {t} \sigma_ {i} \left(\mathcal {R} _ {\mathcal {S} _ {i}} ^ {\mathcal {B}} (\ell , h) - \mathcal {R} _ {\mathcal {D} _ {i}} ^ {\mathcal {B}} (\ell , h)\right) \right] \\ \stackrel {c} {\leq} 2 \mathop {\mathbb {E}} _ {\mathfrak {D} \sim p ^ {t}} \mathop {\mathbb {E}} _ {\boldsymbol {\sigma}} \left[ \sup  _ {h \in \mathcal {H}} \frac {1}{t} \sum_ {i = 1} ^ {t} \sigma_ {i} \mathcal {R} _ {\mathcal {D} _ {i}} ^ {\mathcal {B}} (\ell , h) \right] \\ = 2 \underset {\mathfrak {D} \sim p ^ {t}} {\mathbb {E}} \underset {\boldsymbol {\sigma}} {\mathbb {E}} \left[ \sup  _ {h \in \mathcal {H}} \frac {1}{t} \sum_ {i = 1} ^ {t} \sigma_ {i} \underset {\left(\boldsymbol {x} _ {i}, y _ {i}\right) \sim \mathcal {D} _ {i}} {\mathbb {E}} \left[ \sup  _ {\boldsymbol {x} _ {i} ^ {\prime} \in \mathcal {B} \left(\boldsymbol {x} _ {i}\right)} \ell \left(h \left(\boldsymbol {x} _ {i} ^ {\prime}\right), y _ {i}\right) \right] \right] \\ \stackrel {d} {\leq} 2 \underset {\mathfrak {D} \sim p ^ {t} (\boldsymbol {x} _ {i}, y _ {i}) \sim \mathcal {D} _ {i}} {\mathbb {E}} \underset {\sigma} {\mathbb {E}} \left[ \sup  _ {h \in \mathcal {H}} \frac {1}{t} \sum_ {i = 1} ^ {t} \sigma_ {i} \left[ \sup  _ {\boldsymbol {x} _ {i} ^ {\prime} \in \mathcal {B} (\boldsymbol {x} _ {i})} \ell (h (\boldsymbol {x} _ {i} ^ {\prime}), y _ {i}) \right] \right] \\ = 2 \underset {\mathfrak {D} \sim p ^ {t} (\boldsymbol {x} _ {i}, y _ {i}) \sim \mathcal {D} _ {i}} {\mathbb {E}} \underset {\mathcal {G}} {\mathbb {E}} \left[ \Re_ {t} (\widetilde {\mathcal {G}}) \right], \\ \end{array}
$$
where:  $a$  uses Jensen's inequality;  $b$  comes from the fact that the random variables  $\sigma_{i}\left(\mathcal{R}_{\mathcal{S}_{i}}^{\mathcal{B}}(\ell ,h) - \mathcal{R}_{\mathcal{D}_{i}}^{\mathcal{B}}(\ell ,h)\right)$  and  $\mathcal{R}_{\mathcal{S}_i}^{\mathcal{B}}(\ell ,h) - \mathcal{R}_{\mathcal{D}_i}^{\mathcal{B}}(\ell ,h)$  are identically distributed, where  $\sigma_{i}\sim$  Uniform  $(\{\pm 1\})$  and  $\pmb {\sigma}\in \{\pm 1\} ^t$  has independent elements  $\sigma_1,\dots ,\sigma_t$  ;  $c$  is from the property of sup that  $\sup (a + b)\leq \sup (a) + \sup (b)$  and the fact that  $\pmb{\sigma}$  has the same distribution as  $-\pmb{\sigma}$  .  $d$  uses Jensen's inequality.
Since  $\ell \in [0,U]$ , change in one data point  $(\pmb{x}_i,y_i)$  will cause at most  $\frac{U}{t}$  difference in  $\Re_t(\widetilde{\mathcal{G}})$ , we then use McDiarmid's inequality to get: with probability at least  $1 - \frac{\delta}{4}$ ,
$$
\underset {\mathfrak {D} \sim p ^ {t} \left(\boldsymbol {x} _ {i}, y _ {i}\right) \sim \mathcal {D} _ {i}} {\mathbb {E}} \underset {\mathcal {G}} {\mathbb {E}} \left[ \Re_ {t} (\widetilde {\mathcal {G}}) \right] \leq \Re_ {t} (\widetilde {\mathcal {G}}) + U \sqrt {\frac {\ln 4 / \delta}{2 t}}. \tag {A.2}
$$
Combine (A.1) with (A.2) we get: with probability at least  $1 - \frac{\delta}{2}$ ,
$$
\Phi^ {\mathcal {B}} (\mathfrak {D}) \leq 2 \Re_ {t} (\widetilde {\mathcal {G}}) + 3 U \sqrt {\frac {\ln 4 / \delta}{2 t}}. \tag {A.3}
$$
Step 2, upper bound the difference between  $\mathcal{L}_{\widehat{p}}^{\mathcal{B}}(\ell,h)$  and  $\mathcal{L}_{\widehat{\mathcal{D}}}^{\mathcal{B}}(\ell,h)$  for any  $h \in \mathcal{H}$ .
$$
\mathcal {L} _ {\hat {p}} ^ {\mathcal {B}} (\ell , h) - \mathcal {L} _ {\hat {\mathcal {D}}} ^ {\mathcal {B}} (\ell , h) = \frac {1}{t} \sum_ {i = 1} ^ {t} \left(\mathcal {R} _ {\mathcal {D} _ {i}} ^ {\mathcal {B}} (\ell , h) - \mathcal {R} _ {\hat {\mathcal {D}} _ {i}} ^ {\mathcal {B}} (\ell , h)\right).
$$
Let  $\widehat{\mathfrak{D}} = \widehat{\mathcal{D}}_i\cup \dots \cup \widehat{\mathcal{D}}_t$  and then define:
$$
\Psi^ {\mathcal {B}} (\widehat {\mathfrak {D}}) = \sup _ {h \in \mathcal {H}} \left(\mathcal {L} _ {\widehat {p}} ^ {\mathcal {B}} (\ell , h) - \mathcal {L} _ {\widehat {\mathcal {D}}} ^ {\mathcal {B}} (\ell , h)\right) = \sup _ {h \in \mathcal {H}} \frac {1}{t} \sum_ {i = 1} ^ {t} \left(\mathcal {R} _ {\mathcal {D} _ {i}} ^ {\mathcal {B}} (\ell , h) - \mathcal {R} _ {\widehat {\mathcal {D}} _ {i}} ^ {\mathcal {B}} (\ell , h)\right).
$$
Since  $\ell \in [0,U]$  and there are  $tn$  examples in  $\widehat{\mathfrak{D}}$ , changing the  $j$ -th example of  $\widehat{\mathcal{D}}_i$ ,  $(x_{ij},y_{ij})$ , will cause at most  $\frac{U}{tn}$  change  $\Psi^{\mathcal{B}}(\widehat{\mathfrak{D}})$ . So by McDiarmid's inequality we know: with probability at least  $1 - \frac{\delta}{4}$  over the choice of  $\widehat{\mathfrak{D}}$ ,
$$
\Psi^ {\mathcal {B}} (\widehat {\mathfrak {D}}) \leq \mathbb {E} \left[ \Psi^ {\mathcal {B}} (\widehat {\mathfrak {D}}) \right] + U \sqrt {\frac {\ln 4 / \delta}{2 t n}}. \tag {A.4}
$$
Next we upper bound  $\mathbb{E}\left[\Psi^{\mathcal{B}}(\widehat{\mathfrak{D}})\right]$ , by the symmetric technique, let  $\widehat{\mathfrak{S}} = \{\widehat{S}_1,\dots ,\widehat{S}_t\}$  be a set of independent copies of  $\widehat{\mathfrak{D}}$  and let  $(\widetilde{x}_{ij},\widetilde{y}_{ij})$  be the  $j$ -th example in  $\widehat{S}_i$ , we have:
where:  $a$  is from Jensen's inequality;  $b$  is from the fact that  $\Sigma_{ij}\left(\sup_{\tilde{\boldsymbol{x}}_{ij}^{\prime}\in \mathcal{B}(\tilde{\boldsymbol{x}}_{ij})}\ell (h(\tilde{\boldsymbol{x}}_{ij}^{\prime}),\tilde{y}_{ij}) - \sup_{\boldsymbol{x}_{ij}^{\prime}\in \mathcal{B}(\boldsymbol{x}_{ij})}\ell (h(\boldsymbol{x}_{ij}^{\prime}),y_{ij})\right)$  has the same distribution as  $\sup_{\tilde{\boldsymbol{x}}_{ij}^{\prime}\in \mathcal{B}(\tilde{\boldsymbol{x}}_{ij})}\ell (h(\tilde{\boldsymbol{x}}_{ij}^{\prime}),\tilde{y}_{ij}) - \sup_{\boldsymbol{x}_{ij}^{\prime}\in \mathcal{B}(\boldsymbol{x}_{ij})}\ell (h(\boldsymbol {x}_{ij}^{\prime}),y_{ij})$ , where  $\Sigma_{ij}\sim$  Uniform(\{\pm1\}) and  $\pmb{\Sigma}$  has independent entries;  $c$  is from the property of sup that  $\sup (a + b)\leq \sup (a) + \sup (b)$  and the fact that  $\pmb{\Sigma}$  has the same distribution as  $-\pmb{\Sigma}$ .
Similar as before, if we change one of the  $\{(x_{ij},y_{ij})\}_{i,j = 1}^{n}$ , the change of  $\Re_{tn}(\widetilde{\mathcal{G}})$  is at most  $\frac{U}{tn}$ , so by McDiarmid's inequality, with probability at least  $1 - \frac{\delta}{4}$ , we have:
$$
\underset {\widehat {\mathfrak {D}}} {\mathbb {E}} \left[ \Re_ {t n} (\widetilde {\mathcal {G}}) \right] \leq \Re_ {t n} (\widetilde {\mathcal {G}}) + U \sqrt {\frac {\ln 4 / \delta}{2 t n}}. \tag {A.5}
$$
Combine (A.4) and (A.5) we have: with probability at least  $1 - \frac{\delta}{2}$ ,
$$
\Psi^ {\mathcal {B}} (\widehat {\mathfrak {D}}) \leq 2 \Re_ {t n} (\widetilde {\mathcal {G}}) + 3 U \sqrt {\frac {\ln 4 / \delta}{2 t n}}. \tag {A.6}
$$
Combine (A.3) and (A.6) we have: with probability at least  $1 - \delta$ :
$$
\Phi^ {\mathcal {B}} (\mathfrak {D}) + \Psi^ {\mathcal {B}} (\widehat {\mathfrak {D}}) \leq 2 \Re_ {t} (\widetilde {\mathcal {G}}) + 2 \Re_ {t n} (\widetilde {\mathcal {G}}) + 3 U \sqrt {\frac {\ln 4 / \delta}{2 t}} + 3 U \sqrt {\frac {\ln 4 / \delta}{2 t n}}.
$$
By the property of supremum, we have: with probability at least  $1 - \delta$
$$
\begin{array}{l} \sup  _ {h \in \mathcal {H}} \left(\mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , h) - \mathcal {L} _ {\hat {\mathcal {D}}} ^ {\mathcal {B}} (\ell , h)\right) = \sup  _ {h \in \mathcal {H}} \left(\mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , h) - \mathcal {L} _ {\hat {p}} ^ {\mathcal {B}} (\ell , h) + \mathcal {L} _ {\hat {p}} ^ {\mathcal {B}} (\ell , h) - \mathcal {L} _ {\hat {\mathcal {D}}} ^ {\mathcal {B}} (\ell , h)\right) \\ \leq \sup  _ {h \in \mathcal {H}} \left(\mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , h) - \mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , h)\right) + \sup  _ {h \in \mathcal {H}} \left(\mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , h) - \mathcal {L} _ {\mathcal {D}} ^ {\mathcal {B}} (\ell , h)\right) \\ = \Phi^ {\mathcal {B}} (\mathfrak {D}) + \Psi^ {\mathcal {B}} (\widehat {\mathfrak {D}}) \\ \leq 2 \Re_ {t} (\widetilde {\mathcal {G}}) + 2 \Re_ {t n} (\widetilde {\mathcal {G}}) + 3 U \sqrt {\frac {\ln 4 / \delta}{2 t}} + 3 U \sqrt {\frac {\ln 4 / \delta}{2 t n}}. \\ \end{array}
$$
Proof of Corollary 3.2. The main idea of the proof is to use risk decomposition to reduce  $\mathcal{L}_p^{\mathcal{B}}(\ell, \hat{h}) - \mathcal{L}_p^{\mathcal{B}}(\ell, h^\star)$  to  $\mathcal{L}_p^{\mathcal{B}}(\ell, h) - \mathcal{L}_{\widehat{\mathcal{D}}}^{\mathcal{B}}(\ell, h)$  for some  $h$ . Then we have: with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , \hat {h}) - \mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , h ^ {\star}) = \mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , \hat {h}) - \mathcal {L} _ {\widehat {\mathcal {D}}} ^ {\mathcal {B}} (\ell , \hat {h}) + \mathcal {L} _ {\widehat {\mathcal {D}}} ^ {\mathcal {B}} (\ell , \hat {h}) - \mathcal {L} _ {\widehat {\mathcal {D}}} ^ {\mathcal {B}} (\ell , h ^ {\star}) + \mathcal {L} _ {\widehat {\mathcal {D}}} ^ {\mathcal {B}} (\ell , h ^ {\star}) - \mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , h ^ {\star}) \\ \overset {a} {\leq} \mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , \hat {h}) - \mathcal {L} _ {\widehat {\mathcal {D}}} ^ {\mathcal {B}} (\ell , \hat {h}) + \mathcal {L} _ {\widehat {\mathcal {D}}} ^ {\mathcal {B}} (\ell , h ^ {\star}) - \mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , h ^ {\star}) \\ \leq 2 \sup  _ {h \in \mathcal {H}} \left| \mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , h) - \mathcal {L} _ {\widehat {\mathcal {D}}} ^ {\mathcal {B}} (\ell , h) \right| \\ \stackrel {b} {\leq} 4 \Re_ {t} (\widetilde {\mathcal {G}}) + 4 \Re_ {t n} (\widetilde {\mathcal {G}}) + 3 U \sqrt {\frac {\ln 8 / \delta}{2 t}} + 3 U \sqrt {\frac {\ln 8 / \delta}{2 t n}}, \\ \end{array}
$$