SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\mathcal{L}_{\mathcal{D}}(h) \coloneqq \mathbb{E}_{\mathbf{z} \sim \mathcal{D}} \ell(h, \mathbf{z})$  and  $\mathcal{L}_S(h) \coloneqq \frac{1}{n} \sum_{i=1}^{n} \ell(h, \mathbf{z}_i)$  are, respectively, the population and empirical loss of a hypothesis  $h \in \mathcal{H}$ , and  $\mathcal{R}_n(\mathcal{F}) \coloneqq \mathbb{E}_{S \sim \mathcal{D}^n} \widehat{\mathcal{R}}_n(\mathcal{F}, S)$  is the Rademacher complexity of  $\mathcal{F}$ .
Finally, we can prove Theorem 6.3 restated below
$$
\bar {R} = \inf  \left\{r > 0: \| \overline {{\mathbf {x}}} \| \leq r \text {w i t h p r o b a b i l i t y} 1 \right\}. \tag {151}
$$
$$
\mathbb {P} _ {(\mathbf {x}, y) \sim \mathcal {D}} [ y \neq \operatorname {s i g n} (\langle \mathbf {x}, \boldsymbol {\beta} _ {\text {s t e e r}} ^ {\infty} \rangle) ] \leq \frac {2 \bar {R} \| \boldsymbol {\beta} _ {0} \|}{\sqrt {n}} + \sqrt {\frac {\log (1 / \delta)}{2 n}}. \tag {152}
$$
Proof of Theorem 6.3. Let  $\mathcal{H} = \{\beta \in \mathbb{R}_G^{d_0} : \| \beta \| \leq \| \beta_0\|\}$  and with slight abuse of notation we identify  $\beta \in \mathcal{H}$  with the invariant linear map  $\mathbf{x} \mapsto \langle \mathbf{x}, \beta \rangle$ . According to Theorem 4.1, let  $\beta^* = \tau \beta_{\mathrm{steer}}^\infty$ ,  $\tau > 0$ , be the minimizer of (24). Since by assumption  $\beta_0$  also satisfies the constraint in (24), we have  $\| \beta^* \| \leq \| \beta_0\|$ , and hence  $\beta^* \in \mathcal{H}$ .
Consider the ramp loss  $\ell$  defined as
$$
\ell : \mathcal {H} \times (\mathbb {R} ^ {d _ {0}} \times \{\pm 1 \}) \rightarrow [ 0, 1 ], \quad (\boldsymbol {\beta}, (\mathbf {x}, y)) \mapsto \min  \{1, \max  \{0, 1 - y \langle \mathbf {x}, \boldsymbol {\beta} \rangle \} \}. \tag {153}
$$
It is easy to verify that  $\ell$  upper bounds the 0-1 loss: for all  $\beta \in \mathcal{H}$  and  $(\mathbf{x},y)\in \mathbb{R}^{d_0}\times \{\pm 1\}$ ,
$$
\ell (\boldsymbol {\beta}, (\mathbf {x}, y)) \geq \mathbb {1} _ {\{y \neq \operatorname {s i g n} (\langle \mathbf {x}, \boldsymbol {\beta} \rangle) \}}. \tag {154}
$$
$$
\begin{array}{l} \mathbb {P} _ {(\mathbf {x}, y) \sim \mathcal {D}} [ y \neq \operatorname {s i g n} \left(\langle \mathbf {x}, \boldsymbol {\beta} _ {\text {s t e e r}} ^ {\infty} \rangle\right) ] = \mathbb {P} _ {(\mathbf {x}, y) \sim \mathcal {D}} [ y \neq \operatorname {s i g n} \left(\langle \mathbf {x}, \boldsymbol {\beta} ^ {*} \rangle\right) ] (155) \\ = \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {D}} \mathbb {1} _ {\{y \neq \operatorname {s i g n} (\langle \mathbf {x}, \boldsymbol {\beta} ^ {*} \rangle) \}} (156) \\ \leq \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {D}} \ell \left(\boldsymbol {\beta} ^ {*}, (\mathbf {x}, y)\right) = \mathcal {L} _ {\mathcal {D}} \left(\boldsymbol {\beta} ^ {*}\right). (157) \\ \end{array}
$$
Since  $\mathcal{L}_S(\beta^*)$  is always 0 by definition (24) and  $\beta^{*}\in \mathcal{H}$ , we have by Lemma F.5 that
$$
\begin{array}{l} \mathbb {P} _ {(\mathbf {x}, y) \sim \mathcal {D}} [ y \neq \operatorname {s i g n} (\langle \mathbf {x}, \boldsymbol {\beta} _ {\text {s t e e r}} ^ {\infty} \rangle) ] \leq \mathcal {L} _ {\mathcal {D}} (\boldsymbol {\beta} ^ {*}) = \mathcal {L} _ {\mathcal {D}} (\boldsymbol {\beta} ^ {*}) - \mathcal {L} _ {S} (\boldsymbol {\beta} ^ {*}) \leq \sup  _ {\boldsymbol {\beta} \in \mathcal {H}} \mathcal {L} _ {\mathcal {D}} (\boldsymbol {\beta}) - \mathcal {L} _ {S} (\boldsymbol {\beta}) (158) \\ \leq 2 \mathcal {R} _ {n} (\mathcal {F}) + \sqrt {\frac {\log (1 / \delta)}{2 n}}, (159) \\ \end{array}
$$
where  $\mathcal{F} = \ell (\mathcal{H},\cdot) = \{(\mathbf{x},y)\mapsto \ell (\beta ,(\mathbf{x},y)): \beta \in \mathcal{H}\}$ . Therefore, to prove Theorem 6.3 it suffices to show that  $\mathcal{R}_n(\mathcal{F})\leq \frac{\overline{R}\|\beta_0\|}{\sqrt{n}}$ . To this end, let  $S = \{(\mathbf{x}_i,y_i):i\in [n]\}$  be a set of  $n$  labeled samples, and let  $S_{\mathbf{x}} = \{\mathbf{x}_{i}:i\in [n]\}$  be the corresponding set of unlabeled inputs, then
$$
\begin{array}{l} \widehat {\mathcal {R}} _ {n} (\mathcal {F}, S) = \frac {1}{n} \mathbb {E} _ {\sigma} \sup  _ {f \in \mathcal {F}} \sum_ {i = 1} ^ {n} \sigma_ {i} f \left(\mathbf {x} _ {i}, y _ {i}\right) (160) \\ = \frac {1}{n} \mathbb {E} _ {\boldsymbol {\sigma}} \sup  _ {\boldsymbol {\beta} \in \mathcal {H}} \sum_ {i = 1} ^ {n} \sigma_ {i} \ell (\boldsymbol {\beta}, (\mathbf {x} _ {i}, y _ {i})) (161) \\ = \frac {1}{n} \mathbb {E} _ {\boldsymbol {\sigma}} \sup  _ {\boldsymbol {\beta} \in \mathcal {H}} \sum_ {i = 1} ^ {n} \sigma_ {i} \min  \left\{1, \max  \left\{0, 1 - y _ {i} \langle \mathbf {x} _ {i}, \boldsymbol {\beta} \rangle \right\} \right\} (162) \\ = \frac {1}{n} \mathbb {E} _ {\boldsymbol {\sigma}} \sup  _ {\boldsymbol {\beta} \in \mathcal {H}} \sum_ {i = 1} ^ {n} \sigma_ {i} \Phi_ {i} (\langle \mathbf {x} _ {i}, \boldsymbol {\beta} \rangle), (163) \\ \end{array}
$$
where  $\Phi_i: \mathbb{R} \to \mathbb{R}, a \mapsto \min \{1, \max \{0, 1 - y_i a\}\}$ , is 1-Lipschitz. Lemma F.3 thus implies that
$$
\widehat {\mathcal {R}} _ {n} (\mathcal {F}, S) \leq \widehat {\mathcal {R}} _ {n} (\mathcal {H}, S _ {\mathbf {x}}). \tag {164}
$$
Using Lemma F.4 and the fact that  $\overline{\mathbf{x}}\leq \overline{R}$  with probability 1 over  $(\mathbf{x},y)\sim \mathcal{D}$ , we arrive at
$$
\mathcal {R} _ {n} (\mathcal {F}) = \mathbb {E} _ {S \sim \mathcal {D} ^ {n}} \widehat {\mathcal {R}} _ {n} (\mathcal {F}, S) \leq \mathbb {E} _ {S \sim \mathcal {D} ^ {n}} \widehat {\mathcal {R}} _ {n} (\mathcal {H}, S _ {\mathbf {x}}) \leq \frac {\bar {R} \| \beta_ {0} \|}{\sqrt {n}}. \tag {165}
$$
This completes the proof of Theorem 6.3.