SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathscr {A} _ {\mathcal {G}} (\vartheta) = \left\| \mathcal {G} - \iota \mathcal {G} _ {\vartheta} \right\| _ {L _ {\nu} ^ {2}} ^ {2} + \vartheta \| \mathcal {G} _ {\vartheta} \| _ {\mathcal {H}} ^ {2} \\ = \sum_ {j \in \mathbb {N}} \frac {\vartheta^ {2}}{(\lambda_ {j} + \vartheta) ^ {2}} \langle e _ {j}, \mathcal {G} \rangle_ {L _ {\nu} ^ {2}} ^ {2} + \sum_ {j \in \mathbb {N}} \frac {\vartheta \lambda_ {j}}{(\lambda_ {j} + \vartheta) ^ {2}} \langle e _ {j}, \mathcal {G} \rangle_ {L _ {\nu} ^ {2}} ^ {2} \\ = \sum_ {j \in \mathbb {N}} \left(\frac {\vartheta}{\lambda_ {j} + \vartheta}\right) \langle e _ {j}, \mathcal {G} \rangle_ {L _ {\nu} ^ {2}} ^ {2}. \\ \end{array}
$$
Using  $\vartheta / (\lambda_j + \vartheta) \leq 1$  for each  $j \in \mathbb{N}$  and  $\mathcal{G} \in \overline{\mathcal{H}}^{L_\nu^2}$  (the  $L_\nu^2$ -closure of  $\mathcal{H}$ ), it follows that
$$
\mathscr {A} _ {\mathcal {G}} (\vartheta) = \sum_ {\{j \in \mathbb {N} \mid \lambda_ {j} \neq 0 \}} \left(\frac {\vartheta}{\lambda_ {j} + \vartheta}\right) \langle e _ {j}, \mathcal {G} \rangle_ {L _ {\nu} ^ {2}} ^ {2} \rightarrow 0 \quad \text {a s} \quad \vartheta \rightarrow 0 \tag {B.8}
$$
by dominated convergence.
The rate of convergence of  $\mathcal{A}_{\mathcal{G}}$  to zero can be quantified under an additional regularity assumption.
Lemma B.3 (Convergence rate under Hölder source condition). Suppose  $\mathcal{G} \in \mathrm{Im}(\mathcal{K}^{r/2})$  for some  $r \geq 0$ . Then for any  $\vartheta > 0$ , it holds under the prevailing assumptions of this section that
$$
\mathscr {A} _ {\mathcal {G}} (\vartheta) \leq \| \mathcal {K} ^ {- r / 2} \mathcal {G} \| _ {L _ {\nu} ^ {2} (\mathcal {X}; \mathcal {Y})} ^ {2} \times \left\{ \begin{array}{l l} \vartheta^ {r}, & \text {i f} r \in [ 0, 1 ], \\ \vartheta \| \mathcal {K} \| _ {\mathcal {L} \left(L _ {\nu} ^ {2}\right)} ^ {r - 1}, & \text {i f} r > 1. \end{array} \right. \tag {B.9}
$$
Proof. The proof closely follows the argument of Smale and Zhou [42, Thm. 4, p. 295]. By hypothesis, there exists  $F_{\mathcal{G}} \in L_{\nu}^{2}(\mathcal{X};\mathcal{Y})$  such that  $\mathcal{G} = \mathcal{K}^{r / 2}F_{\mathcal{G}}$ . Then
$$
\mathcal {A} _ {\mathcal {G}} (\vartheta) = \sum_ {j \in \mathbb {N}} \frac {\vartheta \lambda_ {j} ^ {r}}{\lambda_ {j} + \vartheta} \langle e _ {j}, F _ {\mathcal {G}} \rangle_ {L _ {\nu} ^ {2}} ^ {2} \leq \left(\sup _ {j \in \mathbb {N}} \frac {\vartheta \lambda_ {j} ^ {r}}{\lambda_ {j} + \vartheta}\right) \| F _ {\mathcal {G}} \| _ {L _ {\nu} ^ {2}} ^ {2}.
$$
For any  $j \in \mathbb{N}$ , the argument of the supremum equals
$$
\frac {\vartheta \lambda_ {j} ^ {r}}{\lambda_ {j} + \vartheta} = \left(\frac {\lambda_ {j}}{\lambda_ {j} + \vartheta}\right) \left(\frac {\lambda_ {j}}{\lambda_ {j} + \vartheta}\right) ^ {r - 1} \frac {\vartheta}{(\lambda_ {j} + \vartheta) ^ {1 - r}} = \vartheta^ {r} \left(\frac {\lambda_ {j}}{\lambda_ {j} + \vartheta}\right) ^ {r} \left(\frac {\vartheta}{\lambda_ {j} + \vartheta}\right) ^ {1 - r}.
$$
This is bounded above by  $\vartheta^r$  for all  $j\in \mathbb{N}$  if  $r\geq 0$  and  $r\leq 1$ . Otherwise,  $r > 1$  and
$$
\frac {\vartheta \lambda_ {j} ^ {r}}{\lambda_ {j} + \vartheta} = \vartheta \lambda_ {j} ^ {r - 1} \left(\frac {\lambda_ {j}}{\lambda_ {j} + \vartheta}\right) \leq \vartheta \lambda_ {j} ^ {r - 1}.
$$
Taking the supremum over  $j \in \mathbb{N}$  completes the proof.
In Lem. B.3,  $\mathcal{G}$  satisfies  $\mathcal{G} \in \mathcal{H}$  if  $r \geq 1$ , and the rate of convergence of  $\mathcal{A}_{\mathcal{G}}(\vartheta)$  is at least as fast as  $O(\vartheta)$  as  $\vartheta \to 0$ . When  $r \in [0,1)$ , then  $\mathcal{G} \notin \mathcal{H}$  and the rate becomes slower than linear in  $\vartheta$ .
C Proofs for section 3
In this section, we prove Thm. 3.4 and its main consequences.
Proof of Theorem 3.4. Under the hypotheses, (4.2) in Prop. 4.1 holds with probability at least  $1 - \delta$  provided that  $M \geq \lambda^{-1} \log(16 / \delta)$  and  $N \geq \lambda^{-2} \log(8 / \delta)$ . That is,  $\mathcal{R}_N(\widehat{\alpha}; \mathcal{G}) \leq \mathcal{R}_N^\lambda(\widehat{\alpha}; \mathcal{G}) \leq \beta \lambda$ , where  $\beta = \beta(\rho, \lambda, \mathcal{G}_{\mathcal{H}}, \eta)$  is given by (3.3). In particular,  $\widehat{\alpha} \in \mathcal{A}_{\beta}$  (4.3) on the same event (by Cor. 4.2). It follows that, on this event,
$$
\mathcal {R} (\widehat {\alpha}; \mathcal {G}) - \mathcal {R} _ {N} (\widehat {\alpha}; \mathcal {G}) \leq \sup  _ {\alpha \in \mathcal {A} _ {\beta}} \left| \mathcal {R} _ {N} (\alpha ; \mathcal {G}) - \mathcal {R} (\alpha ; \mathcal {G}) \right|.
$$
Application of Prop. 4.7 shows that the right hand side of the above display is bounded above by
$$
3 2 \sqrt {6} e ^ {3 / 2} \big (\| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}} ^ {2} + \| \varphi \| _ {L ^ {\infty}} ^ {2} \beta (\rho , \lambda , \mathcal {G} _ {\mathcal {H}}, \eta) \big) \lambda \leq 7 9 e ^ {3 / 2} \big (\| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}} ^ {2} + \| \varphi \| _ {L ^ {\infty}} ^ {2} \beta (\rho , \lambda , \mathcal {G} _ {\mathcal {H}}, \eta) \big) \lambda
$$
with probability at least  $1 - \delta$  because  $N \geq \lambda^{-2} \log(8 / \delta) \geq \log(2 / \delta)$ . Recalling (4.1), using  $1 \leq 79e^{3/2}$ , and applying a union bound completes the proof.
Proof of Corollary 3.8. Since  $\mathcal{G} = \rho +\mathcal{G}_{\mathcal{H}}$  , we compute
$$
\begin{array}{l} \mathcal {R} (\widehat {\alpha}; \mathcal {G} _ {\mathcal {H}}) = \left\| \mathcal {G} _ {\mathcal {H}} - \Phi (\cdot ; \widehat {\alpha}) \right\| _ {L _ {\nu} ^ {2}} ^ {2} = \left\| - \rho + [ \mathcal {G} - \Phi (\cdot ; \widehat {\alpha}) ] \right\| _ {L _ {\nu} ^ {2}} ^ {2} \\ \leq 2 \| \rho \| _ {L _ {\nu} ^ {2}} ^ {2} + 2 \| \mathcal {G} - \Phi (\cdot ; \widehat {\alpha}) \| _ {L _ {\nu} ^ {2}} ^ {2} \\ = 2 \mathbb {E} _ {u \sim \nu} \| \rho (u) \| _ {\mathcal {Y}} ^ {2} + 2 \mathscr {R} (\widehat {\alpha}; \mathcal {G}). \\ \end{array}
$$
By (3.2) and (3.3), we see that the term  $\mathbb{E}\| \rho (u)\|_{\mathcal{Y}}^2$  also appears in the upper bound for  $\mathcal{R}(\widehat{\alpha};\mathcal{G})$ . Collecting like terms and enlarging constants proves the assertion.
Proof of Theorem 3.10. For  $k \in \mathbb{N}$  and  $\delta \in (0,1)$ , the trained RFM satisfies  $\| \mathcal{G} - \Phi(\cdot; \widehat{\alpha}^{(k)}) \|_{L_\nu^2} \leq c s_k$  for some deterministic constant  $c > 0$ , where the sequence  $s_n \to 0$  as  $n \to \infty$  is given by the right hand side of (3.9) with  $\lambda = \lambda_n$  for each  $n \in \mathbb{N}$  (by Lem. B.2). This inequality holds with probability at least  $1 - \delta$  if  $M \gtrsim \lambda_k^{-1} \log(2/\delta)$  and  $N \gtrsim \lambda_k^{-2} \log(2/\delta)$  by Thm. 3.4. Now choose  $\delta = \lambda_k$ . Then
$$
\sum_ {k \in \mathbb {N}} \Pr \left\{\| \mathcal {G} - \Phi (\cdot ; \widehat {\alpha} ^ {(k)}) \| _ {L _ {\nu} ^ {2}} > c s _ {k} \right\} \leq \sum_ {k \in \mathbb {N}} \lambda_ {k} <   \infty .
$$
The first Borel-Cantelli lemma establishes that there exists an  $\mathbb{N}$ -valued random variable  $k_{0}$  such that  $\| \mathcal{G} - \Phi (\cdot ;\widehat{\alpha}^{(k)})\|_{L_\nu^2}\leq c s_k$  for all  $k > k_0$  almost surely. This implies the desired result.
Proof of Theorem 3.12. Application of Thm. 3.4 leads to the high probability bound (3.9) by the same argument from Sect. 3.3. Using that  $\lambda \lesssim 1$  and Lem. B.3 proves the assertion.
Proof of Corollary 3.13. Apply Thm. 3.12 to get a high probability bound. Choose  $\lambda = \lambda_k = \delta$ . Then the proof follows that of Thm. 3.10 except with  $\{s_k\}$  replaced by  $\{\lambda_k^{(r\wedge 1) / 2}\}$ .
D Proofs for section 4
This section provides proofs of the error bounds for the regularized empirical risk (SM D.1) and the generalization gap (SM D.2).
D.1 Proofs for subsection 4.1: Bounding the regularized empirical risk
We now prove Prop. 4.1. Supporting results used in the proof appear after the argument in the subsequent subsections (SM D.1.1, D.1.2, and D.1.3).
Proof of Proposition 4.1. Our starting point is (4.5). We first note by linearity that
$$
\begin{array}{l} \frac {2}{N} \sum_ {n = 1} ^ {N} \langle \eta_ {n}, \Phi (u _ {n}; \widehat {\alpha}) \rangle_ {\mathcal {Y}} = \| \widehat {\alpha} \| _ {M} \left(\frac {2}{N} \sum_ {n = 1} ^ {N} \langle \eta_ {n}, \Phi (u _ {n}; \widehat {\alpha} / \| \widehat {\alpha} \| _ {M}) \rangle_ {\mathcal {Y}}\right) \\ \leq \| \widehat {\alpha} \| _ {M} \left(2 \sup  _ {\alpha^ {\prime} \in \mathcal {A} _ {1}} \left| \frac {1}{N} \sum_ {n = 1} ^ {N} \langle \eta_ {n}, \Phi (u _ {n}; \alpha^ {\prime}) \rangle_ {\mathcal {Y}} \right|\right), \\ \end{array}
$$
where  $\mathcal{A}_1 = \{\alpha' \in \mathbb{R}^M \mid \| \alpha' \|_M^2 \leq 1\}$ , provided that  $\| \widehat{\alpha} \|_M > 0$ . Otherwise, the inequality in the above display holds trivially. Next, we define
$$
t := \frac {1}{M} \sum_ {m = 1} ^ {M} \left| \widehat {\alpha} _ {m} \right| ^ {2} = \left\| \widehat {\alpha} \right\| _ {M} ^ {2}, \tag {D.1}
$$
$$
A _ {N, M} ^ {\lambda} := \mathcal {R} _ {N} ^ {\lambda} (\alpha ; \mathcal {G}) + \frac {2}{N} \sum_ {n = 1} ^ {N} \langle - \eta_ {n}, \Phi (u _ {n}; \alpha) \rangle_ {\mathcal {Y}}, \quad \text {a n d} \tag {D.2}
$$
$$
c _ {N, M} := \left(2 \sup  _ {\alpha^ {\prime} \in \mathcal {A} _ {1}} \left| \frac {1}{N} \sum_ {n = 1} ^ {N} \langle \eta_ {n}, \Phi \left(u _ {n}; \alpha^ {\prime}\right) \rangle \mathcal {Y} \right|\right) ^ {2}. \tag {D.3}
$$
The inequality in (4.5) and the arithmetic-mean-geometric-mean inequality imply that
$$
\lambda t \leq \mathscr {R} _ {N} ^ {\lambda} (\widehat {\alpha}; \mathcal {G}) \leq A _ {N, M} ^ {\lambda} + \sqrt {c _ {N , M} t} \leq A _ {N, M} ^ {\lambda} + \frac {1}{2} \lambda^ {- 1} c _ {N, M} + \frac {1}{2} \lambda t. \tag {D.4}
$$
Subtracting  $\lambda t / 2$  from both sides and multiplying through by  $2\lambda^{-1}$  yields
$$
t \leq 2 \lambda^ {- 1} A _ {N, M} ^ {\lambda} + \lambda^ {- 2} c _ {N, M}. \tag {D.5}
$$
Substituting (D.5) back into the right-most side of (D.4) gives
$$
\mathcal {R} _ {N} ^ {\lambda} (\widehat {\alpha}; \mathcal {G}) \leq 2 A _ {N, M} ^ {\lambda} + \lambda^ {- 1} c _ {N, M}. \tag {D.6}
$$
All of the above calculations hold with probability one. To complete our estimate of  $\mathcal{R}_N^\lambda (\widehat{\alpha};\mathcal{G})$ , it remains to upper bound the  $A_{N,M}^{\lambda}$  (D.2) and  $c_{N,M}$  (D.3) terms. We begin with the latter.
Lemmas D.4 and D.5 (with  $t = 1$ ) and (A.7) show that
$$
\begin{array}{l} \sqrt {c _ {N , M}} \leq \frac {4 \| \eta \| _ {\psi_ {1} (\mathcal {Y})} \| \varphi \| _ {L ^ {\infty}}}{\sqrt {N}} + 1 6 e ^ {3 / 2} \| \eta \| _ {\psi_ {1} (\mathcal {Y})} \| \varphi \| _ {L ^ {\infty}} \sqrt {\frac {\log (1 / \delta)}{N}} \tag {D.7} \\ \leq 1 6 e ^ {3 / 2} \| \eta \| _ {\psi_ {1} (\mathcal {Y})} \| \varphi \| _ {L ^ {\infty}} \sqrt {\frac {6 \log (2 / \delta)}{N}} \\ \end{array}
$$
with probability at least  $1 - \delta$  if  $N\geq \log (2 / \delta)\geq \log (1 / \delta)$ . We used the inequalities  $4\leq 16e^{3 / 2}$ ,  $\sqrt{a} +\sqrt{b}\leq \sqrt{2(a + b)}$ , and  $1\leq 2\log (2 / \delta)$  to get to the last line.
Continuing, we bound  $A_{N,M}^{\lambda}$ . It has several error contributions originating from two terms. The first term in (D.2) is  $\mathcal{R}_N^\lambda (\alpha ;\mathcal{G})$ , where  $\alpha \in \mathbb{R}^{M}$  is arbitrary. By Assumption 3.2,  $\mathcal{G} = \rho +\mathcal{G}_{\mathcal{H}}$  so that
$$
\mathcal {R} _ {N} ^ {\lambda} (\alpha ; \mathcal {G}) \leq \lambda \| \alpha \| _ {M} ^ {2} + 2 \mathcal {R} _ {N} (\alpha ; \mathcal {G} _ {\mathcal {H}}) + \frac {2}{N} \sum_ {n = 1} ^ {N} \| \rho (u _ {n}) \| _ {\mathcal {Y}} ^ {2} \leq 2 \mathcal {R} _ {N} ^ {\lambda} (\alpha ; \mathcal {G} _ {\mathcal {H}}) + \frac {2}{N} \sum_ {n = 1} ^ {N} \| \rho (u _ {n}) \| _ {\mathcal {Y}} ^ {2}. \tag {D.8}
$$
By Lem. 4.5, it holds with probability at least  $1 - \delta$  that
$$
\frac {2}{N} \sum_ {n = 1} ^ {N} \| \rho (u _ {n}) \| _ {\mathcal {Y}} ^ {2} \leq 4 \mathbb {E} _ {u} \| \rho (u) \| _ {\mathcal {Y}} ^ {2} + \frac {9 \| \rho \| _ {L _ {\nu} ^ {\infty}} ^ {2} \log (2 / \delta)}{N}. \tag {D.9}
$$
Next, we bound the first term on the right hand side in (D.8). Since  $\mathcal{G}_{\mathcal{H}} \in \mathcal{H}$ , there exists  $\alpha_{\mathcal{H}} \in L_{\mu}^{2}(\Theta; \mathbb{R})$  such that
$$
\mathcal {G} _ {\mathcal {H}} = \mathbb {E} _ {\theta \sim \mu} \big [ \alpha_ {\mathcal {H}} (\theta) \varphi (\cdot ; \theta) \big ] \quad \mathrm {a n d} \quad \| \mathcal {G} _ {\mathcal {H}} \| _ {\mathcal {H}} ^ {2} = \mathbb {E} _ {\theta \sim \mu} | \alpha_ {\mathcal {H}} (\theta) | ^ {2}.
$$
With  $\alpha_{\mathcal{H}}$  as in the above display, choose once and for all  $\alpha \equiv \alpha^{\star} \in \mathbb{R}^{M}$  as in (4.7). By Lem. 4.3,
$$
2 \mathcal {R} _ {N} ^ {\lambda} \left(\alpha^ {\star}; \mathcal {G} _ {\mathcal {H}}\right) \leq 1 6 2 \lambda \| \mathcal {G} _ {\mathcal {H}} \| _ {\mathcal {H}} ^ {2} \tag {D.10}
$$
with probability at least  $1 - \delta$  if  $M\geq \lambda^{-1}\log (4 / \delta)$ . On the same event,
$$
\| \alpha^ {\star} \| _ {M} ^ {2} \leq \mathbb {E} _ {\theta} | \alpha_ {\mathcal {H}} (\theta) | ^ {2} \left(1 + \frac {2 \log (2 / \delta)}{\lambda M} + \sqrt {\frac {2 \log (2 / \delta)}{\lambda M}}\right) \leq 5 \mathbb {E} _ {\theta} | \alpha_ {\mathcal {H}} (\theta) | ^ {2} = 5 \| \mathcal {G} _ {\mathcal {H}} \| _ {\mathcal {H}} ^ {2}
$$
by Lem. D.3. This fact and Lem. 4.6 (with  $\alpha \equiv \alpha^{\star}$  as in (4.7)) shows that the second and final term in  $A_{N,M}^{\lambda}$  (D.2) satisfies with probability at least  $1 - \delta$  the upper bound
$$
\frac {2}{N} \sum_ {n = 1} ^ {N} \left\langle - \eta_ {n}, \Phi \left(u _ {n}; \alpha^ {\star}\right) \right\rangle_ {\mathcal {Y}} \leq 4 0 e ^ {3 / 2} \| \mathcal {G} _ {\mathcal {H}} \| _ {\mathcal {H}} \| \eta \| _ {\psi_ {1} (\mathcal {Y})} \| \varphi \| _ {L ^ {\infty}} \sqrt {\frac {\log (2 / \delta)}{N}}. \tag {D.11}
$$
Combining the estimates (D.7), (D.8), (D.9), (D.10), and (D.11), recalling (D.6), and invoking the union bound, we deduce that if  $N \geq \lambda^{-2} \log(2 / \delta)$ , then
$$
\mathcal {R} _ {N} ^ {\lambda} (\widehat {\alpha}; \mathcal {G}) \leq C _ {0} \lambda + 8 \mathbb {E} _ {u \sim \nu} \| \rho (u) \| _ {\mathcal {V}} ^ {2} + 1 8 \| \rho \| _ {L _ {\nu} ^ {\infty}} ^ {2} \lambda^ {2}
$$
with probability at least  $1 - 4\delta$  , where
$$
\begin{array}{l} C _ {0} := 3 2 4 \| \mathcal {G} _ {\mathcal {H}} \| _ {\mathcal {H}} ^ {2} + 8 0 e ^ {3 / 2} \| \eta \| _ {\psi_ {1} (\mathcal {Y})} \| \varphi \| _ {L ^ {\infty}} \| \mathcal {G} _ {\mathcal {H}} \| _ {\mathcal {H}} + 1 5 3 6 e ^ {3} \| \eta \| _ {\psi_ {1} (\mathcal {Y})} ^ {2} \| \varphi \| _ {L ^ {\infty}} ^ {2} \\ \leq (3 2 4 + 4) \| \mathcal {G} _ {\mathcal {H}} \| _ {\mathcal {H}} ^ {2} + 1 9 3 6 e ^ {3} \| \eta \| _ {\psi_ {1} (\mathcal {Y})} ^ {2} \| \varphi \| _ {L ^ {\infty}} ^ {2}. \\ \end{array}
$$
In the last line, we used Young's inequality with  $\varepsilon = 1/8$ , that is,  $ab \leq \varepsilon a^2 / 2 + b^2 / (2\varepsilon)$  with  $a = 80e^{3/2}\|\eta\|_{\psi_1(\mathcal{V})}\|\varphi\|_{L^\infty}$  and  $b = \|\mathcal{G}_{\mathcal{H}}\|_{\mathcal{H}}$ . This proves the asserted upper bound.
D.1.1 Proofs for subsection 4.1.1: Bounding the approximation error
Given a function  $\alpha \in L_{\mu}^{2}(\Theta ;\mathbb{R})$  , we denote its cut-off at level  $T > 0$  by
$$
\theta \mapsto \alpha_ {\leq T} (\theta) := \alpha (\theta) \mathbb {1} _ {\{| \alpha (\theta) | \leq T \}}. \tag {D.12}
$$
This subsection is devoted to the proof of Lem. 4.3, which is based on the following three lemmas.
Lemma D.1. Suppose  $\mathcal{G} \in \mathcal{H}$  belongs to the RKHS  $\mathcal{H}$ . Let  $\alpha \in L_{\mu}^{2}(\Theta; \mathbb{R})$  be such that  $\mathcal{G} = \mathbb{E}_{\theta}[\alpha(\theta)\varphi(\cdot;\theta)]$ . Let  $u_1, \ldots, u_N \sim \nu$  be iid samples and  $\nu_{N} = \frac{1}{N}\sum_{n=1}^{N}\delta_{u_n}$  be the corresponding empirical measure. Then almost surely,
$$
\left\| \mathcal {G} - \mathbb {E} _ {\theta} [ \alpha_ {\leq T} (\theta) \varphi (\cdot ; \theta) ] \right\| _ {L _ {\nu_ {N}} ^ {2}} ^ {2} \leq \frac {\| \varphi \| _ {L ^ {\infty}} ^ {2} \left(\mathbb {E} _ {\theta} | \alpha (\theta) | ^ {2}\right) ^ {2}}{T ^ {2}} \quad f o r a l l \quad T > 0. \tag {D.13}
$$
Proof. Fix  $u \in \mathcal{X}$  and define  $\alpha_{>T} \coloneqq \alpha - \alpha_{< T}$ . The claim follows from
$$
\| \mathcal {G} (u) - \mathbb {E} _ {\theta} [ \alpha_ {\le T} (\theta) \varphi (u; \theta) ] \| _ {\mathcal {Y}} ^ {2} = \| \mathbb {E} _ {\theta} [ \alpha_ {> T} (\theta) \varphi (u; \theta) ] \| _ {\mathcal {Y}} ^ {2} \leq \| \varphi \| _ {L ^ {\infty}} ^ {2} (\mathbb {E} _ {\theta} | \alpha_ {> T} (\theta) |) ^ {2}
$$
and the observation that  $\mathbb{E}_{\theta}|\alpha_{>T}(\theta)|\leq \mathbb{E}_{\theta}|\alpha (\theta)|^2 /T$
The previous lemma controls the error incurred by truncating the coefficient function of elements in the RKHS. The next lemma provides a bound on sample average approximations of these truncations.
Lemma D.2. Let  $u_{1}, \ldots, u_{N} \sim \nu$  be iid samples and let  $\nu_{N} = \frac{1}{N} \sum_{n=1}^{N} \delta_{u_{n}}$  denote the corresponding empirical measure. For  $\alpha \in L_{\mu}^{2}(\Theta; \mathbb{R})$ , let  $Z = Z(\theta)$  be the  $L_{\nu_{N}}^{2}(\mathcal{X}; \mathcal{Y})$ -valued random variable defined for  $\theta \sim \mu$  by
$$
Z = \alpha_ {\leq T} (\theta) \varphi (\cdot ; \theta). \tag {D.14}
$$
If  $Z_{1}, \ldots, Z_{M}$  are  $M$  iid copies of  $Z$ , then it holds with probability at least  $1 - \delta$  that
$$
\left\| \frac {1}{M} \sum_ {m = 1} ^ {M} Z _ {m} - \mathbb {E} Z \right\| _ {L _ {\nu_ {N}} ^ {2}} ^ {2} \leq \frac {3 2 T ^ {2} \| \varphi \| _ {L ^ {\infty}} ^ {2} \log^ {2} (2 / \delta)}{M ^ {2}} + \frac {4 \| \varphi \| _ {L ^ {\infty}} ^ {2} \log (2 / \delta) \mathbb {E} _ {\theta} | \alpha (\theta) | ^ {2}}{M}. \tag {D.15}
$$
Proof. By boundedness of  $|\alpha_{\leq T}| \leq T$ , we have the trivial uniform upper bound  $\| Z_m \|_{L_{\nu_N}^2} \leq T \| \varphi \|_{L^\infty}$  for each  $m$ . The variance is bounded above as
$$
\sigma^ {2} := \mathbb {E} \| Z - \mathbb {E} Z \| _ {L _ {\nu_ {N}} ^ {2}} ^ {2} \leq \mathbb {E} \| Z \| _ {L _ {\nu_ {N}} ^ {2}} ^ {2} \leq \| \varphi \| _ {L ^ {\infty}} ^ {2} \mathbb {E} _ {\theta} | \alpha (\theta) | ^ {2}.
$$
By Lem. A.2 and Thm. A.1, it holds with probability at least  $1 - \delta$  that
$$
\left\| \frac {1}{M} \sum_ {m = 1} ^ {M} Z _ {m} - \mathbb {E} Z \right\| _ {L _ {\nu_ {N}} ^ {2}} \leq \frac {4 T \| \varphi \| _ {L ^ {\infty}} \log (2 / \delta)}{M} + \sqrt {\frac {2 \sigma^ {2} \log (2 / \delta)}{M}}.
$$
Squaring both sides and substitution of the above bound on  $\sigma^2$  yields the claimed estimate.
The third lemma below develops a high probability bound on the empirical approximation of the RKHS norm of truncated elements in the RKHS.
Lemma D.3. Let  $\alpha \in L_{\mu}^{2}(\Theta ;\mathbb{R})$  and  $\{\theta_m\} \sim \mu^{\otimes M}$ . With probability at least  $1 - \delta$ , it holds that
$$
\frac {1}{M} \sum_ {m = 1} ^ {M} | \alpha_ {\leq T} (\theta_ {m}) | ^ {2} \leq \mathbb {E} _ {\theta} | \alpha (\theta) | ^ {2} + \frac {4 T ^ {2} \log (2 / \delta)}{M} + \sqrt {\frac {2 T ^ {2} \mathbb {E} _ {\theta} | \alpha (\theta) | ^ {2} \log (2 / \delta)}{M}}. \tag {D.16}
$$
Proof. We apply Bernstein's inequality (A.2) to the random variable  $Z(\theta) \coloneqq |\alpha_{\leq T}(\theta)|^2$  with  $\theta \sim \mu$  and  $M \in \mathbb{N}$  iid copies  $Z_1, \ldots, Z_M$  of  $Z$  defined by  $Z_m = Z(\theta_m)$  for each  $m$ . We note that  $|Z| \leq T^2$  by definition. The variance of  $Z$  satisfies the upper bound
$$
\sigma^ {2} := \mathbb {E} | Z - \mathbb {E} Z | ^ {2} \leq \mathbb {E} | Z | ^ {2} = \mathbb {E} _ {\theta} | \alpha_ {\leq T} (\theta) | ^ {4} \leq T ^ {2} \mathbb {E} _ {\theta} | \alpha (\theta) | ^ {2}.
$$
It follows from Lem. A.2 and Thm. A.1 that
$$
\frac {1}{m} \sum_ {m = 1} ^ {M} Z _ {m} \leq \mathbb {E} Z + \frac {4 T ^ {2} \log (2 / \delta)}{M} + \sqrt {\frac {2 \sigma^ {2} \log (2 / \delta)}{M}}
$$
with probability at least  $1 - \delta$ . This in turn implies that
$$
\frac {1}{M} \sum_ {m = 1} ^ {M} | \alpha_ {\le T} (\theta_ {m}) | ^ {2} \le \mathbb {E} _ {\theta} | \alpha (\theta) | ^ {2} ] + \frac {4 T ^ {2} \log (2 / \delta)}{M} + \sqrt {\frac {2 T ^ {2} \mathbb {E} | \alpha (\theta) | ^ {2} \log (2 / \delta)}{M}}
$$
with at least the same probability. This is the claim.
We are now in a position to prove Lem. 4.3.
Proof of Lemma 4.3. Write  $T \coloneqq T(\lambda)$  and  $\alpha \coloneqq \alpha_{\mathcal{H}}$ . Next, define  $\alpha_{\leq T} \in L_{\mu}^{2}(\Theta; \mathbb{R})$  by
$$
\theta \mapsto \alpha_ {\leq T} (\theta) := \alpha (\theta) \mathbb {1} _ {\{| \alpha (\theta) | \leq T \}} = \left\{ \begin{array}{l l} \alpha (\theta), & \text {i f} | \alpha (\theta) | \leq T  , \\ 0, & \text {o t h e r w i s e}  . \end{array} \right.
$$
We define  $\alpha_{>T} \coloneqq \alpha \mathbb{1}_{\{| \alpha | > T\}}$  similarly, so that  $\alpha \equiv \alpha_{\leq T} + \alpha_{>T}$  holds true. Then for  $\theta_1, \ldots, \theta_M$ , we have  $\alpha^{\star} \in \mathbb{R}^{M}$  given by  $\alpha_{m}^{\star} = \alpha_{\leq T}(\theta_{m})$  for each  $m \in \{1, \ldots, M\}$ . We claim that  $\mathcal{R}_N^\lambda (\alpha^\star; \mathcal{G}) \leq (74 \| \varphi \|_{L^\infty}^2 + 7) \lambda \mathbb{E}_{\theta} |\alpha_{\mathcal{H}}(\theta)|^2$  with high probability, which implies the asserted bound (4.8). To see this, we make the error decomposition
$$
\begin{array}{l} \mathcal {R} _ {N} ^ {\lambda} \left(\alpha^ {\star}; \mathcal {G}\right) = \frac {1}{N} \sum_ {n = 1} ^ {N} \left\| \mathcal {G} \left(u _ {n}\right) - \frac {1}{M} \sum_ {m = 1} ^ {M} \alpha_ {\leq T} \left(\theta_ {m}\right) \varphi \left(u _ {n}; \theta_ {m}\right) \right\| _ {\mathcal {Y}} ^ {2} + \frac {\lambda}{M} \sum_ {m = 1} ^ {M} | \alpha_ {\leq T} \left(\theta_ {m}\right) | ^ {2} (D.17) \\ \leq \frac {2}{N} \sum_ {n = 1} ^ {N} \| \mathcal {G} (u _ {n}) - \mathbb {E} _ {\theta} [ \alpha_ {\leq T} (\theta) \varphi (u _ {n}; \theta) ] \| _ {\mathcal {Y}} ^ {2} (I) \\ + \frac {2}{N} \sum_ {n = 1} ^ {N} \left\| \frac {1}{M} \sum_ {m = 1} ^ {M} \alpha_ {\leq T} \left(\theta_ {m}\right) \varphi \left(u _ {n}; \theta_ {m}\right) - \mathbb {E} _ {\theta} \left[ \alpha_ {\leq T} (\theta) \varphi \left(u _ {n}; \theta\right) \right] \right\| _ {y} ^ {2} (II) \\ + \frac {\lambda}{M} \sum_ {m = 1} ^ {M} \left| \alpha_ {\leq T} \left(\theta_ {m}\right) \right| ^ {2}. (III) \\ \end{array}
$$
Each of the three terms (I)-(III) is estimated as follows.
By Lem. D.1, we can bound
$$
(I) \leq \frac {2 \| \varphi \| _ {L ^ {\infty}} ^ {2} (\mathbb {E} _ {\theta} | \alpha (\theta) | ^ {2}) ^ {2}}{T ^ {2}} = 2 \lambda \| \varphi \| _ {L ^ {\infty}} ^ {2} \mathbb {E} _ {\theta} | \alpha (\theta) | ^ {2}.
$$
Lem. D.2 delivers the bound
$$
\begin{array}{l} (\mathrm {I I}) \leq \frac {6 4 T ^ {2} \| \varphi \| _ {L _ {\infty}} ^ {2} \log (2 / \delta) ^ {2}}{M ^ {2}} + \frac {8 \| \varphi \| _ {L _ {\infty}} ^ {2} \log (2 / \delta) \mathbb {E} _ {\theta} | \alpha (\theta) | ^ {2}}{M} \\ = \lambda \mathbb {E} _ {\theta} | \alpha (\theta) | ^ {2} \left(\frac {6 4 \| \varphi \| _ {L _ {\infty}} ^ {2} \log (2 / \delta) ^ {2}}{\lambda^ {2} M ^ {2}} + \frac {8 \| \varphi \| _ {L _ {\infty}} ^ {2} \log (2 / \delta)}{\lambda M}\right) \\ \end{array}
$$
Last, Lem. D.3 yields
$$
\begin{array}{l} \left(\text {I I I}\right) \leq \lambda \mathbb {E} | \alpha (\theta) | ^ {2} + \frac {4 \lambda T ^ {2} \log (2 / \delta)}{M} + \lambda \sqrt {\frac {2 T ^ {2} \mathbb {E} | \alpha (\theta) | ^ {2} \log (2 / \delta)}{M}} \\ = \lambda \mathbb {E} _ {\theta} | \alpha (\theta) | ^ {2} \left(1 + \frac {4 \log (2 / \delta)}{\lambda M} + \sqrt {\frac {2 \log (2 / \delta)}{\lambda M}}\right) \\ \end{array}
$$
Combining the three estimates, it follows that if
$$
\frac {\log (2 / \delta)}{\lambda M} \leq 1,
$$