SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {R} _ {N} ^ {\lambda} (\alpha^ {\star}; \mathcal {G} _ {\mathcal {H}}) = \mathcal {R} _ {N} ^ {\lambda} (\alpha^ {\star}; \mathcal {G}) \leq (7 4 \| \varphi \| _ {L ^ {\infty}} ^ {2} + 7) \lambda \mathbb {E} _ {\theta} | \alpha_ {\mathcal {H}} (\theta) | ^ {2}
$$
with probability at least  $1 - 2\delta$ . We used the fact that  $\sqrt{2} \leq 2$ . This is the claimed upper bound.
D.1.2 Proof for subsection 4.1.2: Bounding the misspecification error
Recall that  $\rho \in L_{\nu}^{\infty}$  under Assumption 3.2. We now prove Lem. 4.5.
Proof of Lemma 4.5. Let  $Z_{1} = \| \rho (u_{1})\|_{\mathcal{V}}^{2}$ , which is uncentered. Almost surely,  $Z_{1}\leq \| \rho \|_{L^{\infty}_{\nu}}^{2}$  and
$$
\mathbb {E} | Z _ {1} - \mathbb {E} Z _ {1} | ^ {2} \leq \mathbb {E} Z _ {1} ^ {2} = \mathbb {E} _ {u \sim \nu} \| \rho (u) \| _ {\mathcal {Y}} ^ {4} \leq \| \rho \| _ {L _ {\nu} ^ {\infty}} ^ {2} \mathbb {E} _ {u \sim \nu} \| \rho (u) \| _ {\mathcal {Y}} ^ {2}.
$$
Thus with probability at least  $1 - \delta$ , Cor. A.2 and Thm. A.1 provide the bound
$$
\begin{array}{l} \frac {1}{N} \sum_ {n = 1} ^ {N} \| \rho (u _ {n}) \| _ {y} ^ {2} \leq \mathbb {E} _ {u} \| \rho (u) \| _ {y} ^ {2} + \frac {4 \| \rho \| _ {L _ {\nu} ^ {\infty}} ^ {2} \log (2 / \delta)}{N} + \sqrt {\frac {2 \mathbb {E} _ {u} \| \rho (u) \| _ {\mathcal {Y}} ^ {2} \| \rho \| _ {L _ {\nu} ^ {\infty}} ^ {2} \log (2 / \delta)}{N}} \\ \leq 2 \mathbb {E} _ {u} \| \rho (u) \| _ {\mathcal {Y}} ^ {2} + \frac {\frac {9}{2} \| \rho \| _ {L _ {\nu} ^ {\infty}} ^ {2} \log (2 / \delta)}{N}. \\ \end{array}
$$
To get the last inequality, we used the arithmetic-mean-geometric-mean inequality  $\sqrt{ab} \leq (a + b) / 2$  to obtain
$$
\sqrt {\left(2 \mathbb {E} _ {u} \| \rho (u) \| _ {\mathcal {Y}} ^ {2}\right) \left(\| \rho \| _ {L _ {\nu} ^ {\infty}} ^ {2} \log (2 / \delta) / N\right)} \leq \mathbb {E} _ {u} \| \rho (u) \| _ {\mathcal {Y}} ^ {2} + \frac {\frac {1}{2} \| \rho \| _ {L _ {\nu} ^ {\infty}} ^ {2} \log (2 / \delta)}{N}.
$$
Multiplying the penultimate chain of inequalities through by two completes the proof.
D.1.3 Proofs for subsection 4.1.3: Bounding the noise error
This subsection provides proofs for the lemmas used to control the error stemming from iid noise corrupting the output data as in Assumption 3.3. The estimates themselves could be improved by using (A.10) instead of (A.11) and by tracking the noise variance  $\mathbb{E}\| \eta \|_{\mathcal{Y}}^2$  instead of bounding it above by  $4\| \eta \|_{\psi_1(\mathcal{Y})}^2$ . This would be relevant in settings where the noise is small or tends to zero with the sample size. We defer such considerations to future work.
Proof of Lemma 4.6. Define  $Z_{n}(\alpha) \coloneqq \langle -\eta_{n}, \Phi(u_{n}; \alpha) \rangle_{\mathcal{V}}$  for each  $n$ . Conditioned on  $\{\theta_{m}\}$ , it holds that  $Z_{n}$  is an iid copy of  $Z_{1}$ . By the assumption  $\mathbb{E}[\eta_{1} \mid u_{1}] = 0$ , we have
$$
\begin{array}{l} \mathbb {E} Z _ {1} (\alpha) = \mathbb {E} _ {(u _ {1}, \eta_ {1})} [ \langle - \eta_ {1}, \Phi (u _ {1}; \alpha) \rangle_ {\mathcal {Y}} ] \\ = \mathbb {E} _ {u _ {1} \sim \nu} [ \mathbb {E} [ \langle - \eta_ {1}, \Phi (u _ {1}; \alpha) \rangle_ {\mathcal {Y}} | u _ {1} ] ] \tag {D.18} \\ = \mathbb {E} _ {u _ {1} \sim \nu} [ \langle - \mathbb {E} [ \eta_ {1} \mid u _ {1} ], \Phi (u _ {1}; \alpha) \rangle_ {\mathcal {Y}} ] \\ = 0. \\ \end{array}
$$
Next,
$$
\left| Z _ {1} (\alpha) \right| \leq \| \eta_ {1} \| _ {\mathcal {Y}} \| \Phi (u _ {1}; \alpha) \| _ {\mathcal {Y}} \leq \| \eta_ {1} \| _ {\mathcal {Y}} \| \alpha \| _ {M} \| \varphi \| _ {L ^ {\infty}}
$$
by two applications of the Cauchy-Schwarz inequality, one in  $\mathcal{V}$  and the other in  $\mathbb{R}^M$ . We deduce that  $\| Z_1(\alpha)\|_{\psi_1}\leq \| \eta_1\|_{\psi_1(\mathcal{V})}\| \alpha \| _M\| \varphi \|_{L^\infty}$ , conditioned on  $\{\theta_m\}$ . Prop. A.4, Thm. A.1 (Bernstein's inequality), and a similar argument to that in the proof of Cor. A.5 deliver the asserted bound.
The next two lemmas are used in the proof of Prop. 4.1 to control the third and final term in (4.5).
Lemma D.4 (Linear empirical process: Concentration). Fix  $t > 0$  and  $\delta \in (0,1)$ . Define
$$
Z _ {t} := \sup  _ {\alpha \in \mathcal {A} _ {t}} \left| \frac {1}{N} \sum_ {n = 1} ^ {N} \langle \eta_ {n}, \Phi (u _ {n}; \alpha) \rangle_ {\mathcal {Y}} \right|, \quad \text {w h e r e} \quad \mathcal {A} _ {t} := \left\{\alpha \in \mathbb {R} ^ {M} \mid \| \alpha \| _ {M} ^ {2} \leq t \right\}. \tag {D.19}
$$
If  $N \geq \log(1/\delta)$ , then conditioned on the realizations  $\{\theta_m\}$  in the family  $\Phi$  it holds that
$$
Z _ {t} \leq \mathbb {E} _ {\left\{\left(u _ {n}, \eta_ {n}\right) \right\}} \left[ Z _ {t} \right] + 8 e ^ {3 / 2} \| \eta_ {1} \| _ {\psi_ {1} (\mathcal {Y})} \| \varphi \| _ {L ^ {\infty}} \sqrt {\frac {\log (1 / \delta)}{N}} \sqrt {t} \tag {D.20}
$$
Proof. For any  $\alpha \in \mathcal{A}_t$ , we compute
$$
\begin{array}{l} | \langle \eta_ {1}, \Phi (u _ {1}; \alpha) \rangle_ {\mathcal {Y}} | = \left| \frac {1}{M} \sum_ {m = 1} ^ {M} \alpha_ {m} \langle \eta_ {1}, \varphi (u _ {1}; \theta_ {m}) \rangle_ {\mathcal {Y}} \right| \\ \leq \left(\frac {1}{M} \sum_ {m = 1} ^ {M} | \alpha_ {m} | ^ {2}\right) ^ {1 / 2} \left(\frac {1}{M} \sum_ {m = 1} ^ {M} \langle \eta_ {1}, \varphi (u _ {1}; \theta_ {m}) \rangle_ {\mathcal {Y}} ^ {2}\right) ^ {1 / 2} \\ \leq \sqrt {t} \left(\| \eta_ {1} \| _ {\mathcal {Y}} ^ {2} \frac {1}{M} \sum_ {m = 1} ^ {M} \| \varphi (u _ {1}; \theta_ {m}) \| _ {\mathcal {Y}} ^ {2}\right) ^ {1 / 2}. \\ \end{array}
$$
We used the Cauchy-Schwarz inequality twice. By the boundedness of  $\varphi$ , the above display gives
$$
\left\| \langle \eta_ {1}, \Phi (u _ {1}; \cdot) \rangle \right\| _ {\psi_ {1} (C (\mathcal {A} _ {t}; \mathbb {R}))} = \left\| \sup  _ {\alpha \in \mathcal {A} _ {t}} \big | \langle \eta_ {1}, \Phi (u _ {1}; \alpha) \rangle_ {\mathcal {Y}} \big | \right\| _ {\psi_ {1}} \leq \| \eta_ {1} \| _ {\psi_ {1} (\mathcal {Y})} \| \varphi \| _ {L ^ {\infty}} \sqrt {t}.
$$
The iid random variables  $\langle \eta_n, \Phi(u_n; \cdot) \rangle_{\mathcal{Y}} \colon \alpha \mapsto \langle \eta_n, \Phi(u_n; \alpha) \rangle_{\mathcal{Y}}$  (conditional on  $\{\theta_m\}$ ) are linear and hence continuous. Application of (A.11) in Cor. A.5 to  $\langle \eta_n, \Phi(u_n; \cdot) \rangle_{\mathcal{Y}}$  taking value in the separable Banach space  $C(\mathcal{A}_t; \mathbb{R})$  of continuous functions from compact set  $\mathcal{A}_t \subset \mathbb{R}^M$  into  $\mathbb{R}$ , equipped with the supremum norm, completes the proof.
The previous lemma gives a concentration bound for the linear empirical process and the next lemma estimates its expectation.
Lemma D.5 (Linear empirical process: Expectation). Fix  $t > 0$ . Define  $A_{t}$  as in Lem. D.4. Then
$$
\mathbb {E} _ {\left\{\left(u _ {n}, \eta_ {n}\right) \right\}} \sup  _ {\alpha \in \mathcal {A} _ {t}} \left| \frac {1}{N} \sum_ {n = 1} ^ {N} \langle \eta_ {n}, \Phi (u _ {n}; \alpha) \rangle_ {\mathcal {Y}} \right| \leq \frac {\left\| \eta_ {1} \right\| _ {L _ {\mathbb {P}} ^ {2} (\Omega ; \mathcal {Y})} \| \varphi \| _ {L ^ {\infty}}}{\sqrt {N}} \sqrt {t}. \tag {D.21}
$$
Proof. For any  $\alpha \in \mathcal{A}_t$ , the Cauchy-Schwarz inequality in  $\mathbb{R}^M$  delivers the bound
$$
\begin{array}{l} \left| \frac {1}{N} \sum_ {n = 1} ^ {N} \langle \eta_ {n}, \Phi (u _ {n}; \alpha) \rangle_ {\mathcal {Y}} \right| = \left| \frac {1}{M} \sum_ {m = 1} ^ {M} \alpha_ {m} \frac {1}{N} \sum_ {n = 1} ^ {N} \langle \eta_ {n}, \varphi (u _ {n}; \theta_ {m}) \rangle_ {\mathcal {Y}} \right| \\ \leq \left(\frac {1}{M} \sum_ {m = 1} ^ {M} | \alpha_ {m} | ^ {2}\right) ^ {1 / 2} \left(\frac {1}{M} \sum_ {m = 1} ^ {M} \left[ \frac {1}{N} \sum_ {n = 1} ^ {N} \langle \eta_ {n}, \varphi (u _ {n}; \theta_ {m}) \rangle_ {\mathcal {Y}} \right] ^ {2}\right) ^ {1 / 2}. \\ \end{array}
$$
Let the left hand side of (D.21) be denoted by  $\Xi_t$ . We next note that
$$
\begin{array}{l} \mathbb {E} _ {\left\{\left(u _ {n}, \eta_ {n}\right) \right\}} \left[ \langle \eta_ {n}, \varphi \left(u _ {n}; \theta_ {m}\right) \rangle_ {\mathcal {Y}} \right] = \mathbb {E} _ {u _ {n} \sim \nu} \left[ \mathbb {E} \left[ \langle \eta_ {n}, \varphi \left(u _ {n}; \theta_ {m}\right) \rangle_ {\mathcal {Y}} \mid u _ {n} \right] \right] \\ = \mathbb {E} _ {u _ {n} \sim \nu} [ \langle \mathbb {E} [ \eta_ {n} \mid u _ {n} ], \varphi (u _ {n}; \theta_ {m}) \rangle y ] \\ = 0. \\ \end{array}
$$
Using the independence of  $(u_{n},\eta_{n})$  and  $(u_{n^{\prime}},\eta_{n^{\prime}})$  for any two indices  $n\neq n'$ , together with the above observation, we thus obtain
$$
\begin{array}{l} \mathbb {E} _ {\left\{\left(u _ {n}, \eta_ {n}\right) \right\}} \left[ \langle \eta_ {n}, \varphi \left(u _ {n}; \theta_ {m}\right) \rangle_ {\mathcal {Y}} \langle \eta_ {n ^ {\prime}}, \varphi \left(u _ {n ^ {\prime}}; \theta_ {m}\right) \rangle_ {\mathcal {Y}} \right] \\ = \mathbb {E} _ {\left(u _ {n}, \eta_ {n}\right)} \left[ \left\langle \eta_ {n}, \varphi \left(u _ {n}; \theta_ {m}\right) \right\rangle_ {\mathcal {Y}} \right] \mathbb {E} _ {\left(u _ {n ^ {\prime}}, \eta_ {n ^ {\prime}}\right)} \left[ \left\langle \eta_ {n ^ {\prime}}, \varphi \left(u _ {n ^ {\prime}}; \theta_ {m}\right) \right\rangle_ {\mathcal {Y}} \right] \\  \\ \end{array}
$$
$$
\begin{array}{l} \Xi_ {t} \leq \frac {\sqrt {t}}{N} \mathbb {E} _ {\left\{\left(u _ {n}, \eta_ {n}\right) \right\}} \sqrt {\frac {1}{M} \sum_ {m = 1} ^ {M} \sum_ {n , n ^ {\prime} = 1} ^ {N} \langle \eta_ {n} , \varphi \left(u _ {n}; \theta_ {m}\right) \rangle_ {\mathcal {Y}} \langle \eta_ {n ^ {\prime}} , \varphi \left(u _ {n ^ {\prime}} ; \theta_ {m}\right) \rangle_ {\mathcal {Y}}} \\ \leq \frac {\sqrt {t}}{N} \sqrt {\frac {1}{M} \sum_ {m = 1} ^ {M} \sum_ {n , n ^ {\prime} = 1} ^ {N} \mathbb {E} _ {\left\{\left(u _ {n} , \eta_ {n}\right) \right\}} \left[ \left\langle \eta_ {n} , \varphi \left(u _ {n} ; \theta_ {m}\right) \right\rangle_ {\mathcal {Y}} \left\langle \eta_ {n ^ {\prime}} , \varphi \left(u _ {n ^ {\prime}} ; \theta_ {m}\right) \right\rangle_ {\mathcal {Y}} \right]} \\ = \frac {\sqrt {t}}{\sqrt {N}} \sqrt {\frac {1}{M} \sum_ {m = 1} ^ {M} \mathbb {E} _ {(u _ {1} , \eta_ {1})} \left\langle \eta_ {1} , \varphi (u _ {1} ; \theta_ {m}) \right\rangle_ {\mathcal {Y}} ^ {2}} \\ \leq \frac {\sqrt {t}}{\sqrt {N}} \| \eta_ {1} \| _ {L _ {\mathbb {P}} ^ {2} (\Omega ; \mathcal {Y})} \| \varphi \| _ {L ^ {\infty}}. \\ \end{array}
$$
We used Jensen's inequality in the second line, independence and the zero-mean property of the summands in the third line, and the Cauchy-Schwarz inequality in  $\mathcal{V}$  in the final line.
D.2 Proofs for subsection 4.2: Bounding the generalization gap
This subsection upper bounds the generalization gap with suprema techniques. We begin with the following empirical process concentration inequality. It gives uniform control on the difference between the empirical and population risk functionals. The process, as a function of its index  $\alpha$ , is quadratic because the RFM  $\Phi(\cdot; \alpha)$  is linear in  $\alpha$ .
Lemma D.6 (Quadratic empirical process: Concentration). Fix  $t > 0$  and  $\delta \in (0,1)$ . Define
$$
\begin{array}{l} Z _ {t} := \sup  _ {\alpha \in \mathcal {A} _ {t}} \left| \mathscr {R} _ {N} (\alpha ; \mathcal {G}) - \mathscr {R} (\alpha ; \mathcal {G}) \right| (D.22) \\ = \sup  _ {\alpha \in \mathcal {A} _ {t}} \left| \frac {1}{N} \sum_ {n = 1} ^ {N} \| \mathcal {G} (u _ {n}) - \Phi (u _ {n}; \alpha) \| _ {\mathcal {Y}} ^ {2} - \mathbb {E} _ {u \sim \nu} \| \mathcal {G} (u) - \Phi (u; \alpha) \| _ {\mathcal {Y}} ^ {2} \right|, (D.23) \\ \end{array}
$$
$$
\mathcal {A} _ {t} := \left\{\alpha \in \mathbb {R} ^ {M} \mid \| \alpha \| _ {M} ^ {2} \leq t \right\}. \tag {D.24}
$$
If  $N \geq \log(1/\delta)$ , then conditioned on the realizations  $\{\theta_m\}$  in the family  $\Phi$ , it holds that
$$
Z _ {t} \leq \mathbb {E} _ {\left\{u _ {n} \right\}} \left[ Z _ {t} \right] + 3 2 e ^ {3 / 2} \left(\| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}} ^ {2} + \| \varphi \| _ {L ^ {\infty}} ^ {2} t\right) \sqrt {\frac {\log (1 / \delta)}{N}} \tag {D.25}
$$
Proof. For any  $\alpha \in \mathcal{A}_t$  and  $n\in \{1,\ldots ,N\}$ , let
$$
X _ {n} (t, \alpha) := \| \mathcal {G} (u _ {n}) - \Phi (u _ {n}; \alpha) \| _ {\mathcal {Y}} ^ {2} - \mathbb {E} _ {u \sim \nu} \| \mathcal {G} (u) - \Phi (u; \alpha) \| _ {\mathcal {Y}} ^ {2}.
$$
We compute
$$
\begin{array}{l} | X _ {1} (t, \alpha) | \leq 2 \| \mathcal {G} (u _ {1}) \| _ {\mathcal {Y}} ^ {2} + 2 \mathbb {E} _ {u \sim \nu} \| \mathcal {G} (u) \| _ {\mathcal {Y}} ^ {2} + 2 \| \Phi (u _ {1}; \alpha) \| _ {\mathcal {Y}} ^ {2} + 2 \mathbb {E} _ {u \sim \nu} \| \Phi (u; \alpha) \| _ {\mathcal {Y}} ^ {2} \\ \leq 4 \| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}} ^ {2} + 4 \| \varphi \| _ {L ^ {\infty}} ^ {2} t. \\ \end{array}
$$
We used the fact that for any  $u \in \mathcal{X}$ $\nu$ -almost surely,  $\| \Phi(u; \alpha) \|_y^2 \leq t \| \varphi \|_{L^\infty}^2$  on the set  $\mathcal{A}_t$  (by the Cauchy-Schwarz inequality). This implies that
$$
\left| \left| X _ {1} (t, \cdot) \right| \right| _ {\psi_ {1} (C (\mathcal {A} _ {t}; \mathbb {R}))} = \left| \left| \sup _ {\alpha \in \mathcal {A} _ {t}} | X _ {1} (t, \alpha) | \right| \right| _ {\psi_ {1}} \leq 4 \| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}} ^ {2} + 4 \| \varphi \| _ {L ^ {\infty}} ^ {2} t.
$$
The  $X_{n}(t,\cdot)$  do indeed belong to  $C(\mathcal{A}_t;\mathbb{R})$  almost surely, as they can be written as a sum of affine and quadratic forms on  $\mathbb{R}^M$  in the  $\alpha$  variable. Application of (A.11) in Cor. A.5 (taking the separable Banach space to be  $C(\mathcal{A}_t;\mathbb{R})$  equipped with the supremum norm) completes the proof.
Since the supremum concentrates around its mean, it remains to show that its mean is small as a function of the sample size. The next lemma does this with Rademacher symmetrization.
Lemma D.7 (Quadratic empirical process: Expectation). Fix  $t > 0$ . Define  $Z_{t}$  as in Lem. D.6. Conditioned on the realizations  $\{\theta_m\}$  in the family  $\Phi$ , it holds that
$$
\mathbb {E} _ {\left\{u _ {n} \right\}} \left[ Z _ {t} \right] \leq \frac {4 \| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}} ^ {2}}{\sqrt {N}} + \frac {4 \| \varphi \| _ {L ^ {\infty}} ^ {2}}{\sqrt {N}} t. \tag {D.26}
$$
Proof. By Gine-Zinn symmetrization [see, e.g., 46, Sect. 4.2, Prop. 4.11, pp. 107-108],
$$
\mathbb {E} _ {\left\{u _ {n} \right\}} \left[ Z _ {t} \right] \leq 2 \mathbb {E} \sup  _ {\alpha \in \mathcal {A} _ {t}} \left| \frac {1}{N} \sum_ {n = 1} ^ {N} \varepsilon_ {n} \| \mathcal {G} (u _ {n}) - \Phi (u _ {n}; \alpha) \| _ {\mathcal {Y}} ^ {2} \right|, \quad \text {w h e r e} \quad \varepsilon_ {n} \stackrel {{\mathrm {i i d}}} {{\sim}} \operatorname {U n i f} \left(\{+ 1, - 1 \}\right),
$$
because the original summands (conditioned on  $\{\theta_m\}$ ) are independent. The expectation on the right is interpreted as the conditional expectation given  $\{\theta_m\}$  (i.e.,  $\mathbb{E}_{\{u_n\}, \{\varepsilon_n\}}$  over the data and Rademacher variables only). The right hand side is the Rademacher complexity of the RF model class composed with the square loss. Expanding the square, it is bounded above by
$$
\begin{array}{l} 2 \mathbb {E} _ {\{u _ {n} \}, \{\varepsilon_ {n} \}} \left| \frac {1}{N} \sum_ {n = 1} ^ {N} \varepsilon_ {n} \| \mathcal {G} (u _ {n}) \| _ {\mathcal {Y}} ^ {2} \right| (I) \\ + 4 \mathbb {E} _ {\{u _ {n} \}, \{\varepsilon_ {n} \}} \sup  _ {\alpha \in \mathcal {A} _ {t}} \left| \frac {1}{N} \sum_ {n = 1} ^ {N} \varepsilon_ {n} \langle \mathcal {G} (u _ {n}), \Phi (u _ {n}; \alpha) \rangle_ {\mathcal {Y}} \right| (II) \\ + 2 \mathbb {E} _ {\left\{u _ {n} \right\}, \left\{\varepsilon_ {n} \right\}} \sup  _ {\alpha \in \mathcal {A} _ {t}} \left| \frac {1}{N} \sum_ {n = 1} ^ {N} \varepsilon_ {n} \| \Phi (u _ {n}; \alpha) \| _ {\mathcal {Y}} ^ {2} \right|. (III) \\ \end{array}
$$
We now estimate each term. The first term (I) satisfies the standard Monte Carlo bound
$$
(I) \leq 2 \left(\mathbb {E} \left| \frac {1}{N} \sum_ {n = 1} ^ {N} \varepsilon_ {n} \| \mathcal {G} (u _ {n}) \| _ {\mathcal {Y}} ^ {2} \right| ^ {2}\right) ^ {1 / 2} = \frac {2}{\sqrt {N}} \left(\frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} \| \mathcal {G} (u _ {n}) \| _ {\mathcal {Y}} ^ {4}\right) ^ {1 / 2} \leq \frac {2 \| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}} ^ {2}}{\sqrt {N}}.
$$
For the second term (II), we begin by estimating the empirical average on the set  $\mathcal{A}_t$  as
$$
\begin{array}{l} \left| \frac {1}{N} \sum_ {n = 1} ^ {N} \varepsilon_ {n} \langle \mathcal {G} (u _ {n}), \Phi (u _ {n}; \alpha) \rangle_ {\mathcal {Y}} \right| = \left| \frac {1}{M} \sum_ {m = 1} ^ {M} \alpha_ {m} \left(\frac {1}{N} \sum_ {n = 1} ^ {N} \varepsilon_ {n} \langle \mathcal {G} (u _ {n}), \varphi (u _ {n}; \theta_ {m}) \rangle_ {\mathcal {Y}}\right) \right| \\ \leq \sqrt {t} \left(\frac {1}{M} \sum_ {m = 1} ^ {M} \left| \frac {1}{N} \sum_ {n = 1} ^ {N} \varepsilon_ {n} \langle \mathcal {G} (u _ {n}), \varphi (u _ {n}; \theta_ {m}) \rangle_ {\mathcal {Y}} \right| ^ {2}\right) ^ {1 / 2} \\ \end{array}
$$
by the Cauchy-Schwarz inequality in  $\mathbb{R}^M$ . We deduce by Jensen's inequality and independence that
$$
\begin{array}{l} \left. \mathrm {(I I)} \leq \frac {4 \sqrt {t}}{N} \left(\frac {1}{M} \sum_ {m = 1} ^ {M} \sum_ {n, n ^ {\prime} = 1} ^ {N} \mathbb {E} \left[ \varepsilon_ {n} \varepsilon_ {n ^ {\prime}} \right] \mathbb {E} _ {\left\{u _ {n} \right\}} \left[ \langle \mathcal {G} (u _ {n}), \varphi (u _ {n}; \theta_ {m}) \rangle_ {\mathcal {Y}} \langle \mathcal {G} (u _ {n ^ {\prime}}), \varphi (u _ {n ^ {\prime}}; \theta_ {m}) \rangle_ {\mathcal {Y}} \right]\right) ^ {1 / 2} \right. \\ = \frac {4 \sqrt {t}}{N} \left(\frac {1}{M} \sum_ {m = 1} ^ {M} \sum_ {n = 1} ^ {N} \mathbb {E} _ {u \sim \nu} \langle \mathcal {G} (u), \varphi (u; \theta_ {m}) \rangle_ {\mathcal {Y}} ^ {2}\right) ^ {1 / 2}. \\ \end{array}
$$
A final application of the Cauchy-Schwarz inequality in  $\mathcal{V}$  in the last line shows that the second term (II) is bounded above by  $4\sqrt{t}\| \mathcal{G}\|_{L_\nu^\infty}\| \varphi \|_{L^\infty} / \sqrt{N}$ . By Young's inequality  $ab\leq a^2 /2 + b^2 /2$ , we further bound
$$
\frac {4 \| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}} \| \varphi \| _ {L ^ {\infty}} \sqrt {t}}{\sqrt {N}} = \left(\frac {2 \| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}}}{N ^ {1 / 4}}\right) \left(\frac {2 \| \varphi \| _ {L ^ {\infty}} \sqrt {t}}{N ^ {1 / 4}}\right) \leq \frac {2 \| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}} ^ {2}}{\sqrt {N}} + \frac {2 \| \varphi \| _ {L ^ {\infty}} ^ {2} t}{\sqrt {N}}.
$$
The third term (III) is estimated in a similar manner. Expanding the empirical average on  $\mathcal{A}_t$  yields
$$
\begin{array}{l} \left| \frac {1}{N} \sum_ {n = 1} ^ {N} \varepsilon_ {n} \| \Phi (u _ {n}; \alpha) \| _ {\mathcal {Y}} ^ {2} \right| = \left| \frac {1}{M} \sum_ {m = 1} ^ {M} \alpha_ {m} \left(\frac {1}{M} \sum_ {m ^ {\prime} = 1} ^ {M} \alpha_ {m ^ {\prime}} \beta_ {m, m ^ {\prime}} ^ {(N)}\right) \right|, \quad \text {w h e r e} \\ \beta_ {m, m ^ {\prime}} ^ {(N)} = \frac {1}{N} \sum_ {n = 1} ^ {N} \varepsilon_ {n} \langle \varphi (u _ {n}; \theta_ {m}), \varphi (u _ {n}; \theta_ {m ^ {\prime}}) \rangle_ {\mathcal {Y}}. \\ \end{array}
$$
The first equality in the above display satisfies the upper bound
$$
\begin{array}{l} \sqrt {t} \left(\frac {1}{M} \sum_ {m = 1} ^ {M} \left| \frac {1}{M} \sum_ {m ^ {\prime} = 1} ^ {M} \alpha_ {m ^ {\prime}} \beta_ {m, m ^ {\prime}} ^ {(N)} \right| ^ {2}\right) ^ {1 / 2} \leq \sqrt {t} \left(\frac {1}{M} \sum_ {m = 1} ^ {M} t \left[ \frac {1}{M} \sum_ {m ^ {\prime} = 1} ^ {M} | \beta_ {m, m ^ {\prime}} ^ {(N)} | ^ {2} \right]\right) ^ {1 / 2} \\ = \frac {t}{N} \sqrt {\frac {1}{M ^ {2}} \sum_ {m , m ^ {\prime} = 1} ^ {M} \left| \sum_ {n = 1} ^ {N} \varepsilon_ {n} \langle \varphi (u _ {n} ; \theta_ {m}), \varphi (u _ {n} ; \theta_ {m ^ {\prime}}) \rangle_ {\mathcal {Y}} \right| ^ {2}} \\ \end{array}
$$
by two applications of the Cauchy-Schwarz inequality in  $\mathbb{R}^M$ . Finally, we deduce that
$$
\begin{array}{l} \left(\text {I I I}\right) \leq \frac {2 t}{N} \sqrt {\frac {1}{M ^ {2}} \sum_ {m , m ^ {\prime} = 1} ^ {M} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\{u _ {n} \}} \left\langle \varphi (u _ {n} ; \theta_ {m}), \varphi (u _ {n} ; \theta_ {m ^ {\prime}}) \right\rangle_ {\mathcal {Y}} ^ {2}} \\ \leq \frac {2 t}{\sqrt {N}} \sqrt {\frac {1}{M ^ {2}} \sum_ {m , m ^ {\prime} = 1} ^ {M} \mathbb {E} _ {u} \left[ \| \varphi (u ; \theta_ {m}) \| _ {y} ^ {2} \| \varphi (u ; \theta_ {m ^ {\prime}}) \| _ {y} ^ {2} \right]} \\ \leq \frac {2 t \| \varphi \| _ {L ^ {\infty}} ^ {2}}{\sqrt {N}} \\ \end{array}
$$
by Jensen's inequality, the fact that  $\mathbb{E}[\varepsilon_n\varepsilon_{n'}] = \delta_{n,n'}$ , and the Cauchy-Schwarz inequality in  $\mathcal{V}$ .
Combining the three estimates completes the proof.
The proof of the main generalization gap bound (4.12) is now immediate.
Proof of Proposition 4.7. Lem. D.6 and D.7 (applied with  $t = \beta$ ) show that
$$
\mathcal {E} _ {\beta} \left(\left\{u _ {n} \right\}, \left\{\theta_ {m} \right\}\right) \leq \frac {4 \left(\| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}} ^ {2} + \| \varphi \| _ {L ^ {\infty}} ^ {2} \beta\right)}{\sqrt {N}} + 3 2 e ^ {3 / 2} \left(\| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}} ^ {2} + \| \varphi \| _ {L ^ {\infty}} ^ {2} \beta\right) \sqrt {\frac {\log (1 / \delta)}{N}} \tag {D.27}
$$
with conditional probability (over  $\{\theta_m\}$ ) at least  $1 - \delta$  if  $N \geq \log(1/\delta)$ . Since  $\delta$  does not depend on  $\{\theta_m\}$ , we deduce by the tower rule of conditional expectation that the event implied by (D.27) has  $\mathbb{P}$ -probability at least  $1 - \delta$  as well. Using the inequalities  $4 \leq 32e^{3/2}$  and  $\sqrt{a} + \sqrt{b} \leq \sqrt{2(a + b)}$  shows that the expression in (D.27) is bounded above by
$$
3 2 e ^ {3 / 2} \left(\| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}} ^ {2} + \| \varphi \| _ {L ^ {\infty}} ^ {2} \beta\right) \sqrt {\frac {2 (1 + \log (1 / \delta))}{N}}.
$$
Application of the inequality  $1 \leq 2\log(2/\delta)$ , valid for  $\delta \in (0,1)$ , implies (4.12) as asserted.
E Numerical experiment details
In this appendix, we detail the setup of the numerical experiment from Sect. 5 and provide additional visualization of the function-valued RFM's discretization-independence in Figure 3. All code used to produce the numerical results and figures in this paper are available at
https://github.com/nickhnelsen/incorrect-bounds-for-vvRF.
A RFM with  $M$  features is trained on  $N$  input-output pairs  $\{(u_n, \mathcal{G}(u_n))\}_{n=1}^N$  according to the vector-valued RF-RR algorithm. The ground truth map  $\mathcal{G} \colon L^2(\mathbb{T}; \mathbb{R}) \to L^2(\mathbb{T}; \mathbb{R})$  is a nonlinear operator defined by  $u^{(0)}(\cdot) \mapsto u(\cdot, 1)$ , where  $u = \{u(x, t)\}_{x,t}$  solves the partial differential equation
$$
\frac {\partial u}{\partial t} + \frac {\partial}{\partial x} \left(\frac {u ^ {2}}{2}\right) = 1 0 ^ {- 1} \frac {\partial^ {2} u}{\partial x ^ {2}}, (x, t) \in \mathbb {T} \times (0, \infty),
$$
with initial condition  $u(\cdot, 0) = u^{(0)} \in L^2(\mathbb{T}; \mathbb{R})$ . Here,  $\mathbb{T} \simeq (0, 2\pi)_{\mathrm{per}}$  is the 1D torus which comes with periodic boundary conditions. The initial conditions  $u_n \sim \nu$  are sampled iid from a centered Matérn-like Gaussian process according to [23, Sect. 6.3, p. 32].
The random features are defined in a similar way to the Fourier Space RFs in [32, Sect. 3.1, p. 15]:
$$
\varphi (u ^ {(0)}; \theta) = 2. 6 \cdot \operatorname {E L U} \left(\mathcal {F} ^ {- 1} \left\{1 _ {\left(| k | \leq k _ {\max }\right)} \chi_ {k} \cdot (\mathcal {F} u ^ {(0)}) _ {k} \cdot (\mathcal {F} \theta) _ {k} \right\} _ {k \in \mathbb {Z}}\right) \quad \text {a n d} \quad \theta \sim \mu ,
$$