SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given a number of samples  $N$ , Thm. 3.7 shows that RF-RR with regularization  $\lambda \simeq 1 / \sqrt{N}$  and number of features  $M \gtrsim \sqrt{N}$  leads to a population squared error of size  $1 / \sqrt{N}$  with high probability. This result should be compared to the previous state-of-the-art convergence rates in the literature for RF-RR with iid sampled features [2, 26, 37, 38]. See Table 1, which indicates that our analysis gives the lowest parameter complexity to date. We emphasize that such a convergence rate rests on the assumption that  $\mathcal{G} \in \mathcal{H}$ . This corresponds to a compatibility condition between  $\mathcal{G}$  and the pair  $(\varphi, \mu)$ , i.e., the random feature map  $\varphi$  and the probability measure  $\mu$ , which determine the RKHS  $\mathcal{H}$ . Designing suitable  $\varphi$  and  $\mu$  for a given operator  $\mathcal{G}$  remains an open problem. For an explanation of the poor parameter complexity in Rahimi and Recht's original paper [37], see [44, Appendix E].
Thm. 3.4 also implies convergence of  $\mathcal{R}(\widehat{\alpha};\mathcal{G})$  when  $\mathcal{G}\not\in\mathcal{H}$ , as we will see in Sect. 3.3 and 3.4. But first, the next corollary shows that the same general bound (3.2) also holds for the  $\mathcal{G}_{\mathcal{H}}$ -population squared error  $\mathcal{R}(\widehat{\alpha};\mathcal{G}_{\mathcal{H}})$ , up to enlarged constant factors. The proof is given in SM C.
Corollary 3.8 ( $\mathcal{G}_{\mathcal{H}}$ -population squared error bound). Instantiate the hypotheses and notation of Thm. 3.4. If  $M \geq \lambda^{-1} \log(32 / \delta)$  and  $N \geq \lambda^{-2} \log(16 / \delta)$ , then there exists an absolute constant  $C > 1$  such that with probability at least  $1 - \delta$ , it holds that
$$
\mathbb {E} _ {u \sim \nu} \| \mathcal {G} _ {\mathcal {H}} (u) - \Phi (u; \widehat {\alpha}) \| _ {\mathcal {Y}} ^ {2} \leq C \left(\| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}} ^ {2} + 2 \beta (\rho , \lambda , \mathcal {G} _ {\mathcal {H}}, \eta)\right) \lambda . \tag {3.7}
$$
Although our main goal is to learn  $\mathcal{G}$  from noisy data, there are settings instead in which the learning of  $\mathcal{G}_{\mathcal{H}} \in \mathcal{H}$  as in Cor. 3.8 is of primary interest, but only values of some approximation  $\mathcal{G} \in L_{\nu}^{\infty}(\mathcal{X};\mathcal{Y})$  are available. The following example illustrates this.
Example 3.9 (Numerical discretization error). One practically relevant setting to which Cor. 3.8 applies arises when training a RFM from functional data generated by a numerical approximation  $\mathcal{G} = \mathcal{G}_{\Delta}$  of some underlying operator  $\mathcal{G}_{\mathcal{H}} \in \mathcal{H}$ . Here  $\Delta > 0$  represents a numerical parameter, such as the grid resolution when approximating the solution operator of a partial differential equation. In this setting,  $\rho = \mathcal{G}_{\Delta} - \mathcal{G}_{\mathcal{H}}$  is non-zero and it is crucial to include the discretization error in the analysis, which we define as  $\varepsilon_{\Delta} := \| \rho \|_{L_{\nu}^{\infty}}^{2} = \| \mathcal{G}_{\Delta} - \mathcal{G}_{\mathcal{H}} \|_{L_{\nu}^{\infty}}^{2}$ . Assume  $\eta \equiv 0$ , so that  $\hat{\alpha}$  minimizes  $\mathcal{R}_N^\lambda (\cdot ;Y) = \mathcal{R}_N^\lambda (\cdot ;\mathcal{G}_\Delta)$ . Using Cor. 3.8, it follows that for  $M$  and  $N$  sufficiently large,
$$
\mathbb {E} _ {u \sim \nu} \| \mathcal {G} _ {\mathcal {H}} (u) - \Phi (u; \widehat {\alpha}) \| _ {\mathcal {Y}} ^ {2} \lesssim \lambda \| \mathcal {G} _ {\mathcal {H}} \| _ {\mathcal {H}} ^ {2} + \varepsilon_ {\Delta} \tag {3.8}
$$
with high probability. Thus, as suggested by intuition, in addition to the error contribution that is present when training on perfect data (the first term on the right-hand side), there is an additional discretization error of size  $\varepsilon_{\Delta}$ . We also see that the performance of RF-RR is stable with respect to such discretization errors stemming from the training data. Actually obtaining a rate of convergence would require problem-specific information about the particular numerical solver that is used.
3.3 Statistical consistency
We now return to the objective of recovering  $\mathcal{G}$  from data. In particular, suppose that  $\mathcal{G} \notin \mathcal{H}$ ; the RKHS, viewed as a hypothesis class, is misspecified. Our analysis demonstrates that statistical guarantees for RF-RR are still possible in this setting.
To this end, assume that  $\mathcal{G} \in L_{\nu}^{\infty}(\mathcal{X};\mathcal{Y})$ . It follows that Assumption 3.2 is satisfied with  $\rho \coloneqq \mathcal{G} - \mathcal{G}_{\mathcal{H}}$  and  $\mathcal{G}_{\mathcal{H}} \in \mathcal{H}$  being any element of the RKHS. Applying Thm. 3.4 and minimizing over  $\mathcal{G}_{\mathcal{H}}$  yields
$$
\mathbb {E} _ {u \sim \nu} \| \mathcal {G} (u) - \Phi (u; \widehat {\alpha}) \| _ {\mathcal {Y}} ^ {2} \lesssim \lambda + \inf  _ {\mathcal {G} _ {\mathcal {H}} \in \mathcal {H}} \left\{\mathbb {E} _ {u \sim \nu} \| \mathcal {G} (u) - \mathcal {G} _ {\mathcal {H}} (u) \| _ {\mathcal {Y}} ^ {2} + \lambda \| \mathcal {G} _ {\mathcal {H}} \| _ {\mathcal {H}} ^ {2} \right\} \tag {3.9}
$$
with probability at least  $1 - \delta$  if  $M \gtrsim \lambda^{-1} \log(2 / \delta)$  and  $N \gtrsim \lambda^{-2} \log(2 / \delta)$ . To obtain (3.9), we enlarged constants and used the bound  $\| \rho \|_{L_\nu^\infty}^2 \lesssim \| \mathcal{G} \|_{L_\nu^\infty}^2 + \| \mathcal{G}_{\mathcal{H}} \|_{L_\nu^\infty}^2$  in (3.3).
If  $\mathcal{G}$  is in the  $L_{\nu}^{2}$ -closure of  $\mathcal{H}$ , then with high probability, the population squared error on the left hand side of (3.9) converges to zero as  $\lambda \to 0$  (by application of Lem. B.2 to the second term on the right). This is a statement about the (weak) statistical consistency of the trained RF-RR estimator; it can be upgraded to an almost sure statement, as expressed precisely in the next main result.
Theorem 3.10 (Strong consistency). Suppose that  $\mathcal{G} \in L_{\nu}^{\infty}(\mathcal{X};\mathcal{Y})$  belongs to the  $L_{\nu}^{2}(\mathcal{X};\mathcal{Y})$ -closure of  $\mathcal{H}$  (2.2). Let  $\{\lambda_k\}_{k \in \mathbb{N}} \subset (0,1)$  be a sequence of positive regularization parameters such that  $\sum_{k \in \mathbb{N}} \lambda_k < \infty$ . For  $\Phi$  the RFM (2.1) satisfying Assumption 3.1 and for each  $k$ , let  $\widehat{\alpha}^{(k)} \in \mathbb{R}^{M_k}$  be the trained RFM coefficients that minimize the regularized  $Y$ -empirical risk  $\mathcal{R}_{N_k}^{\lambda_k}(\cdot;Y)$  given by (2.3) with  $M_k \simeq \lambda_k^{-1} \log(2/\lambda_k)$  iid random features and  $N_k \simeq \lambda_k^{-2} \log(2/\lambda_k)$  iid data pairs under Assumption 3.3. It holds true that
$$
\lim  _ {k \rightarrow \infty} \mathbb {E} _ {u \sim \nu} \| \mathcal {G} (u) - \Phi (u; \widehat {\alpha} ^ {(k)}) \| _ {\mathcal {Y}} ^ {2} = 0 \text {w i t h p r o b a b i l i t y o n e .} \tag {3.10}
$$
The proof relies on a standard Borel-Cantelli argument. See SM C for the details.
Remark 3.11 (Universal RKHS). The assumption that  $\mathcal{G}$  belongs to the  $L_{\nu}^{2}$ -closure of the RKHS  $\mathcal{H}$  is automatically satisfied if  $\mathcal{H}$  is dense in  $L_{\nu}^{2}(\mathcal{X};\mathcal{Y})$ . This is equivalent to its kernel being universal [7, 9]. In this case, the trained RFM is a strongly consistent estimator of any  $\mathcal{G} \in L_{\nu}^{\infty}$ . However, we are unaware of any practical characterizations of universality of the kernel in terms of its corresponding random feature pair  $(\varphi ,\mu)$  for the vector-valued setting studied here.
3.4 Convergence rates
The previous subsection establishes convergence guarantees without any rates. We now establish quantitative bounds. Throughout what follows, we denote by  $\mathcal{K}\colon L_{\nu}^{2}(\mathcal{X};\mathcal{Y})\to L_{\nu}^{2}(\mathcal{X};\mathcal{Y})$  the integral operator (B.5) corresponding to the operator-valued kernel function of the RKHS  $\mathcal{H}$  (see SM B).
Theorem 3.12 (Slow rates under misspecification). Suppose that  $\mathcal{G} \in L_{\nu}^{\infty}(\mathcal{X};\mathcal{Y})$  and that Assumption 3.3 holds. Additionally, assume that  $\mathcal{G} \in \mathrm{Im}(\mathcal{K}^{r / 2})$  for some  $r > 0$ , where  $\mathcal{K}$  is the integral operator corresponding to the kernel of RKHS  $\mathcal{H}$  (2.2). Fix  $\delta \in (0,1)$  and  $\lambda \in (0,1)$ . For
$\Phi$  the RFM (2.1) satisfying Assumption 3.1, let  $\widehat{\alpha} \in \mathbb{R}^{M}$  minimize  $\mathcal{R}_N^\lambda (\cdot ;Y)$  given by (2.3). If  $M \geq \lambda^{-1}\log (32 / \delta)$  and  $N \geq \lambda^{-2}\log (16 / \delta)$ , then with probability at least  $1 - \delta$  it holds that
$$
\mathbb {E} _ {u \sim \nu} \| \mathcal {G} (u) - \Phi (u; \widehat {\alpha}) \| _ {\mathcal {V}} ^ {2} \lesssim \lambda^ {r \wedge 1}. \tag {3.11}
$$
The implied constant in (3.11) depends only on  $\| \mathcal{G}\|_{L_\nu^\infty}$  and  $\| \eta \|_{\psi_1(\mathcal{V})}$ .
Thm. 3.12 provides a quantitative convergence rate as  $\lambda \to 0$ . For  $r \geq 1$ , i.e., when  $\mathcal{G} \in \mathcal{H}$ , we recover the linear convergence rate of order  $\lambda$  from Thm. 3.7. The assumption that  $\mathcal{G} \in \mathrm{Im}(\mathcal{K}^{r/2})$  can be viewed as a "fractional regularity" assumption on the underlying operator; indeed, in specific settings it corresponds to a fractional (Sobolev) regularity of the underlying function. In general, it appears difficult to check this condition in practice, which is one limitation of our result.
A quantitative analog to the almost sure statement of Thm. 3.10 also holds.
Corollary 3.13 (Strong convergence rate). Instantiate the hypotheses and notation of Thm. 3.10. Assume in addition that  $\mathcal{G} \in \mathrm{Im}(\mathcal{K}^{r/2})$  for some  $r > 0$ . Let  $\{\lambda_k\}_{k \in \mathbb{N}} \subset (0,1)$  be a sequence of positive regularization parameters such that  $\sum_{k \in \mathbb{N}} \lambda_k < \infty$ . For each  $k$ , let  $\widehat{\alpha}^{(k)} \in \mathbb{R}^{M_k}$  be the trained RFM coefficients with  $M_k \simeq \lambda_k^{-1} \log(2/\lambda_k)$  and  $N_k \simeq \lambda_k^{-2} \log(2/\lambda_k)$ . It holds true that
$$
\lim  _ {k \rightarrow \infty} \left(\frac {\mathbb {E} _ {u \sim \nu} \| \mathcal {G} (u) - \Phi (u ; \widehat {\alpha} ^ {(k)}) \| _ {\mathcal {Y}} ^ {2}}{\lambda_ {k} ^ {r \wedge 1}}\right) <   \infty \text {w i t h p r o b a b i l i t y o n e}. \tag {3.12}
$$
Short proofs of both Thm. 3.12 and Cor. 3.13 may be found in SM C.
4 Proof outline for the main theorem
Our main results are all derived from Thm. 3.4, whose proof, schematically illustrated in Figure 1, we now outline. Following [37], we break the proof into several steps that arise from the error decomposition
$$
\mathcal {R} (\widehat {\alpha}; \mathcal {G}) = \mathcal {R} _ {N} (\widehat {\alpha}; \mathcal {G}) + \left[ \mathcal {R} (\widehat {\alpha}; \mathcal {G}) - \mathcal {R} _ {N} (\widehat {\alpha}; \mathcal {G}) \right]. \tag {4.1}
$$
Sect. 4.1 estimates the first term on the right hand side of (4.1) while Sect. 4.2 estimates the second.
Figure 1: Flow chart illustrating the proof of Theorem 3.4.
4.1 Bounding the regularized empirical risk
The main technical contribution of this work is a tight bound on the  $\mathcal{G}$ -empirical risk  $\mathcal{R}_N(\widehat{\alpha};\mathcal{G})$  for the trained RFM. The analysis involves controlling several sources of error and careful truncation arguments to avoid unnecessarily strong assumptions on the problem. The result is the following.
Proposition 4.1 (Regularized  $\mathcal{G}$ -empirical risk bound). Let Assumptions 3.1 and 3.3 hold. Suppose that  $\mathcal{G} = \rho + \mathcal{G}_{\mathcal{H}}$  satisfies Assumption 3.2. Fix  $\delta \in (0,1)$ ,  $\lambda \in (0,1)$ ,  $M \in \mathbb{N}$ , and  $N \in \mathbb{N}$ .
Let  $\widehat{\alpha} \in \mathbb{R}^{M}$  be the minimizer of the regularized  $Y$ -empirical risk  $\mathcal{R}_N^\lambda (\cdot ;Y)$  given by (2.3). If  $M \geq \lambda^{-1}\log (16 / \delta)$  and  $N \geq \lambda^{-2}\log (8 / \delta)$ , then
$$
\mathcal {R} _ {N} ^ {\lambda} (\widehat {\alpha}; \mathcal {G}) \leq \lambda \beta (\rho , \lambda , \mathcal {G} _ {\mathcal {H}}, \eta) \tag {4.2}
$$
with probability at least  $1 - \delta$ , where the multiplicative factor  $\beta (\rho ,\lambda ,\mathcal{G}_{\mathcal{H}},\eta)$  is given by (3.3).
Since  $\lambda \| \widehat{\alpha} \|_M^2 \leq \mathcal{R}_N^\lambda (\widehat{\alpha};\mathcal{G})$ , the next corollary controlling the norm (2.3) of  $\widehat{\alpha}$  is immediate. It plays a crucial role in developing an upper bound for the second term on the right side of (4.1).
Corollary 4.2 (Trained RFM norm bound). Instantiate the hypotheses and notation of Prop. 4.1. Fix  $\delta \in (0,1)$  and  $\lambda \in (0,1)$ . If  $M \geq \lambda^{-1} \log(16 / \delta)$  and  $N \geq \lambda^{-2} \log(8 / \delta)$ , then
$$
\widehat {\alpha} \in \mathcal {A} _ {\beta} := \left\{\alpha \in \mathbb {R} ^ {M} \mid \| \alpha \| _ {M} ^ {2} \leq \beta \right\} \tag {4.3}
$$
with probability at least  $1 - \delta$ . The radius  $\beta \coloneqq \beta (\rho, \lambda, \mathcal{G}_{\mathcal{H}}, \eta)$  of the norm bound is given by (3.3).
The core elements of the proof of Prop. 4.1 are provided in the next few subsections, with the full argument in SM D.1. The main idea is to upper bound the  $\mathcal{G}$ -empirical risk by its regularized counterpart and then decompose the latter into several (coupled) error contributions.
To do this, first fix any  $\alpha \in \mathbb{R}^{M}$ . It holds that
$$
\mathcal {R} _ {N} ^ {\lambda} (\alpha ; Y) = \mathcal {R} _ {N} ^ {\lambda} (\alpha ; \mathcal {G}) + \frac {2}{N} \sum_ {n = 1} ^ {N} \left\langle \eta_ {n}, \mathcal {G} \left(u _ {n}\right) - \Phi \left(u _ {n}; \alpha\right) \right\rangle_ {\mathcal {Y}} + \frac {1}{N} \sum_ {n = 1} ^ {N} \| \eta_ {n} \| _ {\mathcal {Y}} ^ {2} \tag {4.4}
$$
because  $\mathcal{V}$  is a Hilbert space and  $y_{n} = \mathcal{G}(u_{n}) + \eta_{n}$ . Using this, a short calculation shows that
$$
\begin{array}{l} \mathcal {R} _ {N} ^ {\lambda} (\widehat {\alpha}; \mathcal {G}) = \left[ \mathcal {R} _ {N} ^ {\lambda} (\widehat {\alpha}; Y) - \mathcal {R} _ {N} ^ {\lambda} (\alpha ; Y) \right] + \mathcal {R} _ {N} ^ {\lambda} (\alpha ; \mathcal {G}) + \frac {2}{N} \sum_ {n = 1} ^ {N} \langle \eta_ {n}, \Phi (u _ {n}; \widehat {\alpha}) - \Phi (u _ {n}; \alpha) \rangle_ {\mathcal {Y}} \\ \leq \mathcal {R} _ {N} ^ {\lambda} (\alpha ; \mathcal {G}) + \frac {2}{N} \sum_ {n = 1} ^ {N} \langle - \eta_ {n}, \Phi (u _ {n}; \alpha) \rangle_ {\mathcal {Y}} + \frac {2}{N} \sum_ {n = 1} ^ {N} \langle \eta_ {n}, \Phi (u _ {n}; \widehat {\alpha}) \rangle_ {\mathcal {Y}}. \tag {4.5} \\ \end{array}
$$
In the second line, we used the fact that  $\widehat{\alpha}$  minimizes  $\mathcal{R}_N^\lambda (\cdot ;Y)$ . Since  $\alpha \in \mathbb{R}^M$  is arbitrary, we have the freedom to choose  $\alpha$  so that the first term is small (see Sect. 4.1.1 and 4.1.2). With  $\alpha$  fixed, the second term averages to zero by our assumptions on the noise, and hence, we expect it to be small with high probability (see Sect. 4.1.3).
The third term in (4.5) exhibits high correlation between the noise  $\eta_{n}$  and the trained RFM coefficients  $\widehat{\alpha}$ , making it more difficult to estimate. To control this last term, we first note that it is homogeneous in  $\| \widehat{\alpha} \|_M$ , which can be used to derive an upper bound in terms of a supremum over the unit ball with respect to  $\| \cdot \|_M$ . The resulting expression is then bounded with empirical process techniques (see SM D.1.3). For the complete details of the required argument we refer the reader to SM D.1.
In the remainder of this subsection, we estimate the first two terms on the right hand side of (4.5). Using the fact that  $\mathcal{G} = \rho +\mathcal{G}_{\mathcal{H}}$ , the first term can be split into two contributions,
$$
\mathcal {R} _ {N} ^ {\lambda} (\alpha ; \mathcal {G}) \leq 2 \mathcal {R} _ {N} ^ {\lambda} (\alpha ; \mathcal {G} _ {\mathcal {H}}) + \frac {2}{N} \sum_ {n = 1} ^ {N} \| \rho (u _ {n}) \| _ {\mathcal {Y}} ^ {2}. \tag {4.6}
$$
These contributions to the first term in (4.5) are bounded in Sect. 4.1.1 and 4.1.2. The second term in (4.5) is controlled in Sect. 4.1.3.
4.1.1 Bounding the approximation error
We begin with the term  $\mathcal{R}_N^\lambda (\alpha ;\mathcal{G}_\mathcal{H})$ , which may be viewed as an empirical approximation error due to  $\alpha$  being arbitrary. Its only dependence on the data is through  $\{u_n\}$  in (2.3). Intuitively, this term should behave like its population counterpart. It is then natural to choose a Monte Carlo approximation  $\alpha_{m} = \alpha_{\mathcal{H}}(\theta_{m})$  for  $\alpha$ , where  $\alpha_{\mathcal{H}}\in L_{\mu}^{2}(\Theta ;\mathbb{R})$  is identified with  $\mathcal{G}_{\mathcal{H}}$  as in (2.2). However, our intuition that  $\lambda \| \alpha \| _M^2$  should concentrate around  $\lambda \| \alpha_{\mathcal{H}}\|_{L_\mu^2}^2$  fails because it is generally not possible to control the tail of the random variable  $|\alpha_{\mathcal{H}}(\theta)|^2$ . We next show that this problem can be overcome by a carefully tuned truncation argument combined with Bernstein's inequality.
Lemma 4.3 (Construction of approximator). Suppose that  $\mathcal{G}_{\mathcal{H}} \coloneqq \mathcal{G} \in \mathcal{H}$ . Fix  $\delta \in (0,1)$ ,  $\lambda > 0$ ,  $N \in \mathbb{N}$ , and  $M \in \mathbb{N}$ . Let  $\{\theta_m\} \sim \mu^{\otimes M}$  and  $\{u_n\} \sim \nu^{\otimes N}$ . Define  $\alpha^\star \in \mathbb{R}^M$  componentwise by
$$
\alpha_ {m} ^ {\star} := \alpha_ {\mathcal {H}} \left(\theta_ {m}\right) \mathbb {1} _ {\left\{\left| \alpha_ {\mathcal {H}} \left(\theta_ {m}\right) \right| \leq T \right\}}, \quad \text {w h e r e} \quad T := \sqrt {\lambda^ {- 1} \mathbb {E} _ {\theta \sim \mu} \left| \alpha_ {\mathcal {H}} (\theta) \right| ^ {2}} \tag {4.7}
$$
and  $\mathcal{G}_{\mathcal{H}} = \mathbb{E}_{\theta \sim \mu}[\alpha_{\mathcal{H}}(\theta)\varphi (\cdot ;\theta)]$  with  $\| \mathcal{G}_{\mathcal{H}}\|_{\mathcal{H}}^{2} = \mathbb{E}_{\theta \sim \mu}|\alpha_{\mathcal{H}}(\theta)|^{2}$ . If  $M\geq \lambda^{-1}\log (4 / \delta)$ , then with probability at least  $1 - \delta$  in the random feature parameters  $\theta_{1},\ldots ,\theta_{M}$ , it holds that
$$
\mathcal {R} _ {N} ^ {\lambda} \left(\alpha^ {\star}; \mathcal {G} _ {\mathcal {H}}\right) \leq 8 1 \lambda \| \mathcal {G} _ {\mathcal {H}} \| _ {\mathcal {H}} ^ {2}. \tag {4.8}
$$
SM D.1.1 provides the proof.
Remark 4.4 (Well-specified and noise-free). Lem. 4.3 gives a  $O(\lambda)$  bound on the regularized  $\mathcal{G}_{\mathcal{H}}$ -empirical risk of a RFM trained on well-specified and noise-free iid data  $\{u_n, \mathcal{G}_{\mathcal{H}}(u_n)\}$ .
4.1.2 Bounding the misspecification error
The second contribution to (4.6) is easily bounded by Bernstein's inequality because  $\rho \in L_{\nu}^{\infty}$ . We refer the reader to SM D.1.2 for the detailed proof.
Lemma 4.5 (Concentration of misspecification error). Let  $\rho$  be as in Assumption 3.2. Fix  $\delta \in (0,1)$ . With probability at least  $1 - \delta$ , it holds that
$$
\frac {2}{N} \sum_ {n = 1} ^ {N} \| \rho (u _ {n}) \| _ {\mathcal {Y}} ^ {2} \leq 4 \mathbb {E} _ {u \sim \nu} \| \rho (u) \| _ {\mathcal {Y}} ^ {2} + \frac {9 \| \rho \| _ {L _ {\nu} ^ {\infty}} ^ {2} \log (2 / \delta)}{N}. \tag {4.9}
$$
4.1.3 Bounding the noise error
The second term on the right hand side of (4.5) is a zero mean error contribution due to the noise corrupting the output training data. By the fact that  $\eta$  is subexponential (Assumption 3.3), Bernstein's inequality delivers exponential concentration. The proof details are in SM D.1.3.
Lemma 4.6 (Concentration of noise error cross term). Let Assumptions 3.1 and 3.3 hold. Fix  $\alpha \in \mathbb{R}^M$ ,  $\{\theta_m\} \sim \mu^{\otimes M}$ , and  $\delta \in (0,1)$ . With probability at least  $1 - \delta$ , it holds that
$$
\frac {2}{N} \sum_ {n = 1} ^ {N} \left\langle - \eta_ {n}, \Phi \left(u _ {n}; \alpha\right) \right\rangle_ {\mathcal {Y}} \leq 1 6 e ^ {3 / 2} \| \eta_ {1} \| _ {\psi_ {1} (\mathcal {Y})} \| \alpha \| _ {M} \sqrt {\frac {\log (2 / \delta)}{N}}. \tag {4.10}
$$
SM D.1.3 also details the techniques used to upper bound the third and final term in (4.5).
4.2 Bounding the generalization gap
Having bounded the empirical risk with approximation arguments, it remains to control the estimation error  $\mathcal{R}(\widehat{\alpha};\mathcal{G}) - \mathcal{R}_N(\widehat{\alpha};\mathcal{G})$  due to finite data in (4.1). We call this the generalization gap: the difference between the population test error and its empirical version. If  $\widehat{\alpha}$  satisfies  $\| \widehat{\alpha}\| _M^2\leq t$  for some  $t > 0$ , then one can upper bound the generalization gap by its supremum over this set. The main challenge is to show existence of a (sufficiently small)  $t$  that satisfies this inequality. This is handled by Cor. 4.2. As summarized in the following proposition, the resulting supremum of the empirical process defined by the generalization gap is shown to be of size  $N^{-1 / 2}$  with high probability.
Proposition 4.7 (Uniform bound on the generalization gap). Let Assumption 3.1 hold. Suppose  $\mathcal{G}$  satisfies Assumption 3.2. Let  $\{\theta_m\} \sim \mu^{\otimes M}$  for the RFM  $\Phi$  given by (2.1). Fix  $\delta \in (0,1)$ . For iid input samples  $\{u_n\} \sim \nu^{\otimes N}$ , define the random variable
$$
\mathcal {E} _ {\beta} \left(\left\{u _ {n} \right\}, \left\{\theta_ {m} \right\}\right) := \sup  _ {\alpha \in \mathcal {A} _ {\beta}} \left| \frac {1}{N} \sum_ {n = 1} ^ {N} \| \mathcal {G} (u _ {n}) - \Phi (u _ {n}; \alpha) \| _ {\mathcal {Y}} ^ {2} - \mathbb {E} _ {u} \| \mathcal {G} (u) - \Phi (u; \alpha) \| _ {\mathcal {Y}} ^ {2} \right|, \tag {4.11}
$$
where  $\mathcal{A}_{\beta} \coloneqq \{\alpha' \in \mathbb{R}^M \mid \| \alpha' \|_M^2 \leq \beta\}$  and the deterministic radius  $\beta = \beta(\rho, \lambda, \mathcal{G}_{\mathcal{H}}, \eta)$  is given in (3.3) with  $\mathcal{G}$  as above. If  $N \geq \log(1/\delta)$ , then with probability at least  $1 - \delta$  it holds that
$$
\mathcal {E} _ {\beta} \left(\left\{u _ {n} \right\}, \left\{\theta_ {m} \right\}\right) \leq 3 2 e ^ {3 / 2} \left(\| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}} ^ {2} + \beta (\rho , \lambda , \mathcal {G} _ {\mathcal {H}}, \eta)\right) \sqrt {\frac {6 \log (2 / \delta)}{N}}. \tag {4.12}
$$
The proof of Prop. 4.7 is given in SM D.2. The argument is composed of two steps. The first is to show that  $\mathcal{E}_{\beta} \mid \{\theta_m\}$  concentrates around its (conditional) expectation (Lem. D.6). This follows easily using the boundedness of the summands. The second step is to upper bound the expectation of  $\mathcal{E}_{\beta} \mid \{\theta_m\}$  (Lem. D.7). This is achieved by exploiting the Hilbert space structure of the  $\mathcal{V}$ -square loss and the linearity of the RFM with respect to its coefficients.
4.3 Combining the bounds
Since we now have control over the  $\mathcal{G}$ -empirical risk and the generalization gap, the  $\mathcal{G}$ -population risk is also under control by (4.1). The proof of Thm. 3.4 follows by putting together the pieces (SM C).
5 Numerical experiment
To study how our theory holds up in practice, we numerically implement the vector-valued RF-RR algorithm on a benchmark operator learning dataset. The data  $\{(u_n,\mathcal{G}(u_n))\}_{n = 1}^N$  is noise-free, the  $\{u_{n}\}$  are iid Gaussian random fields, and  $\mathcal{G}\colon L^{2}(\mathbb{T};\mathbb{R})\to L^{2}(\mathbb{T};\mathbb{R})$  is a nonlinear operator defined as the time one flow map of the viscous Burgers' equation on the torus  $\mathbb{T}$ . SM E provides more details about the problem setting and the choice of random feature pair  $(\varphi ,\mu)$ .
(a) Varying  $p,M,\lambda$  for fixed  $N = 1548$