SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Samuel Lanthaler*
California Institute of Technology slanth@caltech.edu
Nicholas H. Nelsen*
California Institute of Technology nnelsen@caltech.edu
This paper provides a comprehensive error analysis of learning with vector-valued random features (RF). The theory is developed for RF ridge regression in a fully general infinite-dimensional input-output setting, but nonetheless applies to and improves existing finite-dimensional analyses. In contrast to comparable work in the literature, the approach proposed here relies on a direct analysis of the underlying risk functional and completely avoids the explicit RF ridge regression solution formula in terms of random matrices. This removes the need for concentration results in random matrix theory or their generalizations to random operators. The main results established in this paper include strong consistency of vector-valued RF estimators under model misspecification and minimax optimal convergence rates in the well-specified setting. The parameter complexity (number of random features) and sample complexity (number of labeled data) required to achieve such rates are comparable with Monte Carlo intuition and free from logarithmic factors.
Supervised learning of an unknown mapping  $\mathcal{G}\colon \mathcal{X}\to \mathcal{Y}$  is a core task in machine learning. The random feature model (RFM), proposed in [36, 37], combines randomization with optimization to accomplish this task. The RFM is based on a linear expansion with respect to a randomized basis, the random features (RF). The coefficients in this RF expansion are optimized to fit the given data of input-output pairs. For popular loss functions, such as the square loss, the RFM leads to a convex optimization problem which can be solved efficiently and reliably.
The RFM provides a scalable approximation of an underlying kernel method [36, 37]. While the former is based on an expansion in  $M$  random features  $\varphi(\cdot; \theta_1), \ldots, \varphi(\cdot; \theta_M)$ , the corresponding kernel method relies on an expansion in values of a positive definite kernel function  $K(\cdot, u_1), \ldots, K(\cdot, u_N)$  on a dataset of size  $N$ . Kernel methods are conceptually appealing, theoretically sound, and have attracted considerable interest [1, 6, 40]. However, they require the storage, manipulation, and often inversion of the kernel matrix  $\mathsf{K}$  with entries  $K(u_i, u_j)$ . The size of  $\mathsf{K}$  scales quadratically in the number of samples  $N$ , which can be prohibitive for large datasets. When the underlying input-output map is vector-valued with  $\dim(\mathcal{V}) = p$ , the often significant computational cost of kernel methods is further exacerbated by the fact that each entry  $K(u_i, u_j)$  of  $\mathsf{K}$  is, in general, a  $p$ -by-  $p$  matrix. Hence, the size of  $\mathsf{K}$  scales quadratically in both  $N$  and  $p$ . This severely limits the applicability of kernel methods to problems with high-dimensional, or indeed infinite-dimensional, output space. In contrast, learning with RF only requires storage of RF matrices whose size is quadratic in the number of features  $M$ . When  $M \ll Np$ , this implies substantial computational savings, with the most extreme case being the infinite-dimensional setting in which  $p = \infty$ .
In the context of operator learning, the underlying target mapping is an operator  $\mathcal{G}\colon \mathcal{X}\to \mathcal{Y}$  with infinite-dimensional input and output spaces. Such operators appear naturally in scientific computing
and often arise as solution maps of an underlying partial differential equation. Operator learning has attracted considerable interest, e.g., [3, 16, 20, 25, 27], and in this context, the RFM serves as an alternative to neural network-based methods with considerable potential for a sound theoretical basis. Indeed, an extension of the RFM to this infinite-dimensional setting has been proposed and implemented in [32]. Although the results show promise, a mathematical analysis of this approach including error bounds and rates of convergence has so far been outstanding.
Related work. Several papers have derived error bounds for learning with RF. Early work on the RFM [37] proceeded by direct inspection of the risk functional, demonstrating that  $M \simeq N$  random features suffice to achieve a squared error  $O(1 / \sqrt{N})$  for RF ridge regression (RR). This result was considerably improved in [38], where  $\sqrt{N} \log N$  random features were shown to be sufficient to achieve the same squared error. This improvement in parameter complexity is based on the explicit RF RR solution formula, combined with extensive use of matrix analysis and matrix concentration inequalities. Similar analysis in [26] sharpens the parameter complexity to  $\sqrt{N} \log d_{\mathsf{K}}^{\lambda}$  random features. Here  $d_{\mathsf{K}}^{\lambda}$  is the number of effective degrees of freedom [2, 6], with  $\lambda$  the RR regularization parameter and  $\mathsf{K}$  the kernel matrix. In this context, we also mention related analysis in [2]. In all these works, the squared error in terms of sample size  $N$  match the minimax optimal rates for kernel RR derived in [6]. Going beyond the above error bounds, [2, 26, 38] also derive fast rates under additional assumptions on the underlying data distribution and/or with improved RF sampling schemes.
Many works also study the interpolatory  $(M\simeq N)$  or overparametrized  $(M\gg N)$  regimes in the scalar output setting [10, 14, 15, 18, 29]. However, when  $p = \dim (\mathcal{V})\gg 1$  or  $p = \infty$ , such regimes may no longer be relevant. This is because the kernel matrix  $\mathsf{K}$  now has size  $Np$ -by- $Np$ , and it is possible that the number of random features  $M$  satisfies  $M\ll Np$  even though  $M\gg N$ . In this case, high-dimensional vector-valued learning naturally operates in the underparametrized regime.
In the area of operator learning for scientific problems, approximation results are common [11, 13, 19, 17, 22, 24, 39, 41] but statistical guarantees are lacking, the main exceptions being [4, 12, 21, 31, 43] in the linear operator setting and [6] in the nonlinear setting. The RFM also has potential for such nonlinear problems. Indeed, vector-valued random Fourier features have been studied before [5, 30]. However, theory is only provided for kernel approximation, not generalization guarantees.
To summarize, while previous analyses have provided considerable insight into the generalization properties of the RFM, they have almost exclusively focused on the scalar-valued setting. Given the paucity of theoretical work beyond this setting, it is a priori unclear whether similar estimates continue to hold when the RFM is applied to infinite-dimensional vector-valued mappings.
Contributions. The primary purpose of this paper is to extend earlier results on learning with random features to the vector-valued setting. The theory developed in this work unifies sources of error stemming from approximation, generalization, misspecification, and noisy observations. We focus on training via ridge regression with the square loss. Our results differ from existing work not only in the scope of applicability, but also in the strategy employed to derive our results. Similar to [37], we do not rely on the explicit random feature ridge regression solution (RF-RR) formula, which is specific to the square loss. One main benefit of this approach is that it entirely avoids the use of matrix concentration inequalities, thereby making the extension to an infinite-dimensional vector-valued setting straightforward. Our main contributions are now listed (see also Table 1).
(C1) Given  $N$  training samples, we prove that  $M \simeq \sqrt{N}$  random features and regularization strength  $\lambda \simeq 1 / \sqrt{N}$  is enough to guarantee that the squared error is  $O(1 / \sqrt{N})$ , provided that the target operator belongs to a specific reproducing kernel Hilbert space (Thm. 3.7);
(C2) we establish that the vector-valued RF-RR estimator is strongly consistent (Thm. 3.10);
(C3) under additional regularity assumptions, we derive rates of convergence even when the target operator does not belong to the specific reproducing kernel Hilbert space (Thm. 3.12);
(C4) we demonstrate that the approach of Rahimi and Recht [37] can be used to derive state-of-the-art rates for the RFM which, for the first time, are free from logarithmic factors.
Outline. The remainder of this paper is organized as follows. We set up the ridge regression problem in Sect. 2. The main results are stated in Sect. 3 and their proofs are sketched in Sect. 4.
Sect. 5 provides a simulation study and Sect. 6 gives concluding remarks. Detailed proofs are deferred to the supplementary material (SM).
Table 1: A summary of available error estimates for the RFM, with regularization parameter  $\lambda$ , output space  $\mathcal{V}$ , and number of random features  $M$ .  $(^{\star})$ : the truth is assumed to be written as  $\mathcal{G}(u) = \mathbb{E}_{\theta}[\alpha^{*}(\theta)\varphi (u;\theta)]$  with restrictive bound  $|\alpha^{*}(\theta)|\leq R$  to avoid explicit regularization.
Paper	Approach	λ	dim(Y)	M	Squared Error
Rahimi & Recht [37]	“kitchen sinks”	n/a (*)	1	N	R/√N
Rudi & Rosasco [38]	matrix concen.	1/√N	1	√N log(N)	1/√N
Li et al. [26]	matrix concen.	1/√N	1	√N log(dK)	1/√N
This work	“kitchen sinks”	1/√N	∞	√N	1/√N
We now set up our vector-valued learning framework by introducing notational conventions, reviewing random features, and formulating the ridge regression problem.
Notation. Let  $(\Omega, \mathcal{F}, \mathbb{P})$  be a sufficiently rich probability space on which all random variables in this paper are defined. Let  $\mathcal{X}$  be the input space,  $\mathcal{V}$  the output space, and  $\Theta$  a set. We consistently use  $u$  to denote elements of  $\mathcal{X}$  and  $\theta$  for RF parameters in  $\Theta$ . The set of probability measures supported on a set  $\mathcal{Q}$  is denoted by  $\mathcal{P}(\mathcal{Q})$ . We write expectation (in the sense of Bochner integration) with respect to  $u \sim \nu \in \mathcal{P}(\mathcal{X})$  as  $\mathbb{E}_u[\cdot]$  and similarly for  $\theta \sim \mu \in \mathcal{P}(\Theta)$ . Independent and identically distributed (iid) samples  $u_1, \ldots, u_N$  from  $\nu$  will be denoted by  $\{u_n\} \sim \nu^{\otimes N}$  and similarly for  $\{\theta_m\} \sim \mu^{\otimes M}$ . We write  $a \simeq b$  to mean that there exists a constant  $C \geq 1$  such that  $C^{-1}b \leq a \leq Cb$  and similarly for the one-sided inequalities  $a \lesssim b$  and  $a \gtrsim b$ . We define  $a \wedge b := \min(a, b)$ .
Random features and reproducing kernel Hilbert spaces. Random features are defined by a pair  $(\varphi, \mu)$ , where  $\varphi \colon \mathcal{X} \times \Theta \to \mathcal{Y}$  and  $\mu \in \mathcal{P}(\Theta)$ . Fixing  $\theta \sim \mu$  defines a map  $\varphi(\cdot; \theta) \colon \mathcal{X} \to \mathcal{Y}$ . Considering linear combinations of such maps leads to the following definition.
Definition 2.1 (Random feature model). The map  $\Phi (\cdot ;\alpha) = \Phi (\cdot ;\alpha ,\{\theta_{m}\})\colon \mathcal{X}\to \mathcal{Y}$  given by
$$
u \mapsto \Phi (u; \alpha) := \frac {1}{M} \sum_ {m = 1} ^ {M} \alpha_ {m} \varphi (u; \theta_ {m}) \tag {2.1}
$$
is a random feature model (RFM) with coefficients  $\alpha \in \mathbb{R}^{M}$  and fixed realizations  $\{\theta_m\} \sim \mu^{\otimes M}$ .
Associated to the pair  $(\varphi, \mu)$  is a reproducing kernel Hilbert space (RKHS)  $\mathcal{H}$  of maps from  $\mathcal{X}$  to  $\mathcal{Y}$  [32, Sect. 2.3]. Under mild assumptions (see SM B) assumed in our main results, it holds that
$$
\mathcal {H} = \left\{\mathcal {G} \in L _ {\nu} ^ {2} (\mathcal {X}; \mathcal {Y}) \mid \mathcal {G} = \mathbb {E} [ \alpha (\theta) \varphi (\cdot ; \theta) ] \text {a n d} \alpha \in L _ {\mu} ^ {2} (\Theta ; \mathbb {R}) \right\} \tag {2.2}
$$
with RKHS norm  $\| \mathcal{G}\|_{\mathcal{H}} = \min_{\alpha}\| \alpha \|_{L_\mu^2}$ , where  $\alpha$  ranges over all decompositions of  $\mathcal{G}$  of the form in (2.2). A minimizer  $\alpha_{\mathcal{H}}$  of this problem always exists [2, Sect. 2.2]. We use this fact to identify any  $\mathcal{G}\in \mathcal{H}$  with its minimizing  $\alpha_{\mathcal{H}}\in L_{\mu}^{2}(\Theta ;\mathbb{R})$  without further comment.
Random feature ridge regression. Let  $\mathcal{P} \in \mathcal{P}(\mathcal{X} \times \mathcal{Y})$  be the joint data distribution. The goal of RF-RR is to estimate an underlying operator  $\mathcal{G} \colon \mathcal{X} \to \mathcal{Y}$  from finitely many iid input-output pairs  $\{(u_n, y_n)\}_{n=1}^N \sim \mathcal{P}^{\otimes N}$ , where typically the  $y_n$  are noisy transformations of the point values  $\mathcal{G}(u_n)$ . To describe RF-RR, we first make some definitions.
Definition 2.2 (Empirical risk). Writing  $Y = \{y_{n}\}$  for the collection of observed output data and fixing a regularization parameter  $\lambda >0$ , the regularized  $Y$ -empirical risk of  $\alpha \in \mathbb{R}^{M}$  is given by
$$
\mathcal {R} _ {N} ^ {\lambda} (\alpha ; Y) := \frac {1}{N} \sum_ {n = 1} ^ {N} \| y _ {n} - \Phi (u _ {n}; \alpha) \| _ {\mathcal {Y}} ^ {2} + \lambda \| \alpha \| _ {M} ^ {2}, \quad \text {w h e r e} \quad \| \alpha \| _ {M} ^ {2} := \frac {1}{M} \sum_ {m = 1} ^ {M} | \alpha_ {m} | ^ {2} \tag {2.3}
$$
is a scaled Euclidean norm on  $\mathbb{R}^M$ . The regularized  $\mathcal{G}$ -empirical risk,  $\mathcal{R}_N^\lambda (\alpha ;\mathcal{G})$ , is defined analogously with  $\mathcal{G}(u_n)$  in place of  $y_{n}$ . In the absence of regularization, i.e.,  $\lambda = 0$ , these expressions define the  $Y$ -empirical risk and  $\mathcal{G}$ -empirical risk, denoted by  $\mathcal{R}_N(\alpha ;Y)$  and  $\mathcal{R}_N(\alpha ;\mathcal{G})$ , respectively.
RF-RR is the minimization problem  $\min_{\alpha \in \mathbb{R}^M}\mathcal{R}_N^\lambda (\alpha ;Y)$ . The minimizer, which we denote by  $\widehat{\alpha}$ , is referred to as trained coefficients and  $\Phi (\cdot ;\widehat{\alpha})$  the trained RFM. For  $M$  and  $N$  sufficiently large and  $\lambda >0$  sufficiently small, we expect the trained RFM to well approximate  $\mathcal{G}$ . This intuition is made precise by quantitative error bounds and statistical performance guarantees in the next section.
The main result of this paper is an abstract bound on the population squared error (Sect. 3.2). From this widely applicable theorem, several more specialized results are deduced. These include consistency (Sect. 3.3) and convergence rates (Sect. 3.4) of the RF-RR estimator trained on noisy data. The assumptions under which this theory is developed are provided next in Sect. 3.1.
Throughout this paper, we assume that the input space  $\mathcal{X}$  is a Polish space and the output space  $\mathcal{V}$  is a real separable Hilbert space. These are common assumptions in learning theory [6]. We view  $\mathcal{X}$  and  $\mathcal{V}$  as measurable spaces equipped with their respective Borel  $\sigma$ -algebras.
Next, we make the following minimal assumptions on the random feature pair  $(\varphi ,\mu)$
Assumption 3.1 (Random feature regularity). Let  $\nu \in \mathcal{P}(\mathcal{X})$  be the input distribution and  $(\Theta, \Sigma, \mu)$  be a probability space. The random feature map  $\varphi: \mathcal{X} \times \Theta \to \mathcal{Y}$  and the probability measure  $\mu \in \mathcal{P}(\Theta)$  are such that (i)  $\varphi$  is measurable; (ii)  $\varphi$  is uniformly bounded; in fact,  $\| \varphi \|_{L^{\infty}} \coloneqq \operatorname*{ess}\sup_{(u,\theta) \sim \nu \otimes \mu} \| \varphi(u;\theta) \|_{\mathcal{Y}} \leq 1$ ; and (iii) the RKHS  $\mathcal{H}$  corresponding to  $(\varphi, \mu)$  is separable.
The boundedness assumption on  $\varphi$  is shared in general theoretical analyses of RF [26, 37, 38]; the unit bound can always be ensured by a simple rescaling. We work in a general misspecified setting.
Assumption 3.2 (Misspecification). There exist  $\rho \in L_{\nu}^{\infty}(\mathcal{X};\mathcal{Y})$  and  $\mathcal{G}_{\mathcal{H}}\in \mathcal{H}$  such that the operator  $\mathcal{G}\colon \mathcal{X}\to \mathcal{Y}$  satisfies the decomposition  $\mathcal{G} = \rho +\mathcal{G}_{\mathcal{H}}$
Since Assumption 3.1 implies that  $\mathcal{H} \subset L_{\nu}^{\infty}(\mathcal{X};\mathcal{Y})$ , any  $\mathcal{G} = \mathcal{G}_{\mathcal{H}} + \rho$  as in Assumption 3.2 is automatically bounded in the sense that  $\mathcal{G} \in L_{\nu}^{\infty}(\mathcal{X};\mathcal{Y})$ . Conversely, any  $\mathcal{G} \in L_{\nu}^{\infty}(\mathcal{X};\mathcal{Y})$  allows such a decomposition. We interpret  $\rho$  as a residual from the operator  $\mathcal{G}_{\mathcal{H}}$  belonging to the RKHS. It may be prescribed by the problem, as we will see later in the context of discretization errors in operator learning (Ex. 3.9), or be arbitrary, as is customary in learning theory when the only information about  $\mathcal{G}$  is that it is bounded.
Our main goal is to recover  $\mathcal{G}$  from iid data  $\{(u_n, y_n)\}$  arising from the following statistical model.
Assumption 3.3 (Joint data distribution). The joint distribution  $\mathcal{P} \in \mathcal{P}(\mathcal{X} \times \mathcal{Y})$  of the random variable  $(u, y) \sim \mathcal{P}$  is given by  $u \sim \nu$  with  $\nu \in \mathcal{P}(\mathcal{X})$  and  $y = \mathcal{G}(u) + \eta$ . Here,  $\mathcal{G}$  satisfies Assumption 3.2. The additive noise  $\eta$  is a random variable in  $\mathcal{V}$  that is conditionally centered,  $\mathbb{E}[\eta | u] = 0$ , and is subexponential:  $\| \eta \|_{\psi_1(\mathcal{V})} < \infty$ ; see (A.7) for the definition of  $\| \cdot \|_{\psi_1(\mathcal{V})}$ .
Assumption 3.3 implies that  $\mathcal{G}(u) = \mathbb{E}[y\mid u]$ . In contrast to related work [2, 26, 37], we allow for unbounded input-dependent noise. In particular, our results also hold for bounded or subgaussian noise, as well as multiplicative noise (e.g.,  $\eta = \xi \mathcal{G}(u)$  with  $\mathbb{E}[\xi \mid u] = 0$  and  $\| \xi \|_{\psi_1} < \infty$ ).
3.2 General error bound
For any  $\mathcal{G}$ , define the  $\mathcal{G}$ -population risk functional or  $\mathcal{G}$ -population squared error by
$$
\mathcal {R} (\alpha ; \mathcal {G}) := \mathbb {E} _ {u \sim \nu} \| \mathcal {G} (u) - \Phi (u; \alpha , \left\{\theta_ {m} \right\}) \| _ {\mathcal {Y}} ^ {2} \quad \text {f o r} \quad \alpha \in \mathbb {R} ^ {M}. \tag {3.1}
$$
The main result of this paper establishes an upper bound for this quantity that holds with high probability, provided that the number of random features and number of data pairs are large enough.
Theorem 3.4 ( $\mathcal{G}$ -population squared error bound). Suppose that  $\mathcal{G} = \rho + \mathcal{G}_{\mathcal{H}}$  satisfies Assumption 3.2. Fix a failure probability  $\delta \in (0,1)$ , regularization strength  $\lambda \in (0,1)$ , and sample size  $N$ . Let  $\{\theta_m\} \sim \mu^{\otimes M}$  be the  $M$  random feature parameters and  $\{(u_n, y_n)\} \sim \mathcal{P}^{\otimes N}$  be the data according to Assumption 3.3. For  $\Phi$  the RFM (2.1) satisfying Assumption 3.1, let  $\widehat{\alpha} \in \mathbb{R}^{M}$  be the minimizer
of the regularized  $Y$ -empirical risk  $\mathcal{R}_N^\lambda (\cdot ;Y)$  given by (2.3). If  $M\geq \lambda^{-1}\log (32 / \delta)$  and  $N\geq$ $\lambda^{-2}\log (16 / \delta)$ , then
$$
\left. \mathbb {E} _ {u \sim \nu} \| \mathcal {G} (u) - \Phi (u; \widehat {\alpha}, \left\{\theta_ {m} \right\}) \| _ {\mathcal {Y}} ^ {2} \leq 7 9 e ^ {3 / 2} \left(\| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}} ^ {2} + 2 \beta (\rho , \lambda , \mathcal {G} _ {\mathcal {H}}, \eta)\right) \lambda \right. \tag {3.2}
$$
with probability at least  $1 - \delta$ , where
$$
\beta (\rho , \lambda , \mathcal {G} _ {\mathcal {H}}, \eta) := 3 2 8 \| \mathcal {G} _ {\mathcal {H}} \| _ {\mathcal {H}} ^ {2} + 2 0 2 3 e ^ {3} \| \eta \| _ {\psi_ {1} (\mathcal {Y})} ^ {2} + 8 \lambda^ {- 1} \mathbb {E} _ {u \sim \nu} \| \rho (u) \| _ {\mathcal {Y}} ^ {2} + 1 8 \lambda \| \rho \| _ {L _ {\nu} ^ {\infty}} ^ {2} \tag {3.3}
$$
is a function of  $\rho, \lambda, \mathcal{G}_{\mathcal{H}}$ , and the law of the noise variable  $\eta$ .
The main elements of the proof of Thm. 3.4 will be explained in Sect. 4.
Remark 3.5 (Excess risk). We note that other work [26, 37, 38] often focuses on bounding the excess risk  $\widehat{\mathcal{E}}\coloneqq \mathcal{E}(\Phi (\cdot ;\widehat{\alpha})) - \inf_{\mathcal{G}_{\mathcal{H}}\in \mathcal{H}}\mathcal{E}(\mathcal{G}_{\mathcal{H}})$ , where  $\mathcal{E}(F)\coloneqq \mathbb{E}\| y - F(u)\|_{\mathcal{Y}}^{2} = \mathbb{E}_{u\sim \nu}\| \mathcal{G}(u) - F(u)\|_{\mathcal{Y}}^{2} + \mathbb{E}\| \eta \|_{\mathcal{Y}}^{2}$ . In particular, this bias-variance decomposition implies that  $\widehat{\mathcal{E}}\leq \mathbb{E}_{u\sim \nu}\| \mathcal{G}(u) - \Phi (u;\widehat{\alpha})\|_{\mathcal{Y}}^{2}$ . Thus, Thm. 3.4 also gives a corresponding bound on the excess risk  $\widehat{\mathcal{E}}$ .
Remark 3.6 (The  $\beta$  factor). In the well-specified setting, that is,  $\mathcal{G} - \mathcal{G}_{\mathcal{H}} = \rho \equiv 0$ , the factor  $\beta$  in Thm. (3.4) satisfies the uniform bound
$$
\beta (\rho , \lambda , \mathcal {G} _ {\mathcal {H}}, \eta) \leq B := 3 2 8 \| \mathcal {G} \| _ {\mathcal {H}} ^ {2} + 2 0 2 3 e ^ {3} \| \eta \| _ {\psi_ {1} (\mathcal {Y})} ^ {2}. \tag {3.4}
$$
In particular, the constant  $B$  does not depend on  $\lambda$  in this case. Otherwise,  $\beta$  in general depends on  $\lambda$ . We can characterize this dependence precisely if it is known that  $\mathcal{G} \in L_{\nu}^{\infty}(\mathcal{X};\mathcal{Y})$ . In this case, Assumption 3.2 is satisfied with  $\rho \coloneqq \mathcal{G} - \mathcal{G}_{\mathcal{H}}$  for any  $\mathcal{G}_{\mathcal{H}} \in \mathcal{H}$ . Choosing  $\mathcal{G}_{\mathcal{H}} = \mathcal{G}_{\vartheta}|_{\vartheta = \lambda}$  as in SM B (which is optimal in the sense described there) and a short calculation delivers the bound
$$
\beta (\rho , \lambda , \mathcal {G} _ {\mathcal {H}}, \eta) \lesssim \lambda^ {- 1} \lambda^ {r \wedge 1} = \lambda^ {- (1 - r) _ {+}} \tag {3.5}
$$
if  $\mathcal{G}$  additionally satisfies a particular  $r$ -th order regularity condition (see Lem. B.3 for the details). Here,  $a_{+} := \max(a, 0)$  for any  $a \in \mathbb{R}$ . Thus,  $\beta$  is uniformly bounded if  $\mathcal{G} \in \mathcal{H}$  ( $r \geq 1$ ) and grows algebraically as a power of  $\lambda^{-1}$  otherwise ( $0 \leq r < 1$ ).
Consequences. The general error bound (3.2) in Thm. 3.4 has several implications for vector-valued learning with the RFM. First, it immediately implies a rate of convergence if  $\mathcal{G} \in \mathcal{H}$ .
Theorem 3.7 (Well-specified). Instantiate the hypotheses and notation of Thm. 3.4. Suppose that  $\rho \equiv 0$  so that  $\mathcal{G} \in \mathcal{H}$  (2.2). If  $M \geq \lambda^{-1} \log(32 / \delta)$  and  $N \geq \lambda^{-2} \log(16 / \delta)$ , then
$$
\mathbb {E} _ {u \sim \nu} \| \mathcal {G} (u) - \Phi (u; \widehat {\alpha}) \| _ {\mathcal {Y}} ^ {2} \leq 7 9 e ^ {3 / 2} \left(\| \mathcal {G} \| _ {L _ {\nu} ^ {\infty}} ^ {2} + 2 B\right) \lambda \lesssim \lambda \tag {3.6}
$$
with probability at least  $1 - \delta$ , where the constant  $B \geq 0$  is defined by (3.4).