SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(b) Varying  $p,N,\lambda$  for fixed  $M = 10^{4}$
Figure 2: Squared test error of trained RFM for learning the Burgers' equation solution operator. All shaded bands denote two empirical standard deviations from the empirical mean of the error computed over 10 different models, each with iid sampling of the features and training data indices.
Figure 2a shows the decay of the relative squared test error as  $M$  increases (with  $\lambda \simeq 1 / M$ ) for fixed  $N$ . The error closely follows the rate  $O(M^{-1})$  until it begins to saturate at larger  $M$ . This is due to either  $\mathcal{G}$  not belonging to the RKHS of  $(\varphi, \mu)$  or the finite data error dominating. As implied by our theory, the error does not depend on the discretized output dimension  $p < \infty$ . Figure 2b displays similar behavior as  $N$  is varied (now with  $\lambda \simeq 1 / \sqrt{N}$  and fixed  $M$ ). Overall, the observed parameter and sample complexity reasonably validate our theoretical insights.
This paper establishes several fundamental results for learning with infinite-dimensional vector-valued random features; these include strong consistency and explicit convergence rates. When the underlying mapping belongs to the RKHS, the rates obtained in this work match minimax optimal rates in the number of samples  $N$ , requiring only a number of random features  $M \simeq \sqrt{N}$ . Despite being derived in a very general setting, to the best of our knowledge, this provides the sharpest parameter complexity in  $M$ , which is free from logarithmic factors for the first time.
There are several interesting directions for future work. These include deriving fast rates, relaxing the boundedness assumption on the features and the true mapping, and accommodating heavier-tailed or white noise distributions. Obtaining fast rates would require a sharpening of several estimates, and in particular, replacing the global Rademacher complexity-type estimate, implicit in our work, by its local counterpart. As our approach does not make use of an explicit solution formula, which is only available for a square loss, this might pave the way for improved rates for other loss functions, such as a general  $L^p$ -loss. We leave such potential extensions of the present approach for future work.
The work of SL is supported by Postdoc.Mobility grant P500PT-206737 from the Swiss National Science Foundation. NHN acknowledges support from the National Science Foundation Graduate Research Fellowship Program under award number DGE-1745301 and from the Amazon/Caltech AI4Science Fellowship, and partial support from the Air Force Office of Scientific Research under MURI award number FA9550-20-1-0358 (Machine Learning and Physics-Based Modeling and Simulation). SL and NHN are also grateful for partial support from the Department of Defense Vannevar Bush Faculty Fellowship held by Andrew M. Stuart under Office of Naval Research award number N00014-22-1-2790.
[1] N. Aronszajn. Theory of reproducing kernels. Transactions of the American mathematical society, 68(3):337-404, 1950. (Cited on page 1)
[2] F. Bach. On the equivalence between kernel quadrature rules and random feature expansions. The Journal of Machine Learning Research, 18(1):714-751, 2017. (Cited on pages 2, 3, 4, and 5)
[3] K. Bhattacharya, B. Hosseini, N. B. Kovachki, and A. M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI journal of computational mathematics, 7: 121-157, 2021. doi: 10.5802/smai-jcm.74. URL https://smai-jcm centre-mersenne.org/articles/10.5802/smai-jcm.74/. (Cited on page 2)
[4] N. Boullé and A. Townsend. Learning elliptic partial differential equations with randomized linear algebra. Foundations of Computational Mathematics, pages 1-31, 2022. (Cited on page 2)
[5] R. Brault, M. Heinonen, and F. Buc. Random fourier features for operator-valued kernels. In Asian Conference on Machine Learning, pages 110–125. PMLR, 2016. (Cited on page 2)
[6] A. Caponnetto and E. De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331-368, 2007. (Cited on pages 1, 2, 4, and 14)
[7] A. Caponnetto, C. A. Micchelli, M. Pontil, and Y. Ying. Universal multi-task kernels. The Journal of Machine Learning Research, 9:1615-1646, 2008. (Cited on page 6)
[8] C. Carmeli, E. De Vito, and A. Toigo. Vector valued reproducing kernel Hilbert spaces of integrable functions and Mercer theorem. Analysis and Applications, 4(04):377-408, 2006. (Cited on pages 16 and 17)
[9] C. Carmeli, E. De Vito, A. Toigo, and V. Umanitá. Vector valued reproducing kernel hilbert spaces and universality. Analysis and Applications, 8(01):19-61, 2010. (Cited on page 6)
[10] Z. Chen, H. Schaeffer, and R. Ward. Concentration of Random Feature Matrices in High-Dimensions. In Mathematical and Scientific Machine Learning, pages 287-302. PMLR, 2022. (Cited on page 2)
[11] A. Cohen, R. Devore, and C. Schwab. Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDE's. Analysis and Applications, 9(01):11-47, 2011. (Cited on page 2)
[12] M. V. de Hoop, N. B. Kovachki, N. H. Nelsen, and A. M. Stuart. Convergence Rates for Learning Linear Operators from Noisy Data. SIAM/ASA Journal on Uncertainty Quantification, 11(2):480-513, 2023. doi: 10.1137/21M1442942. (Cited on page 2)
[13] B. Deng, Y. Shin, L. Lu, Z. Zhang, and G. E. Karniadakis. Approximation rates of DeepONets for learning operators arising from advection-diffusion equations. *Neural Networks*, 153: 411–426, 2022. (Cited on page 2)
[14] W. E, C. Ma, and L. Wu. The generalization error of the minimum-norm solutions for overparameterized neural networks. Pure and Applied Functional Analysis, 5(6):1445-1460, 2020. (Cited on page 2)
[15] B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari. Linearized two-layers neural networks in high dimension. Ann. Statist., 49:1029-1054, 2021. (Cited on page 2)
[16] C. R. Gin, D. E. Shea, S. L. Brunton, and J. N. Kutz. DeepGreen: deep learning of Green's functions for nonlinear boundary value problems. Scientific reports, 11(1):21614, 2021. (Cited on page 2)
[36] A. Rahimi and B. Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007. (Cited on page 1)
[37] A. Rahimi and B. Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. Advances in neural information processing systems, 21, 2008. (Cited on pages 1, 2, 3, 4, 5, and 7)
[38] A. Rudi and L. Rosasco. Generalization properties of learning with random features. Advances in neural information processing systems, 30, 2017. (Cited on pages 2, 3, 4, 5, and 14)
[39] T. D. Ryck and S. Mishra. Generic bounds on the approximation error for physics-informed (and) operator learning. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum? id=bF4eYy3LTR9. (Cited on page 2)
[40] B. Scholkopf and A. J. Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2018. (Cited on page 1)
[41] C. Schwab and J. Zech. Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ. Analysis and Applications, 17(01):19-55, 2019. (Cited on page 2)
[42] S. Smale and D.-X. Zhou. Shannon sampling II: Connections to learning theory. Applied and Computational Harmonic Analysis, 19(3):285-302, 2005. (Cited on pages 16 and 17)
[43] G. Stepaniants. Learning partial differential equations in reproducing kernel Hilbert spaces. Journal of Machine Learning Research, 24(86):1-72, 2023. (Cited on page 2)
[44] Y. Sun, A. Gilbert, and A. Tewari. But how does it work in theory? Linear SVM with random features. Advances in Neural Information Processing Systems, 31, 2018. (Cited on page 5)
[45] R. Vershynin. High-dimensional probability: An introduction with applications in data science, 2018. (Cited on page 15)
[46] M. J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge University Press, 2019. (Cited on page 25)
Supplementary Material for:
Error Bounds for Learning with
Vector-Valued Random Features
A Concentration of measure
In this section, we recall two classical results from [35] that estimate the difference between empirical averages and true averages of random vectors taking values in Banach spaces. These are then specialized to the setting of subexponential random variables, which play a major role in this paper. To set the notation, we use  $\operatorname{Pr}$  to denote probability with respect to the underlying probability space.
The first result is a vector-valued Bernstein concentration inequality with various applications to problems posed in infinite-dimensional Hilbert spaces [6, 28, 38]. It is used throughout this work.
Theorem A.1 (Vector-valued Bernstein inequality in Hilbert space). Let  $Z$  be an  $H$ -valued random variable, where  $(H, \langle \cdot, \cdot \rangle, \| \cdot \|)$  is a separable Hilbert space. Suppose there exist positive numbers  $b > 0$  and  $\sigma > 0$  such that
$$
\mathbb {E} \| Z - \mathbb {E} Z \| ^ {p} \leq \frac {1}{2} p! \sigma^ {2} b ^ {p - 2} \quad f o r a l l \quad p \geq 2. \tag {A.1}
$$
For any  $\delta \in (0,1)$  and  $N\in \mathbb{N}$ , denoting by  $\{Z_n\}_{n = 1}^N$  a sequence of  $N$  iid copies of  $Z$ , it holds that
$$
\Pr \left\{\left\| \frac {1}{N} \sum_ {n = 1} ^ {N} Z _ {n} - \mathbb {E} Z \right\| \leq \frac {2 b \log (2 / \delta)}{N} + \sqrt {\frac {2 \sigma^ {2} \log (2 / \delta)}{N}} \right\} \geq 1 - \delta . \tag {A.2}
$$
Proof. The result is a direct consequence of [35, Cor. 1, p. 144] in the iid Hilbert space setting. In this case, it holds for any  $t > 0$  that
$$
\operatorname * {P r} \big \{\| S _ {N} - \mathbb {E} S _ {N} \| \geq t \big \} \leq 2 \exp \bigg (- \frac {N ^ {2} t ^ {2}}{2 N \sigma^ {2} + 2 N t b} \bigg) = 2 \exp \bigg (- \frac {N t ^ {2}}{2 \sigma^ {2} + 2 b t} \bigg),
$$
where  $S_{n} \coloneqq \frac{1}{N}\sum_{n = 1}^{N}Z_{n}$ . Setting the right hand side equal to  $\delta$ , solving a quadratic equation for  $t = t(\delta)$ , and using the inequality  $\sqrt{a + b} \leq \sqrt{a} +\sqrt{b}$  to bound  $t(\delta)$  from above leads to (A.2).
The most common use case of Bernstein's inequality is in the following bounded setting.
Lemma A.2. Let  $Z$  be a (potentially) uncentered random variable such that
$$
\| Z \| \leq c \text {a l m o s t s u r e l y} \quad \text {a n d} \quad \mathbb {E} \| Z - \mathbb {E} Z \| ^ {2} \leq v ^ {2} \tag {A.3}
$$
for some  $c > 0$  and  $v > 0$ . Then  $Z$  satisfies Bernstein's moment condition (A.1) with  $b = 2c$  and  $\sigma = v$ . If  $\mathbb{E}Z = 0$ , then taking  $b = c$  suffices.
Proof. It holds that  $\| Z - \mathbb{E}Z\| \leq \| Z\| +\mathbb{E}\| Z\| \leq 2c$  almost surely. We compute
$$
\begin{array}{l} \mathbb {E} \| Z - \mathbb {E} Z \| ^ {p} = \mathbb {E} \left[ \| Z - \mathbb {E} Z \| ^ {2} \| Z - \mathbb {E} Z \| ^ {p - 2} \right] \leq \mathbb {E} \| Z - \mathbb {E} Z \| ^ {2} (2 c) ^ {p - 2} \\ \leq v ^ {2} (2 c) ^ {p - 2} \leq \frac {1}{2} p! v ^ {2} (2 c) ^ {p - 2} \\ \end{array}
$$
because  $1 \leq p! / 2$  for all  $p \geq 2$ . The centered improvement is trivial.
The second classic result we present is a one-sided Bernstein-type tail bound in a general Banach space. We invoke this theorem to control the tails of suprema of empirical processes.
Theorem A.3 (Vector-valued Bernstein inequality in Banach space). Let  $Z$  be a  $\mathcal{Z}$ -valued random variable, where  $(\mathcal{Z}, \| \cdot \|)$  is a separable Banach space. Suppose there exist positive numbers  $b > 0$  and  $\sigma > 0$  such that
$$
\mathbb {E} \| Z - \mathbb {E} Z \| ^ {p} \leq \frac {1}{2} p! \sigma^ {2} b ^ {p - 2} \quad f o r a l l \quad p \geq 2. \tag {A.4}
$$
$$
\Pr \left\{\left\| \frac {1}{N} \sum_ {n = 1} ^ {N} Z _ {n} \right\| - \mathbb {E} \left\| \frac {1}{N} \sum_ {n = 1} ^ {N} Z _ {n} \right\| \leq \frac {2 b \log (1 / \delta)}{N} + \sqrt {\frac {2 \sigma^ {2} \log (1 / \delta)}{N}} \right\} \geq 1 - \delta . \tag {A.5}
$$
Proof. The assertion is [35, Cor. 1, p. 144] in the iid Banach space setting (hence only convergence of norms in (A.5) instead of strong convergence). It is proved the same way as Thm. A.1.  $\square$
In Theorem A.3, a random variable  $Z$  satisfying the Bernstein moment condition (A.4) is subexponential in the sense that  $\| Z - \mathbb{E}Z\|$  is subexponential on  $\mathbb{R}$ , i.e., exhibits exponential tail decay. Recall that for a real-valued random variable  $Z$ , its subexponential norm may be defined by
$$
\| Z \| _ {\psi_ {1}} := \sup  _ {p \in [ 1, \infty)} \frac {(\mathbb {E} | Z | ^ {p}) ^ {1 / p}}{p}. \tag {A.6}
$$
See [45, Sect. 2.7] for equivalent definitions. We say that  $Z$  is subexponential if its subexponential norm is finite. Following [31, Sect. 4.3, pp. 19-20], for a random variable  $Z$  with values in a Banach space  $(\mathcal{Z},\| \cdot \|_{\mathcal{Z}})$  we define
$$
\| Z \| _ {\psi_ {1} (\mathcal {Z})} := \left\| \| Z \| _ {\mathcal {Z}} \right\| _ {\psi_ {1}} = \sup  _ {p \in [ 1, \infty)} \frac {\left(\mathbb {E} \| Z \| _ {\mathcal {Z}} ^ {p}\right) ^ {1 / p}}{p} \tag {A.7}
$$
as its subexponential norm. It is known that a random variable has finite subexponential norm if and only if it satisfies the Bernstein moment condition (A.4) [see, e.g., 31, Appendix A.2]. Next, we give explicit constants in the Bernstein moment condition that depend on the subexponential norm.
Proposition A.4 (Subexponential implies Bernstein moment condition). Let  $Z$  be a  $(\mathcal{Z},\| \cdot \|)$ -valued subexponential random variable, that is,  $\| Z\|_{\psi_1(\mathcal{Z})} < \infty$ . Then  $Z$  satisfies
$$
\mathbb {E} \| Z - \mathbb {E} Z \| ^ {p} \leq \frac {1}{2} p! \sigma^ {2} b ^ {p - 2} \quad f o r a l l \quad p \geq 2, \tag {A.8}
$$
$$
\sigma^ {2} := 4 e \sqrt {\mathbb {E} \| Z - \mathbb {E} Z \| ^ {2}} \| Z \| _ {\psi_ {1} (\mathcal {Z})} \quad a n d \quad b := 4 e \| Z \| _ {\psi_ {1} (\mathcal {Z})}. \tag {A.9}
$$
Proof. By the Cauchy-Schwarz inequality,
$$
\mathbb {E} \| Z - \mathbb {E} Z \| ^ {p} = \mathbb {E} [ \| Z - \mathbb {E} Z \| \| Z - \mathbb {E} Z \| ^ {p - 1} ] \leq \| Z - \mathbb {E} Z \| _ {L _ {\mathbb {P}} ^ {2}} (\mathbb {E} \| Z - \mathbb {E} Z \| ^ {2 p - 2}) ^ {1 / 2}.
$$
The inequality  $|a + b|^q \leq 2^{q - 1}(|a|^q + |b|^q)$ , Jensen's inequality, and  $\| Z \|_{\psi_1(\mathcal{Z})} < \infty$  show that
$$
\mathbb {E} \| Z - \mathbb {E}   Z \| ^ {2 p - 2} \leq 2 ^ {2 p - 2} \mathbb {E} \| Z \| ^ {2 p - 2} \leq 2 ^ {2 p - 2} (2 p - 2) ^ {2 p - 2} \| Z \| _ {\psi_ {1} (\mathcal {Z})} ^ {2 p - 2}.
$$
Next, Stirling's approximation  $(q / e)^{q} \leq q!$  and  $q! = q(q - 1)! \geq 2(q - 1)!$  for  $q \geq 2$  yields
$$
\begin{array}{l} \mathbb {E} \| Z - \mathbb {E} Z \| ^ {p} \leq 2 ^ {2 p - 2} \| Z - \mathbb {E} Z \| _ {L _ {\mathbb {P}} ^ {2}} (p - 1) ^ {p - 1} \| Z \| _ {\psi_ {1} (\mathcal {Z})} ^ {p - 1} \\ \leq (p! / 2) \| Z - \mathbb {E} Z \| _ {L _ {\mathbb {P}} ^ {2}} 2 ^ {2 p - 2} e ^ {p - 1} \| Z \| _ {\psi_ {1} (\mathcal {Z})} ^ {p - 1}. \\ \end{array}
$$
Rearranging the exponents to fit the Bernstein moment condition form completes the proof.
This leads to the following corollary, which is useful in the setting that the variance  $\mathbb{E}\| Z - \mathbb{E}Z\|_{\mathcal{Z}}^{2} = \| Z - \mathbb{E}Z\|_{L^2 (\Omega ;\mathcal{Z})}^2$  of random variable  $Z$  is not small or hard to compute.
Corollary A.5 (Subexponential tail bound in Banach space). Fix  $N \in \mathbb{N}$ . Let  $\{Z_n\}_{n=1}^N$  be iid random variables with values in a separable Banach space  $(\mathcal{Z}, \| \cdot \|$ \. Suppose that  $\| Z_1 \|_{\psi_1(\mathcal{Z})} < \infty$ . Let  $S_N \coloneqq \frac{1}{N} \sum_{n=1}^N Z_n$ . Fix  $\delta \in (0, 1)$ . With probability at least  $1 - \delta$ , it holds that
$$
\left\| S _ {N} \right\| - \mathbb {E} \| S _ {N} \| \leq \frac {8 e \| Z _ {1} \| _ {\psi_ {1} (\mathcal {Z})} \log (1 / \delta)}{N} + \sqrt {\frac {8 e \| Z _ {1} - \mathbb {E} Z _ {1} \| _ {L _ {\mathbb {P}} ^ {2} (\Omega ; \mathcal {Z})} \| Z _ {1} \| _ {\psi_ {1} (\mathcal {Z})} \log (1 / \delta)}{N}}. \tag {A.10}
$$
In particular, if  $N \geq \log(1 / \delta)$ , then with probability at least  $1 - \delta$  it holds that
$$
\left\| S _ {N} \right\| - \mathbb {E} \left\| S _ {N} \right\| \leq \sqrt {\frac {6 4 e ^ {3} \left\| Z _ {1} \right\| _ {\psi_ {1} (\mathcal {Z})} ^ {2} \log (1 / \delta)}{N}}. \tag {A.11}
$$
Proof. Apply Prop. A.4 to Thm. A.3 to obtain the first assertion. For the second assertion, first note that  $\mathbb{E}\| Z_1 - \mathbb{E}Z_1\|^2\leq 4\mathbb{E}\| Z_1\|^2$  (by triangle inequality and using  $(a + b)^2\leq 2(a^2 +b^2)$ ) and  $\| Z_1\|_{\psi_1(\mathcal{Z})}^2\geq \mathbb{E}\| Z_1\|^2 /4$  (by (A.7)). Since  $N\geq \log (1 / \delta)$ , we have  $\log (1 / \delta) / N\leq \sqrt{\log(1 / \delta) / N}$ . Combining these facts, it follows that the right hand side of (A.10) is bounded above by
$$
\sqrt {\frac {6 4 e ^ {2} \| Z _ {1} \| _ {\psi_ {1} (\mathcal {Z})} ^ {2} \log (1 / \delta)}{N}} + \sqrt {\frac {3 2 e \| Z _ {1} \| _ {\psi_ {1} (\mathcal {Z})} ^ {2} \log (1 / \delta)}{N}} \leq \sqrt {\frac {6 4 (2 e ^ {2} + e) \| Z _ {1} \| _ {\psi_ {1} (\mathcal {Z})} ^ {2} \log (1 / \delta)}{N}}.
$$
We used  $\sqrt{a} +\sqrt{b}\leq \sqrt{2(a + b)}$  on the right. Noting that  $2e^{2} + e\leq e^{3}$  completes the proof.
Comparing this result to a similar result [28, Prop. 7(i), pp. 4-5, the iid case], we note that (A.10) in Cor. A.5 is sharper in the sense that the constant in the  $N^{-1/2}$  term depends on the variance of the summands instead of just its subexponential norm (which can be much larger than the variance).
B Misspecification error with RKHS methods
Let  $K\colon \mathcal{X}\times \mathcal{X}\to \mathcal{L}(\mathcal{Y})$  be an operator-valued kernel [32] corresponding to a separable $^2$  RKHS  $\mathcal{H}$  of functions mapping  $\mathcal{X}$  to  $\mathcal{Y}$ . Here,  $\mathcal{L}(\mathcal{Y})$  denotes the set of bounded linear operators from  $\mathcal{Y}$  into itself. In this section, we present a general analysis based on [42] for the approximation of elements in  $L_{\nu}^{2}(\mathcal{X};\mathcal{Y})$  by elements in the RKHS  $\mathcal{H}$ . This problem is relevant to our learning theory framework whenever the true data-generating map does not belong to the RKHS (2.2) associated to the given random feature pair  $(\varphi ,\mu)$ . Specializing to this setting, suppose that  $(\varphi ,\mu)$  satisfy Assumption 3.1. Let  $K\colon (u,u^{\prime})\mapsto \mathbb{E}_{\theta \sim \mu}[\varphi (u;\theta)\otimes_{\mathcal{Y}}\varphi (u^{\prime};\theta)]$  be the corresponding limit random feature kernel. It holds that  $K(u,u)$  is trace-class for each  $u\in \mathcal{X}$ $\nu$ -almost surely because
$$
\operatorname {t r} (K (u, u)) = \mathbb {E} _ {\theta} \operatorname {t r} (\varphi (u; \theta) \otimes_ {\mathcal {Y}} \varphi (u; \theta)) = \mathbb {E} _ {\theta} \| \varphi (u; \theta) \| _ {\mathcal {Y}} ^ {2} \leq \| \varphi \| _ {L ^ {\infty}} ^ {2} <   \infty \tag {B.1}
$$
by the Fubini-Tonelli theorem. It follows from [8, Prop. 4.8, p. 394] that  $\mathcal{H}$  is compactly embedded into  $L_{\nu}^{2}(\mathcal{X};\mathcal{Y})$  because
$$
\left. \mathbb {E} _ {u \sim \nu} \| K (u, u) \| _ {\mathcal {L} (\mathcal {Y})} \leq \mathbb {E} _ {u \sim \nu} \operatorname {t r} (K (u, u)) \leq \| \varphi \| _ {L ^ {\infty}} ^ {2} <   \infty . \right. \tag {B.2}
$$
We denote the canonical embedding by  $\iota: \mathcal{H} \hookrightarrow L_{\nu}^{2}(\mathcal{X}; \mathcal{Y})$ . Now let  $\mathcal{G} \in L_{\nu}^{2}(\mathcal{X}; \mathcal{Y})$  be an arbitrary operator. We consider an approximation  $\mathcal{G}_{\vartheta}$  to  $\mathcal{G}$  defined by
$$
\mathcal {G} _ {\vartheta} := \underset {F \in \mathcal {H}} {\arg \min } \left\{\| \mathcal {G} - \iota F \| _ {L _ {\nu} ^ {2}} ^ {2} + \vartheta \| F \| _ {\mathcal {H}} ^ {2} \right\}. \tag {B.3}
$$
This operator has a simple representation.
Lemma B.1. There exists a unique solution  $\mathcal{G}_{\vartheta}$  to (B.3) given by
$$
\mathcal {G} _ {\vartheta} = \left(\iota^ {*} \iota + \vartheta \operatorname {I d}\right) ^ {- 1} \iota^ {*} \mathcal {G}. \tag {B.4}
$$
Proof. The result is a consequence of convex optimization on Hilbert spaces and the fact that  $\iota$  is a bounded linear operator.
The adjoint of the inclusion map,  $\iota^{*}\colon L_{\nu}^{2}(\mathcal{X};\mathcal{Y})\to \mathcal{H}$ , is given by the vector-valued reproducing kernel property as the (Bochner) integral operator
$$
F \mapsto \iota^ {*} F = \mathbb {E} _ {u \sim \nu} [ K (\cdot , u) F (u) ]. \tag {B.5}
$$
Since  $\iota$  is compact, so is  $\mathcal{K} \coloneqq u^{*} \in \mathcal{L}(L_{\nu}^{2}(\mathcal{X};\mathcal{Y}))$ . The action of  $\mathcal{K}$  is the same as that of the integral operator  $\iota^{*}$  above. Since  $\mathcal{K}$  is also symmetric, the spectral theorem yields the operator norm convergent expansion  $\mathcal{K} = \sum_{j} \lambda_{j} e_{j} \otimes_{L^{2}(\nu)} e_{j}$ . The sequence  $\{\lambda_{j}\} \subset \mathbb{R}_{\geq 0}$  is a non-increasing rearrangement of the eigenvalues of  $\mathcal{K}$  and  $\{e_{j}\}$  is its corresponding eigenbasis.
Now define the regularized RKHS approximation error
$$
\mathscr {A} _ {\mathcal {G}} (\vartheta) := \inf  _ {F \in \mathcal {H}} \left\{\| \mathcal {G} - \iota F \| _ {L _ {\nu} ^ {2}} ^ {2} + \vartheta \| F \| _ {\mathcal {H}} ^ {2} \right\} \tag {B.6}
$$
which is parametrized by  $\mathcal{G} \in L_{\nu}^{2}(\mathcal{X};\mathcal{Y})$ . We have the following convergence result for this error.
2The assumption of separability of the RKHS could be removed if additional conditions are placed on its kernel that would imply separability, the most relevant being continuity-type assumptions. In the case  $\mathcal{V} = \mathbb{R}$  it is known that existence of a Borel measurable feature map for the kernel suffices for separability of its RKHS [34], which is much weaker than continuity. However, we are unaware of similar results for general  $\mathcal{V}$ .
Lemma B.2 (Convergence of regularized RKHS approximation error). Suppose that  $\mathcal{G}$  is in the  $L_{\nu}^{2}$ -closure of  $\mathcal{H}$ . Under the prevailing assumptions of this section,
$$
\lim  _ {\vartheta \rightarrow 0} \mathscr {A} _ {\mathcal {G}} (\vartheta) = 0. \tag {B.7}
$$
Proof. By Lem. B.1 and the Woodbury identity [33, Thm. 1], it holds that
$$
\iota \mathcal {G} _ {\vartheta} = \iota \left(\iota^ {*} \iota + \vartheta \operatorname {I d} _ {\mathcal {H}}\right) ^ {- 1} \iota^ {*} \mathcal {G} = \mathcal {K} \left(\mathcal {K} + \vartheta \operatorname {I d} _ {L _ {\nu} ^ {2}}\right) ^ {- 1} \mathcal {G} = \left(\mathcal {K} + \vartheta \operatorname {I d} _ {L _ {\nu} ^ {2}}\right) ^ {- 1} \mathcal {K} \mathcal {G}.
$$
The second equality holds by simultaneous diagonalization. Writing  $\mathcal{G}$  in the eigenbasis of  $\kappa$  yields
$$
\mathcal {G} - \iota \mathcal {G} _ {\vartheta} = \left[ \operatorname {I d} _ {L _ {\nu} ^ {2}} - (\mathcal {K} + \vartheta \operatorname {I d} _ {L _ {\nu} ^ {2}}) ^ {- 1} \mathcal {K} \right] \mathcal {G} = \sum_ {j \in \mathbb {N}} \frac {\vartheta}{\lambda_ {j} + \vartheta} \langle e _ {j}, \mathcal {G} \rangle_ {L _ {\nu} ^ {2}} e _ {j}.
$$
Similarly, using the norm isometry between  $L_{\nu}^{2}$  and the RKHS [see, e.g., 8, pp. 403-404] we obtain
$$
\| \mathcal {G} _ {\vartheta} \| _ {\mathcal {H}} ^ {2} = \| \mathcal {K} ^ {- 1 / 2} \iota \mathcal {G} _ {\vartheta} \| _ {L _ {\nu} ^ {2}} ^ {2} = \| \mathcal {K} ^ {1 / 2} (\mathcal {K} + \vartheta \operatorname {I d} _ {L _ {\nu} ^ {2}}) ^ {- 1} \mathcal {G} \| _ {L _ {\nu} ^ {2}} ^ {2} = \sum_ {j \in \mathbb {N}} \left(\frac {\sqrt {\lambda_ {j}}}{\lambda_ {j} + \vartheta}\right) ^ {2} \langle e _ {j}, \mathcal {G} \rangle_ {L _ {\nu} ^ {2}} ^ {2}.
$$
Since the infimum in (B.6) is attained at  $\mathcal{G}_{\vartheta}$  (B.3), we deduce that