Title: Error Bounds for Learning with Vector-Valued Random Features
Venue: NeurIPS 2023
Accepted: yes
Keywords: random features

random feature model

operator learning

vector-valued
Abstract: This paper provides a comprehensive error analysis of learning with vector-valued random features (RF). The theory is developed for RF ridge regression in a fully general infinite-dimensional input-output setting, but nonetheless applies to and improves existing finite-dimensional analyses. In contrast to comparable work in the literature, the approach proposed here relies on a direct analysis of the underlying risk functional and completely avoids the explicit RF ridge regression solution formula in terms of random matrices. This removes the need for concentration results in random matrix theory or their generalizations to random operators. The main results established in this paper include strong consistency of vector-valued RF estimators under model misspecification and minimax optimal convergence rates in the well-specified setting. The parameter complexity (number of random features) and sample complexity (number of labeled data) required to achieve such rates are comparable with Monte Carlo intuition and free from logarithmic factors.

Error Bounds for Learning with Vector-Valued Random Features

Abstract

This paper provides a comprehensive error analysis of learning with vector-valued random features (RF). The theory is developed for RF ridge regression in a fully general infinite-dimensional input-output setting, but nonetheless applies to and improves existing finite-dimensional analyses. In contrast to comparable work in the literature, the approach proposed here relies on a direct analysis of the underlying risk functional and completely avoids the explicit RF ridge regression solution formula in terms of random matrices. This removes the need for concentration results in random matrix theory or their generalizations to random operators. The main results established in this paper include strong consistency of vector-valued RF estimators under model misspecification and minimax optimal convergence rates in the well-specified setting. The parameter complexity (number of random features) and sample complexity (number of labeled data) required to achieve such rates are comparable with Monte Carlo intuition and free from logarithmic factors.

1 Introduction

Supervised learning of an unknown mapping G : X → Y is a core task in machine learning. The random feature model (RFM), proposed in [36, 37], combines randomization with optimization to accomplish this task. The RFM is based on a linear expansion with respect to a randomized basis, the random features (RF). The coefficients in this RF expansion are optimized to fit the given data of input-output pairs. For popular loss functions, such as the square loss, the RFM leads to a convex optimization problem which can be solved efficiently and reliably.

The RFM provides a scalable approximation of an underlying kernel method [36, 37]. While the former is based on an expansion in M random features φ ( · ; θ 1 ) , . . . , φ ( · ; θ M ) , the corresponding kernel method relies on an expansion in values of a positive definite kernel function K ( · , u 1 ) , . . . , K ( · , u N ) on a dataset of size N . Kernel methods are conceptually appealing, theoretically sound, and have attracted considerable interest [1, 6, 40]. However, they require the storage, manipulation, and often inversion of the kernel matrix K with entries K ( u i , u j ) . The size of K scales quadratically in the number of samples N , which can be prohibitive for large datasets. When the underlying input-output map is vector-valued with dim( Y ) = p , the often significant computational cost of kernel methods is further exacerbated by the fact that each entry K ( u i , u j ) of K is, in general, a p -byp matrix. Hence, the size of K scales quadratically in both N and p . This severely limits the applicability of kernel methods to problems with high-dimensional, or indeed infinite-dimensional, output space. In contrast, learning with RF only requires storage of RF matrices whose size is quadratic in the number of features M . When M ≪ Np , this implies substantial computational savings, with the most extreme case being the infinite-dimensional setting in which p = ∞ .

In the context of operator learning, the underlying target mapping is an operator G : X → Y with infinite-dimensional input and output spaces. Such operators appear naturally in scientific computing

and often arise as solution maps of an underlying partial differential equation. Operator learning has attracted considerable interest, e.g., [3, 16, 20, 25, 27], and in this context, the RFM serves as an alternative to neural network-based methods with considerable potential for a sound theoretical basis. Indeed, an extension of the RFM to this infinite-dimensional setting has been proposed and implemented in [32]. Although the results show promise, a mathematical analysis of this approach including error bounds and rates of convergence has so far been outstanding.

Related work. Several papers have derived error bounds for learning with RF. Early work on the RFM [37] proceeded by direct inspection of the risk functional, demonstrating that M ≃ N random features suffice to achieve a squared error O (1 / √ N ) for RF ridge regression (RR). This result was considerably improved in [38], where √ N log N random features were shown to be sufficient to achieve the same squared error. This improvement in parameter complexity is based on the explicit RF RR solution formula, combined with extensive use of matrix analysis and matrix concentration inequalities. Similar analysis in [26] sharpens the parameter complexity to √ N log d λ K random features. Here d λ K is the number of effective degrees of freedom [2, 6], with λ the RR regularization parameter and K the kernel matrix. In this context, we also mention related analysis in [2]. In all these works, the squared error in terms of sample size N match the minimax optimal rates for kernel RR derived in [6]. Going beyond the above error bounds, [2, 26, 38] also derive fast rates under additional assumptions on the underlying data distribution and/or with improved RF sampling schemes.

Many works also study the interpolatory ( M ≃ N ) or overparametrized ( M ≫ N ) regimes in the scalar output setting [10, 14, 15, 18, 29]. However, when p = dim( Y ) ≫ 1 or p = ∞ , such regimes may no longer be relevant. This is because the kernel matrix K now has size Np -byNp , and it is possible that the number of random features M satisfies M ≪ Np even though M ≫ N . In this case, high-dimensional vector-valued learning naturally operates in the underparametrized regime.

In the area of operator learning for scientific problems, approximation results are common [11, 13, 19, 17, 22, 24, 39, 41] but statistical guarantees are lacking, the main exceptions being [4, 12, 21, 31, 43] in the linear operator setting and [6] in the nonlinear setting. The RFM also has potential for such nonlinear problems. Indeed, vector-valued random Fourier features have been studied before [5, 30]. However, theory is only provided for kernel approximation, not generalization guarantees.

To summarize, while previous analyses have provided considerable insight into the generalization properties of the RFM, they have almost exclusively focused on the scalar-valued setting. Given the paucity of theoretical work beyond this setting, it is a priori unclear whether similar estimates continue to hold when the RFM is applied to infinite-dimensional vector-valued mappings.

Contributions. The primary purpose of this paper is to extend earlier results on learning with random features to the vector-valued setting. The theory developed in this work unifies sources of error stemming from approximation, generalization, misspecification, and noisy observations. We focus on training via ridge regression with the square loss. Our results differ from existing work not only in the scope of applicability, but also in the strategy employed to derive our results. Similar to [37], we do not rely on the explicit random feature ridge regression solution (RF-RR) formula, which is specific to the square loss. One main benefit of this approach is that it entirely avoids the use of matrix concentration inequalities, thereby making the extension to an infinite-dimensional vector-valued setting straightforward. Our main contributions are now listed (see also Table 1).

(C1) Given N training samples, we prove that M ≃ √ N random features and regularization strength λ ≃ 1 / √ N is enough to guarantee that the squared error is O (1 / √ N ) , provided that the target operator belongs to a specific reproducing kernel Hilbert space (Thm. 3.7);

(C2) we establish that the vector-valued RF-RR estimator is strongly consistent (Thm. 3.10);

(C3) under additional regularity assumptions, we derive rates of convergence even when the target operator does not belong to the specific reproducing kernel Hilbert space (Thm. 3.12);

(C4) we demonstrate that the approach of Rahimi and Recht [37] can be used to derive state-ofthe-art rates for the RFM which, for the first time, are free from logarithmic factors.

Outline. The remainder of this paper is organized as follows. We set up the ridge regression problem in Sect. 2. The main results are stated in Sect. 3 and their proofs are sketched in Sect. 4.

Sect. 5 provides a simulation study and Sect. 6 gives concluding remarks. Detailed proofs are deferred to the supplementary material ( SM ).

2 Preliminaries

Wenowset up our vector-valued learning framework by introducing notational conventions, reviewing random features, and formulating the ridge regression problem.

Notation. Let (Ω , F , P ) be a sufficiently rich probability space on which all random variables in this paper are defined. Let X be the input space, Y the output space, and Θ a set. We consistently use u to denote elements of X and θ for RF parameters in Θ . The set of probability measures supported on a set Q is denoted by P ( Q ) . We write expectation (in the sense of Bochner integration) with respect to u ∼ ν ∈ P ( X ) as E u [ · ] and similarly for θ ∼ µ ∈ P (Θ) . Independent and identically distributed (iid) samples u 1 , . . . , u N from ν will be denoted by { u n } ∼ ν ⊗ N and similarly for { θ m } ∼ µ ⊗ M . We write a ≃ b to mean that there exists a constant C ≥ 1 such that C -1 b ≤ a ≤ Cb and similarly for the one-sided inequalities a ≲ b and a ≳ b . We define a ∧ b := min( a, b ) .

Random features and reproducing kernel Hilbert spaces. Random features are defined by a pair ( φ, µ ) , where φ : X × Θ →Y and µ ∈ P (Θ) . Fixing θ ∼ µ defines a map φ ( · ; θ ): X → Y . Considering linear combinations of such maps leads to the following definition.

Definition 2.1 (Random feature model) . The map Φ( · ; α ) = Φ( · ; α, { θ m } ): X → Y given by

is a random feature model (RFM) with coefficients α ∈ R M and fixed realizations { θ m } ∼ µ ⊗ M .

Associated to the pair ( φ, µ ) is a reproducing kernel Hilbert space (RKHS) H of maps from X to Y [32, Sect. 2.3]. Under mild assumptions (see SM B) assumed in our main results, it holds that

with RKHS norm ∥G∥ H = min α ∥ α ∥ L 2 µ , where α ranges over all decompositions of G of the form in (2.2). A minimizer α H of this problem always exists [2, Sect. 2.2]. We use this fact to identify any G ∈ H with its minimizing α H ∈ L 2 µ (Θ; R ) without further comment.

Random feature ridge regression. Let P ∈ P ( X × Y ) be the joint data distribution . The goal of RF-RR is to estimate an underlying operator G : X → Y from finitely many iid input-output pairs { ( u n , y n ) } N n =1 ∼ P ⊗ N , where typically the y n are noisy transformations of the point values G ( u n ) . To describe RF-RR, we first make some definitions.

Definition 2.2 (Empirical risk) . Writing Y = { y n } for the collection of observed output data and fixing a regularization parameter λ > 0 , the regularized Y -empirical risk of α ∈ R M is given by

is a scaled Euclidean norm on R M . The regularized G -empirical risk , R λ N ( α ; G ) , is defined analogously with G ( u n ) in place of y n . In the absence of regularization, i.e., λ = 0 , these expressions define the Y -empirical risk and G -empirical risk , denoted by R N ( α ; Y ) and R N ( α ; G ) , respectively.

RF-RR is the minimization problem min α ∈ R M R λ N ( α ; Y ) . The minimizer, which we denote by ̂ α , is referred to as trained coefficients and Φ( · ; ̂ α ) the trained RFM . For M and N sufficiently large and λ > 0 sufficiently small, we expect the trained RFM to well approximate G . This intuition is made precise by quantitative error bounds and statistical performance guarantees in the next section.

3 Main results

The main result of this paper is an abstract bound on the population squared error (Sect. 3.2). From this widely applicable theorem, several more specialized results are deduced. These include consistency (Sect. 3.3) and convergence rates (Sect. 3.4) of the RF-RR estimator trained on noisy data. The assumptions under which this theory is developed are provided next in Sect. 3.1.

3.1 Assumptions

Throughout this paper, we assume that the input space X is a Polish space and the output space Y is a real separable Hilbert space. These are common assumptions in learning theory [6]. We view X and Y as measurable spaces equipped with their respective Borel σ -algebras.

Next, we make the following minimal assumptions on the random feature pair ( φ, µ ) .

Assumption 3.1 (Random feature regularity) . Let ν ∈ P ( X ) be the input distribution and (Θ , Σ , µ ) be a probability space. The random feature map φ : X × Θ → Y and the probability measure µ ∈ P (Θ) are such that (i) φ is measurable; (ii) φ is uniformly bounded; in fact, ∥ φ ∥ L ∞ := ess sup ( u,θ ) ∼ ν ⊗ µ ∥ φ ( u ; θ ) ∥ Y ≤ 1 ; and (iii) the RKHS H corresponding to ( φ, µ ) is separable.

The boundedness assumption on φ is shared in general theoretical analyses of RF [26, 37, 38]; the unit bound can always be ensured by a simple rescaling. We work in a general misspecified setting.

Assumption 3.2 (Misspecification) . There exist ρ ∈ L ∞ ν ( X ; Y ) and G H ∈ H such that the operator G : X → Y satisfies the decomposition G = ρ + G H .

Since Assumption 3.1 implies that H ⊂ L ∞ ν ( X ; Y ) , any G = G H + ρ as in Assumption 3.2 is automatically bounded in the sense that G ∈ L ∞ ν ( X ; Y ) . Conversely, any G ∈ L ∞ ν ( X ; Y ) allows such a decomposition. We interpret ρ as a residual from the operator G H belonging to the RKHS. It may be prescribed by the problem, as we will see later in the context of discretization errors in operator learning (Ex. 3.9), or be arbitrary, as is customary in learning theory when the only information about G is that it is bounded.

Assumption 3.3 (Joint data distribution) . The joint distribution P ∈ P ( X × Y ) of the random variable ( u, y ) ∼ P is given by u ∼ ν with ν ∈ P ( X ) and y = G ( u ) + η . Here, G satisfies Assumption 3.2. The additive noise η is a random variable in Y that is conditionally centered, E [ η | u ] = 0 , and is subexponential: ∥ η ∥ ψ 1 ( Y ) < ∞ ; see (A.7) for the definition of ∥ · ∥ ψ 1 ( Y ) .

Our main goal is to recover G from iid data { ( u n , y n ) } arising from the following statistical model.

Assumption 3.3 implies that G ( u ) = E [ y | u ] . In contrast to related work [2, 26, 37], we allow for unbounded input-dependent noise. In particular, our results also hold for bounded or subgaussian noise, as well as multiplicative noise (e.g., η = ξ G ( u ) with E [ ξ | u ] = 0 and ∥ ξ ∥ ψ 1 < ∞ ).

3.2 General error bound

For any G , define the G -population risk functional or G -population squared error by

The main result of this paper establishes an upper bound for this quantity that holds with high probability, provided that the number of random features and number of data pairs are large enough.

Theorem 3.4 ( G -population squared error bound) . Suppose that G = ρ + G H satisfies Assumption 3.2. Fix a failure probability δ ∈ (0 , 1) , regularization strength λ ∈ (0 , 1) , and sample size N . Let { θ m } ∼ µ ⊗ M be the M random feature parameters and { ( u n , y n ) } ∼ P ⊗ N be the data according to Assumption 3.3. For Φ the RFM (2.1) satisfying Assumption 3.1, let ̂ α ∈ R M be the minimizer

of the regularized Y -empirical risk R λ N ( · ; Y ) given by (2.3) . If M ≥ λ -1 log(32 /δ ) and N ≥ λ -2 log(16 /δ ) , then

with probability at least 1 -δ , where

is a function of ρ , λ , G H , and the law of the noise variable η .

The main elements of the proof of Thm. 3.4 will be explained in Sect. 4.

Remark 3.5 (Excess risk) . We note that other work [26, 37, 38] often focuses on bounding the excess risk ̂ E := E (Φ( · ; ̂ α )) -inf G H ∈H E ( G H ) , where E ( F ) := E ∥ y -F ( u ) ∥ 2 Y = E u ∼ ν ∥G ( u ) -F ( u ) ∥ 2 Y + E ∥ η ∥ 2 Y . In particular, this bias-variance decomposition implies that ̂ E ≤ E u ∼ ν ∥G ( u ) -Φ( u ; ̂ α ) ∥ 2 Y . Thus, Thm. 3.4 also gives a corresponding bound on the excess risk ̂ E .

Remark 3.6 (The β factor) . In the well-specified setting, that is, G - G H = ρ ≡ 0 , the factor β in Thm. (3.4) satisfies the uniform bound

In particular, the constant B does not depend on λ in this case. Otherwise, β in general depends on λ . We can characterize this dependence precisely if it is known that G ∈ L ∞ ν ( X ; Y ) . In this case, Assumption 3.2 is satisfied with ρ := G - G H for any G H ∈ H . Choosing G H = G ϑ | ϑ = λ as in SM B (which is optimal in the sense described there) and a short calculation deliver the bound

if G additionally satisfies a particular r -th order regularity condition (see Lem. B.3 for the details). Here, a + := max( a, 0) for any a ∈ R . Thus, β is uniformly bounded if G ∈ H ( r ≥ 1) and grows algebraically as a power of λ -1 otherwise (0 ≤ r < 1) .

Consequences. The general error bound (3.2) in Thm. 3.4 has several implications for vector-valued learning with the RFM. First, it immediately implies a rate of convergence if G ∈ H .

Theorem 3.7 (Well-specified) . Instantiate the hypotheses and notation of Thm. 3.4. Suppose that ρ ≡ 0 so that G ∈ H (2.2) . If M ≥ λ -1 log(32 /δ ) and N ≥ λ -2 log(16 /δ ) , then with probability at least 1 -δ , where the constant B ≥ 0 is defined by (3.4) .

Given a number of samples N , Thm. 3.7 shows that RF-RR with regularization λ ≃ 1 / √ N and number of features M ≳ √ N leads to a population squared error of size 1 / √ N with high probability. This result should be compared to the previous state-of-the-art convergence rates in the literature for RF-RR with iid sampled features [2, 26, 37, 38]. See Table 1, which indicates that our analysis gives the lowest parameter complexity to date. We emphasize that such a convergence rate rests on the assumption that G ∈ H . This corresponds to a compatibility condition between G and the pair ( φ, µ ) , i.e., the random feature map φ and the probability measure µ , which determine the RKHS H . Designing suitable φ and µ for a given operator G remains an open problem. For an explanation of the poor parameter complexity in Rahimi and Recht's original paper [37], see [44, Appendix E].

Thm. 3.4 also implies convergence of R ( ̂ α ; G ) when G ̸∈ H , as we will see in Sect. 3.3 and 3.4. But first, the next corollary shows that the same general bound (3.2) also holds for the G H -population squared error R ( ̂ α ; G H ) , up to enlarged constant factors. The proof is given in SM C.

Corollary 3.8 ( G H -population squared error bound) . Instantiate the hypotheses and notation of Thm. 3.4. If M ≥ λ -1 log(32 /δ ) and N ≥ λ -2 log(16 /δ ) , then there exists an absolute constant C > 1 such that with probability at least 1 -δ , it holds that

Although our main goal is to learn G from noisy data, there are settings instead in which the learning of G H ∈ H as in Cor. 3.8 is of primary interest, but only values of some approximation G ∈ L ∞ ν ( X ; Y ) are available. The following example illustrates this.

Example 3.9 (Numerical discretization error) . One practically relevant setting to which Cor. 3.8 applies arises when training a RFM from functional data generated by a numerical approximation G = G ∆ of some underlying operator G H ∈ H . Here ∆ > 0 represents a numerical parameter, such as the grid resolution when approximating the solution operator of a partial differential equation. In this setting, ρ = G ∆ -G H is non-zero and it is crucial to include the discretization error in the analysis, which we define as ε ∆ := ∥ ρ ∥ 2 L ∞ ν = ∥G ∆ -G H ∥ 2 L ∞ ν . Assume η ≡ 0 , so that ̂ α minimizes R λ N ( · ; Y ) = R λ N ( · ; G ∆ ) . Using Cor. 3.8, it follows that for M and N sufficiently large,

with high probability. Thus, as suggested by intuition, in addition to the error contribution that is present when training on perfect data (the first term on the right-hand side), there is an additional discretization error of size ε ∆ . We also see that the performance of RF-RR is stable with respect to such discretization errors stemming from the training data. Actually obtaining a rate of convergence would require problem-specific information about the particular numerical solver that is used.

3.3 Statistical consistency

We now return to the objective of recovering G from data. In particular, suppose that G ̸∈ H ; the RKHS, viewed as a hypothesis class, is misspecified. Our analysis demonstrates that statistical guarantees for RF-RR are still possible in this setting.

To this end, assume that G ∈ L ∞ ν ( X ; Y ) . It follows that Assumption 3.2 is satisfied with ρ := G-G H and G H ∈ H being any element of the RKHS. Applying Thm. 3.4 and minimizing over G H yields

with probability at least 1 -δ if M ≳ λ -1 log(2 /δ ) and N ≳ λ -2 log(2 /δ ) . To obtain (3.9), we enlarged constants and used the bound ∥ ρ ∥ 2 L ∞ ν ≲ ∥G∥ 2 L ∞ ν + ∥G H ∥ 2 L ∞ ν in (3.3).

If G is in the L 2 ν -closure of H , then with high probability, the population squared error on the left hand side of (3.9) converges to zero as λ → 0 (by application of Lem. B.2 to the second term on the right). This is a statement about the (weak) statistical consistency of the trained RF-RR estimator; it can be upgraded to an almost sure statement, as expressed precisely in the next main result.

Theorem 3.10 (Strong consistency) . Suppose that G ∈ L ∞ ν ( X ; Y ) belongs to the L 2 ν ( X ; Y ) -closure of H (2.2) . Let { λ k } k ∈ N ⊂ (0 , 1) be a sequence of positive regularization parameters such that ∑ k ∈ N λ k < ∞ . For Φ the RFM (2.1) satisfying Assumption 3.1 and for each k , let ̂ α ( k ) ∈ R M k be the trained RFM coefficients that minimize the regularized Y -empirical risk R λ k N k ( · ; Y ) given by (2.3) with M k ≃ λ -1 k log(2 /λ k ) iid random features and N k ≃ λ -2 k log(2 /λ k ) iid data pairs under Assumption 3.3. It holds true that

The proof relies on a standard Borel-Cantelli argument. See SM C for the details.

Remark 3.11 (Universal RKHS) . The assumption that G belongs to the L 2 ν -closure of the RKHS H is automatically satisfied if H is dense in L 2 ν ( X ; Y ) . This is equivalent to its kernel being universal [7, 9]. In this case, the trained RFM is a strongly consistent estimator of any G ∈ L ∞ ν . However, we are unaware of any practical characterizations of universality of the kernel in terms of its corresponding random feature pair ( φ, µ ) for the vector-valued setting studied here.

3.4 Convergence rates

The previous subsection establishes convergence guarantees without any rates. We now establish quantitative bounds. Throughout what follows, we denote by K : L 2 ν ( X ; Y ) → L 2 ν ( X ; Y ) the integral operator (B.5) corresponding to the operator-valued kernel function of the RKHS H (see SM B).

Theorem 3.12 (Slow rates under misspecification) . Suppose that G ∈ L ∞ ν ( X ; Y ) and that Assumption 3.3 holds. Additionally, assume that G ∈ Im( K r/ 2 ) for some r > 0 , where K is the integral operator corresponding to the kernel of RKHS H (2.2) . Fix δ ∈ (0 , 1) and λ ∈ (0 , 1) . For

Φ the RFM (2.1) satisfying Assumption 3.1, let ̂ α ∈ R M minimize R λ N ( · ; Y ) given by (2.3) . If M ≥ λ -1 log(32 /δ ) and N ≥ λ -2 log(16 /δ ) , then with probability at least 1 -δ it holds that

The implied constant in (3.11) depends only on ∥G∥ L ∞ ν and ∥ η ∥ ψ 1 ( Y ) .

Thm. 3.12 provides a quantitative convergence rate as λ → 0 . For r ≥ 1 , i.e., when G ∈ H , we recover the linear convergence rate of order λ from Thm. 3.7. The assumption that G ∈ Im( K r/ 2 ) can be viewed as a 'fractional regularity' assumption on the underlying operator; indeed, in specific settings it corresponds to a fractional (Sobolev) regularity of the underlying function. In general, it appears difficult to check this condition in practice, which is one limitation of our result.

A quantitative analog to the almost sure statement of Thm. 3.10 also holds.

Corollary 3.13 (Strong convergence rate) . Instantiate the hypotheses and notation of Thm. 3.10. Assume in addition that G ∈ Im( K r/ 2 ) for some r > 0 . Let { λ k } k ∈ N ⊂ (0 , 1) be a sequence of positive regularization parameters such that ∑ k ∈ N λ k < ∞ . For each k , let ̂ α ( k ) ∈ R M k be the trained RFM coefficients with M k ≃ λ -1 k log(2 /λ k ) and N k ≃ λ -2 k log(2 /λ k ) . It holds true that

Short proofs of both Thm. 3.12 and Cor. 3.13 may be found in SM C.

4 Proof outline for the main theorem

Our main results are all derived from Thm. 3.4, whose proof, schematically illustrated in Figure 1, we now outline. Following [37], we break the proof into several steps that arise from the error decomposition

Sect. 4.1 estimates the first term on the right hand side of (4.1) while Sect. 4.2 estimates the second.

Theorem 3.4

R

(

̂

α

;

G

)

R

(

̂

α

;

G

)

R

λ

N

(

̂

α

;

G

) +

[

R

(

̂

α

;

G

)

-

R

N

(

̂

α

;

G

)

]

Proposition 4.1

:

∃

β

λ

∥

̂

α

∥

2

M

R

λ

N

(

̂

α

;

G

)

βλ

Proposition 4.7

: given

β

from 4.1, bounds

R

(

̂

α

;

G

)

-

R

N

(

̂

α

;

G

)

Lemma 4.3

R

λ

N

(

α

⋆

;

G

H

)

Lemma 4.5

2

N

∑

N

n

=1

∥

ρ

(

u

n

)

∥

2

Lemma 4.6

2

N

∑

N

n

=1

⟨-

η

n

,

Φ(

u

n

;

α

⋆

)

⟩

sup

α

∈A

1

∣

∣

∣

2

N

∑

N

n

=1

⟨

η

n

,

Φ(

u

n

;

α

)

⟩

∣

∣

∣

Lemma D.6 + Lemma D.7

: for any

t

, bounds

sup

α

∈A

t

|

R

(

α

;

G

)

-

R

N

(

α

;

G

)

|

4.1 Bounding the regularized empirical risk

The main technical contribution of this work is a tight bound on the G -empirical risk R N ( ̂ α ; G ) for the trained RFM. The analysis involves controlling several sources of error and careful truncation arguments to avoid unnecessarily strong assumptions on the problem. The result is the following.

Proposition 4.1 (Regularized G -empirical risk bound) . Let Assumptions 3.1 and 3.3 hold. Suppose that G = ρ + G H satisfies Assumption 3.2. Fix δ ∈ (0 , 1) , λ ∈ (0 , 1) , M ∈ N , and N ∈ N .

Let ̂ α ∈ R M be the minimizer of the regularized Y -empirical risk R λ N ( · ; Y ) given by (2.3) . If M ≥ λ -1 log(16 /δ ) and N ≥ λ -2 log(8 /δ ) , then with probability at least 1 -δ , where the multiplicative factor β ( ρ, λ, G H , η ) is given by (3.3) .

Since λ ∥ ̂ α ∥ 2 M ≤ R λ N ( ̂ α ; G ) , the next corollary controlling the norm (2.3) of ̂ α is immediate. It plays a crucial role in developing an upper bound for the second term on the right side of (4.1).

Corollary 4.2 (Trained RFM norm bound) . Instantiate the hypotheses and notation of Prop. 4.1. Fix δ ∈ (0 , 1) and λ ∈ (0 , 1) . If M ≥ λ -1 log(16 /δ ) and N ≥ λ -2 log(8 /δ ) , then

with probability at least 1 -δ . The radius β := β ( ρ, λ, G H , η ) of the norm bound is given by (3.3) .

The core elements of the proof of Prop. 4.1 are provided in the next few subsections, with the full argument in SM D.1. The main idea is to upper bound the G -empirical risk by its regularized counterpart and then decompose the latter into several (coupled) error contributions.

To do this, first fix any α ∈ R M . It holds that

because Y is a Hilbert space and y n = G ( u n ) + η n . Using this, a short calculation shows that

In the second line, we used the fact that ̂ α minimizes R λ N ( · ; Y ) . Since α ∈ R M is arbitrary, we have the freedom to choose α so that the first term is small (see Sect. 4.1.1 and 4.1.2). With α fixed, the second term averages to zero by our assumptions on the noise, and hence, we expect it to be small with high probability (see Sect. 4.1.3).

The third term in (4.5) exhibits high correlation between the noise η n and the trained RFM coefficients ̂ α , making it more difficult to estimate. To control this last term, we first note that it is homogeneous in ∥ ̂ α ∥ M , which can be used to derive an upper bound in terms of a supremum over the unit ball with respect to ∥ · ∥ M . The resulting expression is then bounded with empirical process techniques (see SM D.1.3). For the complete details of the required argument we refer the reader to SM D.1.

In the remainder of this subsection, we estimate the first two terms on the right hand side of (4.5). Using the fact that G = ρ + G H , the first term can be split into two contributions,

These contributions to the first term in (4.5) are bounded in Sect. 4.1.1 and 4.1.2. The second term in (4.5) is controlled in Sect. 4.1.3.

4.1.1 Bounding the approximation error

We begin with the term R λ N ( α ; G H ) , which may be viewed as an empirical approximation error due to α being arbitrary. Its only dependence on the data is through { u n } in (2.3). Intuitively, this term should behave like its population counterpart. It is then natural to choose a Monte Carlo approximation α m = α H ( θ m ) for α , where α H ∈ L 2 µ (Θ; R ) is identified with G H as in (2.2). However, our intuition that λ ∥ α ∥ 2 M should concentrate around λ ∥ α H ∥ 2 L 2 µ fails because it is generally not possible to control the tail of the random variable | α H ( θ ) | 2 . We next show that this problem can be overcome by a carefully tuned truncation argument combined with Bernstein's inequality.

Lemma 4.3 (Construction of approximator) . Suppose that G H := G ∈ H . Fix δ ∈ (0 , 1) , λ > 0 , N ∈ N , and M ∈ N . Let { θ m } ∼ µ ⊗ M and { u n } ∼ ν ⊗ N . Define α ⋆ ∈ R M componentwise by

and G H = E θ ∼ µ [ α H ( θ ) φ ( · ; θ )] with ∥G H ∥ 2 H = E θ ∼ µ | α H ( θ ) | 2 . If M ≥ λ -1 log(4 /δ ) , then with probability at least 1 -δ in the random feature parameters θ 1 , . . . , θ M , it holds that

SM D.1.1 provides the proof.

Remark 4.4 (Well-specified and noise-free) . Lem. 4.3 gives a O ( λ ) bound on the regularized G H -empirical risk of a RFM trained on well-specified and noise-free iid data { u n , G H ( u n ) } .

4.1.2 Bounding the misspecification error

The second contribution to (4.6) is easily bounded by Bernstein's inequality because ρ ∈ L ∞ ν . We refer the reader to SM D.1.2 for the detailed proof.

Lemma 4.5 (Concentration of misspecification error) . Let ρ be as in Assumption 3.2. Fix δ ∈ (0 , 1) . With probability at least 1 -δ , it holds that

4.1.3 Bounding the noise error

The second term on the right hand side of (4.5) is a zero mean error contribution due to the noise corrupting the output training data. By the fact that η is subexponential (Assumption 3.3), Bernstein's inequality delivers exponential concentration. The proof details are in SM D.1.3.

Lemma 4.6 (Concentration of noise error cross term) . Let Assumptions 3.1 and 3.3 hold. Fix α ∈ R M , { θ m } ∼ µ ⊗ M , and δ ∈ (0 , 1) . With probability at least 1 -δ , it holds that

SM D.1.3 also details the techniques used to upper bound the third and final term in (4.5).

4.2 Bounding the generalization gap

Having bounded the empirical risk with approximation arguments, it remains to control the estimation error R ( ̂ α ; G ) -R N ( ̂ α ; G ) due to finite data in (4.1). We call this the generalization gap : the difference between the population test error and its empirical version. If ̂ α satisfies ∥ ̂ α ∥ 2 M ≤ t for some t > 0 , then one can upper bound the generalization gap by its supremum over this set. The main challenge is to show existence of a (sufficiently small) t that satisfies this inequality. This is handled by Cor. 4.2. As summarized in the following proposition, the resulting supremum of the empirical process defined by the generalization gap is shown to be of size N -1 / 2 with high probability.

Proposition 4.7 (Uniform bound on the generalization gap) . Let Assumption 3.1 hold. Suppose G satisfies Assumption 3.2. Let { θ m } ∼ µ ⊗ M for the RFM Φ given by (2.1) . Fix δ ∈ (0 , 1) . For iid input samples { u n } ∼ ν ⊗ N , define the random variable

where A β := { α ′ ∈ R M | ∥ α ′ ∥ 2 M ≤ β } and the deterministic radius β = β ( ρ, λ, G H , η ) is given in (3.3) with G as above. If N ≥ log(1 /δ ) , then with probability at least 1 -δ it holds that

The proof of Prop. 4.7 is given in SM D.2. The argument is composed of two steps. The first is to show that E β | { θ m } concentrates around its (conditional) expectation (Lem. D.6). This follows easily using the boundedness of the summands. The second step is to upper bound the expectation of E β | { θ m } (Lem. D.7). This is achieved by exploiting the Hilbert space structure of the Y -square loss and the linearity of the RFM with respect to its coefficients.

4.3 Combining the bounds

Since we now have control over the G -empirical risk and the generalization gap, the G -population risk is also under control by (4.1). The proof of Thm. 3.4 follows by putting together the pieces ( SM C).

5 Numerical experiment

To study how our theory holds up in practice, we numerically implement the vector-valued RF-RR algorithm on a benchmark operator learning dataset. The data { ( u n , G ( u n )) } N n =1 is noise-free, the { u n } are iid Gaussian random fields, and G : L 2 ( T ; R ) → L 2 ( T ; R ) is a nonlinear operator defined as the time one flow map of the viscous Burgers' equation on the torus T . SM E provides more details about the problem setting and the choice of random feature pair ( φ, µ ) .

10

1

10

2

10

3

M

10

-

3

10

-

2

10

-

1

Relative Bochner squared error

p

= 256

p

= 1024

p

= 4096

M

-

1

10

1

10

2

10

3

N

10

-

3

10

-

2

p

= 256

p

= 1024

p

= 4096

N

-

1

/

2

Figure 2a shows the decay of the relative squared test error as M increases (with λ ≃ 1 /M ) for fixed N . The error closely follows the rate O ( M -1 ) until it begins to saturate at larger M . This is due to either G not belonging to the RKHS of ( φ, µ ) or the finite data error dominating. As implied by our theory, the error does not depend on the discretized output dimension p < ∞ . Figure 2b displays similar behavior as N is varied (now with λ ≃ 1 / √ N and fixed M ). Overall, the observed parameter and sample complexity reasonably validate our theoretical insights.

6 Conclusion

This paper establishes several fundamental results for learning with infinite-dimensional vectorvalued random features; these include strong consistency and explicit convergence rates. When the underlying mapping belongs to the RKHS, the rates obtained in this work match minimax optimal rates in the number of samples N , requiring only a number of random features M ≃ √ N . Despite being derived in a very general setting, to the best of our knowledge, this provides the sharpest parameter complexity in M , which is free from logarithmic factors for the first time.

There are several interesting directions for future work. These include deriving fast rates, relaxing the boundedness assumption on the features and the true mapping, and accommodating heavier-tailed or white noise distributions. Obtaining fast rates would require a sharpening of several estimates, and in particular, replacing the global Rademacher complexity-type estimate, implicit in our work, by its local counterpart. As our approach does not make use of an explicit solution formula, which is only available for a square loss, this might pave the way for improved rates for other loss functions, such as a general L p -loss. We leave such potential extensions of the present approach for future work.

Acknowledgments and Disclosure of Funding

The work of SL is supported by Postdoc.Mobility grant P500PT-206737 from the Swiss National Science Foundation. NHN acknowledges support from the National Science Foundation Graduate Research Fellowship Program under award number DGE-1745301 and from the Amazon/Caltech AI4Science Fellowship, and partial support from the Air Force Office of Scientific Research under MURI award number FA9550-20-1-0358 (Machine Learning and Physics-Based Modeling and Simulation). SL and NHN are also grateful for partial support from the Department of Defense Vannevar Bush Faculty Fellowship held by Andrew M. Stuart under Office of Naval Research award number N00014-22-1-2790.

References

N. Aronszajn. Theory of reproducing kernels. Transactions of the American mathematical society , 68(3):337-404, 1950. (Cited on page 1)

F. Bach. On the equivalence between kernel quadrature rules and random feature expansions. The Journal of Machine Learning Research , 18(1):714-751, 2017. (Cited on pages 2, 3, 4, and 5)

K. Bhattacharya, B. Hosseini, N. B. Kovachki, and A. M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI journal of computational mathematics , 7: 121-157, 2021. doi: 10.5802/smai-jcm.74. URL https://smai-jcm.centre-mersenne. org/articles/10.5802/smai-jcm.74/ . (Cited on page 2)

N. Boullé and A. Townsend. Learning elliptic partial differential equations with randomized linear algebra. Foundations of Computational Mathematics , pages 1-31, 2022. (Cited on page 2)

R. Brault, M. Heinonen, and F. Buc. Random fourier features for operator-valued kernels. In Asian Conference on Machine Learning , pages 110-125. PMLR, 2016. (Cited on page 2)

A. Caponnetto and E. De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics , 7(3):331-368, 2007. (Cited on pages 1, 2, 4, and 14)

A. Caponnetto, C. A. Micchelli, M. Pontil, and Y. Ying. Universal multi-task kernels. The Journal of Machine Learning Research , 9:1615-1646, 2008. (Cited on page 6)

C. Carmeli, E. De Vito, and A. Toigo. Vector valued reproducing kernel Hilbert spaces of integrable functions and Mercer theorem. Analysis and Applications , 4(04):377-408, 2006. (Cited on pages 16 and 17)

C. Carmeli, E. De Vito, A. Toigo, and V. Umanitá. Vector valued reproducing kernel hilbert spaces and universality. Analysis and Applications , 8(01):19-61, 2010. (Cited on page 6)

Z. Chen, H. Schaeffer, and R. Ward. Concentration of Random Feature Matrices in HighDimensions. In Mathematical and Scientific Machine Learning , pages 287-302. PMLR, 2022. (Cited on page 2)

A. Cohen, R. Devore, and C. Schwab. Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDE's. Analysis and Applications , 9(01):11-47, 2011. (Cited on page 2)

M. V. de Hoop, N. B. Kovachki, N. H. Nelsen, and A. M. Stuart. Convergence Rates for Learning Linear Operators from Noisy Data. SIAM/ASA Journal on Uncertainty Quantification , 11(2):480-513, 2023. doi: 10.1137/21M1442942. (Cited on page 2)

B. Deng, Y. Shin, L. Lu, Z. Zhang, and G. E. Karniadakis. Approximation rates of DeepONets for learning operators arising from advection-diffusion equations. Neural Networks , 153: 411-426, 2022. (Cited on page 2)

W. E, C. Ma, and L. Wu. The generalization error of the minimum-norm solutions for overparameterized neural networks. Pure and Applied Functional Analysis , 5(6):1445-1460, 2020. (Cited on page 2)

B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari. Linearized two-layers neural networks in high dimension. Ann. Statist. , 49:1029-1054, 2021. (Cited on page 2)

C. R. Gin, D. E. Shea, S. L. Brunton, and J. N. Kutz. DeepGreen: deep learning of Green's functions for nonlinear boundary value problems. Scientific reports , 11(1):21614, 2021. (Cited on page 2)

L. Gonon, L. Grigoryeva, and J.-P. Ortega. Approximation bounds for random neural networks and reservoir systems. The Annals of Applied Probability , 33(1):28-69, 2023. (Cited on page 2)

A. Hashemi, H. Schaeffer, R. Shi, U. Topcu, G. Tran, and R. Ward. Generalization bounds for sparse random feature expansions. Applied and Computational Harmonic Analysis , 62: 310-330, 2023. (Cited on page 2)

L. Herrmann, C. Schwab, and J. Zech. Neural and gpc operator surrogates: construction and expression rate bounds, 2022. (Cited on page 2)

J. Hesthaven and S. Ubbiali. Non-intrusive reduced order modeling of nonlinear problems using neural networks. Journal of Computational Physics , 363:55-78, 2018. ISSN 0021-9991. doi: https://doi.org/10.1016/j.jcp.2018.02.037. URL https://www.sciencedirect.com/ science/article/pii/S0021999118301190 . (Cited on page 2)

J. Jin, Y. Lu, J. Blanchet, and L. Ying. Minimax Optimal Kernel Operator Learning via Multilevel Training. arXiv preprint arXiv:2209.14430 , 2022. (Cited on page 2)

N. Kovachki, S. Lanthaler, and S. Mishra. On Universal Approximation and Error Bounds for Fourier Neural Operators. Journal of Machine Learning Research , 22(290):1-76, 2021. URL http://jmlr.org/papers/v22/21-0806.html . (Cited on page 2)

N. B. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. M. Stuart, and A. Anandkumar. Neural Operator: Learning Maps Between Function Spaces With Applications to PDEs. J. Mach. Learn. Res. , 24(89):1-97, 2023. (Cited on page 27)

S. Lanthaler, S. Mishra, and G. E. Karniadakis. Error estimates for DeepONets: a deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications , 6(1), 03 2022. ISSN 2398-4945. doi: 10.1093/imatrm/tnac001. URL https://doi.org/10.1093/ imatrm/tnac001 . tnac001. (Cited on page 2)

Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier Neural Operator for Parametric Partial Differential Equations. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum? id=c8P9NQVtmnO . (Cited on page 2)

Z. Li, J.-F. Ton, D. Oglic, and D. Sejdinovic. Towards a unified analysis of random Fourier features. Journal of Machine Learning Research , 22(108), 2021. (Cited on pages 2, 3, 4, and 5)

L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence , 3(3):218-229, 2021. (Cited on page 2)

A. Maurer and M. Pontil. Concentration inequalities under sub-Gaussian and sub-exponential conditions. Advances in Neural Information Processing Systems , 34:7588-7597, 2021. (Cited on pages 14 and 16)

S. Mei, T. Misiakiewicz, and A. Montanari. Generalization error of random feature and kernel methods: hypercontractivity and kernel matrix concentration. Applied and Computational Harmonic Analysis , 59:3-84, 2022. (Cited on page 2)

H. Q. Minh. Operator-valued Bochner theorem, Fourier feature maps for operator-valued kernels, and vector-valued learning. arXiv preprint arXiv:1608.05639 , 2016. (Cited on page 2)

M. Mollenhauer, N. Mücke, and T. Sullivan. Learning linear operators: Infinite-dimensional regression as a well-behaved non-compact inverse problem. arXiv preprint arXiv:2211.08875 , 2022. (Cited on pages 2 and 15)

N. H. Nelsen and A. M. Stuart. The random feature model for input-output maps between Banach spaces. SIAM Journal on Scientific Computing , 43(5):A3212-A3243, 2021. (Cited on pages 2, 3, 16, and 27)

H. Ogawa. An operator pseudo-inversion lemma. SIAM Journal on Applied Mathematics , 48 (6):1527-1531, 1988. (Cited on page 17)

H. Owhadi and C. Scovel. Separability of reproducing kernel spaces. Proceedings of the American Mathematical Society , 145(5):2131-2138, 2017. (Cited on page 16)

I. F. Pinelis and A. I. Sakhanenko. Remarks on inequalities for large deviation probabilities. Theory of Probability & Its Applications , 30(1):143-148, 1986. (Cited on pages 14 and 15)

A. Rahimi and B. Recht. Random features for large-scale kernel machines. Advances in neural information processing systems , 20, 2007. (Cited on page 1)

A. Rahimi and B. Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. Advances in neural information processing systems , 21, 2008. (Cited on pages 1, 2, 3, 4, 5, and 7)

A. Rudi and L. Rosasco. Generalization properties of learning with random features. Advances in neural information processing systems , 30, 2017. (Cited on pages 2, 3, 4, 5, and 14)

T. D. Ryck and S. Mishra. Generic bounds on the approximation error for physics-informed (and) operator learning. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum? id=bF4eYy3LTR9 . (Cited on page 2)

B. Scholkopf and A. J. Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond . MIT press, 2018. (Cited on page 1)

C. Schwab and J. Zech. Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ. Analysis and Applications , 17(01):19-55, 2019. (Cited on page 2)

S. Smale and D.-X. Zhou. Shannon sampling II: Connections to learning theory. Applied and Computational Harmonic Analysis , 19(3):285-302, 2005. (Cited on pages 16 and 17)

G. Stepaniants. Learning partial differential equations in reproducing kernel Hilbert spaces. Journal of Machine Learning Research , 24(86):1-72, 2023. (Cited on page 2)

Y. Sun, A. Gilbert, and A. Tewari. But how does it work in theory? Linear SVM with random features. Advances in Neural Information Processing Systems , 31, 2018. (Cited on page 5)

R. Vershynin. High-dimensional probability: An introduction with applications in data science, 2018. (Cited on page 15)

M. J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press, 2019. (Cited on page 25)

Supplementary Material for:

Error Bounds for Learning with Vector-Valued Random Features

A Concentration of measure

In this section, we recall two classical results from [35] that estimate the difference between empirical averages and true averages of random vectors taking values in Banach spaces. These are then specialized to the setting of subexponential random variables, which play a major role in this paper. To set the notation, we use Pr to denote probability with respect to the underlying probability space.

The first result is a vector-valued Bernstein concentration inequality with various applications to problems posed in infinite-dimensional Hilbert spaces [6, 28, 38]. It is used throughout this work.

Theorem A.1 (Vector-valued Bernstein inequality in Hilbert space) . Let Z be an H -valued random variable, where ( H, ⟨· , ·⟩ , ∥ · ∥ ) is a separable Hilbert space. Suppose there exist positive numbers b > 0 and σ > 0 such that

For any δ ∈ (0 , 1) and N ∈ N , denoting by { Z n } N n =1 a sequence of N iid copies of Z , it holds that

Proof. The result is a direct consequence of [35, Cor. 1, p. 144] in the iid Hilbert space setting. In this case, it holds for any t > 0 that

where S n := 1 N ∑ N n =1 Z n . Setting the right hand side equal to δ , solving a quadratic equation for t = t ( δ ) , and using the inequality √ a + b ≤ √ a + √ b to bound t ( δ ) from above leads to (A.2).

The most common use case of Bernstein's inequality is in the following bounded setting.

Lemma A.2. Let Z be a (potentially) uncentered random variable such that

for some c > 0 and v > 0 . Then Z satisfies Bernstein's moment condition (A.1) with b = 2 c and σ = v . If E Z = 0 , then taking b = c suffices.

Proof. It holds that ∥ Z -E Z ∥ ≤ ∥ Z ∥ + E ∥ Z ∥ ≤ 2 c almost surely. We compute

because 1 p ! / 2 for all p 2 . The centered improvement is trivial.

The second classic result we present is a one-sided Bernstein-type tail bound in a general Banach space. We invoke this theorem to control the tails of suprema of empirical processes.

Theorem A.3 (Vector-valued Bernstein inequality in Banach space) . Let Z be a Z -valued random variable, where ( Z , ∥ · ∥ ) is a separable Banach space. Suppose there exist positive numbers b > 0 and σ > 0 such that

For any δ ∈ (0 , 1) and N ∈ N , denoting by { Z n } N n =1 a sequence of N iid copies of Z , it holds that

Proof. The assertion is [35, Cor. 1, p. 144] in the iid Banach space setting (hence only convergence of norms in (A.5) instead of strong convergence). It is proved the same way as Thm. A.1.

In Theorem A.3, a random variable Z satisfying the Bernstein moment condition (A.4) is subexponential in the sense that ∥ Z -E Z ∥ is subexponential on R , i.e., exhibits exponential tail decay. Recall that for a real-valued random variable Z , its subexponential norm may be defined by

See [45, Sect. 2.7] for equivalent definitions. We say that Z is subexponential if its subexponential norm is finite. Following [31, Sect. 4.3, pp. 19-20], for a random variable Z with values in a Banach space ( Z , ∥ · ∥ Z ) we define

as its subexponential norm. It is known that a random variable has finite subexponential norm if and only if it satisfies the Bernstein moment condition (A.4) [see, e.g., 31, Appendix A.2]. Next, we give explicit constants in the Bernstein moment condition that depend on the subexponential norm.

Proposition A.4 (Subexponential implies Bernstein moment condition) . Let Z be a ( Z , ∥ · ∥ ) -valued subexponential random variable, that is, ∥ Z ∥ ψ 1 ( Z ) < ∞ . Then Z satisfies

where

Proof. By the Cauchy-Schwarz inequality,

The inequality | a + b | q ≤ 2 q -1 ( | a | q + | b | q ) , Jensen's inequality, and ∥ Z ∥ ψ 1 ( Z ) < ∞ show that

Next, Stirling's approximation ( q/e ) q ≤ q ! and q ! = q ( q -1)! ≥ 2( q -1)! for q ≥ 2 yields

Rearranging the exponents to fit the Bernstein moment condition form completes the proof.

This leads to the following corollary, which is useful in the setting that the variance E ∥ Z -E Z ∥ 2 Z = ∥ Z -E Z ∥ 2 L 2 P (Ω; Z ) of random variable Z is not small or hard to compute.

Corollary A.5 (Subexponential tail bound in Banach space) . Fix N ∈ N . Let { Z n } N n =1 be iid random variables with values in a separable Banach space ( Z , ∥ · ∥ ) . Suppose that ∥ Z 1 ∥ ψ 1 ( Z ) < ∞ . Let S N := 1 N ∑ N n =1 Z n . Fix δ ∈ (0 , 1) . With probability at least 1 -δ , it holds that

In particular, if N ≥ log(1 /δ ) , then with probability at least 1 -δ it holds that

Proof. Apply Prop. A.4 to Thm. A.3 to obtain the first assertion. For the second assertion, first note that E ∥ Z 1 -E Z 1 ∥ 2 ≤ 4 E ∥ Z 1 ∥ 2 (by triangle inequality and using ( a + b ) 2 ≤ 2( a 2 + b 2 ) ) and ∥ Z 1 ∥ 2 ψ 1 ( Z ) ≥ E ∥ Z 1 ∥ 2 / 4 (by (A.7)). Since N ≥ log(1 /δ ) , we have log(1 /δ ) /N ≤ √ log(1 /δ ) /N . Combining these facts, it follows that the right hand side of (A.10) is bounded above by

Comparing this result to a similar result [28, Prop. 7(i), pp. 4-5, the iid case], we note that (A.10) in Cor. A.5 is sharper in the sense that the constant in the N -1 / 2 term depends on the variance of the summands instead of just its subexponential norm (which can be much larger than the variance).

B Misspecification error with RKHS methods

Let K : X × X → L ( Y ) be an operator-valued kernel [32] corresponding to a separable 2 RKHS H of functions mapping X to Y . Here, L ( Y ) denotes the set of bounded linear operators from Y into itself. In this section, we present a general analysis based on [42] for the approximation of elements in L 2 ν ( X ; Y ) by elements in the RKHS H . This problem is relevant to our learning theory framework whenever the true data-generating map does not belong to the RKHS (2.2) associated to the given random feature pair ( φ, µ ) . Specializing to this setting, suppose that ( φ, µ ) satisfy Assumption 3.1. Let K : ( u, u ′ ) ↦→ E θ ∼ µ [ φ ( u ; θ ) ⊗ Y φ ( u ′ ; θ )] be the corresponding limit random feature kernel. It holds that K ( u, u ) is trace-class for each u ∈ X ν -almost surely because by the Fubini-Tonelli theorem. It follows from [8, Prop. 4.8, p. 394] that H is compactly embedded into L 2 ν ( X ; Y ) because

We denote the canonical embedding by ι : H ↪ → L 2 ν ( X ; Y ) . Now let G ∈ L 2 ν ( X ; Y ) be an arbitrary operator. We consider an approximation G ϑ to G defined by

This operator has a simple representation.

Lemma B.1. There exists a unique solution G ϑ to (B.3) given by

Proof. The result is a consequence of convex optimization on Hilbert spaces and the fact that ι is a bounded linear operator.

The adjoint of the inclusion map, ι ∗ : L 2 ν ( X ; Y ) → H , is given by the vector-valued reproducing kernel property as the (Bochner) integral operator

Since ι is compact, so is K := ιι ∗ ∈ L ( L 2 ν ( X ; Y )) . The action of K is the same as that of the integral operator ι ∗ above. Since K is also symmetric, the spectral theorem yields the operator norm convergent expansion K = ∑ j λ j e j ⊗ L 2 ( ν ) e j . The sequence { λ j } ⊂ R ≥ 0 is a non-increasing rearrangement of the eigenvalues of K and { e j } is its corresponding eigenbasis.

Now define the regularized RKHS approximation error

which is parametrized by G ∈ L 2 ν ( X ; Y ) . We have the following convergence result for this error.

Lemma B.2 (Convergence of regularized RKHS approximation error) . Suppose that G is in the L 2 ν -closure of H . Under the prevailing assumptions of this section,

Proof. By Lem. B.1 and the Woodbury identity [33, Thm. 1], it holds that

The second equality holds by simultaneous diagonalization. Writing G in the eigenbasis of K yields

Similarly, using the norm isometry between L 2 ν and the RKHS [see, e.g., 8, pp. 403-404] we obtain

Since the infimum in (B.6) is attained at G ϑ (B.3), we deduce that

Using ϑ/ ( λ j + ϑ ) ≤ 1 for each j ∈ N and G ∈ H L 2 ν (the L 2 ν -closure of H ), it follows that

̸

by dominated convergence.

The rate of convergence of A G to zero can be quantified under an additional regularity assumption.

Lemma B.3 (Convergence rate under Hölder source condition) . Suppose G ∈ Im( K r/ 2 ) for some r ≥ 0 . Then for any ϑ > 0 , it holds under the prevailing assumptions of this section that

Proof. The proof closely follows the argument of Smale and Zhou [42, Thm. 4, p. 295]. By hypothesis, there exists F G ∈ L 2 ν ( X ; Y ) such that G = K r/ 2 F G . Then

For any j ∈ N , the argument of the supremum equals

This is bounded above by ϑ r for all j ∈ N if r ≥ 0 and r ≤ 1 . Otherwise, r > 1 and

Taking the supremum over j ∈ N completes the proof.

In Lem. B.3, G satisfies G ∈ H if r ≥ 1 , and the rate of convergence of A G ( ϑ ) is at least as fast as O ( ϑ ) as ϑ → 0 . When r ∈ [0 , 1) , then G / ∈ H and the rate becomes slower than linear in ϑ .

C Proofs for section 3

In this section, we prove Thm. 3.4 and its main consequences.

Proof of Theorem 3.4. Under the hypotheses, (4.2) in Prop. 4.1 holds with probability at least 1 -δ provided that M ≥ λ -1 log(16 /δ ) and N ≥ λ -2 log(8 /δ ) . That is, R N ( ̂ α ; G ) ≤ R λ N ( ̂ α ; G ) ≤ βλ , where β = β ( ρ, λ, G H , η ) is given by (3.3). In particular, ̂ α ∈ A β (4.3) on the same event (by Cor. 4.2). It follows that, on this event,

Application of Prop. 4.7 shows that the right hand side of the above display is bounded above by

with probability at least 1 -δ because N ≥ λ -2 log(8 /δ ) ≥ log(2 /δ ) . Recalling (4.1), using 1 ≤ 79 e 3 / 2 , and applying a union bound completes the proof.

Proof of Corollary 3.8. Since G = ρ + G H , we compute

By (3.2) and (3.3), we see that the term E ∥ ρ ( u ) ∥ 2 Y also appears in the upper bound for R ( ̂ α ; G ) . Collecting like terms and enlarging constants proves the assertion.

Proof of Theorem 3.10. For k ∈ N and δ ∈ (0 , 1) , the trained RFM satisfies ∥G Φ( · ; ̂ α ( k ) ) ∥ L 2 ν ≤ cs k for some deterministic constant c > 0 , where the sequence s n → 0 as n →∞ is given by the right hand side of (3.9) with λ = λ n for each n ∈ N (by Lem. B.2). This inequality holds with probability at least 1 -δ if M ≳ λ -1 k log(2 /δ ) and N ≳ λ -2 k log(2 /δ ) by Thm. 3.4. Now choose δ = λ k . Then

The first Borel-Cantelli lemma establishes that there exists an N -valued random variable k 0 such that ∥G Φ( · ; ̂ α ( k ) ) ∥ L 2 ν ≤ cs k for all k > k 0 almost surely. This implies the desired result.

Proof of Theorem 3.12. Application of Thm. 3.4 leads to the high probability bound (3.9) by the same argument from Sect. 3.3. Using that λ ≲ 1 and Lem. B.3 proves the assertion.

Proof of Corollary 3.13. Apply Thm. 3.12 to get a high probability bound. Choose λ = λ k = δ . Then the proof follows that of Thm. 3.10 except with { s k } replaced by { λ ( r ∧ 1) / 2 k } .

D Proofs for section 4

This section provides proofs of the error bounds for the regularized empirical risk ( SM D.1) and the generalization gap ( SM D.2).

D.1 Proofs for subsection 4.1: Bounding the regularized empirical risk

We now prove Prop. 4.1. Supporting results used in the proof appear after the argument in the subsequent subsections ( SM D.1.1, D.1.2, and D.1.3).

Proof of Proposition 4.1. Our starting point is (4.5). We first note by linearity that

where A 1 = { α ′ ∈ R M ∣ ∣ ∥ α ′ ∥ 2 M ≤ 1 } , provided that ∥ ̂ α ∥ M > 0 . Otherwise, the inequality in the above display holds trivially. Next, we define

The inequality in (4.5) and the arithmetic-mean-geometric-mean inequality imply that

Subtracting λt/ 2 from both sides and multiplying through by 2 λ -1 yields

Substituting (D.5) back into the right-most side of (D.4) gives

All of the above calculations hold with probability one. To complete our estimate of R λ N ( ̂ α ; G ) , it remains to upper bound the A λ N,M (D.2) and c N,M (D.3) terms. We begin with the latter.

Lemmas D.4 and D.5 (with t = 1 ) and (A.7) show that

with probability at least 1 -δ if N ≥ log(2 /δ ) ≥ log(1 /δ ) . We used the inequalities 4 ≤ 16 e 3 / 2 , √ a + √ b ≤ √ 2( a + b ) , and 1 ≤ 2 log(2 /δ ) to get to the last line.

Continuing, we bound A λ N,M . It has several error contributions originating from two terms. The first term in (D.2) is R λ N ( α ; G ) , where α ∈ R M is arbitrary. By Assumption 3.2, G = ρ + G H so that

By Lem. 4.5, it holds with probability at least 1 -δ that

Next, we bound the first term on the right hand side in (D.8). Since G H ∈ H , there exists α H ∈ L 2 µ (Θ; R ) such that

With α H as in the above display, choose once and for all α ≡ α ⋆ ∈ R M as in (4.7). By Lem. 4.3,

with probability at least 1 -δ if M ≥ λ -1 log(4 /δ ) . On the same event,

by Lem. D.3. This fact and Lem. 4.6 (with α ≡ α ⋆ as in (4.7)) shows that the second and final term in A λ N,M (D.2) satisfies with probability at least 1 -δ the upper bound

Combining the estimates (D.7), (D.8), (D.9), (D.10), and (D.11), recalling (D.6), and invoking the union bound, we deduce that if N ≥ λ -2 log(2 /δ ) , then

with probability at least 1 -4 δ , where

In the last line, we used Young's inequality with ε = 1 / 8 , that is, ab ≤ εa 2 / 2 + b 2 / (2 ε ) with a = 80 e 3 / 2 ∥ η ∥ ψ 1 ( Y ) ∥ φ ∥ L ∞ and b = ∥G H ∥ H . This proves the asserted upper bound.

D.1.1 Proofs for subsection 4.1.1: Bounding the approximation error

Given a function α ∈ L 2 µ (Θ; R ) , we denote its cut-off at level T > 0 by

This subsection is devoted to the proof of Lem. 4.3, which is based on the following three lemmas. Lemma D.1. Suppose G ∈ H belongs to the RKHS H . Let α ∈ L 2 µ (Θ; R ) be such that G = E θ [ α ( θ ) φ ( · ; θ )] . Let u 1 , . . . , u N ∼ ν be iid samples and ν N = 1 N ∑ N n =1 δ u n be the corresponding empirical measure. Then almost surely,

Proof. Fix u ∈ X and define α >T := α -α ≤ T . The claim follows from

and the observation that E θ | α >T ( θ ) | ≤ E θ | α ( θ ) | 2 /T .

The previous lemma controls the error incurred by truncating the coefficient function of elements in the RKHS. The next lemma provides a bound on sample average approximations of these truncations.

Lemma D.2. Let u 1 , . . . , u N ∼ ν be iid samples and let ν N = 1 N ∑ N n =1 δ u n denote the corresponding empirical measure. For α ∈ L 2 µ (Θ; R ) , let Z = Z ( θ ) be the L 2 ν N ( X ; Y ) -valued random variable defined for θ ∼ µ by

If Z 1 , . . . , Z M are M iid copies of Z , then it holds with probability at least 1 -δ that

Proof. By boundedness of | α ≤ T | ≤ T , we have the trivial uniform upper bound ∥ Z m ∥ L 2 ν N ≤ T ∥ φ ∥ L ∞ for each m . The variance is bounded above as

By Lem. A.2 and Thm. A.1, it holds with probability at least 1 -δ that

Squaring both sides and substitution of the above bound on σ 2 yields the claimed estimate.

The third lemma below develops a high probability bound on the empirical approximation of the RKHS norm of truncated elements in the RKHS.

Lemma D.3. Let α ∈ L 2 µ (Θ; R ) and { θ m } ∼ µ ⊗ M . With probability at least 1 -δ , it holds that

Proof. We apply Bernstein's inequality (A.2) to the random variable Z ( θ ) := | α ≤ T ( θ ) | 2 with θ ∼ µ and M ∈ N iid copies Z 1 , . . . , Z M of Z defined by Z m = Z ( θ m ) for each m . We note that | Z | ≤ T 2 by definition. The variance of Z satisfies the upper bound

It follows from Lem. A.2 and Thm. A.1 that

with probability at least 1 -δ . This in turn implies that

with at least the same probability. This is the claim.

We are now in a position to prove Lem. 4.3.

Proof of Lemma 4.3. Write T := T ( λ ) and α := α H . Next, define α ≤ T ∈ L 2 µ (Θ; R ) by

We define α >T := α 1 {| α | >T } similarly, so that α ≡ α ≤ T + α >T holds true. Then for θ 1 , . . . , θ M , we have α ⋆ ∈ R M given by α ⋆ m = α ≤ T ( θ m ) for each m ∈ { 1 , . . . , M } . We claim that R λ N ( α ⋆ ; G ) ≤ (74 ∥ φ ∥ 2 L ∞ +7) λ E θ | α H ( θ ) | 2 with high probability, which implies the asserted bound (4.8). To see this, we make the error decomposition

Each of the three terms (I)-(III) is estimated as follows.

By Lem. D.1, we can bound

Lem. D.2 delivers the bound

with probability at least 1 -δ . Last, Lem. D.3 yields

with probability at least 1 -δ .

Combining the three estimates, it follows that if

then

with probability at least 1 -2 δ . We used the fact that √ 2 ≤ 2 . This is the claimed upper bound.

D.1.2 Proof for subsection 4.1.2: Bounding the misspecification error

Recall that ρ ∈ L ∞ ν under Assumption 3.2. We now prove Lem. 4.5.

Proof of Lemma 4.5. Let Z 1 = ∥ ρ ( u 1 ) ∥ 2 Y , which is uncentered. Almost surely, Z 1 ≤ ∥ ρ ∥ 2 L ∞ ν and

Thus with probability at least 1 -δ , Cor. A.2 and Thm. A.1 provide the bound

To get the last inequality, we used the arithmetic-mean-geometric-mean inequality √ ab ≤ ( a + b ) / 2 to obtain

Multiplying the penultimate chain of inequalities through by two completes the proof.

D.1.3 Proofs for subsection 4.1.3: Bounding the noise error

This subsection provides proofs for the lemmas used to control the error stemming from iid noise corrupting the output data as in Assumption 3.3. The estimates themselves could be improved by using (A.10) instead of (A.11) and by tracking the noise variance E ∥ η ∥ 2 Y instead of bounding it above by 4 ∥ η ∥ 2 ψ 1 ( Y ) . This would be relevant in settings where the noise is small or tends to zero with the sample size. We defer such considerations to future work.

Proof of Lemma 4.6. Define Z n ( α ) := ⟨-η n , Φ( u n ; α ) ⟩ Y for each n . Conditioned on { θ m } , it holds that Z n is an iid copy of Z 1 . By the assumption E [ η 1 | u 1 ] = 0 , we have

Next,

by two applications of the Cauchy-Schwarz inequality, one in Y and the other in R M . We deduce that ∥ Z 1 ( α ) ∥ ψ 1 ≤ ∥ η 1 ∥ ψ 1 ( Y ) ∥ α ∥ M ∥ φ ∥ L ∞ , conditioned on { θ m } . Prop. A.4, Thm. A.1 (Bernstein's inequality), and a similar argument to that in the proof of Cor. A.5 deliver the asserted bound.

The next two lemmas are used in the proof of Prop. 4.1 to control the third and final term in (4.5).

Lemma D.4 (Linear empirical process: Concentration) . Fix t > 0 and δ ∈ (0 , 1) . Define

If N ≥ log(1 /δ ) , then conditioned on the realizations { θ m } in the family Φ it holds that

with probability at least 1 -δ .

Proof. For any α ∈ A t , we compute

We used the Cauchy-Schwarz inequality twice. By the boundedness of φ , the above display gives

The iid random variables ⟨ η n , Φ( u n ; · ) ⟩ Y : α ↦→⟨ η n , Φ( u n ; α ) ⟩ Y (conditional on { θ m } ) are linear and hence continuous. Application of (A.11) in Cor. A.5 to ⟨ η n , Φ( u n ; · ) ⟩ Y taking value in the separable Banach space C ( A t ; R ) of continuous functions from compact set A t ⊂ R M into R , equipped with the supremum norm, completes the proof.

The previous lemma gives a concentration bound for the linear empirical process and the next lemma estimates its expectation.

Lemma D.5 (Linear empirical process: Expectation) . Fix t > 0 . Define A t as in Lem. D.4. Then

Proof. For any α ∈ A t , the Cauchy-Schwarz inequality in R M delivers the bound

Let the left hand side of (D.21) be denoted by Ξ t . We next note that

̸

Using the independence of ( u n , η n ) and ( u n ′ , η n ′ ) for any two indices n = n ′ , together with the above observation, we thus obtain

This implies that

We used Jensen's inequality in the second line, independence and the zero-mean property of the summands in the third line, and the Cauchy-Schwarz inequality in Y in the final line.

D.2 Proofs for subsection 4.2: Bounding the generalization gap

This subsection upper bounds the generalization gap with suprema techniques. We begin with the following empirical process concentration inequality. It gives uniform control on the difference between the empirical and population risk functionals. The process, as a function of its index α , is quadratic because the RFM Φ( · ; α ) is linear in α .

Lemma D.6 (Quadratic empirical process: Concentration) . Fix t > 0 and δ ∈ (0 , 1) . Define

where

If N ≥ log(1 /δ ) , then conditioned on the realizations { θ m } in the family Φ , it holds that

with probability at least 1 -δ .

Proof. For any α ∈ A t and n ∈ { 1 , . . . , N } , let

We compute

We used the fact that for any u ∈ X ν -almost surely, ∥ Φ( u ; α ) ∥ 2 Y ≤ t ∥ φ ∥ 2 L ∞ on the set A t (by the Cauchy-Schwarz inequality). This implies that

The X n ( t, · ) do indeed belong to C ( A t ; R ) almost surely, as they can be written as a sum of affine and quadratic forms on R M in the α variable. Application of (A.11) in Cor. A.5 (taking the separable Banach space to be C ( A t ; R ) equipped with the supremum norm) completes the proof.

Since the supremum concentrates around its mean, it remains to show that its mean is small as a function of the sample size. The next lemma does this with Rademacher symmetrization.

Lemma D.7 (Quadratic empirical process: Expectation) . Fix t > 0 . Define Z t as in Lem. D.6. Conditioned on the realizations { θ m } in the family Φ , it holds that

Proof. By Giné-Zinn symmetrization [see, e.g., 46, Sect. 4.2, Prop. 4.11, pp. 107-108],

because the original summands (conditioned on { θ m } ) are independent. The expectation on the right is interpreted as the conditional expectation given { θ m } (i.e., E { u n } , { ε n } over the data and Rademacher variables only). The right hand side is the Rademacher complexity of the RF model class composed with the square loss. Expanding the square, it is bounded above by

We now estimate each term. The first term (I) satisfies the standard Monte Carlo bound

For the second term (II), we begin by estimating the empirical average on the set A t as

by the Cauchy-Schwarz inequality in R M . We deduce by Jensen's inequality and independence that

A final application of the Cauchy-Schwarz inequality in Y in the last line shows that the second term (II) is bounded above by 4 √ t ∥G∥ L ∞ ν ∥ φ ∥ L ∞ / √ N . By Young's inequality ab ≤ a 2 / 2 + b 2 / 2 , we further bound

The third term (III) is estimated in a similar manner. Expanding the empirical average on A t yields

The first equality in the above display satisfies the upper bound

by two applications of the Cauchy-Schwarz inequality in R M . Finally, we deduce that

by Jensen's inequality, the fact that E [ ε n ε n ′ ] = δ n,n ′ , and the Cauchy-Schwarz inequality in Y . Combining the three estimates completes the proof.

The proof of the main generalization gap bound (4.12) is now immediate.

Proof of Proposition 4.7. Lem. D.6 and D.7 (applied with t = β ) show that

with conditional probability (over { θ m } ) at least 1 -δ if N ≥ log(1 /δ ) . Since δ does not depend on { θ m } , we deduce by the tower rule of conditional expectation that the event implied by (D.27) has P -probability at least 1 -δ as well. Using the inequalities 4 ≤ 32 e 3 / 2 and √ a + √ b ≤ √ 2( a + b ) shows that the expression in (D.27) is bounded above by

Application of the inequality 1 ≤ 2 log(2 /δ ) , valid for δ ∈ (0 , 1) , implies (4.12) as asserted.

E Numerical experiment details

In this appendix, we detail the setup of the numerical experiment from Sect. 5 and provide additional visualization of the function-valued RFM's discretization-independence in Figure 3. All code used to produce the numerical results and figures in this paper are available at https://github.com/nickhnelsen/error-bounds-for-vvRF .

A RFM with M features is trained on N input-output pairs { ( u n , G ( u n )) } N n =1 according to the vector-valued RF-RR algorithm. The ground truth map G : L 2 ( T ; R ) → L 2 ( T ; R ) is a nonlinear operator defined by u (0) ( · ) ↦→ u ( · , 1) , where u = { u ( x, t ) } x,t solves the partial differential equation

with initial condition u ( · , 0) = u (0) ∈ L 2 ( T ; R ) . Here, T ≃ (0 , 2 π ) per is the 1D torus which comes with periodic boundary conditions. The initial conditions u n ∼ ν are sampled iid from a centered Matérn-like Gaussian process according to [23, Sect. 6.3, p. 32].

The random features are defined in a similar way to the Fourier Space RFs in [32, Sect. 3.1, p. 15]:

where µ is also a centered Matérn Gaussian measure with covariance operator 1 . 8 2 ( -d 2 dx 2 +15 2 Id) -3 . In the above display, F maps a function to its Fourier series coefficients, and F -1 expresses a Fourier coefficient sequence as a function expanded in the Fourier basis. The filter χ is given by [32, Eqn. 3.6, p. 15] with δ = 0 . 32 and β = 0 . 1 . We take k max = 64 . The feature map φ lifts the notion of hidden neuron in neural network architectures to function space.

In Figures 2 and 3, the quantity represented on the vertical axis is an empirical approximation to the relative Bochner squared error:

In (E.1), N ′ = 500 is the size of the test set { ( u ′ n , G ( u ′ n )) } N ′ n =1 , where { u ′ n } N ′ n =1 ∼ ν ⊗ N ′ is disjoint from the training input set { u n } N n =1 . The input and output spaces are discretized on the same p -point equally spaced grid in (0 , 2 π ) . Thus, the discretized version of any input or output function belonging to X = Y = L 2 ( T ; R ) may be identified with an element of R p . In Figures 2a and 3a, the regularization factor is chosen as λ = 7 · 10 -4 /M and as λ = 3 · 10 -6 / √ N in Figures 2b and 3b.

A priori, it is not clear whether this operator learning benchmark satisfies our theoretical assumptions because we cannot verify that the Burgers' solution operator belongs to the RKHS of ( φ, µ ) (or the range of some power of the RKHS kernel integral operator). At a more technical level, the feature map φ uses an unbounded activation function (ELU, the exponential linear unit), while our theory is only developed for bounded RFs (Assumption 3.1). Nevertheless, the empirically obtained parameter and sample complexity in Figure 2 reasonably fit the main result of our well-specified theory (Theorem 3.7).

10

3

p

10

-

2

10

-

1

Relative Bochner squared error

M

= 10

M

= 32

M

= 100

M

= 316

M

= 1000

(a) Varying

M,λ,p

for fixed

N

= 1548

.

10

3

p

10

-

3

10

-

2

N

= 10

N

= 23

N

= 54

N

= 124

N

= 288

N

= 668

N

= 1548

(b) Varying

N,λ,p

for fixed

M

= 10

4

.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents error analysis for learning with vector-valued random features, where the output takes vector values. Specifically, the paper considers random feature ridge regression and derive a general bound for the population risk functional, based on which the paper gives several results such as convergence rates for well-specified and mis-specified problems, statistical consistency as well as almost sure bounds. The main results show that $O(\\sqrt{N})$ random features are sufficient to derive error bounds of the order $O(1/\\sqrt{N})$, where $N$ is the number of training examples. This extends the existing results from real-valued output to vector-valued outputs.\n\nSTRENGTHS:\nThe paper provides comprehensive analysis and consider approximation, generalization, misspecification and noisy observations. Furthermore, the paper implies minimax optimal convergence rates in the well-specified setting.\n\nThe paper is very well written, and the analysis seems to be rigorous.\n\nThe technique does not rely on the explicit RF-RR solution formula, and avoids the use of matrix concentration inequalities.\n\nWEAKNESSES:\nWhile the technique does not use matrix concentration inequalities, the paper only considers random feature ridge regression with square loss. Therefore, the problem considered in the paper is a bit limited. It is not quite clear to me whether the technique developed here can be extended to learning with other loss functions, e.g., logistic loss. Can we get similar error bounds for vector-valued learning with random features and logistic loss?\n\nIn Theorem 3.12, the error bounds are of the order $\\lambda^{r\\land 1}$. Therefore, the error bounds would improve if we have very nice regularity with $r>1$. In other words, the results suffer from a saturation phenomenon. It is not clear to me whether the technique developed here can overcome this saturation phenomenon.\n\nThe obtained bounds do not show an explicit dependency on the output dimension $p$. It is not clear to me how the output dimension would affect the performance of learning with vector-valued outputs.\n\nTypo:\n\nAbove Eq (3.5): \"optimal the sense\" should be \"optimal in the sense\"\n\nAbove Eq (3.5): \"a short calculation deliver\" should be \"a short calculation delivers\"\n\nQUESTIONS:\nCan the analysis here be extended to learning with logistic loss functions. In this case, we also do not have explicit formula for the solution?\n\nCan we derive improved results if $r>1$ to overcome the saturation phenomenon?\n\nHow would the output dimension affect the convergence rates of the learning with vector-valued outputs.\n\nLIMITATIONS:\nI do not see concerns on negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical error analysis of infinite-dimensional input-output ridge regression with vector-valued random features. This one is the first analysis adapted to infinite-dimensional outputs and even improves existing results for finite-dimensional outputs. Several by-products come with the main error bound, such as strong consistency in the misspecified setting and minimax optimal convergence rate in the well-specified setting. In addition, in the well-specified setting, the proposed analysis provides the sharpest parameter complexity, that is square root of the sample size. The analysis is based on a new proof technique, which is sketched in the last section.\n\nSTRENGTHS:\nThe subject addressed by the authors is topical since there is a lack of theoretical analysis of random Fourier methods with infinite-dimensional output in the literature, and that the random Fourier technique is definitely the most practical way to speed up kernel methods.\nThe contribution of this work is substantial since, besides providing the first result for infinite-dimensional outputs, it also presents an improved parameter complexity ($\\sqrt N$, where $N$ is the sample size) with respect to the literature ($\\sqrt N \\log N$).\nMoreover, this paper is very well written: there is no typo and a clear effort has been made to present clearly the theoretical analysis. Remarks and examples help understanding the results and generally anticipate the questions that may arise while reading the manuscript.\n\nWEAKNESSES:\nExcept for a minor remark (RF-RR Lines 48, 74, 81 is used before being defined Line 112), I think that the paper has no flaw. I must admit that I am used to seeing a numerical experiment confirming the convergence rate obtained theoretically, but I understand that choices have to be made to fit the page limit and I am quite grateful to the authors to try to explain the successive steps of their proof instead. In a way, the paper is more consistent this way.\n\nQUESTIONS:\n1) Could the authors give some details regarding the relaxation of the independence assumption in light of the proof outline (Section 4)?\n2) Why the notation $\\Phi(u; \\hat \\alpha, \\{\\theta_m\\})$ is preferred to $\\Phi(u; \\hat \\alpha)$ before Example 3.9, and the converse after? Besides, why Theorem 3.7 is named as such rather than as a corollary?\n3) Could the authors explain the assumption $\\mathcal G \\in \\operatorname{Im}(\\mathcal K^{r/2})$ and its implications?\n\nLIMITATIONS:\nOverall, technical assumptions are discussed. Nevertheless, Societal impact is not addressed.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the theoretical aspects of learning vector-valued operators with random features. Specifically, it studies the convergence of the random feature estimate $\\Phi(u;\\hat{\\alpha})$ to the true underlying function $\\mathcal{G}$. Its main results are:\n\nTheorem 3.4: Error bound for fixed $\\lambda$ and fixed $N$. \n\nTheorem 3.7: The same error bound in the well-specified case $\\mathcal{G}\\in\\mathcal{H}$. \n\nTheorem 3.10: Almost sure consistency of the random feature estimate to the true function $\\mathcal{G}$ as $N\\rightarrow\\infty$ and $\\lambda\\rightarrow0$. \n\nTheorem 3.12: The convergence rate.\n\nSTRENGTHS:\nThe paper is entirely a theoretical paper, so the value of the paper should lie in the significance of the results and the correctness of its proofs. I believe that the stated results would be good contributions (although see Questions for Theorem 3.10), but perhaps quite limited in scope. Unfortunately, I have not had time to go through all of the proofs in detail (perhaps I will in the subsequent reviewing process).\n\nWEAKNESSES:\nThe paper is in general well-written, and the mathematics is presented quite clearly. However, its structure makes it rather difficult to read. For example, when reading the main body of the paper, we are led to the appendix several times for definitions that are required to read on in the main body. This happens on L151, L177, L179, L222 and L240. Having proofs in the appendix is fine, and is often done, but the way we are led to the appendix in this paper disrupts the flow a bit in my opinion. Also, the proofs are not in one place - we have to jump back and forth between Section 4 and the Appendix to read the proof of the results. \n\nAlso, the results are very abstract, and I think it would have been much better if some concrete examples of $\\phi$, $\\Phi$, $\\theta$, $\\alpha$ and $\\mathcal{H}$ were given. An example of a learning algorithm to which the results could have been applied would have been even better. \n\nMinor comments:\n\nL228: I think the superscript $\\lambda$ should be $\\lambda_k$. \n\nL614, displayed equation: The squared brackets after $\\mathbb{E}$ are not closed.\n\nQUESTIONS:\nTheorem 3.10 and Corollary 3.13: It is, at least in my limited experience, rare to see almost sure statements in statistical learning theory. Could the authors please comment on what made it possible in this case, or whether they believe a simple Borel-Cantelli argument should make all of the results in statistical learning theory almost sure statements?\n\nAlso, none of the results seem to depend on the distribution $\\mu$ of $\\theta$, and always just the number $M$ of them. I could be mistaken, but this is hard to believe - what if $\\mu$ put all mass on one point? I didn't seem to be able to find in Assumptions 3.1, 3.2 and 3.3 that prevented something like this.\n\nLIMITATIONS:\nThe authors barely discuss the limitations of their work, except as \"future work\" in the Conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper proposes a statistical analysis of the risk associated to learning with vector-valued random features (vv-RF) in the context of ridge regression.

The analysis shows that $\sqrt{N}$ random features are enough to attain a $\mathcal{O}(\frac{1}{\sqrt{N}})$ squared error (matching minimax rates) and strong consistency in the well-specified setting, as well as slower convergence rates related to how well the target function can be approximated in the RKHS in case of misspecified setting.

STRENGTHS:
- Topic is highly relevant to the machine learning community
- Paper is remarkably well written despite the technical complexity
- The paper is mathematically sound
- Assumptions are mild and very standard

WEAKNESSES:
Contribution (C4) is a bit exaggerated in my opinion, see questions.

QUESTIONS:
In the contributions listing, you mention in (C4) that the proof approach is not specific to RR. While I agree that you make no use of the closed form solution, you still make use of the fact that the loss is the squared loss (e.g. in 4.4), and the analysis would not hold if another loss was chosen. Same remark applies in the conclusion when you mention that future works could focus on general $L^p$ losses. If the analysis is not specific to the RR setting, for which class of losses does it hold ?

And more of a remark: it would be good to include a non trivial example of a feature map $\phi(x, \theta)$ used to solve a problem where $p = \infty$.

LIMITATIONS:
- Limitations are well discussed in the paper, expect perhaps the point about the specificity to the RR that is mentioned in the "Questions" section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a learning theory for ridge regression in random feature models in the setting when the input-output map $\\mathcal{G}: \\mathcal{X} \\mapsto \\mathcal{Y}$ is vector-valued (potentially infinite-dimensional). The model under consideration is a random feature model $\\phi: \\mathcal{X} \\times \\Theta \\mapsto \\mathcal{Y}$, such that the feature map $\\Phi(\\cdot)$ consists of a linear combination of RFs $\\Phi(\\cdot) = \\sum_{m=1}^M \\alpha_m \\phi(\\cdot, \\theta_m)$,  with different random weights sampled from $\\theta_1, \\dots, \\theta_M \\sim \\mu$. The parameters $\\alpha_1, \\dots, \\alpha_M$ are fitted by minimizing the regularized empirical risk $\\frac{1}{N} \\sum_{n=1}^N \\lVert y_n - \\Phi(u_n) \\rVert_{\\mathcal{Y}}^2 + \\frac{\\lambda}{M} \\lVert{\\alpha}\\rVert_2^2$ given data sampled from some data generating distribution $(x_i, y_i) \\sim \\nu$. The assumed hypothesis class is an RKHS that consists of functions $\\mathcal{F} \\in \\mathcal{H}$ given by weighted averages of the RFs $\\mathbb{E}_{\\mathcal{P}(\\Theta)}\\lbrack \\alpha(\\theta) \\phi(\\cdot, \\theta)\\rbrack$ for $\\alpha(\\cdot) \\in L^2_\\mu(\\Theta, \\mathbb{R})$. However, the considered setting also allows for model misspecification $\\mathcal{G} = \\mathcal{G}_\\mathcal{H} + \\rho$, such that $\\mathcal{G}_\\mathcal{H} \\in \\mathcal{H}$ and $\\rho$ is almost surely bounded. The observation noise $y = \\mathcal{G}(u) + \\eta$ is modelled by a subexponential distribution allowing for heavier noise than the subGaussian.\n\nGiven the previous setting, the main result seems to be Theorem 3.4, which bounds the population squared error $\\mathbb{E}_\\nu\\lbrack \\mathcal{G}(u) - \\Phi(u)\\rbrack$ in terms of various quantities relating to the functions $\\mathcal{G}, \\mathcal{G}_\\mathcal{H}, \\rho$, the observation noise $\\eta$ under mild conditions which roughly state that $M$ is of order $\\sqrt{N}$. In contrast to previous work, the parameter complexity $M$ is free from logarithmic factors, which gives the lowest parameter complexity so far. As an application of the previous bound, the authors results regarding 1) strong statistical consistency of the model, that is, almost sure convergence in mean square of $\\Phi(\\cdot)$ to $\\mathcal{G}(\\cdot)$, and 2) corresponding explicit convergence rates. Sketches of the proofs are given in Section 4 to guide the reader along the main steps.\n\nSTRENGTHS:\nThe paper is well-written and coherent. The area, theoretical properties of infinite-dimensional RF models, is relevant to (contemporary) applications in scientific computing, such as learning the solution operator of PDEs using RF or neural models. The presented results are highly technical and non-trivial to derive. In addition to generalizing the theory of RF models to this setting, the RF parameter complexity is also sharper by a logarithmic factor as opposed to previous works, thatworked with stronger assumptions on the output domain. The presentation is good-ish, I think they did a good job in outlining the setting, then the main results, and presenting the key steps of the proofs in section with full proofs deferred to the appendix.\n\nWEAKNESSES:\nAs a theoretical work, there is no empirical part to speak of. This, on the one hand is common in these types of papers, but on the other, it makes the paper less accessible to a majority of the NeurIPS community. Although it seems like such RF models do have important applications, I am unsure about the size of the community that is involved in these kinds of niche results, as the main application only seems to be in operator learning.\n\nOne other point is that although there are applications listed, the mathematical objects considered in the paper still seem kind of abstract, I think giving more examples and connecting the dots more to the applications would help in this aspect (see question 1).\n\nLastly, no illustrations are provided whatsoever to support comprehension or visualize the results. Some figures would go a long way to make the paper less dense (see question 2)\n\nQUESTIONS:\n- As being familiar with random feature models, I understood the main premise of the paper, but I was still left wondering about how one could construct feature maps into such infinite-dimensional spaces. I think giving some examples about what kind of RFs were used before for operator learning, on what kind of data, etc would go a long way. \n\n- I was wondering if it would be possible to visualize the various hyperparameter complexities and empirically validate the convergence rate appearing in the theorems to have some visual intuition either using synthetic data or one of the datasets appearing in previous work (if the true operator is known)? Another idea which would also help in structuring the results and aid in understanding them is to draw a flowchart about the previous theorems and lemmas showing how they depend on each other.\n\nMinor:\n- In line 32, when the authors state the size of RF matrices is quadratic in the number of features $M$, do they actually mean the covariance matrix that has size $M \\times M$, or the $N \\times M$ feature matrix? In the latter case I am unsure where the quadratic factors comes in.\n- In line 228, there might be a $k$ missing in $\\mathfrak{R}_N^{\\lambda_k}$, otherwise $\\hat \\alpha$ does not actually depend on $k$\n- In Apprendix A, line 489, the authors define the subexponential norm in terms of the moments, which is slightly uncommon, since it's standard to define it in terms of the exponential Orlicz norm. It might be beneficial to state the original definition first, and relate that\n\nLIMITATIONS:\nAs a purely theoretical work, there is no direct societal impact commonly associated with AI models. The authors have addresed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}