Title: Optimal Rates for Vector-Valued Spectral Regularization Learning Algorithms
Venue: NeurIPS 2024
Accepted: yes
Keywords: Kernel methods

nonparametric regression

spectral algorithms
Abstract: We study theoretical properties of a broad class of regularized algorithms with vector-valued output. These spectral algorithms include kernel ridge regression, kernel principal component regression and various implementations of gradient descent.  Our contributions are twofold. First, we rigorously confirm the so-called saturation effect for ridge regression with vector-valued output by deriving a novel lower bound on learning rates; this bound is shown to be suboptimal when the smoothness of the regression function exceeds a certain level.
Second, we present an upper bound on the finite sample risk for general vector-valued spectral algorithms, applicable to both well-specified and misspecified scenarios (where the true regression function lies outside of the hypothesis space), and show that this bound is minimax optimal in various regimes. All of our results explicitly allow the case of infinite-dimensional output variables, proving consistency of recent practical applications.

Optimal Rates for Vector-Valued Spectral Regularization Learning Algorithms

Abstract

We study theoretical properties of a broad class of regularized algorithms with vector-valued output. These spectral algorithms include kernel ridge regression, kernel principal component regression and various implementations of gradient descent. Our contributions are twofold. First, we rigorously confirm the so-called saturation effect for ridge regression with vector-valued output by deriving a novel lower bound on learning rates; this bound is shown to be suboptimal when the smoothness of the regression function exceeds a certain level. Second, we present an upper bound on the finite sample risk for general vector-valued spectral algorithms, applicable to both well-specified and misspecified scenarios (where the true regression function lies outside of the hypothesis space), and show that this bound is minimax optimal in various regimes. All of our results explicitly allow the case of infinite-dimensional output variables, proving consistency of recent practical applications.

1 Introduction

We investigate a fundamental topic in modern machine learning-the behavior and efficiency of learning algorithms for regression in high-dimensional and potentially infinite-dimensional output spaces Y . Given two random variables X and Y , we seek to empirically minimize the squared expected risk

over functions F in a reproducing kernel Hilbert space consisting of vector-valued functions from a topological space X to a Hilbert space Y . The study of this setting as an ill-posed statistical inverse problem is well established: see e.g. 46, 6, 53, 3, 5, 17. In this work, we study the setting when Y is high- or infinite-dimensional, since it has been less well covered by the literature, yet has many applications in multitask regression [7, 2] and infinite-dimensional learning problems, including

Zhu Li

the conditional mean embedding [20, 21, 41], structured prediction [11, 12], causal inference [43], regression with instrumental and proximal variables [42, 35], the estimation of linear operators and dynamical systems [47, 37, 26, 38, 25], and functional regression [24]. Interestingly, the aforementioned infinite-dimensional applications typically use the classical ridge regression algorithm. Our goal is to motivate the use of alternative learning algorithms in these settings, while providing strong theoretical guarantees.

Classically, the ill-posed problem (1) is solved via regularization strategies, which are often implemented in terms of so-called spectral filter functions in the context of inverse problems in Hilbert spaces [16]. When applied to the learning problem given by (1), these filter functions correspond to learning algorithms including ridge regression, a variety of different implementations of gradient descent , principal component regression , and other related methods (we refer the reader to 19 and 2 for overviews of the real-valued and vector-valued output variable case, respectively). Algorithms based on spectral filter functions when Y = R are studied extensively, see e.g. [5, 34]. To the best of our knowledge, the detailed behavior of this general class of algorithms has remained unknown when Y is a general Hilbert space, with the exception of a few results for special cases in the setting of ridge regression [6, 31].

Overview of our contributions. In this manuscript, we aim to theoretically understand vector-valued spectral learning algorithms. The contribution of our work is twofold: (i) we rigorously confirm the saturation effect of ridge regression for general Hilbert spaces Y (see paragraph below) in the context of lower bounds on rates for the learning problem (1) and (ii) we cover a gap in the existing literature by providing upper rates for general spectral algorithms in high- and infinite-dimensional spaces. Our results explicitly allow the misspecified learning case in which the true regression function is not contained in the hypothesis space. We base our analysis on the concept of vector-valued interpolation spaces introduced by [30, 31]. The interpolation space norms measure the smoothness of the true regression function, replacing typical source conditions found in the literature which only cover the well-specified case. To the best of our knowledge, these are the first bounds covering this general setting for vector-valued spectral algorithms.

Saturation effect of ridge regression. The widely-used ridge regression algorithm is known to exhibit the so-called saturation effect: it fails to exploit additional smoothness in the target function beyond a certain threshold. This effect has been thoroughly investigated in the context of Tikhonov regularization in inverse problems [16, Chapter 5], but is generally reflected only in upper rates in the learning literature, see e.g. [34, 5]. Interestingly, existing lower bounds [6, 5, 31] are usually formulated in a more general setting and do not reflect this saturation effect, leaving a gap between upper and lower rates. We leverage the bias-variance decomposition paradigm to lower bound the learning risk of kernel ridge regression with vector-valued output, in order to close this gap.

Learning rates of vector-valued spectral algorithms. Motivated by the fact that the saturation effect is technically unavoidable with vector-valued ridge regression, we proceed to study the generalization error of popular alternative learning algorithms. In particular, we provide upper rates in the vector-valued setting consistent with the known behavior of spectral algorithms in the real-valued learning setting, based on their so-called qualification property [5, 34]. In particular, we confirm that a saturation effect can be bypassed in high and infinite dimensions by algorithms such as principal component regression and gradient descent, allowing for a better sample complexity for high-smoothness problems. Furthermore, we study the misspecified setting and show that upper rates for spectral algorithms match the state-of-the-art upper rates for misspecified vector-valued ridge regression recently obtained by [31]. Those rates are optimal for a wide variety of settings. Moreover, we argue that applications of vector-valued spectral algorithms are easy to implement by making use of an extended representer theorem based on [2], allowing for the numerical evaluation based on empirical data-even in the infinite-dimensional case.

Related Work. The saturation effect of regularization techniques in deterministic inverse problems is well-known. For example, [40, 36, 22] study the saturation effect for Tikhonov regularization and general spectral algorithms. In the kernel statistical learning framework, the general phenomenon of saturation is discussed by e.g. [3, 19]. Recent work by [29] investigates saturation effect in the learning context by providing a lower bound on the learning rate. To the best of our knowledge, however, all studies in the statistical learning context focus on the case when Y is real-valued. General upper bounds of kernel ridge regression with real-valued or finite-dimensional Y have been extensively studied in the literature (see e.g., [6, 50, 8, 17]), where minimax optimal learning

rates are derived. Recent work [30, 31] studies the infinite-dimensional output space setting with Tikhonov regularization and obtains analogous minimax optimal learning rates. [23] later study a setting where both the input and output space is the infinite dimensional Sobolev RKHS and establish the minimax optimal rate. For kernel learning with spectral algorithms, existing work (see e.g., [3, 5, 32, 34, 54, 28]) focuses on real-valued output space setting and obtains optimal upper learning rates depending on the qualification number of the spectral algorithms, where only [54, 28] consider the misspecified learning scenario where the target regression function does not lie in the hypothesis space. For vector-valued output spaces, [27] considers learning with vector-valued random features. However, general investigations of spectral algorithms for vector-valued output spaces are absent in the literature.

Structure of this paper. This paper is structured as follows. In Section 2, we introduce mathematical preliminaries related to reproducing kernel Hilbert spaces, vector-valued regression as well as the concept of vector-valued interpolation spaces. Section 3 contains a brief review the so-called saturation effect and a corresponding novel lower bound for vector-valued kernel ridge regression. In Section 4, we investigate general spectral learning algorithms in the context of vector-valued interpolations spaces and provide our main result: upper learning rates for this setting.

2 Background and Preliminaries

Throughout the paper, we consider a random variable X (the covariate) defined on a second countable locally compact Hausdorff space 2 X endowed with its Borel σ -field F X , and the random variable Y (the output) defined on a potentially infinite dimensional separable real Hilbert space (Y , ⟨⋅ , ⋅⟩ Y ) endowed with its Borel σ -field F Y . We let ( Ω , F , P ) be the underlying probability space with expectation operator E . Let P be the push-forward of P under ( X,Y ) and π and ν be the marginal distributions on X and Y , respectively; i.e., X ∼ π and Y ∼ ν . We use the Markov kernel p ∶ X × F Y → R + to express the distribution of Y conditioned on X as

for all x ∈ X and events A ∈ F Y , see e.g. [15]. We introduce some notation related to linear operators on Hilbert spaces and vector-valued integration; formal definitions can be found in Appendix A for completeness, or we refer the reader to [52, 14]. The spaces of Bochner square-integrable functions with respect to π and taking values in Y are written as L 2 (X , F X , π ; Y) , abbreviated as L 2 ( π ; Y) . We obtain the classical Lebesgue spaces as L 2 ( π ) ∶= L 2 ( π ; R ) . Throughout the paper, we write [ F ] or more explicitly [ F ] π for the π -equivalence class of (potentially pointwise defined) measurable functions from X to Y , which we naturally interpret as elements in L 2 ( π ; Y) whenever they are square-integrable. Let H be a separable real Hilbert space with inner product ⟨⋅ , ⋅⟩ H . We write L( H,H ′ ) as the Banach space of bounded linear operators from H to another Hilbert space H ′ , equipped with the operator norm ∥ ⋅ ∥ H → H ′ . When H = H ′ , we simply write L( H ) instead. We write S 2 ( H,H ′ ) as the Hilbert space of Hilbert-Schmidt operators from H to H ′ and S 1 ( H,H ′ ) as the Banach space of trace class operators (see Appendix A for a complete definition). For two Hilbert spaces H,H ′ , we say that H is (continuously) embedded in H ′ and denote it as H ↪ H ′ if H can be interpreted as a vector subspace of H ′ and the inclusion operator i ∶ H → H ′ performing the change of norms with ix = x for x ∈ H is continuous; and we say that H is isometrically isomorphic to H ′ and denote it as H ≃ H ′ if there is a linear isomorphism between H and H ′ which is an isometry.

Tensor Product of Hilbert Spaces: Denote H ⊗ H ′ the tensor product of Hilbert spaces H , H ′ . The element x ⊗ x ′ ∈ H ⊗ H ′ is treated as the linear rank-one operator x ⊗ x ′ ∶ H ′ → H defined by y ′ → ⟨ y ′ , x ′ ⟩ H ′ x for y ′ ∈ H ′ . Based on this identification, the tensor product space H ⊗ H ′ is isometrically isomorphic to the space of Hilbert-Schmidt operators from H ′ to H , i.e., H ⊗ H ′ ≃ S 2 ( H ′ , H ) . We will hereafter not make the distinction between these two spaces, and treat them as being identical.

Remark 1 (1, Theorem 12.6.1) . Consider the Bochner space L 2 ( π ; H ) where H is a separable Hilbert space. One can show that L 2 ( π ; H ) is isometrically identified with the tensor product space H ⊗ L 2 ( π ) , and we denote as Ψ the isometric isomorphism between the two spaces. See Appendix A for more details on tensor product spaces and the explicit definition of Ψ .

Scalar-valued Reproducing Kernel Hilbert Space (RKHS). We let k ∶ X × X → R be a symmetric and positive definite kernel function and H be a vector space of functions from X to R , endowed with a Hilbert space structure via an inner product ⟨⋅ , ⋅⟩ H . We say that k is a reproducing kernel of H if and only if for all x ∈ X we have k (⋅ , x ) ∈ H and for all x ∈ X and f ∈ H , we have f ( x ) = ⟨ f, k ( x, ⋅)⟩ H . A space H which possesses a reproducing kernel is called a reproducing kernel Hilbert space (RKHS; see e.g. 4). We denote the canonical feature map of H as ϕ ( x ) = k (⋅ , x ) .

We require some technical assumptions on the previously defined RKHS and kernel, which we assume to be satisfied throughout the text:

H is separable: this is satisfied if k is continuous, given that X is separable 3 ;

k (⋅ , x ) is measurable for π -almost all x ∈ X ;

k ( x, x ) ≤ κ 2 for π -almost all x ∈ X .

The above assumptions are not restrictive in practice, as well-known kernels such as the Gaussian, Laplace, and Matérn kernels satisfy them on X ⊆ R d [48]. We now introduce some facts about the interplay between H and L 2 ( π ) , which has been extensively studied by [44, 45], [13] and [51]. We first define the (not necessarily injective) embedding I π ∶ H → L 2 ( π ) , mapping a function f ∈ H to its π -equivalence class [ f ] . The embedding is a well-defined compact operator as long as its Hilbert-Schmidt norm is finite. In fact, this requirement is satisfied since its Hilbert-Schmidt norm can be computed as [51, Lemma 2.2 & 2.3] ∥ I π ∥ S 2 (H ,L 2 ( π )) = ∥ k ∥ L 2 ( π ) ≤ κ. The adjoint operator S π ∶= I ∗ π ∶ L 2 ( π ) → H is an integral operator with respect to the kernel k , i.e. for f ∈ L 2 ( π ) and x ∈ X we have [49, Theorem 4.27]

Next, we define the self-adjoint, positive semi-definite and trace class integral operators

Vector-valued Reproducing Kernel Hilbert Space (vRKHS). Let K ∶ X×X → L(Y) be an operator valued positive-semidefinite (psd) kernel. Fix K , x ∈ X , and h ∈ Y , then ( K x h ) (⋅) ∶= K (⋅ , x ) h defines a function from X to Y . The completion of

with inner product on G pre defined on the elementary elements as ⟨ K x h, K x ′ h ′ ⟩ G ∶= ⟨ h, K ( x, x ′ ) h ′ ⟩ Y , defines a vRKHS denoted as G . For a more complete overview of the vectorvalued reproducing kernel Hilbert space, we refer the reader to [9], [10] and [31, Section 2]. In the following, we will denote G as the vRKHS induced by the kernel K ∶ X × X → L(Y) with

We emphasize that this family of kernels is the de-facto standard for high- and infinite-dimensional applications [20, 21, 41, 11, 12, 42, 35, 43, 37, 26, 38, 25, 24] due to the crucial representer theorem which gives a closed form solution for the ridge regression problem based on the data. We generalize this representer theorem to cover the general spectral algorithm case in Proposition 1.

Remark 2 (General multiplicative kernel) . Without loss of generality, we provide our results for the vRKHS G induced by the operator-valued kernel given by K ( x, x ′ ) = k ( x, x ′ ) Id Y . However, with suitably adjusted constants in the assumptions, our results transfer directly to the more general vRKHS G ̃ induced by the more general operator-valued kernel

where T ∶ Y → Y is any positive-semidefinite self-adjoint operator. The precise characterization of the adjusted constants is given by [31, Section 4.1].

An important property of G is that it is isometrically isomorphic to the space of Hilbert-Schmidt operators between H and Y [31, Corollary 1]. Similarly to the scalar case we can map every element in G into its π -equivalence class in L 2 ( π ; Y) and we use the shorthand notation [ F ] = [ F ] π (see Definition 6 in Appendix A for more details).

Theorem 1 (vRKHS isomorphism) . For every function F ∈ G there exists a unique operator C ∈ S 2 (H , Y) such that F (⋅) = Cϕ (⋅) ∈ Y with ∥ C ∥ S 2 (H , Y) = ∥ F ∥ G and vice versa. Hence G ≃ S 2 (H , Y) and we denote the isometric isomorphism between S 2 (H , Y) and G as ¯ Ψ . It follows that G can be written as G = { F ∶ X → Y ∣ F = Cϕ (⋅) , C ∈ S 2 (H , Y)} .

2.1 Vector-valued Regression

We briefly recall the basic setup of regularized least-squares regression with Hilbert space-valued random variables. The squared expected risk for vector-valued regression is

for measurable functions F ∶ X → Y . The analytical minimizer of the risk over measurable functions is the regression function or the conditional mean function F ⋆ ∈ L 2 ( π ; Y) given by

Throughout the paper, we assume that E [∥ Y ∥ 2 Y ] < +∞ , i.e., the random variable Y is squareintegrable. Note that this implies F ⋆ ∈ L 2 ( π ; Y) . Our focus in this work is to approximate F ∗ with kernel-based regularized least-squares algorithms, where we pay special attention to the case when Y is of high or infinite dimension. We pick G as a hypothesis space of functions in which to estimate F ∗ . Note that by Theorem 1, minimizing the functional E on G is equivalent to minimizing the following functional on S 2 (H , Y) ,

It is shown in [38, Proposition 3.5 and Section 3.4] that the optimality condition can be written as

where C Y X ∶= E [ Y ⊗ ϕ ( X )] is the cross-covariance operator. As discussed in full detail by [38], the problem (5) can be formulated as a potentially ill-posed inverse problem on the space of HilbertSchmidt operators. As such, a regularization is required; we introduce regularized solutions of this problem in Section 4 through the classical concept of spectral filter functions.

2.2 Vector-valued Interpolation Space and Source Condition

We now introduce the background required in order to characterize the smoothness of the target function F ∗ , both in the well-specified setting ( F ∗ ∈ G ) and in the misspecified setting ( F ∗ ∉ G ). We review the results of [51] and [17] in constructing scalar-valued interpolation spaces, and [30] in defining vector-valued interpolation spaces.

Real-valued Interpolation Space: By the spectral theorem for self-adjoint compact operators, there exists an at most countable index set I , a non-increasing sequence ( µ i ) i ∈ I > 0 , and a family ( e i ) i ∈ I ∈ H , such that ([ e i ]) i ∈ I 4 is an orthonormal basis (ONB) of ran I π ⊆ L 2 ( π ) and ( µ 1 / 2 i e i ) i ∈ I is an ONB of ( ker I π ) ⊥ ⊆ H , and we have

For α ≥ 0 , the α -interpolation space [51] is defined by

equipped with the inner product

for ( a i ) i ∈ I , ( b i ) i ∈ I ∈ ℓ 2 ( I ) . The α -interpolation space defines a Hilbert space. Moreover, ( µ α / 2 i [ e i ]) i ∈ I forms an ONB of [H] α and consequently [H] α is a separable Hilbert space. In the following, we use the abbreviation ∥ ⋅ ∥ α ∶= ∥ ⋅ ∥ [H] α .

Vector-valued Interpolation Space: Introduced in [30], vector-valued interpolation spaces generalize the notion of scalar-valued interpolation spaces to vRKHS with a kernel of the form (2).

Definition 1 (Vector-valued interpolation space) . Let k be a real-valued kernel with associated RKHS H and let [H] α be the real-valued interpolation space associated to H with some α ≥ 0 . The vector-valued interpolation space [G] α is defined as (refer to Remark 1 for the definition of Ψ )

The space [G] α is a Hilbert space equipped with the inner product

where C = Ψ -1 ( F ) , L = Ψ -1 ( G ) . For α = 0 , we retrieve ∥ F ∥ 0 = ∥ F ∥ L 2 ( π ; Y) = ∥ C ∥ S 2 ( L 2 ( π ) , Y) .

Remark 3 (Interpolation space inclusions) . Note that we have F ∗ ∈ L 2 ( π ; Y) since Y ∈ L 2 ( P ; Y) by assumption. Furthermore, for 0 < β < α , [17, Eq. (7)] imply the inclusions

Under assumptions 1 to 3 and with X being a second-countable locally compact Hausdorff space, [G] 0 = L 2 ( π ; Y) if and only if H is dense in the space of continuous functions vanishing at infinity, equipped with the uniform norm [31, Remark 4].

Remark 4 (Well-specified versus misspecified setting) . We say that we are in the well-specified setting if F ∗ ∈ [G] 1 . In this case, there exists ¯ F ∈ G such that F ∗ = ¯ F π -almost surely and ∥ F ∗ ∥ 1 = ∥ ¯ F ∥ G , i.e. F ∗ admits a representer in G (see Remark 5 in Appendix A). When F ∗ ∈ [G] β for β < 1 , F ∗ may not admit such a representation and we are in the misspecified setting, as [G] 1 ⊆ [G] β .

Definition 1 and Remarks 3 and 4 motivate the use of following assumption on the smoothness of the target function: there exists β > 0 and a constant B ≥ 0 such that F ∗ ∈ [G] β and

We let C ∗ ∶= Ψ -1 ( F ∗ ) ∈ S 2 ([H] β , Y) . (SRC) directly generalizes the notion of a so-called Höldertype source condition in the learning literature [6, 17, 32, 34] and allows to characterize the misspecified learning scenario.

2.3 Further Assumptions

In addition to (SRC), we require standard assumptions to obtain the precise learning rates for kernel learning algorithms. We list them below. For constants D 2 > 0 and p ∈ ( 0 , 1 ] and for all i ∈ I ,

For constants D 1 , D 2 > 0 and p ∈ ( 0 , 1 ) and for all i ∈ I ,

(EVD) and (EVD+) are standard assumptions on the eigenvalue decay of the integral operator: they describe the interplay between the marginal distribution π and the RKHS H (see more details in 6, 17). (EVD+) is needed in order to establish lower bounds on the excess risk. Note that we have excluded the value p = 1 from (EVD+); indeed, p = 1 is incompatible with the assumption of a bounded kernel, a fact missed by previous works and of independent interest (see Appendix, Remark 7).

For α ∈ [ p, 1 ] , the inclusion I α, ∞ π ∶ [H] α ↪ L ∞ ( π ) is continuous, and ∃ A > 0 such that

Property (EMB) is referred to as the embedding property in [17]. It can be shown that it holds if and only if there exists a constant A ≥ 0 with ∑ i ∈ I µ α i e 2 i ( x ) ≤ A 2 for π -almost all x ∈ X [17, Theorem 9]. Since we assume k to be bounded, the embedding property always hold true when α = 1 .

Furthermore, (EMB) implies a polynomial eigenvalue decay of order 1 / α , which is why we take α ≥ p . (EMB) is not needed when we deal with the well-specified setting, but is crucial to bound the excess risk in the misspecified setting.

Finally, we assume that there are constants σ, R > 0 such that

is satisfied for π -almost all x ∈ X and all q ≥ 2 . The (MOM) condition on the Markov kernel p ( x, dy ) is a Bernstein moment condition used to control the noise of the observations (see 6, 17 for more details). If Y is almost surely bounded, for example ∥ Y ∥ Y ≤ Y ∞ almost surely, then (MOM) is satisfied with σ = R = 2 Y ∞ . It is possible to prove that the Bernstein condition is equivalent to sub-exponentiality, see [38, Remark 4.9].

3 Saturation Effect of Kernel Ridge Regression

The most established way of learning F ∗ is by kernel ridge regression (KRR), which can be formulated as the following optimization problem: given a dataset D = {( x i , y i )} n i = 1 independently and identically sampled from the joint distribution of X and Y ,

where λ > 0 is the regularization parameter. The generalization error of vector-valued KRR is expressed as ˆ F λ -F ∗ , and controlled in different norms: see [31] for an extensive study. We recall here a simplified special case of the key results obtained in this work. In the next Theorem, ≲ , ≳ are inequality up to positive multiplicative constants that are independent of n .

Theorem 2 (Upper and lower bounds for KRR in the well-specified regime) . Let ˆ F λ be the KRR estimator from (7) . Furthermore, let the conditions (EVD+) , (SRC) and (MOM) be satisfied for some 0 < p ≤ 1 and β ≥ 1 . Then, with high probability we have

and furthermore for all learning methods (i.e., measurable maps) of the form D → ˆ F D ,

Theorem 2 shows the minimax optimal learning rate for vector-valued KRR for β ∈ [ 1 , 2 ] . However, when β > 2 , the obtained upper bound saturates at n -2 2 + p , creating a gap with the lower bound. This phenomenon is referred to as the saturation effect of Tikhonov regularization, and has been well investigated in deterministic inverse problems [40]. In the case where Y is real-valued, [29] prove that the saturation effect cannot be avoided with Tikhonov regularization. Below, we give a similar but generalized bound on lower rates for the case that Y is a Hilbert space. For this result only, we assume that X is a compact subset of R d . We give the proof in Appendix B.

Theorem 3 (Saturation of KRR) . Let X be a compact subset of R d . Let λ = λ ( n ) be an arbitrary choice of regularization parameter satisfying λ ( n ) → 0 as n →+∞ and let ˆ F λ be the KRR estimator from (7) . We assume that the noise is non-zero and bounded below, i.e. there exists σ > 0 , such that

is satisfied for π -almost all x ∈ X . We assume in addition and for this result only that k is Hölder continuous (see Definition 11 in the appendix), i.e., k ∈ C θ (X × X) for θ ∈ ( 0 , 1 ] . Suppose that Assumptions (EVD+) and (SRC) hold with p ∈ ( 0 , 1 ) and β ≥ 2 . For τ ≥ 0 , for sufficiently large n > 0 , where the hidden index bound depends on τ , with probability greater than 1 -e -τ , there exists some constant c τ > 0 such that

The assumption that k is Hölder continuous is crucial in lower bounding the variance with a covering number argument. Kernels satisfying this assumption include Gaussian kernels, Laplace kernels and Matérn kernels. Theorem 3 clearly demonstrates that the learning rate from vector-valued KRR cannot reach the information theoretic lower rate given in Theorem 2.

As discussed above, [29] propose a similar lower bound in the real-valued case, and we now highlight two fundamental differences with [29] in the proof. First, while both works adopt the same bias-variance decomposition, we need to lower bound the bias and the variance term with infinite-dimensional output in our setting. Second, we adopt a different and simpler approach in proving the lower bound, since there are a number of issues with the proof of [29], both in the treatment of the bias and of the variance. For a detailed comparison with the earlier work, and an explanation of the differences in our approach, please refer to Remark 6 in the Appendix.

4 Consistency and optimal rates for general spectral algorithms

Regularized population solution : Our goal is to regularize (5) in such a way that we get a unique and well-defined solution that provides a good approximation to F ∗ . We first recall the concept of a filter function (i.e., a function on an interval which is applied on self-adjoint operators to each individual eigenvalue via the spectral calculus, see 16), that will allow to define a regularization strategy. One may think of the following definition as a class of functions approximating the inversion map x ↦ 1 / x while still being defined for x = 0 in a reasonable way. We use the definition given by [34], but equivalent definitions can be found throughout the literature.

Definition 2 (Filter function) . Let Λ ⊆ R + . A family of functions g λ ∶ [ 0 , ∞) → [ 0 , ∞) indexed by λ ∈ Λ is called a filter with qualification ρ ≥ 0 if it satisfies the following two conditions:

There exists a positive constant E such that, for all λ ∈ Λ

There exists a positive constant ω ρ < ∞ such that

Below, we give some standard examples which are discussed by e.g. [19, 5] in the context of kernel regression with scalar output variables, and in [2] for the vector-valued case. A variety of additional algorithms can be expressed in terms of a filter function.

Ridge regression. From the Tikhonov filter function g λ ( x ) = ( x + λ ) -1 , we obtain the known ridge regression algorithm. In this case, we have E = ρ = ω ρ = 1 .

Gradient Descent. From the Landweber iteration filter function given by

we obtain the gradient descent scheme with constant step size τ > 0 , which corresponds to the population gradient iteration given by F k + 1 ∶= F k -τ ∇E( F k ) for k ∈ N . In this case, we have E = 1 and arbitrary qualification with ω ρ = 1 whenever 0 < ρ ≤ 1 and ω ρ = ρ ρ otherwise. Gradient schemes with more complex update rules can be expressed in terms of filter functions as well [39, 32, 34].

Kernel principal component regression. The truncation filter function g λ ( x ) = x -1 1 [ x ≥ λ ] yields kernel principal component regression, corresponding to a hard thresholding of eigenvalues at a truncation level λ . In this case we have E = ω ρ = 1 for arbitrary qualification ρ .

Population solution : Given a filter function g λ , we call g λ ( C X ) 5 the regularized inverse of C X . We may think of the regularized inverse as approximating the pseudoinverse of C X (see e.g. [16]) when λ → 0 . We define the regularized population solution to (4) as

The solution arising from standard regularization strategies leads to well-known statistical methodologies. We refer to [16] for the background on filter functions in classical regularization theory.

Empirical solution : Given the dataset D = {( x i , y i )} n i = 1 , the empirical analogue of (10) is

where ˆ C Y X , ˆ C X are empirical covariance operators define as

Note that (11) is the regularized solution of the empirical inverse problem

which arises as the optimality condition for minimizers on G of the empirical analogue of (3), given by E n ( F ) ∶= 1 n ∑ n i = 1 ∥ y i -F ( x i )∥ 2 Y ; see Proposition 2 in the Appendix for a proof. For the vector-valued kernel given in (2), it is well-known that ˆ F λ can be computed in closed-form for the ridge regression estimator-even in infinite dimensions [47]. For general filter functions, an extended representer theorem is given by [2] in the context of finite-dimensional multitask learning: this approach works in infinite dimensions as well. We give the closed form solution based on [2] below (we include the proof in Appendix D.1).

Proposition 1 (Representer theorem for general spectral filter) . Let ( K ) ij = k ( x i , x j ) , 1 ≤ i, j ≤ n denote the Gram matrix associated to the scalar-valued kernel k . We have

Example 1 (Conditional integration) . Consider now a random variable Z taking values in a topological space Z on which we define a second RKHS H ′ ⊆ R Z with kernel ℓ ∶ Z × Z → R and canonical feature map ψ ∶ Z → H ′ , z ↦ ℓ ( z, ⋅) . The conditional mean embedding [47, 20] is defined as

We immediately see the link with vector-valued regression with Y = ψ ( Z ) and Y = H ′ . The conditional mean embedding allows us to compute the conditional expectation of any element of H ′ . Indeed, using the reproducing property, for f ∈ H ′ , we have for all x ∈ X ,

Given a dataset {( x i , z i )} n i = 1 6 and an estimate of the conditional mean embedding F ∗ with a spectral algorithm ˆ F λ as in Eq. (11) , and substituting the formula in Eq. (12) , we obtain E [ f ( Z ) ∣ X = x ] ≈ ⟨ f, ˆ F λ ( x )⟩ H ′ = ∑ n i = 1 ⟨ f, ψ ( z i )⟩ H ′ α i ( x ) = f ⊺ z α ( x ) , where ( f z ) i = f ( z i ) , 1 ≤ i ≤ n.

Learning rates: We now give our main result, the learning rates for the difference between [ ˆ F λ ] and F ∗ in the interpolation norm, where F λ and ˆ F λ are given by (10) and (11) based on a general spectral filter satisfying Definition 2. The proof is deferred to Section C in the Appendix.

Theorem 4 (Upper learning rates) . Let ˆ F λ be an estimator based on a general spectral filter with qualification ρ ≥ 0 . Furthermore, let the conditions (EVD) , (EMB) , (MOM) be satisfied with 0 < p ≤ α ≤ 1 . With 0 ≤ γ ≤ 1 , if (SRC) is satisfied with γ < β ≤ 2 ρ , we have

in the case β + p ≤ α , let λ n = Θ (( n / log θ ( n )) -1 α ) for some θ > 1 , for all τ > log ( 6 ) and sufficiently large n ≥ 1 , there is a constant J > 0 independent of n and τ such that

is satisfied with P n -probability not less than 1 -6 e -τ .

in the case β + p > α , let λ n = Θ ( n -1 β + p ) , for all τ > log ( 6 ) and sufficiently large n ≥ 1 , there is a constant J > 0 independent of n and τ such that

is satisfied with P n -probability not less than 1 -6 e -τ .

Theorem 4 provides the upper rate for vector-valued spectral algorithms. In particular, in combination with the lower bound in Theorem 2, we see that vector-valued spectral algorithms with qualification ρ achieve an optimal learning rate when the smoothness β of the regression function is in the range ( α -p, 2 ρ ] . For algorithms with infinite ρ such as gradient descent and principal component regression, we confirm that they can exploit smoothness of the target function just as in the real-valued setting [3, 5, 30], while not suffering from saturation. For Tikhonov regularization, where ρ = 1 , the rates recover the state-of-the-art results from [31]. Finally, we point out that obtaining minimax optimal learning rates for β < α -p still remains challenging even in the real-valued output scenario. Note however that for a large variety of RKHS, α is arbitrarily close to p and we obtain optimal rates for the whole range ( 0 , 2 ρ ] : we refer to [31, 54] for a detailed discussion.

We provide a proof sketch for Theorem 4. The key technical challenge in extending the results of [31] to spectral filter functions lies in the analysis of the estimation error. The estimation error in

Appendix C.3). We rely on the fact that Id H = ˆ C X g λ ( ˆ C X ) + r λ ( ˆ C X ) (see Definition 2), to obtain the decomposition ˆ C λ -C λ = ( ˆ C Y X -C λ ˆ C X ) g λ ( ˆ C X ) -C λ r λ ( ˆ C X ) , which yields two terms to be controlled,

To control term (I), we use the definition of the filter function g λ (Definition 2) to obtain that ∥ ˆ C X,λ g λ ( ˆ C X )∥ H→H ≲ 1 . Thus it suffices to control the term ∥( ˆ C Y X -C λ ˆ C X ) C -1 2 X,λ ∥ S 2 (H , Y) =

∥ 1 n ∑ n i = 1 ξ ( x i , y i )∥ S 2 (H , Y) , where ξ ( x, y ) = ( y -C λ ϕ ( x )) ⊗ C -1 2 X,λ ϕ ( x ) . We proceed by bounding E [∥ ξ ( X,X )∥ m S 2 (H , Y) ] for m ≥ 1 , and then use Bernstein's inequality to derive the upper bound on ∥( ˆ C Y X -C λ ˆ C X ) C -1 2 X,λ ∥ S 2 (H , Y) . To control term (II), Lemma 9 in Appendix C.1 shows that ( II ) ≲ ∥ ˆ C 1 2 X,λ r λ ( ˆ C X ) g λ ( C X ) C β + 1 2 X ∥ H→H . The term on the right side is bounded in prior work on scalar-valued spectral method, and we refer the reader to [54, Theorem 16]. The results of Theorem 4 are then obtained by choosing regularization parameter λ = λ ( n ) to optimally trade off approximation and estimation errors.

5 Conclusion

In this work, we have rigorously explored the theoretical properties of vector-valued spectral learning algorithms, focusing on their performance in infinite-dimensional output spaces. We first proved the saturation effect observed in vector-valued kernel ridge regression, highlighting its limitations in exploiting additional smoothness in regression functions. We then presented upper bounds on the finite sample risk for a general class of spectral learning algorithms, demonstrating their minimax optimality across various scenarios, including misspecified learning settings.

Our results open avenues for further research, particularly in developing more efficient implementations for practical use in high-dimensional machine learning problems such as causal inference and functional data analysis.

Acknowledgement: Dimitri Meunier, Arthur Gretton and Zhu Li were supported by the Gatsby Charitable Foundation.

References

J.-P. Aubin. Applied Functional Analysis . John Wiley & Sons, Inc., 2nd edition, 2000.

L. Baldassarre, L. Rosasco, A. Barla, and A. Verri. Multi-output learning via spectral filtering. Machine Learning , 87(3):259-301, 2012.

F. Bauer, S. Pereverzev, and L. Rosasco. On regularization algorithms in learning theory. Journal of Complexity , 23(1):52-72, 2007.

A. Berlinet and C. Thomas-Agnan. Reproducing Kernel Hilbert Spaces in Probability and Statistics . Springer, 2011.

G. Blanchard and N. Mücke. Optimal rates for regularization of statistical inverse learning problems. Foundations of Computational Mathematics , 18(4):971-1013, 2018.

A. Caponnetto and E. De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics , 7(3):331-368, 2007.

A. Caponnetto, C. A. Micchelli, M. Pontil, and Y. Ying. Universal multi-task kernels. Journal of Machine Learning Research , 9:1615-1646, 2008.

A. Caponnetto and Y. Yao. Cross-validation based adaptation for regularization operators in learning theory. Analysis and Applications , 8(02):161-183, 2010.

C. Carmeli, E. De Vito, and A. Toigo. Vector valued reproducing kernel Hilbert spaces of integrable functions and Mercer theorem. Analysis and Applications , 4(04):377-408, 2006.

C. Carmeli, E. De Vito, A. Toigo, and V. Umanitá. Vector valued reproducing kernel Hilbert spaces and universality. Analysis and Applications , 8(01):19-61, 2010.

C. Ciliberto, L. Rosasco, and A. Rudi. A consistent regularization approach for structured prediction. Advances in Neural Information Processing Systems , 29, 2016.

C. Ciliberto, L. Rosasco, and A. Rudi. A general framework for consistent structured prediction with implicit loss embeddings. Journal of Machine Learning Research , 21(1):3852-3918, 2020.

E. De Vito, L. Rosasco, and A. Caponnetto. Discretization error analysis for tikhonov regularization. Analysis and Applications , 4(01):81-99, 2006.

J. Diestel and J. Uhl. Vector Measures . American Mathematical Society, 1977.

R. Dudley. Real Analysis and Probability . Cambridge University Press, 2nd edition edition, 2002.

H. W. Engl, M. Hanke, and A. Neubauer. Regularization of Inverse Problems . Kluwer, 2000.

S. Fischer and I. Steinwart. Sobolev norm learning rates for regularized least-squares algorithms. Journal Of Machine Learning Research , 21:205-1, 2020.

J. I. Fujii, M. Fujii, T. Furuta, and R. Nakamoto. Norm inequalities equivalent to heinz inequality. Proceedings of the American Mathematical Society , 118(3):827-830, 1993.

L. L. Gerfo, L. Rosasco, F. Odone, E. D. Vito, and A. Verri. Spectral algorithms for supervised learning. Neural Computation , 20(7):1873-1897, 2008.

S. Grünewälder, G. Lever, L. Baldassarre, S. Patterson, A. Gretton, and M. Pontil. Conditional mean embeddings as regressors. In International Conference on Machine Mearning , pages 1803--1810, 2012.

S. Grünewälder, G. Lever, L. Baldassarre, M. Pontil, and A. Gretton. Modelling transition dynamics in MDPs with RKHS embeddings. In International Conference on Machine Mearning , pages 535-542, 2012.

T. Herdman, R. D. Spies, and K. G. Temperini. Global saturation of regularization methods for inverse ill-posed problems. Journal of optimization theory and applications , 148(1):164-196, 2011.

J. Jin, Y . Lu, J. Blanchet, and L. Ying. Minimax optimal kernel operator learning via multilevel training. In The Eleventh International Conference on Learning Representations , 2023.

H. Kadri, E. Duflos, P. Preux, S. Canu, A. Rakotomamonjy, and J. Audiffren. Operator-valued kernels for learning from functional response data. Journal of Machine Learning Research , 17(20):1-54, 2016.

V. Kostic, K. Lounici, P. Novelli, and M. Pontil. Sharp spectral rates for koopman operator learning. Advances in Neural Information Processing Systems , 36, 2024.

V. Kostic, P. Novelli, A. Maurer, C. Ciliberto, L. Rosasco, and M. Pontil. Learning dynamical systems via Koopman operator regression in reproducing kernel Hilbert spaces. Advances in Neural Information Processing Systems , 35:4017-4031, 2022.

S. Lanthaler and N. H. Nelsen. Error bounds for learning with vector-valued random features. Advances in Neural Information Processing Systems , 36, 2024.

Y. Li, W. Gan, Z. Shi, and Q. Lin. Generalization error curves for analytic spectral algorithms under power-law decay. arXiv preprint arXiv:2401.01599 , 2024.

Y. Li, H. Zhang, and Q. Lin. On the saturation effect of kernel ridge regression. In The Eleventh International Conference on Learning Representations , 2023.

Z. Li, D. Meunier, M. Mollenhauer, and A. Gretton. Optimal rates for regularized conditional mean embedding learning. In Advances in Neural Information Processing Systems , volume 35, pages 4433-4445, 2022.

Z. Li, D. Meunier, M. Mollenhauer, and A. Gretton. Towards optimal sobolev norm rates for the vector-valued regularized least-squares algorithm. Journal of Machine Learning Research , 25(181):1-51, 2024.

J. Lin and V. Cevher. Optimal distributed learning with multi-pass stochastic gradient methods. In International Conference on Machine Learning , pages 3092-3101. PMLR, 2018.

J. Lin and V. Cevher. Optimal convergence for distributed learning with stochastic gradient methods and spectral algorithms. Journal of Machine Learning Research , 21(147):1-63, 2020.

J. Lin, A. Rudi, L. Rosasco, and V. Cevher. Optimal rates for spectral algorithms with least-squares regression over Hilbert spaces. Applied and Computational Harmonic Analysis , 48(3):868-890, 2020.

A. Mastouri, Y. Zhu, L. Gultchin, A. Korba, R. Silva, M. Kusner, A. Gretton, and K. Muandet. Proximal causal learning with kernels: Two-stage estimation and moment restriction. In International Conference on Machine Mearning , pages 7512-7523. PMLR, 2021.

P. Mathé. Saturation of regularization methods for linear ill-posed problems in Hilbert spaces. SIAM journal on numerical analysis , 42(3):968-973, 2004.

M. Mollenhauer and P. Koltai. Nonparametric approximation of conditional expectation operators. arXiv preprint arXiv:2012.12917 , 2020.

M. Mollenhauer, N. Mücke, and T. Sullivan. Learning linear operators: Infinite-dimensional regression as a well-behaved non-compact inverse problem. arXiv preprint arXiv:2211.08875 , 2022.

N. Mücke, G. Neu, and L. Rosasco. Beating SGD saturation with tail-averaging and minibatching. In Advances in Neural Information Processing Systems , volume 32, 2019.

A. Neubauer. On converse and saturation results for Tikhonov regularization of linear ill-posed problems. SIAM Journal On Numerical Analysis , 34(2):517-527, 1997.

J. Park and K. Muandet. A measure-theoretic approach to kernel conditional mean embeddings. Advances in Neural Information Processing Systems , 33:21247-21259, 2020.

R. Singh, M. Sahani, and A. Gretton. Kernel instrumental variable regression. Advances in Neural Information Processing Systems , 32, 2019.

R. Singh, L. Xu, and A. Gretton. Kernel methods for causal functions: dose, heterogeneous and incremental response curves. Biometrika , 111(2):497-516, 2024.

S. Smale and D.-X. Zhou. Shannon sampling and function reconstruction from point values. Bulletin of the American Mathematical Society , 41(3):279-305, 2004.

S. Smale and D.-X. Zhou. Shannon sampling II: Connections to learning theory. Applied and Computational Harmonic Analysis , 19(3):285-302, 2005.

S. Smale and D.-X. Zhou. Learning theory estimates via integral operators and their approximations. Constructive Approximation , 26(2):153-172, 2007.

L. Song, J. Huang, A. Smola, and K. Fukumizu. Hilbert space embeddings of conditional distributions with applications to dynamical systems. In International Conference on Machine Mearning , pages 961-968, 2009.

B. K. Sriperumbudur, K. Fukumizu, and G. R. Lanckriet. Universality, characteristic kernels and RKHS embedding of measures. Journal of Machine Learning Research , 12(Jul):2389-2410, 2011.

I. Steinwart and A. Christmann. Support Vector Machines . Springer, 2008.

I. Steinwart, D. R. Hush, and C. Scovel. Optimal rates for regularized least squares regression. In COLT , pages 79-93, 2009.

I. Steinwart and C. Scovel. Mercer's theorem on general domains: On the interaction between measures, kernels, and RKHSs. Constructive Approximation , 35(3):363-417, 2012.

J. Weidmann. Linear Operators in Hilbert Spaces . Springer, 1980.

Y. Yao, L. Rosasco, and A. Caponnetto. On early stopping in gradient descent learning. Constructive Approximation , 26(2):289-315, 2007.

H. Zhang, Y. Li, and Q. Lin. On the optimality of misspecified spectral algorithms. Journal of Machine Learning Research , 25(188):1-50, 2024.

H. Zhang, Y. Li, W. Lu, and Q. Lin. On the optimality of misspecified kernel ridge regression. In International Conference on Machine Learning , pages 41331-41353. PMLR, 2023.

Appendices

The appendix is organized as follows. In Section A, we give additional mathematical background and notations. In Section B, we give the proof of Theorem 3 and provide a technical comparison of our proof with [29]. In Section C, we prove Theorem 4. Finally, in Section D, we provide auxiliary results used in the main proofs.

A Additional Background

A.1 Hilbert spaces and linear operators

Definition 3 (Bochner L q -spaces, [14]) . Let H be a separable Hilbert space and π a probability measure on X . For 1 ≤ q ≤ ∞ , L q (X , F X , π ; H ) , abbreviated L q ( π ; H ) , is the space of strongly F X -F H measurable and Bochner q -integrable functions from X to H , with the norms

Definition 4 ( p -Schatten class, e.g. [52]) . Let H,H ′ be separable Hilbert spaces. For 1 ≤ q ≤ ∞ , S p ( H,H ′ ) , abbreviated S p ( H ) if H = H ′ , is the Banach space of all compact operators C from H to H ′ such that ∥ C ∥ S p ( H,H ′ ) ∶= ∥( σ i ( C )) i ∈ I ∥ ℓ p is finite. Here ∥ ( σ i ( C )) i ∈ I ∥ ℓ p is the ℓ p -sequence space norm of the sequence of the strictly positive singular values of C indexed by the at most countable set I . For p = 2 , we retrieve the space of Hilbert-Schmidt operators, for p = 1 we retrieve the space of Trace Class operators, and for p = +∞ , ∥ ⋅ ∥ S ∞ ( H,H ′ ) corresponds to the operator norm ∥ ⋅ ∥ H → H ′ .

Definition 5 (Tensor Product of Hilbert Spaces, [1]) . Let H,H ′ be Hilbert spaces. The Hilbert space H ⊗ H ′ is the completion of the algebraic tensor product with respect to the norm induced by the inner product ⟨ x 1 ⊗ x ′ 1 , x 2 ⊗ x ′ 2 ⟩ H ⊗ H ′ = ⟨ x 1 , x 2 ⟩ H ⟨ x ′ 1 , x ′ 2 ⟩ H ′ for x 1 , x 2 ∈ H and x ′ 1 , x ′ 2 ∈ H ′ defined on the elementary tensors of H ⊗ H ′ . This definition extends to span { x ⊗ x ′ ∣ x ∈ H,x ′ ∈ H ′ } and finally to its completion. The space H ⊗ H ′ is separable whenever both H and H ′ are separable. If { e i } i ∈ I and { e ′ j } j ∈ J are orthonormal basis in H and H ′ , { e i ⊗ e ′ j } i ∈ I,j ∈ J is an orthonormal basis in H ⊗ H ′ .

Theorem 5 (Isometric Isomorphism between L 2 ( π ; Y) and S 2 ( L 2 ( π ) , Y) , Theorem 12.6.1 [1]) . Let H be a separable Hilbert space. The Bochner space L 2 ( π ; H ) is isometrically isomorphic to S 2 ( L 2 ( π ) , Y) and the isometric isomorphism is realized by the map Ψ ∶ S 2 ( L 2 ( π ) , Y) → L 2 ( π ; H ) acting on elementary tensors as Ψ ( f ⊗ y ) = ( ω → f ( ω ) y ) .

A.2 RKHS embbedings into L 2 and Well-specifiedness

Recall that I π ∶H→ L 2 ( π ) is the embedding that maps every function in H into its π -equivalence class in L 2 ( π ) and that we used the shorthand notation [ f ] = I π ( f ) for all f ∈ H . We define similarly I π ∶ G → L 2 ( π ; Y) as the embedding that maps every function in G into its π -equivalence class in L 2 ( π ; Y) .

Definition 6 (Embedding G into L 2 ( π ; Y) ) . Let I π ∶= I Y ⊗ I π be the tensor product of the operator Id Y with the operator I π (see [1, Definition 12.4.1.] for the definition of tensor product of operators). I π maps every function in G into its π -equivalence class in L 2 ( π ; Y) . We then use the shorthand notation [ F ] = I π ( F ) for all F ∈ G .

Remark 5. Let { d j } j ∈ J be an orthonormal basis of Y and recall that { √ µ i [ e i ]} i ∈ I forms an orthonormal basis of [H] 1 . Let F ∈ [G] 1 . Then F can be represented as the element C ∶= ∑ i ∈ I,j ∈ J a ij d j ⊗ √ µ i [ e i ] in S 2 ([H] 1 , Y) by definition of [G] 1 with ∥ C ∥ 2 1 = ∑ i,j a 2 ij . Hence defining ¯ C ∶= ∑ i ∈ I,j ∈ J a ij d j ⊗ √ µ i e i we have C = ¯ C π -a.e. and

Taking the elements identifying ¯ C in G gives a representer ¯ F of F in G .

A.3 Additional Notations

In the following, we fix { d j } j ∈ J an orthonormal basis of Y , where J is at most countable. Recall that { µ 1 / 2 i e i } i ∈ I is an ONB of ( ker I π ) ⊥ in H , and {[ e i ]} i ∈ I is an ONB of ran I π in L 2 ( π ) . Let { ˜ e i } i ∈ I ′ be an ONB of ker I π (with I ∩ I ′ = ∅ ), then { µ 1 / 2 i e i } i ∈ I ∪{ ˜ e i } i ∈ I ′ forms an ONB of H , and { d j ⊗ µ 1 / 2 i e i } i ∈ I,j ∈ J ∪{ d j ⊗ ˜ e i } i ∈ I ′ ,j ∈ J forms an ONB of Y ⊗ H ≃ G .

For any Hilbert space H , linear operator T ∶ H → H and scalar λ > 0 , we define T λ ∶= T + λI H .

B Saturation Effect with Tikhonov Regularization - Proof of Theorem 3

In the following proofs a quantity h n ≥ 0 depending on n ≥ 1 , but independent of τ the confidence level, is equal to o ( 1 ) if h n → 0 when n →+∞ .

We will make extensive use of the following notation in the subsequent analysis.

Definition 7 (Empirical L 2 ( π )-norm) . Denoted by ⟨⋅ , ⋅⟩ 2 ,n , the empirical L 2 ( π )-norm associated to points { x i } n i = 1 independently and identically sampled from the distribution of X , is defined as, for any f, g ∈ H ,

This induces an inner product on H , with associated norm,

Definition 8. Fix x ∈ X and λ > 0 . The regularized canonical feature map is defined as

Recall from Eq. (11) that the ridge estimator ˆ F λ defined in Eq. (7) can be expressed as

where in Theorem 3 we focus on Tikhonov regularization where g λ ( x ) = ( x + λ ) -1 . In that setting we have

Proof of Theorem 3. Since β ≥ 2 , F ∗ ∈ [G] β ⊆ [G] 1 , therefore F ∗ has a representer ¯ F in G such that F ∗ = ¯ F π -a.e. (see Remark 5), and by Theorem 1, ¯ F (⋅) = ¯ Cϕ (⋅) , with ¯ C ∈ S 2 (H , Y) . Define the errors ϵ i ∶= y i -¯ Cϕ ( x i ) , i = 1 , . . . , n , that are i.i.d samples with the same distribution as ϵ ∶= Y -¯ Cϕ ( X ) . By assumption E [∥ ϵ ∥ 2 Y ∣ X ] ≥ σ 2 and by definition E [ ϵ ∣ X ] = 0 . By Eq. (13), we have

The following bias-variance decomposition is the essence of the proof. In the following derivation we abbreviate S 2 ( L 2 ( π ) , Y) to S 2 L 2 ( π ; Y) to L 2 and x 1 , . . . , x n to x n to save space.

The second term is a lower bound on the variance while the first term is a lower bound on the bias .

Bounding the Bias term. The idea is to first show that the population analogue of ∥[ ¯ C ˆ C -1 X,λ ]∥ 2 S 2 ( L 2 ( π ) , Y) can be bounded below by a non-zero constant. We can then bound the difference between the empirical and population version of ∥[ ¯ C ˆ C -1 X,λ ]∥ 2 S 2 ( L 2 ( π ) , Y) using a concentration inequality. By Lemma 1, for λ > 0 , there is a constant c > 0 (see Lemma 1 for the exact value of c ) such that

Furthermore by Lemma 2, there is a constant c 0 > 0 (see Lemma 2 for the exact value of c 0 ) such that for any τ ≥ log ( 4 ) , with probability at least 1 -4 e -τ , for n ≥ ( c 0 τ ) ( 4 + 2 p ) and 1 ≥ λ ≥ n -1 2 + p , we have

Therefore, under the same high probability,

It leads to our final bound on the bias term, for a constant ρ 2 ≥ 0 and for sufficiently large n ≥ 1 , where the hidden index bound depends on τ , we have

Bounding the Variance Term. Using the norm from Definition 7, we have the following chain of identities.

Therefore it suffices to consider ∫ X ∥ ˆ C -1 X,λ k ( x, ⋅)∥ 2 2 ,n dπ ( x ) .

Combining the result of Lemma 4 and Lemma 5 we obtain that for 1 ≥ λ ≥ n -1 2 + p with probability at least 1 -6 e -τ , for n ≥ ( c 0 τ ) 4 + 2 p , the following bounds hold simultaneously for all x ∈ X :

Fix x ∈ X . Using the algebraic identity a 2 -b 2 = ( a -b )( 2 b + ( a -b )) , and recalling that by Definition 7,

we deduce

Using Definition 7 again, this reads

We have

Hence,

Furthermore, by Jensen's inequality,

Therefore we have

Hence

Combined with Eq. (15), it leads to our final bound on the variance term, for a constant ρ 2 ≥ 0 and for sufficiently large n ≥ 1 , where the hidden index bound depends on τ , we have

Putting it together. We are now ready to assemble the lower bounds on the variance and on the bias. For a fixed confidence parameter τ ≥ log ( 10 ) , for sufficiently large n > 0 , where the hidden index

bound depends on τ , with probability at least 1 -10 e -τ , we have by Eq. (14) and Eq. (16), that for λ = λ ( n ) satisfying 1 ≥ λ ≥ n -1 2 + p ,

where ρ 1 , ρ 2 have no dependence on n . Recall Young's inequality, for r, q > 1 satisfying r -1 + q -1 = 1 , we have for all a, b ≥ 0 ,

We apply Young's inequality with r -1 = p /( 2 + p ) and q -1 = 2 /( 2 + p ) , there exists a constant c 1 > 0 such that

To conclude the proof, let λ = λ ( n ) be an arbitrary choice of regularization parameter satisfying λ ( n ) → 0 . We have just covered the case 1 ≥ λ ≥ n -1 2 + p and the case 0 < λ ≤ n -1 2 + p is covered by [29, Section B.4].

Lemma 1. For any λ ≤ 1 and C ∈ S 2 (H , Y) , with C ̸ S 2 ( ran S π , Y) 7 , we have

with a ij ∶= ⟨ d j , C √ µ i e i ⟩ Y , i ∈ I, j ∈ J .

Proof. Recell the notations of Section A.3. Define { a ij } i ∈ I ∩ I ′ ,j ∈ J such that a ij ∶= ⟨ d j , C √ µ i e i ⟩ Y for i ∈ I, j ∈ J and a ij ∶= ⟨ d j , C ˜ e i ⟩ Y for i ∈ I ′ , j ∈ J . Then, on one hand, since C ∈ S 2 (H , Y) ,

On the other hand,

Therefore, noting that ˜ e i = 0 π -a.e. for all i ∈ I ′ , we have,

Therefore the S 2 ( L 2 ( π ) , Y) -norm can be evaluated in closed form using Parseval's identity,

where we used that { d j ⊗[ e i ]} j ∈ J,i ∈ I is orthonormal in Y ⊗ L 2 ( π ) , and λ ≤ 1 . The right hand side has no dependence on λ or n . Furthermore, under assumption (EVD+), µ i > 0 for all i ∈ I , therefore the right hand side term equals zero if and only if a ij = 0 for all i ∈ I, j ∈ J . Since by assumption C ̸ S 2 ( ran S π , Y) , the right hand side is strictly positive.

Lemma 2. Suppose Assumption (EVD) holds with p ∈ ( 0 , 1 ] . Let C ∈ S 2 (H , Y) such that [ C ] ∈ S 2 ([H] 2 , Y) . There is a constant c 0 > 0 such that for any τ ≥ log ( 4 ) , with probability at least 1 -4 e -τ , for n ≥ ( c 0 τ ) ( 4 + 2 p ) and 1 ≥ λ ≥ n -1 2 + p , we have

We have c 0 ∶= 8 κ max { √ c 2 , 1 , 1 } ) where c 2 , 1 is defined in Lemma 16.

Proof. Using the identity A -1 -B -1 = A -1 ( B -A ) B -1 , we obtain

We apply Lemma 22 with γ = 0 ,

We consider each of the four terms in line (17). The last term is bounded above by 1 and the first term is bounded above by λ -1 2 ∥ C ∥ S 2 (H , Y) . By Lemma 20 applied with s = 1 / 2 , we have for the second term

Then, by Lemma 18, for τ ≥ log ( 2 ) , with probability at least 1 -2 e -τ , for √ nλ ≥ 8 τκ √ max {N( λ ) , 1 } , we have

Since N( λ ) ≤ c 2 , 1 λ -p by Lemma 16, and λ ≤ 1 , it suffices to verify that λ satisfies

Since λ ≥ n -1 2 + p by assumption, we deduce the sufficient condition n ≥ ( τc 0 ) 2 ( 2 + p ) , where c 0 ∶= 8 κ max { √ c 2 , 1 , 1 } .

We bound the third term using Lemma 16 [33]. For τ ≥ log ( 2 ) , with probability at least 1 -2 e -τ , we have

where we define

By assumption λ ≥ n -1 2 + p . We thus have

On the other hand, since 1 ≥ λ ≥ n -1 2 + p , using Lemma 16, we have

The first term does not depend on n , and the second term is logarithmic in n . Putting everything together with a union bound, we get a bound on (17). With probability at least 1 -4 e -τ , for n ≥ ( c 0 τ ) ( 4 + 2 p ) , we have

The derivations in the proof of Lemma 1 show that

with a ij ∶= ⟨ d j , C √ µ i e i ⟩ Y , i ∈ I, j ∈ J . Note that since [ C ] ∈ S 2 ([H] 2 , Y) , we have

Hence,

Using the equality a 2 -b 2 = ( a -b )( a + b ) and the reverse triangular inequality, we obtain the following bound, with probability at least 1 -4 e -τ , for n ≥ ( c 0 τ ) ( 4 + 2 p ) ,

where in the second last line we used Equation (18).

Lemma 3. Fix x ∈ X and f x,λ as in Definition 8. For τ ≥ log ( 2 ) , with probability at least 1 -2 e -τ (note that this event depends on x ),

Proof. We start with

We apply Proposition 3 to f = f x,λ , with M = κ 2 λ -1 . For τ ≥ log ( 2 ) , with probability at least 1 -2 e -τ ,

Lemma 4. Suppose that X is a compact set in R d and that k ∈ C θ (X × X) for θ ∈ ( 0 , 1 ] (Definition 11). Assume that 1 ≥ λ ≥ n -1 2 + p . With probability at least 1 -2 e -τ , it holds for all x ∈ X simultaneously that

Proof. The proof follows [29, Lemma C.11]. As we use different notations and tracking of constants, we provide a similar proof in our setting for completeness. By Lemma 24, there exists an ϵ -net F ⊆ K λ ⊆ H with respect to ∥ ⋅ ∥ ∞ such that there exists a positive constant c with

for ϵ to be determined later. Using Lemma 3 and a union bound over the finite set F , with probability at least 1 -2 e -τ , it holds simultaneously for all f ∈ F that

We work in the event where Equation (19) holds for all f ∈ F . By definition of an ϵ -net and K λ , for any x ∈ X , there exists some f ∈ F such that

which in particular implies that

Since ∥ C -1 X,λ k ( x, ⋅)∥ ∞ ≤ κ 2 λ -1 , using the algebraic identity a 2 -b 2 = ( a -b )( 2 b +( a -b )) , we obtain

We therefore have,

We now choose ϵ = 1 n and bound the error term. Recall that 1 ≥ λ ≥ n -1 2 + p , therefore,

Lemma 5. For 1 ≥ λ ≥ n -1 2 + p , with probability at least 1 -4 e -τ , for n ≥ ( c 0 τ ) 4 + 2 p , we have for all x ∈ X simultaneously

where c 0 is the same constant as in Lemma 2.

Proof.

We already saw in the proof of Lemma 2 that the first term is bounded by 1 and there is a constant c 0 > 0 such that for τ ≥ log ( 2 ) , with probability at least 1 -2 e -τ , for n ≥ ( c 0 τ ) 4 + 2 p , the second term is bounded by √ 2 . For the third term we also saw in the proof of Lemma 4 that for τ ≥ log ( 2 ) , with probability at least 1 -2 e -τ , we have

where we defined

Finally, the fourth term is bounded above by λ -1 2 κ . Note that the bound on the fourth term is independent of x , so it holds simultaneously for all x ∈ X . This is in contrast with the setting of Lemma 4 where for each fixed x ∈ X corresponds an element in the ϵ -net of F for which we have a high probability bound, and therefore we must use a union bound in order for the bound to hold simultaneously for all x ∈ X in the proof of Lemma 4. As in the proof of Lemma 4 since 1 ≥ λ ≥ n -1 2 + p , we have

In the bound on ξ δ above, the first term does not depend on n , and the second term is logarithmic in n . Putting everything together by union bound, with probability at least 1 -4 e -τ , for n ≥ ( c 0 τ ) 4 + 2 p , we have

Remark 6 (Comparison to [29]) . We explicit the differences between our proof strategy and the proof strategy of [29].

Scalar versus vector-valued: lower bounding the bias in our case require us to accommodate for the vector-valued setting (see Lemma 1).

New proof of the bias: we lower bound the bias through Lemma 2, while [29] obtain the lower bound in Lemma C.7; however the proof of Lemma C.7 implicitly uses the equality ∥ A -1 ∥ = ∥ A ∥ -1 , with ∥ ⋅ ∥ the operator norm, see Eq. (69) [29] and the preceding equations. It holds that ∥ A -1 ∥ ≥ ∥ A ∥ -1 , but ∥ A -1 ∥ ≤ ∥ A ∥ -1 may not hold in general. We therefore develop a new proof for this step, leading to Lemma 2.

New proof of the variance: we lower bound the variance in Lemma 5, while [29] lower bound the variance in Lemma C.12; to show Eq. (20) , [29] use a covering argument involving N(K λ , ∥ ⋅ ∥ H , ϵ ) (Lemma C.10). However, a close look at the proof of Lemma C.10 (last inequality of the proof) reveals that λ i λ + λ i was mistaken for λ λ + λ i and plugging the correct term in the proof would lead to a vacuous bound. As explained in the proof of Lemma 5, we therefore develop a proof that is free of a covering number argument for this step.

C Learning rates for spectral algorithms

To upper bound the excess-risk, we use a decomposition involving the approximation error expressed as F λ -F ∗ and the estimation error expressed as ˆ F λ -F λ .

where ˆ F λ is the empirical estimator based on general spectral regularization (Eq. (11)) and F λ is its counterpart in population (Eq. (10)). Note that this is a different decomposition than the bias-variance decomposition used in the proof of Theorem 3.

The proof structure is as follows:

Fourier expansion C.1.

Approximation Error C.2.

Estimation error C.3

C.1 Fourier expansion

Recall the notations defined in Appendix A.3. The family { d j } j ∈ J is an ONB of Y , the family { µ 1 / 2 i e i } i ∈ I is an ONB of ( ker I π ) ⊥ and the family { ˜ e i } i ∈ I ′ is an ONB of ker I π such that { µ 1 / 2 i e i } i ∈ I ∪ { ˜ e i } i ∈ I ′ forms an ONB of H . Furthermore, recall that { µ β / 2 i [ e i ]} i ∈ I is an ONB of [H] β , β ≥ 0 .

Lemma 6 (Fourier expansion) . Suppose Assumption (SRC) holds with β ≥ 0 . By definition of the vector-valued interpolation space and by Theorem 5, we have

Then, we have the following equalities with respect to this Fourier decomposition.

The Hilbert-Schmidt operator ( C Y X -C λ C X ) C -1 2 X,λ ∈ S 2 (H , Y) can be written as

3. The Hilbert-Schmidt operator C Y X ∈ S 2 (H , Y) can be written as

Proof. We first derive the Fourier expansion of C Y X ,

where in Eq. (25) we used the tower property of conditional expectation and in Eq. (26) we used the fact that {[ e i ]} i ∈ I forms an orthonormal system in L 2 ( π ) . We can manipulate Eq. (26) to derive Eq. (24),

By the spectral decomposition of C X Eq. (6) and spectral calculus (Definition 9), we have that

where we used r λ ( 0 ) = 1 .

Proof of Eq. (22). Using Eq. (26) and (27), we have

where in Eq. (29), we recall the fact that { √ µ i e i } i ∈ I forms an ONB of ( ker I π ) ⊥ and { ˜ e i } i ∈ I ′ forms an ONB of ker I π .

Proof of Eq. (23). Using Eq. (26) and (28), we have

Lemma 7. Suppose Assumption (SRC) holds with β ≥ 0 , then the following bound is satisfied, for all λ > 0 and 0 ≤ γ ≤ 1 , we have

For the definition of E , see Eq. (8) .

Proof. We adopt the notations of Lemma 6. By Parseval's identity and Eq. (22), we have

In the case of γ ≤ β , we bound g λ ( µ i ) µ i ≤ E using Eq. (8). Then, by Eq. (21),

In the case of γ > β , we apply Eq. (8) to g λ ( µ i ) µ 1 -γ -β 2 i ≤ Eλ -γ -β 2 to obtain, using Eq. (21) again,

Lemma 8. Suppose Assumption (SRC) holds for 0 ≤ β ≤ 2 ρ , with ρ the qualification. Then, the following bound is satisfied, for all λ > 0 , we have

For the definition of ω ρ , see Eq. (9) .

Proof. Recall that in Lemma 6 we used the decomposition

where Assumption (SRC) implies that ∥ F ∗ ∥ 2 β = ∑ ij a 2 ij µ β i < ∞ . Using Eq. (23) in Lemma 6 and Parseval's identity w.r.t. the ONS { d j ⊗ µ 1 / 2 i e i } i ∈ I,j ∈ J in S 2 (H , Y) , we have

Lemma 9. Suppose Assumption (SRC) holds with β ≥ 0 , then for all λ > 0 , we have

where ∥ F ∗ ∥ β = B < ∞ .

Proof. Recall that Lemma 6 we used the decomposition

where ∥ F ∗ ∥ 2 β = ∑ ij a 2 ij µ β i = B 2 < ∞ . Using Eq. (24) in Lemma 6 and C λ = C Y X g λ ( C X ) , we have

where we notice that the S 2 (H , Y) norm of the first term is exactly the β norm of F ∗ , which is given by B . Recalling that C X , ˆ C X are self adjoint, we prove the final result by taking the adjoint and using that an operator has the same operator norm as its adjoint.

C.2 Approximation Error

Lemma 10. Let F λ be given by Eq. (10) based on a general spectral filter satisfying Definition 2 with qualification ρ ≥ 0 . Suppose Assumption (SRC) holds with parameter β ≥ 0 and define β ρ = min { β, 2 ρ } , then the following bound is satisfied, for all λ > 0 and 0 ≤ γ ≤ β ρ ,

Proof. In Eq. (10), we defined F λ (⋅) = C λ ϕ (⋅) . On the other hand, in Lemma 6 we obtained the Fourier expansion of C λ leading to Eq. (22). Thus we have for π -almost all x ∈ X ,

Therefore,

Suppose β ≤ 2 ρ , using Parseval's identity w.r.t. the ONB { d j µ γ / 2 i [ e i ]} i ∈ I,j ∈ J of [G] γ , we have

where we used Eq. (9) in the definition of a filter function, together with 0 ≤ β ≤ 2 ρ and 0 ≤ γ ≤ β , which taken together implies that 0 ≤ β -γ 2 ≤ ρ . Finally, if β ≥ 2 ρ , then since [G] β ⊆ [G] 2 ρ , we can perform the last derivations again with β = 2 ρ to obtain the final result.

C.3 Estimation error

Before proving the main results we recall two embedding properties for the vector-valued interpolation space [G] β (Definition 1). The first embedding property lifts the property (EMB) defined for the scalar-valued RKHS [H] α to the vector-valued RKHS [G] α .

Lemma 11 ( L ∞ -embedding property - Lemma 4 [31]) . Under (EMB) the inclusion operator I α,β π ∶ [G] α ↪ L ∞ ( π ; Y) is bounded with operator norm A ,

Theorem 6 ( L q -embedding property - Theorem 3 [31]) . Let Assumption (EMB) be satisfied with parameter α ∈ ( 0 , 1 ] . For any β ∈ [ 0 , α ) , the inclusion map

is bounded, where q α,β ∶= 2 α α -β .

The L q -embedding property was first introduced in the scalar-valued setting in [55] and then lifted to the vector-valued setting by [31]. Its role is to replace a boundedness condition on the ground truth function F ∗ . We now explain how the L q -embedding property can be combined with Assumption (EMB) and a truncation technique.

Lemma 12. Recall that π is the marginal measure of X on X . For t ≥ 0 , define the measurable set Ω t as follows

Let q > 0 . Assume that F ∗ ∈ L q ( π ; Y) . In other words, there exists some constant c q > 0 such that

Then we have the following conclusions

The π -measure of the complement of Ω can be bounded by

Recall that { x i } n i = 1 are i.i.d. samples distributed according to π . If t = n 1 ˜ q for ˜ q < q , then we can conclude as follows. For a fixed parameter τ > 0 , for all sufficiently large n , where the hidden index bound depends on q ˜ q -1 and τ , we have

Proof. The first claim is a straightforward application of Markov's inequality, as follows

To show the second claim, we first evaluate the probability that there exists some x i 's that lies outside Ω t ,

where in the last inequality we used Bernoulli's inequality, which states that for r ≥ 1 and 0 ≤ x ≤ 1 ,

By assumption t = n 1 q t for some fixed q > q t > 0 . We thus have

for sufficiently large n , where the hidden index bound depends on q q t and τ .

We adapt [31, Lemma 5] to the spectral algorithms setting.

Lemma 13. Suppose Assumptions (SRC) and (EMB) hold for some 0 ≤ β ≤ 2 ρ , with ρ the qualification, then the following bounds are satisfied, for all 0 < λ ≤ 1 ,

Proof. We use Lemma 11 and Lemma 7 to write:

This proves Eq. (31). To show Eq. (30), in the case β ≤ α we use the triangle inequality, Eq. (31) and λ ≤ 1 to obtain

In the case β > α , Eq. (30) is a consequence of Lemma 11 and Lemma 10 with γ = α (here we use the assumption 0 ≤ β ≤ 2 ρ ),

We adapt [54, Theorem 13] to the vector-valued setting.

Theorem 7. Suppose that Assumptions (EMB) , (EVD) , (MOM) and (SRC) hold for 0 ≤ β ≤ 2 ρ , where ρ is the qualification, and p ≤ α ≤ 1 . Denote, for i = 1 , . . . , n ,

Then for all τ ≥ 1 , with probability at least 1 -2 e -τ , we have

where R is the constant from Assumption (MOM) , and

where A is the constant from Assumption (EMB) , and E,ω ρ are defined in Eq. (8) and (9) respectively.

Proof. We wish to apply vector-valued Bernstein's inequality, namely Theorem 10. We thus compute,

First we consider the inner integral, by Assumption (MOM),

Plugging the above inequality into Eq. (32), as well as introducing the shorthand,

and for t ≥ 0 , we have

We bound term (33) using Lemma 15 and Lemma 17 with l = 1 . We have,

Therefore we bound term (33) as follows,

If β ≥ α , by Assumption (EMB),

If β < α , by Lemma 13, we have for π -almost all x ∈ Ω t ,

Therefore, for all β ∈ [ 0 , 2 ρ ] ,

Using Lemma 17 with l = 1 , we have,

Putting everything together,

t

}∥

m

S

2

E [∥ ξ ( X,Y ) ✶ { X ∈ Ω We now apply Theorem 10 with

(H

,

Y)

2

(

R

+

χ

(

t, λ

))

A

2

λ

] ≤

m

!

(

α

)

(

σ

N

1

2

We bound ∥ F ∗ -[ F λ ]∥ L 2 ( π ; Y) using Lemma 10 with γ = 0 ,

The conclusion is, for all τ ≥ 1 , with probability at least 1 -2 e -τ , we have

Lemma 14. Suppose that the same assumptions and notations listed in Theorem 7 hold.

Suppose β + p > α , and λ ≍ n -1 β + p . For any fixed τ ≥ 1 , with probability at least 1 -2 e -τ , suppose that the truncation level t satisfies

then there exists a constant c > 0 such that

m

-

2

(

λ

) + ∥

F

∗

-[

F

λ

]∥

2

L

2

(

π

;

Y)

2

A

λ

2

α

)

.

Suppose β + p ≤ α , and λ ≍ ( n log θ ( n ) ) 1 α for some θ > 1 . For any fixed τ ≥ 1 , with probability at least 1 -2 e -τ , suppose that the truncation level t satisfies

then there exists a constant c > 0 such that

Proof. Note that Theorem 7 yields the same conclusion as in the scalar-valued case proved in [54, Theorem 13]. The Lemma then follows from the analysis for the scalar-valued case in the proof of [54, Theorem 15].

We adapt [55, Theorem 15] to the vector-valued setting.

Theorem 8. Suppose that Assumptions (EMB) , (EVD) , (MOM) and (SRC) hold for 0 ≤ β ≤ 2 ρ , where ρ is the qualification, and p ≤ α ≤ 1 .

In the case of β + p > α , choosing λ ≍ n -1 β + p , for any fixed τ ≥ log ( 4 ) , when n is sufficiently large, with probability at least 1 -4 e -τ , we have

where c is a constant independent of n, τ, λ .

In the case of β + p ≤ α , choosing λ ≍ ( n log θ ( n ) ) -1 α for some θ > 1 . We make the additional assumption that there exists some α ′ < α such that Assumption (MOM) is satisfied for α ′ < α . Then, for any fixed τ ≥ log ( 4 ) , when n is sufficiently large, where the hidden index bound depends on α -α ′ , with probability at least 1 -4 e -τ , we have

where c is a constant independent of n, τ, λ .

Proof. By assumption (EMB) and Theorem 6, if β < α , then the inclusion map

is bounded, where q α,β ∶= 2 α α -β . If β ≥ α , then by Lemma 11 the inclusion map

is bounded and therefore [G] β is continuously embedded into L q ( π ; Y) for any q ≥ 1 . In the rest of the proof, we will use q to denote q α,β , unless otherwise specified. Furthermore, we will use c q = ∥ F ∗ ∥ L q ( π ; Y) .

We first consider the case β + p > α . We can easily verify using β + p > α that the following inequality holds

Choose t = n ˜ q -1 , where

We thus have

Thus the assumptions for both Lemma 12 and Lemma 14 are satisfied.

We then consider the case β + p ≤ α . We now apply Assumption (EMB) and Theorem 6 to α ′ instead of α . We obtain that the inclusion map I q α ′ ,β π is bounded, where we recall that q α ′ ,β is defined to be 2 α ′ α ′ -β . Since x ↦ 2 x x -β is monotonically decreasing for x > β , we obtain the inequality

We choose t = n 1 q α,β . By construction, t satisfies the assumptions in Lemma 14. Furthermore, the assumptions of Lemma 12 are satisfied, with F ∗ ∈ L q ′ ( π, Y) , and t = n 1 q α,β .

Having established the applicability of Lemma 14 and Lemma 12, let us turn our attention to proving the results of the Theorem. Denote

We compute

1. The β + p > α case. Have

✶ We can bound the first term with probability at least 1 -2 e -τ by cτ ( n log θ ( n ) ) -β 2 α , according to Lemma 14. By Lemma 12, with probability at least 1 -e -τ , for sufficiently large n , x i ∈ Ω t for all i ∈ [ n ] , whereby the second term is zero. It remains to bound the third term, where our bound will be deterministic. Using Jensen's inequality, we have,

✶ where in the third line we used Lemma 17. We first split the second term into an approximation error and a noise term using triangle inequality.

E [∥( y -C λ ϕ ( x )) ✶ { x ∉ Ω t }∥ Y ] ≤ E [∥( y -F ∗ ( x )) ✶ { x ∉ Ω t }∥ Y ]+ E [∥( F ∗ ( x ) -F λ ( x )) ✶ { x ∉ Ω t }∥ Y ] We bound the first term using the tower property of conditional expectation,

where in the third inequality we used Assumption (MOM) with q = 2 , and in the fourth inequality we used Lemma 12. We bound the second term using Cauchy-Schwarz inequality and Lemma 10 with γ = 0 ,

Therefore, using Lemma 12, we have,

We now plug in λ ≍ n -1 β + p . Recall that by construction t > n 1 q . Thus,

We've therefore proved inequality (34).

The β + p ≤ α case. We proceed similarly to the β + p > α case. We have,

✶ We can bound the first term with probability at least 1 -2 e -τ by cτ ( n log θ ( n ) ) -β 2 α , according to Lemma 14. By Lemma 12, with probability at least 1 -e -τ , for sufficiently large n , x i ∈ Ω t for all i ∈ [ n ] , whereby the second term is zero. We bound the third term by Eq. (36). We now plug in λ ≍ ( n log θ ( n ) ) -1 α . Recall that by construction t > n 1 q . Thus,

We have therefore proved inequality (35).

We adapt [54, Theorem 16] to the vector-valued setting.

Theorem 9 (Bound of estimation error) . Suppose that assumptions (EMB) , (EVD) , (MOM) and (SRC) hold for 0 ≤ β ≤ 2 ρ , where ρ is the qualification, and p ≤ α < 1 . For 0 ≤ γ ≤ 1 , with γ ≤ β ,

In the case of β + p > α , by choosing λ ≍ n -1 β + p , for any fixed τ ≥ log ( 4 ) , when n is sufficiently large, with probability at least 1 -4 e -τ , we have

where c is a constant independent of n, τ .

In the case of β + p ≤ α , by choosing λ ≍ ( n log θ ( n ) ) -1 α , for any fixed τ ≥ log ( 4 ) , when n is sufficiently large, with probability at least 1 -4 e -τ , we have

where c is a constant independent of n, τ .

Proof. Firstly, we establish the applicability of Lemma 19.

The β + p > α case. Have λ ≍ n -1 β + p , hence

whereas using λ ≤ ∥ C X ∥ H→H for sufficiently large n , as well as Lemma 16,

Therefore, for a fixed τ > 0 , for all sufficiently large n , Eq. (40) in Lemma 19 is satisfied.

whereas similar to the β + p > α case, we ahve

Therefore, for a fixed τ > 0 , for all sufficiently large n , Eq. (40) in Lemma 19 is satisfied.

τ

We thus conclude for all α ∈ ( 0 , 1 ] , with probability ≥ 1 -2 e -, Eq. (41) and (42) are satisfied simultaneously.

We exploit the following decomposition

where in the first inequality we used Lemma 22, in the third inequality we used Eq. (42) and in the last inequality we used Lemma 21. We consider the following decomposition

Term (I) . The high level idea is to bound (I) by exploiting the first axiom of the filter function (8), where g λ ( ˆ C X ) is intuitively a regularized inverse of ˆ C X , by grouping it with ˆ C X,λ .

where the second inequality follows from Eq. (42). We consider the following decomposition

Hence where

We bound the first term by Theorem 8 and the second term by Lemma 8. This yields, for τ ≥ log ( 4 ) , with probability at least 1 -4 e -τ , for some constant c > 0 which does not depend on n, τ, λ ,

where we used that τ > 1 . So collecting all the relevant constants together, we can write the upper bound of term (I) as follows: with probability at least 1 -4 e -τ , for some constant c ′ > 0 (different from the c before) which does not depend on n, τ, λ , we have

Term (II) . Using Lemma 9, we have

The second term is the same as the scalar-valued case, which is bounded in Step 3 of the proof of [54, Theorem 16]. We define

By the proof of [54, Theorem 16], we have, with probability at least 1 -6 e -τ

✶ 1. Case β + p > α . In this case λ ≍ n -1 β + p . We note that for β > 2 , ∆ 1 as a function of n can be written as

Note that

Hence

Therefore we have shown that there exists some constant c ′′ > 0 , independent of n, λ, τ , such that with probability at least 1 -6 e -τ , for sufficiently large n ,

2. Case β + p ≤ α . In this case β ≤ α ≤ 1 , and λ ≍ ( n log θ ( n ) ) -1 α . We have also shown that there exists some constant c ′′ > 0 , independent of n, λ, τ , such that with probability at least 1 -6 e -τ , for sufficiently large n ,

Putting together Lemma 10 and Theorem 9, we have proved Theorem 4.

D Auxiliary Results

D.1 Spectral Calculus, Proof of Proposition 1 and Empirical Solution

Definition 9 (Spectral Calculus; see 16, Chapter 2.3) . Let H be a Hilbert space. Consider g ∶ R → R and a self-adjoint compact operator A ∶ H → H admitting a spectral decomposition written as

whenever this series converges in operator norm.

Proof of Proposition 1. We define the sampling operator S ∶ R n →H and it dual S ∗ ∶H→ R n ,

We can verify that ˆ C X = n -1 SS ∗ and K = S ∗ S . Let Y = ( y i ) n i = 1 ∈ R n . We have, for all x ∈ X ,

To go from (38) to (39), we make the following observation. Consider the singular value decomposition of the compact operator S , there is m ≤ n such that

where ( e i ) i , ( f i ) i are orthonormal sequences in R n and H respectively. We then have

We then define g ( A ) ∶ H → H as

Therefore, we deduce

Similarly,

Hence we have proved on G must satisfy

where F (⋅) = Cϕ (⋅) .

Proof. By Corollary 1, it is equivalent to solve the following optimization problem on S 2 (H , Y) ,

Recall for a Hilbert-Schmidt operator L ∈ S 2 (H , Y) , we have

Using this, we re-write the objective as an inner product in S 2 (H , Y) :

Taking the Fréchet derivative with respect to C and setting in to zero, we obtain the following first order condition

D.2 Properties Related to Assumptions (EMB) and (EVD)

Lemma 15 (Lemma 13 [17]) . Under (EMB) , the following inequality is satisfied, for λ > 0 and π -almost all x ∈ X ,

Definition 10 ( l -effective dimension) . For l ≥ 1 , the l -effective dimension N l ∶ ( 0 , ∞) → [ 0 , ∞) is defined by

The 1 -effective dimension is widely considered in the statistical analysis of kernel ridge regression (see [6], [5], [32], [34], [17]). The following lemma provides upper and lower bounds for the l -effective dimension.

as desired.

Proposition 2. Any minimizer F ∈ G of

Lemma 16. Suppose Assumption (EVD) holds with parameter p ∈ ( 0 , 1 ] , for any λ ∈ ( 0 , 1 ] , there exists a constant c 2 ,l > 0 independent of λ such that

If furthermore, Assumption (EVD+) holds with parameter p ∈ ( 0 , 1 ) , for any λ ∈ ( 0 , 1 ] , there exists a constant c 1 ,l > 0 independent of λ such that

The proof can be found in [29] (Proposition D.1), but as the proof is incomplete we provide a full proof for completeness. This allows us to detect that the value p = 1 in Assumption (EVD+) is not compatible with the assumption of a bounded kernel (see Remark 7 below).

Proof.

Let us now consider the integral. Let us first consider p ≤ 1 < l ,

Therefore, using λ ≤ 1 , we can take c 2 ,l = 1 + D l 2 p l -p . The remaining edge case p = 1 = l , is covered by [17, Lemma 11] with c 2 , 1 = ∥ C X ∥ S 1 (H) . For the lower bound, we proceed similarly. For p ∈ ( 0 , 1 ) (and therefore p < l ),

Therefore, we can take c 1 ,l = ( D 1 D 1 + 1 ) l p l -p .

Remark 7. We note that Assumption (EVD+) with p = 1 is not compatible with the assumption that k is bounded (Assumption 3). Indeed, suppose that µ i ≥ D 1 i -1 , for all i ≥ 1 . Recall that {[ e i ]} i ≥ 1 forms an orthonormal set in L 2 ( π ) . By Mercer's theorem,

which leads to a contradiction.

Lemma 17. For any l ∈ [ 1 , 2 ] , the following equality holds,

Proof. Fix x ∈ X . Since k ( x, ⋅) ∈ H , and { µ 1 / 2 i e i } i ∈ I is an ONB of ( ker I π ) ⊥ , we have that π -almost everywhere

This is Mercer's Theorem [51]. On the other hand, π -almost everywhere,

Therefore, and by Parseval's identity, using that { µ ( 2 -l )/ 2 i [ e i ]} i ∈ I is an ONB of [H] 2 -l ,

In particular for l = 1 , and for l = 2 ,

Therefore,

where we used that ([ e i ]) i ∈ I forms an orthonormal set in L 2 ( π ) .

D.3 Concentration Inequalities

The following Theorem is from [17, Theorem 26 ].

Theorem 10 (Bernstein's inequality) . Let ( Ω , B , P ) be a probability space, H be a separable Hilbert space, and ξ ∶ Ω → H be a random variable with

for all m ≥ 2 . Then, for τ ≥ 1 and n ≥ 1 , the following concentration inequality is satisfied

In particular, for τ ≥ 1 and n ≥ 1 ,

Lemma 18. Let τ ≥ log ( 2 ) , with probability at least 1 -2 e -τ , for √ nλ ≥ 8 τκ √ max {N( λ ) , 1 } , we have

Proof. The proof is identical to [5, Proposition 5.4] with the only difference that in their setting κ = 1 .

Proposition 3 (Proposition C.9 29) . Let π be a probability measure on X , f ∈ L 2 ( π ) and ∥ f ∥ ∞ ≤ M . Suppose we have x 1 , . . . , x n sampled i.i.d. from π . Then, for any τ ≥ log ( 2 ) , the following holds with probability at least 1 -2 e -τ :

where ∥ ⋅ ∥ 2 ,n was defined in Definition 7.

Lemma 19 (Lemma 12 54) . Let Assumptions (EMB) , (SRC) and (MOM) be satisfied. For τ ≥ 1 , if λ and n satisfy that

then the following operator norm bounds are satisfied with probability not less than 1 -2 e -τ

D.4 Miscellaneous results

Lemma 20 (Cordes inequality [18]) . Let A,B be two positive bounded linear operators on a separable Hilbert space H and s ∈ [ 0 , 1 ] . Then

Lemma 21 (Lemma 25 [17]) . For λ > 0 and s ∈ [ 0 , 1 ] , we have

We recall the following basic Lemma from [30, Lemma 2].

Lemma 22. For 0 ≤ γ ≤ 1 and F ∈ G , the inequality

holds, where C = ¯ Ψ -1 ( F ) ∈ S 2 (H , Y) . If, in addition, γ < 1 or C ⊥ Y ⊗ ker I π is satisfied, then the result is an equality.

Definition 11. Let X ⊆ R d be a compact set and θ ∈ ( 0 , 1 ] . For a function f ∶ X → R , we introduce the Hölder semi-norm

where ∥ ⋅ ∥ represents the usual Euclidean norm. Then, we define the Hölder space

which is equipped with the norm

The next lemma is used to prove Lemma 24 below. It appears in [29, Lemma A.3], albeit the use of an erroneous equality in their proof: ∥ k ( x, ⋅) -k ( y, ⋅)∥ 2 H = k ( x, x ) k ( y, y ) -k ( x, y ) 2 . We therefore provide our own proof of this result.

Lemma 23. Assume that H is an RKHS over a compact set X ⊆ R d associated with a kernel k ∈ C θ (X × X) for θ ∈ ( 0 , 1 ] . Then, we have H⊆ C θ 2 (X) and

Proof. For all ( x, y ) ∈ X and f ∈ H , by the reproducing property and Cauchy-Schwarz inequality,

Then, using k ∈ C θ (X × X) , we obtain

which concludes the proof.

We derive as a corollary a quantitative upper bound on the ϵ -covering number of the the set of (spectral) regularized kernel basis function with respect to the ∥ ⋅ ∥ ∞ norm.

Lemma 24 (Lemma C.10 by 29) . Assume that H is an RKHS over a compact set X ⊆ R d associated with a kernel k ∈ C θ (X × X) for θ ∈ ( 0 , 1 ] . Assume that k ( x, x ) ≤ κ 2 for all x ∈ X . Then, we have that for all ϵ > 0 ,

where K λ ∶= { C -1 X,λ k ( x, ⋅)} x ∈X , and c is a positive constant which does not depend on λ, ϵ and only depends on κ and [ k ] θ, X×X . N(K λ , ∥ ⋅ ∥ ∞ , ϵ ) denotes the ϵ -covering number of the set K λ in the norm ∥ ⋅ ∥ ∞ (see [49, Definition 6.19] for the definition of covering numbers).

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We specifically point out the settings and detailed contributions of our work in both abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In describing our assumptions as well as discussing our theoretical results, we specifically list our limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide our assumptions in Section 2.3. The complete proof of our works are listed in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: Our paper does not contain experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our paper mainly focuses on investigating the learning efficiency of spectral algorithms. We therefore believe our paper does not have any negative societal impacts. We explain how our theory can improve our understanding of various learning algorithms in the introduction which could be potential positive societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the vector-valued regression problem. Given $n $ iid samples $\\{(x\\_i,y\\_i)\\}\\_{i=1}^n$ from a distribution $\\mathcal{D}$ on $\\mathcal{X} \\times \\mathcal{Y}$, the goal is to output the estimator $\\hat{f}$ such that $\\mathbb{E}[||\\hat{f}(x)-f^{\\star}(x)||\\_{\\mathcal{Y}}^2]$ is small. Here, $f^{\\star}$ is the optimal regressor, that is $\\mathbb{E}[y \\mid x]$. In this work, the authors consider the case where the estimator $\\hat{f} $ is obtained through spectral regularization methods such as Tikhonov, hard-thresholding, and iteration filters.\n\nSTRENGTHS:\n- In the RKHS framework, the ridge estimator obtained by Tikhonov regularization is a canonical estimator. However, in Theorem 3, the authors show that the ridge estimator cannot exploit the higher-order smoothness of the optimal regressor  $f^{\\star}$ . \n\n- In Theorem 4, the authors show that other spectrally regularized estimators (for example, one based on hard-thresholding) can exploit the higher-order smoothness of the functions. \n\n- The paper also points out some inaccuracies in earlier work. I did not verify the validity of these claims, but if true, this is also an important contribution to the literature.\n\n- Overall, the paper is well-written and is easy to follow.\n\nWEAKNESSES:\nThe proof of Theorem 4 is fully deferred to the Appendix. Given that the proof is highly technical, it would be helpful for the reader to include a high-level proof sketch in the main text of the paper. For example, some discussion of key challenges in generalizing the proof of scalar-valued to general $\\mathcal{Y}$ would be useful.\n\nQUESTIONS:\nIf the proof in [1] were correct or can be readily fixed, can't Theorem 3 be inferred immediately through the result in the scalar-valued case? Here is a sketch of the argument:\n\n Let $\\\\{d\\_j\\\\}\\_{j \\in \\mathbb{N}}$ be the ONB of $\\mathcal{Y}$, and $\\mathcal{Y}\\_1 = \\\\{ \\langle y, d_1 \\rangle \\quad |\\quad  y \\in \\mathcal{Y} \\\\}$ to be the subspace along the direction $d\\_1$.  Consider the probability distribution such that $y \\mid x$ is only supported over $\\mathcal{Y}\\_1$.  \nSuppose $f^{\\star}$ is the optimal regressor and $\\hat{f}\\_{\\lambda}$ is the KRR estimator. Then, we have\n\n$$\\mathbb{E}[||\\hat{f}\\_{\\lambda}-f^{\\star}||\\_{\\mathcal{Y}}^2] = \\int ||\\hat{f}\\_{\\lambda}(x)-f^{\\star}(x) ||\\_{\\mathcal{Y}}^2   p(x, dy)  \\pi(dx) \\geq \\int | \\langle \\hat{f}\\_{\\lambda}(x)-f^{\\star}(x) ,d_1\\rangle |^2  p(x, dy)  \\pi(dx) . $$\n\nNow, this is effectively lower bounding the risk for scalar-valued regression. I believe that this argument can be formalized by defining a one-to-one mapping between $\\mathcal{Y}\\_1 $ and $\\mathbb{R}$. I might be missing something here, so please correct me if I am wrong.\n\n\n\n[1] Y. Li, H. Zhang, and Q. Lin. On the saturation effect of kernel ridge regression. In The Eleventh 390 International Conference on Learning Representations, 2023.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript presents the excess risk upper bound for spectral regularized algorithms whose output might belong to potential infinite-dimensional Hilbert space. Additionally, the saturation effect for a special case of vector-valued spectral algorithms, KRR, is rigorously confirmed.\n\nSTRENGTHS:\n1. The manuscript is easier to follow as it provides enough detail for moving from scalar-valued RKHS to vector-valued RKHS.\n2. The manuscript established a series of well-studied properties/results for spectral algorithms and saturation effect to vector-valued output scenarios, which, to the best of my knowledge, is less concerned and studied in the community.\n3. Identify some issues for proving the lower bound in previous work [1] and provide new techniques to handle the bias and variance.\n\nWEAKNESSES:\n1. I find that the saturation effect, or more exactly, the lower bounds for real-valued spectral algorithms, is proved in [2], seemingly a following work of [1]. Is there any specific obstacle the authors are facing to prove the lower bounds for the vector-valued spectral algorithms?\n2. The (EVD+) condition looks weird to me. The authors state that the lower bound for the eigenvalue depends not only on $p$ but also on the running index $i$, and this is needed for the lower-bound proof. Can authors elaborate more on this? I especially note that [1], who considered the real-valued case, does not have such a requirement of dependence on $i$ in the lower bound. Is this a unique challenge present by the vector-valued setting? Or is this due to your correction for the proof issues in [1]?\n\n[1] Y. Li, H. Zhang, and Q. Lin. On the saturation effect of kernel ridge regression. In The Eleventh International Conference on Learning Representations, 2023.\n\n[2] Li, Yicheng, et al. \"Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decay.\" _arXiv preprint arXiv:2401.01599_ (2024).\n\nQUESTIONS:\n1. Just of independent interest. Consider spectral algorithms with finite qualification context (like KRR). Since there is a gap between the upper and lower bound when the interpolation index $\\beta$ is greater than $2\\rho$, we can consider the following approach to avoid this gap. Based on your misspecification results, one might consider imposing a kernel whose induced space RKHS is much 'smoother' than the RKHS to which the true function belongs. However, without the prior of the true RKHS, it is hard to pick the imposed kernel to make it 'smooth' enough. \n   \n   So, I'm wondering if there are some general approaches to avoid the saturation effect. In practical applications, when one needs to use a specific general algorithm with finite $\\rho$, this seems to be an important issue. (I understand this is a theoretical paper, but this just popped up in my head when I read the theorems).\n   \n2. Is the $Id_{\\mathcal{Y}}$ in Equation (2) the identical element in $\\mathcal{Y}$? If so, I think you should define it for completeness.\n\nLIMITATIONS:\nWhile the authors claim they discussed the limitation in terms of assumptions in the checklist. I don't clearly see them. But, to my knowledge, these assumptions are almost standard in the literature, except the one I raise in W2.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the regression task of learning a mapping where both the input space and the output space can potentially be infinite dimensional. The authors formulate the problem setting by proposing a number of assumptions that can be thought of as the vector-valued counterparts of the standard assumptions in (real-valued) kernel regression. In the well-specified regime, the authors show that a saturation effect exists i.e. the Tikhonov regularized regression estimator is provably suboptimal. The same phenomenon is known to exist in the real-valued setting, but extension to vector-valued output is a novel contribution. Finally, the authors show that for estimators based on a class of filter functions, one can establish error rates that match the best-known upper bounds even in the real-valued setting.\n\nSTRENGTHS:\n1. The paper is well written. Though the authors consider an extension of previous works where the output space is R, all relevant notions and assumptions in the current setting are rigorously defined.\n\n2. The main result of this paper addresses the optimality (at least for some set of smoothness parameters) of a class of regression estimators based on filter functions, going beyond Tikhonov regularization that most often appears in existing literature.\n\nWEAKNESSES:\n1. While the mathematical parts of this paper look sound, and the results appear to be novel, both the error rates and the proof techniques seem to have no difference with the finite-dimensional output setting. I haven't gone through all the proof details and it might be the case that additional challenges arise in the vector-valued setting. If this is the case, then it would be great if the authors can point to the most challenging parts of the proof with several sentences in the main part of the paper.\n\n2. The main part of the paper ends a little abruptly -- it would be better to write a final section summarizing the contributions of the paper and discussing potential future directions. The problem setting and necessary assumptions may be introduced in a more concise way.\n\nQUESTIONS:\nDoes the lower bound for real-valued output ($n^{-\\frac{\\max\\{\\alpha,\\beta\\}-\\gamma}{\\max\\{\\alpha,\\beta\\}+p}}$) directly imply a lower bound in the vector-valued setting? If this is the case, it would be nice to state it as a theorem after Theorem 4.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe  submission explores a class of spectral learning algorithms for regression within the context of supervised learning using random design. The focus is on high-dimensional and potentially infinite-dimensional output spaces. The problem is framed as minimizing the risk associated with the least squares loss, over vector-valued reproducing kernel Hilbert spaces (RKHS).\n\nThe authors make several contributions: \n\n1. Saturation Effect of Ridge Regression: The paper rigorously confirms the saturation effect in ridge regression for general Hilbert output spaces. \n\n2. Convergence Rates for Spectral Algorithms: The paper provides upper bounds on the rates of convergence for a broad range of spectral algorithms even in the misspecified learning case where the target function might not be contained within the RKHS. The smoothness of the target function is characterized using interpolation spaces.\n\nSTRENGTHS:\n1. The paper extends non-parametric regression results involving spectral regularization to a broader setting that includes potentially infinite-dimensional output spaces. \n\n2. The authors mostly clearly explain their contributions, making the advancements in the field accessible. \n\n2.They rigorously introduce the mathematical framework of vector-valued reproducing kernel Hilbert spaces (RKHSs) and regression, ensuring a solid theoretical foundation.\n\n3. One particularly commendable aspect of the approach is the expression of the smoothness of the target function in terms of vector-valued interpolation spaces, which appears to be a natural and effective strategy. \n\n4. Rates of convergence are presented that are shown to be optimal in the well-specified case, effectively closing a gap in the literature.\n\n5. Additionally, the paper demonstrates tight lower bounds for Tikhonov regularization with a Hölder continuous kernel, proving that saturation is an unavoidable phenomenon in this context. The authors extend the results from [28] from real-valued output spaces to infinite dimensional output spaces using the same bias-variance decomposition, however with a simpler approach. \n\nOverall, the research provides valuable insights into spectral learning algorithms and their application in high-dimensional settings.\n\nWEAKNESSES:\n1. While the paper does close a gap in the literature, it follows the usual lines in non-parametric regression over RKHSs. The results are somewhat expected, and the approach is not actually new. \n\n2. The setting seems rather restrictive: The maps that are learned with classical regularization approaches are basically linear operators (Thm. 1) . Perhaps, the authors can state this more clearly to better distinguish between other current streams of operator learning research (in particular non-linear operator learning). \n\n3. The difference between this work and previous work [38], particularly in the well-specified case, could be explained in more detail to highlight the novel contributions.\n\nQUESTIONS:\nQ1: Is the machinery with introducing vvRKHSs really necessary when functions of Hilbert-Schmidt operators are learned?  \n\nQ2: Theorem 2: Can \"with high probability\" be made more precise?\n\nminor: \n* some brackets for references are missing, e.g. p. 1, l.20, 33 and through out the manuscript \n* the references [33] and [34] are the same, also [6] and [7]\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis papers focuses on learning vector-valued functions in reproducing kernel Hilbert spaces (RKHS). The kernel in this case is an operator-valued function instead of a scalar-valued function. The papers considers kernel-based vector-valued regression with spectral regularization, which include ridge regression and kernel PCR. The contribution of the paper is theoretical: learning rates for vector-valued and spectral regularization-based regression are derived. In the case of kernel ridge regression upper and lower rates are provided. In the general case of spectral filter function only upper rates are given.\n\nSTRENGTHS:\n* The paper provides new theoretical results for vector-valued RKHS-based regression.\n* The paper is well written.\n\nWEAKNESSES:\n* Contributions compared to previous work should be made more clear.\n\nQUESTIONS:\n* Section 3 focuses on kernel ridge regression (KRR). Learning rates and results on the saturation effect of vector-valued KRR have been reported in [1]. Could you make clear what are the contribution compared to [1] here? What are the main technical challenges compared to [1]?\n\n[1] Li, Zhu, et al. \"Towards Optimal Sobolev Norm Rates for the Vector-Valued Regularized Least-Squares Algorithm.\" JMLR (2024).\n\n* Excess risk bounds for vector-valued learning with spectral filtering are provided in [2]. How these bounds can be compared to those obtained in this work?\n\n[2] Baldassarre, Luca, et al. \"Multi-output learning via spectral filtering.\" Machine learning (2012).\n\n* Theorem 4 provides upper rates for vector-valued function learning with general spectral function. How optimality is maintained in this case?\n\n* Could the results be extended to other classes of operator-valued kernels (e.g., non-separable kernels)?\n\nLIMITATIONS:\nA detailed discussion on limitations is missing.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}