# Probability Error Bounds for Approximation of Functions in Reproducing Kernel Hilbert Spaces

Authors: Ata Deniz Aydın; Aurelian Gheondea
Journal: Journal of Function Spaces
 (2021)

---

## Abstract

We find probability error bounds for approximations of functionsf in a separable reproducing kernel Hilbert space H with reproducing kernel K on a base space X, firstly in terms of finite linear combinations of functions of type Kxi and then in terms of the projection πxn on spanKxii=1n, for random sequences of points x=xii in X. Given a probability measure P, letting PK be the measure defined by dPKx=Kx,xdPx, x∈X, our approach is based on the nonexpansive operator L2X;PK∋λ↦LP,Kλ≔∫XλxKxdPx∈H, where the integral exists in the Bochner sense. Using this operator, we then define a new reproducing kernel Hilbert space, denoted by HP, that is the operator range of LP,K. Our main result establishes bounds, in terms of the operator LP,K, on the probability that the Hilbert space distance between an arbitrary function f in H and linear combinations of functions of type Kxi, for xii sampled independently from P, falls below a given threshold. For sequences of points xii=1∞ constituting a so-called uniqueness set, the orthogonal projections πxn to spanKxii=1n converge in the strong operator topology to the identity operator. We prove that, under the assumption that HP is dense in H, any sequence of points sampled independently from P yields a uniqueness set with probability 1. This result improves on previous error bounds in weaker norms, such as uniform or Lp norms, which yield only convergence in probability and not almost certain convergence. Two examples that show the applicability of this result to a uniform distribution on a compact interval and to the Hardy space H2D are presented as well.


---

## Body

## 1. Introduction

Several machine learning algorithms that use positive semidefinite kernels, such as support vector machines (SVM), have been analysed and justified rigorously using the theory of reproducing kernel Hilbert spaces (RKHS), yielding statements of optimality, convergence, andLp approximation bounds, e.g., see Cucker and Smale [1]. Reproducing kernel Hilbert spaces are Hilbert spaces of functions associated to a suitable kernel such that convergence with respect to the Hilbert space norm implies pointwise convergence, and in the context of approximation possess various favourable properties resulting from the Hilbert space structure. For example, under certain conditions on the kernel, every function in the Hilbert space is sufficiently differentiable, and differentiation is in fact a nonexpansive linear map with respect to the Hilbert space norm, e.g., see ([2], Subsection 2.1.3).In order to substantiate the motivation for our investigation, we briefly review previously obtained bounds on the approximation of functions as linear combinations of kernels evaluated at finitely many points. The theory of Vapnik and Chervonenkis of statistical learning theory [3–5] relies on concentration inequalities such as Hoeffding’s inequality to bound the supremum distance between expected and empirical risk. The theory considers a data space X⊆ℝd on which an unknown probability distribution P is defined, a hypothesis set H, and a loss function V:H×X⟶ℝ+, such that one wishes to find a hypothesis h∈H that minimizes the expected risk
(1)Rh≔∫XVh,xdPx.SinceP is not known in general, instead of minimizing the expected risk one usually minimizes the empirical risk
(2)R^Sh=1n∑i=1nVh,xi,over a finite set S=xii=1n⊆X of samples. Vapnik-Chervonenkis theory measures the probability with which the maximum distance between R and R^ falls below a given threshold. Recall that the Vapnik-Chervonenkis (VC) dimension of H with respect to V is the maximum cardinality of finite subsets Y⊆X that can be shattered by H, i.e. for each Y′⊆Y, there exist h∈H and α∈ℝ such that
(3)Y′=x∈Y∣Vh,x≥α,Y\Y′=x∈Y∣Vh,x<α.Thus, they prove that, assuming thatA≤Vh,x≤B for each h∈H,x∈X and the VC dimension of H is d<∞, then, for any η∈0,1,
(4)Psuph∈HRh−R^Sh≥B−Adlog2en/d−logη/4n≤η.Girosi, see [6] and ([7], Proposition 2), has used this general result to bound the uniform distance between integrals ∫Jx,yλydy and sums of the form 1/n∑i=1nJx,xi, by reinterpreting H as ℝd, V as J, and dPy as λy/λL1dy. Kon and Raphael [7] then applied this methodology to obtain uniform approximation bounds of functions in reproducing kernel Hilbert spaces. They consider two cases where the Hilbert space is dense in L2ℝd with a stronger norm ([7], Theorem 4), and where it is a closed subspace with the same norm ([7], Theorem 5). Also, Kon et al. [8] extended Girosi’s approximation estimates for functions in Sobolev spaces. While these bounds guarantee uniform convergence in probability, the approximating functions are neither orthogonal projections of f nor necessarily elements of a reproducing kernel Hilbert space and hence may not capture f exactly at xii=1n nor converge monotonically. Furthermore, the fact that the norm is not a RKHS norm means that derivatives of f may not be approximated in general, since differentiation is not bounded with respect to the uniform norm, unlike the RKHS norm associated with a continuously differentiable kernel.The purpose of this article is thus to establish sufficient conditions for convergence and approximation in the reproducing kernel Hilbert space norm. In Section3, we find probability error bounds for approximations of functions f in a separable reproducing kernel Hilbert space H with reproducing kernel K on a base space X, firstly in terms of finite linear combinations of functions of type Kx and then in terms of the projection πxn onto spanKxii=1n, for random sequences of points x=xii in the base space X. Given a probability measure P, letting PK be the measure defined by dPKx=Kx,xdPx, x∈X, we approach these problems by firstly showing the existence of the nonexpansive operator
(5)L2X;PK∋λ↦LP,Kλ≔∫XλxKxdPx∈H,where the integral exists in the Bochner sense. Using this operator, we then define a new reproducing kernel Hilbert space, denoted by HP, that is the operator range of LP,K. Our main result establishes bounds, in terms of the operator LP,K, on the probability that the Hilbert space distance between an arbitrary function f in H and linear combinations of functions of type Kxi, for xii sampled independently from P, falls below a given threshold, see Theorem 8. For sequences of points xii=1∞ constituting a so-called uniqueness set, see Subsection 3.4, the orthogonal projections πxn onto the spanKxii=1n converge in the strong operator topology to the identity operator. As an application of our main result, we show that, under the assumption that HP is dense in H, any sequence of points sampled independently from P yields a uniqueness set with probability 1.The results obtained in this article improve on the results obtained by Kon and Raphael in several senses: the convergence of approximations is in the RKHS norm, which is stronger than the uniform norm whenever the kernel is bounded; the type of convergence with respect to the pointsxii is strengthened from convergence in probability to almost certain convergence; and the separability of H then allows the result to be extended from the approximation of a single function to the simultaneous approximation of all functions in the Hilbert space. In addition, when compared to the existing methods for this kind of problems, our approach based on the operator LP,K defined at (5), that encodes the interplay between the kernel K and the probability measure P, and the associated RKHS HP, is completely new and has the potential to overcome many difficulties.These results are confined to the special case of a separable RKHSH of functions on an arbitrary set X, due to several reasons, one of them being the fact that the Bochner integral is requiring the assumption of separability, but we do not see this as a loss of generality since most of the spaces of interest for applications are separable. In the last section, we present two examples that point out the applicability, and the limitations of our results as well, the first to the uniform probability distribution on the compact interval −π,π, together with a class of bounded continuous kernels, and the second to the Hardy space H2D corresponding to the Szegö kernel which is unbounded. In each case, we can explicitly calculate the space HP, its reproducing kernel KP, and the operator LP,K.


## 2. Notation and Preliminary Results

### 2.1. Reproducing Kernel Hilbert Spaces

In this subsection, we briefly review some concepts and facts on reproducing kernel Hilbert spaces, following classical texts such as Aronszajn [9, 10] and Schwartz [11], or more modern ones such as Saitoh and Sawano ([2], Chapter 2) and Paulsen and Raghupathi [12].Throughout this article, we denote byF one of the commutative fields ℝ or ℂ. For a nonempty set X, let FX denote the set of F-valued functions on X, forming an F-vector space under pointwise addition and scalar multiplication. For each p∈X, the evaluation map at p is the linear functional
(6)evp:FX⟶F;f↦fp.The evaluation maps equipFX with the locally convex topology of pointwise convergence, which is the weakest topology on FX that renders each evaluation map continuous. Under this topology, a generalized sequence in FX converges if and only if it converges pointwise, i.e., its image under each evaluation map converges. Since each evaluation map is linear and hence the vector space operations are continuous, this renders FX into a complete Hausdorff locally convex space. With respect to this topology, if X is a topological space, a map ϕ:X⟶FX is continuous if and only if evp∘ϕ:X⟶F is continuous for all p∈X.We are interested in Hilbert spacesH⊆FX with topologies at least as strong as the topology of pointwise convergence of FX, so that the convergence of a sequence of functions in H implies that the functions also converge pointwise. When X is a finite set, FX≅Fd, where d is the number of elements of X, can itself be made into a Hilbert space with a canonical inner product f,g≔∑p∈Xfpgp¯, or in general by an inner product induced by a positive semidefinite d×d matrix. This leads to the concept of reproducing kernel Hilbert spaces.Recalling the Riesz’s Theorem of representations of bounded linear functionals on Hilbert spaces, if eachevp:H⟶F restricted to H⊆FX is continuous, for each p∈X, then there exists a unique vector Kp∈H such that evp=·,Kp. But, since each vector in H is itself a function X⟶F, these vectors altogether define a map K:X×X⟶F, Kp,q≔Kqp. Also, recall that a map K:X×X⟶F is usually called a kernel.Definition 1.
LetH⊆FX be a Hilbert space, K:X×X⟶F a kernel. For each p∈X define Kp≔K·,p∈FX. K is said to be a reproducing kernel for H, and H is then said to be a reproducing kernel Hilbert space (RKHS), if, for each p∈X, we have
(i)
Kp∈H(ii)
evp=·,Kp, that is, for every f∈H we have fp=f,Kp
The second property is referred to as the reproducing property of the kernelK.We may then summarize the last few paragraphs with the following characterization: LetH⊆FX be a Hilbert space. The following assertions are equivalent:
(i)
The canonical injectioniH:H⟶FX is continuous(ii)
For eachp∈X, the map evp:H⟶F is continuous(iii)
H admits a reproducing kernelIn that case, the reproducing kernel admitted by the Hilbert space is unique, by the uniqueness of the Riesz representativesKp of the evaluation maps. We may further apply the reproducing property to each Kq to obtain that Kp,q=Kq,Kp for each p,q∈X, yielding the following properties:
(i)
For eachp∈X, Kp,p=Kp2≥0(ii)
For eachp,q∈X, Kq,p=Kp,q¯, and(7)Kp,q2≤Kp,pKq,q.(iii)
For eachn∈ℕ, cii=1n∈Fn, pii=1n∈Xn(8)∑i=1n∑j=1nci¯cjKpi,pj=∑i=1nciKpi2≥0.The property in (7) is the analogue of the Schwarz Inequality. As a consequence of it, if Kp,p=0 for some p∈X then Kp,q=Kq,p=0 for all q∈X.For anyK:X×X⟶F, each Kp∈FX so we may define the subspace
(9)H~K≔spanKpp∈X,ofFX. If K is the reproducing kernel of a Hilbert space H, H~K is also a subspace of H and
(10)H~K⊥=f∈H∀p∈X,fp=f,Kp=0=0,therefore, H~K is a dense subspace of H, equivalently, Kpp∈X is a total set for H.The property at item (iii) is known as the positive semidefiniteness property. A positive semidefinite kernelK is called definite if Kp,p≠0 for all p∈X. Positive semidefiniteness is in fact sufficient to characterize all reproducing kernels. By the Moore-Aronszajn Theorem, for any positive semidefinite kernel K:X×X⟶F, there is a unique Hilbert space HK⊆FX with reproducing kernel K.Let us briefly recall the construction of the Hilbert spaceHK in the proof. We first render H~K into a pre-Hilbert space satisfying the reproducing property. Define on H~K the inner product
(11)∑i=1naiKpi,∑j=1mbjKqjH~K≔∑i=1n∑j=1maibj¯Kqj,pi,for any ∑i=1naiKpi,∑j=1mbjKqj∈H~K. It is proven that the definition is correct and provides indeed an inner product.LetH^K be the completion of H~K, then H^K is a Hilbert space with an isometric embedding ϕ:H~K⟶H^K whose image is dense in H^K. It is proven that this abstract completion can actually be realized in FX and that it is the RKHS with reproducing kernel K that we denote by HK.In applications, one of the most useful tools is the interplay between reproducing kernels and orthonormal bases of the underlying RKHSs. Although this fact holds in higher generality, we state it for separable Hilbert spaces since, most of the time, this is the case of interest: lettingH⊆FX be a separable RKHS, with reproducing kernel K, and let ϕnn be an orthonormal basis of H, then
(12)Kp,q=∑n=1∞ϕnpϕnq¯,p,q∈X,where the series converges absolutely pointwise.We now recall a useful result on the construction of new RKHSs and positive semidefinite kernels from existing ones. It also shows that the concept of reproducing kernel Hilbert space is actually a special case of the concept of operator range. LetH be a Hilbert space, ϕ:H⟶FX a continuous linear map. Then ϕH⊆FX with the norm
(13)fϕH≔minuHu∈H,f=ϕu,is a RKHS, unitarily isomorphic to kerϕ⊥. The kernel for ϕH is then given by the map
(14)p,q↦uq,up=evp∘ϕuq=ϕuqp,where uq∈H such that evq∘ϕ=·,uq on H. Applying this proposition to particular continuous linear maps, one obtains useful results for pullbacks, restrictions, sums, scaling, and normalizations of kernels.


### 2.2. Integration of RKHS-Valued Functions

In this article, we use integrals of Hilbert space-valued functions. We first provide fundamental definitions and properties concerning the Bochner integral, an extension of the Lebesgue integral for Banach space-valued functions, following Cohn ([13], Appendix E).LetE;· be a (real or complex) Banach space and X,Σ,μ a finite measure space. On E, we consider the Borel σ-algebra denoted by BE. A map f:X⟶E is called measurable if f−1S∈Σ for all S∈BE, and it is called strongly measurable if it is measurable and its range fX is separable. If E is a separable Banach space then the concepts coincide. Both sets of measurable functions, respectively, strongly measurable functions, are vector spaces. It is proven that a function f:X⟶B is strongly measurable if and only if there exists a sequence of simple functions ϕnn such that ϕn→nf pointwise on X. In addition, in this case, the sequence ϕnn can be chosen such that ϕnx≤fx for all x∈X.A functionf:X⟶E is Bochner integrable if it is strongly measurable and the scalar function X∋x↦fx∈ℝ is integrable. In this case, the Bochner integral of f is defined by approximation with simple functions. Bochner integrable functions share many properties with scalar-valued integrable functions, but not all. For example, the collection of all Bochner integrable functions makes a vector space, and, for any Bochner integrable function f, we have
(15)∫Xfxdμx≤∫Xfxdμx.Also, lettingL1X;μ;E denote the collection of all equivalence classes of Bochner integrable functions, identified μ-almost everywhere, this is a Banach space with norm
(16)f1≔∫Xfxdμx,f∈L1X;μ;E.In addition, the Dominated Convergence Theorem holds for the Bochner integral as well, e.g., see ([13], Theorem E.6).In this article, we will use the following result, which is a special case of a theorem of Hille, e.g., see ([14], Theorem III.2.6). In Hille’s Theorem, the linear transformation is supposed to be only closed, and, consequently, additional assumptions are needed, so we provide a proof for the special case of bounded linear operators for the reader’s convenience.Theorem 2.
LetE be a Banach space, X,μ a measure space, and f:X⟶E a Bochner integrable function. If L:E⟶F is a continuous linear transformation between Banach spaces, then L∘f:E⟶F is Bochner integrable and
(17)∫XL∘fxdμx=L∫Xfxdμx.Proof.
Sincef is Bochner integrable, there exists a sequence ϕnn of simple functions that converges pointwise to f on X and ∥ϕnx∥≤∥fx∥ for all x∈X and all n∈ℕ. Then,
(18)Lϕnx−Lfx=Lϕnx−fx≤Lϕnx−fx→n0,x∈X,hence, the sequence L∘ϕnn converges pointwise to L∘f. Also, it is easy to see that L∘ϕn is a simple function for all n∈ℕ. These show that L∘f is strongly measurable. Since Lfx≤Lfx for all x∈X and f is Bochner integrable, it follows that
(19)∫XLfxdμx≤L∫Xfxdμx<∞,hence, L∘f is Bochner integrable.On the other hand,(20)Lϕnx≤Lϕnx≤Lfx,x∈X,n∈ℕ,hence, by the Dominated Convergence Theorem for the Bochner integral, it follows that
(21)∫XLfxdμx=limn⟶∞∫XLϕnxdμx=limn⟶∞L∫Xϕnxdμx=Llimn∫Xϕnxdμx=L∫Xfxdμx.☐A direct consequence of this fact is a sufficient condition for when a pointwise integral coincides with the Bochner integral, valid not only for RKHSs but also for Banach spaces of functions on which evaluation maps at any point are continuous, e.g.,CY for some compact Hausdorff space Y.Proposition 3.
LetX,Σ,μ be a measure space, B⊆FX a Banach space of functions on X, such that all evaluation maps on B are continuous. Let λ:X×X⟶F be such that for each q∈X we have λq≔λ·,q∈B.If, for eachq∈X, the mapX∋q↦λq∈Bis Bochner integrable, then the scalar mapX∋q↦λp,q∈Fis integrable, for each fixedp∈X.Moreover, in that case, the pointwise integral mapX∋p↦∫Xλp,qdμqlies inBand coincides with the Bochner integral∫Xλqdμq.Proof.
Since, for eachq∈X, the map X∋q↦ϕq≔λ·,q∈B is Bochner integrable, and taking into account that, for all p∈X, the linear functional evp is continuous, by Theorem 2, we have
(22)evp∫Xϕqdμq=∫Xevp∘ϕqdμq.
Sinceevp∘ϕq=λp,q for all p,q∈X, this means that the scalar map X∋q↦λp,q∈F is integrable, for each fixed p∈X, and
(23)evp∫Xϕqdμq=∫Xλp,qdμq,p∈X,hence, the pointwise integral map X∋p↦∫Xλp,qdμq lies in B and coincides with the Bochner integral ∫Xλqdμq.☐


## 2.1. Reproducing Kernel Hilbert Spaces

In this subsection, we briefly review some concepts and facts on reproducing kernel Hilbert spaces, following classical texts such as Aronszajn [9, 10] and Schwartz [11], or more modern ones such as Saitoh and Sawano ([2], Chapter 2) and Paulsen and Raghupathi [12].Throughout this article, we denote byF one of the commutative fields ℝ or ℂ. For a nonempty set X, let FX denote the set of F-valued functions on X, forming an F-vector space under pointwise addition and scalar multiplication. For each p∈X, the evaluation map at p is the linear functional
(6)evp:FX⟶F;f↦fp.The evaluation maps equipFX with the locally convex topology of pointwise convergence, which is the weakest topology on FX that renders each evaluation map continuous. Under this topology, a generalized sequence in FX converges if and only if it converges pointwise, i.e., its image under each evaluation map converges. Since each evaluation map is linear and hence the vector space operations are continuous, this renders FX into a complete Hausdorff locally convex space. With respect to this topology, if X is a topological space, a map ϕ:X⟶FX is continuous if and only if evp∘ϕ:X⟶F is continuous for all p∈X.We are interested in Hilbert spacesH⊆FX with topologies at least as strong as the topology of pointwise convergence of FX, so that the convergence of a sequence of functions in H implies that the functions also converge pointwise. When X is a finite set, FX≅Fd, where d is the number of elements of X, can itself be made into a Hilbert space with a canonical inner product f,g≔∑p∈Xfpgp¯, or in general by an inner product induced by a positive semidefinite d×d matrix. This leads to the concept of reproducing kernel Hilbert spaces.Recalling the Riesz’s Theorem of representations of bounded linear functionals on Hilbert spaces, if eachevp:H⟶F restricted to H⊆FX is continuous, for each p∈X, then there exists a unique vector Kp∈H such that evp=·,Kp. But, since each vector in H is itself a function X⟶F, these vectors altogether define a map K:X×X⟶F, Kp,q≔Kqp. Also, recall that a map K:X×X⟶F is usually called a kernel.Definition 1.
LetH⊆FX be a Hilbert space, K:X×X⟶F a kernel. For each p∈X define Kp≔K·,p∈FX. K is said to be a reproducing kernel for H, and H is then said to be a reproducing kernel Hilbert space (RKHS), if, for each p∈X, we have
(i)
Kp∈H(ii)
evp=·,Kp, that is, for every f∈H we have fp=f,Kp
The second property is referred to as the reproducing property of the kernelK.We may then summarize the last few paragraphs with the following characterization: LetH⊆FX be a Hilbert space. The following assertions are equivalent:
(i)
The canonical injectioniH:H⟶FX is continuous(ii)
For eachp∈X, the map evp:H⟶F is continuous(iii)
H admits a reproducing kernelIn that case, the reproducing kernel admitted by the Hilbert space is unique, by the uniqueness of the Riesz representativesKp of the evaluation maps. We may further apply the reproducing property to each Kq to obtain that Kp,q=Kq,Kp for each p,q∈X, yielding the following properties:
(i)
For eachp∈X, Kp,p=Kp2≥0(ii)
For eachp,q∈X, Kq,p=Kp,q¯, and(7)Kp,q2≤Kp,pKq,q.(iii)
For eachn∈ℕ, cii=1n∈Fn, pii=1n∈Xn(8)∑i=1n∑j=1nci¯cjKpi,pj=∑i=1nciKpi2≥0.The property in (7) is the analogue of the Schwarz Inequality. As a consequence of it, if Kp,p=0 for some p∈X then Kp,q=Kq,p=0 for all q∈X.For anyK:X×X⟶F, each Kp∈FX so we may define the subspace
(9)H~K≔spanKpp∈X,ofFX. If K is the reproducing kernel of a Hilbert space H, H~K is also a subspace of H and
(10)H~K⊥=f∈H∀p∈X,fp=f,Kp=0=0,therefore, H~K is a dense subspace of H, equivalently, Kpp∈X is a total set for H.The property at item (iii) is known as the positive semidefiniteness property. A positive semidefinite kernelK is called definite if Kp,p≠0 for all p∈X. Positive semidefiniteness is in fact sufficient to characterize all reproducing kernels. By the Moore-Aronszajn Theorem, for any positive semidefinite kernel K:X×X⟶F, there is a unique Hilbert space HK⊆FX with reproducing kernel K.Let us briefly recall the construction of the Hilbert spaceHK in the proof. We first render H~K into a pre-Hilbert space satisfying the reproducing property. Define on H~K the inner product
(11)∑i=1naiKpi,∑j=1mbjKqjH~K≔∑i=1n∑j=1maibj¯Kqj,pi,for any ∑i=1naiKpi,∑j=1mbjKqj∈H~K. It is proven that the definition is correct and provides indeed an inner product.LetH^K be the completion of H~K, then H^K is a Hilbert space with an isometric embedding ϕ:H~K⟶H^K whose image is dense in H^K. It is proven that this abstract completion can actually be realized in FX and that it is the RKHS with reproducing kernel K that we denote by HK.In applications, one of the most useful tools is the interplay between reproducing kernels and orthonormal bases of the underlying RKHSs. Although this fact holds in higher generality, we state it for separable Hilbert spaces since, most of the time, this is the case of interest: lettingH⊆FX be a separable RKHS, with reproducing kernel K, and let ϕnn be an orthonormal basis of H, then
(12)Kp,q=∑n=1∞ϕnpϕnq¯,p,q∈X,where the series converges absolutely pointwise.We now recall a useful result on the construction of new RKHSs and positive semidefinite kernels from existing ones. It also shows that the concept of reproducing kernel Hilbert space is actually a special case of the concept of operator range. LetH be a Hilbert space, ϕ:H⟶FX a continuous linear map. Then ϕH⊆FX with the norm
(13)fϕH≔minuHu∈H,f=ϕu,is a RKHS, unitarily isomorphic to kerϕ⊥. The kernel for ϕH is then given by the map
(14)p,q↦uq,up=evp∘ϕuq=ϕuqp,where uq∈H such that evq∘ϕ=·,uq on H. Applying this proposition to particular continuous linear maps, one obtains useful results for pullbacks, restrictions, sums, scaling, and normalizations of kernels.


## 2.2. Integration of RKHS-Valued Functions

In this article, we use integrals of Hilbert space-valued functions. We first provide fundamental definitions and properties concerning the Bochner integral, an extension of the Lebesgue integral for Banach space-valued functions, following Cohn ([13], Appendix E).LetE;· be a (real or complex) Banach space and X,Σ,μ a finite measure space. On E, we consider the Borel σ-algebra denoted by BE. A map f:X⟶E is called measurable if f−1S∈Σ for all S∈BE, and it is called strongly measurable if it is measurable and its range fX is separable. If E is a separable Banach space then the concepts coincide. Both sets of measurable functions, respectively, strongly measurable functions, are vector spaces. It is proven that a function f:X⟶B is strongly measurable if and only if there exists a sequence of simple functions ϕnn such that ϕn→nf pointwise on X. In addition, in this case, the sequence ϕnn can be chosen such that ϕnx≤fx for all x∈X.A functionf:X⟶E is Bochner integrable if it is strongly measurable and the scalar function X∋x↦fx∈ℝ is integrable. In this case, the Bochner integral of f is defined by approximation with simple functions. Bochner integrable functions share many properties with scalar-valued integrable functions, but not all. For example, the collection of all Bochner integrable functions makes a vector space, and, for any Bochner integrable function f, we have
(15)∫Xfxdμx≤∫Xfxdμx.Also, lettingL1X;μ;E denote the collection of all equivalence classes of Bochner integrable functions, identified μ-almost everywhere, this is a Banach space with norm
(16)f1≔∫Xfxdμx,f∈L1X;μ;E.In addition, the Dominated Convergence Theorem holds for the Bochner integral as well, e.g., see ([13], Theorem E.6).In this article, we will use the following result, which is a special case of a theorem of Hille, e.g., see ([14], Theorem III.2.6). In Hille’s Theorem, the linear transformation is supposed to be only closed, and, consequently, additional assumptions are needed, so we provide a proof for the special case of bounded linear operators for the reader’s convenience.Theorem 2.
LetE be a Banach space, X,μ a measure space, and f:X⟶E a Bochner integrable function. If L:E⟶F is a continuous linear transformation between Banach spaces, then L∘f:E⟶F is Bochner integrable and
(17)∫XL∘fxdμx=L∫Xfxdμx.Proof.
Sincef is Bochner integrable, there exists a sequence ϕnn of simple functions that converges pointwise to f on X and ∥ϕnx∥≤∥fx∥ for all x∈X and all n∈ℕ. Then,
(18)Lϕnx−Lfx=Lϕnx−fx≤Lϕnx−fx→n0,x∈X,hence, the sequence L∘ϕnn converges pointwise to L∘f. Also, it is easy to see that L∘ϕn is a simple function for all n∈ℕ. These show that L∘f is strongly measurable. Since Lfx≤Lfx for all x∈X and f is Bochner integrable, it follows that
(19)∫XLfxdμx≤L∫Xfxdμx<∞,hence, L∘f is Bochner integrable.On the other hand,(20)Lϕnx≤Lϕnx≤Lfx,x∈X,n∈ℕ,hence, by the Dominated Convergence Theorem for the Bochner integral, it follows that
(21)∫XLfxdμx=limn⟶∞∫XLϕnxdμx=limn⟶∞L∫Xϕnxdμx=Llimn∫Xϕnxdμx=L∫Xfxdμx.☐A direct consequence of this fact is a sufficient condition for when a pointwise integral coincides with the Bochner integral, valid not only for RKHSs but also for Banach spaces of functions on which evaluation maps at any point are continuous, e.g.,CY for some compact Hausdorff space Y.Proposition 3.
LetX,Σ,μ be a measure space, B⊆FX a Banach space of functions on X, such that all evaluation maps on B are continuous. Let λ:X×X⟶F be such that for each q∈X we have λq≔λ·,q∈B.If, for eachq∈X, the mapX∋q↦λq∈Bis Bochner integrable, then the scalar mapX∋q↦λp,q∈Fis integrable, for each fixedp∈X.Moreover, in that case, the pointwise integral mapX∋p↦∫Xλp,qdμqlies inBand coincides with the Bochner integral∫Xλqdμq.Proof.
Since, for eachq∈X, the map X∋q↦ϕq≔λ·,q∈B is Bochner integrable, and taking into account that, for all p∈X, the linear functional evp is continuous, by Theorem 2, we have
(22)evp∫Xϕqdμq=∫Xevp∘ϕqdμq.
Sinceevp∘ϕq=λp,q for all p,q∈X, this means that the scalar map X∋q↦λp,q∈F is integrable, for each fixed p∈X, and
(23)evp∫Xϕqdμq=∫Xλp,qdμq,p∈X,hence, the pointwise integral map X∋p↦∫Xλp,qdμq lies in B and coincides with the Bochner integral ∫Xλqdμq.☐


## 3. Main Results

Throughout this section, we consider a probability measure spaceX;Σ;P and a RKHS H;·,· in FX, with norm denoted by ·H, such that its reproducing kernel K is measurable. In addition, throughout this section, the reproducing kernel Hilbert space H is supposed to be separable.


### 3.1. The Reproducing Kernel Hilbert SpaceHP

On the measurable spaceX;Σ, we define the measure PK by
(24)dPKx=Kx,xdPx,x∈X,that is, PK is the absolutely continuous measure with respect to P such that the function X∋x↦Kx,x is the Radon-Nikodym derivative of PK with respect to P.With respect to the measure spaceX;Σ;PK, we consider the Hilbert space L2X;PK. Our approach is based on the following natural bounded linear operator mapping L2X;PK to H.Proposition 4.
With notation and assumptions as before, letλ:X⟶F be a measurable function such that the integral ∫Xλx2dPKx is finite. Then, the Bochner integral
(25)∫XλxKxdPx,exists in H.In addition, the mapping(26)L2X;PK∋λ↦LP,Kλ≔∫XλxKxdPx∈H,is a nonexpansive, hence, bounded, linear operator.Proof.
By assumptions, the mapX∋x↦λxKx∈H is measurable, and, since H is separable, it follows that this map is actually strongly measurable. Letting ·H denote the norm on H and using the assumption that ∫Xλx2Kx,xdPx is finite, we have
(27)∫XλxKxH2dPx=∫Xλx2Kx,xdPx<∞,
hence, by the Schwarz Inequality and taking into account thatP is a probability measure, we have
(28)∫XλxKxHdPx≤∫XλxKxH2dPx<∞.By Theorem2, this implies that the Bochner integral ∫XλxKxdPx exists in H. Consequently, the mapping LP,K as in (26) is correctly defined, and it is clear that it is a linear transformation.For arbitraryλ∈L2X;PK, by the triangle inequality for the Bochner integral (15), we then have
(29)∫XλxKxdPxH2≤∫XλxKxdPx2=∫XλxKx,x12dPx2, and applying the Schwarz Inequality for the integral and taking into account that P is a probability measure
(30)≤∫Xλx2Kx,xdPx=λL2X;PK2,hence, LP,K:L2X;PK⟶H is a nonexpansive linear operator.Using the bounded linear operatorLP,K defined as in (26), let us denote its range by
(31)HP≔LP,KL2X;PK,which is a subspace of the RKHSH.Proposition 5.
HP is a RKHS contained in H, hence, in FX, and its reproducing kernel KP is
(32)KPx,y=∫XKx,zKz,yKz,zdPz,x,y∈X,where whenever Kz,z=0, by convention we define Kx,zKz,y/Kz,z=0 for all x,y∈X.Proof.
SinceL2X;PK is a Hilbert space and LP,K is a bounded linear map, by (13) it follows that HP is a RKHS in FX, isometrically isomorphic to the orthogonal complement of kerLP,K⊆L2X;PK, and its norm is given by
(33)gHP≔minλL2X;PKLP,Kλ=g,g∈HP.Let(34)X0≔x∈X∣Kx,x=0,and let us define ux:X⟶F by
(35)uxy≔Ky,xKy,y,y∈X\X0,0,y∈X0.From the Schwarz Inequality for the kernelK, it follows that if x∈X0 then Kx,y=0 for all y∈X. This shows that ux=0 for all x∈X0.For eachx∈X, by the Schwarz inequality and the fact that P is a probability measure, we have
(36)∫Xuxy2Ky,ydPy=∫X\X0Ky,x2Ky,ydPy≤∫X\X0Ky,yKx,xKy,ydPy=Kx,xPX\X0<∞,hence,ux∈L2X,PK. Then, taking into account that Kx,y=0 for all y∈X0 and all x∈X, it follows that, for each λ∈L2X,PK and x∈X, we have
(37)LP,Kλx=∫XλyKx,ydPy=∫X\X0λyKx,ydPy=∫X\X0λyKy,xKy,y¯Ky,ydPy=∫Xλyuxy¯Ky,ydPy=λ,uxL2X,PK.In conclusion,ux is exactly the representative for the functional evxLP,K so, by (13) the kernel of HP is
(38)KPx,y=uy,uxL2X,PK=∫Xuyzuxz¯Kz,zdPz=∫X\X0uyzuxz¯Kz,zdPz,and, using the convention that Kx,zKz,y/Kz,z=0 whenever Kz,z=0 and for arbitrary x,y∈X,
(39)=∫XKx,zKz,y/Kz,zdPz.☐One of the main results of this article, see Theorem11, assumes that the space HP is dense in H. The next proposition provides sufficient conditions for this.Proposition 6.
LetX be a topological space, P a Borel probability measure on X, H⊆FX a RKHS with measurable kernel K, and let PK, LP,K, and HP defined as in (24), (26), and (31), respectively.Suppose thatKis continuous onX, thatH⊆L2X;PK, and thatPis strictly positive on any nonempty open subset ofX. Then, HPis dense inH.Proof.
The assertion is clearly equivalent with showing that the orthogonal complement ofHP in H is the null space. To this end, let f∈H, f⊥HP. That is, for each λ∈L2X;PK, we have
(40)f,LP,KλH=f,∫XλxKxdPx=0.Then noting the fact that∫λxKxdPx is a Bochner integral, and hence, by Theorem 2, it commutes with inner products,
(41)0=f,∫XλxKxdPx=∫Xλx¯f,KxdPx=∫λx¯fxdPx.By assumption,f∈H⊆L2X;PK, so we can take λ=f to obtain
(42)∫fx2dPx=∫Xfx¯fxdPx=0.This implies thatf=0P-almost everywhere, i.e., the set f−1F\0 has zero P measure.SinceK is continuous by assumption, by the Theorem 2.3 in ([2], Section 2.1.3), each f∈H is continuous hence f−1F\0 is an open subset of X. But, since P is assumed strictly positive on any nonempty open set, it follows that f−1F\0 must be empty, hence, f=0 identically.☐


### 3.2. Probability Error Bounds of Approximation

The first step in our enterprise is to find error bounds for approximations of functions in the reproducing kernel Hilbert spaceH in terms of distributional finite linear combinations of functions of type Kx. To do that, we use the celebrated Markov-Bienaymé-Chebyshev Inequality on the concentration of probability measures to obtain regions of large measure with small approximation error, in terms of the Hilbert space norm and not simply the uniform norm.Theorem 7.
(Markov-Bienaymé- Chebyshev’s Inequality) LetX;Σ;P be a probability space, B;· a Banach space, and let f,g:X⟶B be two Borel measurable functions. Then, for any δ>0, we have
(43)Px∈X∣gx≥δ≤1δ2∫Xgx2dPx.The classical Bienaymé-Chebyshev Inequality(44)Px∈X∣fx−Ef≥kσ≤1k2,is obtained from (43) applied for B=ℝ, gx=fx−Ef, and δ=kσ, for k>0, where Ef=∫Xfxdx is the expected value of the random variable f and σ2=Ef−Ef2=Ef2−Ef2>0 is the variance of f.Theorem 8.
With notation and assumptions as before, letλ∈L2X;PK and f∈H. For each n∈ℕ and δ>0, consider the set
(45)An,δ≔x1,⋯,xn∈Xn∣f−1n∑i=1nλxiKxiH≥δ.Then, lettingPndenote the product probability measure onXnand defining the bounded linear operatorLP,Kas in (26), we have(46)PnAn,δ≤1δ2f−LP,KλH2+1nδ2λL2X;PK2−LP,KλH2.Proof.
By Proposition4, the Bochner integral ∫XλxKxdPx exists in H and the linear operator LP,K is well-defined and bounded. In order to simplify the notation, considering g:Xn⟶H the function defined by
(47)gx1,⋯,xn=f−1n∑i=1nλxiKxi,x1,⋯,xn∈Xn,observe that g is measurable and for each δ>0, we have
(48)An,δ=x1,⋯,xn∈Xn∣gx1,⋯,xnH≥δ.Then, we have(49)gx1,⋯,xn2=f−1n∑i=1nλxiKxiH2=fH2−2n∑i=1nRef,λxiKxi+1n2∑i=1n∑j=1nλxiKxi,λxjKxj.SincePn is a probability measure, we have
(50)∫XnfH2dPnx1,⋯,xn=fH2.On the other hand, by Fubini’s theorem and the fact that the Bochner integral commutes with continuous linear operations, see Theorem2, we have
(51)∫XnRef,λxiKxidPnx1,⋯,xn=Ref,∫XnλxiKxidPnx1,⋯,xn=Ref,∫XλxKxdPx=Ref,LP,Kλ.Also, for eachi=1,⋯,n,
(52)∫XnλxiKxi,λxiKxidPnx1,⋯,xn=∫Xnλxi2Kxi,xidPnx1,⋯,xn=∫Xλx2Kx,xdPx,and, for each i,j=1,⋯,n,i≠j,
(53)∫XnλxiKxi,λxjKxjdPnx1,⋯,xn=∫XλxiKxi,∫XλxjKxjdPxjdPxi=∫XλxKxdPx,∫XλxKxdPx=∫XλxKxdPxH2.Integrating both sides of (49) and using all the previous equalities, we therefore have
(54)∫Xngx1,⋯,xnH2dPnx1,⋯,xn=fH2−2n∑i=1nRef,∫XλxKxdPx+1n2∑i=1n∑i≠j=1n∫XλxKxdPxH2+1n2∑i=1n∫Xλx2Kx,xdPx=fH2−2Ref,∫XλxKxdPx+n−1n∫XλxKxdPxH2+1n∫Xλx2Kx,xdPx=f−∫XλxKxdPxH2+1n∫Xλx2Kx,xdPx−∫XλxKxdPxH2=f−LP,KλH2+1n∫Xλx2Kx,xdPx−LP,KλH2.Finally, in view of the Markov-Bienaymé-Chebyshev Inequality as in (43), when X is replaced by Xn and P by Pn, and taking into account the previous equality and (48), we get
(55)PnAn,δ≤1δ2∫Xngx1,⋯,xnH2dPnx1,⋯,xn=1δ2f−LP,KλH2+1nδ2λL2X;PK2−LP,KλH2,which is the required inequality.☐


### 3.3. Convergence in Probability

As with the special case of kernel embeddings, for whichλ=1, see Smola et al. [15], we may use the bound in Theorem 8 to obtain a statement of convergence in probability.With notation and assumptions as before, givenf∈H and fixed xii=1N∈X, the problem of finding the optimal ωiNfi=1N∈FN to minimize ∥f−∑i=1NωiNfKxi∥H is straightforward: ∑i=1NωiNfKxi is the orthogonal projection of f to spanKxii=1N.We may assume without loss of generality thatKxii=1N are linearly independent, by removing points as necessary without affecting spanKxii=1N (or losing any information about f, since ∑i=1NciKxi=0 implies ∑i=1Nci¯fxi=0 by the reproducing property). According to Körezlioglu [16], if xii=1N∈X is a sampling such that Kxii=1N are linearly independent and considering the finite-dimensional subspace HxN≔spanKxii=1N of H, then the orthogonal projection πxN of H onto HxN is given by
(56)πxNf=∑i=1NωiπfKxi≔∑i=1N∑j=1NfxjΓjiNKxi=∑i=1N∑j=1Nf,KxjΓjiNKxifor anyf∈H, where ΓN∈MNF is the inverse of the Gram matrix GN≔Kxj,xii,j=1N=Kxi,Kxji,j=1N of x1,⋯,xN.More generally, ifKxii=1N are not linearly independent, for any subset s=xijj=1K such that Kxijj=1K form a basis for HxN, we have HxN=HsK and
(57)πxN=πsK=∑j=1K∑k=1K·,KxikΓkjKKxij.Note that, in general,ωiπ is not simply a multiple of fxi, hence, setting ωi≔Vifxi for any fixed Vi will not yield the best possible approximation. However, with such coefficients dependent only on xi, it will be easier to bound f−∑iωiKxi across different xiis than f−πxNf. Then, any upper bound on f−∑iωiKxi for some fixed ωii will also be an upper bound on f−πxNf.Theorem 9.
(Convergence in Probability of Projections) LetX, P, K, and H be as in Theorem 8. For each sequence x=xii∈Xℕ and each n∈ℕ, let πxn denote the orthogonal projection of H onto spanKxii=1n. Let f∈H and, for each δ>0 and n∈ℕ, define
(58)Bn,δ≔x1,⋯,xn∈Xn∣f−πxnfH≥δ.
Then, for eachδ>0(59)limsupn⟶∞PnBn,δ≤1δ2dHf,HP2,
wheredHf,HP=infg∈HP∥f−g∥.In particular, iffbelongs toHP¯H, the closure ofHPwith respect to the topology ofH, then(60)limn⟶∞PnBn,δ=0.Proof.
Letλ∈L2X,PK and fix δ>0, arbitrary. Then
(61)f−πxnfH≤f−1n∑i=1nλxiKxiH,hence, with notation as in (45), we have Bn,δ⊆An,δ. By Theorem 8, this implies
(62)PnBn,δ≤1δ2f−LP,KλH2+1nδ2λL2X,PK2−LP,KλH2.
Therefore,(63)limsupn⟶∞PnBn,δ≤limsupn⟶∞1δ2f−LP,KλH2+1nδ2λL2X,PK2−LP,KλH2=1δ2f−LP,KλH2.
Thus, since the left-hand side is independent ofλ,
(64)limsupn⟶∞PnBn,δ≤infλ∈L2X,PK1δ2f−LP,KλH2=1δ2dHf,HP2.
In particular, iff belongs to HP¯H, then dHf,HP=0.☐


### 3.4. Uniqueness Sets and Almost Certain Convergence of Projections

With notation and assumptions as before, we now follow ([2], Subsection 2.4.4) in recalling the strong convergence of πxN to the identity map as N⟶∞ for appropriately chosen xii=1∞. Since H is separable, there exists a countable subset of Kpp∈X which is total in H; thus, there exists a countable set F⊆X such that spanKxx∈F is dense in H. This motivates the following definition: a countable subset xii=1∞ of X is called a uniqueness set for H if Kxii=1∞ is a total set in H, that is, if f∈H such that fxi=0 for all i∈ℕ implies f=0. Then, the so-called Ultimate Realization of RKHSs, compare ([2], Theorem 2.33), reads as follows: if xii=1∞ is a uniqueness set such that Kxii=1∞ is linearly independent, GN is the Gram matrix for xii=1N, ΓN=GN−1, then for each f∈H,
(65)limN⟶∞πxNf=limN⟶∞∑i=1N∑j=1NfxiΓijNKxj=f,under the topology ofH, with distance decreasing monotonically. Consequently,
(66)f,g=limN⟶∞∑i=1N∑j=1NfxiΓijNgxj¯,for f,g∈H, and
(67)fx=f,Kx=limN⟶∞∑i=1N∑j=1NfxiΓijNKx,xj,for f∈H,x∈X. This has implications in interpolation theory, e.g., see ([2], Corollary~2.6).Coming back to our problem, by noting thatf−πxnf, unlike f−1/n∑i=1nλxiKxi, is monotonically nonincreasing with respect to n, our next goal is to strengthen Theorem 9 to almost certain convergence after passing to a single measure space. First, recall that, e.g., see ([13], Proposition 10.6.1), the countably infinite product space Xℕ equipped with the smallest σ-algebra rendering each projection map Xi:Xℕ⟶X measurable admits a unique probability measure Pℕ such that the projection maps are independent random variables with distribution P.Lemma 10.
LetX, P, K, and H be as in Theorem 8 and f∈H. For each δ>0 define
(68)Sn,δ≔x=xkk=1∞∈Xℕ∣f−πxnfH≥δ,n∈ℕ,and
(69)Sδ≔x=xkk=1∞∈Xℕ∣∀N∈ℕ,∃n≥N,f−πxnfH≥δ=⋂N∈ℕ⋃n≥NSn,δ.
Then,(70)PℕSδ≤1δ2dHf,HP2,and, consequently, if f∈HP¯H, then
(71)PℕSδ=0.Proof.
Observe that for eachn,m∈ℕ such that n>m, f−πxnfH≤f−πxmfH, for each x∈Xℕ, and hence Sn,δ⊆Sm,δ for each δ>0. Then,
(72)Sδ=⋂N∈ℕ⋃n≥NSn,δ=⋂N∈ℕSN,δ,
hence, for anyλ∈L2X,PK,
(73)PℕSδ≤infN∈ℕPℕSN,δ≤1δ2f−LP,KλH2,
sincePℕ is monotone and Sδ⊆SN,δ for all N∈ℕ.☐The main result of this subsection is the following.Theorem 11.
(Almost Certain Convergence of Projections) LetX,P,K,H be as in Theorem 8 and suppose HP is dense in H. Then, for each f∈H,
(74)Pℕx∈Xℕ∣πxnf→nf=1,hence,
(75)Pℕx∈Xℕ∀f∈H,πxnf→nf=1.Proof.
Letf∈H. With the same sets Sδ defined in (69),
(76)x∈Xℕ∣πxnf↛f=x∈Xℕ∣∃δ>0,∀N∈ℕ,∃n≥N,f−πxnfH≥δ=⋃δ>0Sδ.Observe further thatSδ⊆Sδ′ whenever δ>δ′, and for each δ>0 there exists m∈ℕ such that δ>1/m, so that
(77)x∈Xℕπxnf↛nf=⋃0<δ≤1Sδ=⋃m∈ℕS1/m,thus, taking into account thatHP is dense in H and using Lemma 10, we get
(78)Pℕx∈Xℕπxnf↛nf≤∑m∈ℕPℕS1/m=∑m∈ℕ0=0.SinceH is separable let D be a countable dense subset of H. Since each πxn is a continuous linear operator with operator norm 1, πxnf⟶nf for all f∈H iff πxnf⟶nf for all f∈D. Thus, by the countable subadditivity of Pℕ,
(79)Pℕx∈Xℕ∃f∈H,πxnf↛nf=Pℕx∈Xℕ∃f∈D,πxnf↛nf=Pℕ⋃f∈Dx∈Xℕπxnf↛nf≤∑f∈DPℕx∈Xℕπxnf↛nf=0.☐In summary, for a given probability measureP under the assumption that it renders the space HP, the image of LP,K, dense in H, a sequence of points sampled independently from P yields a uniqueness set with probability 1. Proposition shows a sufficient condition, valid for many applications, when this assumption holds.


## 3.1. The Reproducing Kernel Hilbert SpaceHP

On the measurable spaceX;Σ, we define the measure PK by
(24)dPKx=Kx,xdPx,x∈X,that is, PK is the absolutely continuous measure with respect to P such that the function X∋x↦Kx,x is the Radon-Nikodym derivative of PK with respect to P.With respect to the measure spaceX;Σ;PK, we consider the Hilbert space L2X;PK. Our approach is based on the following natural bounded linear operator mapping L2X;PK to H.Proposition 4.
With notation and assumptions as before, letλ:X⟶F be a measurable function such that the integral ∫Xλx2dPKx is finite. Then, the Bochner integral
(25)∫XλxKxdPx,exists in H.In addition, the mapping(26)L2X;PK∋λ↦LP,Kλ≔∫XλxKxdPx∈H,is a nonexpansive, hence, bounded, linear operator.Proof.
By assumptions, the mapX∋x↦λxKx∈H is measurable, and, since H is separable, it follows that this map is actually strongly measurable. Letting ·H denote the norm on H and using the assumption that ∫Xλx2Kx,xdPx is finite, we have
(27)∫XλxKxH2dPx=∫Xλx2Kx,xdPx<∞,
hence, by the Schwarz Inequality and taking into account thatP is a probability measure, we have
(28)∫XλxKxHdPx≤∫XλxKxH2dPx<∞.By Theorem2, this implies that the Bochner integral ∫XλxKxdPx exists in H. Consequently, the mapping LP,K as in (26) is correctly defined, and it is clear that it is a linear transformation.For arbitraryλ∈L2X;PK, by the triangle inequality for the Bochner integral (15), we then have
(29)∫XλxKxdPxH2≤∫XλxKxdPx2=∫XλxKx,x12dPx2, and applying the Schwarz Inequality for the integral and taking into account that P is a probability measure
(30)≤∫Xλx2Kx,xdPx=λL2X;PK2,hence, LP,K:L2X;PK⟶H is a nonexpansive linear operator.Using the bounded linear operatorLP,K defined as in (26), let us denote its range by
(31)HP≔LP,KL2X;PK,which is a subspace of the RKHSH.Proposition 5.
HP is a RKHS contained in H, hence, in FX, and its reproducing kernel KP is
(32)KPx,y=∫XKx,zKz,yKz,zdPz,x,y∈X,where whenever Kz,z=0, by convention we define Kx,zKz,y/Kz,z=0 for all x,y∈X.Proof.
SinceL2X;PK is a Hilbert space and LP,K is a bounded linear map, by (13) it follows that HP is a RKHS in FX, isometrically isomorphic to the orthogonal complement of kerLP,K⊆L2X;PK, and its norm is given by
(33)gHP≔minλL2X;PKLP,Kλ=g,g∈HP.Let(34)X0≔x∈X∣Kx,x=0,and let us define ux:X⟶F by
(35)uxy≔Ky,xKy,y,y∈X\X0,0,y∈X0.From the Schwarz Inequality for the kernelK, it follows that if x∈X0 then Kx,y=0 for all y∈X. This shows that ux=0 for all x∈X0.For eachx∈X, by the Schwarz inequality and the fact that P is a probability measure, we have
(36)∫Xuxy2Ky,ydPy=∫X\X0Ky,x2Ky,ydPy≤∫X\X0Ky,yKx,xKy,ydPy=Kx,xPX\X0<∞,hence,ux∈L2X,PK. Then, taking into account that Kx,y=0 for all y∈X0 and all x∈X, it follows that, for each λ∈L2X,PK and x∈X, we have
(37)LP,Kλx=∫XλyKx,ydPy=∫X\X0λyKx,ydPy=∫X\X0λyKy,xKy,y¯Ky,ydPy=∫Xλyuxy¯Ky,ydPy=λ,uxL2X,PK.In conclusion,ux is exactly the representative for the functional evxLP,K so, by (13) the kernel of HP is
(38)KPx,y=uy,uxL2X,PK=∫Xuyzuxz¯Kz,zdPz=∫X\X0uyzuxz¯Kz,zdPz,and, using the convention that Kx,zKz,y/Kz,z=0 whenever Kz,z=0 and for arbitrary x,y∈X,
(39)=∫XKx,zKz,y/Kz,zdPz.☐One of the main results of this article, see Theorem11, assumes that the space HP is dense in H. The next proposition provides sufficient conditions for this.Proposition 6.
LetX be a topological space, P a Borel probability measure on X, H⊆FX a RKHS with measurable kernel K, and let PK, LP,K, and HP defined as in (24), (26), and (31), respectively.Suppose thatKis continuous onX, thatH⊆L2X;PK, and thatPis strictly positive on any nonempty open subset ofX. Then, HPis dense inH.Proof.
The assertion is clearly equivalent with showing that the orthogonal complement ofHP in H is the null space. To this end, let f∈H, f⊥HP. That is, for each λ∈L2X;PK, we have
(40)f,LP,KλH=f,∫XλxKxdPx=0.Then noting the fact that∫λxKxdPx is a Bochner integral, and hence, by Theorem 2, it commutes with inner products,
(41)0=f,∫XλxKxdPx=∫Xλx¯f,KxdPx=∫λx¯fxdPx.By assumption,f∈H⊆L2X;PK, so we can take λ=f to obtain
(42)∫fx2dPx=∫Xfx¯fxdPx=0.This implies thatf=0P-almost everywhere, i.e., the set f−1F\0 has zero P measure.SinceK is continuous by assumption, by the Theorem 2.3 in ([2], Section 2.1.3), each f∈H is continuous hence f−1F\0 is an open subset of X. But, since P is assumed strictly positive on any nonempty open set, it follows that f−1F\0 must be empty, hence, f=0 identically.☐


## 3.2. Probability Error Bounds of Approximation

The first step in our enterprise is to find error bounds for approximations of functions in the reproducing kernel Hilbert spaceH in terms of distributional finite linear combinations of functions of type Kx. To do that, we use the celebrated Markov-Bienaymé-Chebyshev Inequality on the concentration of probability measures to obtain regions of large measure with small approximation error, in terms of the Hilbert space norm and not simply the uniform norm.Theorem 7.
(Markov-Bienaymé- Chebyshev’s Inequality) LetX;Σ;P be a probability space, B;· a Banach space, and let f,g:X⟶B be two Borel measurable functions. Then, for any δ>0, we have
(43)Px∈X∣gx≥δ≤1δ2∫Xgx2dPx.The classical Bienaymé-Chebyshev Inequality(44)Px∈X∣fx−Ef≥kσ≤1k2,is obtained from (43) applied for B=ℝ, gx=fx−Ef, and δ=kσ, for k>0, where Ef=∫Xfxdx is the expected value of the random variable f and σ2=Ef−Ef2=Ef2−Ef2>0 is the variance of f.Theorem 8.
With notation and assumptions as before, letλ∈L2X;PK and f∈H. For each n∈ℕ and δ>0, consider the set
(45)An,δ≔x1,⋯,xn∈Xn∣f−1n∑i=1nλxiKxiH≥δ.Then, lettingPndenote the product probability measure onXnand defining the bounded linear operatorLP,Kas in (26), we have(46)PnAn,δ≤1δ2f−LP,KλH2+1nδ2λL2X;PK2−LP,KλH2.Proof.
By Proposition4, the Bochner integral ∫XλxKxdPx exists in H and the linear operator LP,K is well-defined and bounded. In order to simplify the notation, considering g:Xn⟶H the function defined by
(47)gx1,⋯,xn=f−1n∑i=1nλxiKxi,x1,⋯,xn∈Xn,observe that g is measurable and for each δ>0, we have
(48)An,δ=x1,⋯,xn∈Xn∣gx1,⋯,xnH≥δ.Then, we have(49)gx1,⋯,xn2=f−1n∑i=1nλxiKxiH2=fH2−2n∑i=1nRef,λxiKxi+1n2∑i=1n∑j=1nλxiKxi,λxjKxj.SincePn is a probability measure, we have
(50)∫XnfH2dPnx1,⋯,xn=fH2.On the other hand, by Fubini’s theorem and the fact that the Bochner integral commutes with continuous linear operations, see Theorem2, we have
(51)∫XnRef,λxiKxidPnx1,⋯,xn=Ref,∫XnλxiKxidPnx1,⋯,xn=Ref,∫XλxKxdPx=Ref,LP,Kλ.Also, for eachi=1,⋯,n,
(52)∫XnλxiKxi,λxiKxidPnx1,⋯,xn=∫Xnλxi2Kxi,xidPnx1,⋯,xn=∫Xλx2Kx,xdPx,and, for each i,j=1,⋯,n,i≠j,
(53)∫XnλxiKxi,λxjKxjdPnx1,⋯,xn=∫XλxiKxi,∫XλxjKxjdPxjdPxi=∫XλxKxdPx,∫XλxKxdPx=∫XλxKxdPxH2.Integrating both sides of (49) and using all the previous equalities, we therefore have
(54)∫Xngx1,⋯,xnH2dPnx1,⋯,xn=fH2−2n∑i=1nRef,∫XλxKxdPx+1n2∑i=1n∑i≠j=1n∫XλxKxdPxH2+1n2∑i=1n∫Xλx2Kx,xdPx=fH2−2Ref,∫XλxKxdPx+n−1n∫XλxKxdPxH2+1n∫Xλx2Kx,xdPx=f−∫XλxKxdPxH2+1n∫Xλx2Kx,xdPx−∫XλxKxdPxH2=f−LP,KλH2+1n∫Xλx2Kx,xdPx−LP,KλH2.Finally, in view of the Markov-Bienaymé-Chebyshev Inequality as in (43), when X is replaced by Xn and P by Pn, and taking into account the previous equality and (48), we get
(55)PnAn,δ≤1δ2∫Xngx1,⋯,xnH2dPnx1,⋯,xn=1δ2f−LP,KλH2+1nδ2λL2X;PK2−LP,KλH2,which is the required inequality.☐


## 3.3. Convergence in Probability

As with the special case of kernel embeddings, for whichλ=1, see Smola et al. [15], we may use the bound in Theorem 8 to obtain a statement of convergence in probability.With notation and assumptions as before, givenf∈H and fixed xii=1N∈X, the problem of finding the optimal ωiNfi=1N∈FN to minimize ∥f−∑i=1NωiNfKxi∥H is straightforward: ∑i=1NωiNfKxi is the orthogonal projection of f to spanKxii=1N.We may assume without loss of generality thatKxii=1N are linearly independent, by removing points as necessary without affecting spanKxii=1N (or losing any information about f, since ∑i=1NciKxi=0 implies ∑i=1Nci¯fxi=0 by the reproducing property). According to Körezlioglu [16], if xii=1N∈X is a sampling such that Kxii=1N are linearly independent and considering the finite-dimensional subspace HxN≔spanKxii=1N of H, then the orthogonal projection πxN of H onto HxN is given by
(56)πxNf=∑i=1NωiπfKxi≔∑i=1N∑j=1NfxjΓjiNKxi=∑i=1N∑j=1Nf,KxjΓjiNKxifor anyf∈H, where ΓN∈MNF is the inverse of the Gram matrix GN≔Kxj,xii,j=1N=Kxi,Kxji,j=1N of x1,⋯,xN.More generally, ifKxii=1N are not linearly independent, for any subset s=xijj=1K such that Kxijj=1K form a basis for HxN, we have HxN=HsK and
(57)πxN=πsK=∑j=1K∑k=1K·,KxikΓkjKKxij.Note that, in general,ωiπ is not simply a multiple of fxi, hence, setting ωi≔Vifxi for any fixed Vi will not yield the best possible approximation. However, with such coefficients dependent only on xi, it will be easier to bound f−∑iωiKxi across different xiis than f−πxNf. Then, any upper bound on f−∑iωiKxi for some fixed ωii will also be an upper bound on f−πxNf.Theorem 9.
(Convergence in Probability of Projections) LetX, P, K, and H be as in Theorem 8. For each sequence x=xii∈Xℕ and each n∈ℕ, let πxn denote the orthogonal projection of H onto spanKxii=1n. Let f∈H and, for each δ>0 and n∈ℕ, define
(58)Bn,δ≔x1,⋯,xn∈Xn∣f−πxnfH≥δ.
Then, for eachδ>0(59)limsupn⟶∞PnBn,δ≤1δ2dHf,HP2,
wheredHf,HP=infg∈HP∥f−g∥.In particular, iffbelongs toHP¯H, the closure ofHPwith respect to the topology ofH, then(60)limn⟶∞PnBn,δ=0.Proof.
Letλ∈L2X,PK and fix δ>0, arbitrary. Then
(61)f−πxnfH≤f−1n∑i=1nλxiKxiH,hence, with notation as in (45), we have Bn,δ⊆An,δ. By Theorem 8, this implies
(62)PnBn,δ≤1δ2f−LP,KλH2+1nδ2λL2X,PK2−LP,KλH2.
Therefore,(63)limsupn⟶∞PnBn,δ≤limsupn⟶∞1δ2f−LP,KλH2+1nδ2λL2X,PK2−LP,KλH2=1δ2f−LP,KλH2.
Thus, since the left-hand side is independent ofλ,
(64)limsupn⟶∞PnBn,δ≤infλ∈L2X,PK1δ2f−LP,KλH2=1δ2dHf,HP2.
In particular, iff belongs to HP¯H, then dHf,HP=0.☐


## 3.4. Uniqueness Sets and Almost Certain Convergence of Projections

With notation and assumptions as before, we now follow ([2], Subsection 2.4.4) in recalling the strong convergence of πxN to the identity map as N⟶∞ for appropriately chosen xii=1∞. Since H is separable, there exists a countable subset of Kpp∈X which is total in H; thus, there exists a countable set F⊆X such that spanKxx∈F is dense in H. This motivates the following definition: a countable subset xii=1∞ of X is called a uniqueness set for H if Kxii=1∞ is a total set in H, that is, if f∈H such that fxi=0 for all i∈ℕ implies f=0. Then, the so-called Ultimate Realization of RKHSs, compare ([2], Theorem 2.33), reads as follows: if xii=1∞ is a uniqueness set such that Kxii=1∞ is linearly independent, GN is the Gram matrix for xii=1N, ΓN=GN−1, then for each f∈H,
(65)limN⟶∞πxNf=limN⟶∞∑i=1N∑j=1NfxiΓijNKxj=f,under the topology ofH, with distance decreasing monotonically. Consequently,
(66)f,g=limN⟶∞∑i=1N∑j=1NfxiΓijNgxj¯,for f,g∈H, and
(67)fx=f,Kx=limN⟶∞∑i=1N∑j=1NfxiΓijNKx,xj,for f∈H,x∈X. This has implications in interpolation theory, e.g., see ([2], Corollary~2.6).Coming back to our problem, by noting thatf−πxnf, unlike f−1/n∑i=1nλxiKxi, is monotonically nonincreasing with respect to n, our next goal is to strengthen Theorem 9 to almost certain convergence after passing to a single measure space. First, recall that, e.g., see ([13], Proposition 10.6.1), the countably infinite product space Xℕ equipped with the smallest σ-algebra rendering each projection map Xi:Xℕ⟶X measurable admits a unique probability measure Pℕ such that the projection maps are independent random variables with distribution P.Lemma 10.
LetX, P, K, and H be as in Theorem 8 and f∈H. For each δ>0 define
(68)Sn,δ≔x=xkk=1∞∈Xℕ∣f−πxnfH≥δ,n∈ℕ,and
(69)Sδ≔x=xkk=1∞∈Xℕ∣∀N∈ℕ,∃n≥N,f−πxnfH≥δ=⋂N∈ℕ⋃n≥NSn,δ.
Then,(70)PℕSδ≤1δ2dHf,HP2,and, consequently, if f∈HP¯H, then
(71)PℕSδ=0.Proof.
Observe that for eachn,m∈ℕ such that n>m, f−πxnfH≤f−πxmfH, for each x∈Xℕ, and hence Sn,δ⊆Sm,δ for each δ>0. Then,
(72)Sδ=⋂N∈ℕ⋃n≥NSn,δ=⋂N∈ℕSN,δ,
hence, for anyλ∈L2X,PK,
(73)PℕSδ≤infN∈ℕPℕSN,δ≤1δ2f−LP,KλH2,
sincePℕ is monotone and Sδ⊆SN,δ for all N∈ℕ.☐The main result of this subsection is the following.Theorem 11.
(Almost Certain Convergence of Projections) LetX,P,K,H be as in Theorem 8 and suppose HP is dense in H. Then, for each f∈H,
(74)Pℕx∈Xℕ∣πxnf→nf=1,hence,
(75)Pℕx∈Xℕ∀f∈H,πxnf→nf=1.Proof.
Letf∈H. With the same sets Sδ defined in (69),
(76)x∈Xℕ∣πxnf↛f=x∈Xℕ∣∃δ>0,∀N∈ℕ,∃n≥N,f−πxnfH≥δ=⋃δ>0Sδ.Observe further thatSδ⊆Sδ′ whenever δ>δ′, and for each δ>0 there exists m∈ℕ such that δ>1/m, so that
(77)x∈Xℕπxnf↛nf=⋃0<δ≤1Sδ=⋃m∈ℕS1/m,thus, taking into account thatHP is dense in H and using Lemma 10, we get
(78)Pℕx∈Xℕπxnf↛nf≤∑m∈ℕPℕS1/m=∑m∈ℕ0=0.SinceH is separable let D be a countable dense subset of H. Since each πxn is a continuous linear operator with operator norm 1, πxnf⟶nf for all f∈H iff πxnf⟶nf for all f∈D. Thus, by the countable subadditivity of Pℕ,
(79)Pℕx∈Xℕ∃f∈H,πxnf↛nf=Pℕx∈Xℕ∃f∈D,πxnf↛nf=Pℕ⋃f∈Dx∈Xℕπxnf↛nf≤∑f∈DPℕx∈Xℕπxnf↛nf=0.☐In summary, for a given probability measureP under the assumption that it renders the space HP, the image of LP,K, dense in H, a sequence of points sampled independently from P yields a uniqueness set with probability 1. Proposition shows a sufficient condition, valid for many applications, when this assumption holds.


## 4. Examples

In this final section, we provide detailed examples of the applicability of the results on approximation error bounds obtained in the previous section.


### 4.1. Uniform Distribution on a Compact Interval

Letμjj∈ℤ∈l1ℤ be such that μj>0 for all j∈ℤ and denote μ≔∑j∈ℤμj. For each j∈ℤ define
(80)ϕj:−π,π⟶ℂ,ϕjt≔eiπjt,t∈−π,π, and consider the Hilbert space
(81)H=∑j∈ℤcjϕj∣∑j∈ℤcj2μj<∞,with the inner product
(82)∑j∈ℤcjϕj,∑j∈ℤdjϕj=∑j∈ℤcjdj¯μj.Thenμjϕjj∈ℤ is an orthonormal basis of H and, for an arbitrary function f∈H, we have the Fourier representation
(83)ft=∑j∈ℤcjϕjt,t∈−π,π, with coefficients cjj∈ℤ subject to the condition
(84)fH2≔∑j∈ℤcj2μj<∞,where the convergence of the series from (83) is at least guaranteed with respect to the norm ·H. However, for any m∈ℕ0 and t∈−π,π, by the Cauchy inequality, we have
(85)∑j≥mcjϕjt≤∑j≥mcj2μj1/2∑j≥mμj1/2→m⟶∞0,hence, the convergence in (83) is absolutely and uniformly on −π,π, in particular f is continuous.By (12), H has the reproducing kernel
(86)Ks,t=∑j∈ℤμjeiπjs−t=∑j∈ℤμjϕjsϕjt¯,and the convergence of the series is guaranteed at least pointwise. In addition, for any t∈−π,π, we have
(87)Kt,t=∑j∈ℤμjϕjt2=∑j∈ℤμj=μ,and hence the kernelK is bounded. In particular, this implies that, actually, the series in (86) converges absolutely and uniformly on −π,π, hence, the kernel K is continuous on −π,π×−π,π. That is, Ks,t is given by κs−t where κ:ℝ⟶ℂ is a continuous function with period 2π whose Fourier coefficients μjj∈ℤ are all positive and absolutely summable.LetP be the normalized Lebesgue measure on −π,π, equivalently, the uniform probability distribution on −π,π, and observe that ϕjj∈ℤ is an orthonormal basis of the Hilbert space LP−π,π. With notation as in (24), we have dPKt=Kt,tdPt=μdPt hence LPK2−π,π=LP2−π,π with norms differing by multiplication with μ>0. In particular, ϕj/μj∈ℤ is an orthonormal basis of the Hilbert space LPK2−π,π.We consider now the nonexpansive operatorLP,K:LPK2−π,π⟶H defined as in (26). Then, for any j∈ℤ and t∈−π,π, we have
(88)LP,Kϕjt=∫−ππϕjsKt,sdPs=∫−ππϕjs∑k∈ℤμkϕktϕks¯dPs=∑k∈ℤμkϕkt∫−ππϕjsϕks¯dPs=∑k∈ℤμkϕktδjk=μjϕjt,where the series commutes with the integral either by the Bounded Convergence Theorem for the Lebesgue integral, or by using the uniform convergence of the series and the Riemann integral. Similarly, the Hilbert space HP≔LP,KLPK2−π,π, as in Proposition 5, is a RKHS, with kernel,
(89)KPs,t=12π∫−ππ∑j∈ℤμjϕjsϕjz¯∑l∈ℤμlϕlzϕlt¯∑j∈ℤμjdz=1μ∑j∈ℤ∑l∈ℤμjμlϕjsϕlt¯12π∫−ππϕjzϕlz¯dz=1μ∑j∈ℤ∑l∈ℤμjμlϕjsϕlt¯δjl=∑j∈ℤμj2μϕjsϕjt¯.Thus, lettingμj′≔μj2/μ≤μj, j∈ℤ and noting that ∑j∈ℤμj′≤∑j∈ℤμj<∞, we have
(90)HP=∑j∈ℤcjϕj∣∑j∈ℤcj2μj′<∞=∑j∈ℤcjϕj∣∑j∈ℤcj2μj2<∞.In particular,HP is dense in H since both contain spanϕjj∈ℤ as dense subsets, but this follows from the more general statement in Proposition 6 as well.Let nowλ∈LPK2−π,π=LP2−π,π be arbitrary, hence
(91)λ=∑j∈ℤλjϕj,∑j∈ℤλj2<∞,λLPK2−π,π2=1μ∑j∈ℤλj2.Then,(92)LP,Kλt=LP,K∑j∈ℤλjϕjt=∑j∈ℤλjμjϕjt,t∈−π,π, and, consequently,
(93)LP,KλH2=∑j∈ℤλj2μj2μj=∑j∈ℤμjλj2.Also, for arbitraryf∈H as in (83) and (84), we have
(94)f−LP,KλH2=∑j∈ℤcj−λjμjϕjH2=∑j∈ℤcj−λjμj2μj.Letxnn∈ℕ be a sequence of points in −π,π. By Theorem 8 and taking into account of the inequality (61), for any N∈ℕ and δ>0, we have
(95)PNf−πxNfH≥δ≤PNf−1N∑n=1NλxnKxnH≥δ≤1δ2∑j∈ℤcj−λjμj2μj+1Nδ2∑j∈ℤμ−μjλj2.On the other hand, we observe that in the inequality (95) the left hand side does not depend on λ and hence, for any ε>0 there exists λ∈LPK2−π,π such that
(96)PNf−πxNfH≥δ<ε2+1Nδ2∑j∈ℤμ−μjλj2,and then, for sufficiently large N, we get
(97)PNf−πxNfH≥δ<ε.In particular, iff∈HP, that is, the inequality (84) is replaced by the stronger one
(98)∑j∈ℤcj2μj2<∞,we can choose λj=cj/μj, j∈ℤ, and we have λ∈LPK2−π,π, hence
(99)PNf−πxNfH≥δ≤1Nδ2∑j∈ℤμ−μjcj2μj2.For example, this is the case forf=ϕk for some k∈ℤ, hence, cj=δj,k, j∈ℤ, and letting λ=ϕk/μk, hence, λj=δj,k/μj, j∈ℤ, we have f=LP,Kλ and hence,
(100)PNϕk−πxNϕk≥δ≤1Nδ2μk2∑ℤ∋j≠kμj.This shows that, the largerμk is, the faster ϕk will be approximated but, since μj→j0, ϕjs cannot be approximated uniformly, in the sense that there does not exist a single N to make each ϕj−πxNϕjH bounded by the same δ with the same probability η.This analysis can be applied more generally to kernels that admit an expansion analogous to (86) under basis functions ϕjj which constitute a total orthonormal set in L2X;PK, e.g., as guaranteed by Mercer’s Theorem ([2], Theorem 2.30).


### 4.2. The Hardy SpaceH2D

We consider the open unit disc in the complex planeD=z∈ℂ∣z<1 and the Szegö kernel
(101)Kz,ζ=11−zζ¯=∑n=0∞znζ¯n,z,ζ∈D,where the series converges absolutely and uniformly on any compact subset of D. The RKHS associated to K is the Hardy space H2D of all functions f:D⟶ℂ that are holomorphic in D with power series expansion
(102)fz=∑n=0∞fnzn,such that the coefficients sequencefnn is in ℓℂ2ℕ0. The inner product in H2D is
(103)∑n=0∞fnzn,∑n=0∞gnzn=∑n=0∞fngn¯,with norm
(104)∑n=0∞fnzn2=∑n=0∞fn2.For eachζ∈D, we have
(105)Kζ=∑n=0∞ζ2n1/2=11−ζ2,hence, the kernel K is unbounded.We considerP the normalized Lebesgue measure on D, that is, for z=x+iy=reiθ, we have
(106)dPz=1πdAx,y=rπdθdr,hence,
(107)dPKz=rπ1−r2dθdr.Then,L2D;PK is contractively embedded in L2D;P.Further on, in view of Proposition5 and (101), for any z,ζ∈D, we have
(108)KPz,ζ=1π∫01∫02πr1−r21−zre−iθ1−ζ¯reiθdθdr=1π∫01∫02π∑n=0∞∑k=0∞1−r2rn+k+1ein−kθznζ¯kdθdr, which, by using twice the Bounded Convergence Theorem for the Lebesgue measure, equals
(109)=∑n=0∞∑k=0∞1π∫01∫02π1−r2rn+k+1ein−kθdθdrznζ¯k=∑n=0∞4∫011−r2r2n+1drznζ¯n=∑n=0∞znζ¯nn+1n+2.This shows that the RKHSHP2D induced by KP consists of all functions h that are holomorphic in D with power series representation hz=∑n=0∞hnzn and such that
(110)∑n=0∞n+1n+2hn2<∞.In particular, an orthonormal basis ofHP2D is zn/n+1n+2n≥0 and hence HP2D is dense in the Hardy space H2D.In order to calculate the operatorLP,K:L2D;PK⟶H2D, let λ∈L2D;PK be arbitrary, that is, λ is a complex-valued measurable function on D such that
(111)λL2D;PK2=1π∫01∫02πλreiθ2r1−r2dθdr<∞.Then, in view of Proposition3, we have
(112)LP,Kλz=1π∫01∫02πλreiθKz,reiθrdθdr=1π∫01∫02πλreiθ∑n=0∞znrn+1e−inθdθdr,which, by the Bounded Convergence Theorem, equals
(113)=∑n=0∞1π∫01∫02πλreiθrn+1e−inθdθdrzn=∑n=0∞λnzn,where for each integer n≥0 we denote
(114)λn=1π∫01∫02πλreiθrn+1e−inθdθdr.Observing that, lettingϕnz≔n+1zn, for all integer n≥0 and z∈D, the set ϕnn≥0 is orthonormal in L2D;P, it follows that λn=λ,ϕnL2D;P for all integer n≥0 and, hence, λnn≥0 is the weighted sequence of Fourier coefficients of λ with respect to the system of orthonormal functions ϕnn≥0 in L2D;P. On the other hand, since L2D;PK is contractively embedded in L2D;P, this shows that LP,K is the restriction to L2D;PK of a Bergman type weighted projection of L2D;P onto a subspace of the Hardy space H2D, that happens to be exactly HP2D.Finally, letf∈H2D with power series representation as in (102) and let λ∈L2D;PK with norm given as in (111). Then, by Theorem 8 and taking into account of the inequality (61), for any N∈ℕ and δ>0, we have
(115)PNf−πzNfH2D≥δ≤PNf−1N∑i=1NλziKziH2D≥δ≤1δ2∑n=0∞fn−λn2+1Nδ2λL2D;PK2−∑n=0∞λn2,wherez=zii∈ℕ denotes an arbitrary sequence of points in D and πzN denotes the projection of H2D onto spanKzi∣i=1,⋯,N. By exploiting the fact that the left hand side in (115) does not depend on λ and the density of HP2D in H2D, for any ε>0 there exists λ∈L2D;PK such that
(116)PNf−πzNfH2D≥δ≤ε2+1Nδ2λL2D;PK2−∑n=0∞λn2,and hence, forN sufficiently large, we have
(117)PNf−πzNfH2D≥δ≤ε.Let us consider now the special case when the functionf∈HP2D, that is, with respect to the representation as in (102), we have the stronger condition
(118)∑n=0∞n+1n+2fn2<∞.In this case, letting(119)λz≔∑n=0∞n+1n+21−z2fnzn,z∈D, calculations similar to (108) and (112) show that
(120)1π∫01∫02πλreiθ2r1−r2dθdr=∑n=0∞n+1n+2fn2<∞,hence λ∈L2D;PK, and
(121)LP,Kλz=1π∫0∞∫02πλreiθKz,reiθrdθdr=∑n=0∞fnzn=fz,z∈D, hence, the first term in the right hand side of (115) vanishes and we get
(122)PNf−πzNfH2D≥δ≤1Nδ2∑n=0∞n2+3n+1fn2.For example, iffz=zn for some integer n≥0, then
(123)PNf−πzNfH2D≥δ≤n2+3n+1Nδ2,showing that better approximations are obtained for smallern than for bigger n.


## 4.1. Uniform Distribution on a Compact Interval

Letμjj∈ℤ∈l1ℤ be such that μj>0 for all j∈ℤ and denote μ≔∑j∈ℤμj. For each j∈ℤ define
(80)ϕj:−π,π⟶ℂ,ϕjt≔eiπjt,t∈−π,π, and consider the Hilbert space
(81)H=∑j∈ℤcjϕj∣∑j∈ℤcj2μj<∞,with the inner product
(82)∑j∈ℤcjϕj,∑j∈ℤdjϕj=∑j∈ℤcjdj¯μj.Thenμjϕjj∈ℤ is an orthonormal basis of H and, for an arbitrary function f∈H, we have the Fourier representation
(83)ft=∑j∈ℤcjϕjt,t∈−π,π, with coefficients cjj∈ℤ subject to the condition
(84)fH2≔∑j∈ℤcj2μj<∞,where the convergence of the series from (83) is at least guaranteed with respect to the norm ·H. However, for any m∈ℕ0 and t∈−π,π, by the Cauchy inequality, we have
(85)∑j≥mcjϕjt≤∑j≥mcj2μj1/2∑j≥mμj1/2→m⟶∞0,hence, the convergence in (83) is absolutely and uniformly on −π,π, in particular f is continuous.By (12), H has the reproducing kernel
(86)Ks,t=∑j∈ℤμjeiπjs−t=∑j∈ℤμjϕjsϕjt¯,and the convergence of the series is guaranteed at least pointwise. In addition, for any t∈−π,π, we have
(87)Kt,t=∑j∈ℤμjϕjt2=∑j∈ℤμj=μ,and hence the kernelK is bounded. In particular, this implies that, actually, the series in (86) converges absolutely and uniformly on −π,π, hence, the kernel K is continuous on −π,π×−π,π. That is, Ks,t is given by κs−t where κ:ℝ⟶ℂ is a continuous function with period 2π whose Fourier coefficients μjj∈ℤ are all positive and absolutely summable.LetP be the normalized Lebesgue measure on −π,π, equivalently, the uniform probability distribution on −π,π, and observe that ϕjj∈ℤ is an orthonormal basis of the Hilbert space LP−π,π. With notation as in (24), we have dPKt=Kt,tdPt=μdPt hence LPK2−π,π=LP2−π,π with norms differing by multiplication with μ>0. In particular, ϕj/μj∈ℤ is an orthonormal basis of the Hilbert space LPK2−π,π.We consider now the nonexpansive operatorLP,K:LPK2−π,π⟶H defined as in (26). Then, for any j∈ℤ and t∈−π,π, we have
(88)LP,Kϕjt=∫−ππϕjsKt,sdPs=∫−ππϕjs∑k∈ℤμkϕktϕks¯dPs=∑k∈ℤμkϕkt∫−ππϕjsϕks¯dPs=∑k∈ℤμkϕktδjk=μjϕjt,where the series commutes with the integral either by the Bounded Convergence Theorem for the Lebesgue integral, or by using the uniform convergence of the series and the Riemann integral. Similarly, the Hilbert space HP≔LP,KLPK2−π,π, as in Proposition 5, is a RKHS, with kernel,
(89)KPs,t=12π∫−ππ∑j∈ℤμjϕjsϕjz¯∑l∈ℤμlϕlzϕlt¯∑j∈ℤμjdz=1μ∑j∈ℤ∑l∈ℤμjμlϕjsϕlt¯12π∫−ππϕjzϕlz¯dz=1μ∑j∈ℤ∑l∈ℤμjμlϕjsϕlt¯δjl=∑j∈ℤμj2μϕjsϕjt¯.Thus, lettingμj′≔μj2/μ≤μj, j∈ℤ and noting that ∑j∈ℤμj′≤∑j∈ℤμj<∞, we have
(90)HP=∑j∈ℤcjϕj∣∑j∈ℤcj2μj′<∞=∑j∈ℤcjϕj∣∑j∈ℤcj2μj2<∞.In particular,HP is dense in H since both contain spanϕjj∈ℤ as dense subsets, but this follows from the more general statement in Proposition 6 as well.Let nowλ∈LPK2−π,π=LP2−π,π be arbitrary, hence
(91)λ=∑j∈ℤλjϕj,∑j∈ℤλj2<∞,λLPK2−π,π2=1μ∑j∈ℤλj2.Then,(92)LP,Kλt=LP,K∑j∈ℤλjϕjt=∑j∈ℤλjμjϕjt,t∈−π,π, and, consequently,
(93)LP,KλH2=∑j∈ℤλj2μj2μj=∑j∈ℤμjλj2.Also, for arbitraryf∈H as in (83) and (84), we have
(94)f−LP,KλH2=∑j∈ℤcj−λjμjϕjH2=∑j∈ℤcj−λjμj2μj.Letxnn∈ℕ be a sequence of points in −π,π. By Theorem 8 and taking into account of the inequality (61), for any N∈ℕ and δ>0, we have
(95)PNf−πxNfH≥δ≤PNf−1N∑n=1NλxnKxnH≥δ≤1δ2∑j∈ℤcj−λjμj2μj+1Nδ2∑j∈ℤμ−μjλj2.On the other hand, we observe that in the inequality (95) the left hand side does not depend on λ and hence, for any ε>0 there exists λ∈LPK2−π,π such that
(96)PNf−πxNfH≥δ<ε2+1Nδ2∑j∈ℤμ−μjλj2,and then, for sufficiently large N, we get
(97)PNf−πxNfH≥δ<ε.In particular, iff∈HP, that is, the inequality (84) is replaced by the stronger one
(98)∑j∈ℤcj2μj2<∞,we can choose λj=cj/μj, j∈ℤ, and we have λ∈LPK2−π,π, hence
(99)PNf−πxNfH≥δ≤1Nδ2∑j∈ℤμ−μjcj2μj2.For example, this is the case forf=ϕk for some k∈ℤ, hence, cj=δj,k, j∈ℤ, and letting λ=ϕk/μk, hence, λj=δj,k/μj, j∈ℤ, we have f=LP,Kλ and hence,
(100)PNϕk−πxNϕk≥δ≤1Nδ2μk2∑ℤ∋j≠kμj.This shows that, the largerμk is, the faster ϕk will be approximated but, since μj→j0, ϕjs cannot be approximated uniformly, in the sense that there does not exist a single N to make each ϕj−πxNϕjH bounded by the same δ with the same probability η.This analysis can be applied more generally to kernels that admit an expansion analogous to (86) under basis functions ϕjj which constitute a total orthonormal set in L2X;PK, e.g., as guaranteed by Mercer’s Theorem ([2], Theorem 2.30).


## 4.2. The Hardy SpaceH2D

We consider the open unit disc in the complex planeD=z∈ℂ∣z<1 and the Szegö kernel
(101)Kz,ζ=11−zζ¯=∑n=0∞znζ¯n,z,ζ∈D,where the series converges absolutely and uniformly on any compact subset of D. The RKHS associated to K is the Hardy space H2D of all functions f:D⟶ℂ that are holomorphic in D with power series expansion
(102)fz=∑n=0∞fnzn,such that the coefficients sequencefnn is in ℓℂ2ℕ0. The inner product in H2D is
(103)∑n=0∞fnzn,∑n=0∞gnzn=∑n=0∞fngn¯,with norm
(104)∑n=0∞fnzn2=∑n=0∞fn2.For eachζ∈D, we have
(105)Kζ=∑n=0∞ζ2n1/2=11−ζ2,hence, the kernel K is unbounded.We considerP the normalized Lebesgue measure on D, that is, for z=x+iy=reiθ, we have
(106)dPz=1πdAx,y=rπdθdr,hence,
(107)dPKz=rπ1−r2dθdr.Then,L2D;PK is contractively embedded in L2D;P.Further on, in view of Proposition5 and (101), for any z,ζ∈D, we have
(108)KPz,ζ=1π∫01∫02πr1−r21−zre−iθ1−ζ¯reiθdθdr=1π∫01∫02π∑n=0∞∑k=0∞1−r2rn+k+1ein−kθznζ¯kdθdr, which, by using twice the Bounded Convergence Theorem for the Lebesgue measure, equals
(109)=∑n=0∞∑k=0∞1π∫01∫02π1−r2rn+k+1ein−kθdθdrznζ¯k=∑n=0∞4∫011−r2r2n+1drznζ¯n=∑n=0∞znζ¯nn+1n+2.This shows that the RKHSHP2D induced by KP consists of all functions h that are holomorphic in D with power series representation hz=∑n=0∞hnzn and such that
(110)∑n=0∞n+1n+2hn2<∞.In particular, an orthonormal basis ofHP2D is zn/n+1n+2n≥0 and hence HP2D is dense in the Hardy space H2D.In order to calculate the operatorLP,K:L2D;PK⟶H2D, let λ∈L2D;PK be arbitrary, that is, λ is a complex-valued measurable function on D such that
(111)λL2D;PK2=1π∫01∫02πλreiθ2r1−r2dθdr<∞.Then, in view of Proposition3, we have
(112)LP,Kλz=1π∫01∫02πλreiθKz,reiθrdθdr=1π∫01∫02πλreiθ∑n=0∞znrn+1e−inθdθdr,which, by the Bounded Convergence Theorem, equals
(113)=∑n=0∞1π∫01∫02πλreiθrn+1e−inθdθdrzn=∑n=0∞λnzn,where for each integer n≥0 we denote
(114)λn=1π∫01∫02πλreiθrn+1e−inθdθdr.Observing that, lettingϕnz≔n+1zn, for all integer n≥0 and z∈D, the set ϕnn≥0 is orthonormal in L2D;P, it follows that λn=λ,ϕnL2D;P for all integer n≥0 and, hence, λnn≥0 is the weighted sequence of Fourier coefficients of λ with respect to the system of orthonormal functions ϕnn≥0 in L2D;P. On the other hand, since L2D;PK is contractively embedded in L2D;P, this shows that LP,K is the restriction to L2D;PK of a Bergman type weighted projection of L2D;P onto a subspace of the Hardy space H2D, that happens to be exactly HP2D.Finally, letf∈H2D with power series representation as in (102) and let λ∈L2D;PK with norm given as in (111). Then, by Theorem 8 and taking into account of the inequality (61), for any N∈ℕ and δ>0, we have
(115)PNf−πzNfH2D≥δ≤PNf−1N∑i=1NλziKziH2D≥δ≤1δ2∑n=0∞fn−λn2+1Nδ2λL2D;PK2−∑n=0∞λn2,wherez=zii∈ℕ denotes an arbitrary sequence of points in D and πzN denotes the projection of H2D onto spanKzi∣i=1,⋯,N. By exploiting the fact that the left hand side in (115) does not depend on λ and the density of HP2D in H2D, for any ε>0 there exists λ∈L2D;PK such that
(116)PNf−πzNfH2D≥δ≤ε2+1Nδ2λL2D;PK2−∑n=0∞λn2,and hence, forN sufficiently large, we have
(117)PNf−πzNfH2D≥δ≤ε.Let us consider now the special case when the functionf∈HP2D, that is, with respect to the representation as in (102), we have the stronger condition
(118)∑n=0∞n+1n+2fn2<∞.In this case, letting(119)λz≔∑n=0∞n+1n+21−z2fnzn,z∈D, calculations similar to (108) and (112) show that
(120)1π∫01∫02πλreiθ2r1−r2dθdr=∑n=0∞n+1n+2fn2<∞,hence λ∈L2D;PK, and
(121)LP,Kλz=1π∫0∞∫02πλreiθKz,reiθrdθdr=∑n=0∞fnzn=fz,z∈D, hence, the first term in the right hand side of (115) vanishes and we get
(122)PNf−πzNfH2D≥δ≤1Nδ2∑n=0∞n2+3n+1fn2.For example, iffz=zn for some integer n≥0, then
(123)PNf−πzNfH2D≥δ≤n2+3n+1Nδ2,showing that better approximations are obtained for smallern than for bigger n.


## 5. Conclusions

Certain key properties of Hilbert spaces drive the analysis that has been obtained in this article, as well as the properties of reproducing kernel Hilbert spaces that render them attractive for function approximation. The Hilbert space structure provides orthogonal projections as the unique best approximation, which can be computed using the reproducing property as an exact interpolation and are shown to converge monotonically to the function for uniqueness sets. The monotonicity of convergence is then used to derive almost certain convergence directly from convergence in probability, and thus establish sufficient conditions for almost every sequence of samples from a probability distribution to be a uniqueness set. For the approximation bound itself, stated in Theorem8, the mean squared distance in Chebyshev’s inequality can be calculated explicitly thanks to the norm being induced by an inner product and the existence of the Bochner integral.We did not include in this article an example with the Gaussian kernel, one of the most useful kernels in applications, although calculations similar to those obtained in Section4 are available. One of the reasons for this omission is that the Gaussian kernels have additional invariance and differentiability/analyticity properties that can be used in order to provide stronger results by using slightly different techniques that are in progress and will make the contents of a future research.



---

*Source: 6617774-2021-05-03.xml*