Year: 2021

Abstract:
We characterize the power-law asymptotics of learning curves for Gaussian process regression (GPR) under the assumption that the eigenspectrum of the prior and the eigenexpansion coefficients of the target function follow a power law. Under similar assumptions, we leverage the equivalence between GPR and kernel ridge regression (KRR) to show the generalization error of KRR. Infinitely wide neural networks can be related to GPR with respect to the neural network GP kernel and the neural tangent kernel, which in several cases is known to have a power-law spectrum. Hence our methods can be applied to study the generalization error of infinitely wide neural networks. We present toy experiments demonstrating the theory.

1 INTRODUCTION
Gaussian processes (GPs) provide a flexible and interpretable framework for learning and adaptive inference, and are widely used for constructing prior distributions in non-parametric Bayesian learning. From an application perspective, one crucial question is how fast do GPs learn, i.e., how much training data is needed to achieve a certain level of generalization performance. Theoretically, this is addressed by analyzing so-called “learning curves”, which describe the generalization error as a function of the training set size n. The rate at which the curve approaches zero determines the difficulty of learning tasks and conveys important information about the asymptotic performance of GP learning algorithms. In this paper, we study the learning curves for Gaussian process regression. Our main result characterizes the asymptotics of the generalization error in cases where the eigenvalues of the GP kernel and the coefficients of the eigenexpansion of the target function have a power-law decay. In the remainder of this introductory section, we review related work and outline our main contributions.
Gaussian processes A GP model is a probabilistic model on an infinite-dimensional parameter space (Williams and Rasmussen, 2006; Orbanz and Teh, 2010). In GP regression (GPR), for example, this space can be the set of all continuous functions. Assumptions about the learning problem are encoded by way of a prior distribution over functions, which gets transformed into a posterior distribution given some observed data. The mean of the posterior is then used for prediction. The model uses only a finite subset of the available parameters to explain the data and this subset can grow arbitrarily large as more data are observed. In this sense, GPs are “non-parametric” and contrast with parametric models, where there is a fixed number of parameters. For regression with Gaussian noise, a major appeal of the GP formalism is that the posterior is analytically tractable. GPs are also one important part in learning with kernel machines (Kanagawa et al., 2018) and modeling using GPs has recently gained considerable traction in the neural network community.
Neural networks and kernel learning From a GP viewpoint, there exists a well known correspondence between kernel methods and infinite neural networks (NNs) first studied by Neal (1996). Neal showed that the outputs of a randomly initialized one-hidden layer neural network (with appropriate scaling of the variance of the initialization distribution) converges to a GP over functions in the limit of an infinite number of hidden units. Follow-up work extended this correspondence with analytical expressions for the kernel covariance for shallow NNs by Williams (1997), and more recently for deep fully-connected NNs (Lee et al., 2018; de G. Matthews et al., 2018), convolutional NNs with many channels (Novak et al., 2019; Garriga-Alonso et al., 2019), and more general architectures (Yang, 2019). The correspondence enables exact Bayesian inference in the associated GP model for infinite-width NNs on regression tasks and has led to some recent breakthroughs in our understanding
of overparameterized NNs (Jacot et al., 2018; Lee et al., 2019; Arora et al., 2019; Belkin et al., 2018; Daniely et al., 2016; Yang and Salman, 2019; Bietti and Mairal, 2019). The most prominent kernels associated with infinite-width NNs are the Neural Network Gaussian Process (NNGP) kernel when only the last layer is trained (Lee et al., 2018; de G. Matthews et al., 2018), and the Neural Tangent Kernel (NTK) when the entire model is trained (Jacot et al., 2018). Empirical studies have shown that inference with such infinite network kernels is competitive with standard gradient descent-based optimization for fully-connected architectures (Lee et al., 2020).
Learning curves A large-scale empirical characterization of the generalization performance of state-of-the-art deep NNs showed that the associated learning curves often follow a power law of the form n−β with the exponent β ranging between 0.07 and 0.35 depending on the data and the algorithm (Hestness et al., 2017; Spigler et al., 2020). Power-law asymptotics of learning curves have been theoretically studied in early works for the Gibbs learning algorithm (Amari et al., 1992; Amari and Murata, 1993; Haussler et al., 1996) that showed a generalization error scaling with exponent β=0.5, 1 or 2 under certain assumptions. More recent results from statistical learning theory characterize the shape of learning curves depending on the properties of the hypothesis class (Bousquet et al., 2021). In the context of GPs, approximations and bounds on learning curves have been investigated in several works (Sollich, 1999; Sollich and Halees, 2002; Sollich, 2001; Opper and Vivarelli, 1999; Opper and Malzahn, 2002; Williams and Vivarelli, 2000; Malzahn and Opper, 2001a;b; Seeger et al., 2008; Van Der Vaart and Van Zanten, 2011; Le Gratiet and Garnier, 2015), with recent extensions to kernel regression from a spectral bias perspective (Bordelon et al., 2020; Canatar et al., 2021). For a review on learning curves in relation to its shape and monotonicity, see Loog et al. (2019); Viering et al. (2019); Viering and Loog (2021). A related but complementary line of work studies the convergence rates and posterior consistency properties of Bayesian non-parametric models (Barron, 1998; Seeger et al., 2008; Van Der Vaart and Van Zanten, 2011).
Power-law decay of the GP kernel eigenspectrum The rate of decay of the eigenvalues of the GP kernel conveys important information about its smoothness. Intuitively, if a process is “rough” with more power at high frequencies, then the eigenspectrum decays more slowly. On the other hand, kernels that define smooth processes have a fast-decaying eigenspectrum (Stein, 2012; Williams and Rasmussen, 2006). The precise eigenvalues (λp)p≥1 of the operators associated to many kernels and input distributions are not known explicitly, except for a few special cases (Williams and Rasmussen, 2006). Often, however, the asymptotic properties are known. The asymptotic rate of decay of the eigenvalues of stationary kernels for input distributions with bounded support is well understood (Widom, 1963; Ritter et al., 1995). Ronen et al. (2019) showed that for inputs distributed uniformly on a hypersphere, the eigenfunctions of the arc-cosine kernel are spherical harmonics and the eigenvalues follow a power-law decay. The spectral properties of the NTK are integral to the analysis of training convergence and generalization of NNs, and several recent works empirically justify and rely on a power law assumption for the NTK spectrum (Bahri et al., 2021; Canatar et al., 2021; Lee et al., 2020; Nitanda and Suzuki, 2021). Velikanov and Yarotsky (2021) showed that the asymptotics of the NTK of infinitely wide shallow ReLU networks follows a power-law that is determined primarily by the singularities of the kernel and has the form λp∝p−α with α=1+ 1d , where d is the input dimension.
Asymptotics of the generalization error of kernel ridge regression (KRR) There is a well known equivalence between GPR and KRR with the additive noise in GPR playing the role of regularization in KRR (Kanagawa et al., 2018). Analysis of the decay rates of the excess generalization error of KRR has appeared in several works, e.g, in the noiseless case with constant regularization (Bordelon et al., 2020; Spigler et al., 2020; Jun et al., 2019), and the noisy optimally regularized case (Caponnetto and De Vito, 2007; Steinwart et al., 2009; Fischer and Steinwart, 2020) under the assumption that the kernel eigenspectrum, and the eigenexpansion coefficients of the target function follow a power law. These assumptions, which are often called resp. the capacity and source conditions are related to the effective dimension of the problem and the difficulty of learning the target function (Caponnetto and De Vito, 2007; Blanchard and Mücke, 2018). Cui et al. (2021) present a unifying picture of the excess error decay rates under the capacity and source conditions in terms of the interplay between noise and regularization illustrating their results with real datasets.
Contributions In this work, we characterize the asymptotics of the generalization error of GPR and KRR under the capacity and source conditions. Our main contributions are as follows:
• When the eigenspectrum of the prior decays with rate α and the eigenexpansion coefficients of the target function decay with rate β, we show that with high probability over the draw of n input samples, the negative log-marginal likelihood behaves as Θ(nmax{ 1 α , 1−2β α +1}) (Theorem 7) and
the generalization error behaves as Θ(nmax{ 1 α−1, 1−2β α }) (Theorem 9). In the special case that the model is correctly specified, i.e., the GP prior is the true one from which the target functions are actually generated, our result implies that the generalization error behaves asO(n 1 α−1) recovering
as a special case a result due to Sollich and Halees (2002) (vide Remark 10). • Under similar assumptions as in the previous item, we leverage the equivalence between GPR
and KRR to show that the excess generalization error of KRR behaves as Θ(nmax{ 1 α−1, 1−2β α }) (Theorem 12). In the noiseless case with constant regularization, our result implies that the generalization error behaves as Θ(n 1−2β α ) recovering as a special case a result due to Bordelon et al. (2020). Specializing to the case of KRR with Gaussian design, we recover as a special case a result due to Cui et al. (2021) (vide Remark 14). For the unrealizable case, i.e., when the target function is outside the span of the eigenfunctions with positive eigenvalues, we show that the generalization error converges to a constant.
• We present a few toy experiments demonstrating the theory for GPR with arc-cosine kernel without biases (resp. with biases) which is the conjugate kernel of an infinitely wide shallow network with two inputs and one hidden layer without biases (resp. with biases) (Cho and Saul, 2009; Ronen et al., 2019).
2 BAYESIAN LEARNING AND GENERALIZATION ERROR FOR GPS
In GP regression, our goal is to learn a target function f : Ω 7→ R between an input x ∈ Ω and output y ∈ R based on training samples Dn = {(xi, yi)}ni=1. We consider an additive noise model yi = f(xi) + i, where i
i.i.d.∼ N (0,σ2true). If ρ denotes the marginal density of the inputs xi, then the pairs (xi, yi) are generated according to the density q(x, y) = ρ(x)q(y|x), where q(y|x) =N (y|f(x),σ2true). We assume that there is a prior distribution Π0 on f which is defined as a zero-mean GP with continuous covariance function k : Ω×Ω→R, i.e., f ∼GP(0,k). This means that for any finite set x = (x1,...,xn)T , the random vector f(x) = (f(x1),...,f(xn))T follows the multivariate normal distributionN (0,Kn) with covariance matrixKn=(k(xi,xj))ni,j=1∈Rn×n. By Bayes’ rule, the posterior distribution of the target f given the training data is given by
dΠn(f |Dn)= 1
Z(Dn) n∏ i=1 N (yi|f(xi),σ2model)dΠ0(f),
where Π0 is the prior distribution, Z(Dn) = ∫ ∏n
i=1N (yi|f(xi),σ2model)dΠ0(f) is the marginal likelihood or model evidence and σmodel is the sample variance used in GPR. In practice, we do not know the exact value of σtrue and so our choice of σmodel can be different from σtrue. The GP prior and the Gaussian noise assumption allows for exact Bayesian inference and the posterior distribution over functions is again a GP with mean and covariance function given by
m̄(x)=KTxx(Kn+σ 2 modelIn) −1y,x∈Ω (1) k̄(x,x′)=k(x,x′)−KTxx(Kn+σ2modelIn)−1Kxx′ ,x,x′∈Ω, (2)
whereKxx=(k(x1,x),...,k(xn,x)) T and y=(y1,...,yn)T ∈Rn (Williams and Rasmussen, 2006, Eqs. 2.23-24).
The performance of GPR depends on how well the posterior approximates f as the number of training samples n tends to infinity. The distance of the posterior to the ground truth can be measured in various ways. We consider two such measures, namely the Bayesian generalization error (Seeger et al., 2008; Haussler and Opper, 1997; Opper and Vivarelli, 1999) and the excess mean squared error (Sollich and Halees, 2002; Le Gratiet and Garnier, 2015; Bordelon et al., 2020; Cui et al., 2021). Definition 1 (Bayesian generalization error). The Bayesian generalization error is defined as the Kullback-Leibler divergence between the true density q(y|x) and the Bayesian predictive density pn(y|x,Dn)= ∫ p(y|f(x))dΠn(f |Dn),
G(Dn)=
∫ q(x,y)log
q(y|x) pn(y|x,Dn) dxdy. (3)
A related quantity of interest is the stochastic complexity (SC), also known as the free energy, which is just the negative log-marginal likelihood. We shall primarily be concerned with a normalized version of the stochastic complexity which is defined as follows:
F 0(Dn)=−log Z(Dn)∏n i=1q(yi|xi)
=−log ∫∏n
i=1N (yi|f(xi),σ2model)dΠ0(f)∏n i=1q(yi|xi) . (4)
The generalization error (3) can be expressed in terms of the normalized SC as follows (Watanabe, 2009, Theorem 1.2):
G(Dn)=E(xn+1,yn+1)F 0(Dn+1)−F 0(Dn), (5)
whereDn+1 =Dn∪{(xn+1,yn+1)} is obtained by augmentingDn with a test point (xn+1,yn+1). If we only wish to measure the performance of the mean of the Bayesian posterior, then we can use the excess mean squared error: Definition 2 (Excess mean squared error). The excess mean squared error is defined as
M(Dn)=E(xn+1,yn+1)(m̄(xn+1)−yn+1) 2−σ2true =Exn+1(m̄(xn+1)−f(xn+1))2. (6)
Proposition 3 (Normalized stochastic complexity for GPR). Assume that σ2model =σ2true =σ2. The normalized SC F 0(Dn) (4) for GPR with prior GP(0,k) is given as
F 0(Dn)= 1 2 logdet(In+ Kn σ2 )+ 1 2σ2y T (In+ Kn σ2 ) −1y− 12σ2 (y−f(x)) T (y−f(x)), (7)
where =( 1,..., n)T . The expectation of the normalized SC w.r.t. the noise is given as E F 0(Dn)= 12 logdet ( In+ Kn σ2 ) − 12Tr ( In− ( In+ Kn σ2 )−1) + 12σ2 f(x) T ( In+ Kn σ2 )−1 f(x). (8)
This is a basic result and has applications in relation to model selection in GPR (Williams and Rasmussen, 2006). For completeness, we give a proof of Proposition 3 in Appendix B. Seeger et al. (2008, Theorem 1) gave an upper bound on the normalized stochastic complexity for the case when f lies in the reproducing kernel Hilbert space (RKHS) of the GP prior. It is well known, however, that sample paths of GP almost surely fall outside the corresponding RKHS (Van Der Vaart and Van Zanten, 2011) limiting the applicability of the result.
We next derive the asymptotics of E F 0(Dn), the expected generalization error E G(Dn) = E E(xn+1,yn+1)F 0(Dn+1)−E F 0(Dn), and the excess mean squared error E M(Dn).
3 ASYMPTOTIC ANALYSIS OF GP REGRESSION WITH POWER-LAW PRIORS
We begin by introducing some notations and assumptions. We assume that f ∈L2(Ω,ρ). By Mercer’s theorem (Williams and Rasmussen, 2006, Theorem 4.2), the covariance function of the GP prior can be decomposed as k(x1,x2)= ∑∞ p=1λpφp(x1)φp(x2), where (φp(x))p≥1 are the eigenfunctions
of the operator Lk : L2(Ω,ρ) 7→ L2(Ω,ρ); (Lkf)(x) = ∫
Ω k(x,s)f(s)dρ(s), and (λp)p≥1 are the
corresponding positive eigenvalues. We index the sequence of eigenvalues in decreasing order, that is λ1≥λ2≥···>0. The target function f(x) is decomposed into the orthonormal set (φp(x))p≥1 and its orthogonal complement {φp(x) :p≥1}⊥ as
f(x)= ∞∑ p=1 µpφp(x)+µ0φ0(x)∈L2(Ω,ρ), (9)
whereµ=(µ0,µ1,...,µp,...)T are the coefficients of the decomposition, andφ0(x) satisfies ‖φ0(x)‖2 = 1 and φ0(x) ∈ {φp(x) : p ≥ 1}⊥. For given sample inputs x, let φp(x) = (φp(x1), ... ,φp(xn))T , Φ = (φ0(x),φ1(x),...,φp(x),...) and Λ = diag{0,λ1,...,λp,...}. Then the covariance matrixKn can be written asKn=ΦΛΦT , and the function values on the sample inputs can be written as f(x)=Φµ.
We shall make the following assumptions in order to derive the power-law asymptotics of the normalized stochastic complexity and the generalization error of GPR: Assumption 4 (Power law decay of eigenvalues). The eigenvalues (λp)p≥1 follow the power law
Cλp −α≤λp≤Cλp−α, ∀p≥1 (10)
whereCλ,Cλ and α are three positive constants which satisfy 0<Cλ≤Cλ and α>1.
As mentioned in the introduction, this assumption, called the capacity condition, is fairly standard in kernel learning and is adopted in many recent works (Bordelon et al., 2020; Canatar et al., 2021; Jun et al., 2019; Bietti et al., 2021; Cui et al., 2021). Velikanov and Yarotsky (2021) derived the exact value of the exponent αwhen the kernel function has a homogeneous singularity on its diagonal, which is the case for instance for the arc-cosine kernel. Assumption 5 (Power law decay of coefficients of decomposition). Let Cµ,Cµ>0 and β>1/2 be positive constants and let {pi}i≥1 be an increasing integer sequence such that supi≥1(pi+1−pi)<∞. The coefficients (µp)p≥1 of the decomposition (9) of the target function follow the power law
|µp|≤Cµp−β , ∀p≥1 and |µpi |≥Cµpi−β , ∀i≥1. (11)
Since f ∈L2(Ω,ρ), we have ∑∞ p=0µ 2 p <∞. The condition β > 1/2 in Assumption 5 ensures that
the sum ∑∞ p=0µ 2 p does not diverge. When the orthonormal basis (φp(x))p is the Fourier basis or the spherical harmonics basis, the coefficients (µp)p decay at least as fast as a power law so long as the target function f(x) satisfies certain smoothness conditions (Bietti and Mairal, 2019). Velikanov and Yarotsky (2021) gave examples of some natural classes of functions for which Assumption 5 is satisfied, such as functions that have a bounded support with smooth boundary and are smooth on the interior of this support, and derived the corresponding exponents β. Assumption 6 (Boundedness of eigenfunctions). The eigenfunctions (φp(x))p≥0 satisfy
‖φ0‖∞≤Cφ and ‖φp‖∞≤Cφpτ , p≥1, (12) whereCφ and τ are two positive constants which satisfy τ < α−12 .
The second condition in (12) appears, for example, in Valdivia (2018, Hypothesis H1) and is less restrictive than the assumption of uniformly bounded eigenfunctions that has appeared in several other works in the GP literature, see, e.g., Braun (2006); Chatterji et al. (2019); Vakili et al. (2021).
Define
T1(Dn)= 1 2 logdet ( In+ ΦΛΦT σ2 ) − 12Tr ( In− ( In+ ΦΛΦT σ2 )−1) , (13)
T2(Dn)= 1 2σ2 f(x) T ( In+ ΦΛΦT σ2 )−1 f(x), (14)
G1(Dn)=E(xn+1,yn+1)(T1(Dn+1)−T1(Dn)), (15) G2(Dn)=E(xn+1,yn+1)(T2(Dn+1)−T2(Dn)). (16)
Using (8) and (5), we haveE F 0(Dn)=T1(Dn)+T2(Dn) andE G(Dn)=G1(Dn)+G2(Dn). Intuitively,G1 corresponds to the effect of the noise on the generalization error irrespective of the target function f , whereasG2 corresponds to the ability of the model to fit the target function. As we will see next in Theorems 9 and 11, ifα is large, then the error associated with the noise is smaller. When f is contained in the span of the eigenfunctions {φp}p≥1,G2 decreases with increasingn, but if f contains an orthogonal component, then the error remains constant and GP regression is not able to learn the target function.
3.1 ASYMPTOTICS OF THE NORMALIZED STOCHASTIC COMPLEXITY
We derive the asymptotics of the normalized SC (8) for the following two cases: µ0 = 0 and µ0>0. When µ0 =0, the target function f(x) lies in the span of all eigenfunctions with positive eigenvalues. Theorem 7 (Asymptotics of the normalized SC, µ0 = 0). Assume that µ0 = 0 and σ2model = σ 2 true = σ
2 = Θ(1). Under Assumptions 4, 5 and 6, with probability of at least 1−n−q over sample inputs (xi)ni=1, where 0 ≤ q < min{ (2β−1)(α−1−2τ) 4α2 , α−1−2τ 2α }, the expected normalized SC (8) has the asymptotic behavior: E F 0(Dn)= [ 1 2 logdet(I+ n σ2 Λ)− 1 2Tr ( I−(I+ nσ2 Λ)
−1)+ n2σ2µT (I+ nσ2 Λ)−1µ](1+o(1)) =Θ(nmax{ 1 α , 1−2β α +1}). (17)
The complete proof of Theorem 7 is given in Appendix D.1. We give a sketch of the proof below. In the sequel, we use the notationsO and Θ to denote the standard mathematical orders and the notation Õ to suppress logarithmic factors.
Proof sketch of Theorem 7. By (8), (13) and (14) we have E F 0(Dn) = T1(Dn) + T2(Dn). In order to analyze the terms T1(Dn) and T2(Dn), we will consider truncated versions of these quantities and bound the corresponding residual errors. Given a truncation parameter R ∈ N, let ΦR = (φ0(x),φ1(x),...,φR(x))∈Rn×R be the truncated matrix of eigenfunctions evaluated at the data points, ΛR = diag(0,λ1,...,λR)∈R(R+1)×(R+1) and µR = (µ0,µ1,...,µR)∈RR+1. We define the truncated version of T1(Dn) as follows:
T1,R(Dn)= 1 2 logdet ( In+ ΦRΛRΦ T R σ2 ) − 12Tr ( In−(In+ ΦRΛRΦ T R σ2 ) −1 ) . (18)
Similarly, define Φ>R = (φR+1(x), φR+2(x), ... , φp(x), ...), Λ>R = diag(λR+1, ... , λp, ...), fR(x) = ∑R p=1 µpφp(x), fR(x) = (fR(x1), ... , fR(xn))
T , f>R(x) = f(x) − fR(x), and f>R(x)=(f>R(x1),...,f>R(xn)) T . The truncated version of T2(Dn) is then defined as
T2,R(Dn)= 1 2σ2 fR(x) T (In+
ΦRΛRΦ T R
σ2 ) −1fR(x) T . (19)
The proof consists of three steps:
• Approximation step: In this step, we show that the asymptotics of T1,R resp. T2,R dominates that of the residuals, |T1,R(Dn)−T1(Dn)| resp. |T2,R(Dn)−T2(Dn)| (see Lemma 32). This builds upon first showing that ‖Φ>RΛ>RΦT>R‖2 =Õ(max{nR−α,n 1 2R 1−2α 2 ,R1−α}) (see Lemma 25) and then
choosingR=n 1 α+κ where 0<κ< α−1−2τ2α2 when we have ‖Φ>RΛ>RΦ T >R‖2 =o(1). Intuitively, the choice of the truncation parameterR is governed by the fact that λR=Θ(R−α)=n−1+κα=o(n−1).
• Decomposition step: In this step, we decompose T1,R into a term independent of ΦR and a series involving ΦTRΦR−nIR, and likewise for T2,R (see Lemma 34). This builds upon first showing using the Woodbury matrix identity (Williams and Rasmussen, 2006, §A.3) that
T1,R(Dn)= 1 2 logdet(IR+ 1 σ2 ΛRΦ T RΦR)− 12TrΦR(σ 2IR+ΛRΦ T RΦR) −1ΛRΦ T R, (20) T2,R(Dn)= 1 2σ2µ T RΦ T RΦR(σ 2IR+ΛRΦ T RΦR) −1µR, (21)
and then Taylor expanding the matrix inverse (σ2IR + ΛRΦTRΦR) −1 in (20) and (21) to show that the ΦR-independent terms in the decomposition of T1,R and T2,R are, respectively, 1 2 logdet(IR+ n σ2 ΛR)− 1 2Tr ( IR−(IR+ nσ2 ΛR) −1), and n2σ2µTR(IR+ nσ2 ΛR)−1µR. • Concentration step: Finally, we use concentration inequalities to show that these ΦR-independent
terms dominate the series involving ΦTRΦR−nIR (see Lemma 35) when we have T1,R(Dn)= ( 1 2 logdet(IR+ n σ2 ΛR)− 1 2Tr ( IR−(IR+ nσ2 ΛR) −1))(1+o(1))=Θ(n 1α ), T2,R(Dn)= ( n 2σ2µ T R(IR+ n σ2 ΛR) −1µR ) (1+o(1))= { Θ(nmax{0, 1−2β α +1}), α 6=2β−1,
Θ(logn), α=2β−1.
The key idea is to consider the matrix Λ1/2R (I+ n σ2 ΛR) −1/2ΦTRΦR(I+ n σ2 ΛR) −1/2Λ 1/2 R and show that it concentrates around nΛR(I+ nσ2 ) −1 (see Corollary 22). Note that an ordinary application
of the matrix Bernstein inequality to ΦTRΦR−nIR yields ‖ΦTRΦR−nI‖2 =O(R √ n), which is not
sufficient for our purposes, since this would giveO(R √ n)=o(n) only when α>2. In contrast, our results are valid forα>1 and cover cases of practical interest, e.g., the NTK of infinitely wide shallow ReLU network (Velikanov and Yarotsky, 2021) and the arc-cosine kernels over high-dimensional hyperspheres (Ronen et al., 2019) that have α=1+O( 1d ), where d is the input dimension.
For µ0>0, we note the following result: Theorem 8 (Asymptotics of the normalized SC, µ0 > 0). Assume µ0 > 0 and σ2model = σ 2 true = σ
2 = Θ(1). Under Assumptions 4, 5 and 6, with probability of at least 1−n−q over sample inputs (xi)ni=1, where 0≤q<min{ 2β−1 2 ,α}·min{ α−1−2τ 2α2 , 2β−1 α2 }. the expected normalized SC (8) has the asymptotic behavior: E F 0(Dn)= 12σ2µ 2 0n+o(n).
The proof of Theorem 8 is given in Appendix D.1 and follows from showing that when µ0 > 0, T2,R(Dn) =
( n
2σ2µ T R(IR+ n σ2 ΛR) −1µR ) (1 + o(1)) = 12σ2µ 2 0n + o(n) (see Lemma 38), which
dominates T1(Dn) and the residual |T2,R(Dn)−T2(Dn)|.
3.2 ASYMPTOTICS OF THE BAYESIAN GENERALIZATION ERROR
In this section, we derive the asymptotics of the expected generalization error E G(Dn) by analyzing the asymptotics of the componentsG1(Dn) andG2(Dn) in resp. (15) and (16) for the following two cases: µ0 =0 and µ0>0. First, we consider the case µ0 =0. Theorem 9 (Asymptotics of the Bayesian generalization error, µ0 = 0). Let Assumptions 4, 5, and 6 hold. Assume that µ0 = 0 and σ2model = σ 2 true = σ
2 = Θ(nt) where 1− α1+2τ < t < 1. Then with probability of at least 1−n−q over sample inputs (xi)ni=1 where 0≤ q < [α−(1+2τ)(1−t)](2β−1) 4α2 , the expectation of the Bayesian generalization error (3) w.r.t. the noise has the asymptotic behavior:
E G(Dn)= 1+o(1)2σ2 ( Tr(I+ nσ2 Λ) −1Λ−‖Λ1/2(I+ nσ2 Λ) −1‖2F +‖(I+ nσ2 Λ) −1µ‖22 ) =Θ(nmax{ (1−α)(1−t) α , (1−2β)(1−t) α }). (22)
The proof of Theorem 9 is given in Appendix D.2. Intuitively, for a given t, the exponent (1−α)(1−t)α in (22) captures the rate at which the model suppresses the noise, while the exponent (1−2β)(1−t)α captures the rate at which the model learns the target function. A larger β implies that the exponent (1−2β)(1−t)α is smaller and it is easier to learn the target. A larger α implies that the exponent (1−α)(1−t)α is smaller and the error associated with the noise is smaller as well. A larger α, however, also implies that the exponent (1−2β)(1−t)α is larger (recall that α>1 and β>1/2 by Assumptions 4 and 5, resp.), which means that it is harder to learn the target. Remark 10. If f ∼ GP(0, k), then using the Karhunen-Loève expansion we have f(x) = ∑∞ p=1 √ λpωpφp(x), where (ωp)∞p=1 are i.i.d. standard Gaussian variables. We can bound ωp almost surely as |ωp| ≤ C logp, where C = supp≥1 |ωp| logp is a finite constant. Comparing
with the expansion of f(x) in (9), we find that µp = √ λpωp =O(p
−α/2logp) =O(p−α/2+ε) where ε>0 is arbitrarily small. Choosing β=α/2−ε in (22), we have E G(Dn)=O(n 1 α−1+ 2ε α ). This rate matches that of an earlier result due to Sollich and Halees (2002), where it is shown that the asymptotic learning curve (as measured by the expectation of the excess mean squared error, EfM(Dn)) scales as n 1 α−1 when the model is correctly specified, i.e., f is a sample from the same Gaussian process GP(0,k), and the eigenvalues decay as a power law for large i, λi∼ iα.
For µ0>0, we note the following result: Theorem 11 (Asymptotics of the Bayesian generalization error, µ0>0). Let Assumptions 4, 5, and 6 hold. Assume that µ0 > 0 and σ2model = σ 2 true = σ
2 = Θ(nt) where 1− α1+2τ < t < 1. Then with probability of at least 1−n−q over sample inputs (xi)ni=1, where 0≤ q< [α−(1+2τ)(1−t)](2β−1) 4α2 , the expectation of the Bayesian generalization error (3) w.r.t. the noise has the asymptotic behavior: E G(Dn)= 12σ2µ 2 0+o(1).
In general, if µ0> 0, then the generalization error remains constant when n→∞. This means that if the target function contains a component in the kernel of the operatorLk, then GP regression is not able to learn the target function. The proof of Theorem 11 is given in Appendix D.2.
3.3 ASYMPTOTICS OF THE EXCESS MEAN SQUARED ERROR
In this section we derive the asymptotics of the excess mean squared error in Definition 2. Theorem 12 (Asymptotics of excess mean squared error). Let Assumptions 4, 5, and 6 hold. Assume σ2model =Θ(n t) where 1− α1+2τ <t<1. Then with probability of at least 1−n −q over sample inputs (xi) n i=1, where 0≤q< [α−(1+2τ)(1−t)](2β−1) 4α2 , the excess mean squared error (6) has the asymptotic:
E M(Dn)=(1+o(1)) [ σ2true σ2model ( Tr(I+ n σ2model Λ)−1Λ−‖Λ1/2(I+ n σ2model Λ)−1‖2F ) +‖(I+ n
σ2model Λ)−1µ‖22
] =Θ ( max{σ2truen 1−α−t α ,n (1−2β)(1−t) α } ) when µ0 =0, and E M(Dn)=µ20+o(1), when µ0>0.
The proof of Theorem 12 uses similar techniques as Theorem 9 and is given in Appendix D.3.
Remark 13 (Correspondence with kernel ridge regression). The kernel ridge regression (KRR) estimator arises as a solution to the optimization problem
f̂=argmin f∈Hk
1
n n∑ i=1 (f(xi)−yi)2+λ‖f‖2Hk , (23)
where the hypothesis spaceHk is chosen to be an RKHS, and λ > 0 is a regularization parameter. The solution to (23) is unique as a function, and is given by f̂(x) = KTxx(Kn +nλIn)
−1y, which coincides with the posterior mean function m̄(x) of the GPR (1) if σ2model = nλ (Kanagawa et al., 2018, Proposition 3.6). Thus, the additive Gaussian noise in GPR plays the role of regularization in KRR. Leveraging this well known equivalence between GPR and KRR we observe that Theorem 12 also describes the generalization error of KRR as measured by the excess mean squared error.
Remark 14. Cui et al. (2021) derived the asymptotics of the expected excess mean-squared error for different regularization strengths and different scales of noise. In particular, for KRR with Gaussian design where Λ1/2R (φ1(x),...,φR(x))) is assumed to follow a Gaussian distributionN (0,ΛR), and regularization λ=nt−1 where 1−α≤ t, Cui et al. (2021, Eq. 10) showed that
E{xi}ni=1E M(Dn)=O ( max{σ2truen 1−α−t α ,n (1−2β)(1−t) α } ) . (24)
Let δ = n−q, where 0 ≤ q < [α−(1+2τ)(1−t)](2β−1)4α2 . By Markov’s inequality, this implies that with probability of at least 1 − δ, E M(Dn) = O( 1δ max{σ 2 truen 1−α−t α , n (1−2β)(1−t) α }) = O(nqmax{σ2truen 1−α−t α ,n (1−2β)(1−t) α }). Theorem 12 improves upon this by showing that with probability of at least 1−δ, we have an optimal bound E M(Dn)=Θ(max{σ2truen 1−α−t α ,n (1−2β)(1−t) α }). Furthermore, in contrast to the approach by Cui et al. (2021), we have no requirement on the distribution of φp(x), and hence our result is more generally applicable. For example, Theorem 12 can be applied to KRR with the arc-cosine kernel when the Gaussian design assumption is not valid. In the noiseless setting (σtrue =0) with constant regularization (t=0), Theorem 12 implies that the mean squared error behaves as Θ(n 1−2β α ). This recovers a result in Bordelon et al. (2020, §2.2).
4 EXPERIMENTS
We illustrate our theory on a few toy experiments. We let the input x be uniformly distributed on a unit circle, i.e., Ω = S1 and ρ = U(S1). The points on S1 can be represented by x= (cosθ,sinθ) where θ ∈ [−π,π). We use the first order arc-cosine kernel function without bias, k
(1) w/o bias(x1,x2) = 1 π (sinψ+ (π−ψ)cosψ), where ψ = 〈x1,x2〉 is the angle between x1 and x2. Cho and Saul (2009) showed that this kernel is the conjugate kernel of an infinitely wide shallow ReLU network with two inputs and no biases in the hidden layer. GP regression with prior GP(0,k) corresponds to Bayesian training of this network (Lee et al., 2018). The eigenvalues and eigenfunctions of the kernel are λ1 = 4π2 , λ2 = λ3 = 1 4 , λ2p = λ2p+1 = 4 π2((2p−2)2−1)2 , p ≥ 2 and φ1(θ) = 1, φ2(θ) = √ 2 2 cosθ, φ3(θ) = √ 2 2 sinθ, φ2p(θ) = √ 2 2 cos(2p− 2)θ,φ2p+1(θ) = √ 2
2 sin(2p− 2)θ, p≥ 2. Hence Assumption 4 is satisfied with α= 4, and Assumption 6 is satisfied with ‖φp‖∞≤ √ 2
2 , p≥ 1 and τ=0. We consider the target functions in Table 1, which satisfy Assumption 5 with the indicated β, and µ0 indicates whether the function lies in the span of eigenfunctions of the kernel.
The training and test data are generated as follows: We independently sample training inputs x1, ... ,xn and test input xn+1 from U(S1) and training outputs yi, i = 1, ... ,n from N (f(xi),σ2), where we choose σ = 0.1. The Bayesian predictive density conditioned on the test point xn+1 N (m̄(xn+1),k̄(xn+1,xn+1)) is obtained by (1) and (2). We compute the normalized SC by (7) and the Bayesian generalization error by the Kullback-Leibler divergence betweenN (f(xn+1),σ2) and N (m̄(xn+1),k̄(xn+1,xn+1)). For each target we conduct GPR 20 times and report the mean and standard deviation of the normalized SC and the Bayesian generalization error in Figure 1, which agree with the asymptotics predicted in Theorems 7 and 9. In Appendix A, we show more experiments confirming our theory for zero- and second- order arc-cosine kernels, with and without biases.
k
(1) w/o bias, their values of β and µ0, and theoretical rates for the normalized SC and the Bayesian
generalization error from our theorems.
k
(1) w/o bias and the target functions in Table 1. The orange curves show the linear regression fit for the
experimental values (in blue) of the log Bayesian generalization error as a function of log n.
5 CONCLUSION
We described the learning curves for GPR for the case that the kernel and target function follow a power law. This setting is frequently encountered in kernel learning and relates to recent advances on neural networks. Our approach is based on a tight analysis of the concentration of the inner product of empirical eigenfunctions ΦTΦ around nI . This allowed us to obtain more general results with more realistic assumptions than previous works. In particular, we recovered some results on learning curves for GPR and KRR previously obtained under more restricted settings (vide Remarks 10 and 14).
We showed that when β≥α/2, meaning that the target function has a compact representation in terms of the eigenfunctions of the kernel, the learning rate is as good as in the correctly specified case. In addition, our result allows us to interpret β from a spectral bias perspective. When 12 <β ≤ α 2 , the larger the value of β, the faster the decay of the generalization error. This implies that low-frequency functions are learned faster in terms of the number of training data points.
By leveraging the equivalence between GPR and KRR, we obtained a result on the generalization error of KRR. In the infinite-width limit, training fully-connected deep NNs with gradient descent and infinitesimally small learning rate under least-squared loss is equivalent to solving KRR with respect to the NTK (Jacot et al., 2018; Lee et al., 2019; Domingos, 2020), which in several cases is known to have a power-law spectrum (Velikanov and Yarotsky, 2021). Hence our methods can be applied to study the generalization error of infinitely wide neural networks. In future work, it would be interesting to estimate the values of α and β for the NTK and the NNGP kernel of deep fully-connected or convolutional NNs and real data distributions and test our theory in these cases. Similarly, it would be interesting to consider extensions to finite width kernels.
REFERENCES
S. Amari and N. Murata. Statistical theory of learning curves under entropic loss criterion. Neural Computation, 5(1):140–153, 1993.
S. Amari, N. Fujita, and S. Shinomoto. Four types of learning curves. Neural Computation, 4(4): 605–618, 1992.
S. Arora, S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems, volume 32, pages 8139–8148, 2019.
Y. Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701, 2021.
A. R. Barron. Information-theoretic characterization of Bayes performance and the choice of priors in parametric and nonparametric problems. In D. A. Bernardo J., Berger J. and S. A., editors, Bayesian statistics, volume 6, pages 27–52. Oxford University Press, 1998.
M. Belkin, S. Ma, and S. Mandal. To understand deep learning we need to understand kernel learning. In Proceedings of the 35th International Conference on Machine Learning (ICML), pages 541–549, 2018.
A. Bietti and J. Mairal. On the inductive bias of neural tangent kernels. In Advances in Neural Information Processing Systems, volume 32, pages 12873–12884, 2019.
A. Bietti, L. Venturi, and J. Bruna. On the sample complexity of learning with geometric stability. arXiv preprint arXiv:2106.07148, 2021.
G. Blanchard and N. Mücke. Optimal rates for regularization of statistical inverse learning problems. Foundations of Computational Mathematics, 18(4):971–1013, 2018.
B. Bordelon, A. Canatar, and C. Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 1024–1034, 2020.
O. Bousquet, S. Hanneke, S. Moran, R. van Handel, and A. Yehudayoff. A theory of universal learning. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing (STOC), pages 532–541, 2021.
M. L. Braun. Accurate error bounds for the eigenvalues of the kernel matrix. The Journal of Machine Learning Research, 7:2303–2328, 2006.
A. Canatar, B. Bordelon, and C. Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature communications, 12(1):1–12, 2021.
A. Caponnetto and E. De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331–368, 2007.
N. Chatterji, A. Pacchiano, and P. Bartlett. Online learning with kernel losses. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 971–980, 2019.
Y. Cho and L. K. Saul. Kernel methods for deep learning. In Advances in Neural Information Processing Systems, volume 22, pages 342–350, 2009.
H. Cui, B. Loureiro, F. Krzakala, and L. Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. arXiv preprint arXiv:2105.15004, 2021.
A. Daniely, R. Frostig, and Y. Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. In Advances In Neural Information Processing Systems, volume 29, pages 2253–2261, 2016.
A. G. de G. Matthews, J. Hron, M. Rowland, R. E. Turner, and Z. Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations, 2018.
P. Domingos. Every model learned by gradient descent is approximately a kernel machine. arXiv preprint arXiv:2012.00152, 2020.
S. Fischer and I. Steinwart. Sobolev norm learning rates for regularized least-squares algorithms. Journal of Machine Learning Research, 21:1–38, 2020.
A. Garriga-Alonso, C. E. Rasmussen, and L. Aitchison. Deep convolutional networks as shallow gaussian processes. In International Conference on Learning Representations, 2019.
D. Haussler and M. Opper. Mutual information, metric entropy and cumulative relative entropy risk. The Annals of Statistics, 25(6):2451–2492, 1997.
D. Haussler, M. Kearns, H. S. Seung, and N. Tishby. Rigorous learning curve bounds from statistical mechanics. Machine Learning, 25(2-3):195–236, 1996.
J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. Patwary, M. Ali, Y. Yang, and Y. Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017.
A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, volume 31, pages 8571–8580, 2018.
K.-S. Jun, A. Cutkosky, and F. Orabona. Kernel truncated randomized ridge regression: Optimal rates and low noise acceleration. Advances in Neural Information Processing Systems, 32:15358–15367, 2019.
M. Kanagawa, P. Hennig, D. Sejdinovic, and B. K. Sriperumbudur. Gaussian processes and kernel methods: A review on connections and equivalences. arXiv preprint arXiv:1807.02582, 2018.
L. Le Gratiet and J. Garnier. Asymptotic analysis of the learning curve for Gaussian process regression. Machine Learning, 98(3):407–433, 2015.
J. Lee, J. Sohl-Dickstein, J. Pennington, R. Novak, S. Schoenholz, and Y. Bahri. Deep neural networks as gaussian processes. In International Conference on Learning Representations, 2018.
J. Lee, L. Xiao, S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems, volume 32, pages 8572–8583, 2019.
J. Lee, S. Schoenholz, J. Pennington, B. Adlam, L. Xiao, R. Novak, and J. Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. In Advances in Neural Information Processing Systems, volume 33, pages 15156–15172, 2020.
M. Loog, T. Viering, and A. Mey. Minimizers of the empirical risk and risk monotonicity. In Advances in Neural Information Processing Systems, volume 32, pages 7478–7487, 2019.
D. Malzahn and M. Opper. Learning curves for Gaussian processes regression: A framework for good approximations. In Advances in Neural Information Processing Systems, volume 13, pages 273–279, 2001a.
D. Malzahn and M. Opper. Learning curves for Gaussian processes models: Fluctuations and universality. In International Conference on Artificial Neural Networks, pages 271–276, 2001b.
R. M. Neal. Bayesian Learning for Neural Networks. Springer-Verlag, Berlin, Heidelberg, 1996. ISBN 0387947248.
A. Nitanda and T. Suzuki. Optimal rates for averaged stochastic gradient descent under neural tangent kernel regime. In International Conference on Learning Representations, 2021.
R. Novak, L. Xiao, Y. Bahri, J. Lee, G. Yang, D. A. Abolafia, J. Pennington, and J. Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. In International Conference on Learning Representations, 2019.
M. Opper and D. Malzahn. A variational approach to learning curves. In Advances in Neural Information Processing Systems, volume 14, pages 463–469, 2002.
M. Opper and F. Vivarelli. General bounds on Bayes errors for regression with Gaussian processes. In Advances in Neural Information Processing Systems, volume 11, pages 302–308, 1999.
P. Orbanz and Y. W. Teh. Bayesian nonparametric models. In Encyclopedia of Machine Learning, pages 81–89. Springer, 2010.
K. Ritter, G. W. Wasilkowski, and H. Woźniakowski. Multivariate integration and approximation for random fields satisfying Sacks-Ylvisaker conditions. The Annals of Applied Probability, pages 518–540, 1995.
B. Ronen, D. Jacobs, Y. Kasten, and S. Kritchman. The convergence rate of neural networks for learned functions of different frequencies. Advances in Neural Information Processing Systems, 32:4761–4771, 2019.
M. W. Seeger, S. M. Kakade, and D. P. Foster. Information consistency of nonparametric Gaussian process methods. IEEE Transactions on Information Theory, 54(5):2376–2382, 2008.
P. Sollich. Learning curves for Gaussian processes. In Advances in Neural Information Processing Systems, volume 11, pages 344–350, 1999.
P. Sollich. Gaussian process regression with mismatched models. In Advances in Neural Information Processing Systems, volume 13, pages 519–526, 2001.
P. Sollich and A. Halees. Learning curves for Gaussian process regression: Approximations and bounds. Neural Computation, 14(6):1393–1428, 2002.
S. Spigler, M. Geiger, and M. Wyart. Asymptotic learning curves of kernel methods: empirical data versus teacher–student paradigm. Journal of Statistical Mechanics: Theory and Experiment, 2020 (12):124001, 2020.
M. L. Stein. Interpolation of spatial data: Some theory for kriging. Springer Science & Business Media, 2012.
I. Steinwart, D. R. Hush, C. Scovel, et al. Optimal rates for regularized least squares regression. In Conference on Learning Theory, pages 79–93, 2009.
J. A. Tropp. User-friendly tail bounds for sums of random matrices. Foundations of computational mathematics, 12(4):389–434, 2012.
S. Vakili, K. Khezeli, and V. Picheny. On information gain and regret bounds in Gaussian process bandits. In International Conference on Artificial Intelligence and Statistics, pages 82–90, 2021.
E. A. Valdivia. Relative concentration bounds for the spectrum of kernel matrices. arXiv preprint arXiv:1812.02108, 2018.
A. Van Der Vaart and H. Van Zanten. Information rates of nonparametric Gaussian process methods. Journal of Machine Learning Research, 12(6), 2011.
M. Velikanov and D. Yarotsky. Universal scaling laws in the gradient descent training of neural networks. arXiv preprint arXiv:2105.00507, 2021.
T. Viering and M. Loog. The shape of learning curves: A review. arXiv preprint arXiv:2103.10948, 2021.
T. Viering, A. Mey, and M. Loog. Open problem: Monotonicity of learning. In Conference on Learning Theory, pages 3198–3201, 2019.
S. Watanabe. Algebraic Geometry and Statistical Learning Theory. Cambridge University Press, 2009.
H. Widom. Asymptotic behavior of the eigenvalues of certain integral equations. Transactions of the American Mathematical Society, 109(2):278–295, 1963.
C. K. Williams. Computing with infinite networks. In Advances in Neural Information Processing Systems, volume 9, pages 295–301, 1997.
C. K. Williams and C. E. Rasmussen. Gaussian processes for machine learning. MIT press, 2006.
C. K. Williams and F. Vivarelli. Upper and lower bounds on the learning curve for Gaussian processes. Machine Learning, 40(1):77–102, 2000.
G. Yang. Wide feedforward or recurrent neural networks of any architecture are gaussian processes. In Advances in Neural Information Processing Systems, volume 32, pages 9951–9960, 2019.
G. Yang and H. Salman. A fine-grained spectral perspective on neural networks. arXiv preprint arXiv:1907.10599, 2019.
APPENDIX
A EXPERIMENTS FOR ARC-COSINE KERNELS OF DIFFERENT ORDERS
Consider the first order arc-cosine kernel function with biases,
k (1) w/ bias(x1,x2)= 1 π (sinψ̄+(π−ψ̄)cosψ̄),where ψ̄=arccos ( 1 2 (〈x1,x2〉+1) ) . (25)
Ronen et al. (2019) showed that this kernel is the conjugate kernel of an infinitely wide shallow ReLU network with two inputs and one hidden layer with biases, whose eigenvalues satisfy Assumption 4 with α = 4. The eigenfunctions of this kernel are the same as that of the first-order arc-cosine kernel without biases, k(1)w/o bias in Section 4. We consider the target functions in Table 3, which satisfy Assumption 5 with the indicated β, and µ0 indicates whether the function lies in the span of eigenfunctions of the kernel. For each target we conduct GPR 20 times and report the mean and standard deviation of the normalized SC and the Bayesian generalization error in Figure 3, which agree with the asymptotics predicted in Theorems 7 and 9.
Table 2 summarizes all the different kernel functions that we consider in our experiments with pointers to the corresponding tables and figures.
Summarizing the observations from these experiments, we see that the smoothness of the activation function (which is controlled by the order of the arc-cosine kernel) influences the decay rate α of the eigenvalues. In general, when the activation function is smoother, the decay rate α is larger. Theorem 9 then implies that smooth activation functions are more capable in suppressing noise but slower in learning the target. We also observe that networks with biases are more capable at learning functions compared to networks without bias. For example, the function cos(2θ) cannot be learned by the zero order arc-cosine kernel without biases (see Table 6 and Figure 6), but it can be learned by the zero order arc-cosine kernel with biases (see Table 7 and Figure 7).
k
(1) w/ bias and the target functions in Table 3. The orange curves show the linear regression fit for the
experimental values (in blue) of the log Bayesian generalization error as a function of log n.
k
(2) w/o bias and the target functions in Table 4.
k
(2) w/ bias and the target functions in Table 5.
k
(0) w/o bias and the target functions in Table 6.
k
(0) w/ bias and the target functions in Table 7.
B PROOFS RELATED TO THE MARGINAL LIKELIHOOD
Proof of Proposition 3. Let ȳ = (ȳ1,...,ȳn)T be the outputs of the GP regression model on training inputs x. Under the GP prior, the prior distribution of ȳ isN (0,Kn). Then the evidence of the model is given as follows:
Zn= ∫ Rn ( n∏ i=1 1√ 2πσ e− (ȳi−yi) 2 2σ2 ) 1 (2π)n/2det(Kn)1/2 e− 1 2 ȳ TK−1n ȳdȳ
= 1
(2π)nσndet(Kn)1/2 ∫ Rn e− 1 2 ȳ T (K−1n + 1 σ2 I)ȳ+ 1 σ2 ȳTy− 1 2σ2 yTydȳ.
(26)
Letting K̃−1n =K −1 n + 1 σ2 I and µ= 1 σ2 K̃ny, we have
Zn= 1
(2π)nσndet(Kn)1/2 ∫ Rn e− 1 2 (ȳ−µ) T K̃−1n (ȳ−µ)− 12σ2 y Ty+ 12µ T K̃−1n µdȳ
= 1
(2π)nσndet(Kn)1/2 (2π)n/2det(K̃n)
1/2e− 1 2σ2 yTy+ 12µ T K̃−1n µ
= det(K̃n)
1/2
(2π)n/2σndet(Kn)1/2 e− 1 2σ2 yTy+ 12µ T K̃−1n µ.
(27)
The normalized evidence is
Z0n= Zn
(2π)−n/2σ−ne− 1 2σ2 (y−f(x))T (y−f(x))
= det(K̃n)
1/2
det(Kn)1/2 e− 1 2σ2 yTy+ 12µ T K̃−1n µ+ 1 2σ2 (y−f(x))T (y−f(x)).
(28)
So the normalized stochastic complexity is
F 0(Dn)=−logZ0n
=−1 2 logdet(K̃n) 1/2+ 1 2 logdet(Kn) 1/2+ 1 2σ2 yTy− 1 2 µT K̃−1n µ− 1 2σ2 (y−f(x))T (y−f(x)) =−1 2 logdet(K−1n + 1 σ2 I)−1+ 1 2 logdet(Kn)+ 1 2σ2 yTy− 1 2σ4 yT (K−1n + 1 σ2 I)−1y
− 1 2σ2 (y−f(x))T (y−f(x))
= 1
2 logdet(I+ Kn σ2 )+ 1 2σ2 yT (I+ Kn σ2 )−1y− 1 2σ2 (y−f(x))T (y−f(x)).
= 1
2 logdet(I+ Kn σ2 )+ 1 2σ2 f(x)T (I+ Kn σ2 )−1f(x)+ 1 2σ2 T (I+ Kn σ2 )−1 − 1 2σ2 T
+ 1
2σ2 T (I+ Kn σ2 )−1f(x)
. (29)
After taking the expectation over noises , we get
E F 0(Dn)= 1
2 logdet(I+ Kn σ2 )+ 1 2σ2 f(x)T (I+ Kn σ2 )−1f(x)− 1 2 Tr(I−(I+Kn σ2 )−1). (30)
This concludes the proof.
C HELPER LEMMAS
Lemma 15. Assume that m → ∞ as n → ∞. Given constants a1, a2, s1, s2 > 0, if s1 > 1 and s2s3>s1−1 , we have that
R∑ i=1
a1i −s1
(1+a2mi−s2)s3 =Θ(m
1−s1 s2 ). (31)
If s1>1 and s2s3 =s1−1, we have that R∑ i=1 a1i −s1 (1+a2mi−s2)s3 =Θ(m−s3 logm). (32)
If s1>1 and s2s3<s1−1, we have that R∑ i=1 a1i −s1 (1+a2mi−s2)s3 =Θ(m−s3). (33)
Overall, if s1>1 andm→∞, R∑ i=1 a1i −s1 (1+a2mi−s2)s3 = { Θ(mmax{−s3, 1−s1 s2 }), s2s3 6=s1−1, Θ(m 1−s1 s2 logm), s2s3 =s1−1.
(34)
Proof of Lemma 15. First, when s1>1 and s2s3>s1−1, we have that R∑ i=1 a1i −s1 (1+a2mi−s2)s3 ≤ a1 (1+a2m)s3 + ∫ [1,+∞] a1x −s1 (1+a2mx−s2)s3 dx
= a1
(1+a2m)s3 +m
1−s1 s2 ∫ [1,+∞]
a1( x m1/s2 )−s1
(1+a2( x m1/s2 )−s2)s3
d x
m1/s2
= a1
(1+a2m)s3 +m
1−s1 s2 ∫ [1/m1/s2 ,+∞]
a1x −s1
(1+a2x−s2)s3 dx
=Θ(m 1−s1 s2 ).
On the other hand, we have R∑ i=1 a1i −s1 (1+a2mi−s2)s3 ≥ ∫ [1,R+1]
a1x −s1
(1+a2mx−s2)s3 dx
=m 1−s1 s2 ∫ [1,R+1]
a1( x m1/s2 )−s1
(1+a2( x m1/s2 )−s2)s3
d x
m1/s2
=m 1−s1 s2 ∫ [1/m1/s2 ,(R+1)/m1/s2 ]
a1x −s1
(1+a2x−s2)s3 dx
=Θ(m 1−s1 s2 ).
Second, when s1>1 and s2s3 =s1−1, we have that R∑ i=1 a1i −s1 (1+a2mi−s2)s3 ≤ a1 (1+a2m)s3 +m 1−s1 s2 ∫ [1/m1/s2 ,+∞]
a1x −s1
(1+a2x−s2)s3 dx
≤ a1 (1+a2m)s3 +m 1−s1 s2 O(logm(1/s2)) =Θ(m 1−s1 s2 logn).
On the other hand, we have R∑ i=1 a1i −s1 (1+a2mi−s2)s3 ≥ ∫ [1,R+1]
a1x −s1
(1+a2mx−s2)s3 dx
=m 1−s1 s2 ∫ [1,R+1]
a1( x m1/s2 )−s1
(1+a2( x m1/s2 )−s2)s3
d x
m1/s2
=m 1−s1 s2 ∫ [1/m1/s2 ,(R+1)/m1/s2 ]
a1x −s1
(1+a2x−s2)s3 dx
=Θ(m 1−s1 s2 logn).
Third, when s1>1 and s2s3<s1−1, we have that R∑ i=1 a1i −s1 (1+a2mi−s2)s3 ≤ a1 (1+a2m)s3 +m 1−s1 s2 ∫ [1/m1/s2 ,+∞]
a1x −s1
(1+a2x−s2)s3 dx
≤ a1 (1+a2m)s3 +m 1−s1 s2 Θ(m(−1/s2)(1−s1+s2s3)) =Θ(m−s3).
On the other hand, we have R∑ i=1 a1i −s1 (1+a2mi−s2)s3 ≤ a1 (1+a2m)s3 +m 1−s1 s2 ∫ [2/m1/s2 ,(R+1)/m1/s2 ]
a1x −s1
(1+a2x−s2)s3 dx
≤ a1 (1+a2m)s3 +m 1−s1 s2 Θ(m(−1/s2)(1−s1+s2s3)) =Θ(m−s3).
Overall, if s1>1, R∑ i=1
a1i −s1
(1+a2mi−s2)s3 =
{ Θ(mmax{−s3, 1−s1 s2 }), s2s3 6=s1−1,
Θ(m−s3 logn), s2s3 =s1−1. (35)
Lemma 16. Assume thatR=m 1s2 +κ for κ>0. Given constants a1,a2,s1,s2>0 , if s1≤1, we have that
R∑ i=1
a1i −s1
(1+a2mi−s2)s3 =Õ(max{m−s3 ,R1−s1}). (36)
Proof of Lemma 16. First, when s1≤1 and s2s3>s1−1, we have that R∑ i=1 a1i −s1 (1+a2mi−s2)s3 ≤ a1 (1+a2m)s3 + ∫ [1,R] a1x −s1 (1+a2mx−s2)s3 dx
= a1
(1+a2m)s3 +m
1−s1 s2 ∫ [1,R]
a1( x m1/s2 )−s1
(1+a2( x m1/s2 )−s2)s3
d x
m1/s2
= a1
(1+a2m)s3 +m
1−s1 s2 ∫ [1/m1/s2 ,R/m1/s2 ]
a1x −s1
(1+a2x−s2)s3 dx
= a1
(1+a2m)s3 +Õ(m
1−s1 s2 ( R
m1/s2 )1−s1)
=Õ(max{m−s3 ,R1−s1}). Second, when s1≤1 and s2s3≤s1−1, we have that
R∑ i=1
a1i −s1
(1+a2mi−s2)s3 ≤ a1 (1+a2m)s3 +m 1−s1 s2 ∫ [1/m1/s2 ,R/m1/s2 ]
a1x −s1
(1+a2x−s2)s3 dx
≤ a1 (1+a2m)s3 +m 1−s1 s2 Õ(m(−1/s2)(1−s1+s2s3)+( R m1/s2 )1−s1)
=Õ(max{m−s3 ,R1−s1}). Overall, if s1≤1,
R∑ i=1
a1i −s1
(1+a2mi−s2)s3 =Õ(max{m−s3 ,R1−s1}). (37)
Lemma 17. Assume that f ∈ L2(Ω,ρ). Consider the random vector f(x) = (f(x1), ... ,f(xn))T , where x1,...,xn are drawn i.i.d from ρ. Then with probability of at least 1−δ1, we have
‖f(x)‖22 = n∑ i=1 f2(xi)=Õ ( ( 1δ1 +1)n‖f‖ 2 2 ) ,
where ‖f‖22 = ∫ x∈Ωf 2(x)dρ(x).
Proof of Lemma 17. Given a positive numberC≥‖f‖22, applying Markov’s inequality we have
P(f2(X)>C)≤ 1 C ‖f‖22.
LetA be the event that for all sample inputs (xi)ni=1, f 2(xi)≤C. Then
P(A)≥1−nP(f2(X)>C)≥1− 1 C n‖f‖22. (38)
Define f̄2(x) = min{f2(x),C}. Then Ef̄2(X) ≤ Ef2(X) = ‖f‖22. So |f̄2(X) − Ef̄2(X)| ≤ max{C,‖f‖22}=C Since 0≤ f̄2(x)≤C, we have
E(f̄4(X))≤CE(f̄2(X))≤C‖f‖22. (39) So we have E|f̄2(X)−Ef̄2(X)|2≤E(f̄4(X))≤C‖f‖22. (40) Applying Bernstein’s inequality, we have
P( n∑ i=1 f̄2(xi)>t+nEf̄2(X))≤exp
( − t 2
2(nE|f̄2(X)−Ef̄2(X)|2)+ Ct3 )
)
≤exp ( − t 2
2(nC‖f‖22+ Ct3 )
)
≤exp ( − t 2
4max{nC‖f‖22,Ct3 }
) .
Hence, with probability of at least 1−δ1/2 we have n∑ i=1 f̄2(xi)≤max {√ 4Clog 2 δ1 n‖f‖22, 4C 3 log 2 δ1 } +nEf̄2(X)
≤max {√ 4Clog 2
δ1 n‖f‖22,
4C
3 log
2
δ1
} +n‖f‖22.
(41)
When event A happens, f2(xi) = f̄2(xi) for all sample inputs. According to (38) and (41), with probability at least 1− 1Cn‖f‖ 2 2−δ1/2, we have
n∑ i=1 f2(xi)= n∑ i=1 f̄2(xi)≤max {√ 4Clog 2 δ1 n‖f‖22, 4C 3 log 2 δ1 } +n‖f‖22.
ChoosingC= 2δ1n‖f‖ 2 2, with probability of at least 1−δ1 we have
n∑ i=1 f2(xi)= n∑ i=1 f̄2(xi)≤max {√ 8 δ1 log 2 δ1 n2‖f‖42, 8 3δ1 n‖f‖22log 2 δ1 } +n‖f‖22 =Õ ( ( 1δ1 +1)n‖f‖ 2 2 ) .
Lemma 18. Assume that f ∈ L2(Ω,ρ). Consider the random vector f(x) = (f(x1), ... ,f(xn))T , where x1,...,xn are drawn i.i.d from ρ. Assume that ‖f‖∞= supx∈Ωf(x)≤C. With probability of at least 1−δ1, we have
‖f(x)‖22 =Õ (√ C2n‖f‖22+C2 ) +n‖f‖22,
where ‖f‖22 = ∫ x∈Ωf 2(x)dρ(x).
Proof of Lemma 18. We have |f2(X)−Ef2(X)|≤max{C2,‖f‖22}=C2 Since 0≤ f2(x)≤C, we have E(f4(X))≤C2E(f2(X))≤C2‖f‖22. (42) So we have E|f2(X)−Ef2(X)|2≤E(f4(X))≤C2‖f‖22. (43) Applying Bernstein’s inequality, we have
P( n∑ i=1 f2(xi)>t+nEf2(X))≤exp
( − t 2
2(nE|f2(X)−Ef2(X)|2)+ C2t3 )
)
≤exp ( − t 2
2(nC2‖f‖22+ C 2t 3 )
)
≤exp ( − t 2
4max{nC2‖f‖22,C 2t 3 }
) .
Hence, with probability of at least 1−δ1 we have n∑ i=1 f2(xi)≤max {√ 4C2log 1 δ1 n‖f‖22, 4C2 3 log 1 δ1 } +nEf2(X)
≤Õ ( max {√ C2n‖f‖22,C2 }) +n‖f‖22
≤Õ (√ C2n‖f‖22+C2 ) +n‖f‖22.
(44)
For the proofs in the reminder of this section, the definitions of the relevant quantities are given in Section 3.
Corollary 19. With probability of at least 1−δ1, we have ‖f>R(x)‖22 =Õ ( ( 1δ1 +1)nR 1−2β ) .
Proof of Corollary 19. The L2 norm of f>R(x) is given by ‖f>R‖22 = ∑∞ p=R+1µ 2 p ≤ Cµ 2β−1R
1−2β . Applying Lemma 17 we get the result.
Corollary 20. For any ν∈RR, with probability of at least 1−δ1 we have ‖ΦRν‖22 =Õ ( ( 1δ1 +1)n‖ν‖ 2 2 ) .
Proof of Corollary 20. Let g(x)= ∑R p=1νpφp(x). Then ΦRν=g(x). The L2 norm of g(x) is given
by ‖g‖22 = ∑R p=1ν 2 p =‖ν‖22. Applying Lemma 17 we get the result.
Next we consider the quantity, ΦTRΦR−nI . The key tool that we use is the matrix Bernstein inequality that describes the upper tail of a sum of independent zero-mean random matrices.
Lemma 21. Let D = diag{d1, ... , dR}, d1, ... , dR > 0 and dmax = max{d1, ... , dR}. Let M=max{ ∑R p=0d 2 p‖φp‖2∞,d2max}. Then with probability of at least 1−δ, we have
‖D(ΦTRΦR−nI)D‖2≤max {√ nd2maxM log R δ ,M log R δ ) } . (45)
Proof of Lemma 21. Let Yj = (φ1(xj), ... , φR(xj))T and Zj = DYj . It is easy to verify that E(ZjZTj )=D2. Then the left hand side of (45) is ∑n j=1[ZjZ T j −E(ZjZTj )]. We note that
‖ZjZTj −E(ZjZTj )‖2≤max{‖ZjZTj ‖2,‖E(ZjZTj )‖2}≤max{‖Zj‖22,d2max}.
For ‖Zj‖22, we have
‖Zj‖22 = R∑ p=0 d2pφ 2 p(xj)≤ R∑ p=0 d2p‖φp‖2∞, (46)
we have ‖ZjZTj −E(ZjZTj )‖2≤max{ ∑R p=0d 2 p‖φp‖2∞,d2max}.
On the other hand,
E[(ZjZTj −E(ZjZTj ))2]=E[‖Zj‖22ZjZTj ]−(E(ZjZTj ))2.
Since
E[‖Zj‖22ZjZTj ]4E[ R∑ p=0 d2p‖φp‖2∞ZjZTj ], (by (46))
= R∑ p=0 d2p‖φp‖2∞E[ZjZTj ],
we have ‖E[(ZjZTj −E(ZjZTj ))2]‖2≤max{ ∑R p=0d 2 p‖φp‖2∞‖E[ZjZTj ]‖2,d4max}
≤max{ ∑R p=0d 2 p‖φp‖2∞d2max,d4max}
≤d2maxmax{ ∑R p=0d 2 p‖φp‖2∞,d2max}.
Using the matrix Bernstein inequality (Tropp, 2012, Theorem 6.1), we have
P(‖ n∑ j=1 [ZjZ T j −E(ZjZTj )]‖2>t)
≤Rexp  −t2 2(n‖E[(ZjZTj −E(ZjZTj ))2]‖2+ tmaxj‖ZjZTj −E(ZjZTj )‖2 3 )  ≤Rexp
 −t2 2(nd2maxmax{ ∑R p=0d 2 p‖φp‖2∞,d2max}+ tmax{ ∑R p=0d 2 p‖φp‖2∞,d2max} 3 )  =Rexp ( −t2
O(max{nd2maxmax{ ∑R p=0d 2 p‖φp‖2∞,d2max},tmax{ ∑R p=0d 2 p‖φp‖2∞,d2max}})
) .
Then with probability of at least 1−δ, we have
‖ n∑ j=1 [ZjZ T j −E(ZjZTj )]‖2
≤max {√ nd2maxmax{ ∑R p=0d 2 p‖φp‖2∞,d2max}logRδ ,max {∑R p=0d 2 p‖φp‖2∞,d2max } logRδ } .
Corollary 22. Suppose that the eigenvalues (λp)p≥1 satisfy Assumption 4, and the eigenfunctions satisfy Assumption 6. Assume σ2 = Θ(nt) where 1− α1+2τ < t< 1 Let γ be a positive number such that 1+α+2τ−(1+2τ+2α)t2α(1−t) <γ≤1. Then with probability of at least 1−δ, we have
‖ 1σ2 (I+ n σ2 ΛR) −γ/2Λ γ/2 R (Φ T RΦR−nI)Λ γ/2 R (I+ n σ2 ΛR) −γ/2‖2 ≤O ( n 1+α+2τ−(1+2τ+2α)t 2α −γ(1−t) √ logRδ ) .
(47)
Proof of Corollary 22. Use the same notation as in Lemma 21. Let D = (I + nσ2 ΛR) −γ/2Λ γ/2 R . Then d2max ≤ σ 2γ nγ and ∑R p=0 d 2 p‖φp‖2∞ ≤ ∑R p=0 C 2 φ λγpp 2τ
(1+ n σ2 λp)γ
= O(( nσ2 ) 1−γα+2τ
α ), where the first inequality follows from Assumptions 4 and 6 and the last equality from Lemma 15. Then M=max{ ∑R p=0d 2 p‖φp‖2∞,d2max}=O(( nσ2 ) 1−γα+2τ α ). Applying Lemma 21, we have
‖ 1σ2 (I+ n σ2 ΛR) −γ/2Λ γ/2 R (Φ T RΦR−nI)Λ γ/2 R (I+ n σ2 ΛR) −γ/2‖2 ≤ 1σ2 max {√ nσ 2γ nγ O(( n σ2 ) 1−γα+2τ α )logRδ ,O(( n σ2 ) 1−γα+2τ α )logRδ } =O( 1σ2 ( n σ2 ) 1−2γα+2τ 2α n 1 2 )=O( √ logRδ n (1−2γα+2τ)(1−t) 2α + 1 2−t)
=O (√ logRδ n 1+α+2τ 2α − (1+2τ+2α)t 2α −γ(1−t) ) .
(48)
Corollary 23. Suppose that the eigenvalues (λp)p≥1 satisfy Assumption 4, and the eigenfuctions satisfy Assumption 6. Let Λ̃1,R = diag{1,λ1,...,λR}. Assume σ2 = Θ(nt) where t < 1 Let γ be a positive number such that 1+2τα <γ≤1. Then with probability of at least 1−δ, we have
‖(I+ nσ2 ΛR) −γ/2Λ̃ γ/2 1,R(Φ T RΦR−nI)Λ̃ γ/2 1,R(I+
n
σ2 ΛR)
−γ/2‖2≤O (√ logRδ n 1 2 ) . (49)
Proof of Corollary 23. Use the same notation as in Lemma 21. LetD=(I+ nσ2 ΛR) −γ/2Λ̃ γ/2 1,R . Then d2max≤1 and ∑R p=0d 2 p‖φp‖2∞≤C2φ+ ∑R p=1C 2 φ λγpp 2τ
(1+ n σ2 λp)γ
=C2φ+O(n (1−γα+2τ)(1−t)
α )=O(1) where the first inequality follows from Assumptions 4 and 6 and the second equality from Lemma 15. Then M=max{ ∑R p=0d 2 p‖φp‖2∞,d2max}=O(1). Applying Lemma 21, we have
‖(I+ nσ2 ΛR) −γ/2Λ γ/2 R (Φ T RΦR−nI)Λ γ/2 R (I+ n σ2 ΛR) −γ/2‖2 ≤max {√
logRδ nO(1),log R δ O(1) } =O (√ logRδ n 1 2 ) .
(50)
Corollary 24. Suppose that the eigenvalues (λp)p≥1 satisfy Assumption 4, and the eigenfunctions satisfy Assumption 6. Let ΦR+1:S = (φR+1(x),...,φS(x)), and ΛR+1:S = (λR+1,...,λS). Then with probability of at least 1−δ, we have ‖Λ1/2R+1:S(Φ T R+1:SΦR+1:S−nI)Λ 1/2 R+1:S‖2≤O ( logS−Rδ max{n 1 2R 1−2α+2τ 2 ,R1−α+2τ} ) . (51)
Proof of Corollary 24. Use the same notation as in Lemma 21. Let D = Λ1/2R+1:S . Then d2max≤CλR−α=O(R−α) and ∑S p=R+1C 2 φd 2 pp 2τ ≤ ∑S p=R+1C 2 φCλp −αp2τ =O(R1−α+2τ ), where
the first inequality follows from Assumptions 4 and 6. ThenM=max{ ∑S p=R+1C 2 φd 2 pp
2τ ,d2max}= O(R1−α+2τ ). Applying Lemma 21, we have
‖(I+ n σ2 ΛR) −γ/2Λ γ/2 R (Φ T RΦR−nI)Λ γ/2 R (I+ n σ2 ΛR) −γ/2‖2
≤max {√
logS−Rδ nO(R −α)O(R1−α+2τ ),logS−Rδ O(R 1−α+2τ )) } =O ( logS−Rδ max{n 1 2R 1−2α+2τ 2 ,R1−α+2τ} ) .
(52)
Lemma 25. Under the assumptions of Corollary 24, with probability of at least 1−δ, we have ‖Φ>RΛ>RΦT>R‖2 =Õ(max{nR−α,n 1 2R 1−2α+2τ 2 ,R1−α+2τ}).
Proof of Lemma 25. For S∈N, we have ‖Φ>SΛ>SΦT>S‖2≤ ∞∑
p=S+1
‖Λpφp(x)φp(x)T ‖2
= ∞∑ p=S+1 λp‖φp(x)‖22
≤ ∞∑
p=S+1
λpnC 2 φ
=O(nS1−α).
Let S=R α α−1 . Then we get ‖Φ>SΛ>SΦT>S‖2 =O(nR−α).
Let ΦR+1:S=(φR+1(x),...,φS(x)), ΛR+1:S=(λR+1,...,λS). We then have ‖Φ>RΛ>RΦT>R‖2≤‖Φ>SΛ>SΦT>S‖2+‖ΦR+1:SΛR+1:SΦTR+1:S‖2
≤O(nR−α)+‖Λ1/2R+1:SΦ T R+1:SΦR+1:SΛ 1/2 R+1:S‖2
≤O(nR−α)+n‖ΛR+1:S‖2+‖Λ1/2R+1:S(Φ T R+1:SΦR+1:S−nI)Λ 1/2 R+1:S‖2 ≤O(nR−α)+O(nR−α)+O(logR α α−1−R δ max{n 12R 1−2α+2τ 2 ,R1−α+2τ}) =Õ(max{nR−α,n 12R 1−2α+2τ 2 ,R1−α+2τ}),
where in the fourth inequality we use Corollary 24.
Corollary 26. Assume that σ2 = Θ(1). If R=n 1α+κ where 0<κ< α−1−2τα(1+2τ) , then with probability of at least 1−δ, we have
‖(I+ ΦRΛRΦ T R σ2 ) −1 Φ>RΛ>RΦT>R σ2 ‖2≤‖ Φ>RΛ>RΦ T >R σ2 ‖2 =Õ(n −κα)=o(1).
Proof of Corollary 26. By Lemma 25 and the assumptionR=n 1 α+κ, we have
‖(I+ ΦRΛRΦ T R σ2 ) −1 Φ>RΛ>RΦT>R σ2 ‖2≤‖ Φ>RΛ>RΦ T >R
σ2 ‖2 ≤Õ(max{nR−α,n 12R 1−2α+2τ 2 ,R1−α+2τ})
=Õ(n−κα).
Lemma 27. Assume that ‖ 1σ2 (I+ n σ2 ΛR) −γ/2Λ γ/2 R (Φ T RΦR−nI)Λ γ/2 R (I+ n σ2 ΛR) −γ/2‖2<1 where 1+2τ α <γ≤1. We then have
(I+ 1σ2 ΛRΦ T RΦR)
−1
=(I+ nσ2 ΛR) −1+ ∞∑ j=1 (−1)j ( 1 σ2 (I+ n σ2 ΛR) −1ΛR(Φ T RΦR−nI) )j (I+ nσ2 ΛR) −1.
Proof of Lemma 27. First note that
‖ 1σ2 (I+ n σ2 ΛR) −1/2Λ 1/2 R (Φ T RΦR−nI)Λ 1/2 R (I+ n σ2 ΛR) −1/2‖2 <‖ 1σ2 (I+ n σ2 ΛR) −γ/2Λ γ/2 R (Φ T RΦR−nI)Λ γ/2 R (I+ n σ2 ΛR) −γ/2‖2<1.
Let Λ̃ ,R = diag{ ,λ1,...,λR}. Since ΛR = diag{0,λ1,...,λR}, we have that when is sufficiently small, ‖ 1σ2 (I + n σ2 Λ̃ ,R) −1/2Λ̃ 1/2 ,R(Φ T RΦR − nI)Λ̃ 1/2 ,R(I + n σ2 Λ̃ ,R)
−1/2‖2 < 1. Since all diagonal entries of Λ̃ ,R are positive, we have
(I+ 1σ2 Λ̃ ,RΦ T RΦR)
−1
=(I+ nσ2 Λ̃ ,R+ 1 σ2 Λ̃ ,R(Φ T RΦR−nI))−1
=Λ̃ 1/2 ,R(I+ n σ2 Λ̃ ,R) −1/2 [ I+ 1σ2 (I+ n σ2 Λ̃ ,R) −1/2Λ̃ 1/2 ,R(Φ T RΦR−nI)Λ̃ 1/2 ,R(I+ n σ2 Λ̃ ,R) −1/2 ]−1
(I+ nσ2 Λ̃ ,R) −1/2Λ̃ −1/2 ,R
=(I+ nσ2 Λ̃ ,R) −1
+ ∞∑ j=1 [ (−1)jΛ̃1/2 ,R(I+ n σ2 Λ̃ ,R) −1/2 ( 1 σ2 (I+ n σ2 Λ̃ ,R) −1/2Λ̃ 1/2 ,R(Φ T RΦR−nI)Λ̃ 1/2 ,R(I+ n σ2 Λ̃ ,R) −1/2 )j
(I+ nσ2 Λ̃ ,R) −1/2Λ̃ −1/2 ,R
]
=(I+ nσ2 Λ̃ ,R) −1+ ∞∑ j=1 (−1)j ( 1 σ2 (I+ n σ2 Λ̃ ,R) −1Λ̃ ,R(Φ T RΦR−nI) )j (I+ nσ2 Λ̃ ,R) −1.
Letting →0, we get
(I+ 1σ2 ΛRΦ T RΦR)
−1
=(I+ nσ2 ΛR) −1+ ∞∑ j=1 (−1)j ( 1 σ2 (I+ nσ2 ΛR) −1ΛR(Φ T RΦR−nI) )j (I+ nσ2 ΛR) −1.
This concludes the proof.
Lemma 28. If ‖(I+ ΦRΛRΦ T R σ2 ) −1 Φ>RΛ>RΦ T >R σ2 ‖2<1, then we have
(I+ ΦΛΦ T σ2 ) −1−(I+ ΦRΛRΦ T R σ2 ) −1 ∞∑ j=1 (−1)j ( (I+ ΦRΛRΦ T R σ2 ) −1 Φ>RΛ>RΦT>R σ2 )j (I+ ΦRΛRΦ T R σ2 ) −1.
(53)
In particular, assume that σ2 =Θ(1). LetR=n 1 α+κ where 0<κ< α−1−2τα(1+2τ) . Then with probability of at least 1−δ, for sufficiently large n, we have ‖(I+ ΦRΛRΦ T R
σ2 ) −1 Φ>RΛ>RΦ
T >R
σ2 ‖2<1 and (53) holds.
Proof of Lemma 28. Define Φ>R=(φR+1(x),φR+2(x),...), Λ>R=diag(λR+1,λR+2,...). Then we have
(I+ ΦΛΦ T σ2 ) −1−(I+ ΦRΛRΦ T R σ2 ) −1
=(I+ ΦRΛRΦ
T R
σ2 + Φ>RΛ>RΦ
T >R
σ2 ) −1−(I+ ΦRΛRΦ
T R
σ2 ) −1
= (( I+(I+ ΦRΛRΦ T R
σ2 ) −1 Φ>RΛ>RΦT>R σ2
)−1 −I ) (I+ ΦRΛRΦ T R
σ2 ) −1.
By Corollary 26, for sufficiently large n, ‖(I+ ΦRΛRΦ T R σ2 ) −1 Φ>RΛ>RΦ T >R
σ2 ‖2< 1 with probability of at least 1−δ. Hence
(I+ ΦΛΦ T σ2 ) −1−(I+ ΦRΛRΦ T R σ2 ) −1
= (( I+(I+ ΦRΛRΦ T R
σ2 ) −1 Φ>RΛ>RΦT>R σ2
)−1 −I ) (I+ ΦRΛRΦ T R
σ2 ) −1
= ∞∑ j=1 (−1)j ( (I+ ΦRΛRΦ T R σ2 ) −1 Φ>RΛ>RΦT>R σ2 )j (I+ ΦRΛRΦ T R σ2 ) −1.
Lemma 29. Assume that µ0 =0 and σ2 =Θ(nt) where 1− α1+2τ <t<1. LetR=n ( 1α+κ)(1−t) where 0<κ< α−1−2τ+(1+2τ)tα2(1−t) . Then when n is sufficiently large, with probability of at least 1−2δ we have
‖(I+ 1σ2 ΦRΛRΦ T R) −1fR(x)‖2 =Õ (√ ( 1δ +1)n·n max{−(1−t), (1−2β)(1−t)2α } ) . (54)
Proof of Lemma 29. Let Λ1:R = diag{λ1, ... , λR}, Φ1:R = (φ1(x), φ1(x), ... , φR(x)) and µ1:R = (µ1, ... , µR). Since µ0 = 0, we have (I + 1σ2 ΦRΛRΦ T R) −1fR(x) = (I+ 1σ2 Φ1:RΛ1:RΦ T 1:R) −1Φ1:Rµ1:R. Using the Woodbury matrix identity, we have that
(I+ 1σ2 Φ1:RΛ1:RΦ T 1:R) −1Φ1:Rµ1:R= [ I−Φ1:R(σ2I+Λ1:RΦT1:RΦ1:R)−1Λ1:RΦT1:R ] Φ1:Rµ1:R
=Φ1:Rµ1:R−Φ1:R(σ2I+Λ1:RΦT1:RΦ1:R)−1Λ1:RΦT1:RΦ1:Rµ1:R =σ2Φ1:R(σ 2I+Λ1:RΦ T 1:RΦ1:R)
−1µ1:R. (55)
Let A = (I + nσ2 Λ1:R) −1/2Λ 1/2 1:R(Φ T 1:RΦ1:R − nI)Λ 1/2 1:R(I + n σ2 Λ1:R) −1/2.By Corollary 22, with probability of at least 1−δ, we have ‖ 1σ2A‖2 = √ logRδ n 1−α+2τ 2α − (1+2τ)t 2α . When n is sufficiently large, ‖ 1σ2A‖2 =o(1) is less than 1 because 1− α 1+2τ <t<1. By Lemma 27, we have
(I+ 1σ2 Λ1:RΦ T 1:RΦ1:R)
−1
=(I+ nσ2 Λ1:R) −1+ ∞∑ j=1 (−1)j ( 1 σ2 (I+ n σ2 Λ1:R) −1Λ1:R(Φ T 1:RΦ1:R−nI) )j (I+ nσ2 Λ1:R) −1.
We then have
‖(σ2I+Λ1:RΦT1:RΦ1:R)−1µ1:R‖2
= 1
σ2 ∥∥∥∥∥∥ (I+ nσ2 Λ1:R)−1+ ∞∑
j=1
(−1)j (
1 σ2 (I+ n σ2 Λ1:R) −1Λ1:R(Φ T 1:RΦ1:R−nI) )j (I+ nσ2 Λ1:R) −1 µ1:R ∥∥∥∥∥∥
2
≤ 1 σ2 ‖(I+ nσ2 Λ1:R)−1µ1:R‖2+ ∞∑ j=1 ∥∥∥( 1σ2 (I+ nσ2 Λ1:R)−1Λ1:R(ΦT1:RΦ1:R−nI))j(I+ nσ2 Λ1:R)−1µ1:R∥∥∥ 2 . (56) By Lemma 15 and Assumption 5, assuming that supi≥1pi+1−pi=h, we have
‖(I+ nσ2 Λ1:R) −1µ1:R‖2≤ √√√√ R∑ p=1
C2µp −2β
(1+nCλp−α/σ2)2 =Θ(nmax{−(1−t), (1−2β)(1−t) 2α }logk/2n),
‖(I+ nσ2 Λ1:R) −1µ1:R‖2≥ √√√√bRh c∑ i=1
C2µi −2β
(1+ nσ2Cλ(hi) −α)2
=Θ(nmax{−(1−t), (1−2β)(1−t) 2α }logk/2n)
where k= {
0, 2α 6=2β−1, 1, 2α=2β−1. . Overall we have
‖(I+ nσ2 Λ1:R) −1µ1:R‖2 =Θ(n(1−t)max{−1,
1−2β 2α }logk/2n). (57)
Using the fact that ‖ 1σ2A‖2 = √ logRδ n 1−α+2τ 2α − (1+2τ)t
2α and ‖(I+ nσ2 Λ1:R) −1Λ1:R‖2≤n−1, we have∥∥∥( 1σ2 (I+ nσ2 Λ1:R)−1Λ1:R(ΦT1:RΦ1:R−nI))j(I+ nσ2 Λ1:R)−1µ1:R∥∥∥
2
= ∥∥∥∥(I+ nσ2 Λ1:R)− 12 Λ 121:R( 1σ2A)j(I+ nσ2 Λ1:R)− 12 Λ 121:Rµ1:R∥∥∥∥ 2
≤Õ(n− 1−t
2 )‖ 1σ2A‖ j 2‖(I+ nσ2 Λ1:R) − 12 Λ − 12 1:Rµ1:R‖2
(58)
By Lemma 16 and the assumptionR=n( 1 α+κ)(1−t),
‖(I+ nσ2 Λ1:R) − 12 Λ − 12 1:Rµ1:R‖2≤ √√√√ R∑ p=1 (Cλp−α)−1C2µp −2β (1+nCλp−α/σ2)1
=Õ(max{n−(1−t)/2,R1/2−β+α/2})
=Õ(max{n−(1−t)/2,n( 12 + 1−2β 2α +κ(1/2−β+α/2))(1−t)})
(59)
We then have ∥∥∥( 1σ2 (I+ nσ2 Λ1:R)−1Λ1:R(ΦT1:RΦ1:R−nI))j(I+ nσ2 Λ1:R)−1µ1:R∥∥∥ 2
=‖ 1σ2A‖ j 2Õ(max{n−(1−t),n( 1−2β 2α +κ(1/2−β+α/2))(1−t)})
(60)
By (56), (57) and (60), we have
‖(σ2I+Λ1:RΦT1:RΦ1:R)−1µ1:R‖2
=Θ(n(1−t)max{−1, 1−2β 2α }logk/2n)+ ∞∑ j=1 ‖ 1 σ2 A‖j2Õ(max{n−(1−t),n(1−t) 1−2β 2α +κ(1−t)(1/2−β+α/2)})
=Θ(n(1−t)max{−1, 1−2β 2α }logk/2n)+Õ(n 1−α+2τ 2α − (1+2τ)t 2α )Õ(max{n−(1−t),n(1−t) 1−2β 2α +κ(1−t)(1/2−β+α/2)}). (61)
By assumption κ< α−1−2τ+(1+2τ)tα2(1−t) , we have that
κ(1−t)(1/2−β+α/2)+ 1−α+2τ 2α − (1+2τ)t 2α <κα(1−t)/2+ 1−α+2τ 2α − (1+2τ)t 2α <0.
Using (61), we then get
‖(σ2I+Λ1:RΦT1:RΦ1:R)−1µ1:R‖2 =Θ(n(1−t)max{−1, 1−2β 2α }logk/2n)
= 1+o(1) σ2 ‖(I+ n σ2 Λ1:R)
−1µ1:R‖2. (62)
By Corollary 20, with probability of at least 1−δ, we have ‖Φ1:R(σ2I+Λ1:RΦT1:RΦ1:R)−1µ1:R‖2 =Õ( √ ( 1δ +1)n‖(σ 2I+Λ1:RΦ T 1:RΦ1:R) −1µ1:R‖2)
=Õ( √
( 1δ +1)n·n (1−t)max{−1, 1−2β2α }).
(63)
From (55), we get ‖(I + 1σ2 Φ1:RΛ1:RΦ T 1:R) −1Φ1:Rµ1:R‖2 = Õ( √ ( 1δ +1)n ·n (1−t)max{−1, 1−2β2α }). This concludes the proof.
Lemma 30. Assume that µ0 > 0 and σ2 = Θ(nt) where 1− α1+2τ < t < 1. Let R = n 1 α+κ where 0<κ< α−1−2τ+(1+2τ)tα2 . Then when n is sufficiently large, with probability of at least 1−2δ, we have
‖(I+ 1σ2 ΦRΛRΦ T R) −1fR(x)‖2 =Õ (√ ( 1δ +1)n ) . (64)
Proof of Lemma 30. Using the Woodbury matrix identity, we have that
(I+ 1σ2 ΦRΛRΦ T R) −1fR(x)= [ I−ΦR(σ2I+ΛRΦTRΦR)−1ΛRΦTR ] ΦRµR
=ΦRµR−ΦR(σ2I+ΛRΦTRΦR)−1ΛRΦTRΦRµR =σ2ΦR(σ 2I+ΛRΦ T RΦR) −1µR.
(65)
Let µR,1 =(µ0,0,...,0) and µR,2 =(0,µ1,...,µR). Then µR=µR,1+µR,2. Then we have
‖(σ2I+ΛRΦTRΦR)−1µR‖2 =‖(σ2I+ΛRΦTRΦR)−1µR,1‖2+‖(σ2I+ΛRΦTRΦR)−1µR,2‖2. (66) According to (62) in the proof of Lemma 29, we have ‖(σ2I + ΛRΦTRΦR)−1µR,2‖2 = Õ(nmax{−(1−t),
(1−t)(1−2β) 2α }). Next we estimate ‖σ2ΦR(σ2I+ΛRΦTRΦR)−1µR,1‖2.
Let A=(I+ n
σ2 Λ1:R)
−γ/2Λ γ/2 1:R(Φ T 1:RΦ1:R−nI)Λ γ/2 1:R(I+
n
σ2 Λ1:R)
−γ/2
where 11−t ( 1+α+2τ 2α − (1+2τ+2α)t 2α )<γ< 1. Since 1− α 1+2τ <t< 1, 1 1−t ( 1+α+2τ 2α − (1+2τ+2α)t
2α )< 1 so the range for γ is well-defined.By Corollary 22, with probability of at least 1 − δ, we have ‖ 1σ2A‖2 = Õ( √ logRδ n 1+α+2τ 2α − (1+2τ+2α)t 2α −γ(1−t)) = o(1). When n is sufficiently large, ‖ 1σ2A‖2 is less than 1 because 1− α1+2τ <t<1. By Lemma 27, we have
(I+ 1σ2 ΛRΦ T RΦR)
−1
=(I+ nσ2 ΛR) −1+ ∞∑ j=1 (−1)j ( 1 σ2 (I+ n σ2 ΛR) −1ΛR(Φ T RΦR−nI) )j (I+ nσ2 ΛR) −1.
We then have ‖(σ2I+ΛRΦTRΦR)−1µR,1‖2
= 1
σ2 ∥∥∥∥∥∥ (I+ nσ2 ΛR)−1+ ∞∑
j=1
(−1)j (
1 σ2 (I+ n σ2 ΛR) −1ΛR(Φ T RΦR−nI) )j (I+ nσ2 ΛR) −1 µR,1 ∥∥∥∥∥∥
2
≤ 1 σ2 ‖(I+ nσ2 ΛR)−1µR,1‖2+ ∞∑ j=1 ∥∥∥( 1σ2 (I+ nσ2 ΛR)−1ΛR(ΦTRΦR−nI))j(I+ nσ2 ΛR)−1µR,1∥∥∥ 2 . (67)
By Lemma 15,
‖(I+ n σ2 ΛR) −1µR,1‖2≤ √√√√µ20+ R∑ p=1
C2µp −2β
(1+nCλp−α/σ2)2 =O(1). (68)
Let Λ̃1,R = diag{1, λ1, ... , λR} and I0,R = (0, 1, ... , 1). Then ΛR = Λ̃1,RI0,R. Let B = (I + n σ2 ΛR) −γ/2Λ̃ γ/2 1,R(Φ T RΦR−nI)Λ̃ γ/2 1,R(I+ n σ2 ΛR) −γ/2. According to Corollary 23, we have ‖B‖2 =
O (√
logRδ n 1 2 ) . Using the fact that ‖ 1σ2A‖2 =Õ (√ logRδ n 1+α+2τ 2α − (1+2τ+2α)t 2α −γ(1−t) ) , we have∥∥∥( 1σ2 (I+ nσ2 ΛR)−1ΛR(ΦTRΦR−nI))j(I+ nσ2 ΛR)−1µR,1∥∥∥
2
= 1
σ2j ∥∥∥∥(I+ nσ2 ΛR)−1+γ2 Λ1−γ2R (A(I+ nσ2 ΛR)−1+γΛ1−γR )j−1B(I+ nσ2 ΛR)−1+γ2 µR,1∥∥∥∥ 2
≤ 1 σ2 (n(−1+ γ 2 +(−1+γ)(j−1))(1−t)Õ( √ logRδ n (j−1)( 1+α+2τ2α − (1+2τ+2α)t 2α −γ(1−t))) √ logRδ n 1 2 ‖µR,1‖2
≤n(−1+ γ 2 )(1−t)+ 1 2−tÕ(n [1−α+2τ−(1+2τ)t](j−1) 2α ) √ logRδ ‖µR,1‖2
=Õ(n− 1 2 + γ 2 (1−t)+ [1−α+2τ−(1+2τ)t](j−1) 2α ). (69) Since 11−t ( 1+α+2τ 2α − (1+2τ+2α)t 2α )<γ<1 and− 1 2 + 1 1−t ( 1+α+2τ 2α − (1+2τ+2α)t 2α ) 1−t 2 <0, we can let γ be a little bit larger than 11−t ( 1+α+2τ 2α − (1+2τ+2α)t 2α ) and make− 1 2 + γ 2 (1−t)< 0 holds. By (67), (68), (69), we have ‖(σ2I+ΛRΦTRΦR)−1µR,1‖2
≤O(1)+ ∞∑ j=1 Õ(n− 1 2 + γ 2 (1−t)+ [1−α+2τ−(1+2τ)t](j−1) 2α )
≤O(1)+o(1)=O(1).
(70)
According to (66), we have ‖(σ2I + ΛRΦTRΦR)−1µR‖2 = Õ(nmax{−(1−t), (1−t)(1−2β) 2α }) +O(1) = O(1). By Corollary 20, with probability of at least 1−δ, we have
‖ΦR(σ2I+ΛRΦTRΦR)−1µR‖2 =Õ( √ ( 1δ +1)n‖(σ 2I+ΛRΦ T RΦR) −1µR‖2)
=Õ (√ ( 1δ +1)n ) .
From (65), we get ‖(I+ 1σ2 ΦRΛRΦ T R) −1fR(x)‖2 =Õ (√ ( 1δ +1)n ) . This concludes the proof.
Lemma 31. Assume that σ2 = Θ(1). Let R= n 1α+κ where 0<κ< α−1−2τα2 . Assume that µ0 = 0. Then when n is sufficiently large, with probability of at least 1−3δ we have
‖(I+ ΦΛΦ T σ2 ) −1fR(x)‖2 =Õ( √ ( 1δ +1)n·n max{−1, 1−2β2α }). (71)
Assume that µ0>0. Then when n is sufficiently large, with probability of at least 1−3δ we have
‖(I+ ΦΛΦ T σ2 ) −1fR(x)‖2 =Õ( √ ( 1δ +1)n). (72)
Proof of Lemma 31. We have
(I+ ΦΛΦ T σ2 ) −1fR(x)
=(I+ ΦRΛRΦ
T R
σ2 ) −1fR(x)+
( (I+ ΦΛΦ T
σ2 ) −1−(I+ ΦRΛRΦ
T R σ2 ) −1 ) fR(x).
(73)
When µ0 =0, by Lemma 29, with probability of at least 1−2δ, we have
‖(I+ 1σ2 ΦRΛRΦ T R) −1fR(x)‖2 =Õ( √ ( 1δ +1)n·n max{−1, 1−2β2α }).
Since α−1−2τα2 < α−1−2τ α(1+2τ) , we apply Lemma 28 and Corollary 26 and get that with probability of at least 1−δ, the second term in the right hand side of (73) is estimated as follows:
‖ ( (I+ ΦΛΦ T
σ2 ) −1−(I+ ΦRΛRΦ
T R σ2 ) −1 ) fR(x)‖2
=‖ ∞∑ j=1 (−1)j ( (I+ ΦRΛRΦ T R σ2 ) −1 Φ>RΛ>RΦT>R σ2 )j (I+ ΦRΛRΦ T R σ2 ) −1fR(x)‖2
= ∞∑ j=1 ∥∥∥((I+ ΦRΛRΦTRσ2 )−1 Φ>RΛ>RΦT>Rσ2 )∥∥∥j 2 ‖(I+ ΦRΛRΦ T R σ2 ) −1fR(x)‖2
= ∞∑ j=1 Õ(n−jκα)Õ( √ ( 1δ +1)n·n max{−1, 1−2β2α })
=o( √
( 1δ +1)n·n max{−1, 1−2β2α }).
Overall, from (73), we have that with probability 1−3δ,
‖(I+ ΦΛΦ T σ2 ) −1fR(x)‖2 =Õ( √ ( 1δ +1)n·n max{−1, 1−2β2α }).
When µ0>0, using the same approach and Lemma 30, we can prove that ‖(I+ ΦΛΦ T σ2 ) −1fR(x)‖2 = Õ( √
( 1δ +1)n). This concludes the proof.
D PROOF OF THE MAIN RESULTS
D.1 PROOFS RELATED TO THE ASYMPTOTICS OF THE NORMALIZED STOCHASTIC COMPLEXITY
Lemma 32. Under Assumptions 4, 5 and 6, with probability of at least 1−2δ we have, we have |T1,R(Dn)−T1(Dn)|=Õ ( 1 σ2 (nR 1−α+n1/2R1−α+τ+R1−α+2τ ) ) (74)
If R= n 1 α+κ where κ > 0, we have |T1,R(Dn)−T1(Dn)|= o ( 1 σ2n 1 α ) . If we further assume that 0<κ< α−1−2τα2 , µ0 =0 and σ 2 =Θ(1), then for sufficiently large nwith probability of at least 1−4δ we have
|T2,R(Dn)−T2(Dn)|=Õ ( ( 1δ +1)n max{( 1α+κ) 1−2β 2 ,1+ 1−2β α + (1−2β)κ 2 ,−1−κα,1+ 1−2β α −κα} ) . (75)
Proof of Lemma 32. Define Φ>R = (φR+1(x), φR+2(x), ... , φp(x), ...), and Λ>R = diag(λR+1,...,λp,...). We then have
|T1(Dn)−T1,R(Dn)|= ∣∣∣∣12 logdet(I+ 1σ2 ΦΛΦT )− 12 logdet(I+ 1σ2 ΦRΛRΦTR) ∣∣∣∣ + 1
2 ∣∣∣∣Tr(I+ ΦΛΦTσ2 )−1−Tr(I+ ΦRΛRΦTRσ2 )−1 ∣∣∣∣. (76)
As for the first term in the right hand side of (76), we have∣∣∣∣12 logdet(I+ 1σ2 ΦΛΦT )− 12 logdet(I+ 1σ2 ΦRΛRΦTR) ∣∣∣∣
= ∣∣∣∣12 logdet ( (I+ 1 σ2 ΦRΛRΦ T R) −1(I+ 1 σ2 ΦRΛRΦ T R+ 1 σ2 Φ>RΛ>RΦ T >R) )∣∣∣∣ = ∣∣∣∣12 logdet ( I+ 1 σ2 (I+ 1 σ2 ΦRΛRΦ T R) −1Φ>RΛ>RΦ T >R
)∣∣∣∣ = 1
2 ∣∣∣∣Trlog(I+ 1σ2 (I+ 1σ2 ΦRΛRΦTR)−1Φ>RΛ>RΦT>R )∣∣∣∣.
(77)
Given a concave function h and a matrixB∈Rn×n whose eigenvalues ζ1,...,ζn are all positive, we have that
Trh(B)= ∑n p=1h(ζi)≤nh( 1 n ∑n p=1ζi)≤nh( 1 nTrB), (78)
where we used Jensen’s inequality. Using h(x)=log(1+x) in (78), with probability 1−δ, we have∣∣ 1 2 logdet(I+ 1 σ2 ΦΛΦ T )− 12 logdet(I+ 1 σ2 ΦRΛRΦ T R) ∣∣
≤ n2 log(1+ 1 nTr( 1 σ2 (I+
ΦRΛRΦ T R
σ2 ) −1Φ>RΛ>RΦ T >R))
≤ n2 log(1+ 1 nσ2 ‖(I+
ΦRΛRΦ T R
σ2 ) −1‖2Tr(Φ>RΛ>RΦT>R))
≤ n2 log(1+ 1 nσ2 ∑∞ p=R+1λp‖φp(x)‖22)≤ 1 2σ2 ∑∞ p=R+1λp‖φp(x)‖22
= 12σ2 ∑∞ p=R+1λp ( C2φÕ (√ p2τn‖φp‖22+p2τ ) +n‖φp‖22 ) =Õ( 1σ2n ∑∞ p=R+1λp+n 1/2 ∑∞ p=R+1λpp τ+ ∑∞ p=R+1λpp 2τ )
=Õ (
1 σ2 (nR
1−α+n1/2R1−α+τ+R1−α+2τ ) ) =o ( 1 σ2n 1 α ) ,
(79)
where in the second inequality we use the fact that TrAB≤‖A‖2TrB whenA andB are symmetric positive definite matrices, and in the last inequality we use Lemma 18.
As for the second term in the right hand side of (76), letA=(I+ ΦRΛRΦ T R
σ2 ) −1/2. Then we have
1 2 ∣∣∣Tr(I+ ΦΛΦTσ2 )−1−Tr(I+ ΦRΛRΦTRσ2 )−1∣∣∣ = 12 ∣∣∣∣TrA[I−(I+A(Φ>RΛ>RΦT>Rσ2 )A)−1 ] A
∣∣∣∣ ≤ 12Tr [ I−(I+A(Φ>RΛ>RΦ T >R σ2 )A) −1 ]
≤ n2 (1−(1+ 1 nTrA(
Φ>RΛ>RΦ T >R
σ2 )A) −1)≤ n2 (1−(1+ 1 nTr(
Φ>RΛ>RΦ T >R
σ2 )) −1)
≤ n2 (1−(1+ 1 nσ2 ∑∞ p=R+1λp‖φp(x)‖22))−1)≤ 1 2σ2 ∑∞ p=R+1λp‖φp(x)‖22
=Õ (
1 σ2 (nR
1−α+n1/2R1−α+τ+R1−α+2τ ) ) =o ( 1 σ2n 1 α ) ,
where in the first inequality we use the fact that ‖A‖2< 1 and TrABA≤‖A‖22TrB when A and B are symmetric positive definite matrices, in the second inequality we use h(x)=1−1/(1+x) in (78) and in the last equality we use the last few steps of (79). This concludes the proof of the first statement.
As for |T2(Dn)−T2,R(Dn)|, we have
|T2(Dn)−T2,R(Dn)|= ∣∣∣f(x)T (I+ ΦΛΦTσ2 )−1f(x)−fR(x)T (I+ ΦΛΦTσ2 )−1fR(x)∣∣∣
+ ∣∣∣fR(x)T (I+ ΦΛΦTσ2 )−1fR(x)−fR(x)T (I+ ΦRΛRΦTRσ2 )−1fR(x)∣∣∣. (80)
For the first term on the right-hand side of (80), we have∣∣∣f(x)T (I+ ΦΛΦTσ2 )−1f(x)−fR(x)T (I+ ΦΛΦTσ2 )−1fR(x)∣∣∣ ≤2 ∣∣∣f>R(x)T (I+ ΦΛΦTσ2 )−1fR(x)∣∣∣+∣∣∣f>R(x)T (I+ ΦΛΦTσ2 )−1f>R(x)∣∣∣
≤2‖f>R(x)‖2‖(I+ ΦΛΦ T σ2 ) −1fR(x)‖2+‖f>R(x)‖2‖(I+ ΦΛΦ T σ2 ) −1‖2‖f>R(x)‖2 ≤2‖f>R(x)‖2‖(I+ ΦΛΦ T σ2 ) −1fR(x)‖2+‖f>R(x)‖22.
Applying Corollary 19 and Lemma 31, with probability of at least 1−4δ, we have∣∣∣f(x)T (I+ ΦΛΦTσ2 )−1f(x)−fR(x)T (I+ ΦΛΦTσ2 )−1fR(x)∣∣∣ ≤2Õ (√ ( 1δ +1)nR 1−2β ) Õ( √ ( 1δ +1)n·n max{−1, 1−2β2α })+Õ(( 1δ +1)nR 1−2β)
=2Õ ( ( 1δ +1)n 1+( 1 α+κ) 1−2β 2 +max{−1, 1−2β 2α } ) +Õ(( 1δ +1)n 1+( 1 α+κ)(1−2β))
=2Õ ( ( 1δ +1)n 1+( 1 α+κ) 1−2β 2 +max{−1, 1−2β 2α } ) ,
where the last equality holds because ( 1α+κ) 1−2β 2 < 1−2β 2α when κ>0.
As for the second term on the right-hand side of (80), according to Lemma 28, Corollary 26 and Lemma 29, we have∣∣∣fR(x)T (I+ ΦΛΦTσ2 )−1fR(x)−fR(x)T (I+ ΦRΛRΦTRσ2 )−1fR(x)∣∣∣
= ∣∣∣∣∣∣ ∞∑ j=1 (−1)jfR(x)T ( (I+ ΦRΛRΦ T R σ2 ) −1 Φ>RΛ>RΦT>R σ2 )j (I+ ΦRΛRΦ T R σ2 ) −1fR(x) ∣∣∣∣∣∣ ≤ ∞∑ j=1 ‖(I+ ΦRΛRΦ T R σ2 ) −1‖j−12 ·‖ Φ>RΛ>RΦ T >R σ2 ‖j2 ·‖(I+ ΦRΛRΦ T R σ2 ) −1fR(x)‖22
= ∞∑ j=1 Õ(n−jκα)Õ(( 1δ +1)n 1+max{−2, 1−2βα })
=Õ(( 1δ +1)n 1+max{−2, 1−2βα }−κα).
(81)
By (80), we have |T2(Dn)−T2,R(Dn)|=Õ ( ( 1δ +1)n 1+( 1 α+κ) 1−2β 2 +max{−1, 1−2β 2α } ) +Õ ( ( 1δ +1)n 1+max{−2, 1−2βα }−κα )
=Õ ( ( 1δ +1)n max{( 1α+κ) 1−2β 2 ,1+ 1−2β α + (1−2β)κ 2 ,−1−κα,1+ 1−2β α −κα} ) .
This concludes the proof of the second statement.
In Lemma 32, we gave a bound for |T2,R(Dn)−T2(Dn)| when n 1 α <R<n 1 α+ α−1−2τ α2 . For R>n, we note the following lemma:
Lemma 33. Let R = nC and σ2 = nt. Assume that C >= 1 and C(1−α+ 2τ)− t < 0. Under Assumptions 4, 5 and 6, for sufficiently large n and with probability of at least 1−3δ we have
|T2,R(Dn)−T2(Dn)|=Õ ( ( 1δ +1) 1 σ2nR max{1/2−β,1−α+2τ} ) . (82)
Proof of Lemma 33. Define Φ>R = (φR+1(x), φR+2(x), ... , φp(x), ...), and Λ>R = diag(λR+1,...,λp,...). Then we have
|T2(Dn)−T2,R(Dn)|= ∣∣∣∣f(x)T (I+ ΦΛΦTσ2 )−1f(x)−fR(x)T (I+ ΦΛΦTσ2 )−1fR(x) ∣∣∣∣ + ∣∣∣∣fR(x)T (I+ ΦΛΦTσ2 )−1fR(x)−fR(x)T (I+ ΦRΛRΦTRσ2 )−1fR(x) ∣∣∣∣. (83)
For the first term on the right-hand side of (83), with probability 1−3δ we have∣∣∣∣f(x)T (I+ ΦΛΦTσ2 )−1f(x)−fR(x)T (I+ ΦΛΦTσ2 )−1fR(x) ∣∣∣∣
≤2 ∣∣∣∣f>R(x)T (I+ ΦΛΦTσ2 )−1fR(x) ∣∣∣∣+∣∣∣∣f>R(x)T (I+ ΦΛΦTσ2 )−1f>R(x) ∣∣∣∣
≤2‖f>R(x)‖2‖(I+ ΦΛΦT
σ2 )−1‖2‖fR(x)‖2+‖f>R(x)‖2‖(I+
ΦΛΦT
σ2 )−1‖2‖f>R(x)‖2
≤2‖f>R(x)‖2‖fR(x)‖2+‖f>R(x)‖22
≤2Õ (√ ( 1
δ +1)nR1−2β
) Õ( √ ( 1
δ +1)n·‖f‖2)+Õ((
1 δ +1)nR1−2β)
=Õ ( ( 1
δ +1)nR1/2−β
) ,
where we used Corollary 19 and Lemma 17 for the last inequality.
The assumption C(1− α+ 2τ)− t < 0 means that R 1−α+2τ
σ2 = o(1). For the second term on the right-hand side of (83), by Lemmas 28 and 25, we have∣∣∣∣fR(x)T (I+ ΦΛΦTσ2 )−1fR(x)−fR(x)T (I+ ΦRΛRΦTRσ2 )−1fR(x)
∣∣∣∣ =
∣∣∣∣∣∣ ∞∑ j=1 (−1)jfR(x)T ( (I+ ΦRΛRΦ T R σ2 )−1 Φ>RΛ>RΦ T >R σ2 )j (I+ ΦRΛRΦ T R σ2 )−1fR(x) ∣∣∣∣∣∣ ≤ ∞∑ j=1 ‖(I+ ΦRΛRΦ T R σ2 )−1‖j+12 ·‖ Φ>RΛ>RΦ T >R σ2 ‖j2 ·‖fR(x)‖22
= ∞∑ j=1 Õ( 1 σ2 Rj(1−α+2τ))Õ(( 1 δ +1)n‖f‖22)
=Õ(( 1
δ +1)
1
σ2 nR1−α+2τ ).
(84)
Using (83), we have |T2(Dn)−T2,R(Dn)|=Õ ( ( 1
δ +1)nR1/2−β
) +Õ(( 1
δ +1)n
1
σ2 R1−α+2τ )
=Õ ( ( 1
δ +1)n
1
σ2 Rmax{1/2−β,1−α+2τ}
) .
Next we consider the asympototics of T1,R(Dn) and T2,R(Dn).
Lemma 34. Let A = (I + nσ2 ΛR) −γ/2Λ γ/2 R (Φ T RΦR − nI)Λ γ/2 R (I + n σ2 ΛR) −γ/2. Assume that ‖A‖2<1 where 1+2τα <γ≤1. Then we have
T2,R(Dn)= n 2σ2µ T R(I+ n σ2 ΛR) −1µR+ 1 2 ∑∞ j=1(−1)j+1Ej ,
where
Ej=µ T R 1 σ2 (I+ n σ2 ΛR) −1(ΦTRΦR−nI) ( 1 σ2 (I+ n σ2 ΛR) −1ΛR(Φ T RΦR−nI) )j−1 (I+ nσ2 ΛR) −1µR.
Proof of Lemma 34. Let Λ̃ ,R = diag{ ,λ1,...,λR}. Since ΛR = diag{0,λ1,...,λR}, we have that when is sufficiently small, ‖ 1σ2 (I+ n σ2 Λ̃ ,R) −1/2Λ̃ 1/2 ,R(Φ T RΦR−nI)Λ̃ 1/2 ,R(I+ n σ2 Λ̃ ,R) −1/2‖2 < 1.
Since all diagonal entries of Λ̃ ,R are positive, we have
1
2σ2 µTRΦ T R(I+
1
σ2 ΦRΛ̃ ,RΦ
T R) −1ΦRµR
= 1
2σ2 µTRΦ T R
[ I−ΦR(σ2I+Λ̃ ,RΦTRΦR)−1Λ̃ ,RΦTR ] ΦRµR
= 1
2σ2 µTRΦ T RΦRµR−
1
2σ2 µTRΦ T RΦR(σ 2I+Λ̃ ,RΦ T RΦR) −1Λ̃ ,RΦ T RΦRµR
= 1
2 µTRΦ T RΦR(σ 2I+Λ̃ ,RΦ T RΦR) −1µR
= 1
2 µTRΛ̃ −1 ,RΛ̃ ,RΦ T RΦR(σ 2I+Λ̃ ,RΦ T RΦR) −1µR
= 1
2 µTRΛ̃ −1 ,RµR−
1 2 µTRΛ̃ −1 ,R(I+ 1 σ2 Λ̃ ,RΦ T RΦR) −1µR.
(85)
Using Lemma 27, we have
1 2 µTRΛ̃ −1 ,RµR− 1 2 µTRΛ̃ −1 ,R(I+ 1 σ2 Λ̃ ,RΦ T RΦR) −1µR = 1
2 µTRΛ̃ −1 ,RµR−
1 2 µTRΛ̃ −1 ,R(I+ n σ2 Λ̃ ,R) −1µR
+ 1
2 ∞∑ j=1 (−1)j+1µTRΛ̃−1 ,R ( 1 σ2 (I+ n σ2 Λ̃ ,R) −1Λ̃ ,R(Φ T RΦR−nI) )j (I+ n σ2 Λ̃ ,R) −1µR
= n
2σ2 µTR(I+
n
σ2 Λ̃ ,R)
−1µR
+ 1
2 ∞∑ j=1 (−1)j+1µTR 1 σ2 (I+ n σ2 Λ̃ ,R) −1(ΦTRΦR−nI) ( 1 σ2 (I+ n σ2 Λ̃ ,R) −1Λ̃ ,R(Φ T RΦR−nI) )j−1 (I+ n
σ2 Λ̃ ,R)
−1µR
(86) Letting →0, we get
T2,R(Dn)= 1
2σ2 µTRΦ T R(I+
1
σ2 ΦRΛRΦ
T R) −1ΦRµR
= n
2σ2 µTR(I+
n
σ2 ΛR)
−1µR
+ 1
2 ∞∑ j=1 [ (−1)j+1µTR 1 σ2 (I+ n σ2 ΛR) −1(ΦTRΦR−nI) ( 1 σ2 (I+ n σ2 ΛR) −1ΛR(Φ T RΦR−nI) )j−1 (I+ n
σ2 ΛR)
−1µR ] This concludes the proof.
Lemma 35. Assume that σ2 = Θ(1). LetR=n 1 α+κ where 0<κ< α−1−2τ2α2 . Under Assumptions 4, 5 and 6, with probability of at least 1−δ, we have
T1,R(Dn)= ( 1 2 logdet(I+ n σ2 ΛR)− 1 2Tr ( I−(I+ nσ2 ΛR) −1))(1+o(1))=Θ(n 1α ). (87) Furthermore, if we assume µ0 =0, we have
T2,R(Dn)= ( n 2σ2µ T R(I+ n σ2 ΛR) −1µR ) (1+o(1))=
{ Θ(nmax{0,1+ 1−2β α }), α 6=2β−1,
Θ(logn), α=2β−1. (88)
Proof of Lemma 35. Let
A=(I+ n
σ2 ΛR)
−γ/2Λ γ/2 R (Φ T RΦR−nI)Λ γ/2 R (I+
n
σ2 ΛR)
−γ/2, (89)
where 1+α+2τ2α <γ≤1. By Corollary 22, with probability of at least 1−δ, we have
‖A‖2 =Õ(n 1−2γα+α+2τ 2α ). (90)
When n is sufficiently large, ‖A‖2 is less than 1. LetB=(I+ nσ2 ΛR) −1/2Λ 1/2 R (Φ T RΦR−nI)Λ 1/2 R (I+
n σ2 ΛR) −1/2. Then ‖B‖2 = σ 2(1−γ) n1−γ ‖A‖2 = Õ(n 1−α+2τ
2α ). Using the Woodbury matrix identity, we compute T1,R(Dn) as follows:
T1,R(Dn)= 1 2 logdet(I+ 1 σ2 ΛRΦ T RΦR)− 12TrΦR(σ 2I+ΛRΦ T RΦR) −1ΛRΦ T R
= 12 logdet(I+ n σ2 ΛR)+ 1 2 logdet[I+ 1 σ2 (I+ n σ2 ΛR) −1/2Λ 1/2 R (Φ T RΦR−nI)Λ 1/2 R (I+ n σ2 ΛR) −1/2]
− 12Tr(σ 2I+ΛΦTRΦR) −1ΛΦTRΦR
= 12 logdet(I+ n σ2 ΛR)+ 1 2Trlog[I+ 1 σ2B]− 1 2Tr(I−σ 2(σ2I+ΛΦTRΦR) −1))
= 12 logdet(I+ n
σ2 ΛR)+
1 2Tr ∞∑ j=1 (−1)j−1 j ( 1 σ2B) j
− 12Tr I−(I+ nσ2 ΛR)−1+ ∞∑ j=1 (−1)j ( 1 σ2 (I+ n σ2 ΛR) −1ΛR(Φ T RΦR−nI) )j (I+ nσ2 ΛR) −1  = (
1 2 logdet(I+ n σ2 ΛR)− 1 2Tr ( I−(I+ nσ2 ΛR) −1))+ 12Tr ∞∑ j=1 (−1)j−1 j ( 1 σ2B) j
− 12Tr  ∞∑ j=1 (−1)j 1σ2j (I+ n σ2 ΛR) −1/2Bj(I+ nσ2 ΛR) −1/2 , (91)
where in the last equality we apply Lemma 27.
Let h(x) = log(1+x)−(1− 11+x ). It is easy to verify that h(x) is increasing on [0,+∞). As for the first term on the right hand side of (91), we have
1 2 logdet(I+ n σ2 ΛR)− 1 2Tr ( I−(I+ nσ2 ΛR) −1) = 12
R∑ p=1 ( log(1+ nσ2λp)−(1− 1 1+ n σ2 λp ) )
= 12 R∑ p=1 h( nσ2λp)≤ 1 2 R∑ p=1 h( n σ2 Cλp −α)
≤ 12h( n σ2Cλ)+ 1 2 ∫ [1,R] h( nσ2Cλx −α)dx
= 12h( n
σ2 Cλ)+
1 2n 1/α ∫ [1/n1/α,R/n1/α] h(Cλσ2 x −α)dx
=Θ(n1/α), where in the last equality we use the fact that ∫
[0,+∞]h(x −α)dx<∞. On the other hand, we have
1 2 logdet(I+ n σ2 ΛR)− 1 2Tr ( I−(I+ nσ2 ΛR) −1) = 12
R∑ p=1 h( nσ2λp)≥ 1 2 R∑ p=1 h( nσ2Cλp −α)
≥ 12 ∫
[1,R+1]
h( nσ2Cλx −α)dx
= 12n 1/α ∫ [1/n1/α,(R+1)/n1/α] h( 1σ2Cλx −α)dx
=Θ(n1/α).
Overall, we have 12 logdet(I+ n σ2 ΛR)− 1 2Tr ( I−(I+ nσ2 ΛR) −1)=Θ(n1/α). As for the second term on the right hand side of (91), we have∣∣∣∣∣∣Tr ∞∑ j=1 (−1)j−1 j ( 1 σ2B) j ∣∣∣∣∣∣≤R ∞∑ j=1 ‖ 1σ2B‖ j 2
=R ∞∑ j=1 1 σ2j Õ(n j(1−α+2τ) 2α )
=RÕ(n 1−α+2τ 2α )=Õ(n 1 α+κ+ 1−α+2τ 2α ).
As for the third term on the right hand side of (91), we have∣∣∣∣∣∣Tr  ∞∑ j=1 (−1)j 1σ2j (I+ n σ2 ΛR) −1/2Bj(I+ nσ2 ΛR) −1/2 ∣∣∣∣∣∣ ≤ ∞∑ j=1
∣∣∣Tr( 1σ2j (I+ nσ2 ΛR)−1/2Bj(I+ nσ2 ΛR)−1/2)∣∣∣ ≤R
∞∑ j=1 ∥∥∥ 1σ2j (I+ nσ2 ΛR)−1/2Bj(I+ nσ2 ΛR)−1/2∥∥∥ 2
≤R ∞∑ j=1 ∥∥∥ 1σ2j (I+ nσ2 ΛR)−1/2Bj(I+ nσ2 ΛR)−1/2∥∥∥ 2
≤R ∞∑ j=1 ∥∥ 1 σ2jB j ∥∥ 2 =Õ(n 1 α+κ+ 1−α+2τ 2α ).
Then the asymptotics of T1,R(Dn) is given by
T1,R(Dn)= 1 2 logdet(I+ n σ2 ΛR)− 1 2Tr ( I−(I+ nσ2 ΛR) −1)+Õ(n 1α+κ+ 1−α+2τ2α )+Õ(n 1α+κ+ 1−α+2τ2α ) =Θ(n1/α)+Õ(n 1 α+κ+ 1−α+2τ 2α )
=Θ(n 1 α ),
where in the last inequality we use the assumption that κ< α−1−2τ2α . Since Õ(n 1 α+κ+ 1−α+2τ 2α ) is lower order term compared to Θ(n 1 α ), we further have
T1,R(Dn)= ( 1 2 logdet(I+ n σ2 ΛR)− 1 2Tr ( I−(I+ nσ2 ΛR) −1))(1+o(1)). This concludes the proof of the first statement.
Let Λ1:R=diag{λ1,...,λR}, Φ1:R=(φ1(x),φ1(x),...,φR(x)) and µ1:R=(µ1,...,µR). Since µ0 =0, we have T2,R(Dn)= 12σ2µ T 1:RΦ T 1:R(I+ 1 σ2 Φ1:RΛ1:RΦ T 1:R)
−1Φ1:Rµ1:R. According to Lemma 34, we have T2,R(Dn)= n
2σ2 µT1:R(I+
n
σ2 Λ1:R)
−1µ1:R
+ 1
2 ∞∑ j=1 (−1)j+1µT1:R 1 σ2 (I+ n σ2 Λ1:R) −1(ΦT1:RΦ1:R−nI) ( 1 σ2 (I+ n σ2 Λ1:R) −1Λ1:R(Φ T 1:RΦ1:R−nI) )j−1 = n
2σ2 µT1:R(I+
n
σ2 Λ1:R)
−1µ1:R
+ 1
2 ∞∑ j=1 [ (−1)j+1 1 σ2j µT1:R(I+ n σ2 Λ1:R) −1+γ/2Λ −γ/2 1:R A ( (I+ n σ2 Λ1:R) −1+γΛ1−γ1:R A )j−1
(I+ n
σ2 Λ1:R)
−1+γ/2Λ −γ/2 1:R µ1:R ] (92)
where in the second to last equality we used the definition ofA (89). As for the first term on the right hand side of (92), by Lemma 15, Assumption 4 and Assumption 5, we have
n
2σ2 µT1:R(I+
n
σ2 Λ1:R)
−1µ1:R≤ n
2σ2 R∑ p=1 C2µp −2β 1+ nσ2Cλp −α =
{ Θ(nmax{0,1+ 1−2β α }), α 6=2β−1,
Θ(logn), α=2β−1.
On the other hand, by Assumption 5, assuming that supi≥1pi+1−pi=h, we have
n
2σ2 µT1:R(I+
n
σ2 Λ1:R)
−1µ1:R≥ n
2σ2 bRh c∑ i=1 C2µp −2β i 1+ nσ2Cλp −α i
≥ n 2σ2 bRh c∑ i=1
C2µi −2β
1+ nσ2Cλ(hi) −α
=
{ Θ(nmax{0,1+ 1−2β α }), α 6=2β−1,
Θ(logn), α=2β−1.
Overall, we have n
2σ2 µT1:R(I+
n
σ2 Λ1:R)
−1µ1:R=Θ(n max{0,1+ 1−2βα }logkn),
where k= {
0, α 6=2β−1, 1, α=2β−1.
By Lemma 16, we have
‖(I+ n σ2 Λ1:R) −1+γ/2Λ −γ/2 1:R µ1:R‖ 2 2≤ R∑ p=1 C2µp −2β(Cλp −α)−γ (1+ nσ2Cλp −α)2−γ
=Õ(max{n−2+γ ,R1−2β+αγ})
=Õ(nmax{−2+γ, 1−2β α +γ+κ(1−2β+αγ)}).
(93)
Using (90), the second term on the right hand side of (92) is computed as follows:
1
2 ∞∑ j=1 [ (−1)j+1 1 σ2j µT1:R(I+ n σ2 Λ1:R) −1+γ/2Λ −γ/2 1:R A ( (I+ n σ2 Λ1:R) −1+γΛ1−γ1:R A )j−1
(I+ n
σ2 Λ1:R)
−1+γ/2Λ −γ/2 1:R µ1:R ] ≤1
2 ∞∑ j=1 1 σ2j ‖A‖j ( n σ2 )(−1+γ)(j−1) ‖(I+ n σ2 Λ1:R) −1+γ/2Λ −γ/2 1:R µ1:R‖ 2 2
≤1 2 ∞∑ j=1 1 σ2j Õ(n j(1−2γα+α+2τ) 2α ) ( n σ2 )(−1+γ)(j−1) Õ(nmax{−2+γ, 1−2β α +γ+κ(1−2β+αγ)})
=Õ(nmax{−2+γ+ 1−2γα+α+2τ 2α , 1−2β α +γ+ 1−2γα+α+2τ 2α +κ(1−2β+αγ)})
=Õ(nmax{−2+ 1+α+2τ 2α , 1−2β α + 1+α+2τ 2α +κ(1−2β+αγ)}).
(94)
Since 1+α+2τ2α < 1+α+2τ α+1+2τ =1, we have−2+ 1+α+2τ 2α <0.Also we have
1−2β α + 1+α+2τ 2α +κ(1−2β+αγ)
= 1−2β α +1+ 1−α+2τ 2α +κ(1−2β+αγ) ≤1−2β α +1+ 1−α+2τ 2α +καγ < 1−2β α +1,
(95)
where the last inequality holds because κ< α−1−2τ2α2 and γ≤1. Hence we have
T2,R(Dn)= n
2σ2 µT1:R(I+
n
σ2 Λ1:R)
−1µ1:R+Õ(n max{−2+ 1+α+2τ2α , 1−2β α + 1+α+2τ 2α +κ(1−2β+αγ)})
=Θ(nmax{0,1+ 1−2β α }logkn)+Õ(nmax{−2+ 1+α+2τ 2α , 1−2β α + 1+α+2τ 2α +κ(1−2β+αγ)})
=Θ(nmax{0,1+ 1−2β α }logkn).
where k= {
0, α 6=2β−1, 1, α=2β−1. . Since Õ(n max{−2+ 1+α+2τ2α , 1−2β α + 1+α+2τ 2α +κ(1−2β+αγ)}) is lower order
term compared to Θ(nmax{0,1+ 1−2β α }logkn), we further have T1,R(Dn)= ( n
2σ2 µT1:R(I+
n
σ2 Λ1:R)
−1µ1:R ) (1+o(1))
= ( n
2σ2 µT (I+
n
σ2 Λ)−1µ
) (1+o(1))
This concludes the proof of the second statement.
Lemma 36. Under Assumptions 4, 5 and 6, with probability of at least 1−5δ, we have
T1(Dn)=
( 1
2 logdet(I+
n
σ2 ΛR)−
1 2 Tr ( I−(I+ n σ2 ΛR) −1 )) (1+o(1))=Θ(n 1 α ), (96)
Furthermore, let δ=n−q where 0≤q<min{ (2β−1)(α−1−2τ)4α2 , α−1−2τ 2α }. If we assume µ0 =0, we have
T2(Dn)= ( n
2σ2 µT (I+
n
σ2 Λ)−1µ
) (1+o(1))= { Θ(nmax{0,1+ 1−2β α }), α 6=2β−1,
Θ(logn), α=2β−1. (97)
Proof of Lemma 36. LetR=n 1 α+κ where 0≤κ< α−1−2τ2α2 . By Lemmas 32 and 35, with probability of at least 1−5δ we have |T1,R(Dn)−T1(Dn)|=Õ(n 1 α+κ(1−α)), (98)
and |T2,R(Dn)−T2(Dn)|=Õ ( ( 1
δ +1)nmax{( 1 α+κ) 1−2β 2 ,1+ 1−2β α + (1−2β)κ 2 ,−1−κα,1+ 1−2β α −κα}
) (99)
as well as
T1,R(Dn)=
( 1
2 logdet(I+
n
σ2 ΛR)−
1 2 Tr ( I−(I+ n σ2 ΛR) −1 )) (1+o(1))=Θ(n 1 α ), (100)
and T2,R(Dn)= ( n
2σ2 µT (I+
n
σ2 Λ)−1µ
) (1+o(1))= { Θ(nmax{0,1+ 1−2β α }), α 6=2β−1,
Θ(logn), α=2β−1. (101)
We then have
T1(Dn)=T1,R(Dn)+T1,R(Dn)−T1(Dn)=Θ(n 1 α )+Õ(n 1 α+κ(1−α))=Θ(n 1 α ).
Since Õ(n 1 α+κ(1−α)) is lower order term compared to Θ(n 1 α ), we further have
T1(Dn)=
( 1
2 logdet(I+
n
σ2 ΛR)−
1 2 Tr ( I−(I+ n σ2 ΛR) −1 )) (1+o(1))=Θ(n 1 α )
This concludes the proof of the first statement.
As for T2(Dn), we have
T2(Dn)=T2,R(Dn)+T2,R(Dn)−T2(Dn)
=Θ(nmax{0,1+ 1−2β α }logkn)+Õ ( ( 1
δ +1)nmax{( 1 α+κ) 1−2β 2 ,1+ 1−2β α + (1−2β)κ 2 ,−1−κα,1+ 1−2β α −κα} ) =Θ(nmax{0,1+ 1−2β α }logkn)+Õ ( nq+max{( 1 α+κ) 1−2β 2 ,1+ 1−2β α + (1−2β)κ 2 ,−1−κα,1+ 1−2β α −κα} )
where we use δ=n−q , k= {
0, α 6=2β−1, 1, α=2β−1. .
Since 0≤κ< α−1−2τ2α2 and 0≤q<min{ (2β−1)(α−1−2τ) 4α2 , α−1−2τ 2α }, we can choose κ< α−1−2τ
2α2 and κ is arbitrarily close to α−1−2τ2α2 such that 0≤q<min{ (2β−1)κ 2 ,κα}. Then we have ( 1 α+κ) 1−2β 2 +q<0, −1−κα+q<0, (1−2β)κ2 +q<0 and−κα+q<0. So we have
T2,R(Dn)=Θ(n max{0,1+ 1−2βα }logkn).
Since Õ (
( 1δ +1)n max{( 1α+κ) 1−2β 2 ,1+ 1−2β α + (1−2β)κ 2 ,−1−κα,1+ 1−2β α −κα}
) is lower order term
compared to Θ(nmax{0,1+ 1−2β α }logkn), we further have
T2(Dn)= ( n
2σ2 µT (I+
n
σ2 Λ)−1µ
) (1+o(1))
This concludes the proof of the second statement.
Proof of Theorem 7. Using Lemma 36 and noting that 1α>0, with probability of at least 1−5δ̃, we have
E F 0(Dn)=T1(Dn)+T2(Dn)
=
[ 1
2 logdet(I+
n
σ2 ΛR)−
1 2 Tr ( I−(I+ n σ2 ΛR) −1 )
+ n
2σ2 µTR(I+
n
σ2 ΛR)
−1µR ] (1+o(1))
=Θ(nmax{ 1 α , 1−2β α +1})
Furthermore, we have
logdet(I+ n σ2 Λ)−logdet(I+ n σ2 ΛR)
= ∞∑ p=R+1 log(1+ n σ2 λp)≤ n σ2 ∞∑ p=R+1 λp≤ n σ2 ∞∑ p=R+1 Cλp −α= n σ2 O(R1−α)
= n
σ2 O(n(1−α)( 1 α+κ))
=o(n 1 α ).
Then we have log det(I + nσ2 ΛR) = log det(I + n σ2 Λ)(1 + o(1)). Similarly we can prove Tr ( I−(I+ nσ2 Λ) −1) = Tr(I−(I+ nσ2 ΛR)−1)(1 + o(1)) and µT (I + nσ2 Λ)−1µ = µTR(I+ n σ2 ΛR) −1µR(1+o(1)). Letting δ=5δ̃, we get the result.
In the case of µ0>0, we have the following lemma:
Lemma 37. Assume that σ2 = Θ(1). Let R= n 1α+κ where 0<κ< α−1−2τα2 . Assume that µ0 > 0. Under Assumptions 4, 5 and 6, for sufficiently large nwith probability of at least 1−4δ we have
|T2,R(Dn)−T2(Dn)|=Õ ( ( 1
δ +1)nmax{1+( 1 α+κ) 1−2β 2 ,1−κα}
) .. (102)
Proof of Lemma 37. As for |T2(Dn)−T2,R(Dn)|, we have |T2(Dn)−T2,R(Dn)|= ∣∣∣∣f(x)T (I+ ΦΛΦTσ2 )−1f(x)−fR(x)T (I+ ΦΛΦTσ2 )−1fR(x) ∣∣∣∣ +
∣∣∣∣fR(x)T (I+ ΦΛΦTσ2 )−1fR(x)−fR(x)T (I+ ΦRΛRΦTRσ2 )−1fR(x) ∣∣∣∣. (103)
For the first term on the right-hand side of (103), we have∣∣∣∣f(x)T (I+ ΦΛΦTσ2 )−1f(x)−fR(x)T (I+ ΦΛΦTσ2 )−1fR(x) ∣∣∣∣
≤2 ∣∣∣∣f>R(x)T (I+ ΦΛΦTσ2 )−1fR(x) ∣∣∣∣+∣∣∣∣f>R(x)T (I+ ΦΛΦTσ2 )−1f>R(x) ∣∣∣∣
≤2‖f>R(x)‖2‖(I+ ΦΛΦT
σ2 )−1fR(x)‖2+‖f>R(x)‖2‖(I+
ΦΛΦT
σ2 )−1‖2‖f>R(x)‖2
≤2‖f>R(x)‖2‖(I+ ΦΛΦT
σ2 )−1fR(x)‖2+‖f>R(x)‖22.
Applying Corollary 19 and Lemma 31, with probability of at least 1−4δ, we have∣∣∣∣f(x)T (I+ ΦΛΦTσ2 )−1f(x)−fR(x)T (I+ ΦΛΦTσ2 )−1fR(x) ∣∣∣∣
≤2Õ (√ ( 1
δ +1)nR1−2β
) Õ( √ ( 1
δ +1)n)+Õ((
1 δ +1)nR1−2β)
=2Õ ( ( 1
δ +1)n1+(
1 α+κ) 1−2β 2 ) +Õ(( 1
δ +1)n1+( 1 α+κ)(1−2β))
=2Õ ( ( 1
δ +1)n1+(
1 α+κ) 1−2β 2 ) .
As for the second term on the right-hand side of (80), according to Lemma 28, Corollary 26 and Lemma 30, we have∣∣∣∣fR(x)T (I+ ΦΛΦTσ2 )−1fR(x)−fR(x)T (I+ ΦRΛRΦTRσ2 )−1fR(x) ∣∣∣∣ =
∣∣∣∣∣∣ ∞∑ j=1 (−1)jfR(x)T ( (I+ ΦRΛRΦ T R σ2 )−1 Φ>RΛ>RΦ T >R σ2 )j (I+ ΦRΛRΦ T R σ2 )−1fR(x) ∣∣∣∣∣∣ ≤ ∞∑ j=1 ‖(I+ ΦRΛRΦ T R σ2 )−1‖j−12 ·‖ Φ>RΛ>RΦ T >R σ2 ‖j2 ·‖(I+ ΦRΛRΦ T R σ2 )−1fR(x)‖22
= ∞∑ j=1 Õ(n−jκα)Õ(( 1 δ +1)n)
=Õ(( 1
δ +1)n1−κα).
(104)
By (80), we have |T2(Dn)−T2,R(Dn)|=Õ ( ( 1
δ +1)n1+(
1 α+κ) 1−2β 2 ) +Õ(( 1
δ +1)n1−κα)
=Õ ( ( 1
δ +1)nmax{1+( 1 α+κ) 1−2β 2 ,1−κα}
) .
Lemma 38. Assume that σ2 = Θ(1). Let R= n 1α+κ where 0<κ<min{α−1−2τ2α2 , 2β−1 α2 }. Assume that µ0>0. Under Assumptions 4, 5 and 6, with probability of at least 1−δ, we have
T2,R(Dn)= n
2σ2 µ20+Õ(n
max{ 1+7α+2τ8α ,1+ 1−2β α }). (105)
Proof of Lemma 38. Let
A=(I+ n
σ2 ΛR)
−γ/2Λ γ/2 R (Φ T RΦR−nI)Λ γ/2 R (I+
n
σ2 ΛR)
−γ/2, (106)
where 1+α+2τ2α <γ≤1. By Corollary 22, with probability of at least 1−δ, we have
‖A‖2 =Õ(n 1−2γα+α+2τ 2α ). (107)
When n is sufficiently large, ‖A‖2 is less than 1. Let µR,1 = (µ0,0,...,0) and µR,2 = (0,µ1,...,µR). Then µR=µR,1+µR,2. Let Λ̃1,R=diag{1,λ1,...,λR} and I0,R=(0,1,...,1). Then ΛR=Λ̃1,RI0,R. Let B = (I + nσ2 ΛR) −1/2Λ̃ 1/2 1,R(Φ T RΦR − nI)Λ̃ 1/2 1,R(I + n σ2 ΛR) −1/2. By Corollary 23, we have
‖B‖2 =O( √ logRδ n 1 2 ). By Lemma 34, we have
T2,R(Dn)= n
2σ2 µTR(I+
n
σ2 ΛR)
−1µR
+ 1
2 ∞∑ j=1 [ (−1)j+1µTR 1 σ2 (I+ n σ2 ΛR) −1(ΦTRΦR−nI) ( 1 σ2 (I+ n σ2 ΛR) −1ΛR(Φ T RΦR−nI) )j−1 (I+ n
σ2 ΛR)
−1µR ] (108)
As for the first term on the right hand side of (108), by Lemma 15, we have
n
2σ2 µT (I+
n σ2 Λ)−1µ≤ n 2σ2
( µ20+ R∑ p=1 C2µp −2β 1+ nσ2Cλp −α ) = n 2σ2 µ20+Õ(n max{0,1+ 1−2βα }).
We defineQ1,j ,Q2,j andQ3,j by
Q1,j=µ T R,1
1
σ2 (I+
n
σ2 ΛR)
−1(ΦTRΦR−nI) ( 1
σ2 (I+
n
σ2 ΛR)
−1ΛR(Φ T RΦR−nI) )j−1 (I+ n
σ2 ΛR)
−1µR,1
Q2,j=µ T R,1
1
σ2 (I+
n
σ2 ΛR)
−1(ΦTRΦR−nI) ( 1
σ2 (I+
n
σ2 ΛR)
−1ΛR(Φ T RΦR−nI) )j−1 (I+ n
σ2 ΛR)
−1µR,2
Q3,j=µ T R,2
1
σ2 (I+
n
σ2 ΛR)
−1(ΦTRΦR−nI) ( 1
σ2 (I+
n
σ2 ΛR)
−1ΛR(Φ T RΦR−nI) )j−1 (I+ n
σ2 ΛR)
−1µR,2
(109)
The quantity Q3,j actually shows up in the case of µ0 = 0 in the proof of Lemma 35. By (92), (94) and (95), we have that
| ∞∑ j=1 (−1)j+1Q3,j |= | ∞∑ j=1 (−1)j+1Õ(n (j−1)(1−α+2τ) 2α )o(nmax{0,1+ 1−2β α })|=o(nmax{0,1+ 1−2β α }).
(110) ForQ1,j , we have
Q1,1 = 1
σ2j µTR,1(I+
n
σ2 ΛR)
−1+ γ2B(I+ n
σ2 ΛR)
−1+ γ2 µR,1
≤ 1 σ2j ‖µR,1‖22‖(I+ n σ2 ΛR) −1+ γ2 ‖22‖B‖2
=O( √ log R
δ n
1 2 ),
where in the last equality we use ‖B‖2 =O( √ logRδ n 1 2 ). For j≥2, we have
Q1,j= 1
σ2j µTR,1(I+
n
σ2 ΛR)
−1+ γ2B ( (I+ n
σ2 ΛR)
−1+γΛ1−γR A )j−2 (I+ n
σ2 ΛR)
−1+γΛ1−γR
B(I+ n
σ2 ΛR)
−1+ γ2 µR,1
≤ 1 σ2j ‖µR,1‖22‖(I+ n σ2 ΛR) −1+ γ2 ‖22‖B‖22‖A‖ j−2 2 ‖(I+ n σ2 ΛR) −1+γΛ1−γR ‖ j−1 2
=O(log R
δ n·n
(j−2)(1−2γα+α+2τ) 2α ·n−(1−γ)(j−1))
=O(log R
δ nγ ·n
(j−2)(1−α+2τ) 2α ).
Then we have
| ∞∑ j=1 (−1)j+1Q1,j |≤O( √ log R δ n 1 2 )+ ∞∑ j=2 O(log R δ nγ ·n (j−2)(1−α+2τ) 2α )=O(log R δ nγ) (111)
ForQ2,j , we have
Q2,j= 1
σ2j µTR,1(I+
n
σ2 ΛR)
−1+ γ2B ( (I+ n
σ2 ΛR)
−1+γΛ1−γR A )j−1 (I+ n
σ2 Λ)−1+ γ 2 Λ̃ − γ2 1,RµR,2
≤ 1 σ2j ‖µR,1‖2‖B‖2‖A‖j−12 ‖(I+ n σ2 ΛR) −1+γΛ1−γR ‖ j−1 2 ‖(I+ n σ2 Λ)−1+ γ 2 Λ̃ − γ2 1,RµR,2‖2
=O( √ log R
δ n
1 2 ·n (j−1)(1−α+2τ) 2α )‖(I+ n
σ2 Λ)−1+ γ 2 Λ̃ − γ2 1,RµR,2‖2.
Since ‖(I+ nσ2 Λ) −1+ γ2 Λ̃ − γ2 1,RµR,2‖2 is actually the case of µ0 = 0, we can use (93) in the proof of Lemma 35 and get
‖(I+ n σ2 Λ)−1+ γ 2 Λ̃ − γ2 1,RµR,2‖ 2 2 =‖(I+ n σ2 Λ1:R) −1+γ/2Λ −γ/2 1:R µ1:R‖ 2 2
=Õ(nmax{−2+γ, 1−2β α +γ+κ(1−2β+αγ)}
=Õ(nmax{−2+γ, 1−2β α +γ+κ(1−2β+αγ)})
=o(nγ),
(112)
where in the last equality we use κ< 2β−1α2 . Then we have
| ∞∑ j=1 (−1)j+1Q2,j |≤ ∞∑ j=1 o( √ log R δ n 1+γ 2 ·n (j−1)(1−α+2τ) 2α )=o( √ log R δ n 1+γ 2 ) (113)
Choosing γ= 12 (1+ 1+α+2τ 2α )= 1+3α+2τ 4α <1, we have
T2,R(Dn)= n
2σ2 µTR(I+
n
σ2 ΛR)
−1µR+ ∞∑ j=1 (−1)j+1(Q1,j+Q2,j+Q3,j)
= n
2σ2 µ20+Õ(n
max{0,1+ 1−2βα })+o(nmax{0,1+ 1−2β α })+O(log R
δ nγ)+o(
√ log R
δ n
1+γ 2 )
= n
2σ2 µ20+Õ(n
max{ 1+γ2 ,1+ 1−2β α })
= n
2σ2 µ20+Õ(n
max{ 1+7α+2τ8α ,1+ 1−2β α }).
Proof of Theorem 8. Let R = n 1 α+κ where 0 < κ < min{α−1−2τ2α2 , 2β−1 α2 }. Since 0 ≤ q < min{ 2β−12 , α} · min{ α−1−2τ 2α2 , 2β−1 α2 }, we can choose κ < min{ α−1−2τ 2α2 , 2β−1 α2 } and κ
is arbitrarily close to κ<min{α−1−2τ2α2 , 2β−1 α2 } such that 0≤ q <min{ (2β−1)κ 2 ,κα}. Then we have ( 1α+κ) 1−2β 2 +q<0, and−κα+q<0. As for T2(Dn), we have
T2(Dn)≤T2,R(Dn)+|T2,R(Dn)−T2(Dn)|
= n
2σ2 µ20+Õ(n
max{ 1+7α+2τ8α ,1+ 1−2β α })+Õ ( ( 1δ +1)n max{1+( 1α+κ) 1−2β 2 ,1−κα} )
= n
2σ2 µ20+Õ(n
max{ 1+7α+2τ8α ,1+ 1−2β α })+Õ ( nq+max{1+( 1 α+κ) 1−2β 2 ,1−κα} ) = n
2σ2 µ20+o(n).
By Lemma 36, we have T1(Dn) = O(n 1 α ). Hence E F 0(Dn) = T1(Dn) + T2(Dn) =
n 2σ2µ 2 0+o(n).
D.2 PROOFS RELATED TO THE ASYMPTOTICS OF THE GENERALIZATION ERROR
Lemma 39. Assume σ2 = Θ(nt) where 1− α1+2τ < t < 1. Under Assumptions 4, 5 and 6, with probability of at least 1−δ over sample inputs (xi)ni=1, we have
G1(Dn)= 1+o(1)
2σ2 ( Tr(I+ nσ2 ΛR) −1ΛR−‖Λ1/2R (I+ n σ2 ΛR) −1‖2F ) =Θ ( n (1−α)(1−t) α ) . (114)
Proof of Lemma 39. Let G1,R(Dn) = E(xn+1,yn+1)(T1,R(Dn+1)− T1,R(Dn)), where R = nC for some constant C. By Lemma 32, we have that
|G1(Dn)−G1,R(Dn)|= ∣∣E(xn+1,yn+1)[T1(Dn+1)−T1,R(Dn+1)]−[T1(Dn)−T1,R(Dn)]∣∣
= ∣∣E(xn+1,yn+1)O((n+1)R1−α)∣∣+∣∣O(nR1−α)]∣∣ =O( 1σ2nR 1−α).
(115)
Define ηR=(φ0(xn+1),φ1(xn+1),...,φR(xn+1))T and Φ̃R=(ΦTR,ηR) T . As forG1,R(Dn), we have
G1,R(Dn)=E(xn+1,yn+1)(T1,R(Dn+1)−T1,R(Dn))
=E(xn+1,yn+1)
( 1
2 logdet(I+
Φ̃RΛRΦ̃ T R
σ2 )− 1 2 Tr(I−(I+ Φ̃RΛRΦ̃
T R
σ2 )−1)
)
− ( 1
2 logdet(I+
ΦRΛRΦ T R
σ2 )− 1 2 Tr(I−(I+ ΦRΛRΦ
T R
σ2 )−1)
)
= 1
2
( E(xn+1,yn+1)logdet(I+ Φ̃RΛRΦ̃R T
σ2 )−logdet(I+ ΦRΛRΦ
T R
σ2 )
)
− 1 2
( E(xn+1,yn+1)Tr(I−(I+ Φ̃RΛRΦ̃ T R
σ2 )−1)−Tr(I−(I+ ΦRΛRΦ
T R
σ2 )−1)
) .
(116)
As for the first term in the right hand side (116), we have
1
2
( E(xn+1,yn+1)logdet(I+ Φ̃RΛRΦ̃ T R
σ2 )−logdet(I+ ΦRΛRΦ
T R
σ2 )
)
= 1
2
( E(xn+1,yn+1)logdet(I+ ΛRΦ̃ T RΦ̃R σ2 )−logdet(I+ ΛRΦ T RΦR σ2 ) )
= 1
2
( E(xn+1,yn+1)logdet(I+ ΛRΦ T RΦR+ηRη T R
σ2 )−logdet(I+ ΛRΦ T RΦR σ2 ) ) = 1
2
( E(xn+1,yn+1)logdet ( (I+ ΛRΦ T RΦR σ2 )−1(I+ ΛRΦ T RΦR σ2 + ΛRηRη T R σ2 ) )) = 1
2
( E(xn+1,yn+1)logdet ( I+(I+ ΛRΦ T RΦR σ2 )−1 ΛRηRη T R σ2 )) = 1
2
( E(xn+1,yn+1)log ( 1+ 1
σ2 ηTR(I+ ΛRΦ T RΦR σ2 )−1ΛRηR
))
Let
A=(I+ n
σ2 ΛR)
−1/2Λ 1/2 R (Φ T RΦR−nI)Λ 1/2 R (I+
n
σ2 ΛR)
−1/2. (117)
According to Corollary 22, with probability of at least 1 − δ, we have ‖ 1σ2A‖2 = O( √
logRδ n 1−α+2τ 2α − (1+2τ)t 2α ) = o(1). When n is sufficiently large, ‖ 1σ2A‖2 is less than 1. By Lemma 27, we have
ηTR(I+ ΛRΦ T RΦR σ2 )−1ΛRηR
=ηTR(I+ n
σ2 ΛR)
−1ΛRηR+ ∞∑ j=1 (−1)jηTR ( 1 σ2 (I+ n σ2 ΛR) −1ΛR(Φ T RΦR−nI) )j (I+ n σ2 ΛR) −1ΛRηR
=ηTR(I+ n
σ2 ΛR)
−1ΛRηR+ ∞∑ j=1 (−1)j 1 σ2j ηTR(I+ n σ2j ΛR) −1/2Λ 1/2 R A j(I+ n σ2 ΛR) −1/2Λ 1/2 R ηR
≤ηTR(I+ n
σ2 ΛR)
−1ΛRηR+ ∞∑ j=1 ‖A‖j2‖(I+ n σ2 ΛR) −1/2Λ 1/2 R ηR‖ 2 2
≤ R∑ p=1 φ2p(xn+1) Cλp −α 1+nCλp−α/σ2 + ∞∑ j=1 ‖ 1 σ2 A‖j2(logR)j/2) R∑ p=1 φ2p(xn+1) Cλp −α 1+nCλp−α/σ2 ≤ R∑ p=1 Cλp −αp2τ 1+nCλp−α/σ2 + ∞∑ j=1 ‖ 1 σ2 A‖j2(logR)j/2) R∑ p=1 Cλp −αp2τ 1+nCλp−α/σ2
≤O(n (1−α)(1−t) α )+ ∞∑ j=1 ‖ 1 σ2 A‖j2(logR)j/2)O(n (1−α)(1−t) α )
=O(n (1−α)(1−t)
α )=o(1), (118)
where we use Lemma 15 in the last inequality. Next we have
1
2
( E(xn+1,yn+1)logdet(I+ Φ̃RΛRΦ̃ T R
σ2 )−logdet(I+ ΦRΛRΦ
T R
σ2 )
)
= 1
2
( E(xn+1,yn+1)log ( 1+ 1
σ2 ηTR(I+ ΛRΦ T RΦR σ2 )−1ΛRηR )) = 1
2
( E(xn+1,yn+1) ( 1
σ2 ηTR(I+ ΛRΦ T RΦR σ2 )−1ΛRηR
) (1+o(1)) ) = 1
2σ2
( Tr(I+
ΛRΦ T RΦR σ2 )−1ΛR
) (1+o(1)),
where in the last equality we use the fact that E(xn+1,yn+1)ηRηTR=I . By Lemma 27, we have
Tr(I+ ΛRΦ T RΦR σ2 )−1ΛR
=Tr(I+ n
σ2 ΛR)
−1ΛR+ ∞∑ j=1 (−1)jTr ( 1 σ2 (I+ n σ2 ΛR) −1ΛR(Φ T RΦR−nI) )j (I+ n σ2 ΛR) −1ΛR
=Tr(I+ n
σ2 ΛR)
−1ΛR+ ∞∑ j=1 (−1)jTr 1 σ2 (I+ n σ2 ΛR) −1/2Λ 1/2 R A j(I+ n σ2 ΛR) −1/2Λ 1/2 R .
By Lemma 15, we have
Tr(I+ n
σ2 ΛR) −1ΛR≤ R∑ p=1
Cλp −α
1+nCλp−α/σ2 =Θ(n
(1−α)(1−t) α )
Tr(I+ n
σ2 ΛR) −1ΛR≥ R∑ p=1
Cλp −α
1+nCλp−α/σ2 =Θ(n
(1−α)(1−t) α ).
Overall,
Tr(I+ n
σ2 ΛR)
−1ΛR=Θ(n (1−α)(1−t) α ). (119)
Since ‖ 1σ2A‖ j 2 =o(1), we have that the absolute values of diagonal entries of 1 σ2jA j are at most o(1)). Let (Aj)p,p denote the (p,p)-th entry of the matrixAj . Then we have∣∣∣∣Tr 1σ2 (I+ nσ2 ΛR)−1/2Λ1/2R Aj(I+ nσ2 ΛR)−1/2Λ1/2R
∣∣∣∣ =
∣∣∣∣∣ R∑ p=1 λp 1 σ2j (A j)p,p 1+nλp/σ2 ∣∣∣∣∣≤ R∑ p=1 λp‖A‖j2 1+nλp/σ2 =Θ(n (1−α)(1−t) α )Õ(n j(1−α+2τ−(1+2τ)t) 2α (logR)j/2), (120)
where in the last step we used (119). According to (119) and (120), we have
1
2
( E(xn+1,yn+1)logdet(I+ Φ̃RΛRΦ̃ T R
σ2 )−logdet(I+ ΦRΛRΦ
T R
σ2 )
)
= 1
2σ2
( Tr(I+
ΛRΦ T RΦR σ2 )−1ΛR
) (1+o(1))=Θ(n (1−α)(1−t) α )+ ∞∑ j=1 Θ(n (1−α)(1−t) α )Õ(n j(1−α+2τ−(1+2τ)t) 2α (logR)j/2)
=Θ(n (1−α)(1−t) α )+Θ(n (1−α)(1−t) α )o(1)=Θ(n (1−α)(1−t) α )
= 1
2σ2
( Tr(I+ n
σ2 ΛR)
−1ΛR ) (1+o(1)).
(121)
Using the Woodbury matrix identity, the second term in the right hand side (116) is given by
1
2
( E(xn+1,yn+1)Tr(I−(I+ Φ̃RΛRΦ̃ T R
σ2 )−1−Tr(I−(I+ ΦRΛRΦ
T R
σ2 )−1
)
= 1
2
( E(xn+1,yn+1)Tr( 1
σ2 Φ̃R(I+
1
σ2 ΛRΦ̃
T RΦ̃R) −1ΛRΦ̃ T R−Tr(
1
σ2 ΦR(I+
1
σ2 ΛRΦ
T RΦR) −1ΛRΦ T R ) = 1
2
( E(xn+1,yn+1)Tr( 1
σ2 (I+
1
σ2 ΛRΦ̃
T RΦ̃R) −1ΛRΦ̃ T RΦ̃R−Tr(
1
σ2 (I+
1
σ2 ΛRΦ
T RΦR) −1ΛRΦ T RΦR ) =−1
2
( E(xn+1,yn+1)Tr(I+ 1
σ2 ΛRΦ̃
T RΦ̃R) −1−Tr(I+ 1 σ2 ΛRΦ T RΦR)
−1 )
=−1 2
( E(xn+1,yn+1)Tr(I+ 1
σ2 ΛRΦ
T RΦR+
1
σ2 ΛRηRη
T R) −1−Tr(I+ 1
σ2 ΛRΦ
T RΦR)
−1 )
= 1
2σ2
( E(xn+1,yn+1)Tr (I+ 1σ2 ΛRΦ T RΦR) −1ΛRηRη T R(I+ 1 σ2 ΛRΦ T RΦR) −1
1+ 1σ2 η T R(I+ 1 σ2 ΛRΦ T RΦR) −1ΛRηR
) ,
where the last equality uses the Sherman–Morrison formula. According to (118), we get
1
2σ2
( E(xn+1,yn+1)Tr (I+ 1σ2 ΛRΦ T RΦR) −1ΛRηRη T R(I+ 1 σ2 ΛRΦ T RΦR) −1
1+ 1σ2 η T R(I+ 1 σ2 ΛRΦ T RΦR) −1ΛRηR ) = 1
2σ2
( E(xn+1,yn+1)Tr(I+ 1
σ2 ΛRΦ
T RΦR) −1ΛRηRη T R(I+
1
σ2 ΛRΦ
T RΦR) −1(1+o(1)) ) = 1+o(1)
2σ2 Tr(I+
1
σ2 ΛRΦ
T RΦR) −1ΛR(I+ 1
σ2 ΛRΦ
T RΦR)
−1
= 1+o(1)
2σ2 TrΛ
1/2 R (I+
1
σ2 Λ
1/2 R Φ T RΦRΛ 1/2 R ) −1Λ 1/2 R (I+
1
σ2 ΛRΦ
T RΦR)
−1
= 1+o(1)
2σ2 Tr(I+
1
σ2 Λ
1/2 R Φ T RΦRΛ 1/2 R ) −1Λ 1/2 R (I+
1
σ2 ΛRΦ
T RΦR) −1Λ 1/2 R
= 1+o(1)
2σ2 Tr(I+
1
σ2 Λ
1/2 R Φ T RΦRΛ 1/2 R ) −1ΛR(I+ 1
σ2 Λ
1/2 R Φ T RΦRΛ 1/2 R ) −1
= 1+o(1)
2σ2 ‖Λ1/2R (I+
1
σ2 Λ
1/2 R Φ T RΦRΛ 1/2 R ) −1‖2F
= 1+o(1)
2σ2 ‖Λ1/2R (I+
n
σ2 ΛR)
−1/2(I+ 1
σ2 A)−1(I+
n
σ2 ΛR)
−1/2‖2F ,
where in the penultimate equality we use Tr(BBT )=‖B‖2F , ‖B‖F is the Frobenius norm ofA, and in the last equality we use the definition ofA (117). Then we have
1+o(1)
2σ2 ‖Λ1/2R (I+
n
σ2 ΛR)
−1/2(I+ 1
σ2 A)−1(I+
n
σ2 ΛR)
−1/2‖2F
= 1+o(1)
2σ2 ‖Λ1/2R (I+
n
σ2 ΛR)
−1/2(I+ ∞∑ j=1 (−1)j 1 σ2j Aj)(I+ n σ2 ΛR) −1/2‖2F
= 1+o(1)
2σ2 ‖Λ1/2R (I+
n
σ2 ΛR)
−1+ ∞∑ j=1 (−1)j 1 σ2j Λ 1/2 R (I+ n σ2 ΛR) −1/2Aj(I+ n σ2 ΛR) −1/2‖2F .
(122) By Lemma 15, we have
‖Λ1/2R (I+ n
σ2 ΛR)
−1‖F ≤ √√√√ R∑ p=1
Cλp−α
(1+nCλp−α/σ2)2 =Θ(n
(1−α)(1−t) 2α )
‖Λ1/2R (I+ n
σ2 ΛR)
−1‖F ≥ √√√√ R∑ p=1
Cλp−α
(1+nCλp−α/σ2)2 =Θ(n
(1−α)(1−t) 2α ).
Overall, we have
‖Λ1/2R (I+ n
σ2 ΛR)
−1‖F =Θ(n (1−α)(1−t) 2α ). (123)
Since ‖ 1σ2A‖2 =O( √ logRδ n 1−α+2τ 2α − (1+2τ)t 2α )=o(1) , we have
‖ 1 σ2j Λ 1/2 R (I+ n σ2 ΛR) −1/2Aj(I+ n σ2 ΛR) −1/2‖F ≤‖Λ1/2R (I+ n
σ2 ΛR)
−1/2‖F ‖ 1
σ2 A‖j2‖(I+
n
σ2 ΛR)
−1/2‖2
=O(n (1−α)(1−t) 2α )Õ(n j(1−α+2τ−(1+2τ)t) 2α (logR)j/2),
(124)
where in the first inequality we use the fact that ‖AB‖F ≤ ‖A‖F ‖B‖2 when B is symmetric. By Lemma 15, we have
1
σ2j ∣∣∣TrΛ1/2R (I+ nσ2 ΛR)−1Λ1/2R (I+ nσ2 ΛR)−1/2Aj(I+ nσ2 ΛR)−1/2∣∣∣ =
∣∣∣∣∣ R∑ p=1 λp(( 1 σ2A) j)p,p (1+nλp/σ2)2 ∣∣∣∣∣≤ R∑ p=1 λp‖ 1σ2A‖ j 2 (1+nλp/σ2)2 =Θ(n (1−α)(1−t) α )Õ(n j(1−α+2τ−(1+2τ)t) 2α (logR)j/2),
(125) According to (123), (124) and (125), we have
1
2
( E(xn+1,yn+1)Tr(I−(I+ Φ̃RΛRΦ̃ T R
σ2 )−1−Tr(I−(I+ ΦRΛRΦ
T R
σ2 )−1
)
= 1+o(1)
2σ2 Tr(I+
1
σ2 ΛRΦ
T RΦR) −1ΛR(I+ 1
σ2 ΛRΦ
T RΦR)
−1
= 1+o(1)
2σ2 ‖Λ1/2R (I+
n
σ2 ΛR)
−1+ ∞∑ j=1 (−1)j 1 σ2j Λ 1/2 R (I+ n σ2 ΛR) −1/2Aj(I+ n σ2 ΛR) −1/2‖2F
= 1+o(1)
2σ2
( ‖Λ1/2R (I+ n
σ2 ΛR) −1‖2F + ∞∑ j=1 ∥∥∥∥ 1σ2j Λ1/2R (I+ nσ2 ΛR)−1/2Aj(I+ nσ2 ΛR)−1/2 ∥∥∥∥2 F
+2TrΛ 1/2 R (I+
n
σ2 ΛR) −1 ∞∑ j=1 (−1)j 1 σ2j Λ 1/2 R (I+ n σ2 ΛR) −1/2Aj(I+ n σ2 ΛR) −1/2 )
= 1+o(1)
2σ2
( Θ(n (1−α)(1−t) α )+ ∞∑ j=1 1 σ2j O(n (1−α)(1−t) α )Õ(n j(1−α+2τ−(1+2τ)t) 2α (logR)j/2)
+2 ∞∑ j=1 1 σ2j Θ(n (1−α)(1−t) α )Õ(n j(1−α+2τ−(1+2τ)t) 2α (logR)j/2) ) =Θ(n (1−α)(1−t) α )= 1+o(1)
2σ2 ‖Λ1/2R (I+
n
σ2 ΛR) −1‖2F . (126)
Combining (121) and (126) we get that G1,R(Dn) = 1+o(1) 2σ2 (Tr(I + n σ2 ΛR) −1ΛR + ‖Λ1/2R (I + n σ2 ΛR) −1‖2F ) = Θ(n (1−α)(1−t)
α ). From (115) we have that G1(Dn) ≤ G1,R(Dn) + |G1(Dn) − G1,R(Dn)| = Θ(n (1−α)(1−t) α ) +O(n 1σ2R 1−α). Choosing R = n( 2α−1 α(α−1) +1)(1−t) we conclude the proof.
Lemma 40. Assume σ2 =Θ(nt) where 1− α1+2τ <t<1. Let S=n D. Assume that ‖ξ‖2 =1. When n is sufficiently large, with probability of at least 1−2δ we have
‖(I+ 1σ2 ΦSΛSΦ T S ) −1ΦSΛSξ‖2 =O( √ ( 1δ +1)n·n −(1−t)). (127)
Proof of Lemma 40. Using the Woodbury matrix identity, we have that
((I+ 1
σ2 ΦSΛSΦ
T S ) −1ΦSΛSξ= [ I−ΦS(σ2I+ΛSΦTSΦS)−1ΛSΦTS ] ΦSΛSξ
=ΦSΛSξ−ΦS(σ2I+ΛSΦTSΦS)−1ΛSΦTSΦSΛSξ =σ2ΦS(σ 2I+ΛSΦ T SΦS) −1ΛSξ.
(128)
Let A= (I+ nσ2 ΛS) −γ/2Λ γ/2 S (Φ T SΦS−nI)Λ γ/2 S (I+ n σ2 ΛS) −γ/2, where γ > 1+α+2τ−(1+2τ+2α)t2α(1−t) . By Corollary 22, with probability of at least 1−δ, we have ‖ 1σ2A‖2 =Õ(n 1+α+2τ−(1+2τ+2α)t
2α −γ(1−t)). When n is sufficiently large, ‖ 1σ2A‖2 is less than 1. By Lemma 27, we have
(I+ 1
σ2 ΛSΦ
T SΦS)
−1
=(I+ n
σ2 ΛS)
−1+ ∞∑ j=1 (−1)j ( 1 σ2 (I+ n σ2 ΛS) −1ΛS(Φ T SΦS−nI) )j (I+ n σ2 ΛS) −1.
Then we have
‖(σ2I+ΛSΦTSΦS)−1ΛSξ‖2
= 1
σ2 ∥∥∥∥∥∥ (I+ n σ2 ΛS) −1+ ∞∑ j=1 (−1)j ( 1 σ2 (I+ n σ2 ΛS) −1ΛS(Φ T SΦS−nI) )j (I+ n σ2 ΛS) −1 ΛSξ ∥∥∥∥∥∥
2
≤ 1 σ2 ‖(I+ n σ2 ΛS) −1ΛSξ‖2+ ∞∑ j=1 ∥∥∥∥∥ ( 1 σ2 (I+ n σ2 ΛS) −1ΛS(Φ T SΦS−nI) )j (I+ n σ2 ΛS) −1ΛSξ ∥∥∥∥∥ 2 . (129) For the first term in the right hand side of the last equation, we have
‖(I+ n σ2 ΛS) −1ΛSξ‖2≤‖(I+ n σ2 ΛS) −1ΛS‖2‖ξ‖2≤ σ2 n =O(n−1). (130)
Using the fact that ‖ 1σ2A‖2 = Õ(n 1+α+2τ−(1+2τ+2α)t 2α −γ(1−t)) and ‖(I+ nσ2 ΛS) −1ΛS‖2 ≤ n−1, we have∥∥∥∥∥ ( 1 σ2 (I+ n σ2 ΛS) −1ΛS(Φ T SΦS−nI) )j (I+ n σ2 ΛS) −1ΛSξ ∥∥∥∥∥ 2
= 1
σ2j ∥∥∥∥(I+ nσ2 ΛS)−1+ γ2 Λ1− γ2S (A(I+ nσ2 ΛS)−1+γΛ1−γS )j−1A(I+ nσ2 ΛS)−1+ γ2 Λ− γ2S ΛSξ ∥∥∥∥
2
≤n(1−t)(−1+ γ 2 +(−1+γ)(j−1))Õ(n j(1+α+2τ−(1+2τ+2α)t) 2α −jγ(1−t))‖(I+ n
σ2 ΛS)
−1+ γ2 Λ 1− γ2 S ξ‖2
=Õ(n− γ 2 (1−t)+ (1−α+2τ−(1+2τ)t)j 2α )‖(I+ n
σ2 ΛS)
−1+ γ2 Λ 1− γ2 S ‖2‖ξ‖2
=Õ(n− γ 2 (1−t)+ (1−α+2τ−(1+2τ)t)j 2α )O(n(−1+γ/2)(1−t))
=Õ(n−(1−t)+ (1−α+2τ−(1+2τ)t)j
2α ). (131)
Using (129), (130) and (131), we have
‖(σ2I+ΛSΦTSΦS)−1ΛSξ‖2
=σ−2Õ(n−1)+ ∞∑ j=1 Õ(n−1+ (1−α+2τ−(1+2τ)t)j 2α )
=Õ(n−(1−t))+Õ(n−1+ 1−α+2τ−(1+2τ)t 2α )
=Õ(n−(1−t)).
(132)
By Corollary 20, with probability of at least 1−δ, we have ‖ΦS(σ2I+ΛSΦTSΦS)−1ΛSξ‖2 =Õ( √ ( 1
δ +1)n‖(σ2I+ΛSΦTSΦS)−1ΛSξ‖2)
=Õ( √ ( 1
δ +1)n·n−(1−t)).
From (128) we get ‖(I + 1σ2 ΦSΛSΦ T S ) −1fS(x)‖2 = Õ( √ ( 1δ +1)n ·n
−(1−t)). This concludes the proof.
Lemma 41. Assume σ2 = Θ(nt) where 1 − α1+2τ < t < 1. Let δ = n −q where 0≤q< [α−(1+2τ)(1−t)](2β−1)4α2 . Under Assumptions 4, 5 and 6, assume that µ0 =0. Then with probability of at least 1−6δ over sample inputs (xi)ni=1, we haveG2(Dn)= (1+o(1)) 2σ2 ‖(I+ n σ2 ΛR) −1µR‖22 =
Θ(nmax{−2(1−t), (1−2β)(1−t) α }logk/2n), where k= {
0, 2α 6=2β−1, 1, 2α=2β−1. .
Proof of Lemma 41. Let S = nD. Let G2,S(Dn) = E(xn+1,yn+1)(T2,S(Dn+1) − T2,S(Dn)). By Lemma 33, when S is large enough, with probability of at least 1−3δ we have that
c (133)
Let Λ1:S = diag{λ1, ... , λS}, Φ1:S = (φ1(x), φ1(x), ... , φS(x)) and µ1:S = (µ1, ... , µS). Since µ0 = 0, we have T2,S(Dn) = 12σ2µ T 1:SΦ T 1:S(I + 1 σ2 Φ1:SΛ1:SΦ T 1:S) −1Φ1:Sµ1:S . Define η1:S = (φ1(xn+1),...,φS(xn+1)) T and Φ̃1:S=(ΦT1:S ,η1:S) T . In the proof of Lemma 34, we showed that
T2,S(Dn)= 1
2σ2 µT1:SΦ T 1:S(I+
1
σ2 Φ1:SΛ1:SΦ
T 1:S) −1Φ1:Sµ1:S
= 1
2 µT1:SΛ −1 1:Sµ1:S−
1 2 µT1:SΛ −1 1:S(I+ 1 σ2 Λ1:SΦ T 1:SΦ1:S) −1µ1:S .
We have
G2,S(Dn)=E(xn+1,yn+1)(T2,S(Dn+1)−T2,S(Dn)) =E(xn+1,yn+1) ( 1
2 µT1:SΛ −1 1:Sµ1:S−
1 2 µT1:SΛ −1 1:S(I+ 1 σ2 Λ1:SΦ̃ T S Φ̃S) −1µ1:S ) − ( 1
2 µT1:SΛ −1 1:Sµ1:S−
1 2 µT1:SΛ −1 1:S(I+ 1 σ2 Λ1:SΦ T 1:SΦ1:S) −1µ1:S) ) =E(xn+1,yn+1) ( 1
2 µT1:SΛ −1 1:S(I+
1
σ2 Λ1:SΦ
T 1:SΦ1:S) −1µ1:S− 1
2 µT1:SΛ −1 1:S(I+
1
σ2 Λ1:SΦ̃
T S Φ̃S) −1µ1:S ) =E(xn+1,yn+1) ( 1
2σ2 µT1:SΛ −1 1:S
(I+ 1σ2 Λ1:SΦ T 1:SΦ1:S) −1Λ1:Sη1:Sη T 1:S(I+ 1 σ2 Λ1:SΦ T 1:SΦ1:S) −1
1+ 1σ2 η T 1:S(I+ 1 σ2 Λ1:SΦ T 1:SΦ1:S) −1Λ1:Sη1:S µ1:S) ) =E(xn+1,yn+1) ( 1 2σ2 µT1:S(I+ 1 σ2 Φ T 1:SΦ1:SΛ1:S) −1η1:Sη T 1:S(I+ 1 σ2 Λ1:SΦ T 1:SΦ1:S) −1µ1:S
1+ 1σ2 η T 1:S(I+ 1 σ2 Λ1:SΦ T 1:SΦ1:S) −1Λ1:Sη1:S ) ) =E(xn+1,yn+1) ( 1+o(1)
2σ2 µT1:S(I+
1
σ2 ΦT1:SΦ1:SΛ1:S) −1η1:Sη T 1:S(I+
1
σ2 Λ1:SΦ
T 1:SΦ1:S) −1µ1:S ) = 1+o(1)
2σ2 µT1:S(I+
1
σ2 ΦT1:SΦ1:SΛ1:S)
−1(I+ 1
σ2 Λ1:SΦ
T 1:SΦ1:S) −1µ1:S
= 1+o(1) 2σ2 ‖(I+ 1 σ2 Λ1:SΦ T 1:SΦ1:S) −1µ1:S‖22, (134)
where in the fourth to last equality we used the Sherman–Morrison formula, in the third inequality we used (118) , and in the last equality we used the fact that E(xn+1,yn+1)η1:SηT1:S=I .
Let µ̂1:R=(µ1,...,µR,0,...,0)∈RS . Then we have
‖(I+ 1 σ2 Λ1:SΦ T 1:SΦ1:S) −1µ1:S‖2≤‖(I+ 1 σ2 Λ1:SΦ T 1:SΦ1:S) −1µ̂1:R‖2+‖(I+ 1 σ2 Λ1:SΦ T 1:SΦ1:S) −1(µ1:S−µ̂1:R)‖2, ‖(I+ 1 σ2 Λ1:SΦ T 1:SΦ1:S) −1µ1:S‖2≥‖(I+ 1 σ2 Λ1:SΦ T 1:SΦ1:S) −1µ̂1:R‖2−‖(I+ 1 σ2 Λ1:SΦ T 1:SΦ1:S)
−1(µ1:S−µ̂1:R)‖2. (135)
ChooseR=n( 1 α+κ)(1−t) where 0<κ< α−1−2τ+(1+2τ)t2α2(1−t) . In Lemma 29, (62), we showed that with probability of at least 1−δ,
‖(σ2I+Λ1:RΦT1:RΦ1:R)−1µ1:R‖2 =Θ(n(1−t)max{−1, 1−2β 2α }logk/2n)
= 1+o(1) σ2 ‖(I+ n σ2 Λ1:R)
−1µ1:R‖2, (136)
where k= {
0, 2α 6=2β−1, 1, 2α=2β−1. . The same proof holds if we replace Φ1:R with Φ1:S , Λ1:R with Λ1:S ,
and µ1:R with µ̂1:R. We have
‖(σ2I+Λ1:SΦT1:SΦ1:S)−1µ̂1:R‖2 =Θ(n(1−t)max{−1, 1−2β 2α }logk/2n)
= 1+o(1) σ2 ‖(I+ n σ2 Λ1:S)
−1µ̂1:R‖2. (137)
Next we bound ‖(I + 1σ2 Λ1:SΦ T 1:SΦ1:S) −1(µ1:S − µ̂1:R)‖2. By Assumption 5, we have that ‖µ1:S−µ̂1:R‖2 =O(R 1−2β 2 ). For any ξ∈RS and ‖ξ‖2 =1, using the Woodbury matrix identity, with probability of at least 1−2δ we have
|ξT (I+ 1 σ2 Λ1:SΦ T 1:SΦ1:S) −1(µ1:S−µ̂1:R)|
= |ξT ( I− 1
σ2 Λ1:SΦ
T 1:S(I+
1
σ2 Φ1:SΛ1:SΦ
T 1:S) −1Φ1:S ) (µ1:S−µ̂1:R)|
= |ξT (µ1:S−µ̂1:R)− 1
σ2 ξTΛ1:SΦ T 1:S(I+
1
σ2 Φ1:SΛ1:SΦ
T 1:S) −1Φ1:S(µ1:S−µ̂1:R)|
≤‖ξ‖2‖µ1:S−µ̂1:R‖2+ 1
σ2 |ξTΛ1:SΦT1:S(I+
1
σ2 Φ1:SΛ1:SΦ
T 1:S) −1Φ1:S(µ1:S−µ̂1:R)|
≤O(R 1−2β 2 )+ 1 σ2 ‖(I+ 1 σ2 Φ1:SΛ1:SΦ T 1:S) −1Φ1:SΛ1:Sξ‖2‖Φ1:S(µ1:S−µ̂1:R)‖2
=O(R 1−2β 2 )+ 1
σ2 O(
√ ( 1
δ +1)n·n−(1−t))O(
√ ( 1
δ +1)nR
1−2β 2 )
=O(( 1
δ +1)R
1−2β 2 ),
where in the second to last step we used Corollary 20 to show ‖Φ1:S(µ1:S − µ̂1:R)‖2 = O( √
( 1δ +1)nR 1−2β 2 ) with probability of at least 1 − δ, and Lemma 40 to show that
‖(I + 1σ2 Φ1:SΛ1:SΦ T 1:S) −1Φ1:SΛ1:Sξ‖2 = O( √ ( 1δ +1)n · n −1) with probability of at least 1−δ. SinceR=n( 1α+κ)(1−t), we have
|ξT (I+ 1 σ2 Λ1:SΦ T 1:SΦ1:S) −1(µ1:S−µ̂1:R)|=O(( 1 δ +1)n (1−2β)(1−t) 2α + (1−2β)(1−t)κ 2 ).
Since ξ is arbitrary, we have ‖(I + 1σ2 Λ1:SΦ T 1:SΦ1:S) −1(µ1:S − µ̂1:R)‖2 = O(( 1δ + 1)n (1−2β)(1−t) 2α +
(1−2β)(1−t)κ 2 ). Since 0 ≤ q < [α−(1+2τ)(1−t)](2β−1)4α2 and 0 < κ < α−1−2τ+(1+2τ)t 2α2(1−t) ,
we can choose κ < α−1−2τ+(1+2τ)t2α2(1−t) and κ is arbitrarily close to κ < α−1−2τ+(1+2τ)t 2α2(1−t) such that
0≤q< (2β−1)(1−t)κ2 . Then we have (1−2β)(1−t)κ 2 +q<0. From (135) and (137), we have
‖(I+ 1 σ2 Λ1:SΦ T 1:SΦ1:S) −1µ1:S‖2 =Θ(nmax{−(1−t), (1−2β)(1−t) 2α }logk/2n)+O(( 1 δ +1)n (1−2β)(1−t) 2α + (1−2β)(1−t)κ 2 )
=Θ(nmax{−(1−t), (1−2β)(1−t) 2α }logk/2n)+O((nq+ (1−2β)(1−t) 2α + (1−2β)(1−t)κ 2 )
=Θ(nmax{−(1−t), (1−2β)(1−t) 2α }logk/2n)
=(1+o(1))‖(I+ n σ2 Λ1:S) −1µ̂1:R‖2 =(1+o(1))‖(I+ n σ2 ΛR) −1µR‖2.
(138) Hence G2,S(Dn) = 1+o(1) 2σ2 ‖(I + 1 σ2 Λ1:SΦ T 1:SΦ1:S) −1µ1:S‖22 = Θ(n(1−t)max{−2, 1−2β α } logk/2 n). Then by (133),G2(Dn)=Θ(nmax{−2(1−t), (1−2β)(1−t) α }logk/2n)+Õ ( ( 1δ +1)n 1 σ2S
max{1/2−β,1−α}). Choosing S=n ( 1+min{2, 2β−1 α } min{β−1/2,α−1}+1 ) (1−t) , we get the result.
Proof of Theorem 9. From Lemmas 39 and 41 and 1α −1>−2, we have that with probability of at least 1−7δ̃,
E G(Dn)= 1+o(1)
2σ2 (Tr(I+
n
σ2 ΛR)
−1ΛR−‖Λ1/2R (I+ n
σ2 ΛR)
−1‖2F +‖(I+ n
σ2 ΛR)
−1µR‖22)
=Θ(n (1−α)(1−t) α )+Θ(nmax{−2(1−t), (1−2β)(1−t) α }logk/2n)
=Θ(nmax{ (1−α)(1−t) α , (1−2β)(1−t) α }) (139)
where k= {
0, 2α 6=2β−1 1, 2α=2β−1 . Furthermore, we have
Tr(I+ n σ2 Λ)−1Λ−Tr(I+ n σ2 ΛR) −1ΛR
= ∞∑ p=R+1 λp 1+ nσ2λp ≤ ∞∑ p=R+1 Cλp −α 1+ nσ2Cλp −α ≤ ∞∑ p=R+1 Cλp −α= n σ2 O(R1−α)
=O(n(1−α)(1−t)( 1 α+κ))
=o(n (1−α)(1−t) α ).
Then we have
Tr(I+ n
σ2 ΛR)
−1ΛR=Tr(I+ n
σ2 Λ)−1Λ(1+o(1)). (140)
Similarly we can prove
‖Λ1/2R (I+ n
σ2 ΛR)
−1‖2F =‖Λ1/2(I+ n
σ2 Λ)−1‖2F (1+o(1)) (141)
‖(I+ n σ2 ΛR) −1µR‖22 =‖(I+ n σ2 Λ)−1µ‖22(1+o(1)) (142)
Letting δ=7δ̃, the proof is complete.
In the case of µ0>0, we have the following lemma:
Lemma 42. Let δ = n−q where 0 ≤ q < [α−(1+2τ)(1−t)](2β−1)4α2 . Under Assumptions 4, 5 and 6, assume that µ0 > 0. Then with probability of at least 1− 6δ over sample inputs (xi)ni=1, we have G2(Dn)= 1 2σ2µ 2 0+o(1).
Proof of Lemma 42. Let S = nD. Let G2,S(Dn) = E(xn+1,yn+1)(T2,S(Dn+1) − T2,S(Dn)). By Lemma 33, when S is large enough, with probability of at least 1−3δ we have that
|G2(Dn)−G2,S(Dn)|= ∣∣E(xn+1,yn+1)[T2(Dn+1)−T2,S(Dn+1)]−[T2(Dn)−T2,S(Dn)]∣∣
= ∣∣∣∣E(xn+1,yn+1)Õ((1δ+1)(n+1) 1σ2Smax{1/2−β,1−α} )∣∣∣∣
+ ∣∣∣∣Õ((1δ+1)n 1σ2Smax{1/2−β,1−α} )∣∣∣∣
=Õ ( ( 1
δ +1)n
1
σ2 Smax{1/2−β,1−α}
) .
(143)
Let ΛS = diag{λ1, ... , λS}, ΦS = (φ1(x), φ1(x), ... , φS(x)) and µS = (µ1, ... , µS). Define ηS = (φ0(xn+1),φ1(xn+1),...,φS(xn+1)) T and Φ̃S = (ΦTS ,ηS) T . By the same technique as in the proof of Lemma 34, we replace ΛR by Λ̃ ,R=diag{ ,λ1,...,λR}, let →0 and show the counterpart of the result (134) in the proof of Lemma 41:
G2,S(Dn)=E(xn+1,yn+1)(T2,S(Dn+1)−T2,S(Dn)) =E(xn+1,yn+1) ( 1 2σ2 µTS (I+ 1 σ2 Φ T SΦSΛS) −1ηSη T S (I+ 1 σ2 ΛSΦ T SΦS) −1µS
1+ 1σ2 η T S (I+ 1 σ2 ΛSΦ T SΦS) −1ΛSηS ) ) =E(xn+1,yn+1) ( 1+o(1)
2σ2 µTS (I+
1
σ2 ΦTSΦSΛS) −1ηSη T S (I+
1
σ2 ΛSΦ
T SΦS) −1µS ) = 1+o(1)
2σ2 µTS (I+
1
σ2 ΦTSΦSΛS)
−1(I+ 1
σ2 ΛSΦ
T SΦS) −1µS
= 1+o(1) 2σ2 ‖(I+ 1 σ2 ΛSΦ T SΦS) −1µS‖22, (144)
where in the fourth to last equality we used the Sherman–Morrison formula, in the third inequality we used (118) , and in the last equality we used the fact that E(xn+1,yn+1)η1:SηT1:S=I .
Let µ̂R=(µ0,µ1,...,µR,0,...,0)∈RS . Then we have
‖(I+ 1 σ2 ΛSΦ T SΦS) −1µS‖2≤‖(I+ 1 σ2 ΛSΦ T SΦS) −1µ̂R‖2+‖(I+ 1 σ2 ΛSΦ T SΦS) −1(µS−µ̂R)‖2, ‖(I+ 1 σ2 ΛSΦ T SΦS) −1µS‖2≥‖(I+ 1 σ2 ΛSΦ T SΦS) −1µ̂R‖2−‖(I+ 1 σ2 ΛSΦ T SΦS)
−1(µS−µ̂R)‖2. (145)
ChooseR=n( 1 α+κ)(1−t) where 0<κ< α−1−2τ+(1+2τ)tα2(1−t) . In Lemma 29, (62), we showed that with probability of at least 1−δ,
‖(σ2I+Λ1:RΦT1:RΦ1:R)−1µ1:R‖2 =Θ(n(1−t)max{−1, 1−2β 2α }logk/2n)
= 1+o(1) σ2 ‖(I+ n σ2 Λ1:R)
−1µ1:R‖2, (146)
where k= {
0, 2α 6=2β−1, 1, 2α=2β−1. . The same proof holds if we replace Φ1:R with Φ1:S , Λ1:R with Λ1:S ,
and µ1:R with µ̂1:R. We have
‖(σ2I+Λ1:SΦT1:SΦ1:S)−1µ̂1:R‖2 =Θ(n(1−t)max{−1, 1−2β 2α }logk/2n)
= 1+o(1) σ2 ‖(I+ n σ2 Λ1:S)
−1µ̂1:R‖2. (147)
So we have
‖(I+ 1 σ2 ΛSΦ T SΦS) −1µ̂R‖2 =µ0+Θ(n(1−t)max{−1, 1−2β 2α }logk/2n)
=µ0+o(1). (148)
Next we bound ‖(I + 1σ2 ΛSΦ T SΦS) −1(µS − µ̂R)‖2. By Assumption 5, we have that ‖µS − µ̂R‖2 = O(R 1−2β 2 ). For any ξ ∈ RS and ‖ξ‖2 = 1, using the Woodbury matrix identity, with probability of at least 1−2δ we have
|ξT (I+ 1 σ2 ΛSΦ T SΦS) −1(µS−µ̂R)|
= |ξT ( I− 1
σ2 ΛSΦ
T S (I+
1
σ2 ΦSΛSΦ
T S ) −1ΦS ) (µS−µ̂R)|
= |ξT (µS−µ̂R)− 1
σ2 ξTΛSΦ T S (I+
1
σ2 ΦSΛSΦ
T S ) −1ΦS(µS−µ̂R)|
≤‖ξ‖2‖µS−µ̂R‖2+ 1
σ2 |ξTΛSΦTS (I+
1
σ2 ΦSΛSΦ
T S ) −1ΦS(µS−µ̂R)|
≤O(R 1−2β 2 )+ 1 σ2 ‖(I+ 1 σ2 ΦSΛSΦ T S ) −1ΦSΛSξ‖2‖ΦS(µS−µ̂R)‖2
=O(R 1−2β 2 )+ 1
σ2 O(
√ ( 1
δ +1)n·n−(1−t))O(
√ ( 1
δ +1)nR
1−2β 2 )
=O(( 1
δ +1)R
1−2β 2 ),
where in the second to last step we used Corollary 20 to show‖ΦS(µS−µ̂R)‖2 =O( √ ( 1δ +1)nR 1−2β 2 ) with probability of at least 1− δ, and Lemma 40 to show that ‖(I + 1σ2 ΦSΛSΦ T S ) −1ΦSΛSξ‖2 =
O( √
( 1δ +1)n·n −(1−t)) with probability of at least 1−δ. SinceR=n( 1α+κ)(1−t), we have
|ξT (I+ 1 σ2 ΛSΦ T SΦS) −1(µS−µ̂R)|=O(( 1 δ +1)n (1−2β)(1−t) 2α + (1−2β)(1−t)κ 2 ).
Since ξ is arbitrary, we have ‖(I + 1σ2 ΛSΦ T SΦS) −1(µS − µ̂R)‖2 = O(( 1δ + 1)n (1−2β)(1−t) 2α +
(1−2β)(1−t)κ 2 ). Since 0 ≤ q < [α−(1+2τ)(1−t)](2β−1)4α2 and 0 < κ < α−1−2τ+(1+2τ)t 2α2(1−t) ,
we can choose κ < α−1−2τ+(1+2τ)t2α2(1−t) and κ is arbitrarily close to κ < α−1−2τ+(1+2τ)t 2α2(1−t) such that 0≤q< (2β−1)(1−t)κ2 . Then we have (1−2β)(1−t)κ 2 +q<0. From (145) and (148), we have
‖(I+ 1 σ2 ΛSΦ T SΦS) −1µS‖2 =µ0+Θ(n(1−t)max{−1, 1−2β 2α }logk/2n)+O(( 1 δ +1)n (1−2β)(1−t) 2α + (1−2β)(1−t)κ 2 )
=µ0+Θ(n (1−t)max{−1, 1−2β2α }logk/2n)
=µ0+o(1). (149)
Hence G2,S(Dn) = 1+o(1) 2σ2 ‖(I + 1 σ2 ΛSΦ T SΦS) −1µS‖22 = 12σ2µ 2 0 + o(1). Then by (143),
G2(Dn)= 1 2σ2µ 2 0+o(1)+Õ ( ( 1δ +1)nS max{1/2−β,1−α}). Choosing S=n( 1+min{2, 2β−1α }min{β−1/2,α−1}+1)(1−t), we get the result.
Proof of Theorem 11. According to Lemma 42, G2(Dn) = 12σ2µ 2 0 +o(1). By Lemma 39, we have G1(Dn)=Θ(n (1−α)(1−t) α ). Then E G(Dn)=G1(Dn)+G2(Dn)= 12σ2µ 2 0+o(1).
D.3 PROOFS RELATED TO THE EXCESS MEAN SQUARED GENERALIZATION ERROR
Proof of Theorem 12. For µ0 =0, we can show that
E M(Dn)=E Exn+1 [m̄(xn+1)−f(xn+1)]2
=E Exn+1 [Kxn+1x(Kn+σ2modelIn)−1y−f(xn+1)]2 =E Exn+1 [ηTΛΦT [ΦΛΦT +σ2modelIn)−1(Φµ+ )−ηTµ]2 =E Exn+1 [ηTΛΦT (ΦΛΦT +σ2modelIn)−1 ]2
+Exn+1 [ ηT ( ΛΦT (ΦΛΦT +σ2modelIn) −1Φ−I ) µ ]2
=σ2trueTrΛΦ T (ΦΛΦT +σ2modelIn) −2ΦΛ +µT ( I+ 1
σ2model ΦTΦΛ
)−1( I+ 1
σ2model ΛΦTΦ
)−1 µ
= σ2true σ2model Tr(I+ ΛΦ TΦ σ2model )−1Λ−Tr(I+ ΛΦ TΦ σ2model )−2Λ+‖(I+ 1 σ2model ΛΦTΦ)−1µ‖22.
According to (138) from the proof of Lemma 41, the truncation procedure (133) and (142), with probability of at least 1−δ we have
‖(I+ 1 σ2model ΛΦTΦ)−1µ‖22 =Θ(n max{−2(1−t),
(1−2β)(1−t) α }logk/2n)=(1+o(1))‖(I+ n
σ2model Λ)−1µ‖22,
where k= {
0, 2α 6=2β−1, 1, 2α=2β−1. .
According to (121) and (126) from the proof of Lemma 39, the truncation procedure (115), (140) and (141), with probability of at least 1−δ we have
Tr(I+ ΛΦ TΦ σ2model )−1Λ−Tr(I+ ΛΦ TΦ σ2model )−2Λ = (
Tr(I+ n σ2model
Λ)−1Λ )
(1+o(1))−‖Λ1/2(I+ n σ2model Λ)−1‖2F (1+o(1))
=Θ(n (1−α)(1−t) α ).
Combining the above two equations we get E M(Dn)=(1+o(1)) ( σ2true σ2model ( Tr(I+ n σ2model Λ)−1Λ−‖Λ1/2(I+ n σ2model Λ)−1‖2F ) +‖(I+ n σ2model Λ)−1µ‖22 ) =
σ2true σ2model
Θ(n (1−α)(1−t) α )+Θ(nmax{−2(1−t), (1−2β)(1−t) α }logk/2n)
=σ2trueΘ(n 1−α−t α )+Θ(nmax{−2(1−t), (1−2β)(1−t) α }logk/2n)
=Θ ( max{σ2truen 1−α−t α ,n (1−2β)(1−t) α } ) When µ0>0, according to (149) in the proof of Lemma 42 and the truncation procedure (133), with probability of at least 1−δ we have
E M(Dn)=Θ(n (1−α)(1−t) α )+µ20+o(1)
=µ20+o(1).

Review summaries:
Working under the nonparametric regression model, the authors established asymptotic rates for the Bayesian generalization error, normalized stochastic complexity and excess mean square error. The authors adopted the Bayesian approach by endowing the unknown regression function with a Gaussian process prior, and the covariance function is parameterized by a kernel function. By diagonalizing this kernel, the authors used the resulting eigenfunctions to represent the target function. They assumed that the kernel eigenvalues and the target basis coefficients decay according to the power-law, i.e., polynomial decay. Lastly, a simulation experiment was performed to verify the claims.

This paper considers the problem of fitting a target function corrupted by additive iid Gaussian noise using with a zero mean Gaussian Process (GP).   It provides an asymptotic characterisation of the typical Bayesian and mean-squared generalisation errors under the assumption of power-law decay of the covariance eigenvalues and the target function coefficients when expressed in the covariance eigenbasis.    The main result is to derive the exponents characterising the rate of decay of the aforementioned errors with the number of samples $n$, as a function of the spectrum and target function decays. The key technical step is to prove the concentration of $\Phi_{R}^{\top}\Phi_{R}\in\mathbb{R}^{R\times R}$ around $n I$ where $\Phi_{R}\in\mathbb{R}^{n\times R}$ the truncated matrix of eigenvectors evaluated at the data points, and $R=n^{\frac{2}{1+\alpha}-\kappa}$ with $0\leq \kappa < \frac{\alpha-1}{\alpha(\alpha+1)}$.  Finally, the authors provide one numerical experiment with the arc-cosine kernel and data uniformly distributed in the unit circle.

The paper consider asymptotic properties of Gaussian process models where the eigenvalues from Mercer's theorem exhibit polynomial decay. Unlike the earlier analysis of Sollich, the authors do not assume that the model is correctly specified. The authors provide a short experiment to illustrate the theory.