A probabilistic Taylor expansion with Gaussian processes
Authors: Toni Karvonen
Jon Cockayne
Filip Tronarp
Simo Särkkä
Venue: Accepted by TMLR ()

Abstract: We study a class of Gaussian processes for which the posterior mean, for a particular choice of data, replicates a truncated Taylor expansion of any order. The data consist of derivative evaluations at the expansion point and the prior covariance kernel belongs to the class of Taylor kernels, which can be written in a certain power series form. We discuss and prove some results on maximum likelihood estimation of parameters of Taylor kernels. The proposed framework is a special case of Gaussian process regression based on data that is orthogonal in the reproducing kernel Hilbert space of the covariance kernel.

Published in Transactions on Machine Learning Research (8/2023)
A probabilistic Taylor expansion with Gaussian processes
Toni Karvonen
ton.karvonnenShccssnnn.k.
Deportment of Mothematics and Statistics University of Helsinki
Jon Cockayne
jon. jon.cockaymeessooon.cc.kk
Department Mathematicnl Sciences University of Southampton.
Filip Tronarp
Centre for Mathematioal Sciences
Lund University
Simo Särkkä
simo.sarkkaaualke
Department of Electrical Engineering and Aulomation
Aalto University
Reviewed on OpenReview: https: // openreview. net/
bet/forunt forum? id= 2TheniEIDB
Abstract
We study class of Gaussian processes for which the posterior mean, for particular choice of data, replicates truncated Taylor expansion of any order. The data consist of derivative evaluations at the expansion point and the prior covariance kernel belongs to the class of Taylor kernels, which can be written in a certain power series form. We discuss and prove some results on maximum likelihood estimation of parameters of Taylor kernels. The proposed framework a special case of Gaussian process regression based on data that is orthogonal in the reproducing kernel Hilbert space of the covariance kernel.
1 Introduction
Taylor's thcorem is among the most fundamental results in analysis. one dimension, Taylor's thcorem states that any function f: R + that sufficiently smooth at a E R can be written as
f(x) = Tn,a (x) + Pm)a(x),
(1.1)
where (x) Ep=o pl fa) a) is the nth order Taylor polynomial and (z.a(z) remainder term which has the property that (x) O( a 1 as x - al + 0. Multidimensional generalisations are readily available and will be introduced Section Approximations derived from (1.1), in particular the first and second order Taylor approximations
1 f(x) ~ f(a) (z(a)(== a) and f(x) f(a) f'))(x a) f" aa².
play an important role in numerical algorithms for mumber of reasons. Firstly, Taylor approximations provide straightforward and principled means of linearising function of interest, which can often dramatically accelerate otherwise costly computations. Secondly, they require only information about a function and its derivatives at a single point; information that particular algorithms may already be collecting. Particular applications of Taylor's thcorem in numerical algorithms include optimisation (Moré, 1978; Conn al., 2000), state estimation (Särkkä, 2013, Ch. 5), ordinary differential equations (Hairer al., 1993, Ch. II), and approximation of exponential integrals in Bayesian statistics (Raudenbush et al., 2000), to name but few.

---

Published in Transactions on Machine Learning Research (8/2023)
A crucial challenge when applying Taylor series in this way, however, is their locality. The approximation is valid only near a, and apart from trivial examples approximation quality decays rapidly away from this point. When a numerical algorithm attempts to use a Taylor approximation to explore function behaviour around a particularly novel point, far from a, the behaviour of the algorithm can be difficult to predict and control.
This paper proposes a remedy for this by introducing a Gaussian process (GP) model (Rasmussen and Williams, 2006) whose posterior mean, given the derivative data f'(),, (a)), is exactly the Taylor polynomial Ta.a: and whose posterior variance plays role analogous to the remainder term Pa.a- In the spirit of probabilistic numerics (Diaconis, 1988; Cockayne et al., 2019b; Hennig et al., 2022), the posterior variance can then be used for principled probabilistic quantification of epistemic uncertainty in the Taylor approximation (x) ~ (x) at I & a, which can be exploited and propagated forward. In effect, the variance may be used to encode into algorithms a degree of "scepticism" about the validity of the Taylor approximation away from a. Taylor approximation thus joins the ranks of classical numerical methods, such as algorithms for spline interpolation (Diaconis, 1988; Kimeldorf and Wahba, 1970), numerical quadrature (Diaconis, 1988; Karvonen and Särkkä, 2017; Karvonen et al., 2018), differential equations (Schober et al., 2014; 2019; Teymur et al., 2016), and linear algebra (Cockayne et al., 2019a; Hennig, 2015), that can be cast as statistical inference. Even though the use of derivative information in Gaussian process modelling is rather standard and the prior that we use is relatively well known, we are unaware of any prior attempts at deriving the Taylor approximation in a Gaussian process framework.
1.1 Related Literature
The Gaussian process priors we use to construct a probabilistic Taylor expansion are determined by positive- definite and non-stationary Taylor kernels which, at an expansion point a, take the form
Cp2" (pl)z (ry))
Ka(z,y) = K(x a, y a)), where K(æ,v)= = 0²
(1.2)
P=0
for non non-negative constants Cp and positive parameters and A. For multidimensional input points, the index p E No is replaced with a multi-index a E Ný and ry usually with the Euclidean inner product; see Section 2.1 for details. The canonical example is the eaponential kernel
K(x,y) = 0²
(xy) = d² exp(AAy))
(1.3)
p=0 p!
which is obtained by setting Cp = p! in (1.2). The exponential kernel is closely connected to the popular
Gaussian kernel K(x, v) = 0² 22cxp( cxp(->2(z - - /)2/
Taylor kernels, often under the name power series kernels, have been used and their approximation propert analysed- in the numerical analysis and scattered data approximation literature; see Dick (2006); Zwicknagl (2009); De Marchi and Schaback (2010); Zwicknagl and Schaback (2013) and Fasshancr and McCourt (2015, Sec. 3.3.1). Section 1 in Zwicknagl and Schaback (2013) has been of particular inspiration for the present work. The Szegö kernel and the Beryman kernel K(x, v) 1/(1 myy and K K(x, v) = 1/(1 - =zy)², respectively, are particularly well studied in the approximation theory literature because their reproducing kernel Hilbert spaces (RKHSs) are important in complex analysis; see, for example, Larkin (1970); Richter-Dyn (1971b;a) and Oettershagen (2017, Sec. 6.2). These kernels are defined on the open interval (-1,1) and obtained from (1.2) by sctting (Szcgö) Gp = (pl)² and (Bergman) Czp = (p!)² and C2p+1 = for p E No in (1.2).
Taylor kernels occasionally appear in the machine learning and statistics literature but, to the best of our knowledge, have not been used in conjunction with derivative data in the way proposed here. We refer to Minka (2000, Sec. 4); Steinwart and Christmann (2008, Example 4.9) and (Liang and Rakhlin, 2020) for a few of their appearances. Gaussian process regression based on derivative evaluations has been explored (e-g., Solak et al., 2002; Prüher and Särkkä, 2016; Wu et al., 2017; Eriksson et al., 2018), though typically for "standard" kernels such as the Gaussian kernel. The approach in this paper differs from the prior Gaussian process literature in two key ways, which enable a probabilistic replication of the Taylor expansion: First, for kernels used in the literature the posterior mean cannot coincide with the Taylor polynomial. Secondly, in
2

---

Published in Transactions on Machine Learning Research (8/2023)
the literature the data typically consist of function and derivative evaluations number of different points, whereas we are specifically interested in derivatives at single point.
1.2 Contributions
The main contributions of the paper are contained in Sections 2 and 3. In Section 2, we derive probabilistic Taylor expansion and a basic error bound; these results are given in Theorems 2.1 and 2.3. We also discuss how inclusion of observation noise affects probabilistic Taylor expansions. In Section 3, we derive expressions for maximum likelihood estimates of the Taylor kernel parameters a and A. Perhaps the most interesting result that we obtaín is Theorem 3.2, which states that derivative data that could have been generated by constant function yield the estimate AML O. As mentioned above, the exponential kernel is related to the Gaussian kernel. In Section 4, we show how to derive closed form expression for the posterior mean and covariance given derivative data when the covariance kernel is Gaussian. Section 5 outlines generalisations of probabilistic Taylor expansions derived in Scction 2 for data that are orthogonal in the reproducing kernel Hilbert space of the covariance kernel. Fourier coefficients constitule an example of orthogonal data when the kernel periodic. Some simple numerical toy examples are included Section
We emphasisc that the purpose this paper is not propose new Gaussian process algorithms but rather to provide Gaussian process interpretation for classical and well-known Taylor approximations. Such competitive algorithms.
interpretation is intrinsically interesting even if it yields
1.3 Notation
We use No to denote the set of non-negative integers and Ng to denote the collection of non-negative d-dimensional multi-indices (a((),.... a(d)), where a(j) E No is the jth index of a. We also use the standard notation |a| a(1) + ......... ((()) and ad = a(1)! x ................................ a(d)!.
2
A Probabilistic Taylor Expansion
In this section, we derive a probabilistic Taylor expansion using Gaussian processes. We discuss a generalisation of this derivation for orthogonal data in Section
2.1 Taylor Kernels
Let a E Rd and r E c0, Define Qa,r = fxr E IRd || - all2 < <r}. A multidimensional Taylor kernel on Pa.r X Pay is definod as
Cada (ae)z
Ka (x, y) = K(x a, y a) for K(x,y) v) 0² Σ
y
(2.1)
aEN
where a 0 and À Ry are kernel hyperparameters. The coefficients Ca are non-negative constants such that Co 0 for infinitely many No and
Aa (a!)
CaAa alva! ad!
< i
Σ aEN
clal <000 if ==
(2.2)
GEN
The conditions (2.2) are sufficient ensure the series defining Ka via (2.1) converges absolutely for all æ, y Na.r, which, together with ca for infinitely many a, guarantees that Taylor kernels are positive- definite (Zwicknagl and Schaback, 2013, Thm. 2.2). If only finititely many Ca positive, the kernel is posilive-semiddefiilee However, 2 0 is not necessary for posiliie-deffnniennss of K in (2.1) (see Zwicknagl, 2009, Sec. 2). To ensure that the diagonal covariance matrix in (2.11) is invertible we always assume that
Ca > 0 for every E No.
Note that a and could be subsumed into the coefficients ca. However, as we shall see in Scction 3, the parametrisation that use leads to convenient. and useful hyperparameter estimation. Specifically, maximum
3

---

Published in Transactions on Machine Learning Research (8/2023)
likelihood estimation of the parameters a and X is possible and the estimators have some intuitive properties. In contrast, it is either useless impossible to estimate the coefficients Ca, which should therefore be fixed. An important subclass of Taylor kernels are inner product kernels, defined by
20
(ax y) = o²
Σ p=0
a
y/a.
where (a, = Σ A,T,Ali-
(2.3)
(p!)²
It is easy to show that inner product kernels are Taylor kernels: From the multinomial theorem we have
K(x,y) = 0²
20
Cp (p!)
(x,
v)
2
Cp (pl)²
Aen.yi )
0²
sc
Cp
ya
(p!)²
a
p=0
p=
p=0
al=p
0²
Σ
ya
a!! |a||
aENS
lail! We will discuss estimation of the parameters and À (as well the coefficients cp) in Section for now, we assume the parameters are given and proceed to show how Taylor kernels may be used to derive probabilistic Taylor expansion.
which we recognise as Taylor kernel in (2.1) with ca - 9aj /a///////!
The multidimensional version of the exponential kernel in (1.3) is
K(x, y) o² exp((xx 0² v)
(2.4)
The exponential kernel is defined on Pa.r = Rd. In Section 4, we discuss a close connection that the exponential kernel has to the commonly used Gaussian kernel. By setting cp = 1 we obtain the Bessel kernel ((x, y) d² - (zz, / (2x, y) 1/2), where Io is the modified Bessel function of the first kind, which is another Taylor kernel defined on the whole of Rd.
2.2 Gaussian Process Regression with Derivative Data
A Gaussian process fGP ~ GP R) characterised by mean function m: Pa.r + and covariance kernel R: Da.r Pa.r + RR is a stochastic process such that for any points of (fap(21),.... fGP (x))) and covariance matrix R (R(æ,, (1)) E NNN (Rasmussen and Williams, 2006). In particular, [[[] = m(x) and Cov(S(z), = R(x, y) for all æ, E Parr- Let f: Pa,r + R be an n times differentiable function on Pa meaning that the partial derivatives
ZN Saar the joint distribution N-dimensional Gaussian with mean vector (m(x1),.... m=,x))
R
glalf
DOS=
0x
Ox a(d)
exist for all E Ng such that |a]] S n. Suppose also that the prior mean m is times different and that R is times differentiable in both arguments, in the sense that the derivative
|la|+101
Owo R(v, w)
exists for all x, E Pa,r and all multi-indices and B such that |a|,10| S The noiseless derivative data are
fa = ((a)) ((a) N f(a)).
(2.5)
where we use an arbitrary ordering of the set aN! = (a E Ng : |al S n), which contains
n2 1n
(n (n+c) d)!
Nd
n! d!
4

---

Published in Transactions on Machine Learning Research (8/2023)
elements. When conditioned on these derivative data, the posterior faP fa is Gaussian process (Särkkä, 2011; Travelletti and Ginsbourger, 2022). That is, fGP fa ~ GP(sn,a, Pa.a) with mean and covariance
(1)a(1) m(x) + (a)) R. (fa R ma) and (a, y) R( - ra( Ruyr)
(2.6)
Here Ra E NExd and ra(x)) E R. are each given by (Ra)sj = D R(x R( æ=a and (ra(z)), = D R(xr, y) y) y=a
(2.7)
where subscripts denote the differentiation variable, and ma (Da ..... Nf v m(a)). When f has multidimensional range, one may model each of its components independently, though we note that this modelling choice may be readily generalised using vector- valtu Gaussian processes (Alvarez et al., 2012).
2.3 Replicating the Taylor Expansion Using Taylor Kernels
The next theorem combines what was described in Sections 2.1 and 2.2 to give a probabilistic version of the Taylor expansion.
Theorem 2.1. Let Ka be Taylor kernel defined as in (2.1). Let SCP ~ GP(m, Ka) and fa = (D* -DD)(aa))a Then fGP fa ~ GP(snn,, Pn.a), where
D*[f(a) - -m(a)
(x a) and (n, y) = 0² Σ Cada
(n,a(x) = +m(x)++++++++ + Σ
(a!) (x) (y-
a)" (2.8)
|a|sn ad!
|a|>n
If is polynomial of degree at most n, then
Do Das(a) aa) a)a
8n,a (x)
(n:
(2.9)
ad!
which is identical to the multidimensional version of the Taylor polynomial in (1.1).
Proof. It straightforward to compute that, for any B,y E No.
Ca Ca - ( aa)- a)a-
D D Ka y) o²
(2.10)
(a- 3))((a - -7)!
a2BAy
where y (ma)( (()),........ (ax) If or y a, all terms with a B + 0
a 7 0, respectively, in (2.10) vanish. Therefore in the context of (2.7) have
(Ra); = Aa and (ra(w)); = و² Sa aa:
(Ø a)
(2.11)
a.!
Consequently, the matrix Ra is diagonal and the ith element the row vector Ta (a in (2.6) is (ag!)- a)a, It follows that the posterior mean and covariance are as in (2.8). From (2.1) we recognise the covariance Pn.a as the remainder the kernel expansion. To prove (2.9) is sufficient to obscrve that m(x) Ejaign D
a when m is polynomial of degree at most n. By inspection is clear that Sn,a in (2.9) is identical to the Taylor expansion given in (1.1) (and its multivariate version), which completes the proof.
Note that the covariance is not identically zero-in fact, P x)) - oc as ||== while (x, y) takes the form of an infinite sum, provided Ka has a closed form it can be computed by subtracting the terms with [x] S n in the summation form of Ka from that closed form. For illustration, some posterior processes are displayed in Figure 1.
| - co. Furthermore,
Whether or not the explosion of the posterior variance away from is desirable depends on what one is trying to achieve and what kind of prior information is available. If one is trying to extrapolate, it seems
5

---

Published in Transactions on Machine Learning Research (8/2023)
entirely natural to us, at least in the absence of additional knowledge about f, that the variance should be very large far away from a. But if there is additional prior information that the function f has, for example, approximately the same magnitude everywhere on its domain, then it may make sense to use a stationary kernel for which the variance tends to a constant value as the distance to the nearest data point increases.
The expressions (2.8) for the posterior mean and covariance show that computational complexity of inference with Taylor kernels and derivative data is linear in the number of data points, N if the derivatives of m are cheap to compute (e.g., if m is a polynomial). A generic kernel for which no special structure is present in the covariance matrix Ra incurs cubic computational cost because linear system of equations needs to solved when the mean and variance are computed directly from (2.6). This seeming advantage of Taylor kernels is lost if the data do not consist of derivatives at & single point.
Remark 2.2. Recall that in Section 2.1 assumed that Co 0 for every E Ng. However, from (2.11) we easily see that Theorem 2.1 remains valid as long as ca > 0 for all |0|| S n because this ensures that the diagonal covariance matrix Ra is invertible. Section 2.1 therefore applics also to polynomial kernels, which are Taylor kernels with finitely many non-zero coefficients co if n remains sulficiently small.
2.4 Error Bounds
Each positive-semidefintte kernel R on Pa.r Ma.r associated to a unique reproducing kernel Hilbert space (RKHS), H(R), equipped with inner product ((() and norm [|-lbx())- The RKHS a Hilbort space of functions f: Pa.r + IR such that R(,,)) E H(R) for every a E Pa, and in which the kernel R has the reproducing property
(wx) R(, ()) for all J' H(R) and a E Pa.r-
See Berlinet and Ahooaa--Agaan (2004) for more information on RKHSs. It often difficult to characterise the functions which lie in the RKHS. Fortunately, for Taylor kernels one may use results such Theorem in Minh (2010) to show that
Va H(Ka) [((Ka)= -(1(o)= Σ fa v
ad ax [1 = Σ Sa < o0 o
NENg
aE
See also Zwicknagl and Schaback (2013) and Paulsen and Raghupathi (2016, Sec. 2.1). For example, all polynomials are contained in the RKHS of any Taylor kernel for any Rd
The next theorem shows that the posterior variance has similar interpretation to the Taylor remainder term if f is in H(Ka).
Theorem 2.3. Let faP fa be in Theorem 2.1, and let the assumptions of that theorem hold. If E H(Ka), then 8n.a and Pr.a salisfy
((x)) (x)) S ( xx) < On, a [l || all2 1+1
(2.12)
for all x E Dar, where oo is a positive sequence such that Cn,r - 0 as n n oc.
Proof. By the standard equivalence between Gaussian process regression in the noiseless setting and worst-case optimal approximation (e-g., Scheuerer et al., 2013; Kanagawa al., 2018) the posterior mean 8n,a H(K.) is the minimum-norm approximant of ff such that D ) DO f(a) for every |o| < n and the posterior standard deviation at (x, 12)1/2 equals the worst-case approximation error at x in the RKHS. Hence
If(x) ( < || sa *all (x. 1/2 < [[]l)(()))) (x
for all x E Da.r if f E H(Ka) (egg Wendland, 2005, Thm. 16.3). To prove the upper bound for the posterior variance observe that from the general inequality | S ||>|| ||ا|||||| || |a] for Rd and Ng it follows
6

---

Published in Transactions on Machine Learning Research (8/2023)
4
5
Exponential
Gaussaa
-1
-1
-1.0
0.5
0.0
0.5
1.0
-1.0 -0.5
0.0
0.5
1.0
Figure 1: Gaussian process posterior means and 95% credible intervals given derivative data for f(() = sin (in
at 0. The priors have m = and use either (blue) the exponential kernel (xx, y) r² exp(Ary) with A 3/2 or (orange) the Gaussian kernel K K(x, y) = o² exp(->²(x /22. Maximum likelihood estimation used to select the scaling parameter We shall revisit this example in Section 6.
that, for any x E Qa,r = fx= E Rd
|| - all2 <r},
aada aa
Pu.a (x,
Σ
(x: 2a
(a!)2
|a/
cada
Cada
0²
(a!)3 (x -a)aa
Σ
(x
a)2a
(a!)²
|a/=n+1
|a/sn
cada
cada
< 0
(a!)²
|| - al12
2(a+1)
+
(a!)
||00 -al2 2/ax
|a|=n+1 1
|alsn+1
cada
cada
o²
1
((x+1))
|| - aa122 2lal- 2
(a!)
(al)2
|a|=n+1
|a/2n#1
cada
Cada y22
S 0²² x all2(xt1 ((++1)
Σ
+r + 2(++1)
(al)2
(a!)2
|a|=nt1
=On,
The summability assumption (2.2) ensures that Cn, is finite and that Ca,r - 0 as n - oc.
The bound (2.12) is valid for every element of H(Ka). However, the bound uncomputable because it is not possible to compute the norm []])))))))))) from the derivative data Sa without some additional information about the function f. For example, when d = 1 and 0, the functions
= = 1 2x 3x² + 4x + 5x and f2(x) = 1 2x 3x + 4x + cr?
are different if c * 0 but provide the same derivative data whenever 3. In practice, what one has to do is estimate the parameters of Ka in a data-dependent way, use the standard deviation (z, x)) to compute, say, the 95% credible interval around the point estimate (x))(;) of f(xx), and conclude that it is likely that f(x) falls within the resulting credible interval. Any such uncertainty estimates are bound fail occasionally- and the severity of the failure can be arbitrary. For example, when n 3, credible intervals formed from derivative data generated the function fa above do depend on even though fa(x)) 83,0(2)) cx does.

---

Published in Transactions on Machine Learning Research (8/2023)
2.5 Noisy Observations
Suppose that the observations are corrupled by Gaussian noise. That is, the data vector is fa = (Ya ( where Vo = D f(a + Fa with independent 2 =(0,=2) for Ea > 0. While unrealistic in practice, the assumption that the noise terms are independent allows for explicit computation of the posterior mean and variance. In this setting the posterior mean and covariance for a general sufficiently differentiable prior mean m and covariance kernel are
- mma) and (a, y)= R(x, ().x,y)-10 rr) E) (2.13) a (x) = m(x) +ra(z)) [R (R (e.
where E is diagonal Ná Nº matrix containing the noise variances eaa and ra (2), Ra, and ma were defined in Section 2.2. Recall then from Section 2.3 that when R Taylor kernel Ka have
(Ra)i = ca.11:.0 6 and (ra(z))) 0² Sa, ta;
(x -a)
ag!
Therefore (Ra + E)-1 (o2 aa: = =22)) - and plugging this into (2.13) yields
la Dam(()
(z m(x) +
allo²cade a + =2) (a a)a
l
and
c 20
Pn,a(x,)) K2 - o²
Σ
C Aa a ay a
a)a
(a!)² (a2
aalsn
Cada (al)2
02 x 20
2
(x (y a)
Σ
(x ax- a)a
(a!) +
a In
c2
Cada
(a a)
a) +
(a!)² (x - 23 - a)a
(o²) a + +
|a/sn
Note that by setting Ea = 0 for every No such that |a| S n we recover the noiseless posterior mean and covariance in (2.8).
3 Parameter Estimation
Observe from (2.8) that, although they do not affect the posterior mean, proper selection of the Taylor kernel parameters À and a is a prerequisite for useful and meaningful uncertainty quantification via the posterior variance Pr.a (x, æ). In this section we consider maximum likelihood estimation of these parameters. For the Gaussian process model in Scction 2.2, the negative log-likelihood function that is be minimisod with respect to generic vector of kernel hyperparameters 0 is
1 (0) = (Ja m.) "RR " ma) + log det R. + Nd
log(2r).
2
By discarding terms and coefficients that do not depend on and using (2.11) the maximum likelihood estimate (MLE) OML is any minimiser of the function
that for Taylor kernel
Σ (D*lf(a) -
m(a))) (P(f(a))((a){?
[(0)
= G2 La
ca
+ Nd log 0² + Σ log A
Σ ca.
(3.1)
JalSn
In principle, every coefficient ca of a Taylor kernel in (2.1) may be considered a free parameter to be estimated. However, maximum likelihood estimation of these coefficient is either useless or impossible: From (2.8) we
see that the posterior process depends on Co only for |a] > n. However the objective function (3.1) does not depend on these ca. making it impossible to estimate those parameters that actually influence posterior uncertainty. We encountered simple example of this phenomenon in Section 2.4.
8

---

Published in Transactions on Machine Learning Research (8/2023)
3.1
Estimation of
From (3.1) it is easy to calculate OML, the maximum likelihood estimate of o, for any fixed A E IR4 and No by differentiating and setting its derivative zero. This gives
(Do m(a)))²
ML
Σ S
(3.2)
N
CaA
3.2 Estimation of X
Estimation of X for fixed is more complicated and the maximum likelihood estimate docs not appear to admit closed form expression akin to that for OML in (3.2). However, something interesting can be said. We write À = (AI.....Aa).
E d. Then lima;- ((A) -o if ((((())) -m(a]] = 0 for every [x| S n such that adi)) > and lim,, E(A) = oo otherwise.
Lemma 3.1. Suppose that n 2 and let sis
Proof. Assume first that (a{ m m(a)] for every |ol| S such that ax(i) > Tt. follows that the first term in (3.1) does not depend on A₁ Because Ejalsn log Xa + -00 as A + 0, we have {(A) + -00
A 0. Assume then that D& [f(a) - -m(a)] + 0 for some |8| S n such that B(i) > 0. Therefore
(D*I((a) mm(a))²
1 (D*If(a) m(a)))²
Σ |a St
log a
>
Σ |a|Sn
log A
cada
o²
cBA0
O Sn
log aa
aa/S
for certain positive constant and X S Since 1/2 + a log for any a > as + from the right, conclude from the above lower bound that {(A) Do as X; - 0. This concludes the proof.
Lemma 3.1 states that ((A) attains minimum X; 0 when the data consistent with m being equal to f up to constant along dimension i. From Lemma 3.1 and the fact that ((A) can tend to negative infinity only if component of A tends to zero we obtain the following theorem, which is essentially special case of Theorem 5.2 in Karvonen and Oates (2023). See also Proposition 4.3 in Ben Salem et al. (2019).
Theorem 3.2. Suppose that n 2 and let i d. Fir AS for j j Then
AL,ML arg min É((A)...... Xa)) =0 2.20
is and only if ()(aa)) (ap((aa-m(aa))) mm(a]] for every |a| S such that ax > 0. In particular, if d 1, then
AML = arg min ((A) 0
A
if and only if (() - mm) (a) = 0 for every 1 s
For simplicity, let d = 1. If X = 0, we see from (2.8) that Pn,l(,,)) = 0 for all a, E IR. Moreover, if f(0) (a) m!P) (a) = 0 for every S p < n, then
n
f(0) (a) (m() (a) (x (x- a) = f(a) ({))
Sn,a (x)
p!
0=0
for every x E R. That is, Sn,a is constant function. The interpretation of Theorem 3.2 is thus that when the data look like they could have been generated by the function f(x) = m(x)) + e for some E IR (i.e., by a constant shift of the prior), maximum likelihood estimation returns AML = because this value of X both explains the data and yield the simplest model, one of zero variance. When the postcrior covariance is identically zero, the resulting degenerate posterior fop fa ~ GP(f(a) - m(a), 0) does not provide useful uncertainty quantification it is unreasonable to expect perfect predictions from finite set of data.
9

---

Published in Transactions on Machine Learning Research (8/2023)
3.3 On Simultaneous Estimation of a and X
The purpose of this section is to remonst.ate that simultaneous maximum likelihood estimation of and X is likely to cause problems. We consider inner product kernels of the form (2.3) with coefficients and n = 1. Let 0f f()) denote the pth order partial derivative of f at æ with respect to the ith coordinate.
C/a al/ |a|!
Note from Ca = ///a///// |a|! that ca = co for a = and ca = cı when |a|== =. By differentiating (3.1) with respect to the ith component of À we see that to obtain AML
we need to solve
(())(D"ff(a) m(a))²
a (d.(f(a) m(a)))² 1 -0
(3.3)
Aa
d²ciA;
Sa
for each l,...,d. d. Equation (3.3) readily gives the maximum likelihood estimates
AL (ai(((a) m(a)))
(3.4)
0²c1
for fixed o O. Inserting these to the expression for the maximum likelihood estimate ML in (3.2) yields
(f((a) m(a)} 2
A1 _ ( (0.[(a) m(aa))²)
(f(()) ( mm(a)² + +do
Ml d d+1
co
d +
co
which is solved by oML = (f(a) - m(a)]²/ /co. By plugging this in (3.4) we obtain the final estimates
[f(a] - mm and A,ML cu (a(f(a) - 2 f(a) m(a)
ML
(3.5)
Co
It is clear what the problem is: If [f()) - m(a) is small relative to [a[[(a) mma)] for some i (i.e., m R ʃ but 0m & 0, at a), the estimate for À becomes large, which may cause numerical problems and yields a large posterior variance. For example, let d = 1 and insert the estimates in (3.5) to the posterior variance in (2.8). This gives
P-1
(["(a)-
[F'(a) - m m2
P Cp (pl)2
(n,
Σ p=2
(x -a)
[f(a] m(a)}²
C1
In practice is therefore safest to fix one of the parameters and estimate the other. In the examples in Scction 6 we fix À and estimate a.
4 Comparison to the Gaussian Kernel
It is common condition a Gaussian process defined by the Gaussian kernel on evaluations of a function and its derivative number of different points (Solak et al., 2002; Prüher and Särkkä, 2016; Wu et al., 2017). However, convenient expressions for the posterior mean and variance are available in this setting. The purpose of this scction is to exploit an expression from Xu and Stein (2017) and derive explicit expressions for the mean and variance when in the setting of Section 2.2 (i.e., when the data consist of derivative evaluations at a single point). Because the Gaussian kernel is not a Taylor kernel, the posterior mean and variance, although available in closed form, are more complicated than the ones for Taylor kernels in (2.8).
Let I1-11x x- It is well known that the ubiquitous Gaussian kernel
R(x, y) exp (- -
1 | y
is closely connected to the exponential kernel in (2.4) via the equation
R(x, y) = exp (- 2
1 (2r, 2)> - -2(x, y) + (x., (o,,))2)] = exp (- 2/2 ||x:1/2 eep(((r, v)a) exp
-
- 11/113
10

---

Published in Transactions on Machine Learning Research (8/2023)
Given such relationship to Taylor kernel it should come surprise that, given derivative data, the posterior mean and covariance in (2.6) are available in closed form for the Gaussian kernel even though the matrix Rg is not diagonal.
Let us consider the univariate case. We get
Xa (-1)) (- 1-22
20
A2p
g=a (-1) exp
=
Σ
11)
Di+s
2p
2pp!
2=0
1==0
p=0
When i + j is odd, all derivatives in the sum vanish, so that in this case DjD,R for & E No, we have
=== O.
If
2k
y=a MeD w- X2h(2
2
This provides us with relatively straightforward expression for the matrix R. in (2.7). From the Rodrigues" formula Hp(x) = (-1)Pe" Dae =²/2 for the probabilist's Hermite polynomials is easy compute ra(z):
= =Di w) y=u Di, exp (-
X2 (2 v)²
=xpexp (-
X2 2/ 20 - a)) a)²) H. (M(x - aa)d- (4.1)
1y=
Finding Snea and Pn. requires the inverse of R.. Fortumately, Xu and Stein (2017, Proposition 3.2) have computed the Cholesky decomposition of the inverse of R., The inverse of R, has the Cholesky decomposition R_ = LLT, where L E R(n+1)x(*+1) is a lower triangular matrix with non-zero elements (L),, = V -)!! when i2 j and i+ j is even. Here i!! is the double factorial, the product of positive integers up to i that have the same parity as i. Thus
p! djj(p- - f)!((p j)m
(R,')s (LL";; x- 10(0+))((
where Q(i.))
Σ ptma p+i is even
(4.2)
Consequently, inserting (4.1) and (4.2) in (2.6) gives the convenient closed form expressions
(z) (ra(z)) ) (() exp (- 1-2( (x and
X2
a
(a) " Q(..) H. (A(2 a))
-
a)²
Σ
Ni!
1=0
X2 (e=-))²) 11 11-e-x((e-)000-
n n
Q(i,))
H, Hi(A(x a, -a)) n)))
Pn.a(x. y) cxp
( (x
Σ>
Σ
ilj!
0,50
In particular, the posterior variance is
n
12
Q(i.)) alj
Pn.a (x,
=1 xx (x 0)
ΣΣ ==
H, (A a)).
These expressions resemble those in (2.8) for Taylor kernels. However, notable difference is that for Taylor kernels the posterior variance blows up as jæ al grows but for the Gaussian kernel the variance tends to a constant as [x --d a) + cc. As discussed in Scction 2.3, both postcrior variancos which blow up and those that remain bounded have their uses. Similarly, the posterior mean for Taylor kernels is unbounded, while for the Gaussian kernel the mean reverts to zero (i.c., the prior mean; recall that we set m = 0) far away from a.
5 General Orthogonal Data
In this section we discuss how simple posterior formulae analogous to those derived in Section 2.3 are available for any data that are orthogonal in the sense that the data are obtained by taking RKHS inner products of f with respect to functions that are orthogonal in the RKHS
Note that the denominator in Equation (3.1) of Xu and Stein (2017) should have (t 2) in the place of (i 5)-
11

---

Published in Transactions on Machine Learning Research (8/2023)
5.1 Generic Construction
Let Ω be an arbitrary non-emply sel, P countable index set, and (0p)peP a collection of linearly independent basis functions on Ω such that Epep (0)(2))² < oo for every Q. We may then define (at least formally) Gaussian process fGP Ω by setting fGP(x) Epep Zz0p(z) for every 92 where Zp are iid standard normal random variables. It is then straightforward to compute that
[[] = O and R(x, v) = (or(fc(())
Σ (p(r)pp()). PEP
(5.1)
The kernel R is pesitive-semidefnite.. Assume that (Øp)pEP are an orthonormal basis the RKHS H(R) (see Section 2.4 for RKHSs).² Then each f H(R) has the pointwise convergent expansion f(x) = Epep fpop(z) for the coefficients fp = (f. (())()) Suppose that one observes 7p5p for p in a finite collection N C of indices and some constants 7p- These data orthogonal because they are obtained by taking inner products of with collection of functions 7p that pairwise orthogonal in the RKHS. That is, each observation Tp may be written as
Jp5p (fi 7p0p) H(N) for functions such that (pppp 7 if p
The orthogonality of 7p0p implies that the corresponding covariance matrix is diagonal. derivation similar to that in Section 2.3 then shows that the posterior mean and covariance are simply
Øp(x) Tp
s(x) = Σ (ppp())= = Σ Tofo
and P(a,v)= E o)(x)o,(1) PEPLN
(5.2)
PEN
PEN
for all y, E Q. See (Wendland, 2005, Ch. 16) and (Oettershagen, 2017, Cor. 3.6) for general formulae when the data consists of applications to f of arbitrary linear functionals. Orthogonal data are known to be optimal in a certain sense and settings (Novak and Wožniakowski, 2008, Sec. 4.2.3). To connect (5.2) to the derivations for Taylor kernels, suppose for instance that f: E IR has the Taylor expansion
Σ fo0p(x), where 6p(x) = oVG,AP p!
f (0)
f(x) Σ
x
and fp=
ovCAA
In this case we therefore have the index set No- Observing the N first derivatives (()) 7p so that W (0, ,......N and 7o = eedd and using (5.2) yields (2.8) with d = 1, a = 0, and m = Moreover, Cov[fcp (y) En- (p()))p()) (xx, y) for K the univariate Taylor kernel in (1.2). We mention two other of examples orthogonal data.
5.2 Mehler Kernel
Let p No and Ω R. Let Hp be the pth probabilist's Hermite polynomial and P (0, Set ov
(x(x) = p0 (n!) Hz(()) Then Mehler's formula yields the Mehler kernel
R(x v) = Σ (z()) = o² Σ p H,(+))H(()) = VI
d²
(- p2 + y²2 - 2pxr
exp
p²
2(1 - p²)
p=0
See Trrgeher and Leobacher (2015) and Oettershagen (2017, Sec. 3.6.4) for the Mehler kernel the context of kernel-based approximation. If f(x) = fn))(),, then by the orthogonality with respect to Gaussian integration, other basic properties of the Hermite polynomials, and properties of Gaussian integrals of derivatives (e.g., Bogachev, 1998, Rmk. 1.3.5),
Pp5p
2T (())))()))XD) (- x2
1
(-2
dr dx= 1 1
V (i) (- x2 (-2)) dr. where 7pp = app.
Here orthogonal data are therefore oblained by Gaussian integration of the derivatives of f-
2 Any functions (PP) for which the expansion of the kernel (5.1) converges pointwise Parseval frome for the RKHS H(R) (Paulsen and Raghupathi, 2016, Thm. 2.10).
12

---

Published in Transactions on Machine Learning Research (8/2023)
Posterior processes
Error and credible interval
10-2-
6
10-10 10
Absolute error
95% CI half-width
10-14 14
-1.0
0.5
0.0 0.5
1.0
25
x
Figure 2: Left: Posterior means and 95% credible intervals given derivative data for f(x) sin(xx) at O. The zero-mean prior uses the exponential kernel K(x, v) 0² exp(Ary) with À 3/2 and scale o set using maximum likelihood. Righl: Maximal absolute errors max [f(x)) (n)) and half-widths
1.96 x max (Pna(I,I) of the 95% credible interval over the domain Ω = [-,
5.3 Periodic Kernel
Let p E, Ω [0, and s E N. Set 6p(z) = 0V2((2)) cos(2xpz) and Φ-p(x) = 2(Tp)))) (sn((mrrr) for p N. Morcover, set Φu c. Then we obtain the periodic Sobolev kernel (or the Korobov kernel)
R(x,y) Σ (x(()))(1)) = 0² (1(1+2) DEZ
00
2Tp)2 cos(22 w)) 0² 22(1 (-1) (2s) B2o 31) (5.3)
for x, y 1), where B2s is the Bernoulli polynomial of degree 2s; see, for example, Wahba (1990, Sec. 2.1). If f: [0, 1] + IR has the expansion f(x) Epez fr0p(x), then it is straightforward to compute that
Jnln -2 S( [(x)) dr and -2 2 S/ f(x) sin (2TpH) dr for PEN
(5.4)
for "p = -2 and rofo (x) de for 7 These orthogonal data the Fourier coefficients.
6 Two Toy Examples
This section contains two numerical toy examples. Figure 2 displays a number of posterior processes and the bchaviour of maximal error and standard deviation when zero-mean Gaussian process with the Taylor kernel K(r,8) v) = o² exp(Acy) with À = 3/2 is used to infer the function f(x) sin(TT) based on noiseless derivative evaluations at a = 0, as described in Section 2. See also Figure 1. The scaling parameter a was taken to be the maximum likelihood estimate in (3.2). From the right panel we see that the Gaussian process model is well-calibratod in the weak sense that, except for small n, fx) never further away from the posterior mean than maximal half-width of the 95% eredible interval over the domain f [-1,1] of interest.: max fz) ( [n)) 1.96 max V Pn,aala (2,
Our second example uses the periodic kernel (5.3) with 2 and scaled Fourier data in (5.4), so that the posterior mean and covariance given by (5.2). We use index sets of the form = {-n,..., -1, 0, 1,..., n) for E N and again use maximum likelihood to set the scaling paramcter, which in this case simply yields oh 1
2 Σm=-n(
The function being inferred is f(z) exp(x), and we compute that
Ppfp Sp (2erp (2mp) + cos(22p)
and 7-2f-p sp (Tp + esin 2eTp (2TTp)
for p E N, where sp 2/(4 + 1), and 20.0 = Figure depicts some of the resulting posterior processes. Except the boundaries where the Gibbs phenomenon caused by the non-periodicity of f occurs, the posteriors fare well and appear provide reasonable quantification of predictive uncertainty.
13

---

Published in Transactions on Machine Learning Research (8/2023)
Posterior processes
Posterior processes
2.5
2.5
2.0
2.0
1,5
1,5
16
17
1.0
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.1
0.6
0.8
1.0
x
x
Figure 3: Posterior means and 95% credible intervals given Fourier data for f(x) exp(x) f= [0, 1]. The zero-mean prior uses the periodic kernel in (5.3) with
and scale set using maximum likelihood.
7 Conclusion
We have proposed Gaussian process model based on Taylor kernels which gives rise to probabilistic version of the classical Taylor expansion when the data consist of derivative evaluations. Using Taylor kernels in Bayesian optimisation (Snoek et al., 2012) would be an interesting future application, where they might be expected to inherit properties from both standard Bayesian optimisation algorithms based on commonly used stationary kernels, such as the Gaussian and Matérns, and classical optimisation algorithms. Because their uncertainly explodes away from the expansion point, Taylor kernels might prove a useful alternative stationary kernels which have tendency be over-exploitative in Bayesian optimisation (Bull, 2011).
Acknowledgments
TK was supported by the Academy Finland postdoctoral researcher grant #338567 "Scalable, adaptive and reliable probabilistic integration". FT was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) fundod by the Knut and Alice Wallenberg Foundation. We thank Chris J. Oales, Onur Teymur, and Matt Graham for helpful comments on early version of this article.
References
Álvarez, M. A., Rosasco, L., and Lawrence, N. D. (2012). Kernels for vector-valued functions: A review. Foundations and Trendso in Machine Learning, 4(3):195 166.
266.
Ben Salem, M., Bachoc, F., Roustant, O., Gamboa, F., and Tomaso, L. (2019). Gaussian process-based dimension reduction for goal-oriented sequential design. SIAM/ASA Journal on Uncertointy Quantification, 7(4):1369 1397.
Berlinet, A. and Thomas- Thomas-Agnan. C. (2004). Reproducing Kernel Hilbert Spaces in Probability and Statistics. Springer.
Bogachev, 1. (1998). Gaussian Measures. Number 62 in Mathematical Surveys and Monographs. American Mathematical Society.
Bull, A. D. (2011). Convergence rates of efficient global optimization algorithms. Journal of Machine Learning Research, 12:2879-200 2904.
Cockayne, J., Oates, C. J., Ipsen, I. C. F., and Girolami, M. (2019a). A Bayesian conjugate gradient method (with discussion). Bayesian Analysis, 14(3):937-1011 1012.
Cockayne, J., Oates, C. J., Sullivan, T., and Girolami, M. (2019b). Bayesian probabilistic numerical methods. STAM Revieu, 61 (144:7566-78))
14

---

Published in Transactions on Machine Learning Research (8/2023)
Conn, A. R., Gould, N. I., and Tointi, P. L. (2000). Trust- Prust-Region Methods, volume of MPS-SIAM Sertes Oplimization. Society for Industrial and Applied Mathematics.
De Marchi, S. and Schaback, R. (2010). Nonstandard kernels and their applications. Dolomites Research Notes on Approrimation, 2:16-43.
Diaconis, P. (1988). Bayesian numerical analysis. In Statistical decision theory and related topics IV, volume 1, pages 163-175. Springer-Verlag New York.
Dick, J. (2006). A Taylor space for multivariate integration. Monte Carlo Methods and Applications, 12(2):99-1)) 112.
Eriksson, D., Dong, K., Lee, E., Bindel, D., and Wilson, A. G. (2018). Scaling Gaussian process regression with derivatives. In Advances in Newral Information Processing Systems, volume 31, pages 6867-6877.
Fasshauer, G. and McCourt, M. (2015). Kernel- Based Approzimation Methods Using MATLAB. Number 19 in Interdisciplinary Mathematical Sciences. World Scientific Publishing.
Hairer, E., Nørsett, S. P., and Wanner, G. (1993). Solving Ordinary Differential Equations I: Nonstiff Problems, volume of Springer Series in Computational Mathematics. Springer.
Hennig, P. (2015). Probabilistic interpretation of linear solvers. SIAM Journal on Optimization, 25(1):234-260)
Hennig, P., Osborne, M. A., and Kersting, H. P. (2022). Probabilistic Numerics. Cambridge University Press.
Irrgeher, C. and Loobacher, G. (2015). High-dimensional integration on Rd, weighted Hermite spaces, and orthogonal transforms. Journal of Complenity, 31(2):174-25) 205.
Kanagawa, M., Hennig, P., Sejdinovic, D., and Sriperumbudur, B. K. (2018). Gaussian processes and kernel methods: A review connections and equivalences. arXiu:1807 107.:08877.02522.1
Karvonen, T. and Oates, C. J. (2023). Maximum likelihood estimation in Gaussian process regression is ill-posed. Journal of Machine Learning Reseorch, 24(120):1-47.
Karvonen, T., Oates, C. J., and Särkkä, S. (2018). A Bayes Sard cubature method. In Advances in Neural Information Processing Systems, volume 31, pages 5882 5893.
Karvonen, T. and Särkkä, S. (2017). Classical quadrature rules via Gaussian processes. In 27th IEEE International Workshop on Machine Learning for Signal Processing.
Kimeldorf, G. S. and Wahba, G. (1970). A correspondence between Bayesian estimation on stochastic processos and smoothing by splines. The Annals of Mathematical Stotistics, -12):4955-50..
Larkin, F. M. (1970). Optimal approximation in Hilbert spaces with reproducing kernel functions. Mathematics of Compulation, 24(112):911 921.
Liang, T. and Rakhlin, A. (2020). Just interpolate: Kernel "ridgeless" regression can generalize. Annals of Slatistics, 48(3) 1347.
Minh, H. Q (2010). Some properties of Ganssian reproducing kernel Hilbert spaces and their implications for function approximation and learning theory. Constructive Approzimation, 32(2):307-338.
Minka, T. (2000). Deriving quadrature rules from Gaussian processes. Technical report, Statistics Department, Carnegic Mellon University.
Moré, J. J. (1978). The Gevenberg-Marquarde algorithm: Implementation and theory. In Numerical Analysis, volume 630 of Leclure Noles in Mathemalics, pages 105 116. Springer.
Novak, E. and Wodniakowski, H. (2008). Tractability of Multivoriate Problems, Volume I: Linear Information. Number 6 in EMS Tracts in Mathematics. European Mathematical Society.
15

---

Published in Transactions on Machine Learning Research (8/2023)
Octtershagen, J. (2017). Construction of Optimal Cuboture Algorithms with Applications to Econometrics and Uncertainly Quantification. PhD thesis, Institut für Numerische Simulation, Universität Bonn.
Paulsen, V. and Raghupathi, M. (2016). An Introduction to the Theory of Reproducing Kernel Hilbert Spaces. Number 152 in Cambridge Studies in Advanced Mathematics. Cambridge University Press.
Prüher, J. and Särkkä, S. (2016). On the use of gradient information in Gaussian process quadratures. In 26th IEEE International Workshop on Machine Learning for Signal Processing.
Rasmussen, C. E. and Williams, C. K. 1. (2006). Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. MIT Press.
Raudenbush, S. W., Yang, M.-.,, and Yosef, M. (2000). Maximum likelihood for generalized linear models with nested random effects via high-order, multivariate Laplace approximation. Journal of Computational and Gruphical Slatistics, 9(1):141-155 1157771-17
Richter-Dyn, N. (1971a). Minimal interpolation and approximation in Hilbert spaces. SIAM Journal Numerical Analysis, 8(3):583-55) 597.
Richter-Dyn, N. (1971b). Properties of minimal integration rules. II. SIAM Journal on Numerical Analysis, 8(3):497-508.
Särkkä, S. (2013). Bayesion Filtering and Smoothing, volume of IMS Teztbooks. Cambridge University Press.
Scheuerer, M., Schaback, R., and Schlather, M. (2013). Interpolation of spatial data A stochastic or a deterministic problem? European Journal of Applied Mathematics, 24(4):601-6%) 6629
Schober, M., Duvenaud, D. K., and Hennig, P. (2014). Probabilistic ODE solvers with Runge- Runge-Kutta Kutta means. In Aduances in Neural Information Processing Systems, volume 27, pages 739-747.
Schober, M., Särkkä, S., and Hennig, P. (2019). probabilistic model for the numerical solution of initial value problems. Statistics and Computing, 29:99-122.
Snoek, J., Larochelle, H., and Adams, R. P. (2012). Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, volume 25, pages 2951-2959.
Solak, E., Murray-Smith, R., Leithead, W. E., Leith, D. J., and Rasmussen, C. (2002). Derivative observations in Gaussian process models of dynamic systems. In Advances in Neural Information Processing Systems, volume 15, pages 1057-1064.
Steinwart, I. and Christmann, A. (2008). Support Vector Machines. Information Science and Statistics. Springer.
Särkkä, S. (2011). Linear operators and stochastic partial differential equations in Gaussian process regression. In International Conference on Artificial Neural Networks, pages 151 158.
Teymur, O., Zygalakis, K., and Calderhead, B. (2016). Probabilistic linear multistep methods. Advances in Neural Information Processing Systems, volume 29, pages 4321 4328.
Travelletti, C. and Ginsbourger, D. (2022). Disintegration of Gaussian measures for sequential assimilation of linear operator data. urXiv:2207.1385 13581v.
Wahba, G. (1990). Spline Models for Observational Data. Number 59 in CBMS-NSF Regional Conference Series in Applied Mathematics. Society for Industrial and Applied Mathematics.
Wendland, H. (2005). Scattered Data Approrimation. Number 17 in Cambridge Monographs Applied and Computational Mathematics. Cambridge University Press.
Wu, J., Poloczek, M., Wilson, A. G., and Frazier, P. I. (2017). Bayesian optimization with gradients. In Aduances in Newral Information Processing Systems, volume 30, pages 5267-5278.
16

---

Published in Transactions on Machine Learning Research (8/2023)
Xu, W. and Stein, M. L. (2017). Maximum likelihood estimation for smooth Gaussian random field model. STAM/ASA Journal on Uncertainly Quantificntion, 5(1):138-175) 175.
Zwicknagl, B. (2009). Power series kernels. Constructive Approzimation, 29(1):11-84.
Zwicknagl, B. and Schaback, R. (2013). Interpolation and approximation in Taylor spaces. Journal of Approrimation Theory, 171:65-83. 83.
17


Reviews:
[{"review_id": "ELrQBjWRtK", "reviewer": "Reviewer_cMBi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is reasonably well written apart from missing a few definitions, e.g. I'm still unclear what \"orthogonal data\" are. It is on the long side and mathematically dense, I wonder if there are intermediate steps that the authors could move to an appendix to allow the reader to focus on the key results. \n\nI'm unclear if the key result (mentioned above) is novel: if so that is very elegant and in my mind the major strength of the work. \n\nPerhaps the main weakness is that practical utility is not demonstrated well. There are just two toy numerical examples, approximating sin(pi x) and exp(x). Neither test multivariate inputs. Relatedly, their own conclusion is that simultaneous estimation of sigma (signal variance) and lambda (inverse length scale) is problematic. How should this be handled in practice? Would Bayesian inference help (at some computational cost)? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider using GPs to model epistemic uncertainty when approximating a function locally using a Taylor series expansion. They introduce Taylor kernels for both univeriate and multivariate input GPs. Perhaps the key result is that when using these kernels and conditioning on a number of known derivatives at a point, the resulting posterior GP has mean function equal to the usual Taylor expansion. That in itself of course is not useful, but the GP also provides an analytic expression for the posterior variance, which grows to infinity far from the expansion point. Further theory is developed covering error bound, noisy data, hyperparameter estimation and \"orthogonal data\". Two small numerical examples are presented. \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Ideally I would like to see a more \"real\" motivating example where the theory can be put to practice. \n\nMinor: \n- I believe K in Theorem 2.1 is missing a K_a - term. \n- In the proof for thm 2.1 there is no need to repeat the final s and P equations. \n- just before section 5.2: examples OF orthogonal data", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HzAnmKicH7", "reviewer": "Reviewer_zF4Z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- strengths\n\nDeveloping methods to estimate Taylor's approximation error empirically is a good idea. Interestingly, the uncertainty depends explicitly on the kernel hyperparameters. \n\nCould the provided uncertainty quantification be exploited as a general tool for formal proofs involving the Taylor expansion?\n\n- weaknesses\n\nThe clarity of the exposition should be improved. Sometimes, it is hard to understand the goal of an entire section.  For example, introducing RKHS does not seem to be strictly necessary.\n\nThe authors should give a practical example where estimating the uncertainty of the Taylor expansion is useful.\n\nIt is unclear what is the net contribution of the paper.  Gaussian Process regression with derivative data already exists. But, according to the authors, previous work only focuses on the Gaussian Kernel. Is the novelty here the extension to the more Taylor Kernels defined in Equation 2.3? If so, the authors should point out the advantages of such an extension.\n\nCompared to Gaussian kernels, Taylor kernels may diverge or grow exponentially when |x-a| is large. The authors do not discuss the practical and theoretical implications of this. It would be nice to include a set of conditions (e.g. on the coefficients of the Taylor expansion) such that the good behaviour of Gaussian kernels is reproduced.\n\nThe experiments are limited. The very brief experiment section suggests general confusion about the practical impact of the proposed scheme.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- summary\n\nThe article establishes a theoretical link between Gaussian Process regression with derivative data and a probabilistic version of Taylor's theorem expansion. The Gaussian Process framework helps quantify the uncertainty of Taylor approximations when the target and reference points are not so close. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "questions:\n\n- What is a possible application of the proposed method?\n- How expansive is the computation of the differential data? What is the overall complexity of the proposed approach? Are Taylor kernels computationally cheaper than Gaussian kernels or standard derivative-based regression methods?  \n- Is assuming the independence of the errors as in Section 2.5 standard? How can this happen, intuitively, when all derivatives are obtained from the same process?\n- Which parameters can be estimated (through ML)? Which are fixed by the Taylor structure? Is the \"estimation of c_p\" equivalent to choosing a specific kernel class? Does c_p need to be an explicit function of p?\n- What is the main difference between the proposed approach and existing methods for regression with derivative data?\n- Why is the possible uncontrolled behaviour of Taylor Kernels when |x-a| grows a good feature?  \n\n", "broader_impact_concerns": "I do not have any ethical concert about this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8yw19pYYmf", "reviewer": "Reviewer_Hfrv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n - The paper has well-chosen graphical examples.\n - While not being mathematical complicated, the results are mathematically interesting and well-interpreted.\n - The paper is clear in its language. Precice, when it needs to be; while still giving intution whenever possible.\n - The paper checks the boxes in correctness and suitable citations.\n - All claims are verified, mostly theoretically, but also with (few, due to space constraints) examples.\n - The paper was interesting to read to me and I am certain that it is interesting to be read in a wider machine learning audience.\n\nSorry for the boring review. In my opinion, the paper is mostly ready to be published, apart from a few minor suggestions below.\n\nWeaknesses:\n - One weakness for practical application is the difficulty when applying the dicussed covariance functions in practice. The hyperparameters corresponding to higher order Taylor coeffcients are not determinable. Since the error estimates depend strongly on these non-observable values, the error estimates are mostly guesses - even though they work out nicely in the examples. While not hidden, this point could be stressed or more discussed in the paper.\n\nQuestions:\n - Why is $c_\\alpha>0$ infinitely often? Why exclude polynomial kernels from your class?\n - Why do you need the hyperparameter lambda? Why not subsume it into the $c_\\alpha$'s?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper discusses the connection of Gaussian process covariance functions and Taylor series. It prooves certain connections between convergence radii, discusses fitting hyperparameter, in particular those corresponding to Taylor coefficients, and it proves and generalizes orthogonality conditions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": " - The characterization of GPs at the beginning of Subsection 2.2 is wrong. It is only correct, if you also demans the subsequent properties about evaluations of GPs.\n - Also in section 2.2, you probably want more than \"R is n times differentiable in both arguments\", but also that all mixed derivatives exist (and perhaps even continuity of those).\n - As described above, the weakness about non-observable hyperparameters and error bounds could be stressed more.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)