Policy Gradient with Kernel Quadrature
Authors: Satoshi Hayakawa
Tetsuro Morimura
Venue: Accepted by TMLR ()

Abstract: Reward evaluation of episodes becomes a bottleneck in a broad range of reinforcement learning tasks. Our aim in this paper is to select a small but representative subset of a large batch of episodes, only on which we actually compute rewards for more efficient policy gradient iterations. We build a Gaussian process modeling of discounted returns or rewards to derive a positive definite kernel on the space of episodes, run an ``episodic" kernel quadrature method to compress the information of sample episodes, and pass the reduced episodes to the policy network for gradient updates. We present the theoretical background of this procedure as well as its numerical illustrations in MuJoCo tasks.

Published in Transactions on Machine Learning Research (02/2024)
Policy Gradient with Kernel Quadrature
Satoshi Hayakawa
baggakaun@@mutttsooo.oo.
Mathematical Institute, University of Oxford
Tetsuro Morimura
morimara morimurre_tessurueeeeleee hetp://eeeesso@@@eegggga.......
Cyber,gent, Inc.
Reviewed on OpenReview: https: // openreview net/ forum? id= WFI9zhJraF
Abstract
Reward evaluation of episodes becomes bottleneck in broad range of reinforcement learning tasks. Our aim in this paper is to select small but representative subset of a large batch of episodes, only on which we actually compute rewards for more efficient policy gradient iterations. We build a Gaussian process modeling of discounted returns or rewards to derive a positive definite kernel on the space of episodes, run an "episodic" kernel quadrature method to compress the information sample episodes, and pass the reduced episodes the policy network for gradient updates. We present the theoretical background of this procedure well as its numerical illustrations in MuJoCo tasks.
1 Introduction
Reinforcement learning (RL) aims to learn policy model that maximizes the cumulative average of rewards Policy gradient algorithms operate based on gradient ascent in the policy parameter
(Sutton & Barto 2018
Schulman et al. 2017) and have greatly benefited from the latest ad advancements in neural network models.
space [Gullapalij 1990
Williams
1992
These algorithms have found widespread applications in domains
such as robotics Peters Schaal 2008),
large language models (Ouyang et al. 2022 medical diagnosis
[Xia al. 2020). and many others.
Despite the broad applications of RT, significant. challenge, often overlooked, is the extensive computational monetary cost associated with reward evaluations in real-world RL scenarios. Notably, in domains like material science and fluid dynamics, physical simulators are often used for evaluation of policy decisions Fan et al. 2020 Rajak et al., 2021). These simulations tend to be computationally intensive. For example, reward caleulation using flow simulation in Fan et al. 2020)) requires about 1.1 hours for each episode. Other computationally demanding tasks include Neural Architecture Search (Zoph & Le 2017) and Ordering- Based Causal Discovery (Wang al.) 2021). Furthermore, tasks like person re-identification (Liu al.) 2019) and RL with Human Feedback (RLHF) (Ouyang al. 2022) demand human annotation for reward computation. This human annotation process often becomes bottleneck, emphasizing the need to minimize such instances.
While recent approaches like RL with AT feedback (RLAIF) [Lee el al. 2023) show promise, querying external AI typically incurs a cost, reinforcing the desire to minimize the number of reward evaluations.
Given this backdrop, our primary motivation to alleviate the computational and monetary burdens of reward evaluations in the online RL setup. We try to accelerate policy gradient methods by employing a novel approach that efficiently selects representative subset of episodes for reward computations.
Let us start with introducing necessary notations in RL and policy gradient methods.
1.1
MDP and policy
We are given state space S, an action space A, a reward function r: P:8AA R, and transition probability p(11s,00 a) over S conditioned each state-action pair along with distribution over initial states p(s); the

---

Published in Transactions on Machine Learning Research (02/2024)
tuple of these defines Markov decision process (MDP). In the RL setting, it is typically the case that both the transition probabilities p(-1s,a) a) and the initial state distribution p(s) are unknown to the agent, requiring the agent to learn an effective policy through interaction with the environment.
Stochastic policy T--ss) probability density over A (with a canonical reference measure) conditioned on each 8 S. Given p and v, we can generate Markov chain that starts from a state 8p E possibly drawn from a probability distribution associated with the MDP and continues as dr [T|ss)) and 5(+1 ~ p(-1st, az) for 0. We call such Markov chain an episode (8t, ae)120 it can be of finite length T = T(e) due to some termination rule of MDP. Since p is fixed in our setting, we might abuse the notation reprosent the stochasticity of e.
to
Given an episode e = (x)20 the discounled return at time / is defined as (e(e) = == 7 ' a.); where E (0,1) is a discount rate. When the dependency on e is apparent, we might simply denote r(e) = (r:)e20 = (r(st, a)))eo and R, = Eu>r?" "m- We finally define the Q-function Q S A R and the value function y=:8. R associated with the policy π by
Q" (8, Rn (R((e) 8, ao a). V=(s) B Ro(e)]oo (1) (s,,
1.2 Policy gradient
With parameterized stochastic policy me, aim to maximize (TTO) == Benng [Ro[e]] with respect to 0. Its gradient can be written as
(TT( = Eern
Σ 120
7¹ logg)Vologoollll = Eeor
Σ 120
A" a Vo
where A a= (= (s) the advantage function, introduced here for variance reduction.
In practice, usually approximate the gradient by Monte Carlo integration by generating episodes Eiy...,EN mid
1
VOJ(TT)) R
((e:),
(1)
N
G(e) = Σ 7 Ar(e) log (a(()))))
(2)
120
where Ar(e) is an approximation of (st, a:), which requires evaluations of r(s,, a.) for 2 0. A typical choice for is A((e)= = (e(c) - Vst)),, where V₂ is a parametric approximation of the value function, often referred to as the baseline and updated iteratively (Williams) 1992). We refer to this method as the vanilla policy gradient (vpg).
We shall denote the policy parameter by and all the other parameters including bascline and kernel hyperparameters by s. In order to highlight our motivation, we assume that we can separate running an episode and evaluating the reward r(e) (and so Á,(e)); this policy gradient method is summarized in Algorithm
Algorithm Policy gradient
Input: A policy ne. advantage estimator Å for ileration = .... do Generate episodes (e:) Iid me Compute (r(e:)) and then (A(e:))
4: 0-0 aN E G(e,) (a: learning rate)
Update Å by using (e,, ((e:))
6: end for
2

---

Published in Transactions on Machine Learning Research (02/2024)
1.3 Contribution
To speed up the usual policy gradient methods, we propose combining policy gradient algorithms and kernel quadrature for reducing episodes, particularly aiming at expensive-reward situations mentioned in Section [ A positive definite kernel K on a domain is a two-variable function K E + I such that, given any points ep....,cc E S with any n, the n x n matrix (K(s,, ej e)))r=1 is symmetric and positive semi-defintte Such a kernel gives an embedding of & into a wider space called reproducing kernel Hilbert space, and ((e, e') is an "inner product" between the points and e' in that space, so kernel over episodes basically enables to mcasure the magnitude and similarity of episodes.
Kernel quadrature in this setting runs as follows: given positive definite kernel of episodes ((e,e)) over space of episodes ε, we approximate the N- N-point empirical measure of episodes by a weighted n-point subset with a certain criterion based on K: N Ei= 6e, = Eier w,de, with I C fI...., NY and | = n, where 5, is the delta measure at e E &, which corresponds to a one one-point distribution on & almost surely taking e. The empirical mcasure episodes we use from rcalistic number of samples (e+), from the stochastic policy and the MDP, which is used in the usual policy gradient algorithms. Thus, kernel quadrature works data compression of the empirical measure to reduce the number of reward evaluations.
This process in our implementation is treated as a (black-box) function KQuad (K, (e:), the convex kernel quadrature algorithm given by Hayakawa et al. (2022), which is briefly explained in Section 3.1 However, we can use any other efficient kernel quadrature method as the KQuad function as long as it applicable given an empirical measure and a kernel (a small ablation study given in Appendix
Algorithm Vanilla PGKQ
Input: < N, a policy wa, advantage estimator Å, and episodic kernel K for iteration = 1,2.... do
Generate episodes (e:), riid I, (wi)ies + KQuad(K, (e with 1|
3:
Compute (r(e,))ee and then (A(e.), G(e,))ve
5:
O - 0 a Eier ((e())
Update Å by using r(e))ee
Update K by using (w,,,, r(s)))e
end for
The proposed algorithm is given in Algorithm 2 as policy gradient with kernel quadrature (PGKQ). See Remark (3] for the computational complexity. While the detailed explanation of kernel quadrature under an MDP is described in Section 3 our contributions are summarized as follows:
We develop theory to make available the kernel quadrature over episodes, by introducing Gaussian process modeling of returns or rewards, which can also be combined with policy gradient relatives including PPO (Schulman et al. 2017).
We compare two versions of our PGKQ method with existing policy gradients with small and large batch sizes, and see the efficiency of PGKQ in catching up with the large-bated batch policy gradient only by small-batch reward observations.
2 Related literature
2.1
Bayesian quadrature for policy gradient
The most directly relevant to our study is the application of Bayesian quadrature (O'Hagan 1991] for estimating policy gradient (BQPG; Ghavamzadch & Engel 2007 Ghavamzadch et al. 2016 Akella et al. 2021). They model the Q-function Q by a Gaussian process GP: see Section 3.11ton a formal definition), and estimate the policy gradient (m)(m)) as posterior of vector-valued GP based upon (noisy) observations
3

---

Published in Transactions on Machine Learning Research (02/2024)
of Q". Their main contribution from the viewpoint of gradient estimate is placing better weight than the uniform weight in
Although we also use GP modeling and weighted gradient estimate, our methods are different from these existing studies in the following two points.
Episodes reduction. As already mentioned in Section 1.3 our objective is to approximate large batch of episodes by smaller batch of weighted episodes in order to reduce the nunber of reward computations, while the variants of BQPG are on how to better estimate the policy gradient by a fixed batch of episodes.
Flexible kernel selection. As is always the case with the use of Bayesian quadrature, we need to know the exact values of some integrals associated with the covariance kernel of the GP (e.g., Eq. (16) in Ghavamzadeh & Engel 2007) to execute BQPG. They need to use specifie class of kernel due to this limitation, while our method is valid for any choice of kernel because of the existence of larger empirical mcasure that we want to approximate.
2.2 Numerical integration in data science
Estimating intractable integrals with a small number of integrand evaluations classically ranges from Monte Carlo (Metropolis & Ulam 1949). cubatuue (Stroud 1971] to QMC (Dick et al. 2013), while the recent
literature also includes Bayesian/ kernel quadrature (O Hagan
2010 Bach 2017), recom- bination (Litterer & Lyons] 2012 Tchernychova 2016), coresets (Bachem et al.] 2017), determinantal point
1991 Chen et al.
processes (DPPs; Bardenet al.) 2020), or dataset distillation (Wang et al. 2018). It is impossible to explain each method in detail, but they all agree on approximating (probability) distribution, which is typically continuous distribution or large discrete data, by small (weighted) set (i.e., a discrete measure with small support).
Let us mention some existing applications of these methods towards better/faster gradient ostimates in data science, stochastic gradient descents (SGDs; Robbins & Monro 1951 in particular. DPPs have already been applied to acquire better mini-batches for gradient estimates SGDs (Zhang et al. 2017 2019 Bardenet et al. 2021), while there also is an application of recombination with the same motivation (Cosentino al, 2020 Small-GAN (Sinha et al. 2020) uses coresets for a better minibatch selection when training generative
adversarial networks (GANs; Goodlellw et al.
2020). Kernel quadrature, method of seeking coreset based on kernel-based discrepancy, has also been applied to iterative updates in data science; minibatch selection
in a warped Bayesian quadrature (Adachi et al. 2022) and Bayesian optimization (Adachi et al. 2023a
and dictionary compression in model-based RL CCakraborty et al.) 2023
Our contribution compared with these studies is on the GP modeling specific to MDPs and the resulting application of kernel quadrature to suited for efficiently estimating the policy gradient. The main difficulty in applying existing sample- -effcceeffi gradient estimators for policy gradient is that points in our setting (i.e., episodes) are not simply a vector in a fixed space: their length can differ (and be very long) and each timestep affects the behavior afterward. This is where we benefit from introducing stepwise GPs and obtaining the overall kernel for a pair of episodes in Section 3.2 Therefore, none of the existing sample reduction techniques are readily applicable to our setting, to the best of our knowledge. Our framework introduced in Section 3.2 then enables us to apply general kernel quadrature methods (not limited KQuad from Hayakawa et al. ); see Appendix C to policy gradient algorithms, but selecting the best-fit kernel quadrature from the literature is not our main objective: our primary contribution is the introduction of a general framework that connects policy gradient with kernel quadrature, particularly useful in scenarios where reward evaluation is bottleneck, and confirm that the idea itself works with the standard benchmarks in RI.
3
Kernel quadrature for reducing episodes
3.1 Kernel quadrature in general
Kernel quadrature is a way of approximating a large or continuous distribution by a small discrete distribu- tion, where the following function space plays an essential role.
4

---

Published in Transactions on Machine Learning Research (02/2024)
The reproducing kernel Hilbert space (RKHS) H associated with positive definite kernel K over space X' is the (smallest) Hilbert space of functions from X' to R satisfying the following properties:
For each fixed x E X, the univariate function K₂ such that Kx(+) K K(x,)) is contained in H.
For each I, y E X, the inner product between K2 and Ky satsfies (Ka; K F = K(r,y).
The first property says that the space H is basically spanned by the embeddings ky of points x E S, and allows linear algebra over the abstract space X. The second property corresponds to what we wrote above: k(r, y) is an "inner product" between the points x and y in that space (i.c., K₂ and Ky). In our context, by finding an appropriate RKHS for episodes (this is equivalent to finding a positive definite kernel over episodes); we can introduce the established kernel quadrature techniques, which enables us to find an efficient discretization of randomness given by π.
In general, for positive definite kernel K : X' x X' - I2 and probability distribution μe, kernel quadrature
aims to find a good quadrature rule H4n = x;) - a set of weights w; E IR and points I; E X that makes the following the worst-case error small: woolpun K, == sup (mn(F) - ~(f)), 10
where H is the reproducing kernel Hilbert space (RKHS) associated with the kernel K, pa(s) = E (x;), and u(f)= == (x))
There are many algorithms for this problem, including but not limited to a greedy algorithm called herding
(Chen et al.] 2010 Huszár & Duvenand 2012 Bach et al. 2012 Tsuji et al. 2022), weighted sampling
methods (Bach, 2017; Belhadji et al. [019)] 2020 Belhadji 2021 Epperly & Morcno, 2023). a subsampling
methods called Chinning (Dwivedi Mackey 2021: 2022; Shetty el. al.] 2022); see Hayakawa et al. (0022
Table 1) for comparison of these methods. We use the convex kernel quadrature (Hayakawa et al 2022;
2023) for its empirical competence, but our method can incorporate any kernel quadrature method feasible with a general kernel and discrete space.
Convex kernel quadrature with empirical measure. Let us briefly explain a version of the KQuad algorithm in Hayakawa et al. (2022), which we use in our implementation of PGKQ. First, we are given a positive definite kernel K on a set of points (x.), Let uI...., un-1 E EN be the eigenvectors (in implementation, given by the randomized SVD (Halko al. 2011)) corresponding largest eigenvalues of the Gram matrix (K(z,, (1,,
the first to (n - 1)-th Then, we find a vector w RV such that w u, $1 u, for i = I,...,,. n - 1, w 2 0, 1 1Tw = 1, and |wlo S n (where = (l,.... 1)T 0 0= (). E R. and | is the number of nonzero entries), by solving a linear programming problem (particularly using the recombination algorithm (Litterer & Lyons 2012 Teheruychova, 2016)). The nonzero entries correspond to the points and weights used in j,n; the index set 7 and weights w/; mentioned in Section 1.3 See the code
and the original paper for the actual implementations and the theoretical guarantees.
Kernels and GPs. The choice of the kernel K is essential for the applications of kernel quadrature Briol et al. 2017), and we propose exploiting the covariance kernels of GPs of the discounted returns Re(e) or the reward function r in Section 3.2 in order to set an appropriate kernel over the space of episodes.
Formally, a real-valued GP on X' is a distribution over the space of functions X + R, associated with a mean function m : X + R and covariance kernel function K X' X + I. denoted as GP(m, K). It satisfies that, for a random function f ~ GP(m, K) and any 21...., In E X. (f(x)))= follows normal distribution with the mean vector (m(x:)) and the covariance matrix (z),) We write the expectation with respect to the distribution of the Gaussian process we have as Egp (e.g. Egrls] = m), and call / ~ GP(m, K) centered if m = 0 as a function.
Before going into details of the modeling the case of MDP, we give an important relation between kernel quadrature and GP. The following is an immediate generalization of Huszár & Duvenaud 2012 Proposi- tion 1):
5

---

Published in Transactions on Machine Learning Research (02/2024)
Proposition 1. Let f (P(O, be centered GP X and j be probability distribution satisfying Ix K(x, a) dp(x)) < c0. Then, for a kernel quadrature rule Pmu the same space, we have
Egp((un()) (ac(n(())~(((())))) wee(pns 1.)2
Note that the GP being centered essential since otherwise an additional term of (un(m) su(m))² with m = Eggl/ appears. We give its proof in Section A.1 for completeness. In the context of our methods, p is typically given by a large batch of episodes and Mn a quadrature rule with a smaller support, with which we estimate the policy gradient. For more details on the role of this proposition, see the following sections, e.g.: the description after Proposition 2
3.2 Gaussian process modeling of MDP
Recall the notations for MDPs introduced in Section 1 In particular, we are given parametrie policy Te, A((e) is approximation of the advantage (s,,, az), and Gx(e) is an episodic gradient defined by
Also, recall that our objective is to find a good subset of episodes before evaluating rewards. Our use of GP is based on the following heuristic:
For a set of episodes (es) and weights (wi)ier with c {I...., Nh, if the uncer- tainty of 1/2 Ei=1E.20 ' Ares)) Eier wi Σi20 7 Ac(es) is small, then the uncertainty of E ((es) - Eier ((s))) is also expected to be small.
The uncertainty mentioned above is not mathematical term, but can artificially be introduced as a variance of a Gaussian variable based on the GP we are working with. We assume above that reducing the GP variance of the scalar-valued quantity based on is likely to result in a reduced GP (co)variance of the gradient estimate based on G, which of the vector-valued. By looking at (2), we are assuming above that ignoring the vector-valued term V log (e(aelst) does not so much harm the quality of the reduced episodes.
This heuristic not only simplifies the derivation of kernels into the MDP context, but also provides unified
treatment to similar policy-gradient methods such as TRPO and PPO (Schulman et al.) 2015 2017 where the loss fonctions are given by multiplying the advantage iunction and other score-related terins.
Under the above heuristic, what we need to do is derive GP for the following functional of an episode:
(e) = Σ A'Ar())
(3)
120
as we can then use kernel quadrature via Proposition 1 We want to directly model F, but will start by modeling smaller components such as RG and for more data and flexibility.
In the following, we introduce two ways of modeling Á with episodic GP. These are based on simpler base GP for either the return R2 or a reward Although we primarily consider the estimator Å.(e) Re(e)- - Vst(s,, our argument can be generalized to more complicated estimators such as R&(e) - Vs() - yno (R,, - (Sta)) (for a certain to) introduced by Mnih et al. (2016).
For simplicity of notation, we shall write
(s, 21 (5t, E
and = (st, (3 1>00 = (2))22 for state-action pairs in the following. We present two ways of GP modeling
in Sections 3.2. 3.2.2 and we explain how to update the GPs (mean function, covariance kernel) over
iterations Scction 3.2.3
3.2.1 Option 1: modeling R, with GP
In this model, our base GP on Z given by
R4 Rela,=z ~ GP((V, kv);
(4)
6

---

Published in Transactions on Machine Learning Research (02/2024)
where V= = (s) is the baseline function in the policy gradient algorithm and Ky is positive definite kernel on the domain 2 S A with hyperparameter vb. It. formally means, for episodes = (z)) e' = (z()220 and time ł, u, we have (Cov denotes covariance)
= Vo((s))
(5)
Cover (Re)) = kwt x
(6)
Our intuition behind this modeling is two-fold. First, we Q ~ GP ko to quantify the uncertainly of Q". Second, we consider Rg as an estimator of (*, az), whose variance modeled by another independent GP, ie., Re ~ GP(Q" (Q("((a (z), ki;)); this randomness is of a different kind from the above uncertainty, since R2 still a random variable when fixing a stochastic policy (or stochastic environment), while Q is a deterministic function. The modeling 4 is obtained by combining these GPs (Rip. ko + k1.
Given the modeling for Re, the advantage estimator Ar(e) = Re(e) - Vx((s) follows a centered GP of zt (and then the episode e) with a covariance kernel k and we have the following kernel for F. Similar computations also apply to other modeling such as Re(e) - Va(5t) - -00
(=(o)))).
Proposition 2. Suppose (Ro)ezo (as functions of an episode) sollow GP delermined by 5 and 1 with
ki being bounded. Then, the functiona II (e)) = Ei (Ac(e)) with A,(e) = Re(e) - Vo( follows centered GP with a covariance kernel K given by
K(e,e))= = Σ y 1 (ip
4,220
where (21)120, e' (#()u20 are episodes.
Given F ~ GP(O, K) with we can apply Proposition to see how kernel quadrature works. Indeed, by letting f = and μ N Σm Se, in Proposition 1 for quadrature rule pn = (wi, ei)ver, we have
N
27 )
Egp
Σ
f(e)) Eurfles) NEl
wwe(Mni 11).
N
Thus, provided the heuristic in Section worst-case error) gives us good gradient estimate Σier W;G((e;) while the amount of actual reward com- putations is kept small.
3.2
expect that a good kernel quadrature (i.e., that of small
The actual algorithm looks like Algorithm where we first update by (see Section 3.2.3 and then K as
Remark We could choose to use a parametric mean function m²(=) instead of using Vx(() in for more detailed modeling of R.. However, then f could not represent the sum of advantage estimators and would not necessarily follow a centered GP, so we could not use Proposition | One way to circumvent this issue is using a hybrid gradient estimate described in the following section.
3.2.2 Option 2: modeling with GP
The object modeled by the GP in the previous section is not static over the iterations of the policy gradient in that Rs (and Q") is dependent. on the policy и. So the update of ky might be inaccurate for future policies.
Our second option is thus modeling an static object, the reward function r:
r ~ GP(mp kw),
where ve is a hyperparameter for the mean function my and the positive definite kernel KGs on Z = S x A.
Strictly speaking, we only justify that the sum in F converges in the 12(GP) space, when with infinite horixon; Section A.2 The same applies Proposition g

---

Published in Transactions on Machine Learning Research (02/2024)
However, under the modeling 8 the estimator A, R. - VV((s) does not necessarily follow centered GP because Egp AA(e)] Euze 7" m))())) (st), As already pointed out in Remark we cannot simply
use Proposition
[ to run an episodic kernel quadrature in this case. Instead, we can observe the following
decomposition:
(TT(
Σ 120
(9)
BenT
Σ (R, -Egp(Re))0((a)
Bewt
Σ 120
7 (Egp [R,] b(s)))
(10)
I: centered GP term
II: bias term
where b: R is any (integrable) baseline function and go (24) == e log To(aelst).
This dccomposition can nicely be understood by introducing a fake reward my (instead of r) and the corresponding fake return (e) == Eizo""
Egg Indeed, the term is given just by replacing Rg with R2 the right-hand side of (9) and the remaining term I works as a gradient modification. Let us consider the context of reducing episodes from large batch (e)) a weighted small batch phn (wi; es)ee. Since we can compute the fake swards/returrs without access to r, can estimate the bias term II by using all the episodes (e:)
For the term I, now that the integrands R. EEgp[R RR] are centered, we can apply Proposition 1 with the following representation of the GP for the modified functional Fop = Σ.20 ( Eop(R.).
Proposition 3. Let the reward r follows a GP given by ( with my and Kip being bounded. Then, the functional Fgp(e)= Ei2o ( - Egp (e)))) follows centered GP with covariance kernel K given by
k(e,e)) Σ (1+1)C ()(1 "og
(11)
6020
where (2))200, e' = (E/)a20 are episodes.
Let us now define Fo = E120 ('e - b(st)) as a generalization of f in the previous sections and Fy := E b((e)) (= Pb Fop) be fake counterpart. Similarly to the previous section, given episodes (e)) and p= 2/2 ΣN Se,, we can approximate N EE Po(ee) by Eier w.Fgp(e)) 1/ EN F (es). where An (wires)vel is quadrature rule, given by running kernel quadrature algorithm only regarding the term 1. Note that this approximation is computable only with the reward evaluations over episodes (e:)ier.
By using Proposition [ their mean squared error in terms of GP, which is actually just about the error of N ENn (es( - Eier we (e)( can again be represented as wee(un; K,,,², where K is given by 11 regarding the term I this time.
However, the gradient estimate is not as simple as Eier w.G((s) in the previous section (or the heuristic in Section 3.2 due to the use of a non-centered GP. Under the notation of (10 let define
Gz(e):= = Σ (Re((e) - b(s)))90
120
Ggr(e):= = Σ (Re(ee) - (R)(())))))))))
120
(e) :- Σ (IR b(s)))go(zz,
10
and then G (= Go Ggr) is computable without access to the true rewards. By using these notations, N Go(e,) with a lighter gradient estimate Σier w,Ggg(e)) + ΣN (e.). The overall
we replace n N algorithm is given by Algorithm 3 The way we update GP parameters is described in Section 3.2.3
8

---

Published in Transactions on Machine Learning Research (02/2024)
Algorithm 3 PGKQ with non-centered GP
Input: < N, policy ne, baseline b, GP-mean me and covariance ky modeling GP GP(mp, 1: for ileralion = 1,2....d do
2: Generate episodes (e:),=1 "ii W
3: Compute with [11
4:
I, (wi))er t (K, (e)) with =n
5:
Compute (r(ei))eer and then (Gop(ei))ker
0- 0 + aiEi (.GGg((e) + + Eisl Cg (e:))
7: Update b by using (c:) and mi
8: Update my, ki by using (wi, ei, r(ei)iEE
9:
end for
Remark 2. We propose using b = Vi taken from the base policy gradient algorithm. We could update VG by using the weighted episodes (w;,es)ie like the updates of my (see Section 3.2.3 but we actually propose updating Ve based on fake rewards my, which clearly separates the roles of the policy gradient (PG) and kernel quadrature (KQ), since 6% is the usual PG-estimate given the fake rewards. Indeed, the KQ side then receives episodes (e:) and outputs smaller batch of weighted episodes eike,, while the PG side runs a usual PG with all the episodes (c:) associated with the fake reward me, with a modification of its gradient estimate by adding Eier w;GGp(es).
By following this formulation, we can not only use Go (or Gg)) but also incorporate any gradient estimator (in the PG-side for the fake-reward MDP) in Algorithm 3 See also Appendix B.1 for more implementation details, including how combine PGKQ with PPO.
3.2.3 Updating GP networks
Let us describe how we update kip (and m.) during the iterations of PGKQ.
Updating ky. Although we should ideally update the GP based on the Bayes rule in a purely Bayesian setting, our GPs have to treat either a non-static target function (like Rt) or too many data points since each timostep the MDP is a data point for the GP. Following the maximum likelihood estimation (MLE)
and the deep kernel learning (Wilson al. 2016) framework, set the loss function for lig.
Taker (ka) = y (z, z)y + log (z,
(12)
and conduct the gradient descont over the iterations, where z is given by arranging data points 2; (se, 44) from all the episodes e E (ei)sez, which are after the reduction of kernel quadrature, and y is the corresponding observed values, such as R2 - Vç(s)) in Option 1 and r(z)) (x)) in Option Tn practice, we conduct the gradient descent by splitting (z, y) into minibatches avoid computing the full ky 2).
Updating my for non-centered GPs. We either pass the optimization of my into the deep kernel learning by using the same loss in 12]. or just independently learn my by least-squares as we do in practice. For the least-squares learning of mp, we can make use of the quadrature measure Pn to incorporate the episode weights as
E(+ (m)(=)))))+))))))))))) - 120
I Tmmaan Beowa
(13)
where the discount factor (1 +1+00 t)y comes from
Remark (Computational complexity). Let us consider the overall complexity of the kernel quadra- ture with kernel learning part in PGKQ. In both options, the computation of the single K(e, e') requires -i-times computation of kp (T is the episode length), and so, by combining with the kernel quadra- ture algorithm in Hayakawa al. (022) (where we all the episodes for Nyström's method), takes
9

---

Published in Transactions on Machine Learning Research (02/2024)
Hoopervv
HattCheetah-v4
Wa Walkee2dlvv
reward (opa)
210
2220
rooordd (000)
2044
252 20
itoratinn
Iteration
Iteracion
Iteration
Figure 1: Learning curves in Mujoco tasks: reward vs iteration (comparing {vpg, ppp} (blue), (vpg. pp (orange), {vpg, no (green), and {vpg, ppo}-large (red))
(Nog + 72) + n² log(N/n) to get an n-episode quadrature from an N-episode samplc. If we set the batch size to M in the kernel learning, then the kernel update takes ( in each iteration and has O(nT/M) iterations. Thus, the overall complexity for single iteration of the PGKQ algorithm (for kernel quadrature and kernel learning) is given by (log + T + n²N log(N/n) + nTMM).
4 Numerical experiments
To demonstrate the effectiveness of our proposed methods, we conducted experiments on MuJoCo tasks since they are widely recognized as standard benchmarks in RT, even though the reward calculation for them is lightweight.
We evaluated PGKQ using two options, both designed reduce large batch of N episodes to a smaller batch of episodes. We adopted two conventional PG methods, vpg and PPO, the bases for PGKQ. We also compared these methods without kernel quadrature, using either small or large N batch sizes. Throughout this section, maintained N 64 and 8. Specifically, compared the following algorithmss
{vpg, ppo}: Corresponding algorithm with n episodes per iteration.
{vpg, ppo}-KQ: Corresponding algorithm with PGKQ by modeling (Option 2).
{upg, ppo}-KQ-no-mean Corresponding algorithm with PGKQ by modeling R, (Option 1).
{vpg, ppo}-large: Corresponding algorithm with N episodes per iteration.
In each experiment (either based on vpg or ppo), for each of the four algorithms, we ran a 500-iteration optimization (whose one iteration based on observing n N episodes, depending on the algorithm) ten times for statistical purposes. In all the figures, the empirical average of the total reward E.>0 r(st, az) (which is actually finite sum in all the experiments) with its standard error (shaded region) are shown. All the experiments
conducted by using PyTTorc (Paszke et al. 2019) and Adam (Kingma & Ba 2015
Our motivation for conducting these experiments as follows: How well can we catch up with the learning curve of {vpg, ppo}-large by using PGKQ rather than the small-batch {vpg, ppo} ? So we compared the rewards against iterations this section, but we also compare rewards against steps in Appendix B.2
In all the experiments, we used three-layer fully connected ReLU neural networks (NNs) for each of m and kig, where (z,=²) was computed by passing the NN-embeddings of state-action pairs and z' to the
10

---

Published in Transactions on Machine Learning Research (02/2024)
Gaussian kernel with additional scale and noise parameters. See the end of this section and Appendix B.1 for implementation details.
MuJoCo tasks. We used MuJoCo (v2.1.0, Todorov et al., 2012) with the Gymnasium API Tow- We compared all the aforementioned algorithms in the four following tasks: Hopper-v4, HalfCheetah-v4, Walker2d-v4. In all the tasks, the maximum episode length is 1000, where the tasks except HalfCheetah-v4 can terminate earlier when the state enters predefined unkealthy region.
ers et al. 2023 7
In these tasks, for the vpg (vanilla policy gradient) and ppo (proximal policy optimization with clipping, Schulman et al., 2017), we used the implementation of the machina library. The learning rates of the policy, baseline, and GP-related networks were all set to 3 x 10-4. This value was the default value of the machina library, but also chosen as the most successful learning rate when comparing several different learning rates with the 300-iteration, n-episode ppo in the Hopper-v4 task. The discount rate was y = 0.995.
The results shown in Figure 1 In most tasks and base algorithms {vpg. ppo]; the order of total reward followed {vpg, ppo}-large > {vpg, ppo}-KQ > {vpg, noo--KQ-no-mean > {vpg, ppoł. In particu- lar, in the experiments with vpg, PGKQ methods often get out of the plateau where the vpg is stuck.
Network architecture. For the policy and baseline networks, we used the default implementations of the library (machina). Let us explain the part with our original implementation, my and by in the GP modeling.
(i) Inputs: In all the MuJoCo tasks, we simply concatenated two vectors 8 and a to make 2,, and used it as imputs to our networks. Let denote by D the dimension of in the following.
(ii) Implementation of kØz In both experiments, we implemented hip as follows
( 2/) exp Ap [[f((z)) - fo {o(2)) + (10-5 + exxp(o)))
20
where ||-|| is the Euclidean norm, 5(21, =y) equals 1 if 2 = z and 0 otherwise, Auo Tw I are respectively the scale and noise parameters, and f : RDD + 10 is the embedding function explained the following.
For for in all the MuJoCo tasks, we used fully connected neural network with two hidden layers with D (so RR + RD + R - R' 1)) and the ReLU activations except for the final layer.
(ili) Implementation of my: We used fully connected neural network with two hidden layers with dimensions 200 and 100 (so RD + 2200 + 1000 + R21) and the ReLU activations except for the final layer.
Ablation study. We can use other KQuad functions than that of Hayakawa et al. 2002 2023 Kernel thinning and kernel herding are also tested against the task Hopper-ve in Appendix as a small ablation study.
5 Concluding remarks
We have developed method called PGKQ, which combines existing policy gradient algorithm with kernel quadrature over episodes, aiming at better gradient estimates while keeping the number of actual reward computations small. PGKQ is based on GP models of the return R2 (Option 1) the reward function r (Option 2), whose covariance kernel accumulated over episodes gives a kernel over the space of episodes, which allows the use of kernel quadrature for selecting a small but informative subset from a large batch of episodes.
We also confirmed the competitiveness of PGKQ, especially the Option 2, by performing experiments in MuJoCo tasks.
All the experiments with MuJoCo were conducted with Google Cloud Vertex Al notcbook with NVIDIA T4 (16-core vCPU, 60 GB RAM). hhttps//ggtthub.. http:/tthuubccc/eepeeeee cww/DeepX- inniepXX-nic//chiiaa
11

---

Published in Transactions on Machine Learning Research (02/2024)
As future direction, it should be beneficial to study generalization of PGKQ based vector-valued GPs as in existing RQPG approaches /Ghavanzadeh & Engel 2007 Ghavanzadeh et al.) 2016 Akella et al. 2021) to avoid the heuristic introduced in Section 3.2 for allowing the use of conventional KQ methods.
While we have been focusing on online RL, extending our approach to offine RL represents another inter- esting direction. In such setup, determining which samples from a large batch are essential for reward computation, given the current learning policy, becomes critically important.
References
Masaki Adachi, Satoshi Hayakawa, Martin Jørgensen, Harald Oberhauser, and Michael Osborne. Fast Bayesian inference with batch Bayesian quadrature via kernel recombination. In Advonces in Neural Information. Processing Systems, volume 35, pp. 16533- 2022.
Masaki Adachi, Satoshi Hayakawa, Saad Hamid, Martin Jørgensen, Harald Oberhauser, and Micheal A Osborne. SOBER: Highly parallel Bayesian optimization and bayesian quadrature over discrete and mixed spaces. arxe preprint XxX::2301.11832 2023a.
Masaki Adachi, Satoshi Hayakawa, Xingchen Wan, Martin Jørgensen, Harald Oberhauser, and Michael A Osborne. agnosic batch Bayesian optimization with diverse constraints via Bayesian quadrature. arXiv preprint Xie:2000 05843, 2023b.
Ravi Tej Akella, Kamyar Azizzadenesheli, Mohammad Ghavamzadeh, Animashree Anandkumar, and Yisong Yuc. Deep Bayesian quadrature policy optimization. In Proceedings of the AAAI conference artificial intelligence, volume 35, pp. 6600 6608, 2021.
Francis Bach. On the equivalence between kernel quadrature rules and random feature expansions. The Journal of Machine Learning Research, 18(1):714-751, 2017.
Francis Bach, Simon Lacoste-Julien, and Guillaume Obozinski. On the equivalence between herding and
conditional gradient algorithms. In International Conference on Machine Learning, pp. 1355-1362, 2012.
Olivier Bachem, Mario Lucic, and Andreas Krause. Practical coreset constructions for machine learning. arXiv preprint arXic:1733.06. 76, 2017.
Rémi Bardenet, Adrien Hardy, et al. Monte carlo with determinantal point processes. The Annols of Applied Probability, 30(1):368 417, 2020.
Rémi Bardenet, Subhroshekhar Ghosh, and Meixia Lin. Determinantal point processes based on orthogonal polynomials for sampling minibatches in SGD. In Advances in Neural Information Processing Systems, 2021.
volume 34, pp.
Ayoub Belhadji. An analysis of Ermakov-Zool Zolotukhin quadrature using kernels. In Advances in Newral Information Processing Systems, volume 34, 2021.
Ayoub Belhadji, R. Bardenet, and Pierre Chainais. Kernel quadrature with DPPs. In Advances in Neural Information Processing Systems, volume 32, pp. 12907 12917, 2019.
Ayoub Belhadji, Rémi Bardenet, and Pierre Chainais. Kernel interpolation with continuous volume sampling, In International Conference on Machine Learning, pp. 725-735. PMLR, 2020.
François- Trançois-Xavior Xavior Briol, Chris J Oates, Jon Cockayne, Wilson Ye Chen, and Mark Girolami. On the sampling problem for kernel quadrature. In International Conference on Machine Learning, pp. 586-595. PMLR, 2017.
Souradip Chakraborty, Amrit Singh Bedi, Pratap Tokekar, Alec Koppel, Brian Sadler, Furong Huang, and Dinesh Manocha. Posterior coresct construction with kernelized stein discrepancy for model-based re- inforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 6980-66 6988, 2023.
12

---

Published in Transactions on Machine Learning Research (02/2024)
Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. In Conference on Uncer- lainly in Arlificial Intelligence, pp. 109-116, 2010.
Francesco Cosentino, Harald Oberhauser, and Alessandro Abate. Carathéodory sampling for stochastic gradient descent. arXiv preprint arXi::2006.0188 018199, 2020.
Josef Dick, Frances Y. Kuo, and lan H. Sloan. figh-dimensional integration: The quasi- Carlo way. Acta Numerica, 22:133 288, 2013.
Raaz Dwivedi and Lester Mackey, Kernel thinning. In Proceedings of Thirty Fourth Conference on Learning Theory, volume 134, pp. 1753. PMLR, 2021.
Raaz Dwivedi and Lester Mackey. Generalized kernel thinning. In International Conference on Learning Representations, 2022.
Ethan N Epperly and Elvira Moreno. Kernel quadrature with randomly pivoted cholesky. In Aduances in Newral Information Processing Systems, volume 36, 2023.
Dixia Fan, Liu Yang, Zhicheng Wang, Michael S. Triantafyllou, and George Em Karniadakis. Reinforcement lcarning for bluff body active flow control in experiments and simulations. Proceedings of the National Academy of Sciences of the United Stales of America, 117(42), 2020.
Mohammad Ghavamzadeh and Yaakov Engel. Bayesian actor-critic algorithms. In Proceedings of the 24th International Conference on Machine Learning, pp. 297-304, 2007.
Mohammad Ghavanzadeh, Yaakov Engel, and Michal Valko. Bayesian policy gradient and actor-critic algorithms. The Journal of Machine Learning Research, 17(1):2319 2371,, 2016.
Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 139-144, 2020.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. kernel method for the pwo-sample-problem In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006.
Vijaykumar Gullapalli. A stochastic reinforcement learning algorithm for learning real-valued functions. Newral Networks, 366):671, 1990.
Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. STAM review, 53(2):21772))) 288, 2011.
Satoshi Hayakawa, Harald Oberhauser, and Terry Lyons. Positively weighted kernel quadrature via subsam- pling. Tn Advances in Neural Information. Processing Systems, volume 35, pp.
2022.
Satoshi Hayakawa, Harald Oberhauser, and Terry Lyons. camppling-based Nyström approximation and kernel quadrature. In Proceedings of the 40th International Conference on Machine Learning, pp. 12678-1681 126999, 2023.
Ferene Huszár and David Duvenaud. weeihtely-weighted herding is Bayesian quadrature. In Conference Uncertainty in Artificial Intelligence, pp. 377-386, 2012.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), Proceedings of the 3rd International Conference on Learning Representations, 2015.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Car- bunc, and Abhinav Rastogi. RLAIF: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint ar arXim:2309000265 2023.
Christian Litterer and Terry Lyons High order recombination and an application to cubature on Wiener space. The Annals of Applied Probabilily, 22(41:13011133 2012.
13

---

Published in Transactions on Machine Learning Research (02/2024)
Zimo Liu, Jingya Wang, Shaogang Gong, Huchuan Lu, and Dacheng Tao. Deep reinforcement active learning for human-in-the-loop person re-identification. In Proceedings of the TEEE/CVP International Conference on Computer Vision (ICCV), 2019.
Nicholas Metropolis and Stanislaw Ulam. The Monte Carlo method. Journal of the American Statistical Association, 44(247):335- 1949.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Pro- ceedings of The 33rd International Conference on Machine Learning, volume 48, pp. 1928 1937. PMT.R, 2016.
Anthony O'Hagan. Bayes Hermite quadrature. Journal of Statistical Planning and Inference, 29(3):245 260, 1991.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, J. Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, P. Welinder, P. Christiano, J. Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pp. 27730 27744, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurth, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Newral Information Processing Syslems, volume 32, 2019.
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Newral Nelworks, 21(4):682 697, 2008
Pankaj Rajak, Aravind Krishnamoorthy, Ankit Mishra, Rajiv Kalia, Aiichiro Nakano, and Priya Vashishta. Autonomous reinforcement learning agent for chemical vapor deposition synthesis of quantum materials. vpj Compulational Materials, 7(1), 2021.
Herbert Robbins and Sutton Monro. A stochastic approximation method. Annals of Mathematical Statistics, 22(3):400-401 1951.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimiza- tion algorithms. arXiv preprint arXiv: 0ao:1777.00 707.06 2017.
Abhishek Shetty, Raaz Dwivedi, and Lester Mackey. Distribution compression in ncar-linear time. In International Conference on Learning Representations, 2022.
Samarth Sinha, Han Zhang, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, and Augustus Odena. Small- GAN: Speeding up GAN training using core-sets. In Proceedings of the 37th International Conference Machine Learning, pp. 9005 9015. PMLR, 2020.
Bharath K Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Schölkopf, and Gert RG Lanckriet. Hilbort space embeddings and metrics on probability measures. The Journal of Machine Learning Research, 11:1517-1561, 2010.
Arthur H Stroud. Approzimate calculation of multiple integrals. Prentice-Han 1971
Richard S. Sutton and Andrew G. Barto. Reinforcement LeannnenennnntntLeneee An Introduction. The MIT Press, second edition, 2018. URL http://ccmmpeeeiieaeaaeeeee/////
html
14

---

Published in Transactions on Machine Learning Research (02/2024)
Maria Tchernychova. Carathéodory cubature measures. PhD thesis, University of Oxford, 2016.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012
TEEF/RSJ international conference on intelligend robols and syslems, pp. 5026 5033. IEEE, 2012.
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL
https://zzenee wwwpooo/or////////////
Kazuma Tsuji, Ken ren'ic Tanaka, and Sebastian Pokutta. Pairwise conditional gradients without swap steps and sparser kernel herding. In International Conference on Machine Learning, pp. 11864-21882 PMLR, 2022.
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:8111.10959, 2018.
Xiaoqiang Wang, Yali Du, Shengyu Zhu, Liangjun Ke, Zhitang Chen, Jianye Hao, and Jun Wang. Ordering- based cansal discovery with reinforcement learning. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 3566 3573. International Joint Conferences on Artifi- cial Intelligence Organization, 2021.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229-256 1992.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. Arthur Gretton and Christian C. Robert (eds.), Proceedings of the 19th International Conference on Artificial Intelligence and Statisties, volume 51, pp. 370-378, PMLR, 09-11 May 2016.
Yuan Xia, Jingbo Zhou, Zhenhui Shi, Chao Lu, and Haifeng Huang. Generative adversarial regularized mutual information policy gradient framework for automatic diagnosis. In AAAI Conference on Artificial Intelligence, pp. 1062 1069-1069, 2020.
Cheng Zhang, Hedvig Kjellström, and Stephan Mandi. Determinantal point processes for mini-batch diver- sification. In Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence, 2017.
Cheng Zhang, Cengiz Öxtireli, Stephan Mandt, and Giampiero Salvi. Active mini-batch sampling using repulsive point processes. In Proceedings of the AAAI conference on Artificial Intelligence, volume 33, pp. 5741-5748, 2019.
Barret Zoph and QuocQuoe Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017.
A Proofs
A.1 Proof of Proposition
Proof. Suppose GP(O, K). We first compute the following term:
Egplu(f))) Egp J] f())(())
( (p(e)
By Canchy-Schwarz, we have ap( - I f(x)² du(r), so we have
f(v) dn(s))
Egp an()) s Egp
[ ((0))a((e) = HOP [f(x)?] du(x)) = / K(c,z) < oc,
15

---

Published in Transactions on Machine Learning Research (02/2024)
where we have used Fubini's theorem (thanks to the nonnegativity of the integrand) in the first equality, and the assumption on the kernel in the last inequality. Thus, we can use Fubini's theorem for the original multiple integral, and we have, by using (epl(())f()) (z,,),
(spif())((V) -ffor/(te))) dp(x) du(y)
II K(x, v) du(a)
Egp [a([f) =
From the same use Fubini's theorem by (partially) replacing μ with Jrn we can compute the other quadratic terms and obtain
Egp (()](( (1))(())))((())) 1f ((z:))
Σ ws ((F duly)
12
w,wy (z45
s,=1
which is well-known formula for the wee(Un; K, 1)? Gretton et al. 2006
Sriperumbudur et al. 2010
In general, we can also treat the non-centered case ~ GP(m, K) if m is integrable with respect to Indeed, we can apply the above computation to f - m ~ GP(O, K) to obtain
Epp [[nn(s mm) wLf - ( = wee(Mn; K, k,
From the linearity of the integral, we actually have
Egp (M m) p(f m
= Egp ((H4n()) - m(f)) (pn(m) (=
= Egp (1) - -(())}] - 2((m((m) (mm)) Egplux()) - +(un(m) m(m))²
m(s]
-p)(m))u(mm)
Egg [(u((f) (()))T] - (mm(m) m(m))²
Thus, in general, have
Egp (())()) - ((())a = woolfun K, (Mn =(m))²²
A.2 Proof of Proposition
2
Proof. From the modeling
5 and 6 A, formally follows a GP on the variables (e, E x T such that
Egp A((e) = 0, Egp (r(e)Au(ee)] = (z, 2!),
where & is the space of episodes and T = 10, is the space of time indices.
Let us define the space LGPGP) as the space of valued centered Gaussian variables given by taking the com- pletion of the linear space span(Ar(e) (e,t) E E x T} with respect to the norm |||] := Egp [x][ 1/2 Then, since the kernel is bounded, we have Ei (e))) A infinite sum F(e) = E' A((e) is well-defined in (GP).. By letting (r = Eizo 7 Ac(e), we have Fr F in (GP).
E >'k(=,, < and so the
Therefore, we have that (f(e) | EJ family centered (jointly) Gaussian variables with
T
Esp [((e)((e)]
lim
lim Egp [T(e)fo(ee)) = lim
lim
t+"E Ac(e)An())))
T-x0 U-+
T
U-00c
f=0 ==0
lim
lim
++ * (xe, 2n);
(14)
(=0 w=0
where e = (z))20, (z/)u2 are episodes. Since the kernel is bounded, the sum En Eluso + u 24,) is absolutely convergent, and coincides with the right-hand side of (14), which completes the proof. -
16

---

Published in Transactions on Machine Learning Research (02/2024)
A.3 Proof of Proposition
Proof. The flow of the proof is mostly the samne the previous one. The base GP is now GP ki). This time we define LgP) as the space of R-valued Gaussian variable given by the completion of the linear space span([]} U (x(2) | E 2}) with respect to the norm |||] = Egp [x Since my and by are bounded, we first have that Re(e) Enzi 7" (r(z) is well-defined in (gP as
Σ 02t
"ra""ll = End't Vm(zz) + ka zu) < oc.
a2t
Note that (GP) is Hilbert space containing 1 with the inner product (X, X') LP(GP) = Egp Sxp(XX"], Thus, the expectation, which is the inner product with 1, is continuous with respect to the norm, and so have
[R = lim
Egp
Σ (r(u)r u2t
= lim T-
Σn" mmp(a) = En"-mmo(zu))
T-500
a2t
where the last infinite sum is absolutely convergent thanks to the boundedness of mg- We can also prove that (e) [Rc(]] = Enzi 7" (r(2x) mz()a)) by combining two convergent sequences.
By letting (c) == Re(e) Egp Rd (c)], we have family of centered jointly Gaussian variables (A (e) (e,t) (x J such that
| (e)lle S ((nr"((r(zx)) m $ Σnn hip( 2n <C CC
u2t
u21
for a constant C > 0 independent of t and e, where & and T are the spaces of episodes and time indices as defined in the previous proof. Thus, Pop(e) Eizo A' (c is well-defined in L22(PP) and (Psp(e)
E E} is family of centered jointly Gaussian variables. Since the double sum En 7 E "aaa:ll""(rrr (r((())( (x()))) is absolutely convergent, we can exchange the sum as
(ep(e)
(r(z(
(r(zu) - mz))))) ΣΣ (r((z)) (z))
Σ +1++ (1+u))((r(z)) mm((a))
u20
Therefore, the covariance kernel for PGp(e) can formally be computed as
Epp (Gp(())(opee) = Σ (1+ ()(1 (ergteErr(aaa - (m -mp
1uu20
Σ
(1 + + +) (k,
f,u20
for episodes e = (21)220, el (z 122, which justified by the same logic as the previous proof.
B Experimental details
B.1 Combining policy gradient with kernel quadrature
Recall that we have introduced in 2 the single-episode gradient estimate
G(e) = Σ 7
(15)
120
with advantage estimator dg. Although have written that use N EN G(es) as the Monte Carlo gradient estimate, what we write in the actual code is the computation of the (one-dimensional) loss
(Loss))
1
Σ Typp [A,((es), where Typs (Ao((e) = Σ (A((e) log (o(ae)))):
N
200
17

---

Published in Transactions on Machine Learning Research (02/2024)
and running an automatic diffcrentiation to get its gradient with respect to the parameter A. Here, A is treated as just functional of an episode in the definition of [rp] Let us explain how we actually compute the policy loss in the PGKQ given kernel quadrature rule μn = (wi eiver. We start from VPG (vanilla policy gradient).
(v1) VPG with kernel quadrature with a centered GP (Egp[A.) = 0, Option 1) is the easiest case. Given Ma: we just replace N Ex= Lygg (A](ee) with Eier Wy (es((s) in loss computation.
(v2) When we combine VPG and a non-centered GP modeling of r (Option 2), we just exploit the decomposition (10) for loss computation (with baseline function b):
(Loss)=
Σ (es)
Σ Turrees;
N
SEI
Lwi(e) = Eizo [ru(e) Eizo
(e log no (a(a))))) (Egr - [R(( (( log no (arls)))
where
As the baseline, use the value estimator Vi trained with fake rewards as explained in Remark 2 The observation Lv1 = LN Egp [Ro]] and Lv11 = LEg [R.] b) allows a unified implemen- tation.
We can also apply PGKQ to PPO (Shulman et al. 2017). In the usual PPO, we use the probability ratio (as a functional of an episode) (e) == To (4g] st) Tatia (ar (st,, where Oold is the policy parameter at which we assume the episodes c1...., eN have been drawn. Given an advantage estimator A, as a functional of an episode, we compute the loss as follows:
1 (Loss)= N
Σ Topol (Ae(ee), where Lppa [ppo(Ai((e)= = Σ min (e) e, (())))
Here, clip(a, b, c) = min b), and (= 0.2 the implementation) is a clipping parameter.
(pl) PPO with the centered GP modeling (Option is given by a straightforward replacement of Typg by Lppo
(p2) When combining PPO with the Option 2, also imitate the decomposition of (v2):
1 N Σ Le)(es), where LpI == Lppo RRe Egp [Rl], 4pll Eppo[Egr [R(-
(Loss) EwILpiee)t
NET
Here, b is again the value estimator Vo trained by fake rewards.
We can also consider the use of other advantage estimators than R₁ - Indeed, Ai(e) = (e) - (R,, - (s))) with ty being the final time was adopted in the MuJoCo tasks (also following the original code of machina). The modification of (v1) and (pl) straightforward as they formally do not depend on the specific form of Å, For (v2) and (p2), though there are other possibilities, we just replaced Lvl and LpIt with Li LvpalEGp| [A]]] and 4n == Lppo [Epp[A] where Egp[A[]()) Egp[Rc(e)] - (RGr RRo (e)) Vi (8).)) is regarded as functional of an episode.
-
B.2 Reward vs step
Though we only compared the rewards against iterations in the main body, since it is common to compare the rewards against observed steps (c.g., Schulman al. 2017), we here present our experimental results that regard. Figure 2 is on each of the four MuJoCo tasks corresponding Figure Note that this is not necessarily fair comparison to ppo in terms of the learning rate per step.
18

---

Published in Transactions on Machine Learning Research (02/2024)
Hopper-4d
Hal Chhheethhhr
Walker 2ch-o
Fohaard Dengg
Velao 3000
Sialo
335
1540
1330
E00000 (0010)
635
600
140
20g
240
stop
stop
st0p
Figure 2: Learning curves in Mujoco Tasks: reward vs step
C
Ablation study
To see how the performance is dependent on the choice of concrete kernel quadrature algorithms, we addi- tionally ran the PGKQ algorithms with the kernel thinning and kernel herding (explained below), instead of the one from Hayakawa et al. 2022 2023). The experiments were all run on the Hopper-v4 task, and all the experimental details other than algorithms are the same in Section except that we ran each algorithm five times instead of ten.
We compared the following algorithms:
{vpg, ppo)-KQ-(thin, herd}: PGKQ given by combining the corresponding base policy gradient (PG) and kernel quadrature (KQ) algorithms, with a GP modeling of (Option 2, Algorithm 3
{vpg, ppo)-KQ-(thin, herd)-no-mean: PGKQ given by combining the corresponding base PG and KQ algorithms, with a GP modeling RG (Option 1, Algorithm 2
We can implement these algorithms by simply using thimning herding the KQuad function in Algorithm 2 AlgorithmB We shall brielly explain these algorithms below. In our setting, both algorithms just require set En of N points (episodes) that makes an empirical measure and positive definite kernel K over the set, and then returns an equally weighted set of n points (N = 64 and n = 8 in our experiments).
Kernel thimning. The option thin corresponds to kernel thinning Dwivedi & Mackey 2021 2022 with algorithmic acceleration based on (Shetty et al. (2022). This is a stochastic algorithm that runs in O(N log N) computational steps for approximating an N-point empirical measure (given by Ex) by an n- point equally weighted measure (with VN). We do not describe the whole algorithm here; it is given by recursive applications of clever randomized algorithm that divides the given set into two balanced subsets, which they call kernel halving.
Kernel herding. The option herd corresponds to kernel herding (Chen et al. 2010 Bach et al. 2012 This is greedy scquential algorithm such that, at the t-th itcration, we select the point z; & such that wee(==) (-] + K, μ) is minimized over the choice of x;, where & is the set of candidate points (episodes), & is the zero measure, K a given positive definite kernel over E, and μ is the target measure, which we want to approximate. Since we have no access the true mean embodding in our sctting, we let = EN and p: 1 IN de. be the uniform measure over Ex (- The minimization in each step done by exhaustive search over En, and the algorithm runs in ((N2) computational steps in our implementation.
19

---

Published in Transactions on Machine Learning Research (02/2024)
Hopper-v4
VCp R0p-R0-
K19 RQ hord
Vig RQ herd
<28 tht--1
503
V2g tntive
*29 1000--nnn-oooomee
VPg 1o0---r-mmmm-mmmea
reward (vpg)
602
603
400
200
01
1.0
200
300
400
502
105000 201000 300000 cabess scc00s 620035 750020-800310 aco050
Spe-ktQ-thin
spo
2003
350-K02-kandssc-mmaa
reward (ppo)
15.0
1500
500
200
2000
300
50c
0.00
0.25
0.50
0.75
1.00
1.25
3.50
1.75
2:0
iteration
step
Figure Learning curves of PCKQ variants Hopper-v4: reward vs iteration and step
The results are given in Figure By comparing it with Figure D and Figure 2 we can see that the thinning- based methods perform well; they give similar results to the PGKQ algorithms presented in the main body of the paper. On the contrary, the herding-based methods perform worse than those based the other two KQ algorithmss, while {vpg, ppo]-XQ-heee -erd certainly improve upon the small-batch {vpg, ppo}. Thus, we propose to use the original PGKQ the main body or the thinning-based variant as the first choice. However, note that this is a small ablation study based on a particular task and requires further investigation to determine which KQ algorithm should be adopted in PGKQ.
20


Reviews:
[{"review_id": "IRHttOaZ1B", "reviewer": "Reviewer_FRMS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea proposed is a good fit for settings in which ground truth reward evaluation is expensive. I would say the main weakness is that the paper is benchmarked using online RL algorithms. In which case, we are still paying the cost of collecting online experience, and only saving on reward evaluation. In most real-world RL settings I believe the online experience generation dominates runtime compared to reward evaluation (although I'm sure you can construct counterexamples, such counterexamples are rare.)\n\nThe authors do acknowledge this in the concluding remarks, and I don't think it is a strike against the technical accuracy of the paper, it's more a strike against the potential impact. This is less of a factor in TMLR reviewing though, aside from that I think the paper is acceptable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors assume a problem setting where we wish to do RL, and getting examples is easy, but evaluating reward is more difficult or computationally expensive.\n\nThe key idea of the paper is to leverage ideas from Kernel Quadrature literature. At a high level, the problem Kernel quadrature tries to solve is to find a discrete set of weights and points $w_i,x_i$ such that the empirical estimate of $E[f] \\approx \\sum w_i f(x_i)$ is close to the true expectation $E_\\mu[f]$ where $\\mu$ is the true probability distribution.\n\nUsing an off-the-shelf method for this, we can take a full batch of $N$ episodes, reducing it to a smaller weighted set of $n$ episodes, and only compute the gradient update over those $n$ episodes.\n\nFor the kernel, this paper proposes using Gaussian Processes. We assume we use an advantage based policy gradient method where $V$ is learned, $R_t = \\sum_{t'=t}^\\infty r_{t'}$ is summed over the episode, and $A = R_t - V$. Two methods are compared, using a GP to estimate $R_t$ directly and estimating $r_t$ individually to sum together later\n\nResults wise, using the GP to estimate $r_t$ performs better empirically. They compare three approaches: using $N$ episodes per batch, using $n$ episodes per batch, and the weighted batch of $n$ filtered from a full $N$ episode batch using their method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N/A", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kEGykZySPw", "reviewer": "Reviewer_ushZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I should preface this by noting that I am not particularly familiar with the Kernel Quadrature literature, so that will affect my judgments\n\nStrengths:\n- The motivation is quite clear, in particular it seems quite intuitive to use these tools for better estimation on a budget\n- The review of related literature in Section 2 is quite nice, especially as I am unfamiliar with the subfield\n- The empirical results in Figure 1 seem to largely fulfill the promise of the introduction\n\nWeaknesses:\n- The description of the method is sometimes a bit hard to follow, in particular:\n  - I believe all details of how to sample transitions and the choice of weights (KQuad), which are crucial to implementing the method, are deferred to an entirely different manuscript (Hayakawa et al. 2022), which is itself quite difficult to follow given the description in that Algorithm 1 is \"Apply a recombination algorithm,\" which links to further citations. This is particular strange given the entire point of the paper is to provide a subsampling method.\n   - the use of Assumption A in its informal fashion is confusing to me, I would prefer to see a definition of \"uncertainty\" or for this to be omitted entirely\n  - the definition of k_\\phi is deferred to the Appendix, I would prefer it introduced in the main text for concreteness\n- I don't think the empirical evaluation is sufficient to justify the claim in the conclusion that the paper \"confirmed the competitiveness of PGKQ\":\n  - besides the vanilla algorithm with different batch sizes, there is not other sample-efficient policy-gradient method tested, I am not the most familiar with this area, but either there is no prior work that fits (which I doubt given the description in Section 2.2), or there is missing justification as to why it does not solve the proposed problem in Section 1\n  - Section 1 describes a much different problem setting (expensive reward functions) than the experiment domain. In particular, preference learning in offline RL with a small language model seems like a perfect application for the method so I am confused why it was not chosen", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes to take tools from Kernel Quadrature (approximation of integrals via kernels) and apply them to policy gradient methods, which allows for better estimation of policy gradients with smaller batch sizes. This has application to settings where reward labeling is expensive or otherwise difficult. Some theory is developed in the specific policy gradient setting, and there is a small empirical justification of the proposed method, with good results.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the Weaknesses, in particular the following seem essential:\n- a short description of KQuad in the text (or please point out to me if I have missed this)\n- another baseline\n- another domain for which the reward is expensive to evaluate (any of the ones in Section 1 would be fine with me)", "broader_impact_concerns": "I don't have any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "G72ag5hTE6", "reviewer": "Reviewer_LoRv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem of the paper is very relevant and the formulation is simple and clear. The paper is well-written and targeted and the contributions are very clear. Most importatnly, the proposed method is empirically shown to be effective in well-thought experiments that show that the method is competitive with the vanilla method that uses N reward evaluations of episodes while only using $n <N$ episodes at each iteration. \n\nI do not have major weaknesses to point out. A general comment would be to simplify the exposition in the introduction, and not assume the reader has knowledge of some terms and notions (see requested changes below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the problem of high-costs of reward evaluation of episodes that are required in policy gradient methods in reinforcement learning. The paper proposes a method for selecting a subset of n episodes of a batch of N episodes for computing the gradients and the corresponding weighted update rule. The method is shown effective empirically.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My recommendations are minor. They are not critical for my recommendation for the outcome of the submission. \n\nMy recommendation is that the authors clarify, in the document, the questions and comments I have at the moment:\n- What is $r_t$ in the background? \n- How can $R_t$ be defined without depending on the episode, in the background? \n- Why is the definition of $R_t$ important if $R_t(e)$ is already defined, and how do they differ?\n- Coud the method be used with stochastic rewards? What would the authors expect from the performance?\n- When \"kernel\" is first mentioned (1.2), it is not defined nor given intuition. It would be beneficial to have an intuition of what the kernel might be, as it is just defined to be a function of pairs of episodes. Does it measure similarity between episodes?\n- \"Positive definite kernel\" is not defined and appears very early in the document (2.2);\n- What is the \"empirical measure of episodes\" used, intuitively?\n- What are the $\\delta_{e_i}$, as they do not seem to be defined when they appear (2.2)?\n- What is the \"reproducing kernel Hilbert pace\"?", "broader_impact_concerns": "I do not foresee any ethical concerns for the submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers praised the paper for the relevance of the considered problem and the originality of the solution, which is especially valuable in a field (policy optimization) where countless techniques to improve the overall efficiency have been tried already. I agree with the reviewers that this is a paper worth publishing and I am asking the authors to implement a couple (hopefully easy) changes:
- I also found the Assumption A a little odd (as Reviewer ushZ did). I think this is not really an assumption, but rather a condition under which we can expect the method to provide benefits. I would not present it with a formal statement but rather with a few lines of text;
- The manuscript says more than once that any method for KQ can be used, but then only experiments with the one by Hayakawa et al. (2022). I think that adding to the appendix a small ablation with an alternative KQ method would go a long way in clarifying that the reported performance benefit is not tied to this particular implementation.
Decision: Accept (poster)