Local policy search with Bayesian optimization
Authors: Sarah Müller
Alexander von Rohr
Sebastian Trimpe
Venue: NeurIPS 2021 Poster (2021)
Keywords: bayesian optimization
active learning
reinforcement learning
policy search

Abstract: Reinforcement learning (RL) aims to find an optimal policy by interaction with an environment. Consequently, learning complex behavior requires a vast number of samples, which can be prohibitive in practice. Nevertheless, instead of systematically reasoning and actively choosing informative samples, policy gradients for local search are often obtained from random perturbations. These random samples yield high variance estimates and hence are sub-optimal in terms of sample complexity. Actively selecting informative samples is at the core of Bayesian optimization, which constructs a probabilistic surrogate of the objective from past samples to reason about informative subsequent ones. In this paper, we propose to join both worlds. We develop an algorithm utilizing a probabilistic model of the objective function and its gradient. Based on the model, the algorithm decides where to query a noisy zeroth-order oracle to improve the gradient estimates. The resulting algorithm is a novel type of policy search method, which we compare to existing black-box algorithms. The comparison reveals improved sample complexity and reduced variance in extensive empirical evaluations on synthetic objectives. Further, we highlight the benefits of active sampling on popular RL benchmarks.

Local policy search with Bayesian optimization
Sarah Müller 1,4
Alexander von R,hr*1,2.3 Sebastian Trimpe 1.2
Max Planck Institute for Intelligent Systems. Stuttgart, Germany Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Germany
"AV GmbH, Gifhom, Germany
Institute for Ophthalmic Research, University of Tübingen, Tübingen, Germany sar. wewellee@euiitttbinneeee {vonrohr, trimpe)@dame wwwh-aachen..de.. de
Abstract
Reinforcement learning (RL) aims to find optimal policy by interaction with an environment. Consequently, learning complex behavior requires a vast number of samples, which can be prohibitive in practice. Nevertheless, instead of systemat- ically reasoning and actively choosing informative samples, policy gradients for local search are often obtained from random perturbations. These random sam- ples yicld high variance estimates and hence are sub-optimal in terms of sample complexity. Actively selecting informative samples is at the core of Bayesian optimization, which constructs a probabilistic surrogate of the objective from past samples to reason about informative subsequent ones. In this paper, propose to join both worlds. We develop an algorithm utilizing a probabilistic model the objective function and its gradient. Based the model, the algorithm decides where to query a noisy zeroth-order oracle to improve the gradient estimates. The resulting algorithm is novel type of policy search method, which compare existing black-box algorithms. The comparison reveals improved sample complex- ity and reduced variance in extensive empirical evaluations synthetic objectives. Further, we highlight the benefits of active sampling on popular RL benchmarks.
Introduction
Reinforcement learning (RL) a notoriously data-hungry machine learning problem, where state-of- the-art methods easily require tens of thousands of data points to learn given task [1]. For every data point, the agent has to carry out potentially complex interactions with its environment, either in simulation in the physical world. This expensive data collection motivates the development of sample-efficient algorithms. Herein, we consider policy search problems, a type of RL technique where directly optimize the parameters of policy with respect to the cumulative reward over finite episode. The collected data is utilized to estimate the direction of local policy improvement, enabling the use of powerful optimization techniques such stochastic gradient descent. Policy gradient methods (e.g.. [2-5]) usually rely on random perturbations for data generation, e.g., in the form of exploration noise in the action space or stochastic policies, and do reason about uncertainty in their gradient estimation. However, innate in the RL setting is the ability to actively generate data, allowing the agent to decide on informative queries, thereby potentially reducing the amount of data needed to find a (local) optimum. Active sampling has the potential to allow those algorithms to improve sample complexity, reducing the number of environment interactions.
'Equal contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In contrast to random sampling, Bayesian op- timization (BO) [6] is a paradigm optimize expensive-to-evaluate and noisy functions in sample-efficient manner.
5
J(0)
the core of BO is the question of how to query the objective function efficiently to maximize the information contained in each sample. By building a prob- abilistic model of the objective using past data and, critically, prior knowledge, the algorithm can reason about how to query a noisy oracle to solve the optimization task. Since RL can be framed as a black-box optimization problem, we BO learn policies in sample-efficient way. However, even though BO has been used to tackle RI,, these approaches are often restricted to low-dimensional problems. One reason is that BO aims find global optimum; hence, with- out further assumptions, BO algorithms need to model and search the entire domain, which needs lot of data and gets exponentially more difficult as the dimensionality increases. Additionally, the amount data grows so does the computational complexity of prohabilistic models, which becomes a significant problem. However, the success of RL algorithms using policy gradient methods indicates that for many problems sufficient to find a locally optimal policy.
V
220
-155
-1.
-0.5
0.0
0.5
1.0
1.3
2.0
Figure 1: Estimation of a Jacobian GP model (bottom) of a 1-dimensional objective function (top). The model has function observations (black crosses), but is able form a posterior belief over the gradient. Uncertainty for the Jacobian model is reduced between samples. An active sample strategy can improve gradient estimates.
Our proposed algorithm combines the strength of gradient-based policy optimization with active sampling of policy parameters using BO. We thereby improve the computational complexity of BO methods the one hand, and the sample-inefficiency of gradient-based methods on the other hand, especially when proper prior knowledge is available. We achieve these improvements by explicitly learning probabilistic model of the objective in the form Gaussian process (GP) [7]. From this model, we can jointly infer the objective and its gradients with a tractable probabilistic posterior. The resulting Jacobian estimate includes all data points, rendering data usage more efficient. Further, the algorithm infers informative queries from the uncertainty of the probabilistic model to improve the estimate of the local gradient. While in this paper we adapt the setting of Mania al. [1] and assume access to zeroth-order information only, the algorithm extends straightforwardly to policy gradient algorithms where additional first-order information is available. summary, the contribution of this paper is local BO-like optimizer called Gradient Information with BO (GIBO). The queries of GIBO are chosen optimally to minimize uncertainty about Jacobian. GIBO uses local GP model for active sampling and gradient estimation and can be used with existing policy search algorithms. Using only zeroth order information, GIBO able to
significantly improves sample complexity in extensive within-model comparisons, i.e., when
accurate prior knowledge available: is able to solve RL benchmark problems in sample efficient manner, and reduces variance the rewards when compared to non-active sampling baselines.
Preliminaries
This work presents a local optimizer with active sampling. The objective function and its derivative's joint distribution are modeled using GP. Since we have developed the optimizer with the RL application mind, we also introduce the RL problem. For the sake of brevity, we refer the reader to [7] and [8] for an introduction into GPs and BO, respectively.
2.1 Problem setting
In the following we phrase policy search as a black-box optimization problem. For parameterized policy To θ + A that maps states s E and the statie policy parameters E to actions E A. we use the same performance measure as in policy gradient methods for the episodic case.

---

Hence, the objective function J V:R: RR - R is defined as
J(0) = Bxg Σ
Σn
where IET, is the expectation under policy Te, is the reward at time step i, and I the length of episode, A BO query is equivalent to the return of one rollout following the policy Ti in the environment. The expected episodic reward is entirely determined by choice of policy parameters (and the initial conditions). Thus, the optimizer explores the reward function in the parameter space rather than in the action space. Since initial conditions might vary and the environment can be non-deterministic. reward evaluations are noisy.
Policy search herein is abstracted as zeroth-order optimization problem of the form
0r arg max (0),
(1)
dge
where 0 is the variable and @ C Rd bounded To solve (1), an optimization algorithm can query an oracle for a noisy function evaluation J(0) w. We assume an i.id. noise variable w E IR to follow normal distribution ~ (0, o²) with variance o². We do not assume access gradient information or other higher-order oracles for conciseness. Albeit, GIBO requires that the following critical assumption is fulfilled:
Assumption 1. The objective function J is sample from known GP prior J (m) k(O 6)), where the mean function at least once differentiable and the covariance function ls is at least twice differentiable, w.r.t. 0.
This is the standard setting for BO with the addition that the mean and kernel need to be differentiable, which is satisfied by some of the most common kernels such as the squared exponential (SE) kernel. In the empirical section, we investigate the performance of the developed algorithm with and without Assumption holding true.
2.2 Jacobian GP model
Since GPs are closed under linear operations, the derivative of a GP is again a GP [7]. This enables us derive an analytical distribution for the objective's 's Jacobian, which we can use as proxy for gradient estimates and enable gradient- based optimization.
Following Rasmussen and Williams [7], the joint distribution between GP and its derivative the point 0, is
1 ~ ~N -ovn((11 ((-M)AX)))) ] Vo
X(XXX)+ + 02I K( Vo. K(0,,
Vg (B.,
(2)
where y are the n zeroth-order observations, X C are the locations of these observations = [0....., On]. and K the covariance matrix given by the kernel function k θ x - IR. The posterior can be derived by conditioning the joint Gaussian prior distribution on the observation [7]
Jo x. ~ NW', E.)
P._ V.. ((.)) + V.. K(O,, X)
(XX (g mXX)) E Rd
ERC
ERexn
ERux"
E
Σ, V2 K(0,, 0.)
V Vo, K(,,, X)
(K(,X) +
Vo. K(X,0.) E Rdxd
ERIxx
ERAX.
ERaxn
ERuxa
Remark 1. Note that the term (K(X + o2 021)-1 with the highest computational cost (O(N3)
the same term that is used to compute the posterior over J. Therefore, calculating the Jacobian does not add to the computational complexity once a GP posterior has been computed.
Any twice differentiable kernel is sufficient for the presented framework, but we assume SE kernel for the remainder of the paper. For the derivatives of the SE kernel function see Appendix A. For visual example of function- and the Jacobian Jacobian-posterior, refer to Fig. 1. The figure indicates that

---

a zeroth-order oracle is enough to form a reasonable belief over the function's 's gradient. Moreover, Fig. 1 shows that the uncertainty about the Jacobian gets reduced between query points more so than at the query points themselves. To minimize uncertainty about the Jacobian at specific point, it intuitively makes sense to space out query points in its immediate surrounding. Herein, we formalize this intuition and formulate an optimization problem that sequentially decides on query points that provide the most information about the Jacobian.
2.3 Related work
In the presented contribution we focus on the benefits of active sampling in policy search, specifically on sample efficiency. Therefore. this section focuses on active sampling in model-free RL setting using probabilistic uncertainty estimations. Most of the literature in this setting is based on BO, but generating informative samples is also discussed in literature regarding evolutionary strategies as well as in policy gradient methods.
Bayesian optimization as an active sampling method has been used for global policy search, mostly in lower dimensional parameter spaces from 2-15 dimensions [9-13]. Global BO for RL, exemplified by the mentioned literature and without additional assumptions, is limited to relatively low dimensional problems for two reasons: (i) the computational complexity of global probabilistic models does not scale well with the number of data points. (ii) global optimization of high-dimensional non-convex objectives is challenging problem solve in general. To combat these problems local variants of BO have been proposed and applied to RL problems, see c.g., [14-17]. These works rely on restricting the search space of BO by a probabilistic belief over the optimums location [14] using rectangular trust-regions [15], learning partition [16], or by staying close (as defined by the GP kernel) to past samples [17]. Restrictions the parameter space avoid "over-exploration of high-dimensional search spaces and thereby encourage exploitation of (local) minima. Our proposed method delegates the exploitation to gradient- based optimizer after exploring local property. the function's 's derivative at the current iterate. for which local search (and model) is sufficient. McLeod et al. [18] and more recently Shekhar and Javidi [19] suggest switching from global BO to a local gradient-basee method once a locally convex region containing low-regret solution has been identified, thereby improving convergence properties of BO. In [18] GIBO can replace the local optimizer of choice and in Shekhar and Javidi [19] GIBO can be used for optimal uncertainty reduction in the gradient estimate.
In general, a GP posterior can incorporate gradient information if the kernel is differentiable and a first-order oracle is available. Bayesian optimization methods that utilize gradient observations are known as first-order BO, and different approaches on how to include the derivative information the Since computing the joint posterior using first- and zeroth-order information computationally expensive, Ahmed et al. [21] and Wu et al. [22] are using a single directional derivative instead of all partial derivatives. A first-order BO approach for RL,, where the gradient information actively used to decide on the following query, is introduced by Prabuchandran et al. [23]. The method therein actively searches for local optima by querying points where the gradient is expected to be zero. In contrast to this approach, we actively reduce local uncertainty of the Jacobian model and afterwards a gradient- -ased optimizer decides on the next location.
model and acquisition functions have been proposed
Reinforcement learning problems in the form of (1) can also be used by evolutionary methods such as [1, 24-26] and recently by policy gradient methods Faccio et al. [27]. These methods typically explore through random perturbations in the parameter space of the policy instead of active sampling. However, generating more informative samples improves evolutionary strategies. Maheswaranathan et al. [25] shows this by adapting the sampling distribution using surrogate gradient information such as previous estimates, and Choromanski et al. [26] uses determinantal point processes for informative samples.
Policies that generate more informative samples have helped improve model-free RL algorithms' performance during the past decade; we mention three examples here: Levine and Koltun [28] propose so-called guiding samples in high reward areas using differential dynamic programming and model knowledge. Soft actor-critic (SAC) methods [3] add the policy's 's entropy to the reward function to encourage exploration and improve the variance of gradient estimates. Based on SAC an optimistic actor-critic algorithm is introduced in [29] with a different exploration strategy that samples more informative actions. To reduce variance in the gradient estimate, it possible to use GIBO as layer between the policy gradient estimator such as SAC and gradient- based optimizer, c.g., stochastic

---

VoJ
a(0)
0
0
0
Figure 2: We visualize GIBO's active sampling process with simple 1-dimensional function. The blue filled circle refers to the current parameter 0,. The figure shows steps of the algorithm, where in the first two steps, the acquisition function (solid green line) proposes two query points (green stars) of the objective function J (solid light grey line). With the history of sampling points (black crosses), the model of the Jacobian VoJ (in blue with confidence intervals) is updated, reducing uncertainty around the analytic Jacobian (dashed light grey line). The next step show a gradient ascent update step to 0 (blue star) and the last step is again suggested query after the update.
gradient ascent or Adam [30]. In future work GIBO can be extended to utilize state-of-the art policy gradient methods as an additional oracle for first-order information and help these methods reducing the variance of their gradient estimates through active sampling. Based on the posterior conditioned on all collected rewards, our algorithm can supply posterior gradient estimates and subsequent queries to evaluate.
To demonstrate the benefits GIBO in a simple setup, we adopt the setting proposed by Mania et al. [1] as baseline. Augmented Random Search (ARS) [1] assumes black-box setting without access gradient samples and estimates the gradient from the finteeddifference of random perturbations, effectively solving RL problems. We replace the random sampling strategy of ARS with active sampling and the gradient estimation with GP model. These changes improve the sample complexity and variance of ARS, especially when prior knowledge about the objective function is available.
3 Gradient informative Bayesian optimization
In this section, we introduce the proposed method GIBO. First, we define an acquisition function to reduce uncertainty for the Jacobian. Second, we outline the basic GIBO algorithm, including implementation choices.
3.1 Maximizing gradient information
We employ the BO framework to design a set iterative queries maximizing gradient information. To this extend, propose novel acquisition function Gradient Information (Gl) actively suggesting query points most informative for the gradient at the current parameters 9,. Acquisition functions measure the expected utility of sample point based surrogate model conditioned the observed data. The utility U Rd + IR of our method depends on Jacobian GP model, the objective's observation data D. and the current parameter O,. measures the decrease in the derivative's variance at Og when observing a new point of the objective function. Hence, we define the utility as the expected difference between the Jacobian's variance ('O,.D) before and the Jacobian's 's variance n' (D,) (0, v)}) after observing a new point (0, y)
aGG(( (0 0,, D) = E [U(0]0,, D)] = E [Ir (E)(),))) Tr (Σ' (0, (D, (0, v))))
(4)
where Tr denotes the trace operator and M' ('(e( is the variance of the Jacobian's 's GP model evaluated 0,
VoJ 0-0, ~ (()
(5)
5

---

The Jacobian's 's variance (0, (6, 8)}) depends on the extended dataset (0, y)). ) A property of the Gaussian distribution is, that the covariance function is independent the observed targets y as shown in Equation (3). Hence, we simplify the optimization over the expectation (see Appendix A.2)
(0,
to
arg max acr (0 D) = arg min Tr (2" (0el (),
(6)
where the variance only depends on a virtual data set X , On 0 =: [X, 0.. In conclusion, the most informative new parameter query is only dependent where sample next and is independent of its outcome f(())
When we replace the Jacobian's 's variance in (6) with (3) and leave out constant factors get
g arg max Tr (o, ((,,, [K +
(Vo,K(0..x)))
Since the acquisition function only depends on the virtual data set, its optimization can be handled computationally efficcent by performing the matrix inversion in (7) with Cholesky factor updates. Furthermore, since the Jacobian a local property we can optimize (7) effectively using the of-the- shelf optimizer supplied by BoTorch [31] (L-BFGS-BB B) using multiple restarts.
3.2
The GIBO algorithm
The guided sequential search of the acquisition function for gradient estimates divides the resulting algorithm into two loops: An outer loop for iterative parameter updates and an inner loop where the acquisition function queries points to increase gradient information. The basic algorithm is given in Alg. 1.
Algorithm/ GIBO
I: Hyperparameters: stepsize 7) hyperpriors for GP hyperparameters, number of iterations N and number of samples for gradient estimate M.
2: Initialize: place GP prior on (0), set 0g and D O--
for 0..... do
Parameter updates.
4: Sample noisy objective function: yt = ((.) +
5: Extend data set: D +- (D) (0.,
6: GP hyperparameter optimization.
for 1,2..... M do
Sample points for a gradient estimate.
8: Get query point: 0 arg max; (0)(0))
9:
Sample noisy objective function: 9 ((²) + w.
Optionally: Use policy gradient method for additional derivative observations.
Extend data set: D {D, (0, 9)). Update the posterior probability distribution of V.J.
11:
12: end for
13: Dc+1 + 17 E VoJ 6_
D Gradient ascent, or any other gradient based optimizer.
14:
end for
3.3 Implementation choices
In the following. we introduce some details of our implementation of Algorithm that further improve the performance and computational efficiency of our method.
Local GP model. Sparse approximation of GPs can be applied on BO when the computational burden of exact inference is too big [32]. our case, however, we are only interested in estimating the local Jacobian the current parameter 0.. We define sparse approximation of the posterior the current parameter 0, heuristically with the last N,, sampled points. Estimating a local model has the additional benefit of making the model selection and hyperparameter optimization simpler. We can approximate non-stationary processes locally by dynamically adapting hyperparameters.
Local optimization of GI. Following similar reasoning as above, we do not have optimize the GI acquisition function globally since we expect informative points to relatively close the current parameter 0r when using a SE kernel. Hence, we define our search bounds locally as (0, ōs, O, + 5z].

---

ARS
Vanilla BO
CMA-ES
GIBO
4-dim. domain
8-dim. domain
16-dim. domain
20-dim. domain
109
10
24-dim, domain
28-dim. domaii
32-dimn domain
36-dim. domait
10a
10-1
100
200
300
0
100
200
300
100
200
300
0
100
200
300
# of evaluations
of evaluations
# of ovaluations
# of evaluations
Figure Within-model comparison: Mean of the normalized distance of the function value at optimizers' best guesses from the true global maximum for eight different dimensional function domains. 40 runs. Logarithmic scale.
Gradient normalization. The gradient is normalized with the Mahalanobis norm using the length- scales of the SE kernel. Hence, the stepsize is adapted automatically to scale with the correlation between points. For the details see Appendix A.3.
State normalization. In the RL. setting we can apply state normalization before we evaluate the policy determine the next action. This has the same effect as data whitening for regression tasks and is beneficial when performing GP regression in unknown policy spaces. In case of a linear policy TT I2 TM Te (s) As + with bias b R", states IR,, means of states μla IRP, state normalization can be defined by ma شلا =
RP and
variances of states
+b=
a
A Ls s- A A b. State normalization is implemented in efficient way that does not require the storage of all states. Also, only keep track of the diagonal of the state's covariance matrix with Welford's online algorithm [33].
Empirical results
We empirically evaluate the performance of GIBO in three types of experiments. In the first experi- ment, we compare our algorithm on several functions sampled from a GP prior so that Assumption is satisfied. In these within-model comparisons [34), we show that GIBO outperforms the benchmark methods in terms of sample complexity and variance of regret, especially in higher dimension. In second experiment, we perform policy search for linear quadratic regulator (LQR) problem proposed by Mania et al. [1]. Finally, for RL environments of Gym [35] and MuJoCo [36], show that GIBO reaches acceptable rewards thresholds faster and with significantly less variance than ARS. All data and source code necessary to reproduce the results are published at https://gi thub. wtts/elleeleerrttii
4.1 Within-model comparison
We evaluate GIBOs performance as a general black-box optimizer on functions that satisfy Assump- tion 1. A straightforward way to guarantee this is by sampling the objective from a known GP prior. This approach has been called within-model comparison by Hennig and Schuler [34] but has likewise been used in other BO literature (e.g., [37, 38]). To show that GIBO scales particularly well to dinhes-ai search spaces, we analyze synthetic benchmarks for up to 36 dimensions.
The experiment was carried out over d-dimensional unit domain / = [0, 1]d, For each domain, we generate 40 different test functions. For each function, 1000 values were jointly sampled from a GP prior with SE kernel and unit signal variance. To cover the space evenly, we used a quasi-random
7

---

ARS
Vanilla BO
CMA-ES
GIBO
0.75
0.50
0.25
0
2 30
260
0.00
1
12
20
28
36
12
20
28
36
12
20
28
36
12
20
28
36
dimensions
dimensions
dimenssoon
dimensions
Figure 4: Within-model comparison: Boxplots (40 runs) show the normalized distance of optimizers' best found values after 300 function evaluations from the global maximum. The whiskers lengths are 1.5 the interquartile range; the black horizontal lines represent medians, green dots the means.
Sobol sampler. To perform experiments with comparable difficulty across different dimensional domains, we increase the lengthscales in higher dimensions by sampling them from the distribution ((d, introduced in Appendix A.4. The resulting posterior means were the objective function. All algorithms were started in the middle of the domain noised function evaluations. The noise was Gaussian distributed with standard deviation 0.1. A more detailed description of the experiments, including the true global maximum search and an out-of-model comparison, is given in Appendix A.4.
[0.5] and had limited budget of 300
We compared our algorithm GIBO to ARS, CMA-ES [24] and standard BO with expected improve- ment [39] as acquisition function ('Vanilla BO'). To ensure a fair comparison, domain knowledge was passed to the ARS and CMA-ES algorithms by scaling the space- memeedeeeendett hyperparameters with the mean of the lengthscale distribution (d). For details about the hyperparameters see Appendix A.7. The unknown hyperparameters were hand-tuned on a low dimensional example.
Fig. 3 shows the normalized difference between the global optimum and the function values of the optimizer's 's best guesses. The within-model comparison shows that our algorithm outperforms vanilla BO on all test functions, except for the 4-dimensional domain. With limited budget of 300 function evaluations the proposed method, GIBO, achieved lower regret than the baseline methods, especially in higher dimensions. Further, GIBO was able reduce the variance of obtained regret significantly, as shown in Figure 4, which indicates consistently better performance.
4.2 Linear quadratic regulator
The classic LQR with known dynamics is fundamental problem in control theory. In this setting, agent seeks to control a linear dynamical system while minimizing a quadratic cost. With available
10a
Fequency of Stability
1.0
GIBO
0.8
Relative Cost.
LSPI
10"
ARS
0.6
0.4
GIRO
10
0.2
LSPI
0.0
10-2
10000
20000
30000
40000
10000
20000
30000
40000
Number Timesteps
Number Timesteps
Figure 5: Results for the LQR experiment. Left: How frequently GIBO found stabilizing controllers in comparison to ARS and LSPI. The frequencies are estimated from 100 trials. Right: The sub- optimality gap the controllers produced by GIBO compared ARS and LSPI. The points along the dashed line denote the median cost, and the shaded region covers 2-nd to 98-th percentile out of 100 trials. Values for the benchmark methods in both images are estimated from [1].

---

Cartpole-v1
Swimmer-vl
Hopper-v1
Average Reward
400
3000
400
200
2000
GIBO
200
1000
25
50
75
100
500
1000
1500
2000
0
2000
4000
6000
8000
of evaluations
# of evaluations
# of evaluations
Figure 6: Training curves of GIBO and ARS for classic control and MuJoCo tasks, averaged over 10 trails (thin lines). The shaded regions show the standard deviation.
dynamics, the LQR problem has an efficiently determinable optimal solution. LQQ with unknown dynamics, on the other hand, is less well understood. As argued in Mania et al. [1], this offers new type of benchmark problems, where one can set up LQR problems with challenging dynamics, and compare model-free methods to known optimal costs. We compare GIBO against ARS and LSPI [40] on a challenging LQR instance with unknown dynamics, proposed by Dean et al. [41]. The reader is referred Appendix A.6 for complete introduction to the setup.
Fig. shows the frequency of stable controllers found and the cost compared to the optimal cost for GIBO, ARS, and LSPI. On the left in Fig. 5 we observe that GIBO requires significantly fewer samples than ARS, equivalent to LSPI, to find a stabilizing controller. But we note that LSPI requires an initial controller Ko. which stabilizes discounted version of the LQR problem. Neither GIBO nor ARS require any special initialization. All algorithms achieve similar regrets.
4.3 Gym and MuJoCo
Lastly, we evaluate the performance of GIBO on classic control and MuJoCo tasks included in the OpenAI Gym [35, 36]. The OpenAI Gym provides benchmark reward functions that use to evaluate our policies' performance compared to policies trained by ARS. Mania et al. [1] showed that deterministic linear policies, Ze TR R. (T(s)) As are sufficiently expressive for MuloCo locomotion tasks. Consequently, we define our parameter space by 0 (A, b) E R dimensions. For all environments, normalize the reward axis. For the Hopper environment, we additionally subtract survival bonus and use state normalization; find further details in Appendix A.5. We hand-tuned the hyperparameter of GIBO within a reasonable degree, where the hyperparameter for ARS are taken from [1]. In the following, we use the reward over function evaluations (calls of RL environment) as evaluation metric for sample efficiency. We averaged the reported policy rewards over ten trials. In Fig. 6 we observe that GIBO reaches the reward thresholds faster and with significantly less variance than ARS.
For the CartPole-vl we need 4, for the Swimmer-vl 16 and for the Hopper-vl
GIBO
GIBO w/o grad. norm.
12-dim. domain
24 di-iim. domain
36 dim. domain
GIBO
GIBO w/o grad. norm.
10a
1.00
0.75
0.50
0.25
0.00
100
200
300
100
200
300
0
100
200
300
12
24
36
12
36
# of evaluations
# of evaluations of evaluations
dimensions
dimensions
Figure 7: Ablation study for within-model comparison. GIBO with and without gradient normal- ization. Left: Regret over 300 function evaluations. Right: Distribution of regret after 300 function evaluations.

---

Cartpole-vl
Swimmer-vl
Hopper-vl
Average Reward
600
400
GTBO
400
2000
w/o
200
w/o state norm.
0
50
75
100
500
# of evaluations
evaluations
# of evaluations
Figure 8: Ablation study for RL environments. Training curves of GIBO and its ablated variants on different RL environments, averaged over 10 trials.
4.4 Ablation study
In this section, we investigate different implementation choices of the GIBO algorithm. We conduct ablation experiments the within-model comparison with synthetic objective functions as well as on RL environments. Gradient normalization using the known GP lengthscales leads to significant improve mean performance as well as reduced variance, see Fig. When optimizing policies for the RL benchmarks the GP lengthscales are not known and are learned during training. Fig. 8 shows that even in the case of learned pyperaarameters gradient normalization proves to be important for the performance. We applied state normalization only to the Hopper environment and found that for this task is not possible to learn well pellorerformng policies without this extension. This shows that the normalization of an unknown policy space can be crucial for GP regression.
5 Conclusion
introduce GIBO, a gradient-based optimization algorithm with a BO-type active sampling strategy to improve gradient estimates for black-box optimization problems. When the model assumptions of BO are satisfied, we show that the algorithm is significantly more sample-efficient especially in higher dimensions, compared baseline algorithms for black-box optimization.
Additionally, we show the benefits of active sampling and probabilistic gradient estimates with GIBO by solving popular RL. benchmarks for which the model assumptions do not hold exactly. When compared random sampling, GIBO is more sample efficient and has lower variance. Yet, the performance benefits are less pronounced in the RL task. This highlights that GIBO especially shines when prior knowledge is available while it still performs reasonably otherwise. Nonetheless, we want to remark that the prior biases the gradient estimates and wrong assumptions about the objective function can deteriorate performance. However, in some sense, all hyperparameters in RL algorithms encode some form of prior knowledge about the problem at hand. In our view, explicit probabilistic priors are an appropriate and intuitive form of prior knowledge to obtain, e.g., from domain knowledge available data from prior experiments.
Since is straightforward to include derivative observations into GIBO, we expect similar improve- ments for other existing RL methods when integrating our method as an additional layer between gradient estimators and optimizers. The proposed framework can suggest different exploration policies and combine all available data into posterior belief over the Jacobian. For future research, we want to utilize GIBO with state-of-the-art actor-critic algorithms to improve sample complexity of these methods.
In a more general context, our active sampling methodology makes a step towards autonomous decision makinn. GIBO decides on learning experiment for the autonomous agent. Whenever decision process is automated, the responsibility for legal and ethical consequences these decisions must be resolved. However, we do not discuss how the decision-maker GIBO, can be constrained to ensure compliance with regulatory requirements, which is relevant aspect for future research.
10

---

Acknowledgments and Disclosure of Funding
The authors thank D. Baumann, Berens, A. R. Geist, H. Heidrich and F. Solowjow for their helpful comments and discussions. This work supported in part by the Cyber Valley Initiative; the Max Planck Society: by the German Federal Ministry of Education Research (BMBF): Tübingen AI Center, FKZ: 011S1 18099A; and by the Deutsche Forschungsgemeinscaaff (DFG, German Re- search Foundation) under Germany's 's Excellence Strategy EXC number 2064/1 - Project number 390727645. The authors thank the International Max Planck Research School for Intelligent Systems for supporting A. von Rohr and S. Müller.
References
[1] Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies competitive for reinforcement learning. In Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[2] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. International Conference on Machine Learning, pages 1587-1596. PMLR, 2018.
[3] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.
[4] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, arXiv preprint MxXiv:15090029970 2015.
[5] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[6] Donald R. Jones, Matthias Schonlau, and William J. Welch. Efficient Global Optimization of Expensive Black-Boo -ox Functions. Journal of Global Optinization, 13(4):555- 492, 1998.
[7] Carl E. Rasmussen and Christopher K. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
[8] Bobak Shahriari, Kevin Swersky, Ziyu Wang. Ryan P. Adams, and Nando de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, pages 148- 175, 2016.
[9] Daniel J. Lizotte, Tao Wang, Michael H. Bowling, and Dale Schuurmans. Automatic gait optimization with gaussian process regression. In International Joint Conferences on Artificial Intelligence, volume 7, pages
2007.
[10] Aaron Wilson, Alan Fern, and Prasad Tadepalli. Using Trajectory Data Improve Bayesian Optimization for Reinforcement Learning. Journal of Machine Learning Research, pages 253-282, 2014.
[11] A. Marco, P. Hennig, J. Bohg, S. Schaal, and S. Trimpe. Automatic LQR tuning based on Gaussian process global optimization. IEEE International Conference on Robotics and Automation, pages 270-277, 2016.
[12] Ruben Martinez-Cantin. Bayesian optimization with adaptive kernels for robot control. In IEEE International Conference on Robotics and Automation, pages 3350-3356, 2017.
[13] Alexander von Rohr, Sebastian Trimpe, Alonso Marco, Peer Fischer, and Stefano Palagi. Gait learning for soft microrobots controlled by light fields. In International Conference on Intelligent Robots and Systems, pages 6199-6206, 2018.

---

[14] Riad Akrour, Dmitry Sorokin, Jan Peters, and Gerhard Neumann. Local Bayesian optimization of motor skills. In Proceedings of the 34th International Conference on Machine Learning. volume 70, pages 41-50. PMLR, 06-11 11 Aug 2017.
[15] David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local bayesian optimization. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[16] Linnan Wang. Rodrigo Fonseca, and Yuandong Tian. Learning search space partition for black-box optimization using monte carlo tree search. In Advances in Neural Information Processing Systems, volume 33, pages 1951 195522. Curran Associates, Inc., 2020.
[17] Lukas P. Fröhlich, Melanie N. Zeilinger, and Edgar D. Klenske, Cautious bayesian optimization for efficient and scalable policy search. In Proceedings of the 3rd Conference on Learning for Dynamics and Control, volume 144, pages 227-240. PMLR, 07 -8 June 2021.
[18] Mark McLeod, Stephen Roberts, and Michael A. Osborne. Optimization, fast and slow: optimally switching between local and Bayesian optimization. In Proceedings of the 35th International Conference Machine Leurning, volume 80, pages 3443-3452, PMLR, 10-15 Jul 2018.
[19] Shubhanshu Shekhar and Tara Javidi. Significance of gradient information in bayesian opti- mization. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130, pages 2836-2844. PMLR, 13-15 Apr 2021.
[20] Michael A. Osborne, Roman Garnett, and Stephen J. Roberts. Gaussian processes for global optimization, In 3rd International Conference on Learning and Intelligent Optimization, pages 1-15, 2009.
[21] Mohamed O. Ahmed, Bobak Shahriari, and Mark Schmidt. Do we need "harmless" bayesian optimization and "first-order" bayesian optimization. In NeurIPS Workshop on Bayesian Optimization, 2016.
[22] Jian Wu, Matthias Poloczek, Andrew G. Wilson, and Peter Frazier. Bayesian Optimization with Gradients. In Advances in Neural Information Processing Systems, pages 5267-5277777 2017. 2017.
[23] K. J. Prabuchandran, Santosh Penubothula, Chandramouli Kamanchi, and S. Bhatnagar, Novel First Order Bayesian Optimization with an Application to Reinforcement Learning. Applied Intelligence, pages 1579, 2021.
[24] Nikolaus Hansen and Andreas Ostermeier. Completely Derandomized Self-Adaptation in Evolution Strategies. Evolutionary Computation, pages 159-195, 2001.
[25] Niru Maheswaranathan, Luke Metz, George Tucker, Dami Choi, and Jascha Sohl-Dickstein. Guided evolutionary strategies: augmenting random search with surrogate gradients. In Proceed- ings of the 36th International Conference on Machine Learning, volume 97, pages 4264 4273. PMLR, 09-15 Jun 2019.
[26) Krzysztof Choromanski, Aldo Pacchiano, Jack Parker-Holder and Yunhao Tang. Practical nonisotropic monte carlo sampling in high dimensions via determinantal point processes. In Proceedings of the Twenty Third International Conference Artificial Intelligence and Statistics, volume 108, pages 1363-1374. PMLR, 26-28 Aug 2020.
[27] Francesco Faccio, Louis Kirsch, and Jürgen Schmidhuber. Parameter-basee value functions, 2021.
[28) Sergey Levine and Vladlen Koltun. Guided policy search. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference Machine Learning, volume 28, pages 1-9, Atlanta, Georgia, USA, 17-19 Jun 2013, PMLR.
[29] Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
12

---

[30] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations. 2015.
[31] Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carie Bayesian Optimization. In Advances in Neural Information Processing Systems 33, 2020.
[32] Mitchell McIntire, Daniel Ratner, and Stefano Ermon. Sparse gaussian processes for bayesian optimization. In Thirty-Second Conference Uncertainty in Artificial Intelligence, page 517-526, 2016.
[33] B. P. Welford. Note on a Method for Calculating Corrected Sums of Squares and Products. Technometrics, pages 419-420, 1962.
[34] Philipp Hennig and Christian Schuler. Entropy Search for Information- Effficient Global Optimization. Journal of Machine Learning Research, pages 1809 1837, 2012.
[35] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym. arXiv preprint arXiv:1606.01540 2016.
[36] E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control. JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.
[37] José Miguel Hernández-Lobato, Michael A. Gelbart, Ryan Adams, Matthew W. Hoffman, and Zoubin Ghahramani. A General Framework for Constrained Bayesian Optimization using Information-bail Search. Journal of Machine Learning Research, pages 1-53, 2016.
[38] Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient Bayesian optimization. In Proceedings of the 34th International Conference on Machine Learning, pages 3627-3635, 2017.
[39] Donald R. Jones, A taxonomy of global optimization methods based on response surfaces. Journal of Global Optimization, pages 345-383, 2001.
[40] Stephen Tu and Benjamin Recht. Least-squares temporal difference learning for the linear quadratic regulator. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 5005-5014. PMLR, 10-15 Jul 2018.
[41] Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu. On the Sample Complexity of the Linear Quadratic Regulator. Foundations of Computational Mathematics, pages 633-679, 679, 2020.
[42] Geoffrey Hinton. Lecture: Neural Networks for Machine Learning, 2012. https://www... wawpntto..dd/hiinttnnt
html.
[43] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature, pages 533-536, 1986.
[44] John Duchi, Flad Hazan, and Yoram Singer. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, pages 2121-2159, November 2011.
[45] R. S. Anderssen, R. P. Brent, D. J. Daley, and P. A. P. Moran. Concerning h .... So (of +.... + =z)) 1/2 dx ... dry and a Taylor Series Method. SIAM Journal on Ap- plied Mathematics, pages 22 22-30, 1976.
[46] Stephen Tu. Sample Complexity Bounds for the Linear Quadratic Regulator. Technical Report UCB/EECS-20N -2019-22, University of California at Berkeley, 2019.
[47] Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems, 2018
13


Reviews:
[{"review_id": "sf6XGWXYwml", "reviewer": "Reviewer_up9D", "summary": "The paper introduces GIBO, a modification of Bayesian Optimization (BO) that performs local policy search. Compared to classic BO, the strengths of this method are that it does not require storing all the past data points, making it more computational efficient, and that it is more stable when moving to higher dimensional problems. The authors compare their method with standard BO and Augmented Random Search, an approach based on finite difference policy search.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nI found the paper well written and easy to follow. The authors related their method to previous approaches using BO for Reinforcement Learning. Since the experiments compare GIBO with CMA-ES and ARS, I would have expected to see a discussion about the similarities and differences with other evolutionary approaches.  Another method that is closely related is Parameter-based Value Functions [1], which directly tries to estimate $J(\\theta)$ using a deterministic function. They also include a comparison with ARS using linear and nonlinear policies. \nBesides the related work section that could be improved, there are some main issues that prevents acceptance:\n\n- Section 3.3 presents many improvements on the basic idea of GIBO, but there are no ablations in the paper supporting these additions. Is also Vanilla BO using these improvements? Ablating these components in GIBO and Vanilla BO seems necessary here to understand why the proposed method can be considered as an improvement\n\n- One main claim in the paper is that it has low variance compare with other methods (ARS). However, in the experimental evaluation, only 3 trails are used for each environment. I would expect to see at least 10 or 20 runs, especially in these relatively small tasks involving very low dimensional linear policies. Using 3 runs is not enough to assess if the variances are really difference. \n \n- I am a bit concerned about the hyperparameters for GIBO in Table 3. How were these chosen? Are the 3 runs in Figure 6 using different seeds with respect to those that found when optimizing the hyperparameters?\n\nIf the authors can expand their related work section and solve the main concerns in the experimental section, I am willing to increase the score for this submission.\n\n[1] Francesco Faccio, Louis Kirsch, and Jurgen Schmidhuber. Parameter-based value functions. In ¨ Int. Conf. on Learning Representations (ICLR), Virtual only, May 2021.\n\n------------------------------------------------------------------------------------------------------------------------------------------------------------\nEdit post-rebuttal: The authors clarified my concerns", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "_TysRBKBf08", "reviewer": "Reviewer_h27d", "summary": "This paper concerns the high sample complexity of policy gradient methods and the computational complexity of Bayesian optimization. The authors propose a novel algorithm called GIBO (and its local variants) to minimize uncertainty in gradient estimation. Experiments are conducted and compared with some naive baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality and significance:\nThe idea itself is interesting. However, the contributions and improvements over GP algorithms are unknown. See detailed comments below.\n\nClarity:\nThe paper has clear structures and is easy to follow.\n\nDetailed comments:\n\n1. The authors claim that BO methods have been restricted to low-dimensional problems. But there are modern Bayesian RL algorithms that are both theoretically grounded and verified experimentally in high-dimensional tasks such as Mujoco [1]. It's thus not clear the contributions of this work.\n\n2. The experiments are conducted and compared with only very simple baselines. Since the paper aims to address the drawbacks of policy gradient with random perturbations algorithms and BO algorithms, it will be more convincing to compare with more SOTA methods in these two areas.\n\n3. In Fig. 3, it seems that the curves are not converging in the plots. So the improvements of GIBO are not clear.\n\n\n[1] Sebastian Curi et al. Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning. NeurIPS 2020.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors state the limitations of their algorithms that rely on prior knowledge of the objective. Negative social impacts are also listed in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "RGSUXSrv-l6", "reviewer": "Reviewer_b2oQ", "summary": "In this paper, the authors propose a new acquisition function for Bayesian optimization based on gradient information. A GP model estimates the Jacobian and its estimation is used for gradient update. This method is implemented in the setting of policy search for reinforcement learning. Empirical evaluations are provided for a synthetic setting, LQR and MuJoCo gym environments.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Post-rebuttal comments: thanks for providing a detailed response. I still think the comparison with [1] is valuable. Or as the authors mentioned, demonstrate how GIBO can be used in conjunction with [1]. As a result, I will keep my score.\n\nOriginality: I really like the perspective the authors have taken to connect policy search with Bayesian optimization. The acquisition function based on uncertainty reduction on gradient is novel and fits the use case quite well. \n\nThe related work section is comprehensive. I would like to draw the authors’ attention to the following paper [1] Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search, Linnan Wang, Rodrigo Fonseca, Yuandong Tian, NeurIPS 2020, which also considers a blackbox setting for policy search. \n\nQuality: The paper is technically sound. I have some technical questions on the algorithm. \n1. Can the authors provide more details on how the optimization for equation (7) is done? The appendix mentioned Cholesky factorization to model K(\\hat{X}, \\hat{X}) but it is unclear to me how the whole equation is optimized. Were you able to compute the gradient analytically and perform gradient descent?\n\n2. Notation-wise, should the \\theta_t in equation (7) be \\theta_n instead since \\hat{X} indicates that n different thetas have been queries?\n\n3. In section 3.3 for the uncertainty threshold, the Sigma’s are matrices, what does it mean to bound the difference between two matrices with a scalar \\epsilon?\n\n4. On the hyperparameters in section 3.3, e.g., the bound (\\delta) and the threshold (\\epsilon), how sensitive is the performance of the GIBO algorithm to them?\n\n5. Can you include error bars for the experiment in section 4.1?\n\nClarity:This paper is well-written and easy to read. Good job! \n\nSignificance: I enjoy the novelty of this paper. However, I think the significance can be improved by improving the empirical section. Comparison with [1] should be included as it is competitive with ARS on Swimmer and Hopper for MuJoCo. \n\nMoreover, I would like to know if it is possible to consider higher dimensional environments, such as the humanoid. Is there a potential scalability issue with the proposed method?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The potential scalability limitation needs more discussion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "7YSCB6UJD-F", "reviewer": "Reviewer_h5kV", "summary": "This paper modifies the standard BayesOpt pipeline in 2 ways:\n\n1. Defines a new acquisition function, the \"Gradient Information\" (GI) which is the reduction in the variance of the Jacobian variance of $J$.\n2. Exploration and Exploitation become disjointed: During the BayesOpt process, a batch of $M$ queries are made via argmax of the GI acquisition, purely for exploration (rather than a mix of exploitation and exploration as is standard in BayesOpt), with the goal of producing a final gradient update step $\\nabla_{\\theta}J$. \n\nThis defines a \"GIBO\" algorithm, which is then experimentally evaluated over the following benchmarks: \n\n1. Sampled functions from GP's\n2. LQR Controllers\n3. Linear controllers for standard Mujoco tasks.\n\nwhich show that GIBO outperforms previous ES/ARS and standard BayesOpt methods especially in high dimensional settings, most likely due to the use of \"local search\"/gradients.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. I overall think this is a promising and good paper, with many potential future directions in BayesOpt + ES/RL research, especially due to its strengths on high dimensional optimization.\n2. In some sense, the method can be seen as a way of \"intelligently\" picking a batch of neighborhood points to form a gradient, rather than e.g. simply via sampling perturbations from an Isotropic Gaussian (as is standard in ARS/ES methods).\n3. Paper is well-written, and well-understandable in general for someone familiar with BayesOpt methods.\n\nQuestions/Cons:\n1. I'm assuming that because the acquisition function is based on the variance of the Jacobian at the \"current\" point $\\theta_{t}$, this means that future argmax query points in the batch would likely be very close to $\\theta_{t}$ right? (Since local information around $\\theta_{t}$ probably contributes more to the gradient estimation at $\\theta_{t}$). I believe that this could potentially cause problems with global exploration / have the same issues with gradient estimation methods converging too fast to local minima.\n2. Citation requests: You may want to cite/compare e.g. [1] (which uses a k-DPP to maintain unbiased gradient estimation in ES, but reduce variance and improve exploration) and other related \"manual\" methods on improving the variance of ES gradient estimation. Similar to the previous point as well: how \"exploratory\" are the queries really, when optimizing the GI acquisition?\n3. How well does GIBO perform in the basic neural network policy setting (where there will be ~200-1000 parameters from a e.g. 2-layer MLP)? Such scenarios are still quite common even in ES + RL methods (e.g. A nonlinear network is needed for training on Pendulum with ES from my experience; a linear policy doesn't always work)\n4. How parallelizable is the method, especially with the $M$ batch query section? A major benefit of ARS/ES is that they can be parallelized with an arbitrary number of workers, (analogous to \"$M$\"), but it seems like the dataset update $\\mathcal{D} \\leftarrow  (\\theta,y) $ forces sequentiality. Could this be moved outside of the batch-for-loop? \n\n[1] Structured Monte Carlo Sampling for Nonisotropic Distributions via Determinantal Point Processes \n\nSmall notation issues:\n* In Algorithm 1, $m$ is already used as a variable denoting the mean function before; perhaps change it for clarity.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciate the novelty of the main contribution of the paper, ie the acquisition function defined as reduction in gradient variance under a GP model. The paper could be strengthened further by applying the proposed method to higher-dimensional problems.
Nevertheless, the paper is acceptance.
Decision: Accept (Poster)