Local policy search with Bayesian optimization
Year: 2021

Abstract:
Reinforcement learning (RL) aims to find an optimal policy by interaction with an environment. Consequently, learning complex behavior requires a vast number of samples, which can be prohibitive in practice. Nevertheless, instead of systematically reasoning and actively choosing informative samples, policy gradients for local search are often obtained from random perturbations. These random samples yield high variance estimates and hence are sub-optimal in terms of sample complexity. Actively selecting informative samples is at the core of Bayesian optimization, which constructs a probabilistic surrogate of the objective from past samples to reason about informative subsequent ones. In this paper, we propose to join both worlds. We develop an algorithm utilizing a probabilistic model of the objective function and its gradient. Based on the model, the algorithm decides where to query a noisy zeroth-order oracle to improve the gradient estimates. The resulting algorithm is a novel type of policy search method, which we compare to existing black-box algorithms. The comparison reveals improved sample complexity and reduced variance in extensive empirical evaluations on synthetic objectives. Further, we highlight the benefits of active sampling on popular RL benchmarks.

heading: 1 Introduction
text: Reinforcement learning (RL) is a notoriously data-hungry machine learning problem, where state-ofthe-art methods easily require tens of thousands of data points to learn a given task [1]. For every data point, the agent has to carry out potentially complex interactions with its environment, either in simulation or in the physical world. This expensive data collection motivates the development of sample-efficient algorithms. Herein, we consider policy search problems, a type of RL technique where we directly optimize the parameters of a policy with respect to the cumulative reward over a finite episode. The collected data is utilized to estimate the direction of local policy improvement, enabling the use of powerful optimization techniques such as stochastic gradient descent. Policy gradient methods (e.g., [2–5]) usually rely on random perturbations for data generation, e.g., in the form of exploration noise in the action space or stochastic policies, and do not reason about uncertainty in their gradient estimation. However, innate in the RL setting is the ability to actively generate data, allowing the agent to decide on informative queries, thereby potentially reducing the amount of data needed to find a (local) optimum. Active sampling has the potential to allow those algorithms to improve sample complexity, reducing the number of environment interactions.
∗Equal contribution
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
In contrast to random sampling, Bayesian optimization (BO) [6] is a paradigm to optimize expensive-to-evaluate and noisy functions in a sample-efficient manner. At the core of BO is the question of how to query the objective function efficiently to maximize the information contained in each sample. By building a probabilistic model of the objective using past data and, critically, prior knowledge, the algorithm can reason about how to query a noisy oracle to solve the optimization task. Since RL can be framed as a black-box optimization problem, we can use BO to learn policies in a sample-efficient way. However, even though BO has been used to tackle RL, these approaches are often restricted to low-dimensional problems. One reason is that BO aims to find a global optimum; hence, without further assumptions, BO algorithms need to model and search the entire domain, which needs a lot of data and gets exponentially more
difficult as the dimensionality increases. Additionally, as the amount of data grows so does the computational complexity of probabilistic models, which becomes a significant problem. However, the success of RL algorithms using policy gradient methods indicates that for many problems it is sufficient to find a locally optimal policy.
Our proposed algorithm combines the strength of gradient-based policy optimization with active sampling of policy parameters using BO. We thereby improve the computational complexity of BO methods on the one hand, and the sample-inefficiency of gradient-based methods on the other hand, especially when proper prior knowledge is available. We achieve these improvements by explicitly learning a probabilistic model of the objective in the form of a Gaussian process (GP) [7]. From this model, we can jointly infer the objective and its gradients with a tractable probabilistic posterior. The resulting Jacobian estimate includes all data points, rendering data usage more efficient. Further, the algorithm infers informative queries from the uncertainty of the probabilistic model to improve the estimate of the local gradient. While in this paper we adapt the setting of Mania et al. [1] and assume access to zeroth-order information only, the algorithm extends straightforwardly to policy gradient algorithms where additional first-order information is available. In summary, the contribution of this paper is a local BO-like optimizer called Gradient Information with BO (GIBO). The queries of GIBO are chosen optimally to minimize uncertainty about the Jacobian. GIBO uses a local GP model for active sampling and gradient estimation and can be used with existing policy search algorithms. Using only zeroth order information, GIBO is able to
• significantly improves sample complexity in extensive within-model comparisons, i.e., when accurate prior knowledge is available; • is able to solve RL benchmark problems in a sample efficient manner; and • reduces variance in the rewards when compared to non-active sampling baselines.

heading: 2 Preliminaries
text: This work presents a local optimizer with active sampling. The objective function and its derivative’s joint distribution are modeled using a GP. Since we have developed the optimizer with the RL application in mind, we also introduce the RL problem. For the sake of brevity, we refer the reader to [7] and [8] for an introduction into GPs and BO, respectively.

heading: 2.1 Problem setting
text: In the following we phrase policy search as a black-box optimization problem. For a parameterized policy πθ : Θ × S → A that maps states s ∈ S and the static policy parameters θ ∈ Θ to actions a ∈ A, we use the same performance measure as in policy gradient methods for the episodic case.
Hence, the objective function J : Rd → R is defined as
J(θ) = Eπθ
[ I∑
i=0
ri
] ,
where Eπθ is the expectation under policy πθ, ri is the reward at time step i, and I the length of the episode. A BO query is equivalent to the return of one rollout following the policy πθ in the environment. The expected episodic reward is entirely determined by choice of policy parameters (and the initial conditions). Thus, the optimizer explores the reward function in the parameter space rather than in the action space. Since initial conditions might vary and the environment can be non-deterministic, reward evaluations are noisy.
Policy search herein is abstracted as a zeroth-order optimization problem of the form
θ∗ = arg max θ∈Θ J(θ), (1)
where θ is the variable and Θ ⊂ Rd a bounded set. To solve (1), an optimization algorithm can query an oracle for a noisy function evaluation y = J(θ) + ω. We assume an i.i.d. noise variable ω ∈ R to follow a normal distribution ω ∼ N (0, σ2) with variance σ2. We do not assume access to gradient information or other higher-order oracles for conciseness. Albeit, GIBO requires that the following critical assumption is fulfilled: Assumption 1. The objective function J is a sample from a known GP prior J ∼ GP (m(θ), k(θ, θ′)), where the mean function is at least once differentiable and the covariance function k is at least twice differentiable, w.r.t. θ.
This is the standard setting for BO with the addition that the mean and kernel need to be differentiable, which is satisfied by some of the most common kernels such as the squared exponential (SE) kernel. In the empirical section, we investigate the performance of the developed algorithm with and without Assumption 1 holding true.

heading: 2.2 Jacobian GP model
text: Since GPs are closed under linear operations, the derivative of a GP is again a GP [7]. This enables us to derive an analytical distribution for the objective’s Jacobian, which we can use as a proxy for gradient estimates and enable gradient-based optimization.
Following Rasmussen and Williams [7], the joint distribution between a GP and its derivative at the point θ∗ is
[ ȳ
∇θ∗J∗
] ∼ N ([ m(X) ∇θ∗m(θ∗) ] , [ K(X,X) + σ2I ∇θ∗K(X, θ∗) ∇θ∗K(θ∗, X) ∇2θ∗K(θ∗, θ∗) ]) , (2)
where ȳ are the n zeroth-order observations, X ⊂ Θ are the locations of these observations X = [θ1, . . . , θn], andK the covariance matrix given by the kernel function k : Θ×Θ→ R. The posterior can be derived by conditioning the joint Gaussian prior distribution on the observation [7]
p ( ∇θ∗J∗ ∣∣θ∗, X, ȳ ) ∼ N (µ′∗,Σ′∗)
µ′∗ = ∇θ∗m(θ∗)︸ ︷︷ ︸ ∈Rd +∇θ∗K(θ∗, X)︸ ︷︷ ︸ ∈Rd×n
( K(X,X) + σ2I )−1 ︸ ︷︷ ︸
∈Rn×n
(ȳ −m(X))︸ ︷︷ ︸ ∈Rn ∈ Rd
Σ′∗ = ∇2θ∗K(θ∗, θ∗)︸ ︷︷ ︸ ∈Rd×d −∇θ∗K(θ∗, X)︸ ︷︷ ︸ ∈Rd×n
( K(X,X) + σ2I )−1 ︸ ︷︷ ︸
∈Rn×n
∇θ∗K(X, θ∗)︸ ︷︷ ︸ ∈Rn×d
∈ Rd×d.
(3)
Remark 1. Note that the term (K(X,X) + σ2I)−1 with the highest computational cost (O(N3)) is the same term that is used to compute the posterior over J . Therefore, calculating the Jacobian does not add to the computational complexity once a GP posterior has been computed.
Any twice differentiable kernel is sufficient for the presented framework, but we assume a SE kernel for the remainder of the paper. For the derivatives of the SE kernel function see Appendix A.1. For a visual example of function- and the Jacobian-posterior, refer to Fig. 1. The figure indicates that
a zeroth-order oracle is enough to form a reasonable belief over the function’s gradient. Moreover, Fig. 1 shows that the uncertainty about the Jacobian gets reduced between query points more so than at the query points themselves. To minimize uncertainty about the Jacobian at a specific point, it intuitively makes sense to space out query points in its immediate surrounding. Herein, we formalize this intuition and formulate an optimization problem that sequentially decides on query points that provide the most information about the Jacobian.

heading: 2.3 Related work
text: In the presented contribution we focus on the benefits of active sampling in policy search, specifically on sample efficiency. Therefore, this section focuses on active sampling in model-free RL setting using probabilistic uncertainty estimations. Most of the literature in this setting is based on BO, but generating informative samples is also discussed in literature regarding evolutionary strategies as well as in policy gradient methods.
Bayesian optimization as an active sampling method has been used for global policy search, mostly in lower dimensional parameter spaces from 2–15 dimensions [9–13]. Global BO for RL, exemplified by the mentioned literature and without additional assumptions, is limited to relatively low dimensional problems for two reasons: (i) the computational complexity of global probabilistic models does not scale well with the number of data points, (ii) global optimization of high-dimensional non-convex objectives is a challenging problem to solve in general. To combat these problems local variants of BO have been proposed and applied to RL problems, see e.g., [14–17]. These works rely on restricting the search space of BO by a probabilistic belief over the optimums location [14] using rectangular trust-regions [15], learning a partition [16], or by staying close (as defined by the GP kernel) to past samples [17]. Restrictions in the parameter space avoid ’over-exploration’ of high-dimensional search spaces and thereby encourage exploitation of (local) minima. Our proposed method delegates the exploitation to a gradient-based optimizer after exploring a local property, the function’s derivative at the current iterate, for which a local search (and model) is sufficient. McLeod et al. [18] and more recently Shekhar and Javidi [19] suggest switching from global BO to a local gradient-based method once a locally convex region containing a low-regret solution has been identified, thereby improving convergence properties of BO. In [18] GIBO can replace the local optimizer of choice and in Shekhar and Javidi [19] GIBO can be used for optimal uncertainty reduction in the gradient estimate.
In general, a GP posterior can incorporate gradient information if the kernel is differentiable and a first-order oracle is available. Bayesian optimization methods that utilize gradient observations are known as first-order BO, and different approaches on how to include the derivative information in the model and acquisition functions have been proposed [20–23]. Since computing the joint posterior using first- and zeroth-order information is computationally expensive, Ahmed et al. [21] and Wu et al. [22] are using a single directional derivative instead of all partial derivatives. A first-order BO approach for RL, where the gradient information is actively used to decide on the following query, is introduced by Prabuchandran et al. [23]. The method therein actively searches for local optima by querying points where the gradient is expected to be zero. In contrast to this approach, we actively reduce local uncertainty of the Jacobian model and afterwards a gradient-based optimizer decides on the next location.
Reinforcement learning problems in the form of (1) can also be used by evolutionary methods such as [1, 24–26] and recently by policy gradient methods Faccio et al. [27]. These methods typically explore through random perturbations in the parameter space of the policy instead of active sampling. However, generating more informative samples improves evolutionary strategies. Maheswaranathan et al. [25] shows this by adapting the sampling distribution using surrogate gradient information such as previous estimates, and Choromanski et al. [26] uses determinantal point processes for informative samples.
Policies that generate more informative samples have helped to improve model-free RL algorithms’ performance during the past decade; we mention three examples here: Levine and Koltun [28] propose so-called guiding samples in high reward areas using differential dynamic programming and model knowledge. Soft actor-critic (SAC) methods [3] add the policy’s entropy to the reward function to encourage exploration and improve the variance of gradient estimates. Based on SAC an optimistic actor-critic algorithm is introduced in [29] with a different exploration strategy that samples more informative actions. To reduce variance in the gradient estimate, it is possible to use GIBO as a layer between the policy gradient estimator such as SAC and a gradient-based optimizer, e.g., stochastic
gradient ascent or Adam [30]. In future work GIBO can be extended to utilize state-of-the art policy gradient methods as an additional oracle for first-order information and help these methods reducing the variance of their gradient estimates through active sampling. Based on the posterior conditioned on all collected rewards, our algorithm can supply posterior gradient estimates and subsequent queries to evaluate.
To demonstrate the benefits of GIBO in a simple setup, we adopt the setting proposed by Mania et al. [1] as a baseline. Augmented Random Search (ARS) [1] assumes a black-box setting without access to gradient samples and estimates the gradient from the finite-difference of random perturbations, effectively solving RL problems. We replace the random sampling strategy of ARS with active sampling and the gradient estimation with a GP model. These changes improve the sample complexity and variance of ARS, especially when prior knowledge about the objective function is available.

heading: 3 Gradient informative Bayesian optimization
text: In this section, we introduce the proposed method GIBO. First, we define an acquisition function to reduce uncertainty for the Jacobian. Second, we outline the basic GIBO algorithm, including implementation choices.

heading: 3.1 Maximizing gradient information
text: We employ the BO framework to design a set of iterative queries maximizing gradient information. To this extend, we propose a novel acquisition function Gradient Information (GI) actively suggesting query points most informative for the gradient at the current parameters θt. Acquisition functions measure the expected utility of a sample point based on a surrogate model conditioned on the observed data. The utility U : Rd → R of our method depends on a Jacobian GP model, the objective’s observation data D, and the current parameter θt. It measures the decrease in the derivative’s variance at θt when observing a new point θ of the objective function. Hence, we define the utility as the expected difference between the Jacobian’s variance Σ′(θt|D) before and the Jacobian’s variance Σ′(θt|{D, (θ, y)}) after observing a new point (θ, y) αGI(θ|θt,D) = E [U(θ|θt,D)] = E [Tr (Σ′(θt|D))− Tr (Σ′ (θt| {D, (θ, y)}))], (4) where Tr denotes the trace operator and Σ′(θt|D) is the variance of the Jacobian’s GP model evaluated at θt
∇θJ ∣∣ θ=θt ∼ GP (µ′(θt|D),Σ′(θt|D)) . (5)
The Jacobian’s variance Σ′(θt|{D, (θ, y)}) depends on the extended dataset {D, (θ, y)}. A property of the Gaussian distribution is, that the covariance function is independent of the observed targets y as shown in Equation (3). Hence, we simplify the optimization over the expectation (see Appendix A.2) to
arg max θ αGI(θ|θt,D) = arg min θ
Tr (Σ′ (θt| [X, θ])), (6)
where the variance only depends on a virtual data set X̂ = [θ1, . . . , θn, θ] =: [X, θ]. In conclusion, the most informative new parameter θ to query is only dependent on where we sample next and is independent of its outcome f(θ) = y.
When we replace the Jacobian’s variance in (6) with (3) and leave out constant factors we get
θ∗ = arg max θ Tr
( ∇θtK(θt, X̂) ( K(X̂, X̂) + σ2nI )−1 ( ∇θtK(θt, X̂) )T) . (7)
Since the acquisition function only depends on the virtual data set, its optimization can be handled computationally efficient by performing the matrix inversion in (7) with Cholesky factor updates. Furthermore, since the Jacobian is a local property we can optimize (7) effectively using the of-theshelf optimizer supplied by BoTorch [31] (L-BFGS-B) using multiple restarts.

heading: 3.2 The GIBO algorithm
text: The guided sequential search of the acquisition function for gradient estimates divides the resulting algorithm into two loops: An outer loop for iterative parameter updates and an inner loop where the acquisition function queries points to increase gradient information. The basic algorithm is given in Alg. 1.
Algorithm 1 GIBO 1: Hyperparameters: stepsize η, hyperpriors for GP hyperparameters, number of iterations N and number of
samples for a gradient estimate M . 2: Initialize: place a GP prior on J(θ), set θ0 and D = {}. 3: for t = 0, . . . , N do . Parameter updates. 4: Sample noisy objective function: yt = J(θt) + t 5: Extend data set: D ← {D, (θt, yt)} 6: GP hyperparameter optimization. 7: for m = 1, 2, . . . ,M do . Sample points for a gradient estimate. 8: Get query point: θ̂ = argmaxθ̂ αGI(θ̂|θt,D). 9: Sample noisy objective function: ŷ = J(θ̂) + ω. . Optionally: Use a policy gradient method
for additional derivative observations. 10: Extend data set: D ← {D, (θ̂, ŷ)}. 11: Update the posterior probability distribution of∇θJ . 12: end for 13: θt+1 = θt + η · E [ ∇θJ ∣∣ θ=θt ] . Gradient ascent, or any other gradient based optimizer. 14: end for

heading: 3.3 Implementation choices
text: In the following, we introduce some details of our implementation of Algorithm 1 that further improve the performance and computational efficiency of our method.
Local GP model. Sparse approximation of GPs can be applied on BO when the computational burden of exact inference is too big [32]. In our case, however, we are only interested in estimating the local Jacobian at the current parameter θt. We define a sparse approximation of the posterior at the current parameter θt heuristically with the last Nm sampled points. Estimating a local model has the additional benefit of making the model selection and hyperparameter optimization simpler. We can approximate non-stationary processes locally by dynamically adapting hyperparameters.
Local optimization of GI. Following similar reasoning as above, we do not have to optimize the GI acquisition function globally since we expect informative points to be relatively close to the current parameter θt when using a SE kernel. Hence, we define our search bounds locally as [θt− δb, θt+ δb].
Gradient normalization. The gradient is normalized with the Mahalanobis norm using the lengthscales of the SE kernel. Hence, the stepsize η is adapted automatically to scale with the correlation between points. For the details see Appendix A.3.
State normalization. In the RL setting we can apply state normalization before we evaluate the policy to determine the next action. This has the same effect as data whitening for regression tasks and is beneficial when performing GP regression in unknown policy spaces. In case of a linear policy πθ : Rp → Rm, πθ(s) = As + b with bias b ∈ Rm, states s ∈ Rp, means of states µs ∈ Rp and variances of states σs ∈ Rp, state normalization can be defined by πθ ( s−µs σs ) = A ( s−µs σs ) + b = A · 1σs s−A · µs σs
+ b. State normalization is implemented in an efficient way that does not require the storage of all states. Also, we only keep track of the diagonal of the state’s covariance matrix with Welford’s online algorithm [33].

heading: 4 Empirical results
text: We empirically evaluate the performance of GIBO in three types of experiments. In the first experiment, we compare our algorithm on several functions sampled from a GP prior so that Assumption 1 is satisfied. In these within-model comparisons [34], we can show that GIBO outperforms the benchmark methods in terms of sample complexity and variance of regret, especially in higher dimension. In a second experiment, we perform policy search for a linear quadratic regulator (LQR) problem proposed by Mania et al. [1]. Finally, for RL environments of Gym [35] and MuJoCo [36], we show that GIBO reaches acceptable rewards thresholds faster and with significantly less variance than ARS. All data and source code necessary to reproduce the results are published at https://github.com/sarmueller/gibo.

heading: 4.1 Within-model comparison
text: We evaluate GIBOs performance as a general black-box optimizer on functions that satisfy Assumption 1. A straightforward way to guarantee this is by sampling the objective from a known GP prior. This approach has been called within-model comparison by Hennig and Schuler [34] but has likewise been used in other BO literature (e.g., [37, 38]). To show that GIBO scales particularly well to higher-dimensional search spaces, we analyze synthetic benchmarks for up to 36 dimensions.
The experiment was carried out over a d-dimensional unit domain I = [0, 1]d. For each domain, we generate 40 different test functions. For each function, 1000 values were jointly sampled from a GP prior with a SE kernel and unit signal variance. To cover the space evenly, we used a quasi-random
Sobol sampler. To perform experiments with comparable difficulty across different dimensional domains, we increase the lengthscales in higher dimensions by sampling them from the distribution `(d), introduced in Appendix A.4. The resulting posterior means were the objective function. All algorithms were started in the middle of the domain x0 = [0.5]d and had a limited budget of 300 noised function evaluations. The noise was Gaussian distributed with standard deviation σ = 0.1. A more detailed description of the experiments, including the true global maximum search and an out-of-model comparison, is given in Appendix A.4.
We compared our algorithm GIBO to ARS, CMA-ES [24] and standard BO with expected improvement [39] as acquisition function (‘Vanilla BO’). To ensure a fair comparison, domain knowledge was passed to the ARS and CMA-ES algorithms by scaling the space-dependent hyperparameters with the mean of the lengthscale distribution `(d). For details about the hyperparameters see Appendix A.7. The unknown hyperparameters were hand-tuned on a low dimensional example.
Fig. 3 shows the normalized difference between the global optimum and the function values of the optimizer’s best guesses. The within-model comparison shows that our algorithm outperforms vanilla BO on all test functions, except for the 4-dimensional domain. With a limited budget of 300 function evaluations the proposed method, GIBO, achieved lower regret than the baseline methods, especially in higher dimensions. Further, GIBO was able to reduce the variance of obtained regret significantly, as shown in Figure 4, which indicates a consistently better performance.

heading: 4.2 Linear quadratic regulator
text: The classic LQR with known dynamics is a fundamental problem in control theory. In this setting, an agent seeks to control a linear dynamical system while minimizing a quadratic cost. With available
dynamics, the LQR problem has an efficiently determinable optimal solution. LQR with unknown dynamics, on the other hand, is less well understood. As argued in Mania et al. [1], this offers a new type of benchmark problems, where one can set up LQR problems with challenging dynamics, and compare model-free methods to known optimal costs. We compare GIBO against ARS and LSPI [40] on a challenging LQR instance with unknown dynamics, proposed by Dean et al. [41]. The reader is referred to Appendix A.6 for a complete introduction to the setup.
Fig. 5 shows the frequency of stable controllers found and the cost compared to the optimal cost for GIBO, ARS, and LSPI. On the left in Fig. 5 we observe that GIBO requires significantly fewer samples than ARS, equivalent to LSPI, to find a stabilizing controller. But we note that LSPI requires an initial controller K0, which stabilizes a discounted version of the LQR problem. Neither GIBO nor ARS require any special initialization. All algorithms achieve similar regrets.

heading: 4.3 Gym and MuJoCo
text: Lastly, we evaluate the performance of GIBO on classic control and MuJoCo tasks included in the OpenAI Gym [35, 36]. The OpenAI Gym provides benchmark reward functions that we use to evaluate our policies’ performance compared to policies trained by ARS. Mania et al. [1] showed that deterministic linear policies, πθ : Rp → Rm, πθ(s) = As + b, are sufficiently expressive for MuJoCo locomotion tasks. Consequently, we define our parameter space by θ = (A, b) ∈ Rp×m+m. For the CartPole-v1 we need 4, for the Swimmer-v1 16 and for the Hopper-v1 36 dimensions. For all environments, we normalize the reward axis. For the Hopper environment, we additionally subtract the survival bonus and use state normalization; find further details in Appendix A.5. We hand-tuned the hyperparameter of GIBO within a reasonable degree, where the hyperparameter for ARS are taken from [1]. In the following, we use the reward over function evaluations (calls of RL environment) as evaluation metric for sample efficiency. We averaged the reported policy rewards over ten trials. In Fig. 6 we observe that GIBO reaches the reward thresholds faster and with significantly less variance than ARS.

heading: 4.4 Ablation study
text: In this section, we investigate different implementation choices of the GIBO algorithm. We conduct our ablation experiments on the within-model comparison with synthetic objective functions as well as on RL environments. Gradient normalization using the known GP lengthscales leads to a significant improve in mean performance as well as reduced variance, see Fig. 7. When optimizing policies for the RL benchmarks the GP lengthscales are not known and are learned during training. Fig. 8 shows that even in the case of learned hyperparameters gradient normalization proves to be important for the performance. We applied state normalization only to the Hopper environment and found that for this task it is not possible to learn well-performing policies without this extension. This shows that the normalization of an unknown policy space can be crucial for GP regression.

heading: 5 Conclusion
text: We introduce GIBO, a gradient-based optimization algorithm with a BO-type active sampling strategy to improve gradient estimates for black-box optimization problems. When the model assumptions of BO are satisfied, we show that the algorithm is significantly more sample-efficient, especially in higher dimensions, compared to baseline algorithms for black-box optimization.
Additionally, we show the benefits of active sampling and probabilistic gradient estimates with GIBO by solving popular RL benchmarks for which the model assumptions do not hold exactly. When compared to random sampling, GIBO is more sample efficient and has lower variance. Yet, the performance benefits are less pronounced in the RL task. This highlights that GIBO especially shines when prior knowledge is available while it still performs reasonably otherwise. Nonetheless, we want to remark that the prior biases the gradient estimates and wrong assumptions about the objective function can deteriorate performance. However, in some sense, all hyperparameters in RL algorithms encode some form of prior knowledge about the problem at hand. In our view, explicit probabilistic priors are an appropriate and intuitive form of prior knowledge to obtain, e.g., from domain knowledge or available data from prior experiments.
Since it is straightforward to include derivative observations into GIBO, we expect similar improvements for other existing RL methods when integrating our method as an additional layer between gradient estimators and optimizers. The proposed framework can suggest different exploration policies and combine all available data into a posterior belief over the Jacobian. For future research, we want to utilize GIBO with state-of-the-art actor-critic algorithms to improve sample complexity of these methods.
In a more general context, our active sampling methodology makes a step towards autonomous decision-making. GIBO decides on a learning experiment for the autonomous agent. Whenever a decision process is automated, the responsibility for legal and ethical consequences of these decisions must be resolved. However, we do not discuss how the decision-maker, GIBO, can be constrained to ensure compliance with regulatory requirements, which is a relevant aspect for future research.
Acknowledgments and Disclosure of Funding
The authors thank D. Baumann, P. Berens, A. R. Geist, H. Heidrich and F. Solowjow for their helpful comments and discussions. This work was supported in part by the Cyber Valley Initiative; the Max Planck Society; by the German Federal Ministry of Education Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A; and by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC number 2064/1 – Project number 390727645. The authors thank the International Max Planck Research School for Intelligent Systems for supporting A. von Rohr and S. Müller.

Review summaries:
The paper introduces GIBO, a modification of Bayesian Optimization (BO) that performs local policy search. Compared to classic BO, the strengths of this method are that it does not require storing all the past data points, making it more computational efficient, and that it is more stable when moving to higher dimensional problems. The authors compare their method with standard BO and Augmented Random Search, an approach based on finite difference policy search.

In this paper, the authors propose a new acquisition function for Bayesian optimization based on gradient information. A GP model estimates the Jacobian and its estimation is used for gradient update. This method is implemented in the setting of policy search for reinforcement learning. Empirical evaluations are provided for a synthetic setting, LQR and MuJoCo gym environments.

This paper concerns the high sample complexity of policy gradient methods and the computational complexity of Bayesian optimization. The authors propose a novel algorithm called GIBO (and its local variants) to minimize uncertainty in gradient estimation. Experiments are conducted and compared with some naive baselines.

This paper modifies the standard BayesOpt pipeline in 2 ways:  1. Defines a new acquisition function, the "Gradient Information" (GI) which is the reduction in the variance of the Jacobian variance of $J$. 2. Exploration and Exploitation become disjointed: During the BayesOpt process, a batch of $M$ queries are made via argmax of the GI acquisition, purely for exploration (rather than a mix of exploitation and exploration as is standard in BayesOpt), with the goal of producing a final gradient update step $\nabla_{\theta}J$.   This defines a "GIBO" algorithm, which is then experimentally evaluated over the following benchmarks:   1. Sampled functions from GP's 2. LQR Controllers 3. Linear controllers for standard Mujoco tasks.  which show that GIBO outperforms previous ES/ARS and standard BayesOpt methods especially in high dimensional settings, most likely due to the use of "local search"/gradients.