An Experimental Design Perspective on Model-Based Reinforcement Learning
Authors: Viraj Mehta
Biswajit Paria
Jeff Schneider
Stefano Ermon
Willie Neiswanger
Venue: ICLR 2022 Poster (2022)
Keywords: reinforcement learning
acquisition function
information gain

Abstract: In many practical applications of RL, it is expensive to observe state transitions from the environment. For example, in the problem of plasma control for nuclear fusion, computing the next state for a given state-action pair requires querying an expensive transition function which can lead to many hours of computer simulation or dollars of scientific research. Such expensive data collection prohibits application of standard RL algorithms which usually require a large number of observations to learn. In this work, we address the problem of efficiently learning a policy while making a minimal number of state-action queries to the transition function. In particular, we leverage ideas from Bayesian optimal experimental design to guide the selection of state-action queries for efficient learning. We propose an \emph{acquisition function} that quantifies how much information a state-action pair would provide about the optimal solution to a Markov decision process. At each iteration, our algorithm maximizes this acquisition function, to choose the most informative state-action pair to be queried, thus yielding a data-efficient RL approach. We experiment with a variety of simulated continuous control problems and show that our approach learns an optimal policy with up to $5$ -- $1,000\times$ less data than model-based RL baselines and $10^3$ -- $10^5\times$ less data than model-free RL baselines. We also provide several ablated comparisons which point to substantial improvements arising from the principled method of obtaining data.

Published as a conference paper at ICLR 2022
AN EXPERIMENTAL DESIGN PERSPECTIVE ON MODEL- -BASED REINFORCEMENT LEARNING
Viraj Mehta, Biswajit Paria, & Jeff Schneider Robotics Insitute & Machine Learning Department Carnegie Mellon University Pinsburgh, PA, USA (virajm, bparia, schneide) cmu. edu
Stefano Ermon & Willie Neiswanger Computer Science Department Stanford University Stanford, CA, USA (ermon, meiswanger (css wewsssstaford...m edu
ABSTRACT
In many practical applications of RL. it is expensive to observe state transitions from the environment. For example, in the problem of plasma control for nuclear fusion, computing the next state for given state-action pair requires querying an expensive transition function which can lead to many hours of computer simulation dollars of scientific research. Such expensive data collection prohibits application standard RL algorithms which usually require a large number of observations learn. In this work, we address the problem of efficiently learning a policy while making a minimal number state-action queries to the transition function. In particular, we leverage ideas from Bayesian optimal experimental design guide the selection of state-action queries for efficient learning. We propose an acquisition function that quantifies how much information state-action pair would provide about the optimal solution to Markov decision process. At each iteration, our algorithm maximizes this acquisition function, to choose the most informative state-action pair to be queried, thus yielding a data-efficient RL approach. We experiment with a variety of simulated continuous control problems and show that our approach learns optimal policy with up 5 1, 000x less data than model- based RL baselines and 10x 10 x less data than model-free RL baselines. We also provide several ablated comparisons which point to substantial improvements arising from the principled method of obtaining data.
INTRODUCTION
Reinforcement learning (RL) has suffered for years from curse of poor sample complexity. State-of- the-art model-free reinforcement learning algorithms routinely take tens of thousands of sampled transitions to solve very simple tasks and millions to solve moderately complex ones (Haarnoja et al., 2018; Lillicrap et al., 2015). The current best basee-baeed reinforcement learning (MBRL) algorithms are better, requiring thousands of samples for simple problems and hundreds of thousands of samples for harder ones (Chua et al., 2018). In settings where each sample is expensive, even this smaller cost can be prohibitive for the practical application of RI. For example, in the physical sciences, many simulators require the solution of computationally demanding spatial PDEs in plasma control (Breslau et al., 2018: 88; Char et al., 2019) or aerodynamics applications (Jameson & Fatica, 2006). In robotics, due to the cost of simulating more complicated objects (Heiden al., 2021). RL methods are typically constrained to fast but limited rigid-body simulators (Todorov et al., 2012). These costly transition functions prompt the question: "If we were to collect one additional datapoint from anywhere in the state-action space to best improve our solution to the task, which one would it he?" An answer to this question can be used to guide data collection in RI.
Across the fields of black-box optimization and experimental design, techniques have been developed which choose data to collect that are particularly useful in improving the value of the objective function of the problem. For example, Bayesian optimization (BO) focuses maximizing an unknown (black-box) function where queries are expensive (Frazier, 2018; Shahriari et al., 2015). More generally, Bayesian optimal experimental design (BOED) aims choose data collect which are maximally informative about the value of some derived quantity (Chaloner & Verdinelli. 1995). We aim to leverage these ideas for data-efficiency in reinforcement learning. Along these lines, several

---

Published as a conference paper at ICLR 2022
works in the realm of Bayesian RL address this problem in the sequential setting. A Bayes-adaptive MDP (Ross et al., 2007) constructs a modified MDP by augmenting the state space with the posterior belief of the MDP, leading to a policy that optimally trade off between acquiring more information and exploiting the knowledge it already has. However such an MDP is intractable to exactly solve in large spaces approximations and heuristics have been developed for the solution (Smith, 2007; Guez et al., 2012). A particularly relevant heuristic is the value of perfect information (VPI) from Dearden al. (1998), which attempts to capture the potential change in value of a state if the value of a particular action at that state was perfectly known, specifically in a tabular setting. However, VPI doesn't 't attempt to distinguish between states that are visited during the execution of the optimal policy and those that aren't. This is a critical distinction when collecting data in continuous spaces, as queries may be wasted learning an optimal policy in irrelevant parts of the state space.
Motivated by our opening question, in this paper we study the setting where the agent collects data by sequentially making queries to the transition function with free choice of both the initial state and the action. We refer to this setting transition-quee query reinforcement learning (TQRL) and formally define it in Section 3.1. Although this setting has been studied in the tabular case, to the best of our knowledge it has not been studied in the continuous MDP literature.
In this work, we draw a connection between MBRL and the world of BOED by deriving an acquisition function that quantifies how much information state-action pair would provide about the optimal solution to MDP. Like the techniques in Bayesian RL,, our acquisition function is able to determine which state-action pairs are worth acquiring in way which takes into account the reward function and the uncertainty in the dynamics. However, like the Adaative MDP and unlike the VPI heuristic, this function takes into account the current estimates of which states the optimal policy will visit and values potential queries accordingly. Furthermore, our acquisition function is scalable enough to apply to multidimensional continuous control problems. In particular, our acquisition function is the expected information gain (EIG) about the trajectory taken by an optimal policy in the MDP that would be achieved if we were to query the transition function at given state-action pair.
Finally, we assess the performance of our acquisition function as data selection strategy in the TQRL setting. Using this method we are able to solve several continuous reinforcement learning tasks (including nuclear fusion example) using orders of magnitude less data than a variety of competitor methods. In summary, the contributions of our paper are:
We construct novel acquisition function that quantifies how much information a state-action pair would provide about the optimal solution to a continuous MDP if the next state were observed from the ground truth transition function. Our acquisition function is able to select relevant datapoints for control purposes leading to improved data efficiency.
We propose a practical algorithm for computing this acquisition function and use it to solve continuous MDPs in the TQRL setting.
We evaluate the algorithm on five diverse control tasks, where it is often orders of magnitude more sample-efficient than competitor methods and reaches similar asymptotic performance.
RELATED WORK
Transition Query Reinforcement Learning
In many RL algorithms, data is collected by initializing policy at start state and executing actions in the environment in an episodic manner. Kearns et al. (2002) introduced the setting where the agent collects data by sequentially sampling transitions from the ground truth transition model by querying state and action of its choice, which they refer to as RL with access to a generative model. We refer to this setting for brevity as TQRL. This setting is relevant in a variety of real-world applications where there is simulator of the transition model available. In particular, we see the setting in nuclear modeled by solving large partial differential equations where 200ms of plasma time can take up to an hour in simulation (Breslau et al., 2018).
fusion research, where plasma dynamics
There is substantial theoretical work TQRL for finite MDPs. In particular, Azar et al. (2013) give matching lin-linear upper and lower PAC sample complexity bounds, a substantial speedup to the upper bound for the standard problem which is quadratic in state size (Kakade, 2003). This is achieved simply by the naive algorithm of learning a transition model by uniformly sampling the space and then performing value iteration on the estimate of the MDP for optimal policy. More
2

---

Published as a conference paper at ICLR 2022
recently, the bound for this setting was tightened to hold for smaller numbers of samples by Li et al. (2020), meaning that for any dataset size in a continuous problem, the PAC performance can be quantified. Finally, Agarwal et al. (2020) show that the naive 'plug-in' estimator used in the previous works is minimax optimal for this setting. In summary, this setting is thoroughly understood for finite MDPs and it gives sample complexity reduction from quadratic linear in the state space size.
To our knowledge there do not exist works specifically solving the TQRL setting for continuous MDPs. In this work, we give an algorithm specifically designed for this setting, which shows sample complexity benefits reminiscent of those theoretically shown in the tabular setting.
Exploration in Reinforcement Learning To encourage exploration in RI., agents often use an e- greedy approach (Mnih et al., 2013), upper confidence bounds (UCB) (Chen et al., 2017), Thompson sampling (TS) (Osband al., 2016), added Ornstein-Uhlenbeck action noise (Lillicrap et al., 2015), or entropy bonuses (Haarnoja et al., 201 8) to add noise to a policy which is otherwise optimizing the RL objective. Although UCB, TS, and entropy bonuses all try to adapt the exploration strategy to the problem, they all tackle which action to take from a predetermined state and don't t explicitly consider which states would be best to acquire data from.
An ideal method of exploration would be to solve the intractable Bayes-adaptive MDP (Ross et al., 2007), giving an optimal tradeoff between exploration and exploitation. Kolter Ng (2009); Guez et al. (2012) show that even approximating these techniques in the sequential setting can result in substantial theoretical reductions in sample complexity compared to frequentist PAC-MDP bounds as in Kakade (2003). Other methods stemming from Dearden et al. (1998; 1999) address this by using the myopic value of perfect information as a heuristic for similar Bayesian exploration. However, these methods don't scale continuous problems and don't provide way to choose states to query. These methods were further extended with the development of knowledge gradient policies (Ryzhov et al., 2019; Ryzhov & Powell, 201 1), which approximate the value function of the Bayes-adaptive MDP. and coormationndirecter sampling (IDS) (Russo & Van Roy, 2014), which takes actions based on minimizing the ratio between squared regret and information gain over dynamics. This was extended to continuous-sian solve fully continuous problems, operates in the rollout setting rather than TQRL, and computes the information gain with respect to the dynamics rather than some notion of the optimal policy. In similar spirit, Arumugam & Van Roy (2021) provide a further generalization of IDS which can also be applied to RL. One recent work very close ours is Lindner et al. (2021), which actively queries an expensive reward function (instead of dynamics as in this work) to learn a Bayesian model of reward. Another very relevant recent paper (Ball et al., 2020) gives an acquísition strategy in policy space that iteratively trains a data-collection policy in the model that trades off exploration against exploitation using methods from active learning. Achterhold & Stueckler (2021) use techniques from BOED to efficiently calibrate a Neural Process representation of a distribution of dynamics to a particular instance, but this calibration doesn't 't include information about the task. A tutorial on Bayesian RL methods can be found in Ghavamzadeh et al. (2016) for further reference.
finite-action settings in Nikolov et al. (2019). However, this work doesn't 't
Separate from the techniques used in RL for a particular task, several methods tackle the problem of unsupervised exploration (Schmidhuber, 1991), where the goal is to learn as much as possible about the transition model without task or reward funetion. One approach synthesizes a reward from modeling errors (Pathak et al., 2017). Another estimates learning progress by estimating model accuracy (Lopes et al., 2012). Others use an information gain-motivated formulation of model disagreement (Pathak et al., 2019; Shyam et al., 2019) as reward. Other methods incentivize the policy to explore regions it hasn'l It been before using hash-based counts (Tang et al., 2017), predictions mimicking a randomly initialized network (Burda et al., 2019), density estimate (Bellemare et al., 2016), or predictive entropy (Buisson-Fera there no reward function and are inefficient for the setting of this paper, as they spend time exploring areas of state space which can be quickly determined to be bad for maximizing reward on a task.
et al., 2020). However, these methods all assume that
Bayesian Algorithm Execution and BOED
Recently, a flexible framework known as Bayesian algorithm execution (BAX) (Neiswanger et al., 2021) has been proposed for efficiently estimating properties of expensive black-box functions, which builds off of a large literature from Bayesian Optimal Experiment Design (Chaloner & Verdinelli, 1995). The BAX framework gives a general procedure for sampling points which are informative about the future execution of an algorithm. In this paper, we extend this framework to the setting
3

---

Published as a conference paper at ICLR 2022
of model-ppeeicttee control, when have expensive dynamics (i.e. transition function) which treat as a black-box function in the BAX framework. Via this strategy, we are able to use similar techniques to develop acquisition functions for data collection in reinforcement learning.
Gaussian Processes (GPs) in Reinforcement Learning There has been substantial prior work using GPs in reinforcement learning. Most well-known is PILCO (Deisenroth & Rasmussen, 201 I), which computes approximate analytic gradients of policy parameters through the GP dynamics model while accounting for uncertainty. Most related our eventual MPC method (Kamthe & Deisenroth, 2018), 1), which gives principled probabilistic model -reedccvee control algorithm for GPs.
PRELIMINARIES
In this work deal with consist of a tuple (S, ... T, r. Pu, H) where S is the state space, AA is the action space, T is the - P(S) (using the convention that P(X)) is the set of probability measures over X), r: S x A S - & is reward function, (s is a distribution over S of start states, and H N is horizon. We always assume A,A, Po: H are known. We also assume the reward is known, though our development of the method can casily be generalized to the case where r is unknown. Our primary function of interest the transition function T,, which we learn from data, Our aim is to find a policy æ S - P(A) that maximizes the objective given below. We can describe the execution of π in the MDP as finite collection of random variables generated by 8p Pa and a; π (8:), 8;
discrete-time Markov decision processes (MDPs) which
transition function T
T( a;-1) for i E 1, ...... H. Then this objective can be written
(x) E Eplso.n 50.
r( at $6+1) Σ t=0
We aim to maximize this objective while minimizing the number of samples from the ground truth transition function T' that are required to reach good performance. We denote the optimal policy as π² argmax, (r)) which we can assume to be deterministic (Sutton & Barto, 1998) but not necessarily unique. Finally, we assume that there is some prior P(T) for which the posterior P(T D) is available for sampling given a dataset D.
3.1 TRANSITION QUERY REINFORCEMENT LEARNING (TQRL)
In the standard online RL setting, one assumes data D = {(s,, az, s() hien] must be collected in length-// trajectories (rollouts) where the initial state Su Po, and after an action a, is chosen, the next state s
8;+1 is sampled from T(s;, 0:) up to i H, at which point the process repeats.
In this work, we consider the TQRL setting. where the agent sequentially acquires data (s;, 0z, s() in arbitrary order by querying a state action pair (s,, a,) from S x A and recieving sample s ~ T(s, from the black-box transition function T (Kearns et al., 2002; Kakade, 2003; Azar al., 2013). The goal in both settings is to find a policy which optimizes the objective in Equation (1).
has been shown for finite MDPs (Azar et al., 2013) that the PAC sample complexity, which the number of samples required to identify with high probability a policy that achieves almost optimal value, of this setting is (((S) ignoring the PAC factors. This is notably better than the bound of ((())))41) in the online RL setting given in Section 8.3 of Kakade (2003). The improvement shown in finite cases suggests that there could be similar reductions available in a continuous setting.
3.2 MODEL-PREDICTIVE CONTROL
Though our acquisition function is derived for any policy search procedure including planning or model-free reinforcement learning algorithms, we focus on poeel-predictive control (MPC) as it is simple and requires minimal components to function. MPC is a standard technique in model-based reinforcement learning (Chua et al., 2018; Wang & Ba, 2020). Using estimated dynamics model
T : S x A + S and an optimization algorithm, an MPC strategy with planning horizon h e N choosing an action at a state 8o solves the planning problem
h
max Eg A
(~T(a)))))
Σ
r(ss ai 81+1)
(2)
E.A
4

---

Published as a conference paper at ICLR 2022
Planning is typically redone periodically. often every timestep, actions are executed in the real environment. The cross cross-entropy method (CEM) is often used to solve this optimization problem. In this work, we use an improved variant of CEM described in Pinneri et al. (2020). Here we will refer to Ty as the stochastic policy obtained by running MPC over a dynamics function T. The randomness in the policy is due to any randomness in T and the randomness used by the optimizer. We give details on the hyperparameters involved in this optimization problem and their values in Section B.
4 AN ACQUISITION FUNCTION FOR MODEL-BASER RL
We draw inspiration from BO and BOED in constructing acquisition function suitable for control applications. For our purposes, an acquisition function is computationally tractable function S x A - R that describes the marginal improvement in the performance of the policy the MDP (conditioned on all previously observed data) when observing one additional state itate-action pair (s, a) E S A. As acquisition functions are greedy, they aren't necessarily optimal data-selection strategies given a fixed budget, compared to non-myopic strategies such as solving the Bayes-adaptive MDP. However, greedy strategies using mutual information are tractable and are often effective due to the submodularity of the expected information gain. More specifically, our acquisition function is an expected information gain (EIG) or equivalently the mutual information (MI) between a query of our transition model and representation of the optimal policy, as we elaborate in this section.
typical approach in a Bayesian setting might be to gather data such that the entropy H[x*] of the belief of the optimal policy x² is minimized, However, a full specification of π includes the behavior of the policy in all parts of the state space including states that are not visited at all, or visited less often in rollouts of x² when the start state is sampled from Po- As result, not all points in the state space are equally important when learning an optimal policy aimed at maximizing the expected reward. Optimizing the entropy of xt would lead uniform treatment of all the points in the state space, and hence would be far from optimal for the standard goal in RL.
We instead propose minimize the entropy of the optimal trajectory T' { defined random vector of states generated by first sampling Su ~ Po then sampling a; = '-" (s;). $ ~ T(s;, a.) for H timesteps. The optimal trajectory is completely specified by X² and the randomness arising from the MDP. Furthermone, 7" contains the necessary information needed about the transition function T to solve the MDP, since any state that could ever be visited by n² is in the support of 7" We empirically observe that this leads to an efficient strategy for active RL.
The randomness in arises from three sources: the start state distribution po. the dynamics T constituting the aleatoric uncertainty, and the uncertainty in our estimate of the model T due to our limited experience which constitutes the epistemic uncertainty. The first two sources of uncertainty being alcatoric in nature cannot be reduced by experience. Our proposed acquisition function based on information gain naturally leads to reduction in the epistemic uncertainty about 7 as desired. Finally, our acquisition function for a given state-action pair (8, a) is given as
FIG,, (8, = Eyy (T),a
D] - H((" | DD ((s,0,s')) [
(3)
Eaorpo
E (x,,(D)
[H|T" [D, - H[("" DDU ((s,,,5))
Here we assume posterior model of the dynamics T(s, a D) for a dataset D we have observed. The second equality true because 80 1 s' 8, a. In this paper, we assume the MPC policy using the ground truth transition function is approximately optimal, i.e. कт RS π², though in principle n² could be approximated using any method. Of course, our method never actually has access to RT or n².
4.1 ESTIMATING EIG,, VIA POSTERIOR FUNCTION SAMPLING
For EIG,- to be of practical benefit, we must be able to tractably approximate it. Here we show how to obtain such an approximation. By the symmetry of MI, we can rewrite Equation (3) as
,. (s, a)= = Esorpo ((-() [[H[s' 8, a, D, spl - H|ss s, s,a,
(4)
Since Hi[s' s, a, D, sul = His' s. a. Dj doesn't depend on 7* or Su, we can simply compute as the entropy of the posterior predictive distribution P(s' s, D) given by our posterior over the transition function P(T D). In order to compute the other term, we must take samples
Ti P(7" D) To do this, we first sample m start states si from Pa (we always set m = 1 in
5

---

Published as a conference paper at ICLR 2022
15
Agent
Observe
Draw samples from postetior of?
Ron RL on samples
Environment
-10
Maximize EIG
Query (s, a)
-30 -30
-20
(a)
(b)
Figure 1: (a) A diagram of the BARL data-collection loop. (b) An illustration of the EIG,- computation over several sample paths of (multi-colors) sampled from P(r* D) for dataset of past queries (grey points). The optimizer (in pink) is a point that is maximally informative when learning model for crossing the path between the lava pools (orange rectangles) the goal (green).
experiments but derive the procedure in general) and for each start state independently sample n posterior functions Tij P(T D) from our posterior over dynamics models. We then run the MPC procedure on each of the posterior functions from si using Tú for T and v for x" (using our assumption that n" z TT), giving our sampled - This is an expression of the generative process for 7" described in the previous section that accounts for the uncertainty in T. Formally, we approximate EIG,- via Monte-Carlo as
ETG,- HH(S s, DJ vE(m) jE(m]
E
Hiss's,, T.J. DJ.
(5)
mn
Finally, we must calculate the entropy H[5'[8, T, D). For this, we follow similar strategy as Neiswanger et al. (2021). In particular, since T." is set of states output from the transition model, we can treat them as additional noiseless datapoints for our dynamics model and condition on them. In the following section we describe our instantiation of this EIG estimate, and how reinforcement learning procedures. Though inspired by the work cited here, we modify the computation of the acquisition function to factor pu irreducible source of uncertainty. We also extend the function being queried to be vector-valued.
use
5 BAYESIAN ACTIVE REINFORCEMENT LEARNING
In this work, take simple approach for nonlinear control in continuous spaces and assume Gaussian process (GP) prior P(T") to model the dynamics. Though computationally expensive, this choice ensures that we can easily approximate all necessary quantities. However, note that the development of the acquisition function is general and any Bayesian model could be used in principle.
The transition function T S A p(S) (dynamics) can be modeled with GP due its non- parametric nature and ability to capture uncertainties in T. The transition function takes state action pair (s, a) E gd=- as input, and produces a d-dimensional output denoting the next state. We model each of the d dimensions of the output independent GPs. More specifically, we model the change in state A (s. a) T(s, a) rather than the final state T(s, a) directly. This is helpful for continuous control problems since the state often changes by only small magnitude. Given observations D {(1,, as, s()). our approach requires posterior sample of the transition function conditioned on D. We follow the approach of Wilson et al. (2020), based sparse- GPs and random fourier approximations of kernels (Rahimi et al., 2007), allowing us to approximately but efficiently sample from the GP posterior conditioned on the observations.
Assuming access to generative model and an initial dataset D (for which, in practice, we use one randomly sampled datapoint (s, a, s')), compute EIG,. for D by running MPC on posterior function samples and approximate argmax.es.oe. EIG,- (s, a) by zeroth order approximation. Then we query s' T(s, a) and add the subsequent triple to the dataset D and repeat the process. To evaluate, simply perform the MPC procedure in Equation (2) and execute TE[T)D] the real

---

Published as a conference paper at ICLR 2022
environment. We refer to this procedure as Bayesian active reinforcement learning (BARL). Details are given in Algorithm 1 (here, denotes the uniform distribution) and a schematic diagram in Figure la. We discuss details of training hyperparameters and the GP model in Appendix A.
Algorithm Bayesian active reinforcement learning (BARL) using EIG,,
Inputs: transition function query budget b, number of points for optimization k, number of posterior function samples n. Initialize (Zo. Mo) US A), x/ T yo). and < {(20: yo, x/)}-
for i E [6] do
Sample posterior functions {TT)}=1 ~ P(T") D).
Sample start state So Po.
Compute {T e by executing MPC policy Try on the dynamics of Tj starting from 8o-
(x1, v1)........ (Ix, Uk) US x A).
(xt . + argmax (( EIG,- (2i, (xi,
+- DU ((x" ,y", )) where x T(x","" end for return Tp where f(s, a) = EEwPorp [T(s,a)]
6 EXPERIMENTS
The aim of our study of acquisition functions for RL is to reduce the sample complexity of learning good policies in continuous spaces, under expensive dynamics. Here, we demonstrate the effectiveness of using EIG,- to leverage transition queries by comparing against variety of state-of-the-art RL algorithms. particular, we compare the average return across five evaluation episodes across five runs with differing random seeds of each algorithm on five continuous control problems data is collected. We also assess the amount of data taken by each algorithm to 'solve' the problem, which is taken to mean performing well our MPC procedure using the ground truth dynamics. Our proposed method, BARL., greatly outperforms other methods across the board. In particular, BARL uses 5 1, x00 less data to solve problems than state-of-the-arr model-based RL algorithms and 10s 100 x less data than model-free RL algorithms. In this section we primarily focus on the performance of the controller, and section A. we also discuss the runtime of the algorithm.
Comparison Methods. We use as our model- comparison methods in this work PETS (Chua al., 2018) as implemented by Pineda al. (2021), which does MPC using probabilistic ensemble of neural networks and particle sampling for stochastic dynamics and a similar MPC method using the mean of the same GP model we use for BARL to execute T. to collect data as in the standard RL setting. We also compare against PILCO (Deisenroth & Rasmussen, 201 1), which also leverages a GP to directly optimize a policy that maximizes an uncertainty-awan long term reward. For model-free methods, we use Soft Actor-Critic (SAC) (Haarnoja et al., 2018), which is an actor-critic method that uses an entropy bonus for the policy to encourage evaluation, TD3 (Fujimoto et al., 2018) which addresses the stability questions of actor-critic methods by including twin networks for value and several other modifications, and Proximal Policy Optimization (PPO) (Schulman et al., 2017), which addresses stability by forcing the policy change slowly in KL so that the critic remains accurate. As a baseline TQRL method and to better understand the GP performance, we use a method we denote EIGT, which chooses points which maximize the predictive entropy of the transition model to collect data. We believe that when given access to transition queries many unsupervised exploration methods like Pathak et al. (2019); Shyam et al. (2019) or methods which value information gain over the transtion function (Nikolov et al., 2019) would default to this behavior.
Control Problems. We tackle five control problems: the standard underactuated pendulum swing-up problem (Pendulum-v0) from Brockman et al. (2016)), cartpole swing-up up problem. 2D lava path navigation problem, a 2-DOF robot arm reacher problem with 8-dimensional state (Reacher-v2 from Brockman et al. (2016)), and a simplified beta tracking problem from plasma control (Char et al., 2019; Mehta et al., 2020) where the controller must maintain a fixed normalized plasma pressure using as GT dynamics a model learned similarly to Abbate et al. (2021). The lava path is intended to test stability and exploration of algorithms. The goal is to reach a fixed goal state from a narrow uniform distribution over start states. As shown in Figure 1b, the state space contains 'lava' region which gives large negative rewards for every timestep. When not in lava, the reward is simply the negative squared distance to the goal, forcing the agent navigate the goal quickly possible.
7

---

Published as a conference paper at ICLR 2022
Groad
Postiki
Pursite
175
0.75
150
0.52
0.30
125
-1.33
-15
-4.55
-0.0
170
Figure 2: Progress and sampled points BARL, showing trajectories through the normalized state space of the pendulum problem from four fixed start points for the optimal controller and MPC with 20 and 40 datapoints, respectively. It is clear that even with very few points the controller is able to closely track the optimal paths. Here color is used only disambiguate the trajectories.
Environment
BARL
MPC
FIGT
PETS
SAC
TD3
PPO
Lava Path
11
41
41
600
N/A
N/A
N/A
Pendulum
16
46
46
5200
6000
57000
13000
Cartpole
91
161
121
1625
31000
18000
N/A
Beta Tracking
96
36
N/A
300
9000
6000
16000
Reacher
251
751
N/A
700
23000
13000
N/A
Table 1: Sample Complexity: Median number samples across seeds required reach 'solved' performance, averaged across trials. We determine 'solved' performance by running an MPC policy (similar the one used for evaluation) on the ground truth dynamics to predict actions. We record 'N/A' when the median run is unable to solve the problem by the end of training.
Since there is narrow path through the lava, we want to explore a policy which crosses efficiently and safely. Agents who fail to find this solution will be forced to go around, incurring penalties.
We see in both the sample complexity figures in Table 1. the learning curves in Figure 3, and visually in Figure that BARL leverages EIG,, significantly reduce the data requirements of learning controllers on the problems presented. We'd like to additionally point out several failure cases of related algorithms that BARL. avoids. Though performs well on the simplest environments (pendulum and cartpole), EIG, suffers from an inability to focus on acquiring data relevant to the control problem and not just learning dynamics as the state space becomes higher- dimensional in the reacher problem, or less smooth in the beta tracking problem. The MPC method performs reasonably well across the board and competitive with BARL on the plasma problem but requires relatively more samples in smaller environments where the model uncertainty can point to meaningfully underexplored areas. PETS is strong across the board but suffers from more required samples due both its neural network dynamics model and its inability to make transition queries.
All algorithms besides BARL suffer substantial instability on the lava path problem, which designed be challenging to explore in a sequential fashion and require precise understanding which areas are safe to enter. BARL manages to learn where it is safe to operate in handful of queries, which is an exciting result and will bear further investigation. Figure 1b gives some intuition as why: points are initally queried close to the start and as those dynamics are understood they are subsequently queried farther and farther along the execution paths. This allows BARL use transition queries to avoid traversing well-understood areas of state space to reach the areas which are worth learning. We see a speedup sample complexity reminiscent some of the theoretical improvements given in the prior work discussed on tabular methods.
move from quadratic to linear, which mirrors
We further support our assertion that BARL is picking 'meaningful' points to the control problem by the evidence in Figure 4. Here, BARL is able to solve the reacher problem while EIG, is not. However, BARL has much worse model predictions on random data than EIG, while doing a much better job modeling data used by the MPC procedure. Clearly, the EIG,- acquisition function captures in some way which data would be valuable to acquire to not just learn about the transition function but actually solve the control problem. We this pattern across other tasks well. In
8

---

Published as a conference paper at ICLR 2022
Coured Patoraacc Praddaa:
Cornd Paforeacce Cutpedo
Charsd Peckerazze Budter
-12
1..0
Perkeraazze
Pafarmance Traidea
MPC
P20
150
EICr
PETS 42
SAC
Figure 3: Learning Curves RL methods, showing control performance averaged across seeds. In each, the z-axis is logarithmic scale to account for widely varying data requirements. We see that though most algorithms end up reaching roughly the same performance on each task, BARL is substantially more efficient in most cases. The shaded region is the standard error of the average performance across the seeds. We additionally include a plot of the performance of the PILCO algorithm (Deisenroth & Rasmussen, 2011) Pendulum. PILCO makes assumptions about the initial state distribution and suffers from numerical instability under long control horizon so we were unable to reach representative performance on the other problems.
Cuntral and Modsligg Performance Roncher
Moai bust un Canont
Moid MSE ca Bualhur
10-9
-15.
8x 10-1
37.5
10:19
30:00
500
1500
1500
Eestronmeer
Elactrvorannt
Figure 4: For single run of BARL. and of EIGT using the same prior model, we evaluate control performance, as well as modeling error, on both the predictions used by the MPC procedure and on set of points uniformly sampled from the state-action space.
section B.I, study whether the acquisition function controller on posterior samples of the dynamics. Our experiments show that FIG,- seems to work well even when the policy used to generate 7* is suboptimal.
here is able to work with suboptimal
7 DISCUSSION AND FUTURE WORK
In this work, we proposed an acquisition function for reinforcement learning, and applied the setting of TQRL., leading to novel algorithm for addressing the problem of efficient data collection RI. We experimented with several control problems and demonstrated that this approach leads to substantial improvements with respect to the sample efficiency.
However, there are some drawbacks to this method as well. Computing the proposed acquisition function relies on executing the entire control algorithm over a set of posterior function samples, leading to high computational requirements. In our current implementation we use Gaussian processes to model the transition function, which computationally expensive and do not scale well to higher dimensions and large datasets. In the future, we plan to extend this idea to use other types of Bayesian models such Bayesian neural networks and take advantage of GPU compute for better scalability.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We would like to acknowledge anonymous reviewers for valuable feedback. VM acknowledges Swapnil Pande for providing the idea for and implementation of the Lava Path environment and lan Char for providing the trained model for the Beta Tracking environment. This work was funded in part by DOE grant number DE-SC00214 140021114. WN acknowledges the helpful feedback from members the Ermon Group studying RL. WN was supported in part by NSF (H655156)) ONR (N000141912145) AFOSR (FA95501910024), ARO (W91 DOE NOE-AC02-76SFOOD and Sloan Fellowship.
REFERENCES
J. Abbate, R. Conlin, and E. Kolemen. Data-driven profile prediction for DIII-d. Nuclear Fusion, 61 (4):046027, mar 2021. doi: 10. 10.10888/171114332//b00888. URL https: http://oooooogooo http://ii..g//..11///111 1741 1741-4326/abe085
Jan Achterhold and Joerg Stueckler. Explore the context: Optimal data collection for context- conditional dynamics models. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 3529-35601 3337. PMLR, 13-15 Apr 2021. URL html.
https: http:///pccceedneg.eee.e mlr
www.e.ll..r..s..ee..meee.eeeerre press v1 30/ http://11///////////e/ee www.aachtthhccc
Alekh Agarwal, Sham Kakade, and Lin F. Yang. Model-based reinforcement learning with generative model is minimax optimal. In Jacob Abernethy and Shivani Agarwal (eds.), Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pp. 67-83. PMLR, 09-1099111102 12 Jul 2020. URL https: /tthttpsss/prroceddiio v125 2/25///arwaa. 120.. html.
mlr press/
Dilip Arumugam and Benjamin Van Roy. The value of information when deciding what to learn, 2021.
Mohammad Gheshlaghi Azar, Rémi Munos, and Hilbert J Kappen. Minimax pac bounds the sample complexity of reinforcement learning with a generative model. Machine learning, 91(3): 229-349,15 2013.
Philip Ball, Jack Parker-Holder, Aldo Pacchiano, Krzysztof Choromanski, and Stephen Roberts. Ready policy one: World building through active learning, In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 591-601. PMLR, 13-18 Jul 2020. URL ptttps://proceedi wwweeeiiinnsmm...... www. press. pttprsss/vi/////ee vt1//vv1////// htl1//bblll00 html.
Mare G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. CoRR, abs/1606.01866, 2016.
URL httt:://aaaaae
org/ abs /1606 01868.
Joshua Breslau, Marina Gorelenkova, Francesca Poli, Jai Sachdev, Alexei Pankin, Gopan Perumpilly, and USDOE Office of Science. Transp, 6 2018. URL https: /wwww//wwww/////////////// osti wwt.os.t.....bbbltoo/ 1489900.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Mona Buisson-Fenet, Friedrich Solowjow, and Sebastian Trimpe. Actively learning gaussian process dynamics. In Alexandre M. Bayen, Ali Jadbabaie, George Pappas, Pablo A. Parrilo, Benjamin Recht, Claire Tomlin, and Melanie Zeilinger (eds.), Proceedings of the 2nd Conference on Learn- ing for Dynamics and Control, volume 120 of Proceedings of Machine Learning Research, pp. 5-15. PMLR, 10-11 Jun 2020. URL https::///// http://pooeedddeeee mtt.....s/ buisson-
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representationss 2019. URL https: wttprrrewwweeeeeeeooooerteeaeeee
//openreviee ew n forum? d= 1 IJJnR5mm.
10

---

Published as a conference paper at ICLR 2022
Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: A review. Staristical Science, pp. 273-304, 1995.
Ian Char, Youngseog Chung, Willie Neiswanger, Kirthevasan Kandasamy, Andrew O Nelson, Mark Boyer, Egemen Kolemen, and Jeff Schneider. Offline contextual bayesian optimization. Advances in Neural Information Processing Systems, 32:4627 4638, 2019.
Richard Y. Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. UCB and infogain exploration via g-ensembles. CORR, abs/1706.01502, 2017. URL http:///aaax www.aawwww................oooete org.org///aa... haag///b///// 1706. 01502.
Kurtland Chua, Roberto Calandra, Rowan McAllister. and Sergey Levine. Deep rein- forcement learning in a handful of trials using probabilistic dynamics models. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL https::/ppoceeddeeett peeeeiinnnnneeuiseeeeeeeeeereeee $de568f8597b940 311 Paee0950c--pperree http://aagg.a..s..spppaa///////a
Richard Dearden, Nir Friedman, and Stuart Russell, Bayesian q-learning. Aaaifiaai, pp. 761-768, 1998.
Richard Dearden, Nir Friedman, and David Andre. Model-based bayesian exploration. CoRR, abs/1301.6613 1999. URL http: htta/axxvvv.r.oraa/e//////a 11011 6690.
Marc Peter Deisenroth and Carl Edward Rasmussen. Pilco: A model-based and data-efficient approach policy search. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'11, 11, pp. 465-472, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306195,
Peter Frazier. A tutorial on bayesian optimization. arXiv preprint 18X7v:1807.0255 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation emror in actor- critic methods. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 1587-1596. PMLR, 10-15 Jul 2018. URL https: /pwoced/odeeeeeeeeeo ings ml www...prss/// / fujimoto Ba. html.
Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforcement learning: A survey. CoRR, abs/1 (0/1104494466 2016. URL 04436.
/609. 609.
Arthur Guez, David Silver, and Peter Dayan. Efficient bayes adaptive reinforcement learning using sample-based search. In Proceedings of the 25th International Conference on Neural Information Processing Systems Volume 1, NIPS 12, pp. 1025-1033, Red Hook, NY, USA, 2012. Curran Associates Inc.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning. pp. 1861-1870. PMLR, 2018.
Eric Heiden, Miles Macklin, Yashraj S Narang, Dieter Fox, Animesh Garg, and Fabio Ramos. DiSECU: A Differentiable Simulation Engine for Autonomous Robotic Cutting. In Proceedings of Robotics: Science and Systems, Virtual, July 2021. doi: 10. N0.156077RSS.2221... 200.070...
Antony Jameson and Massimiliano Fatica. Using computational fluid dynamics for aerodynamies, 2006.
Sham Machandranath Kakade. On the sample complexity of reinforcement learning. University of London, University College London (United Kingdom), 2003.
Sanket Kamthe and Mare Deisenroth. Data-efficient reinforcement learning with probabilistic model predictive control. In Amos Storkey and Fernando Perez-Cruz (eds.), Proceedings of the Twenty-Fing First International Conference Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pp. 1701-1710. PMLR, 09-11 Apr 2018. URL https: / proceed ngs ml r. press v84 he 8a html.

---

Published as a conference paper at ICLR 2022
Michael Kearns, Yishay Mansour, and Andrew Y Ng. A sparse sampling algorithm for near-optimal planning in large markov decision processes. Machine learning, 49(2):193-22 -081 2002.
J Zico Kolter and Andrew Y Ng. Near-bayesian exploration in polynomial time. In Proceedings of the 26th annual international conference on machine learning, pp. 513-520, 2009.
Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Breaking the sam- ple size barrier in model-based reinforcement learning with a generative model. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neurul Information Processing Systems, volume 33, pp. 12861-12872. Curran Asso- ciates, Inc., 2020. URL
nww.ipuurp.... cc wttppccee/00c////////e// /fle// Papeeee
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.0220 2015.
David Lindner, Matteo Turchetta, Sehastian Tsshiatschek, Kamil Ciosek, and Andreas Krause. Information directed reward learning for reinforcement learning. 35, 2021.
Manuel Lopes, Tobias Lang, Mare Toussaint, and Pierre-yves Oudeyer. Exploration in model-based reinforcement learning by empirically estimating learning progress. In F. Pereira, C. C. Burges, L.. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https: htttttsss/pppcceeiinnt http///proeeeen...eee/ee/o nwwrips. paper/2012/ htt.://///...e///000000332365522 3btpeef1133aad1133133233
daper.pdE.
David JC MacKay et al. Bayesian nonlinear modeling for the prediction competition. ASHRAE transactions, (00(2):1033110)) 1994.
Viraj Mehta, Ian Char, Willie Neiswanger, Youngscog Chung, Andrew Oakleigh Nelson, Mark D Boyer, Egemen Kolemen, and Jeff Schneider. Neural dynamical systems: Balancing structure and flexibility in physical prediction. arXiv preprint arXiv:2006. /2682, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13Xiv:131..5602, 2013.
Radford M Neal. Bayesian Learning for Neural Networks. PhD thesis, University of Toronto, 1995.
Willie Neiswanger, Ke Alexander Wang, and Stefano Ermon. Bayesian algorithm execution: Esti- mating computable properties of black-box functions using mutual information. In International Conference Machine Learning. PMLR, 2021.
Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, and Andreas Krause. Information-directed exploration for deep reinforcement learning. In International Conference Learning Representa- tions, 2019. URL /ttttts:://ppereeiiieee net Forum? dd= 33 9Km.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In D. Lee, M. Sugiyama. U. Luxburg. I. Guyon, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Asso- ciates, Inc., 2016. URL 8d8818c8e14 46644474111135555775 Papeee pdf.
wwwpinggn..uuirrre nwwripu httpppppccpppppe//1/////////
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference machine learning, pp. 2778-2787. PMLR, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning. volume 97 of Proceedings of Machine Learning Research. pp. 5062-5077 PMLR, 09-15 Jun 2019. URL https: httpt:://prcceeeeeeieieee /ttp///eeic..c.cppe/eepp/os...sp mlr. press//97. pathak1 9a. html.
12

---

Published as a conference paper at ICLR 2022
Luis Pineda, Brandon Amos, Amy Zhang, Nathan O. Lambert, and Roberto Calandra. Mbrl- lib: A modular library for model-based reinforcement learning. Arxiv, 2021. URL httpa: //arxiv. http.//////......//////////..t.a 101599.
Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Joerg Stueckler, Michal Rolinek, and Georg Martius. Sample-efficient cross-entropy method for real-time planning. arXiv preprint arXiv:2008. 06389, 2020.
Ali Rahimi, Benjamin Recht, et Random features for large-scale kernel machines. NIPS, volume 3, pp. 5. Citeseer, 2007.
Stephane Ross, Brahim Chaib-draa, and Joelle Pineau. Bayes-adaptive pomdps. In NIPS, pp. 1225-1232, 2007.
Daniel Russo and Benjamin Van Roy. Learning to optimize via information-directe sam- pling. In Z. Ghahramani, M. Welling, Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Asso- ciates, Inc., 2014. URL httpps://proceeeeeee 301
www..ips.c./ppprr/2211.... Paper. pdf.
Ilya O Ryzhov and Warren B Powell. Information collection on graph. Operations Research, 59(1): 188 188-201, 2011.
Ilya O Ryzhov, Martijn RK Mes. Warren B Powell, and Gerald van den Berg. Bayesian exploration for approximate dynamic programming. Operations research, 67(1):198-22) 224,, 2019.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp. 222-227, 1991.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (00.:06770.064777 2017.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1): 148-175 2015.
Pranav Shyam, Wojeiech Jaškowski, and Faustino Gomez. Model-based active exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5779-5788. PMLR, 09-15 Jun 2019. URL https: http::/s//ppc/eeee/g//e/ www.mmrr...s.......mmmmmmmmmmm.e v shyam1 9a. html.
Trey Smith. Probabilistic planning for robotic exploration. Carnegie Mellon University. 2007.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, USA, 1998. ISBN 0-262-19398-1 URL http:/ttp/// /twp////ww///////////////////. cs. ua wewlbbera... ca/ $7Esutton/book www.k/to....ooooo theook/tteeeeeee wwttheebeee html.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schul- man, Filip De Turck, and Pieter Abbeel. #exploration: A study of count-based explo- ration for deep reinforcement learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Ben- gio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. neurips wwtpccpppp///oeee /hash/
2753-2762, 2017. URL testtps:/proceeeeeee 3a20f62a0a 525.aaa5577000335022222222 feed- Abstract html.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 0026-5033, 2012. doi: 00.119//ROOS.2022.6661
13

---

Published as a conference paper at ICLR 2022
Tingwu Wang and Jimmy Ba. Exploring model-base planning with policy networks. In International Conference on Learning Representations, 2020. URL https:: http////wwwwweeeeeeeeeeeeeeeeeee id=Hlexf 6 64wH.
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for regression. 1996.
James Wilson, Viacheslav Borovitskiy, Alexander Terenin, Peter Mostowsky, and Mare Deisenroth. Efficiently sampling functions from gaussian process posteriors. In International Conference on Machine Learning, pp. 00292-10302 PMLR, 2020.
14

---

Published as a conference paper at ICLR 2022
Control Problem
Pendulum
Cartpole
Lava Path
Reacher
Beta Tracking
Budget
200
300
100
1500
300
# of points for optimization
1000
1000
1000
1000
1000
# of posterior function samples n
15
15
15
15
15
Table BARL hyperparameters used for each control problem.
Control Problem
Pendulum
Cartpole
Lava Path
Reacher
Beta Tracking
Sample n times
20.7 18.5
-3.9
33.4 35.8
216 276
7.5 4.9
Evaluate EIG,- at k points
3.3 5.5
12.7
7.6 9.3
21.3 147
7.5 13.6
Total for BARL Iteration
24.24.04
46.2 46.5
40.1 45.1
237 423
18.5
Evaluation for one episode
7.2
2.5 10.3
18 47.9
26.2 913.7
0.9 3.5
Table 3: Runtime in seconds for the phases of the BARL algorithm all problems when the author's s 24-core CPU machines. The ranges given show the runtime for the operation at the beginning and the end of training, as some operations run longer as more data added.
A TRAINING DETAILS
We vary our budget based on our understanding of how much data would be required to solve the problem. The other hyperparameters of the BARL algorithm are constant but listed for completeness in Table 2.
For all of our experiments, we use a squared exponential kernel with automatic relevance determina- tion (MacKay et al., 1994; Neal, 1995). The parameters of the kernel were estimated by maximizing the likelihood of the parameters after marginalizing over the posterior GP (Williams Rasmussen, 1996).
To optimize the transition function, we simply sampled set points from the domain, evaluated acquisition function, and chose the maximum of the set. This set was chosed uniformly for every problem but Reacher, for which we chose a random subset of Uj Uy Tú (the posterior samples of the optimal trajectory) since the space of samples is 10-dimensional and uniform random sampling will not get good coverage of interesting regions of the state space.
A.1 RUNTIME DETAILS
Based on these choices and the MPC hyperparameters below in Section B, each of these problems results in varying runtime for the BARL algorithm. In Table 3, report the time taken for an iteration of BARL and how it breaks down by step. We give these as ranges, as the computational time requires increases the learning process continues since GP computational costs scale with the size the dataset. We also include for completeness the time taken to execute the MPC policy on the ground truth problem, which is strictly part of the BARL algorithm but still relevant to practitioners.
Clearly, BARL is relatively slow algorithm computationally. But in settings where samples scarce, BARL is much cheaper than alternative methods which might use less compute for the RL algorthms but require many more samples. When compared to the costs of running an hour-long simulation or running costly experiments, spending a few minutes computing the acquisition function seems like a good use of resources.
B MPC DETAILS
As we've ve discussed, we use preeciireeiceive control in this work to choose actions which maximize future reward given model of the dynamics. In particular Method from Pinneri et al. (2020) to solve the optimization problem in Equation 2, which uses several tricks including colored noise samples and caching reduce the number of queries to the planning
the improved Cross-Entropy
15

---

Published as a conference paper at ICLR 2022
Control Problem
Pendulum
Cartpole
Lava Path
Reacher
Beta Tracking
Base number of samples
25
30
25
100
Number of clites
3
6
15
Planning horizon
20
15
20
15
Number of iCEM iterations
3
5
3
Replanning Period
6
1
6
2
Table 4: Hyperparameters used for optimization in MPC procedure for control problems.
model. There is natural trade-off in any search method between computational cost and quality of actions found terms of predicted reward. In this work, we chose hyperparameters for each task that were as computationally light as possible which attained similar reward to larger hyperparameters when executing MPC using the ground-truth model (TT, in our terms). As recommended by the original paper, we use 3 = 3 for the scaling exponent of the power spectrum density of sampled noise for action sequences, the amount of caching.
1.25 for the exponential decay population size, and 5 = 0.3 for
manually tuned the base number of samples, planning horizon, number elites to take from the sampled action sequences, number of iterations of planning, and the replanning period of the model. Here we give the ultimate values for those parameters, which were used for all ablations using our GP model and MPC. The values used across all experiments for each problem given in Table 4.
B. ROBUSTNESS OF EIG,, TO SUBOPTIMAL CONTROLLER
In order to compute E - in this work, we perform model-predictive control on posterior transition function samples (execute Try on Ty. in our notation). We assume that ттς close the optimal policy for the MDP with transition function Ty. However, this assumption could lead to pathologies in the method if it doesn't 't hold in practice. In this section, we emprically investigate the consequences of using suboptimal controller when finding samples of 7" on posterior samples of the transition function.
In order to understand the sensitivity of EIG,- to the MPC policy executed on posterior samples, we ran experiments where we reduced the planning budget or horizon for the posterior function policy in order to see whether the acquisition function fails. In particular, we ran the reacher and cartpole experiments from the main paper with varying MPC budgets for the posterior function policy TTy and a fixed MPC budget at test time. This allows us to isolate the effectt of a suboptimal policy generating samples of y"
On the reacher experiment, we vary the number of CEM iterations ranging from 1 to This is straightforwardyy linked the amount of search the policy conducts before executing action. On the cartpole experiment, we varied the length of the planning horizon, which affects how far in the future the policy will consider actions both cases, we see in Figure 5 that performance hardly changes as the budget for MPC is reduced. Only on the reacher problem when the number of CEM iterations is reduced to (effectively reducing CEM to simple random search) do we see a significant drop in performance. This supports the notion that the quality of the approximation of the optimal policy in EIGo is not critical the performance of the acquisition function as data selection strategy. We also plot in the figure the performance of clear that if we were to execute these degraded policies at test time, they would be much worse. We find it interesting and intend to further study the robustness of using cheaper policies to decide where to acquire data.
is deciding what is the good immediate next action. In
an MPC controller on the ground truth dynamics with these reduced MPC budgets.
C DESCRIPTION OF CONTINUOUS CONTROL PROBLEMS
Lava Path. The lava path has 4-dimensional state (position and velocity) and 2-dimensional action (an applied force in the plane). The goal is to reach a fixed goal state from a relatively narrow uniform distribution over start states. As shown in Figure 1b, the state space contains 'lava' region which
16

---

Published as a conference paper at ICLR 2022
Control Performmnee Cartpole
Control Performancs Rescher
-20
-12
+60
-10
-1.
-16
1000
-173
Ty
-200
231
100
2.31
Entrmoaaca Suge
Brrimsaact Shipe
Figure 5: Performance of BARL when MPC budget for posterior function samples is varied while MPC test time budget is held constant. The error regions are the standard error of the return seen across 5 trials of the policy. The dashed lines are the performances that MPC with the equivalent hyperparameters achieves if executed test time given the ground truth dynamics.
gives very large negative rewards for every timestep. Other than when in lava, the reward is simply the negative squared distance to the goal, forcing the agent navigate to the goal as quickly as possible. Since the lava has a narrow path through, the actor is forced to explore policy which will realize that is safe and efficient to cross. Agents who fail to find this solution will be forced to go around, incurring penalties.
Pendulum. The pendulum swing-up problem is the standard one found in the OpenAl gym (Brock- man et al., 2016). The state space contains the angle of the pendulum and its first derivative and action space simply the scalar torque applied by the motor on the pendulum. The challenge in this problem that the motor doesn't 't have enough torque to simply rotate the pendulum up from all positions and often requires reward function here penalizes deviation from an upright pole and squared torque.
and-forth swing to achieve a vertically balanced position. The
The cartpole swing-up problem has 4-dimensional state (position of the cart and its velocity, angle of the pole and its angular velocity) and a I-dimensional action (horizontal force applied to the cart). Here, the difficulty lies in translating the horizontal motion of the cart into effective torque on the pole. The reward function here is negative sigmoid function penalizing the distance betweent the tip of the pole and centered upright goal position.
Cartpole.
Reacher. The reacher problem simulates a 2-DOF robot arm aiming to move the end effector to a randomly resampled target provided. The problem requires joint angles and velocities as well as an indication of the direction of the goal, giving an 8-dimensional state space with the mentioned 2-D control. Our results on this problem are particularly encouraging as they show that BARL. can scale to some problems with higher dimensionalities.
Beta Tracking (Nuclear Fusion). Finally, the beta tracking problem has 4-dimensional state consisting of the current normalized plasma performance Sy in the DIIIID tokamak. By is given by an appropriately normalized ratio between the plasma pressure and the magnetic pressure and is a common figure of merit in fusion energy research. In addition 3n the state space contains its most recent change as well as the current power injection level and its most recent change. The action is the next change in the power injection level. The "ground-truth" dynamics for this problem are given by a neural network model learned from data processed as in Abbate et al. (2021). ). Control is done at a timestep of 200ms and the reward function is the negative absolute deviation from B, Reliably controlling plasmas to sustain high performance is major goal of research efforts for fusion energy. and though this is very much simplification the problem, we intend to extend and apply BARL to more realistic settings in the immediate future.
17


Reviews:
[{"review_id": "XaUE4EUmNf0", "reviewer": "Reviewer_n6c6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Simple and easy to understand method for active learning in TQRL, that is based on the principle of expected information gain (EIG) from Bayesian experimental design. While not optimal, similar greedy maximization approaches have a strong track record in related problems such as active information gathering.\n- A nice reformulation of the problem as reducing uncertainty about the optimal trajectory, which leads to a decomposition according to the sources of uncertainty.\n- Empirically demonstrated to reach good performance with only a handful of samples, outperforming model-free methods and in some cases model-based competitors such as PETS.\n\nWeaknesses:\n1. The empirical evaluation only concerns simple domains where a Gaussian process (GP) model easily suffices. It is not clear if the methodology will scale to larger or more complex models, where evaluation of the EIG objective function itself might pose challenges.\n2. To the best of my understanding, the performance of the proposed method is also capped to the performance of an MPC solution in a problem. This also shows in the results, e.g., Fig. 3 showing PETS on beta tracking and reacher obtaining a higher final reward, SAC & TD3 in other domains. While this is not necessarily an issue in the domains considered in the paper, generally MPC cannot be guaranteed to reach the same performance as an optimal solution. This is a limitation of the proposed approach.\n3. There are a few points that need additional clarification, details below.\n\n\nDetailed comments:\n\n\nQuestions on Algorithm 1:\n- Why is # of points for optimization k an input when k does not appear in the algorithm?\n- Line 2: does U refer to uniform distribution? In the initialization, should s_0 be sampled from p_0?\n- Line 7: I don't understand why X is drawn from an uniform distribution over S x A. Then, X would be a pair of a state and an action. This does not really make sense for how it is used on Line 8 as a set.\n\nPresentation of the results:\n- Fig 1 (b), the colored trajectories are distracting, are they necessary?\n- The significance of the colors in Fig. 2 is not clear.\n- Could you clarify what exactly is the MPC method shown in Table 1, Fig. 3? I did not get it from the paragraph \"Comparison Methods\".\n\n\n\nOther/Minor:\n- This comment is just for information and needs no response: Considering the GP models, submodularity arguments might be helpful to further justify the greedy sampling approach. See, e.g., Krause et al. \"Near-optimal sensor placements in Gaussian processes: Theory, efficient algorithms and empirical studies\", JMLR 2008. \n- Just before Eq. (1), there range 1, ..., H for the index i has a mismatch. That is, a_0 is not accounted for and no action should be drawn at step H.\n- \"The target the minimal information...\" -- sentence seems ill-formed, please revise\n- Eqns. (3)-(4): to remove any guessing, please mention here that blackboard H is the entropy\n- \"Given sufficient computational resources and an accurate model, model-predictive control will typically find very good if not optimal actions in an MDP.\" -- could you point to a source for this claim and/or make precise what this means? AFAIK, no error bound for the performance of MPC compared to an optimal solution is known.\n\n**Added after the author response**\nI thank the authors for their response, which clarifies my remaining concerns. I have increased the score to reflect this.", "summary_of_the_paper": "This paper proposes an active learning method for transition query RL (TQRL) in continous domains. In TQRL, an agent queries the dynamics of the transition function at arbitrary state-action pairs in order to estimate the system dynamics. The objective is to query pairs that are most useful in solving the underlying control problem. The paper formulates TQRL as an expected information gain (EIG) maximization given the current dataset of observed state transitions, and applies posterior function sampling to evaluate EIG. The method is empirically evaluated in five continuous control tasks, with improvements in empirical sample complexity.", "summary_of_the_review": "This paper proposes a simple but seemingly effective method for model-based RL in cases where obtaining transitions for training from the real environment is extremely costly. The method is based on well-founded principles of Bayesian experimental design. The paper itself is clearly written and easy to understand, and technically sound. The empirical results sufficiently clearly show the advantages of the proposed methods. More realistic experimental settings could be considered in future work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G0foYWY_CqJ", "reviewer": "Reviewer_gjNV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n1. The paper is very well written.\n\n2. The proposed framework and algorithm are well motivated with repeated grounding in practical considerations around problems where a ground truth model exists but is expensive to compute samples from. Independent of empirical evaluation, there are reasonable intuitive arguments for why one might expect improvements in sample complexity.\n\n3. A wide variety of relevant baselines were compared against, including some ablations of the proposed algorithm. The empirical results presented appear promising.\n\n## Weaknesses\n\n1. The current setup they used for BARL appears relatively computationally expensive (e.g., using GPs to model the dynamics, running MPC over every posterior transition function, etc.)\n\n2. The domains are on the simpler end, which are reasonable for the scope of the paper but leave questions about how the approach might scale to larger domains.\n\n3. Only five random seeds were evaluated. Recent criticisms have emphasized how five seeds might not be representative of the data distribution- can the authors comment on the statistical significance of the results? Along those lines, I couldn't find in the text what the shaded regions represent in Figure 3? Can the authors further comment on how hyperparameters were selected across each algorithm and whether they provide a fair comparison?\n\n## Minor stuff which didn't affect the review:\n\n\"the the\" near the start of Section 3.1\n\"evaluatation episdoes\" near the start of Section 6", "summary_of_the_paper": "This paper uses insight from Bayesian optimal experimental design and define an *acquisition function* which quantifies information gain about an MDP's optimal solution given a state-action pair. Such an acquisition function would provide a mechanism for selecting which transitions to query from a ground truth model in model-based RL, in hopes to greatly improve sample efficiency of a learning algorithm. They propose a method for computing this acquisition function, and evaluate it empirically.", "summary_of_the_review": "Taking the above into account, I recommend acceptance of the paper. While I feel the empirical evaluation could be improved, I find the motivation and intuitive arguments behind the proposed algorithm convincing enough to be a useful contribution in the literature.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3CbaLNYH9tf", "reviewer": "Reviewer_X5up", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n-The paper provides a principled and interesting approach to an impactful problem. The idea of approximating the optimal policy and then computing information gain on the sequence of states visited by the optimal policy is both novel and clever.\n\nWeaknesses:\n\n-The paper uses independent GPs to model the transition dynamics of the MDP. Could multi-task GPs be used here to improve the model’s capability? For example, [1] allows for  high-dimensional, correlated outputs, which could be used to scale your method to larger problems.\n\n-I’m curious about the sensitivity to the approximation of the optimal policy. For example, such a study could be carried out on a domain where the optimal policy is computable and then various approximations to it (e.g., MPC, MCTS, myopic policies) could be used as the approximation.\n\n-Even though the motivation is for problems with expensive transitions, wall times for the new method should be included so that readers can get a sense for how computationally expensive the method is.\n\n-There are references that might be relevant. Please see [2, 3] from the OR literature.\n\n[1] https://arxiv.org/abs/2106.12997\n[2] https://pubsonline.informs.org/doi/abs/10.1287/opre.2018.1772\n[3] https://pubsonline.informs.org/doi/abs/10.1287/opre.1100.0873\n", "summary_of_the_paper": "This paper considers the very relevant (in my opinion) problem of data-efficient RL and is potentially applicable in nearly all situations where RL is to be applied in the real-world rather than a simulator. The authors propose an acquisition function based on expected information gain (EIG) approach. The overall framework consists of a (1) Gaussian process model of the transition dynamics, (2) an MPC approximation of the optimal policy, (3) the EIG acquisition function where the target variable is the sequence of states visited by the optimal policy. Empirical results are shown for a number of continuous control domains.", "summary_of_the_review": "The paper develops a novel method for an important problem. I, however, have a few comments above that I believe would improve the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0j7H0AizODm", "reviewer": "Reviewer_AKFY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea of the paper is really interesting. Most model based RL approaches are based on data efficiency, so clearly the idea of applying active learning is interesting. Furthermore, the paper focuses on active learning for policy optimality, instead of the most traditional exploration/exploitation approach.\n\nHowever, I have trouble following the approach in the TQRL setting that the authors proposed. In that setting, there is already a generative model/oracle/simulator that allows querying arbitrary transitions. If we already have a transition oracle that we can query, what is the point of learning a transition model? Most planning approaches, like the ones presented in the paper and compared against are based on MPC with sampling of the transition model. If the control part was done in a fashion that required the full distribution, I could imagine some possible contribution, but given the use of CEM, it seems that the paper is trying to solve a problem that it is already solved at the beginning. For example, the 3 papers that the paper cites to justify the TQRL setting, one of them is for planning, not RL, and the other 2 are about theoretical analysis on sample complexity. There is not a practical implementation in an RL setting.\n\nBeing familiar with the RL and BO literature, the paper can be hard to follow. For example, the acquisition function is presented (s4) before the context where it is applied (s5). Also, in the Algorithm it is hard to find where the reward is optimized until the reader realizes that the 3rd line in the loop (executing pi) implies computing the optimal policy by MPC/CEM and then, generating the trajectories. It is not just executing a policy. The acquisition function also needs more clarification: the authors choose to compute O instead of a fully defined pi* to avoid sampling unnecessary states that are never visited from p0. This affirmation implies that both the initial distribution and the transition distributions are considerably limited. If the initial state is unknown or the noise level is reasonably high, the advantage of using O vanishes, because the trajectories end up visiting most of the state space. But the choice of O might have some limitations: what happens if the initial policy/trajectories are completely wrong, driving the queries to irrelevant state/actions for the goal? Is there any convergence guarantee?\n\nThe state of the art seems to miss an important part of literature on intrinsic motivation, curiosity, which have addressed similar problems before. I recommend the works of Pierre-Yves Oudeyer, Jürgen Schmidhuber, etc. For example, the work on Lopes et al. seems very related to the paper at hand.\n\nLopes M, Lang T, Toussaint M, Oudeyer PY. Exploration in model-based reinforcement learning by empirically estimating learning progress. InNeural Information Processing Systems (NIPS) 2012 Dec 3.\n\nFurthermore, given that the surrogate model in this paper is based on GPs, there should be a mention and comparison to GP-based model based RL methods, such as PILCO, which has been, and they still are, the standard of sample efficient RL. \n\nDeisenroth MP, Fox D, Rasmussen CE. Gaussian processes for data-efficient learning in robotics and control. IEEE transactions on pattern analysis and machine intelligence. 2013 Nov 4;37(2):408-23.\n\nIn fact, the MPC methods that the authors include in the comparison, seems to be a stripped down implementation of this work.\n\nKamthe S, Deisenroth M. Data-efficient reinforcement learning with probabilistic model predictive control. InInternational conference on artificial intelligence and statistics 2018 Mar 31 (pp. 1701-1710). PMLR.\n\nIn contrast, it seems that the inclusion of model-free methods in the comparison just adds noise to the plots. Maybe you can include the performance in the table and supplementary material and, instead, include GP-based RL in the plots, like those mentioned before.\n\nMinor details:\n\n-For fair comparison, given that the competitors cannot exploit the transition oracle, it would be interesting to see performance BARL in environmental steps, including the steps needed to reach the desired query. That is, the algorithm is able to sample arbitrarily any transition, but the agent has to reach it sequentially.\n\n-The most complex problem in terms of exploration space seems to be the reacher which is 8D. How does the algorithm behave in higher dimensional problems, such as half-cheeta, ant, humanoid, etc. Or even some exploration challenging scenarios such as the ant-maze.\n\n-The authors use EIGt as a proxy for the exploration algorithms. Why not use the full algorithms instead? For example, the code of Shyam et al is available.\n\n-What is the implementation of the cartpole swing-up? Why haven't the authors used a standard benchmark such as the OpenAI gym version with mujoco or pybullet?\n\n-Second paragraph, third line -> repeated “on”\n\nUpdated review: \n----\nAfter reading the updated version of the paper, I see the relevance of the proposed method and the TQRL setup: The paper is now much more readable and I have updated my score accordingly.\n\nMy only remaining major concern is with the comparison. Being a novel setup I understand the complexity of a fair comparison and the addition of PILCO clearly solves some of the questions. However, there are still some questionable choices. For example, in the submitted code, there are multiple versions of the cartpole and it is unclear which one is used, and the fact that there are some versions with the names of specific methods could be interpreted as different versions where used (which I assume was not the case). It's true that the OpenAI gym cartpole is not swing-up, but there is one in pybullet and there are many well known benchmarks such as dm_control. Using previously defined, unaltered tasks, like the Pendulum and Reacher would make all the experiments much more stronger.", "summary_of_the_paper": "This paper presents an active learning algorithm for model based reinforcement learning. The algorithm is based on a new acquisition function that finds the most informative transition to query an oracle model in order to achieve better performance with respect to the optimal task.", "summary_of_the_review": "The main idea of the paper is interesting and relevant for the field. However, I feel that the setting and the comparison needs to be revisited to have practical application.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors introduce an exploration method for RL according to experimental design perspective via designing an acquisition function, which quantifies how much information a state-action pair would provide about the optimal solution to a Markov decision process, and the state-action that maximizes such acquisition function will be used for sampling for policy update. The empirical evidences show the proposed method is promising. 

Since most of the reviewers support the paper, I recommend acceptance of this submission. 

However, besides the questions raised by the reviewers, e.g., computation cost and planning quality from CEM, there is a major issue need to be clarified in the paper:

>The algorithm designed for RL with generative model, which makes the state-action reset can be conducted (this is sometimes impossible in practice where the agent must start from initial state). This is different from the common RL setting, and thus reduce the complexity of RL. This should be emphasized in the paper. Meanwhile, for a fair comparison, this should be explicitly specified in experiment setting.
Decision: Accept (Poster)