The guide and the explorer: smart agents for resource-limited iterated batch reinforcement learning
Authors: Othman Gaizi
Albert Thomas
Balázs Kégl
Gabriel Hurtado
Venue: Submitted to ICLR 2023 (2023)
Keywords: Model-based reinforcement learning
Dyna
exploration
planning
offline
growing batch
iterated batch

Abstract: Iterated (a.k.a growing) batch reinforcement learning (RL) is a growing subfield fueled by the demand from systems engineers for intelligent control solutions that they can apply within their technical and organizational constraints. Model-based RL (MBRL) suits this scenario well for its sample efficiency and modularity. Recent MBRL techniques combine efficient neural system models with classical planning (like model predictive control; MPC). In this paper we add two components to this classical setup. The first is a Dyna-style policy learned on the system model using model-free techniques. We call it the guide since it guides the planner. The second component is the explorer, a strategy to expand the limited knowledge of the guide during planning. Through a rigorous ablation study we show that combination of these two ingredients is crucial for optimal performance and better data efficiency. We apply this approach with an off-policy guide and a heating explorer to improve the state of the art of benchmark systems addressing both discrete and continuous action spaces.

Under review as a conference paper at ICLR 2023
THE GUIDE AND THE EXPLORER: SMART AGENTS FOR RESOURCE-LIMITED ITERATED BATCH REINFORCE- MENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Iterated (a.k.a growing) batch reinforcement learning (RL) growing subfield fueled by the demand from systems engineers for intelligent control solutions that they can apply within their technical and organizational constraints. Mlodel-based RL (MBRL) suits this scenario well for its sample efficiency and modularity. Re- cent MBRI. techniques combine efficient neural system models with classical planning (like model predictive control; MPC). In this paper we add two com- ponents to this classical setup. The first is a Dyna-style policy learned on the system model using model-free techniques. We call it the guide since it guides the planner. The second component is the explorer, a strategy to expand the limited knowledge of the guide during planning. Through rigorous ablation study we show that combination of these two ingredients is crucial for optimal performance and better data efficiency. We apply this approach with off off-policy guide and heating explorer to improve the state of the art of benchmark systems addressing both discrete and continuous action spaces.
INTRODUCTION
John is telecommunication engineer. His day job is to operate mobile antenna. He has about forty knobs to turn, in principle every five minutes, based on about a hundred external and internal system observables. His goal is keep some performance indicators within operational limits while optimizing some others. In the evenings John dreams about using reinforcement learning (RL) to help him with his job. He knows that he cannot put an untrusted model-free agent on the antenna control (failures are very costly), but he manages to convince his boss to run live tests couple of days every month.
John's 'ss case is arguably on the R&D table of lot of engineering companies today. AI adoption slow, partly because these companies have little experience with AI, but partly also because the algorithms we develop fail to address the constraints and operational requirements of these systems. What are the common attributes of these systems?
They are physical, not getting faster with time, producing tiny data compared what model-free RL (MFRL) algorithms require for training.
System access is limited to small number of relatively short live tests, each producing logs that can be used evaluate the current policy and can be fed into the training of the next.
They are relatively small-dimensional, and system observables were designed to support human control decisions, so there is no need to filter them or to learn representations (with the exception when the engineer uses complex images, e.g., driver).
Rewards are non-sparse, performance indicators come continually. Delays are possible but usually not long.
The RL setup that fits this scenario is neither pure offline (batch RL; Levine et al. (2020)), since interacting with the system is possible during multiple live tests, nor pure online, since the policy can only be deployed a limited number times on the system (Fig 1). After each deployment on the real system, the policy is updated offline with access all the data collected during the

---

Under review as a conference paper at ICLR 2023
Live test on real system with fixed learned policy
Add live test data
Limited number of live tests
Deploy
Learn offline
Data history
Policy
Figure 1: Iterated (a.k.a growing) batch RL. The policy updated offline between scheduled live tests where the latest learned policy can be deployed on the real system to collect data and further improve itself at the next offline update.
previous deployments, each update benefiting from a larger and more diverse data set. This setup also assumes that the policy cannot be updated online while it is being deployed on the system. We refer to it as iterated batch RL (also called growing batch (Lange et al., 2012) or semi-batch (Singh are interested in model-
al., 1995; Matsushima et al., 2021) ) in the literature). Furthermore, based RL (Deisenroth & Rasmussen, 2011; Chua et al., 2018; Moerland et al., 2021). With limited access to the real system, model of the system transitions can be used to simulate trajectories, either at decision-tion time to search for the best action (decision-time planning, e.g., model predictive control (MPC)) or when learning the policy (background planning, e.g., Dyna-style algorithms that learn model-free agents with the model), which makes model-based RL sample efficient (Chua et al., 2018; Wang et al., 2019). Additionally, model-based RL works well on mmall-dimensional systems with dense rewards, and the system model (data-drive) simulator, digital twin) itself is object of interest because it can case the adoption of data- algorithms by systems engineers.
Given robust system model, simple model predictive control (MPC) agents using random shooting (RS; Richards (2005); Rao (2010)) or the cross entropy method (CEM; de Boer et al. (2004)) have been shown perform remarkably well on many benchmark systems (Nagabandi et al., 2018; Chua et al., 2018; Wang et al., 2019; Hafner et al., 2019; Kégl et al., 2021) and real-life domains such as robotics (Yang al., 2020). However these methods can be time consuming run at decision-time on the real system, especially when using large models and with the search budgets required for complex action spaces. On the other hand, implementing successfully the seemingly elegant Dyna- style approach (Sutton, 1991; Kurutach et al., 2018; Clavera et al., 2018; Luo et al., 2019), when we learn fast reactive model-free agents on the system model and apply them on the real system, remains challenging especially on systems that require planning with long horizon. Our main findings are that i) the Dyna-style approach can still be an excellent choice when combined with decision-time planning or, looking at it from the opposite direction, ii) the required decision-time planner can be made resource efficient by guiding it with the Dyna-style policy and optionally bootstrapping with its associated value function: this allows an efficient exploration of the action search space (given the limited resource budget) where fewer and shorted rollouts are needed to find the best action to play. We also innovate on the experimental framework (metrics, statistically rigorous measurements). so we can profit from the modularity of the Dyna-style approach by tuning ingredients (model, the MFRL guide policy, exploration, planning, bootstrapping: explained in Section 3.2) independently. This modular approach makes engineering easier (as opposed to monolithic approaches like Alp- haZero (Silver al., 2017)), which is an important aspect if we want to give the methodology to non-expert systems engineers.
1.1 SUMMARY OF CONTRIBUTIONS
A conceptual framework with interchangeable algorithmic bricks for iterative batch re- inforcement learning, suitable to bring intelligent control into slow, physical, low- dimensional engineering systems and the organizational constraints surrounding them.
A case study that indicates that the combination of a Dyna-style approach and resource- limited planning can mutually improve each other.
An ablation study that helped us find the combination of a neural model, a bootstrapping off- off-policy algorithm guide, and heating explorer, which brings significant improvement over vanilla agents (MPC, pure Dyna without planning) on both discrete and continuous actions systems.
2

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
The MBRL subfield has seen a proliferation of powerful methods, but most of them miss the spe- cific requirements (solving problems irrelevant in this scenario like representation learning or sparse rewards) and missing others (limited and costly system access; time constraints for action search; data taking and experimentation through campaigns, live tests; safety) (Hamrick, 2019).
The Dyna framework developed by Sutton (1991) consists in training an agent from both real ex- perience and from simulations from a system model learned from the real data. Its efficient use of system access makes it natural candidate for iterated batch RL. The well- limitation of this approach is the agent overfitting the imperfect system model (Grill et al., 2020). A first solution is to done in Model-Based Policy Opti-
use short rollouts the model to reduce error accumulation mization (MBPO; Janner et al. (2019)), Another solution is to rely on ensembling techniques for the model as done in ME-TRPO (Kurutach et al., 2018) and MP-MPO (Clavera et al., 2018). Instead of learning the model and then the policy from the model, Stochastic Lower Bound Optimization (SLBO; Luo et al. (2019)) alternates between model and policy updates. In our ITERATED schema (Fig 2), this strategy would couple the LEARN and ACTOR steps, which we do not study in this paper. Finally, Yu et al. (2020) and Kidambi et al. (2020) use Dyna-style approach in the context of pure batch RL where no further data collection and therefore no further model updates are assumed.
The idea of using a guide and a value function when planning is not novel (Silver et al., 2017; Schrittwieser et al., 2020; Wang & Ba, 2020; Argenson Dulac-Arold, 2021; Sikchi al., 2021).
We were greatly inspired by these elements in our objective of building smarter agents as they can make the search more efficient and lead to a better performance. POPLIN-A (Wang & Ba, 2020) relies on behavior cloning (using only real experience, unlike our Dyna-style approach that mainly uses the model), but their decision-time planner is similar to our approach. During the planning, they add a Gaussian noise to the actions recommended by deterministic policy network and update the noise distribution using a CEM strategy. In a similar way our GUIDE&EXPLORE strategy also adds a carefully controlled amount of noise to the recommended actions. Our results highlight the importance of a well-calibrater exploration. Additionally, our planner does not require to specify the amount of noise beforehand. Argenson & Dulac-Arnold (2021); Lowrey et al. (2019); Sikchi et al. (2021) found that bootstrapping with a value estimate improves the performance of simple guided MPC strategies (see also Bhardwaj et al. (2021)). The popular AlphaZero (Silver et al., 2017) and MuZero (Schrittwieser et al., 2020) algorithms also rely on guide and a value function for their Monte Carlo Tree Search (MCTS). The principal issue of MuZero (Schritwwieser et al., 2020) in our micro-data iterated batch RL context is that it does not control the number of system access steps: needs simulate a lot from the real environment to establish the targets for the value function. In these two algorithms the guide is updated from the results obtained during the search that it guided, a procedure known as dual policy Iteration (Anthony et al., 2017; Sun et al., 2018). Furthermore, most of the computation to grow the search tree is done sequentially which results in a slower planner compared to the natural parallelized implementation of our agent. We prefer experimenting with Dyna-style approaches first leverage popular MFRL. algorithms and defer the study of dual policy iteration to future work.
Our results show that decision-time planning is an important ingredient, claim already made by Hamrick et al. (2021) and Springenberg et al. (2020) among others. Hamrick et al. (2021) use MuZero run their ablation study while we prefer using an explicit model for practical reasons explained in the introduction. Springenberg et al. (2020) study where additional computation is best spent between policy update and decision- -ime planning. In our case we are however less concerned by computational resources required to update the policy, as it is done offline, but rather by the time spent at decision-time while interacting with the real system.
3
THE FRAMEWORK FOR DESOURCE-LIMTTER ITERATIVE BATCH RL
3.1 THE FORMAL SETUP
Let Tr = ((81, (1),,.)....... (sT,aT)) aT)) be a system trace consisting of T steps of observable-action pairs (sc. a:): given an observable 8; of the system state at time t, an action a: was taken, leading to a new system state observed as 8,41- The observable vector 8, = (s!...., s,²²) contains d,
numerical categorical variables, measured the system at time l. The action vector a, contains
3

---

Under review as a conference paper at ICLR 2023
d numerical or categorical action variables, typically set by control function a, ((st) of the current observable 8t (or by a stochastic policy a ~ T(:)); we will also use the notation : 8, ~ a²). The performance of the policy is measured by the reward r, which is function of 8, and dg. Given a trace Tr and a reward ry obtained at each step i, we define the mean reward as ((T))) T Ei=rr.' re. The transition p (st, Qz) ~ St+1 can be deterministic (a function) or probabilistic (generative). The transition may either be the real system p = Preat or system model p p. When the model is probabilistie, besides the point prediction E (5(ss++1(*s)) (t)) also provides information on the uncertainty of the prediction and/or to model the randomness of the system (Deisenroth & Rasmussen, 2011; Chua et al., 2018). Finally, in the description of the algorithms we index a trace T ((5), (s),...,(s7,ar))) ar) as follows: for E {L...., ,T, T. i) 8, and [ at.
3.2 A NOTE ON TERMINOLOGY
By model we will consistently refer to the learned transition or system model á (never to any policy). Rollout is the procedure of obtaining a trace T from initial state 81 by alternating a model real system p and a policy π (Fig 2). We decided to rename what Silver et al. (2017) calls the prior policy guide since prior clashes with Bayesian terminology (as, e.g., Grill et al. (2020); Hamrick et al. (2021) also note), and guide expresses well that the role of this policy is to guide the search at decision time. Sometimes the guide is also called the reactive policy (Sun et al., 2018) since it is typically an explicit function or conditional distribution & 8 ~ that can be executed or drawn from rapidly. We will call the (often implicit) policy 8 plan/search the actor (sometimes also called the non-reactive policy since it takes time to simulate from the model, before each action). Planning generally refers the use of a model ý to generate imaginary plans and in that sense, planning is part of training the guide. However, in the rest of the paper we will use the term planning to refer to the guided search procedure that results in acting on the real system (this is sometimes called decision- time planning). We will explicitly use the term background planning to refer to the planning used at training time as it is done in Sutton & Barto (2018) and Hamrick et al. (2021).
resulting from the guided
3.3 EXPERIMENTAL SETUP: THE ITERATED BATCH MBRI
For rigorously studying and comparing algorithms and algorithmic ingredients, we need to fix not only the simulation environment but also the experimental setup. We parameterize the iterated batch RL loop (the pseudocode in Fig 2 is the formal definition) by four parameters:
the number of episodes N,
the number of system access steps T per episode,
the planning horizon L, and
the number of generated rollouts n at each planning step.
N and are usually set by hard organizational constraints (number N and length T live tests) that part of the experimental setup. Our main goal is to measure the performance of our algorithms a given (and challengingly small) number of system access steps N x T for given planning budget (n and L) determined by the (physical) time between two steps and the available computational resources.
In benchmark studies, we argue that fixing N, T, n. and is important for making the problem well defined (taking some of the usual algorithmic choices out of the input of the optimizer), affording meaningful comparison across papers and steady progress of algorithms. As in all benchmark de- signs, the goal is to make the problem challenging but not unsolvable. That said, we are aware that these choices may change the task and the research priorities implicitly but significantly (for exam- ple, a longer horizon I, will be more challenging for the model but may make the planning easier although more expensive), so it would make sense to carefully design several settings (quadruples N - T
L) on the same environment.
We use the mean reward (as opposed to the total reward, a.k.a return), since it is invariant to episode length and its unit is more meaningful to systems engineers.
First practical note: the pseudocode in Fig 2 and the subroutines in Figs 3-5 contain our formal definition. They are ordered top-down, but can also be read in reverse order according to the reader's preference.
4

---

Under review as a conference paper at ICLR 2023
RT P. S):
1 1
2 fort - to T:
D for ' steps draw action from policy update trace D draw next state
3 at ~- ((()
4 TU (8,, a4)
5 8141 ~ p(1, ax)
6 return T
ITERATEDMBRL (Preat: So (1) NN,,, n):
81 ~ So
draw initial state random trace > for N episodes learn system model
2 (()) & ROLLOUT (T((), Preal: T)
3 for r + to N:
4 p(7) < LLEARN U+ T(r))
5 x(1) ( -ACTOR (T(0), T( (1-1)) p(()
1,, ((()) ,0A n.))
6 81 Su
D draw initial state new trace
7
T(++1) ROLLOUT (T(1), Preal 8 81
return U T((
The iterated batch MBRL loop. Preal (8t, (e) ~ 8(41 is the real system (so Line what dominates the cost) and p (st, az) ~e 8c+-1 is the transition in ROLLOUT that can be either the real system Preal or the system model p. Sg is the distribution of the initial state of the real system. T()) 8, ~ Or an initial (typically random) policy and in ROLLOUT क : 8, < a, is any policy. N is the number of episodes; 7' is the length of the episodes; L is the planning horizon and is the number of planning trajectories used by the actor policies x x(7), whereas 1 ...... T is the system (or model) access step index. LEARN (probabilisticc or deterministic time-series forecasting) algorithm applied to the collected traces and ACTOR a wrapper of the various techniques that we experiment with in this paper (Fig 4), An ACTOR typically updates π(r-1) using the model f(()) in background-planning loop, but can also access the initial policy (0) and the trace - T(()) collected on Preal up to episode T.
Figure 2:
L...., N is the episode index supervised learning
HEATINGEXPLORE [))
for to
((ale)1/T;
a is discrete
Ea 1:177
2
pı(als)
(( if a is continuous.
3 return [p:]
Figure 3: Heating exploration strategy. HEATINGEXpLORR heats the guide action distribution
E(a s) to n different temperatures. The temperatures [T,] are hyperparameters.
Our main operational cost system access step so we are looking for any-time algorithms that achieve the best possible performance any episode for given decision-time planning budget (n and L). Hence, in the MBRL iteration (Fig 2), use the same traces (T)), rolled out in each iteration (Line 7), to i) update the model ş and the actor policy (Line 4 and 5) and ii) to measure the performance of the techniques (Section 3.5).
3.4 MODEL-BAL BASED ACTOR POLICIES: GUIDE AND EXPLORE
Our main contribution is Dyna-style GUIDE&EXPLORE strategy (Fig 4). This strategy consists in learning a guide policy & for the decision-ting planner (TRAJOPT in Fig 5) using a model-free RL technique the model ý and the traces collected the real system T. It is known that the guide ç. executed as an actor _ = on the real system, does not usually give the best performance (we also confirm it in Section 4), partly because & overfits the model (Fig in Kurutach al. (2018); Grill al. (2020)), partly because the goal of is not only to exploit the traces T - ((() collected so far and the model P = LTARN(T). but also to collect data to self-improve P and ε/π in the next episode This second reason is particular in our iterated batch setup: contrary pure batch
5

---

Under review as a conference paper at ICLR 2023
RSACTOR (T (T),,, -hrry, ,p T, L, 12):
return TRAIOPT ([T(0)) 1= T,,
> planning
GUDD&&EXPLOREACTOOR (0) TTO PP,T, L.m):
1 g < (T PP.))
b model free guide policy
2 [p:] HEATINGEXPLORE 1)
explorer policies
3 return TRAJOPT ( (p:))=1- I.,)
planning
Figure 4: Model-based ACTORS (policies executed on the real system). RSACTOR is a classical random shooting planner that uses the random policy 7(0) for all rollouts. first learns Dyna-style guide policy contained in TP((). It can also use the traces collected on the real system. then "decorates" the guide by (possibly different) exploration strategies, and runs these reactive guide&explore policies [pi]i=1 in the TRAJOPT planner in Fig 5.
the transition p (more precisely, updates the previous guide
TEWALREWARD (T): I return 7' x R(T)
total reward (a.k.a return)
BOOTSTRAP (T):
return T x R(T) aV
total reward value of last state
[RA) [s]]
1 for i - to
(()) ROOLLOUT(P,, (oi,
ith roll-out trace total reward of T(() b index of the best trace
((i) VALUE (T(1)
4 1" + argmax; V(()
5 return T (1" [1]
> first action of the best trace
Figure 5: VALUE estimates on rollout traces and TRAJOPT: trajectory optimization using ran- dom shooting with a set of policies. TotalRReaard and BooSTRAP are two ways to evaluate the value of a rollout trace. The latter adds the value of the last state the total reward, according to value estimate V + I weighted by a hyperparameter a. They are called in Line 3 of TRAJOPT which is a random shooting planner that accepts n different shooting policies [p:] for the rollouts used in the search. As usual, it returns the first action a; T [1] of the best trace ((²) ((s), aj).... (s7, af)). Its parameters the shooting policies [p.] the tran- sition p. and the number n and length L rollouts, but to properly define it, we also need the a double argument list ()l)- We recall here that for trace (ST, ar)) and for t {1. T}. [a S; and [T ag.
state which plan from, T
RL, exploration here is crucial. When the guide & is probabilistic, we explore implicitly because of the sampling step (Line in ROLLOUT, Fig 1), and partly also because of the draw from the imperfect and possibly stochastic model (Line 5). Nevertheless, exploration explicitly. To show this, we experiment with HEATINGEXPLORE strategy which consists in modulating the temperature of the guide distribution {(als) (Fig 4). The novelty of approach that, instead of constant randomness further diversify the search and let the planner explore promising regions far from the distribution of trajectories where the guide may have falsely converged. Finally, similarly to Lowrey et al. (2019); Argenson & Dulac-Arold (2021), we found that bootstrapping the planning with the learned value function at the end of each rollout trace (BoOTSTRAP in Fig 5) can be helpful for optimizing the performance with a short horizon.
found that it helps if control
the exploration, we use a ser of temperatures [T]] to
3.5 METRICS
We use two rigorously defined and measured metrics (Kégl et al., 2021) to assess the performance of the different algorithmic combinations. MAR measures the asymptotic performance after the

---

Under review as a conference paper at ICLR 2023
learning has converged, and MRCP measures the convergence pace. Both be averaged over seeds, and MAR is also an average over episodes, so we can detect statistically signifieant differences even when they are tiny, leading to proper support for experimental development.
MEAN ASYMPTOTIC REWARD (MAR). Our measure of asymptotic performance, the mean asymptotic reward, is the mean reward MR(7) (Tx") in the second half of the episodes (after convergence; set N in such way that the algorithms converge after less than N/2 episodes) MAR N/= (R)
MEAN REWARD CONVERGENCE PACE (MRCP(F)). To assess the speed of convergence, we define the mean reward convergence pace MRCP(T) the number of steps needed to achieve mean reward F, smoothed over window of size 5: MRCP(F) T x argmin, Er'=r ( r))
The umit of (FCP(F) (7) is system access steps, not episodes, first to make it invariant to episode length, and second because in micro-data RL the unit of cost a system access step. For Acrobot, we use F 1.8 in our experiments, which is roughly 70% of the best achievable mean reward.
EXPERIMENTS
4.1 ACROBOT
Acrobot is an underactuated double pendulum with four observables s, = [61, 02 02. 01. 02] which are usually augmented to six by taking sine and cosine of the angles (Brockman al., 2016); 0, the angle to the vertical axis of the upper link; 02 is the angle of the lower link relative to the upper link, both being clipped to [- m|: O, and 0g are the corresponding angular momenta. For the starting position 81 of each episode, all four state variables are sampled uniformly from an approximately hanging and stationary position of E [-0.11 The action is a discrete torque on the lower link E 0, The reward is the height of the tip of the lower link over the hanging position r(s) = - cos 0 - cos(O1 + 022) E [0,4].
Acrobot is small but relatively difficult and fascinating system, so it is an ideal benchmark for continuous-rwaard engineering systems. Similarly to Kégl et al. (2021), we set the number of episodes to N 100, the number of steps per episode to T 200, the number of planning rollouts to n = 100, and the horizon to L = 10. With these settings, we can identify four dis- tinctively different regimes (see the attached vidcos): i) the random uniform policy π (0) achieves MAR 0.1 0.2 (Acrobot keeps approximately hanging). ii) reasonable models with random shooting or pure Dyna-style controllers achieve MAR = 1.4 - 1.6 (Acrobot gains energy but moves its limb quite uncontrollably), iii) random shooting n 100,
10 with good models such as PETS (Chua et al., 2018; Wang et al., 2019) or DARMDN (Kégl et al., 2021) keep the limb up and
manage have its tip above horizon on average MAR 2.0 2.1 (previous state of the art), and iv) in our experiments we could achieve a quasi perfect policy (Acrobot moves up like a gymnast and stays balanced the top) MAR 2.7 2.8 using random shooting with 100K, L 20 on the real system, giving us target and possibly large margin of improvement.
Acrobot is an ideal benchmark for making our point for the two following reasons. First, it turned out to be a quite difficult system for pure model-free baselines and associated Dyna-style algorithms (we achieve a MAR & 2.1 with a DQN, see Table 1, and Wang et al. (2019) report a MAR x 1.6 1.7 for other Dyna-style algorithms). Second, decision-time planning can achieve quasi-perfect policy (MAR 2 2.7 - 2.8) but doing so while being data efficient and with a limited planning budget appears to be challenging.
4.2 CARTPOLE SWING-UP
Cartpole swing-up from the DeepMind Control Suite (Tunyasuvunakool et al., 2020) is an under- actuated pendulum attached by frictionless pivot cart that moves freely along horizontal line. Observations include the cart position, cosine and sine of the pendulum angle, and their time derivatives 8, = [x. à cos 0, sin 0, 0j. The cart is initialized at a position x and a velocity close to
We chose this rather than the sparse variable-episode-lenge version r(s) T (2 cos 0 cos(#: 02) > 3} (Sutton, 1996) since it corresponds better to the continuous aspect of engineering systems.
7

---

Under review as a conference paper at ICLR 2023
O, and an angle close to 0 (hanging position). The goal is to swing-up the pendulum and sta- bilize it upright by applying a continuous horizontal force a E [-1, to the cart at each timestep t. The reward in [0, is obtained by the multiplication of four reward components: one depending on the height of the pendulum (in [1/2. 1).. one on the cart position (in [1/2, 1), one on its velocity (in [1 /2, 1)) and one the amplitude of the force (in [4/5, 1). The maximum reward is obtained when the pendulum is centered (x = 0), upright with no velocity and an applied force of O. This task has been widely used by the literature as standard benchmark system for nonlinear control and complex action space due to its potential generalization to different domains (Boubaker, 2012; Nagendra et al., 2017). We set the number of episodes to N 35, the number of steps per episode to T = 1000, the number of planning rollouts to n 500, and the horizon to L = 20. A mean reward of 0.8 corresponds to a pole that succeeds at standing upright and being stable.
We chose the Cartpole system as vanilla MPC agents (RS or CEM) require long planning horizon (at least = 100) to succeed at swinging up the pendulum and stabilize upright. It also illustrates how our approach extends to the continuous action setting which increases the complexity of the optimization search space and requires sensitive controllers.
4.3 MODELS, GUIDES, AND ACTORS
Following Kégl et al. (2021), we tried different system models (Fig 2/Line 4) from the family of Deep (Autoregressive) Mixture Density Networks (D(A)RMDN) and selected the ones giving the best results on the Acrobot and Cartpole swing-up systems.
In principle, any MFRL technique providing a value function (for bootstrapping) and a policy can be used as a guide & when applying ITERATEDMBRL (Fig 4/Line 1). We argue though that an off-policy algorithm is better suited here since it can leverage all the (off-policy) data coming from interac- tion with the real system. In partieular, those traces are generated using planning and represent stronger learning signal for the agent (planning during training vs. at test time; Hamrick et al. (2021)). Thus, we experimented with Deep Q Networks (DQN; Mnih et al. (2015)) for the discrete action Acrobot system and Soft Actor-Critic (SAC) (Haarnoja et al., 2018) for the continuous action Cartpole swing-up task. For SAC on Cartpole following Janner et al. (2019). short rollouts starting from real observations are performed experience replay buffer, along with the real transitions observed during the rollouts (Fig 2/1 2/Line 7). The SAC is then updated by sampling batches from this buffer. Details on the implementation and hyperparameters of the DQN and SAC agents are given in Appendix A.
the model to sample transitions which are then placed in an
For actors (Fig 2/Line 5, Fig 4), we start from the simple model-free guide (DQN SAC) which is trained with data generated by the model and is interacting with the system without planning. We refer to these agents as MBPO(DQN) and MBPO(SAC). Adding (n. L) to the name of the agent means that we use the agent to guide a planning of n rollouts with horizon L.
Appendix A contains detailed information on the various algorithmic choices.
4.4 RESULTS
Table and Fig 6 compare the results obtained with GUIDE&EXPLORE, a vanilla RSACTOR agent using the same budget as the one we consider for GUIDE&EXPLORE, the pure model free guides trained on the real system and their Dyna-style version (no planning, MBPO). We see that the GUIDE&EXPLORE algorithm gives the best performance. On Acrobot it is almost matching the costly RSACTOR(2 = 100K, L = 20) that we include as a target even though it would not be officially ac- cepted in our benchmark as we restrict n to 100 and L to 10. On Cartpole reaches the performance reported in Lee et al. (2020) and approaches the one from (Springenberg et al., 2020) which are, to the best of our knowledge, the state of the art on Cartpole We note that Springenberg et al. (2020) reports the median performance whereas report the mean performance Lee al. (2020).
We ran an ablation study on Acrobot showing that all ingredients add to the performance. Although MBPO(DQN) performs reasonably well and is comparable to RSACTOR(2 achieve the performance of RSACTOR(2 = 100K, L = 20). Fig 6 also shows that adding a heating explorer to the guide significantly improves the performance of RSACTOR(R MBPO(DQN). We found that allowing the planner to choose the right amount of exploration (the right temperature) is robust and safe approach (see Appendix B and Fig 8 for more results with
100, L = 10) it fails to
100, L = 10) and
8

---

Under review as a conference paper at ICLR 2023
Table 1: Agent evaluation results. MAR is the mean asymptotic reward showing the asymptotic performance of the agent and MRCP(L88) is the mean reward convergence pace showing the sample- efficiency (the number of system access steps required achieve mean reward of 1.8). The model- free agents are trained on the real system and the corresponding MAR shows the asymptotic perfor- mance obtained after convergence. + and t mean lower and higher the better, respectively.
Acrobot
MAR t
MRCP(1.8) 4
MBPO(DQN)
2.113::0.02
4140
=720
RSACTOR(n = 100, L 10) DQN(n 100, L 10J-GUIDE&EXPLORE
2.075=00001
2620
=320
2.404
1800
+:90
RSACTOR(π 100K, L 20)
2.474110022
2280
+5500
Model free DQN
2.16 +0.018
1.58M+0.20M
Cartpole
MAR 1
MRCP(0..5)
MBPO(SAC)
0.566=#0.055
17750=1743
RSACTOR(1 500, L 20)
0.304=00001
inf tinf
SAC(m = 500, 20)-GUIDE&EXPLOREE
0.022
10500==5020
Model free SAC
0.781 +0.05
72250=111000
2.5
0.8
2.0
0.7
mean reward
mean reward
0.6
1.5
0.5
1.0
Agent
0.3
NEPC(DOH
0.2
Random Shooting (1:100, L=17)
0.1
COOL
0.0
0.0
8,000
16,000
0
10,000
20,000
30,000
system access step
system access step
(a) Acrobot
(h) Cartpole
Figure 6: Learning curves obtained with different agents. Mean reward curves are averaged across at least four seeds. Areas with lighter colors show the 90% confidence intervals and dashed lines represent the score of the best converged model-free algorithms.
(a) Mean reward between
(hanging) and 4 (standing up). Episode length is T 200, number of epochs is N 100 with one
episode per epoch. (b) Mean reward is between and Episode length epochs is N 35 with one episode per epoch.
1000, number of
suboptimal DQN guide). The final improvement was attained by adding value bootstrapping to the heated explorer/planner.
5 CONCLUSION
In this paper we show that an offline Dyna-style approach can be successfully applied on benchmark systems where previously Dyna-style algorithms were failing. Our empirical results exhibit the im- portance of guiding the decision-time planning with the correct amount of exploration achieve the best performance under a planning budget constraint. More precisely, our decision-time planner explores varied range of trajectories around the guide's prior distribution and bootstraps with value function estimate to further improve the performance.
This combination leads achieving state-of-the-art performance while respecting reasonable source constraints. Future work includes modelling the uncertainties of the value estimates so as to use them for better exploration.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
In order to ensure reproducibility we will release the code at <URL hidden for review>, the paper has been accepted. We also provide details the hyperparameter optimization of the agents and models as well as the best ones in the Appendix.
REFERENCES
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Arthur Argenson and Gabriel Dulac-Arnold. Model-based offline planning. In International Con- ference Learning Representations, 2021.
Mohak Bhardwaj, Sanjiban Choudhury, and Byron Boots. Blending {mpe} & value function ap- proximation for efficient reinforcement learning. International Conference on Learning Rep- resentations, 2021. URL https / www.ooeeeevieeeeeeeeee net forum? waetiimiiiiicccooo d=RgCC_00Rgy%.
Olfa Boubaker. The inverted pendulum: A fundamental benchmark in control theory and robotics. In Proceedings of the International Conference on Education and e-Learning Innovations, pp- 1-6, 2012.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl gym, 2016.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems 31, pp. 4754-4765
Curran Associates, Inc., 2018.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-Based Reinforcement Learning via Meta-Policy Optimization. In 2nd Annual Confer- ence on Robot Learning, CoRL 2018, Züriic, Switzerland, 29-31 October 2018, Proceedings. volume 87 of Proceedings of Machine Learning Research, pp. 617-629. PMLR, 2018.
Pieter- Tjerk de Boer, Dirk P. Kroese, Shie Mannor, and Reuven Y. Rubinstein. tutorial on the cross-entropy method. Annals of Operations Research, 134, 2004.
Mare Peter Deisenroth and Carl Edward Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In Proceedings of the International Conference on Machine Learning, 2011.
Jean-Bastien Grill, Florent Altché, Yunhao Tang, Thomas Hubert, Michal Valko, loannis Antonoglou, and Remi Munos. Monte-Carlo tree search as regularized policy optimization. In Proceedings of the 37th International Conference on Machine Learning, pp. 3769-3778, PMLR, 2020. URL https: wwwppooeeiinnoo ngs mlr. press 1eee11///rllii
html.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with Stochastic Actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings the 36th In- ternational Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2555 2019.
Jessica Hamrick, Abram Friesen, Feryal Behbahani, Arthur Guez, Fabio Viola, Sims Witherspoon, Thomas Anthony, Lars Buesing, Petar Veličković, and Theophane Weber. On the role of plan- ning in model-based based deep reinforcement learning. In 9th International Conference on Learning Representations, ICLR 2021, To appear, 2021. URL https: wpwpeeeeeeeeeeeeeo/e/ee net forum? (1=TrM64DGB33
10

---

Under review as a conference paper at ICLR 2023
Jessica B Hamrick. Analogues of mental simulation and imagination in deep learning. Current Opinion in Behavioral Sciences, 29, 2019. ISSN 2352- 1546.1 doi: https///oi.org//0.110000 cobeha.2018. 12.011. URL https: / /ww. sciencedirect. com/ wwwasacceinnncrii////ee/ p1i/5235215 64 8301 670. Artificial Intelligence.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty- Seccod--e AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Ed- ucational Advances in Artificial Intelligence (EAAI-18), pp- 3207-3214. AAAI Press, 2018.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Balázs Kégl, Gabriel Hurtado, and Albert Thomas. Model-based micro-data reinforcement learn- ing: what are the crucial model properties and which model to choose? In 9th International Conference on Learning Representations, ICLR 2021, 2021. URL https::///// / /operre/ee.r..e...
net/ wow.mnidd-ppyyll9e...
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model- Based Offline Reinforcement Learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 21810- 21823, Curran Associates, Inc., 2020.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations, 2018. URL htw.t: http://eeeeeeeeeeeeeeeeeeeeeeeee n forum? Sd=
Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Rein- forcement Learning, 2012.
Kuang-Huei Lee, lan Fischer, Anthony Z. Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in rl. In Neural Information Processing Systems (NeurIPS), 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model-basee -ased control. In International Conference Learning Representations, 2019. URL https: wwwppppeeee.nneeeeeeeeeeeeeeeeee forum?
id=Byee7n05FQ.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorith- mie Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees. In International Conference on Learning Representationss 2019. URL https: opppreviee.....ee n net/forum:s forum? de:iddd=eeee2 1K2KKXXX
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. In International Confer- ence on Learning Representations, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Joannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 51807540):559-533, 2015.
Thomas M. Moerland, Joost Brockens, and Catholijn M. Jonker. Model-based reinforcement leam- ing: A survey. arXiv preprint arXiv:2006.. 127 2021.

---

Under review as a conference paper at ICLR 2023
Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural network dy- namics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 JEEE International Conference Robotics and Automation, ICRA 2018, pp. 7559-7566. IEEE, 2018.
Savinay Nagendra, Nikhil Podil, Rashmi Ugarakhod, and Koshy George. Comparison of reinforce- ment learning algorithms applied to the cart-pole problem. In Proceedings of the International Conference on Advances in Computing, Communications and Informatics, pp. 26-32, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, reg-ppeffrrance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dor- mann. Stable baselines3. https::////..... https://gittttt ctw/ ROLLLRRRM/--0 wttp/ttabb-bbbeeeeeeeee.ee 2019
Anil Rao. A survey of numerical methods for optimal control. Advances in the Astronautical Sciences, 135, 01 2010.
Arthur George Richards. Robust constrained model predictive control. PhD thesis, Massachusetts Institute of Technology, 2005.
Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis. Thore Graepel, and et al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature, 588(7839):604 609, Dec 2020. ISSN 1476-4687. doi:
URL http: dxx doi org/
10. 10.1038/s4158 020 03051-4. 4.
Harshit Sikchi, Wenxuan Zhou, and David Held. Learning off-policy with online planning. In 5th Anmual Conference on Robot Learning, 2021.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel. Timothy Lillicrap. Karen Si- monyan, and Demis Hassabis. Mastering chess and shogi by self-play with general reinforce- ment learning algorithm, 2017.
Satinder Singh, Tommi Jaakkola, and Michael Jordan. Reinforcement learning with soft state ag- gregation. In G. Tesauro, D. Touretzky, and T. Leen (eds.), Advances in Neural Information Processing Systems, volume 7. MIT Press, 1995.
Jost Tobias Springenberg, Nicolas Heess, Daniel Mankowitz, Josh Merel, Arunkumar Byravan, Ab- bas Abdolmalcki, Jackie Kay. Jonas Degrave, Julian Schrittwieser, Yuval Tassa, Jonas Buchli, Dan Belov, and Martin Riedmiller. Local search for policy iteration in continuous control. arXiv preprint ar a000::201. 2020.
Wen Sun, Geoffrey Gordon, Byron Boots, and J. Bagnell. Dual policy iteration. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bull., 164):160- July 1991. ISSN 0163-5719.
Richard S Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo (eds.), Advances in Neural Information Processing Systems 8, pp. 1038-1044. MIT Press, 1996.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http: http:///nnonppeeeeeedneeoo net /book//o//////oo ttokkkoohoookk 2nd. html.
12

---

Under review as a conference paper at ICLR 2023
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts. 6:100022, 2020. ISSN 2665-9638.
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. In 8th Inter- national Conference on Learning Representations, ICLR 2020, 2020.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforce- ment learning. arXiv preprint (rXXv:19070020577 2019.
Yuxiang Yang, Ken Caluwaerts, Atil Iscen, Tingnan Zhang, Jie Tan, and Vikas Sindhwani. Data efficient reinforcement learning for legged robots. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), Proceedings of the Conference Robot Learning, volume 100, pp. 1-10, 2020.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based Offline Policy Optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc., 2020.
volume 33, pp. 14129-14142.
Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan O. Lambert, André Biedenkapp, Kurtland Chua, Frank Hutter, and Roberto Calandra. On the importance of hyperparameter optimization for model-based reinforcement learning. In Arindam Banerjee and Kenji Fukumizu (eds.), The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, volume 130 of Proceedings of Machine Learning Research, pp. 4015-4023, PMLR, 2021.
13

---

Under review as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
A.1 CODE AND DEPENDENCIES
Our code will be made publicly available after publication to ease the reproducibility of all our results. We use Pytorch (Paszke et al., 2019) to build and train the neural network system models and policies. To rum the ITERATEDMBRL experiments we use the ii_ssmulatoo (https:// githus, com/ r comp-xits/ee simulator) python library developed by Kégl al. (2021) which relies on Open AI Gym (Brockman et al., 2016) for the Acrobot dynamics and dm cocoootolll (Tunyasuvunakool al., 2020) for the Cartpole swing-up task. For the DQN and SAC agents rely on the StableBaselines3 implementations (Raffin et al., 2019).
A.2 MODELS AND AGENTS
is known that carefully tuning hyperparameters of deep reinforcement learning algorithms is cru- cial for success and fair comparisons (Henderson al., 2018; Zhang al., 2021). To reduce the computational cost and consider a reasonable search space the models and the agents were optimized independently. For the Acrobot system models we use the same hyperparameters in Kégl et al. (2021). Please refer to Appendix D in Kégl et al. (2021) for complete description of the hyperparameter search and the selected hyperparameters. We decided to use DMDN de trained on the first 2000 and last 3000 collected samples as it lead to similar performance with limited training time. For Cartpole we use a DARMDN model with hidden layer and 128 neurons trained on the full dataset. These models are trained by minimizing the negative log-likelihood. The 'det' suffix means that the model is sampled from deterministically, returning the mean of the predicted distribution. The reader can also refer to Kégl et al. (2021) for complete description of these models.
the ones used
For the DQN optimized its hyperparameters with random search of 1000 trials and a parallel training on 10 copies of the real system for 10 million steps (Table 2). We then selected the DQN with the best mean reward (Table 3). When training DQN on the system model, iteratively update it with 100 000 steps at each episode using the most recent system model. We do not use short rollouts this was not necessary. When bootstrapping with the value function used discount factor of 0.95 as it lead to the best performance.
For the SAC agent we performed similar hyperparameter optimization with a random search of 1000 trials and a parallel training on 10 copies of the real system for million steps (Table 4). The best SAC parameters are given Table When training SAC on the system model, we train from scratch at each episode for 250 000 steps and perform short rollouts of length 100.
Table 2: Random search parameters for the model free DQN
Parameter
Values
Discount factor
[0.9, 0.99, 1]
Polyak update parameter
[0.1, 0.5, 1]
Learning rate
[0.0001, 0.001]
Final training epsilon value
[0.01, 0.1]
Exploration fraction
[0.1, 0.5)
Buffer size
[10000, 100000]
Update frequency of the network
[2, 10]
Batch size
[64, 128]
Update frequency of the target network
[500, 1000]
Network architecture
[MLP(1188, 128), MI.P(256, 256]
Gradient steps
[1,2,5, 10]
A.3 EXPLORATION
The multi-temperature HEATINGEXPLORE strategy depends on whether actions are discrete or con-
tinuous:
14

---

Under review as a conference paper at ICLR 2023
Table 3: Best parameters for the model free DQN
Parameter
Values
Discount factor
0.99
Polyak update parameter
0.5
Learning rate
0.001
Final training epsilon value
0.01
Exploration fraction
0.1
Buffer size
10000
Update frequency of the network
Batch size
128
Update frequency of the target network
1000
Network architecture
MIP((26,, 256)
Gradient steps
10
Table 4: Random search parameters for the model free SAC
Parameter
Values
Discount factor
[0.9, 0.95, 0.99, [0.1. 0.5, 1]
Polyak update parameter
Learning rate
[0.0001, 0.0005, 0.001]
Buffer size
[10000, 100000, 1000000]
Update frequency of the network
[4. 10, 20]
Batch size
[64, 128)
Update frequency of the target network
[100, 1000, 10000]
Network architecture
[MLP(64, 64), MLP((18,, 128), MLP(256, 256]
Gradient steps
[1. 5, 10, 20]
Table 5: Best parameters for the model free SAC
Parameter
Values
Discount factor
1.0
Polyak update parameter
0.1
Learning rate
0.0005
Buffer size
1000000
Update frequency of the network
10
Batch size
128
Update frequency of the target network
1000
Network architecture
MLP(256, 256)
Gradient steps
20
For environments with discrete actions, we learn a Q function and first normalize the Q values by their maximum value, Q(s, a) Q(s, a)/ max Qss,a'), before applying softmax:
((
Pr(a)s;) Ene Qlsaat/T
For continuous actions environments, learn a Gaussian policy E(-ls) ((((ss) and define the explorer policies as:
(r(-(s)) ~ (N((s) T:)
(T.1iis1 n) is increasing sequence of temperatures. A large temperature gives uniform distribution, whereas a low temperature corresponds to taking argmax, Q(,,, a) or u(s). Different shapes of sequences were tried (linear, logarithmie, exponential, polynomial, logistic), and best
15

---

Under review as a conference paper at ICLR 2023
performance obtained with logistic schedule (with linear end). The exact values will be provided in the code.
For Acrobot and the DQN guide also played with multi-s exploration strategy based on ExSPRRE (Fig 7) where we use one E value for each of the n 100 rollouts: (0.001, 0.01, 0.02, ..... 0.99). Refer to Appendix B for experiments with suboptimal DQN guide using this exploration strategy.
EPSGREEDYEXPLOCM ()) for f- to n: Pc(aass argmax, E(als) with probability E π (a)
with probability 6t-
return [p:li=1
Figure 7:
Exploration strategies. BysGreEdyExPLLOG changes the action to random action ()) ~ a with different probabilities. The probabilities [e:]ā= are hyperparameters.
B IMPORTANCE OF THE EXPLORATION: STUDY WITH A SUBOPTIMAL DQN GUIDE ON ACROBOT
We ran an ablation study with suboptimal DQN guide on Acrobot (achieving asymptotic perfor- mance of on the real system) and a multi-e greedy explorer (EpsGreeDy (xp). to claim the importance of the explorer. EpSGREEDY makes easy to control and interpret the degree of exploration through the parameter.
We consider the following agents. We start from the simple DQN guide which is interacting with the system without planning. Adding (n, L) to the name of the agent means that we use agent to guide planning of rollouts with horizon L. It is important to note here that planning without exploration using the greedy guide is, in our case, equivalent to no planning since both the model deterministic. NONNFEPSGREEDYXXPP...
p and the guides associated exploration strategy (Fig 7). When fixed Æ is used for the exploration strategy, we add an explicit parameter, c.g., EPSGREEDYEXPLOREE)) No parameters means that a different e or temperature is used for each of the n rollouts. Setting ε to 0 corresponds to no exploration and is equivalent to using the guide greedily without planning deterministically when sampled from. Setting to corresponds to full exploration and is equivalent the purely random RSACTOR(R 100. L 10).
refers the additional use of the
1 and = 1) our model used
Table 6 reports the results obtained by DQN alone, DQN with planning and fixed & values (DQN(n 100, = FEn
for & E [0.0001, 0.01, 0.05, 0.2, 0.4, 0.8. 0.99, 0.9999)) The closer 5 is to 0 the closer the performance is to DQN, and the closer & is to 1 the closer the performance is to RSACToR(N = 100, L 10). With a well-chosen between these two extremes, say = 0.4, we obtain better performance than either extremes. We can thus claim that planning required with a correct amount exploration. Our EpsGREEDyEXPLOOR exploration strategy, used with multiple values allows for the automatic and dynamic selection of the good amount of exploration. It would indeed be possible that using a fixed e value would not give the best performance different values would be required at different epochs or different steps of episode. We illustrate this by plotting the selected epsilon vs the episode step for 3 different epochs (Fig 8). Even though the guide is suboptimal the exploration scheme can make the agent benefit from the guide where it is good and discard it where is bad.
C RSACTOR PERFORMANCE ON THE REAL ACROBOT AND CARTPOLE SYSTEMS
C.I ACROBOT
We present the results one can obtain on the real system with an RSACTOR and different values of the planning horizon I. and the number of generated rollouts in Fig 9. For the considered planning
16

---

Under review as a conference paper at ICLR 2023
Table Importance of the explorer. MAR is the Mean Asymptotic Reward showing the asymptotic performance of the agent and MRCP(118) is the Mean Reward Convergence Space showing the sample-effciincy performance as the number of system access steps required to achieve a reward of 1.8. J and f mean lower and higher the better, respectively, The + values are 90% Gaussian confidence intervals.
Agent
MAR t
MRCP(1.)) 1
DQN
1.44210.014
inf Hiff
DQN(n 100, L 10-BBPGGRRBDYYEXPLOORE 0.0001)
1.475
inf Hinf
DQN(nz 100. = LDD-EBrSGREEDDYEELLORRC 0.01)
1.66410.032
inf #inf
DQN(" 100.
LD-EESSGREEDYEXALOORG 0.05)
1.932=0.012
3540#5200
DQN(n 100, I. = 10)-EBSSHRAHEDYEXPPLOREE 0.1)
2.009++0.031
2400
RSACTOR(N 100, = 10)
2.075+0.01
2620+320
DQN(" 100, T. = 10>-EBSGGRREEDYYXPPLORRR 0.9999)
2.107
+
(QN(R 100.
10--EPSGRREBDYEEPLORRE 0.99)
2.11810.046
2400
DQN(R2 -000, L 10-EEPSGREEDYYEXPLORGE 0.2)
2.151
2400
DQN("E - 100. 10--EPrSGREEDDYEXPPLOOCC 0.8)
2.196+0.037
2000
DQN("z 100. L (Q.EEESSRREEEYEEXP.ORR
0.4)
2.204=00001
1400=140
DQN(n 100, 10J-EPSGGREEDDEEXPP.RR
2.203
1880+110
epoch epoch 50
0.6
epoch 100
median epsilon
selected epsilon
0.5
0.4
0.3
25
50
75
100
125
150
175
200
timestep
Figure 8: Moving average evolution of the selected epsilon (from the best trace in Line in Fig 5) across timesteps and epochs in the Acrobot environment (1 epoch = 200 timesteps). In early epochs (epoch 0) when the guide weak, selected epsilons are roughly uniform around the median, but as the guide is getting better (epoch 50), late timesteps require more exploration than first ones already learnt well by the guide. Eventually (epoch 100), the guide would further improve and less exploration is needed.
horizons a larger number of generated rollouts lead to a better performance. We also observed in our simulations that for the Acrobot to stay balanced, was necessary (although not always sufficient) to have a reward larger than 2.6. We see from Fig that this can be achieved with simple agent such as RSACTOR but at the price very large number of generated rollouts. The goal therefore to design smarter agent that can come as close as possible to this performance with limited budget.
C.2 CARTPOLE
Fig 10 reports the performance obtained with RSACTOR and a CEM agent for different values of and L. For the CEM agent 5 iterations with the given and L so the total budget is x x L. mean reward 0.8 corresponds to a pole that succeeds at standing upright and being stable. We that achieving such a performance requires CEM agent and planning horizon larger than
= 100.
17

---

Under review as a conference paper at ICLR 2023
2.8
2.6
2.4
Average reward
2.2
2.0
1.8
Planning horizon L
1.6
10
20
1.4
30
1.2
100
1,000
10,000 100,000 1,000,000
Number of generated rollouts n
Figure 9: Performance obtained with RSACTOR on the real Acrobot system for different planning horizons and number of generated rollouts n. The plot shows the mean rewards obtained for sev- eral randomly initialized episodes of 200 steps. The error bars give the associated 90% confidence intervals. Note that since Acrobot has a discrete action space with three actions, the total number of different rollouts for h 10 is 3 59,049, The performance shown for h 10 and n 100,000 thus only requires n = 59,049 rollouts.
0.8
0.7
0.6
Mean Reward
0.5
0.3
0.1
CEM N-100 H-150
CEM N=100 2=0
N-100 100 H-50
Rs N-100 H=70
RS N-100 H-108
RS N=150 H=150
N-000 H- too
50000 H- 150
H-200-
N=1000 He50
N-1000 Ha60
AS N=1000 H=100
as Ne:1000 H=200
H-150
H-100
Discrete N-5000 H-50
Discrete N-500 1t-100
Discrete N=500 H=150
Discrete N- 1000 H- 100
N=500
RS N-5000
Discrete N-100
Figure 10: Performance obtained with RSACTOR and CEM on the real Cartpole system for different planning horizons L and number of generated rollouts The plot shows the mean rewards obtained for several randomly initialized episodes of 1000 steps. The error bars give the associated 90% confidence intervals.
18


Reviews:
[{"review_id": "tvfJmrx6Cn", "reviewer": "Reviewer_qJN2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores approaches to iterative batch reinforcement learning, whereby relatively low-dimensional data is collected from physical systems and relatively simple actions/decisions must be made in response to these observations. This includes consideration of how the agent may only be given limited real-world decision making experience, but potentially more access to off-policy data. The paper shows that a novel \"Guide&Explore\" strategy achieves better performance than the baselines on two standard tasks: acrobot and cartpole.", "summary_of_the_review": "While the paper presents an interesting look at using and modifying RL methods for iterated batch RL problems that have some similarities to real world intelligent control problems, it does not yet have the precise and rigorous problem setup and mathematical theory-backed analysis expected for ICLR papers in this area. It could also use more experimental results ideally exploring more diverse test problems and more state-of-the-art baselines (e.g. in imitation learning and other fields).", "strength_and_weaknesses": "# Strengths\n## Novelty\n- The heating explore based on using a set of different temperatures to create more exploratory policies seems novel and effective\n## Significance\n- The general classes of real-world problems considered by this paper are widespread, and this is an interesting investigation into using deep reinforcement learning to tackle them\n## Clarity\n- The algorithms and results figures are clear and informative\n- The text is well edited and makes good use of references to prior works; both of these are well appreciated\n\n# Weaknesses\n## Novelty\n- Novel algorithms and/or algorithmic changes are not clearly marked, making it harder to identify what is novel\n## Significance\n- It is hard to see how performance on cartpole and acrobot relate at all to the motivating example of a telecommunications engineer; cartpole and acrobot do not seem to reflect the considerations listed in section 1\n- This approach seems to rely on determining the optimal temperature set for exploration -- it is not clear how one could feasibly identify a temperature set to use for novel problems\n- There are no new theorems or proofs presented in this paper to rigorously explain why these problems are difficult or unsuited to other deep learning methods, or why the proposed methods are better\n## Clarity\n- The problems discussed in the first half of the paper at a high level and in vague terms, making it unclear what this paper is precisely trying to contribute or how its insights can/should be used by future works", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "knIR0JhVoKm", "reviewer": "Reviewer_PxDX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed an efficient guide and exploration strategy to improve the performance of iterated batch RL. A Dyna-style explore, and guide method is used where an exploration model is learnt first using a model-based guiding approach that is then used to collect future trajectories. An end-to-end algorithm is proposed that learns a model free guide policy, learn exploration policy, and then collect optimized trajectories. Experimental results on Acrobat and Cartpole tasks demonstrate that the proposed method outperforms existing methods like DQN, SAC and RSActor.", "summary_of_the_review": "Iterative batch RL is an interesting and growing research area that can make significant impact in adoption of RL. The ideas presented in the paper are interesting and novel in the sense that an efficient exploration with a learned guiding model can accelerate performance. Experiments demonstrate performance improvement in Acrobat and Cartpole environment. It would be really appreciated if some theoretical results were presented in terms of performance vs. organizational time constraints. Experiments on other real-world domains will further ground the claims. Adding ablation studies with different parameters (e.g., L, n) would be great as well, as my main concern is that choosing an optimal value for these parameters would be challenging. ", "strength_and_weaknesses": "The paper extends the research on combining model-based and model-free RL methods for improving sample efficiency and resource utilization. Combining model-free RL with MPC is a well-studied problem but exploring this in an iterated batch RL setting is the key novelty of this paper. This is an extension of Kegl et. al. (2021), where rather than using simple random shooting, a Dyna-style approach is used to learn a dynamic model from gathered data that is used to collect future imaginary traces. Experimental results support the claim that such Dyna-style approach with efficient exploration improves the performance over random shooting.\nWhile I appreciate the approach of using sophisticated Dyna-style approach for generating future traces, I have some reservations regarding the contributions of the paper. Firstly, both Dyna-style approach, and using the concept of model-based exploration are well-studied; this paper just used these ideas in the context of iterated batch RL setting. The paper lacks any theoretical results that can demonstrate the efficacy (in terms of improving two evaluation metrics) of the proposed models in a generic setting. The experiments are conducted on synthetic Acrobat and Cartpole problems, so without theoretical guarantees, it is not clear to me whether the claims are generalizable to other real-world offline RL problems. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "W1RdFPVjzKT", "reviewer": "Reviewer_fuzN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This algorithm proposes an algorithm for iterated batch reinforcement learning. The algorithm uses model-free RL to learn a guide policy, and then uses decision-time planning to improve the policy. The decision-time planning uses some exploration method and a rollout procedure to get a good action.", "summary_of_the_review": "The paper lacks novelty since it is very similar to AlphaZero, and in fact, it is almost a simplified version of AlphaZero. The experiments are performed only in simple environments. Therefore, I don't think this paper makes a significant contribution to the community.", "strength_and_weaknesses": "Strength:\n\nThe paper is overall well-written and studies an important problem.\n\nWeakness:\n\nThe paper is very similar to the AlphaZero algorithm by Silver et al. (2017), therefore, I don't think this paper has enough novelty.\nThe guide policy is similar to the prior policy in AlphaZero as the authors mentioned in Section 3.2. Instead of learning the guide during MCTS training, this paper learns the guide using simpler model-free RL algorithms.\nThe decision-time planning is also similar to AlphaZero. In fact, AlphaZero uses a more advanced MCTS technique whereas this paper uses a simple rollout procedure and chooses the action that leads to the best return. In other words, this paper only uses one step of MCTS.\nTherefore, I think this paper is a simple modification of the AlphaZero algorithm and thus it is not novel enough as an ICLR paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "19GMF4CbpoZ", "reviewer": "Reviewer_F5c7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the  iterated/growing batch RL problem. Authors proposes a Dyna-Style Guide & Explore strategy, which most novel component seems to be an explorer (heating explorer) based on a temperature parameter. The proposed strategy is validated in two classical simple problems: Acrobot (discrete actions) and Cartpole Swing-Up (continuous actions).\n", "summary_of_the_review": "In the context of iterated/growing batch RL, authors propose a Dyna-Style Guide & Explore strategy, which most novel components seems to be an explorer (heating explorer) based on a temperature parameter. The proposed strategy is validated in two simple, simulated problems: Acrobot (discrete actions) and Cartpole Swing-Up (continuous actions). The simplicity of these problems do not allow to assess properly the relevance of the proposed approach.", "strength_and_weaknesses": "Strengths: \n- Motivation\n- Relevance of the problem being addressed\n\nWeaknesses:\n- The novelty of the proposed strategy needs to be better explained. I suggest to improve sub section 3.4\n- The two problems used for the validation are rather simple. I recommend to use a much more complex problem to validate the proposes strategy.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject