The guide and the explorer: smart agents for resource-limited iterated batch reinforcement learning
Authors: Albert Thomas
Balázs Kégl
Othman Gaizi
Gabriel Hurtado
Venue: ICLR 2022 Submitted (2022)
Keywords: Model-based reinforcement learning
Dyna
exploration
planning
DQN

Abstract: Iterated batch reinforcement learning (RL) is a growing subfield fueled by the demand from systems engineers for intelligent control solutions that they can apply within their technical and organizational constraints. Model-based RL (MBRL) suits this scenario well for its sample efficiency and modularity. Recent MBRL techniques combine efficient neural system models with classical planning (like model predictive control; MPC). In this paper we add two components to this classical setup. The first is a Dyna-style policy learned on the system model using model-free techniques. We call it the guide since it guides the planner. The second component is the explorer, a strategy to expand the limited knowledge of the guide during planning. Through a rigorous ablation study we show that exploration is crucial for optimal performance. We apply this approach with a DQN guide and a heating explorer to improve the state of the art of the resource-limited Acrobot benchmark system by about 10%.

Under review as a conference paper at ICLR 2022
THE GUIDE AND THE EXPLORER: SMART AGENTS FOR RESOURCE-LIMITED ITERATED BATCH REINFORCE- MENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Iterated batch reinforcement learning (RL) is growing subfield fueled by the de- mand from systems engineers for intelligent control solutions that they can apply within their technical and organizational constraints. Model-based RL (MBRL) suits this scenario well for its sample efficiency and modularity. Recent MBRL techniques combine efficient neural system models with classical planning (like model predictive control; MPC). In this paper we add two components to this classical setup. The first is Dyna-style policy learned on the system model using model-free techniques. We call the guide since it guides the planner. The second component is the explorer, strategy expand the limited knowledge of the guide during planning. Through a rigorous ablation study we show that exploration is crucial for optimal performance. We apply this approach with DQN guide and a heating explorer to improve the state of the art of the resource- limited Acrobot benchmark system by about 10%.
INTRODUCTION
John telecommunication engineer. His day job is to operate a mobile antenna. He has about forty knobs turn, in principle every five minutes, based about hundred external and internal system observables. His goal is to keep some performance indicators within operational limits while optimizing some others. In the evenings John dreams about using reinforcement learning (RL) to help him with his job. He knows that he cannot put an untrusted model-free agent on the antenna control (failures are very costly), but he manages to convince his boss to live tests couple of days every month.
John's case arguably on the R&D table of lot of engineering companies today. AI adoption is slow, partly because these companies have little experience with AI, but partly also because the algorithms we develop fail address the constraints and operational requirements of these systems. What are the common attributes of these systems?
They are physical, not getting faster with time, producing tiny data compared what model-free RI. (MFRL) algorithms require for training.
System access is limited to a small number of relatively short live tests, each producing logs that can be used evaluate the current policy and can be fed into the training of the next.
They are relatively miml--dimensional, and system observables were designed to support human control decisions, there is no need to filter them or to learn representations (with the exception when the engineer uses complex images, e.g., a driver).
Rewards are non-sparse, performance indicators come continually. Delays are possible but usually not long.
Safe operation even during live tests is crucial. Simulators and digital twins are more and more available, but some systems are too complex to simulate from first principles.
The RL setup that fits this scenario is neither pure batch (offline; Levine et al. (2020)) as interacting with the system is possible during short periods of time pure online as the policy only be

---

ease Under review as a conference paper at ICLR 2022
updated offline between two interaction periods. We refer to it iterated batch RL (Lange et al. (2012) call it growing batch). Furthermore, we are interested in model-based RL (Deisenroth & Rasmussen, 2011; Chua et al., 2018; Moerland et al., 2021) because i) it is sample efficient (Chua et al., 2018; Wang et al., 2019), ii) it works well on small-dimensional systems with dense rewards, and iii) the system model (data-driven simulator) itself is an object of interest because the adoption of data-driven algorithms by systems engineers.
Given robust system model, simple model predictive control (MPC) agents using random shooting (RS; Richards (2005); Rao (2010)) or the cross entropy method (CEM; de Boer et al. (2004)) have been shown to perform remarkably well on many benchmark systems (Nagabandi et al., 2018; Chua et al., 2018; Wang et al., 2019; Hafner et al., 2019; Kégl et al., 2021) and real-life domains such as robotics (Yang et al., 2020). On the other hand, implementing successfully the seemingly elegant Dyna-style approach (Sutton, 1991; Kurutach et al., 2018; Clavera et al., 2018; Luo et al., 2019), when we learn model-free agents on the system model and apply them on the real system, remains challenging especially systems that require planning with long horizon. Our main finding is that the Dyna-style approach can be an excellent choice for iterated batch RL by adding to it a decision- time planning guided by the model-free policy and a dynamically self-tuning exploration. The key that exploration and bootstrapping with a value function estimate affords us shorter rollouts so reduce the accumulation of errors that plagues the Dyna-style approach with long horizons (Janner et al., 2019). We also innovate on the experimental framework (metrics, statistically rigorous measurements), so we can profit from the modularity of the Dyna-style approach, tuning ingredients (model, the MFRL guide policy, exploration, planning, bootstrapping) independently. This modular approach makes engineering easier (as opposed monolithic approaches like AlphaZero (Silver et al., 2017)), which is an important aspect if we want to give the methodology to non-expert systems engineers.
We demonstrate our QUIDE&EXPLORE approach on the mmal--dimensional but difficult Acrobot system which known to be especially tough for Dyna-style techniques (Wang et al., 2019). We show in rigorous ablation study how each ingredient adds a small but significant improvement, achieving a 10% total margin over the current state of the art (Wang et al., 2019; Wang Ba, 2020; Kégl et al., 2021). DYNAZERO (pure Dyna-style with an AlphaZero guide) also matches the asymptotic performance with a larger sample complexity (3x) and computational (5x) price tag.
Table 1: Summaay of previous scores and our results on resource-limited Acrobot (n: planning rollouts, L: horizon). MAR is the episodewise mean reward measured on the second half of the episodes where the algorithms have safely achieved their asymptotic performance. The two RS rows random shooting planning the real system, requiring orders of magnitudes more system access steps than MBRL. and mean lower and higher the better, respectively.
Method
Style
Sys acc steps
MAR
POPLIN-A (Wang& & Ba, 2020)
model-based MPC
50 K
2.116+000010
PETS-RS (Chua al.. 2018: Wang al., 2019)
model-based MPC
50 K
1.9081100010
DARMDN-RS (Kégl et al., 2021)
model-based MPC
20
2.0751100010
RS 100, L 10)
model-free MPC
200 K
2.106
RS(n = 100 10 20)
model-free MPC
400
2.583-
GUNDE&EXPLORE(( 100, L = 10)
model-based Dyna
20 K
2.284+:0.012
DYNAZERO(n = 500)
model-based Dyna
20 K
2.304+t0.031
1.1 SUMMARY OF CONTRIBUTIONS
A conceptual framework with interchangeable algorithmic bricks for iterative batch re- inforcement learning, suitable to bring intelligent control into slow, physical, low- dimensional engineering systems and the organizational constraints surrounding them.
A rigorous experimental framework optimize such systems.
An ablation study that helped us find the combination of neural model, a bootstrapping DQN guide, and a heating explorer, which lead to a 10% jump in the state of the art on the limited Acrobot system.
2

---

Under review as a conference paper at ICLR 2022
2
RELATED WORK
The MBRL subfield has seen proliferation of powerful methods, but most of them miss the spe- cific requirements (solving problems irrelevant this scenario like representation learning or sparse rewards) and missing others (limited and costly system access; data taking and experimentation through campaigns, live tests; safety) (Hamrick, 2019).
The Dyna framework developed by Sutton (1991) consists in training an agent from both real ex- perience and from simulations from a system model learned from the real data. Its efficient use of system access makes it natural candidate for iterated batch RL. The well-known limitation of this approach is the agent overfitting the imperfect system model (Grill et al., 2020). A first solution is to use short rollouts the model to reduce error accumulation as done in Model-Based Policy Optimization (MBPO; Janner et al. (2019)). Another solution is to rely on ensembling techniques for the model. Kurutach et al. (2018)'s ME-TRPO is based on an ensemble of models and Trust Region Policy Optimization (TRPO; Schulman et al. (2015)), At each TRPO step a transition is sampled from a randomly picked model of the ensemble, preventing the policy from overfitting one model. An ensemble is also used in Model-Based Meta-Policy Optimization (MP-MPO; Clavera et al. (2018)) where each model of the ensemble seen as a different task used to meta-learn a pol- icy. The meta-policy is then able to quickly adapt to any of the dynamics model and is more robust to model inaccuracies. Instead of learning the model and then the policy from the model, Stochastic Lower Bound Optimization (SLBO; Luo et al. (2019)) alternates between model and policy updates. In our ITERATEDMB RL skeleton (Fig 1), this strategy would couple the LEARN and MFRL steps, which we do not study in this paper. We note that according to the results shown in Fig 1.(a) and Table in Wang et al. (2019), ME-TRPO, SLBO and MB-MPO alone (pure Dyna, without plan- ning) are clearly suboptimal on the Acrobot system', similar to the performance obtained by our pure Dyna DQN (Section 4). Finally, Yu et al. (2020) and Kidambi et al. (2020) use Dyna-style approach in the context of pure batch RL where no further data collection and therefore no further model updates are assumed.
The idea of using a guide and a value function when planning is not novel (Silver et al., 2017: Schrit- twieser et al., 2020; Wang & Ba, 2020; Argenson & Dulac-Amold, 2020). We were greatly inspired by these elements in our objective of building smarter agents as they can make the search more efficient and thus lead to better performance. POPLIN-A (Wang & Ba, 2020) relies on behavior cloning (using only real experience, unlike our Dyna-style approach that mainly uses the model), but their guide is similar to our approach. During the planning, they add random noise to the actions recommended by a deterministic policy network and update the noise distribution using CEM strategy. In similar way our GUIDE&EXPLORE strategy also adds carefully controled amount of noise to the recommended actions. Our results highlight the importance of a well-calibrated ex- ploration, which also contributes to the understanding of POPLIN-A. Argenson & Dulac-Arnold (2020) and Lowrey et al. (2019) both found that bootstrapping with a value estimate improves the performance of simple guided MPC strategies. The popular AlphaZero (Silver et al., 2017) and MuZero (Schrittwieser et al., 2020) algorithms also rely on a guide and a value function for their Monte Carlo Tree Search (MCTS). We implement a Dyna-style version of AlphaZero, which we call DYNAZERO. The principal issue of MuZero (Schrittwieser et al., 2020) in our micro-data iterated batch RL context is that does not control the number of system access steps: it needs to simulate a lot from the real environment to establish the targets for the value function. In these two algorithms the guide updated from the results obtained during the search that it guided, a procedure known as Dual Policy Iteration (Sun et al., 2018). We prefer experiencing with Dyna-style approaches first to leverage popular MFRL algorithms and defer the study of Dual Policy Iteration to future work.
Our results show that planning is an important ingredient, claim already made by Hamrick et al. (2021). They use MuZero to run their ablation study while we prefer using an explicit model for practical reasons explained in the introduction. Besides planning also study the importance of exploration.
'The conversion from Wang et al. (2019): our mean reward per episode of 200 steps their return 1200m/200+11..
3

---

Under review as a conference paper at ICLR 2022
3 THE FRAMEWORK FOR RESOURCE- ITERATIVE BATCH RL
3.1 THE FORMAL SETUP
Let Tr- (1)),a))...-- (sT,aT) a)) be system trace consisting of T steps of observable-action pairs (st, az): given an observable of the system state at time t, an action a, was taken, leading to a new system state observed as 8-+1- The observable vector 8t (st,...,,87 55)) contains d, numerical or categorical variables, measured on the system at time f. The action vector a, contains d numerical or categorical action variables, typically set by control function at <T({)) of the current observable 8; (or by stochastic policy a₂ T8;);; we will also use the notation π : 8, at). The performance of the policy measured by the reward ra which we assume to be function of the observables St. Given a trace Tr and a reward re obtained at each step t, we define the mean reward as R(Tr) = -i= TT.2 The system model p : (st, aq) ~ 81+1 can be deterministic point predictor or a probabilistic (generative) model that, besides the point prediction $t41
E {p($4+1((*) a+) }. also provides information on the uncertainty of the prediction and/or model
the randomness of the system (Deisenroth & Rasmussen, 2011 I; Chua et al., 2018).
3.2 A NOTE ON TERMINOLOGY
By model will consistently refer to the learned transition system model p (never any policy). Rollout is the procedure of obtaining a trace T from an initial state 81 by alternating a model or real system p and a policy π (Fig 1). We decided rename what Silver et al. (2017) calls the prior policy to guide since prior clashes with Bayesian terminology (as, e.g. Grill et al. (2020); Hamrick et al. (2021) also note), and guide expresses well that the role of this policy to guide the search/planning. Sometimes the guide is also called the reactive policy (Sun et al., 2018) since typically an explicit function or conditional distribution & 8 r that can executed or drawn from rapidly. We will call the (often implicit) policy 8 plan/search the actor (sometimes also called the non-reactive policy since it takes time to simulate from the model, before each action). We will learn the guide & using MFRL mainly on the model p, but, when possible, off-policy (since x + {) data (system trace T) obtained by acting on the real system will also be used. In that sense, through the data from the real system, planning is part of training the guide. However, since from the point of view of the MFRL guide, the model is the world, will not use the term planning refer the rollouts of the model p and the guide & when training the guide on rather we will reserve planning to the guided search procedure that results in acting on the real system.
resulting from the guided
3.3 EXPERIMENTAL SETUP: THE ITERATED BATCH MBRL
For rigorously studying and comparing algorithms and algorithmic ingredients, we need to fix not only the simulation environment but also the experimental setup. We parameterize the iterated batch RL loop (Fig 1) by four parameters:
the number of episodes N,
the number of system access steps T per episode,
the planning horizon L, and
the number of generated rollouts at each planning step.
N and are usually set by hard organizational constraints (number N and length of live tests) that part of the experimental setup. Our main goal is to measure the performance of our algorithms at a given (and challengingly small) number of system access steps N x T. Planning happens in silico, and n and L (and so the total number of calls N x T x L to the simulator/system model p) are softer constraints determined by the (physical) time between two steps and the computational resources available for the planning.
In benchmark studies, such as this paper, we argue that fixing N, T, n, and L is important for making the problem well defined (taking some of the usual algorithmic choices out of the input of
We use the mean reward (as opposed to the total reward, a.k.a return), since it is invariant to episode length and its unit is more meaningful systems engineers.
4

---

Under review as a conference paper at ICLR 2022
T 0
2
for 1 + to T:
3
4 T (8,)
draw action from policy >> update trace
4
T T
5
8141 p(s,,
> draw next state
6
return T
ITERATEDMBRL So, (1), T, L, n):
1
81 S0
draw initial state π (0) Preal: 81, T
2
T()) < ROLLOUT
b initial random trace
3
for < 1 to N:
for N episodes
4
p(5) + LEARN
D learn system model 7(7) + ACTOR T()), (T(11) pp(T) 1
UFi=1 T(r))
5
6
81 ~ So T(++1) COLLOOT((117)) Preal, 81, T)
b draw initial state
7
D episode trace
8
return - T(T)
Figure 1: The iterated batch MBRL loop. Preat (st, az) ~ 8t+1 is the real system (so Line is what dominates the cost) and (st, at) ~ 81+1 can be the real system or the system model in ROLLOUT. So is the distribution of the initial state of the real system. π(1) 8t ~ is an initial (typically random) policy and in ROLLOUT π 8; ~us at is any policy.
the number of episodes; T is the length of the episodes; L is the planning horizon and n is the number of planning trajectories used by the actor policies T(7), = 1,...,A N is the episode index whereas / = 1,..., is the system (or model) access step index. LEARN is supervised learning (probabilistic deterministic time-series forecasting) algorithm applied to the collected traces and ACTOR is a wrapper of the various techniques that we experiment with in this paper (Fig 2). An ACTOR typically updates (7-1) using the model in L planning loop, but it can also access the initial policy π (0) and the trace T(")) collected on Preal up to episode T.
the optimizer), affording meaningful comparison across papers and steady progress of algorithms. As in all benchmark designs, the goal is make the problem challenging but not unsolvable. That said, we are aware that these choices may change the task and the research priorities implicitly but significantly (for example, longer horizon L will be more challenging for the model but may make the planning easier), so if the MBRL community can agree, it would make sense to carefully design several settings (quadruples N T n L.) the same environment.
Our main operational cost is system access step so we are looking for any-time algorithms that achieve the best possible performance at any episode т. Hence, in the MBRL iteration (Fig 1), we use the same traces T(=), rolled out in each iteration (Line 7), to i) update the model p (Line and the actor policy (Line 5) and ii) to measure the performance of the techniques (Section 3.5).
3.4 MODEL-BASED ACTOR POLICIES; GUIDE AND EXPLORE
Our main contribution is Dtna- GUIDE&EXPLOC strategy (Fig 2). The gist is to learn guide policy & using a model-free RL technique on the model p and on traces collected on the real system T. It is known that the guide &, executed an actor not work (we also confirm it in Section 4), partly because & overfits the model (Fig 5 in Kurutach al. (2018); Grill et al. (2020)), partly because the goal of is not only to exploit the traces T = U- (T) collected so far and model p LEARN(T), but also to collect data to self-improve p and ε/π in the next episode т. This second reason is particular in our iterated batch setup: in pure batch RL, exploration is not issue. We explore implicitly because of the randomness of planning HETERoGENEOUSRS in Fig 3), but it turns out that needs to be augmented by explicit exploration strategies. To show this, we experiment with strategies, i) HEATING: modulating
on the real system, does
5

---

Under review as a conference paper at ICLR 2022
the temperature of the guide distribution {(a)s), and ii) EPSGREEDY: choosing random action with probability E (Fig 4). The novelty of our approach is that, instead of constant T and E, tuned as set of temperatures [T]] and probabilities [s:][_ to further diversify the search and to let the planner to dynamically choose the right amount of randomization when selecting the best trace in Line 4 in Fig 3. Finally, similarly Lowrey al. (2019); Argenson & Dulac-Arnole (2020), we found that bootstrapping the planning with the learned value function at the end of each rollout trace (BOOTSTRAP in Fig 3) crucial for optimizing the performance with short horizon.
hyperparameters,
The main competitor of our GUIDE&EXPLORE actor is an actor that delegates all planning and exploration to the Monte-Carlo tree search of Silver et al. (2017)'s ALPHAZERO (Fig 4),
RSACTOR (0), Pre -P, T, L, n)):
return HETEROGENEEOSRS, (IT 1 11=1: L,
D planning
GUIDE&EXPLOREACTTRR( TTROC ,P, T,T,T. L,
guide policy ( ("Dyna-stylee) a set of guided explorer policies 3 return HETEROGENEOUSRSS ((Pi) 11,, I.,.
E- MFRL. PP. T)
2
[pi] < EXPLORE E.n)
b planning
ALPHAZZRROATOR (0), preyy .p. P,T.1. L,, n))
1
return ALPHAZERO (T)) mreey P. T, Tx
Figure 2: Model-based ACTORS (policies executed on the real system). RSACTOR is a classical random shooting planner that uses the random policy z(0) for all rollouts. Since the random policy has no value estimate, only TOTALREWARD can be used as VALUE Fig 3/Line 3 of HETEROGE- NEOUSRS. GUIDE& EXPLORE precisely, updates previous guide contained in P It can also use the traces T collected on the real system. It then "decorates" the guide by (possibly n. different) exploration strategies (Fig 4), and runs these reactive guide&cxplore policies [p:]== in the HETEROGENEOUSRS planner, either using the raw TOTALREWARD or the total reward bootstrapped by the value estimate of the guide policy & (BOOTSTRAP) in Fig 3/Line 3. ALPHAZEROACTOR calls ALPHAZERO which plans and explores internally, using Monte- Carlo tree search, with budget of L simulator calls.
first learns a Dyna-style guide policy on the model p (more
3.5 METRICS
We use two rigorously defined and measured metrics (Kégl et al., 2021) assess the performance of the different algorithmic combinations. MAR measures the asymptotic performance after the learning has converged, and MRCP measures the convergence pace. Both can be averaged over seeds, and MAR is also an average over episodes, so we can detect statistically significant differences even when they are tiny, leading to a proper support for experimental development.
MEAN ASYMPTOTIC REWARD (MAR). Our measure of asymptotic performance, the mean asymptotic reward, is the mean reward MR(T) = (T in the second half of the episodes (after
convergence; we set in such way that the algorithms converge after less than N/2 episodes) MAR = N E/=N/2 MR(T).
MEAN REWARD CONVERGENCE PACE (MRCP(F)). To assess the speed of convergence, we define the mean reward convergence pace MRCP(T) as the number of steps needed to achieve mean
reward F. smoothed over a window of size 5: MRCP(F) T' arg min, Er MR(() > F)
The unit of MRCP(F) is system access steps, not episodes, first to make invariant to episode length, and second because in micro-data RL the unit of cost a system access step. For Acrobot, we use 1.8 in experiments, which is roughly 70% of the best achievable mean reward.

---

Under review as a conference paper at ICLR 2022
TOTALREWARD (T):
1
return I x R(T)
>> total reward (a.k.a return) of trace
BOOTSTRAP(V,
return R (T()) + ( total reward value of last state
HETEROGENEOUSRS ( (fp L, ms)ss)
1
for +- to n:
2
T ROLLOUT Pi, 8, L)
ith roll-out trace total reward of T()) bootstrap
3
V & VALUE
4 1" 6- arg max, V(
index of the best trace
5 return [1,2]
> first action of the best trace
Figure 3: VALUE estimates rollout traces and HE HETEROGENEOSRRS, random shooting with a set of policies. TOTAI REWARD and BOOSTRAP are two ways to evaluate the value of a rollout trace. The latter adds the value of the last state to the total reward, according to a value estimate : + R weighted by a hyperparameter a. They are called in Line 3 of HETEROGENEOUSRS which random shooting planner that accepts n different shooting policies [a,],=1 for the rollouts used the search. As usual, it returns the first action aj = T [1, 2] of the best trace T()) (($1, aa)..................... (sir, a.)). parameters are the shooting policies [p: n the model p. and the number n and length of rollouts, but to properly define it, we also need the state which we plan from, so we use double argument list Ol-
HEATINGEXPLORE ( [))
for i < 1 to E(als)!/T: pi(als)
Ea' {(a4
return [p:]E=1
EPSGREEDYEXPLORE (0)) [s]:
for - to n:
arg max E(a|s)
with probability (1 Ei), with probability Et.
2
Pi(als) return [p:]
((0) (a)
Figure 4: Exploration strategies. HEATINGEXPLORE heats the guide action distribution {(a)s) to n different temperatures, and EpSGREEDYEXPLORE changes the best action to random ac- tion (0) a with different probabilities. The temperatures [T,]%= and probabilities [e,] are hyperparameters.
EXPERIMENTS
4.1
THE ACROBOT BENCHMARK EN ENVIRONMENT
Acrobot is an underactuated double pendulum with four observables 8, = [0,, 02, 0,, 02] which are usually augmented to six by taking sine and cosine of the angles (Brockman al., 2016); 02 the angle to the vertical axis of the upper link; 0g is the angle of the lower link relative to the upper link, both being clipped to [- n|; 01 and bg are the corresponding angular momenta. For the starting position s of each episode, all four state variables are sampled uniformly from an approximately hanging and stationary position s!
1.0.1.0.11 The action discrete torque on the lower link E 0, The reward is the height of the tip of the lower link over the hanging position r(s) = - cos 0, - cos + 02) [0, 4].3
We chose this rather than the sparse variable-episode-lnnge version r(s) T {2 cos 01 cos(#: 02) > 3} (Sutton, 1996) since it corresponds better to the continuous aspect of engineering systems.

---

Under review as a conference paper at ICLR 2022
Acrobot is small but relatively difficult and fascinating system, so it is an ideal benchmark for Sesiiuous-rewanr engineering systems. Similarly to Kégl et al. (2021), we set the number of episodes to N 100, the number of steps per episode to T 200, the number of planning rollouts to n = 100, and the horizon to I = 10. With these settings, we can identify four dis- tinctively different regimes (see the attached videos): i) the random uniform policy π (0) achieves MAR 0.1 0.2 (Acrobot keeps approximately hanging). ii) reasonable models with random shooting or pure Dyna-style controllers achieve MAR = 1.4 - 1.6 (Acrobot gains energy but moves its limb quite uncontrollably), iii) random shooting n 100, L 10 with good models such as PETS (Chua et al., 2018; Wang et al., 2019) or DARMIN (Kégl et al., 2021) keep the limb up and manage have its tip above horizon in our experiments we could achieve a quasi perfect policy (Acrobot moves up like a gymnast and stays balanced at the top) MAR R 2.7 2.8 using random shooting with 100 K, L 20 on the real system, giving us a target and possibly large margin of improvement. Acrobot is also an ideal benchmark for making our point since it turned out quite challenging for Dyna-style techniques in Wang et al. (2019)'s s benchmarks (MAR R 1.6 1.7).
average MAR x 2.1-2.1 (previous state of the art), and iv)
4.2 MODELS, GUIDES, AND ACTORS
Following Kégl et al. (2021), we tried different system models (Fig 1/Line 4) from the family of Deep Autoregressive Mixture Density Networks (DARMDN) and decided to use DARMDN(1 Dan- an autoregressive network trained probabilistically (the output of the net for each observable di- mension is a single Gaussian learned by minimizing the negative log-likelihood loss) and sampled deterministically (the mean of the Gaussian). In Kégl et al. (2021) this was the best model using random shooting; Table in the Appendix confirms this using our best actor.
In principle, any MFRL technique working with discrete action space and providing a value function and a policy can be used as a guide & when applying I'ERATEDMBRL to the Acrobot system (Fig 2/Line 1). We experimented with Deep Q Networks (DQN; Mnih et al. (2015)) and PPO (Schulman et al., 2017). For the DQN, following Janner et al. (2019), short rollouts starting from real observations are performed on the model to sample transitions which are then placed in an experience replay buffer, along with the real transitions observed during the rollouts (Fig i/Lene 7)- The DQN is then updated by sampling batches from this buffer. For DQN alone (without planning), an e-greedy strategy is used for its training (s decaying as more training samples are used) and at decision time (s 0.05).
For actors (Fig I/Line 5, Fig 2), we start from the simple DQN(s = 0.05) or PPO guide which is in- teracting with the system without planning. We then add different ingredients these simple agents to improve the performance. Adding (n. L) to the name of the agent means that we use the agent to guide a planning of n rollouts with horizon T.. It is important to note here that planning without ex- ploration using the greedy guide is, in our case, equivalent to no planning since both the model p and and XXX-EPSGREEDYEER
the guides { are deterministic. to the additional use of the associated exploration strategies (Fig 1/Line 2, Fig 4). When a fixed is used for the exploration strategy, we add it as an explicit parameter, e.g., TEPSGREDYYEPPLORIE No parameters means that different the value estimate is the TEWALREWARD (Fig 3). We add BOOTSTRAP when we use bootstrapping by the value function estimate. Finally, we also apply AlphaZero (Silver et al., 2017) in a Dyna-style fashion (Fig 2) which we refer to as DYNAZERO.
refer
temperature is used for each of the rollouts. By default
Appendix A contains detailed information on the various algorithmic choices.
4.3 RESULTS
Table 2 and Figure 5 present our the results with the main actors. We see that planning with explo- ration and bootstrapping gives the best results among all the DQN actors. The approach beats the previous state of the art: both RSACTOR(1 100, L = 10) from Kégl et al. (2021) and POPLIN-A (Wang& & Ba (2020); Table 1) by 10% in terms of MAR. PPO is slightly suboptimal compared to DQN. DYNAZERO matches the best MAR but converges three times slower and takes about five times more computational time, mainly because, unlike random shooting, MCTS cannot easily be parallelized.
Table 2 and the detailed ablation study in Appendix B show that all ingredients add to the perfor- mance. Although DQN alone (DQN(s- 0.05)) is far better than a random policy, it is clearly not
8

---

Under review as a conference paper at ICLR 2022
sufficient to obtain reasonable actor. Using it planning guide (DQN(n 100, L 10)- EPSGREEDY EXPLORE(E = 0.05)) significantly improves the performance but remains worse than random shooting RSACTOR(n 100, L 10). To propel the Dyna-style actor above simple MPC, we need to fine tune exploration. We found that allowing the planner to choose the right amount is a robust and safe approach. The final improvement heated explorer/planner; bootstrapping does not improve the s-greedy actor, possibly because the value estimate of the guide is off random actions, unlike the heating exploration when the explorer "softly" increases the probability of actions, given by the guide. As a reference, we also include simple but costly RSACTOR(n 100 K, 10). would be officially accepted in our benchmark as we restrict to 100 and L to 10, but can serve target to reach.
attained by adding value bootstrapping the
Table 2: Agent evaluation results. MAR is the Mean Asymptotic Reward showing the asymp- totic performance the agent and MRCP(1.5 the Mean Reward Convergence Pace showing the sample-efficiency (the number of system access steps required to achieve a mean reward of 1.8). + and mean lower and higher the better, respectively. Except for DYNAZERO all the agents were run for 10 seeds with the + giving the 90% confidence interval.
Agent
MAR t
MRCP(1.8) +
RSACTOR(n 100 K, 20)
2.474=0.022 2.075=0.01
2280.0#580.0
RSACTOR(n = 100, 10)
2620.0+320.0
DQN(E 0.05)
1.442=00.014
NaN =NaN
DQN(n 100, L = 100-100)--- EPSGREEDYEXPLORRIC 0.05)
1.932=0.012
3540.0+520.0
DQN(" = 100, L 10)-HHEATINGEXXPLORE
2.196:10.014
190000++800.
DQN(" 100, -= DDD-EESSGRREDDYEXP.OR
2.203-000012
1800.0+
DQN(n 100, I. HD-HHEATIINEEXPLOR---OOOOM BOOTTSTRAP
2.283+-00012
2180.0+-280.0
PPO
1.578++0.03
NaN +NaN
PPO(n 100, L 10)- HE-HHIIGEEXLLORE
2.161
2520.01600.0
DYNAZERO
2.304
6000.01NaN
2.5
Agent
DQN
DQN planning with muli-teemerrauuee exploration
2.0
DCN planning with nuuttttameeraure explorstion and value bootstrap
DynaZer
mean reward
Random Shooting (previous state of the art)
1.5
1.0
0.5
0.0
0
8,000
16,000
system access step
Figure 5: Learning curves obtained with different agents. Mean reward between (hanging) and 4 (standing up). Episode length is T 200, number of epochs is N 100 with one episode per epoch. Except for DYNAZERO, mean reward curves are averaged across ten seeds. Areas with lighter colors show the 90% confidence intervals.
5 CONCLUSION
In this paper we show that an offline Dyna-style approach can be successfully applied on the Ac- robot system where previously Dyna-style algorithms were failing, Our empirical results exhibit the importance of using a planning guided by a policy with the correct amount of exploration. We thus propose a strategy to automatically fine-tune exploration and add bootstrapping with a value func- tion estimate to further improves the performance. This combination leads an improvement over the previous state of the art by 10% while respecting the same resource constraints. Future work includes modelling the uncertainties of the value estimates so to use them for better exploration.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
All the code to produce our results will be made publicly available after publication. Details for the implementations of ITERATEDMBRL, the different exploration strategies, the system models, and the different MFRL agents are also provided in Section A.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Arthur Argenson and Gabriel Aulac-Amold. Model-based offline planning. CoRR, abs/2008.05556 2020. URL https: /tte:s//aaxiiii org/ abs/2000 05556.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI gym, 2016.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems 3/, pp. 4754-4765. Curran Associates, Inc., 2018.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-Based Reinforcement Learning via Meta-Policy Optimization. In 2nd Annual Confer- ence on Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, volume 87 of Proceedings of Machine Learning Research, pp. 617-629. PMLR, 2018.
Pieter-Tjerk de Boer, Dirk P. Kroese, Shie Mannor, and Reuven Y. Rubinstein. A tutorial on the cross-entropy method. Annals of Operations Research, 134, 2004.
Marc Peter Deisenroth and Carl Edward Rasmussen. PILCO: A model-based and data-efficient approach policy search. In Proceedings of the International Conference on Machine Learning, 2011.
Jean-Bastien Grill, Florent Altché, Yunhao Tang, Thomas Hubert, Michal Valko, Ioannis Antonoglou, and Remi Munos. Monte-Carlo tree search as regularized policy optimization. In Proceedings of the 37th International Conference on Machine Learning, pp. 3769-3778. PMLR, 2020. URL https://///.. htt:://pooeeeeiig/s//eee www..ppres/ee.. http//v11gggggr/ httggrrll1200 html.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the 36th In- lernational Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2555-2565, 2019.
Jessica Hamrick, Abram Friesen, Feryal Behbahani, Arthur Guez, Fabio Viola, Sims Witherspoon, Thomas Anthony, Lars Buesing, Petar Velikovi, and Theophane Weber. On the role of planning in model-based deep reinforcement learning. In 9th International Conference on Learning Rep- resentations, ICLR 2021, To appear, 2021. URL https: /atte///ppeeeeee wpwpreeeeeeeeeeee///eeeee net forum?id= IrM64DGB21.
Jessica B Hamrick. Analogues of mental simulation and imagination in deep learning. Current Opinion in Behaviorul Sciences, 29:8-16, 2019. ISSN 2352-1546. doi: https://doi.org//..001//// cobeha.2018. 12.011. URL https: / www. sciencedirecte wttp/cccc/cc.eeeee/e http///iinnn/rrrtllllee ppi/s235215 51222154688888 46183016 670. Artificial Intelligence.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Sheila Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (AA1-18), and the 8th AAAI Symposium on Ed- ucational Advances in Artificial Intelligence (FAAI-18), pp. 3207-3214. AAAI Press, 2018.
G. Zacharias Holland, Erin J. Talvitie, and Michael Bowling. The Effect of Planning Shape Dyna-style Planning in figh-dimensional State Spaces. arXiv preprint wxXiv1806.01822 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché- Buc, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Balázs Kégl, Gabriel Hurtado, and Albert Thomas. Model-based micro-data reinforcement learn- ing: what the crucial model properties and which model choose? In 9th International Conference on Learning Representations, ICLR 2021, 2021. URL https: oppenrvvie.. ftt.formmmm11ppyyycc4556666
11

---

Under review as a conference paper at ICLR 2022
Rahul Kidambi, Aravind Rajeswaran, Pranceth Netrapalli, and Thorsten Joachims. MOReL: Model- Based Offline Reinforcement Learning. In H. Larochelle, M. Ranzato. R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21810- 21823. Curran Associates, Inc., 2020.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations, 2018. URL https::p////mmm opwnrevieweeeeeeeeeeee net forttt/orm??ii id=SSJinnWR...
Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Rein- forcement Learning, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.06643 2020.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model -ased control. In International Conference on Learning Reprexentations, 2019. htRp:t/s:/// hwtp///ppppeeeeeeeeeeeeeeeeeeee. hwtp:/.eeooooooooo forum? Ad=Byeyyn055F)
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorith- mie Framework for Model- Deep Reinforcement Learning with Theoretical Guarantees. In International Conference on Learning Representations, 2019. URL pesttss:/oppereeeee net/ fort/f i d== 1E2R5KX.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529553. 2015.
Thomas M. Moerland, Joost Broekens, and Catholijn M. Jonker. Model- reinforcement learn- ing: A survey. arXiv preprint arXiv:2006. 16712, 2021.
Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural network dy- namics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, pp. 7559-7566. IEEE, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8035-8035. Curran Associates, Inc., 2019.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dor- mann. Stable baselines3. https: : //github..b/./////// com/ DLR- RM/ www.aaleebbbllleeee liness, 2019.
Anil Rao. A survey of numerical methods for optimal control. Advances in the Astronautical Sciences, 135, 01 2010.
Arthur George Richards. Robust constrained model predictive control. PhD thesis, Massachusetts Institute of Technology, 2005.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, and et al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature, 588(7839))604609, Dec 2020. ISSN 1476-4687. doi: 10. 10.0338411866-000000001111 10. / s41///41550 s11///1158866661 5-1-1886-2222 020 020-03051-44
URL http: /ttpp::/dxx.eo .tw.dxx.xdooooooo dof orgg
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust Region Policy Optimization. In Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889-1897, Lille, France, 07-09 Jul 2015. PMLR.
12

---

Under review as a conference paper at ICLR 2022
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Si- monyan, and Demis Hassabis. Mastering chess and shogi by self-play with general reinforce- ment learning algorithm, 2017.
Wen Sun, Geoffrey Gordon, Byron Boots, and J. Bagnell. Dual policy iteration. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gareet (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bull., 264):16, July 1991. ISSN 0163-5719.
Richard S Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo (eds.), Advances in Neural Information Processing Systems 8, pp. 1044. MIT Press, 1996.
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. In 8th Inter- national Conference Learning Representations, ICLR 2020, 2020.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforce- ment learning. arXiv preprint arXiv: 00000090000077 2019.
Yuxiang Yang, Ken Caluwaerts, Atil Iscen, Tingnan Zhang, Jie Tan, and Vikas Sindhwani. Data efficient reinforcement learning for legged robots. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), Proceedings of the Conference on Robot Learning, volume 100, pp. 1-10, 2020.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based Offline Policy Optimization, In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc., 2020.
volume 33, pp. 14129-14142.
Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan O. Lambert, André Biedenkapp, Kurtland Chua, Frank Hutter, and Roberto Calandra. On the importance of hyperparameter optimization for model-based reinforcement learning. In Arindam Banerjee and Kenji Fukumizu (eds.). The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, volume 130 of Proceedings of Machine Learning Research, pp. 4015-4023. PMLR, 2021.
13

---

new real Under review as a conference paper at ICLR 2022
A IMPLEMENTATION DETAILS
A.1 CODE AND DEPENDENCIES
Our code will be made publicly available after publication to ease the reproducibility of all our results. We use Pytorch (Paszke et al., 2019) to build and train the neural network system models and policies. To run the ITERATEDMBRL experiments we use the ri s__ssmulato (https: / //github. com/ ramp-kits/
simulatoon python library developed by Kégl et al. (2021) which relies on Open Al Gym (Brockman et al., 2016) for the Acrobot dynamics. For the PPO agent we use the StableBaselines3 implementation (Raffin et al., 2019).
A.2 MODELS AND AGENTS
is known that carefully tuning hyperparameters of deep reinforcement learning algorithms is cru- cial for success and fair comparisons (Henderson al., 2018; Zhang al., 2021). To reduce the computational cost and consider a reasonable search space the models and the agents were opti- mized independently. For the systems models we use the same hyperparameters Kégl et al. (2021). Please refer to Appendix D in Kégl et al. (2021) for a complete description of the hyperparameter search and the selected hyperparameters.
the ones used in
The DQN is neural network with one hidden layer of 64 neurons. The input and hidden layers are both made of a linear layer followed by the ReLU activation function. The output layer is a linear layer. At the start of each epoch of ITERATED the system model is trained using all the data collected at the previous epochs and the DQN updated offline from both transitions generated with the model and the real observed transitions. Following Janner et al. (2019) and Holland et al. (2019), short rollouts starting from real observations are performed on the model to sample transitions which are then placed in experience replay buffer with total size 25,000. The real transitions observed during the rollouts on the real system are also placed in this buffer. We train and act with E-greedy strategy with
decaying as see more samples. The decay schedule is given by max(0.6 0.0013756, 0.05) where is the current number of steps realized on both the model and the real system. The DQN is updated by sampling batches of size 64 from this buffer. To generate transitions from the model we use a total number of 1800 steps. Each rollout starts from randomly sampled real observation and performs 50 steps before being reset observation. The DQN is updated after each two steps. We use a target network which is updated after every 400 steps. Finally the learning rate is set to 0.0001 and the discount factor to 1. To obtain these values for the different hyperparameters, we started from default values and carefully tuned them to achieve the best performance.
The PPO agent is also updated offline at the start of each epoch with the model. We rely on the Sta- bleBaselines3 (Raffin et al., 2019) implementation using the default Multi-Layer Perceptron policy. A total of 15000 steps are generated and each rollout resets from real observation each 20 steps.
For DYNAZERO we modified the version of AlphaZero available at https: /ttp::://ghhuut com/ tmoer/ / a alppaaeroo weahhaeroo_ssnggpplllleeeeeeeeee singleplaayrr. For the policy-value network we use two hidden layers with 64 neurons each. The input and hidden layers are both made of a linear layer followed by the ELU activation function. The output layer for the policy is softmax layer and a linear layer for the value. The constant e in the UCT rule is set to 1.5 and the discount factor to 0.9. The policy-value network is updated at the beginning of each epoch by performing rollouts the model, reset after each 50 steps. The total number of steps performed on the model at each update is 400. Data are put in buffer of size 10,000. The batch size set to 64 and the learning rate to 0.001. Finally the number of MCTS searches at each step is set to 500.
A.3 EXPLORATION
For the multi-a exploration strategy based on EPSGREEDYEXPPLOE we use one s value for each of the = 100 rollouts: (0.001, ..., For the multi-temperature HEATING- EXPLORE strategy, we first normalized the values by their maximum value, Q(8, a)
14

---

Under review as a conference paper at ICLR 2022
(s,,a)/ max a(s, before applying softmax:
Pi(als:) Ee x1/T:
where 1 n is an increasing sequence of temperatures. A large temperature gives uniform distribution, whereas low lemperature corresponds taking arg max ((, st). Different shapes of sequences were tried (linear, logarithmic, exponential, polynomial, logistic), and best performance was obtained with logistic schedule (with linear end). The exact values will be provided in the code.
A.4 BOOTSTRAPPING
For DQN the value estimate is computed by taking the maximum of the Q values over the actions. The a parameter is 0.6.
B ABLATION STUDY
B.1 ACTORS
We show in this section that guided planning and exploration needed obtain the best result. Remove either planning or exploration and you will obtain suboptimal performance. To support this claim we ran ablation study with EPSGRREDDEXPPOORR EPSCREEDYEXPI.OR makes it easy to control the degree of exploration through the parameter. Setting to 0 corresponds to no exploration and is equivalent using the guide greedily without planning (n and L as our model is used deterministically when sampled from. Setting ε to 1 corresponds to full exploration 100, = 10). Figure shows the results
and is equivalent the purely random RSACTOR(M obtained by DQN(s = 0.05), DQN with forced e-greedy scheme at decision time with 0.4 and no planning (DQN(E 0.4)), and DQN with planning and fixed & values (DQN(n = 100, L 10) EPSGREEDG EXPLORE(:) for {0.0001, 0.01, 0.05, 0. 0.2, 0.4, 0.8, 0.99, 0.9999)). All the results are also put in Table 3.
First,
that planning definitely helps when comparing the performance of DQN(= 0.4) which uses no planning but forces the exploration with e 0.4 and the performance of DQN(n = 100, L 10) EEPSGREEDYEXPLOREE Second, as expected, the closer E to 0 the closer the performance is to DQN(E = 0.05), and the closer is the closer the performance is to ACTOOR 100, T, 10). With well-chosen e between these two extremes, say = 0.4, we obtain a better performance than either extremes.
0.4) which uses planning and the same fixed value of e.
We thus claim that planning is required and planning with correct amount of exploration. Our EpSGREEDYEXPH or HEATEDEXPLORE exploration strategies, used with multiple & or tem- peratures values allows for the automatic and dynamic selection of the good amount of exploration.
B.2 MODELS
In Table compare the performances of models from Kégl al. (2021) when applied with our best agent DQN(n = 100, L = long to the family of Deep (Autoregressive) Mixture Density Networks with D components, D(AR) MMDN(D that the model sampled from deterministically, returning the mean of the predicted distribution. DARNNs are Deep Autoregressive Neural Networks point-estimating a mean by minimizing the mean squared error and estimating a constant variance using the residual errors. These models are homoscedastic with a variance independent of the input. The reader can also refer to Kégl et al. (2021) for a complete description of these models. In the paper we use the second best model DARMDN(1)ac over the non-autoregressive DMDN(1) because we find autoregressive models to be easier to use in practice on real-life projects due to their ability to model dependence between the observable dimensions and to model different feature types. We also note that, as observed in Kégl et al. (2021) with RCAOOoR = 100, L = 10), the deterministic versions perform slightly better than the stochastic ones, in general.
Most models be-
trained by minimizing the negative log-likelihood. The 'det' suffix means
15

---

Under review as a conference paper at ICLR 2022
DQN(n= = 100, == 10--Epsrredyyxxloreede
2.2
Mean Asymptotic Reward
1.8
1.4
DON 0.05): DQN e decision
1.0
DONie 0.4): DON with = 0.4 at decision time
Random Shooting (previous state art)
planning multi-epsilon exploration
0.8
DONin 100, 10) Epsrrredddyyyy
Figure 6: Mean asymptotic rewards (MAR) of DQN(n = 100, L. 10)-EEPSRREEDYEXPP.OOO obtained for different values of Error bars and areas in lighter colors represent the 90% confi- dence intervals. When is close to 0 the performance is close to DQN while for e close to the performance close the RSACTOR agent. Our multi-e exploration strategy able to select the best E automatically and dynamically.
Table 3: Importance of planning and exploration. MAR is the Mean Asymptotic Reward showing the asymptotic performance of the agent and MRCP(1.8) is the Mean Reward Convergence Space showing the sample-efficies setileefffciiccy performance as the number of system access steps required to achieve a reward 1.8. + and t mean lower and higher the better, respectively. The + values are 90% Gaussian confidence intervals.
Agent
MAR
MRCP(L8) J
DQN(E = 0.4)
1.209100001
NaN NNaN
DQN(E = 0.05)
1.442+0.014
NaN NNaN
DQN(n 100, I EEEEERGGREEEEDEPPPRREIII 0.0001)
1.475=+0.062
NaN NNaN
DQN(n 100,
EO-EEEESGRREEEEEPPOOREEEE 0.01)
1.664+00032
NaN #NaN
DQN(1 100, 10--EPSGRREEDYEXXPLOOEE 0.05)
1.932-10001
3540.0-.500.0
DQN(1 = 100, L = 10)-EISGRREEDYEXPPLOGG = 0.1)
2.009110.031
2400.011-
RSACTOR(T: 100, L 10)
2.075.10.01
2620.0-120..0
DQN(n 100,
LUD-EPSGGREEDEEXPPLOEE -0.9999)
2.107
2000.0
DQN(n 100,
DDD-EESGRREDDEEXPORR
0.99)
2.118100.46
2400.0+
DQN(n 100,
EDDEEESSRREEDYEEPLOREEE = -0.2)
2.151+00034
2400.0+
DQN(1 = 100, 10)-EPSGRRHEDYEXXPLORR = 0.8)
2.196++0.037
2000.0+
DQN(n 100, L 10-FPSSGRREEYYXXPP.ORR ERRR 0.4)
2.204+0.011
1910.0+140.00
C RSACTOR PERFORMANCE ON THE REAL SYSTEM
We present the results one obtain on the real system with an RSACTOR and different values of the planning horizon L and the number generated rollouts n in Fig For the considered planning horizons a larger number of generated rollouts lead better performance. We also observed in simulations that for the Acrobot to stay balanced, was necessary (although not always sufficient) to have reward larger than 2.6. We from Fig that this be achieved with simple agent such as ACTOR but at the price very large number of generated rollouts. The goal therefore to design smarter agent that can come as close as possible this performance with limited budget.
16

---

Under review as a conference paper at ICLR 2022
Table 4: Model evaluation results with for our best guide&explore strategy DQN(n = 100, HEEATINGEXPLORE-BOOTSS
MAR the Mean Asymptotic Reward showing the asymptotic performance of the agent and MRCP(1.8) is the Mean Reward Convergence Space show- ing the sample-efficiency performance as the number of system access steps required to achieve reward of 1.8. + and f mean lower and higher the better, respectively. The performances com- puted from 10 random repetitions of ITERATEDMBRL.
Model
MAR
MRCP(1.8)
DMDN (1) 1)
2.308=10.01
1860.0=240.0
DARMDN
2.297=0.012
2180.0=170.0
DARMDN(1)
2.294+000012
2060.0-190.0
(1D
2.271++0.011
2120.0-400.0
DARMDN(10)
2.264+00.011
2840.0-720.0
DARMDN (10))
2.244110.013
1960.0-160.0
DARNN
2.222-10.011
2640.0-510.0
DARNN.
2.177-1.011
3380.0-430.0
(10)(10)
2.148:10.015
1070.0
2.8
2.6
2.4
Average reward
2.2
2.0
1.8
Planning horizon L
1.6
10
1.4
30
1.2
100
1,000
10,000 100,000 1,000,000
Number of generated rollouts n
Figure 7: Performance obtained with RSACTOR the real Acrobot system for different planning horizons and number of generated rollouts n. The plot shows the mean rewards obtained for sev- eral randomly initialized episodes of 200 steps. The error bars give the associated 90% confidence intervals. Note that since Acrobot has a discrete action space with three actions, the total number 310 59,049. The performance shown for h 10 and
of different rollouts for h 10
100 thus only requires n = 59,049 rollouts.
17


Reviews:
[{"review_id": "yzqYWfJib04", "reviewer": "Reviewer_jVBs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The studied problem on an approach to sample-efficient MBRL is clearly a very significant problem in modern research on RL. The authors have nice ideas, they enhance the traditional MBRL approach with two models (called the guide and the explorer), the guide is based on a Dyna-style policy, applying of which is known to be challenging in a long horizon setting. The paper is nicely written and easy to follow. I think that the designed methodology has a significant potential of being useful in the research on MBRL, but only after the presentation and experimental evaluation are mature enough.\n\nI would like that the researchers address my concerns listed below and related mostly to correct mathematical formalization of results, and validating the methodology in a more broad set of benchmarks and environments.\n\n### 1. Lack of proper formalization of the setting.\nAfter reading the studied problem statement in the introduction, it is still unclear for me what precisely can be characterized using the term ‘iterated batch RL’ coined by the authors. In particular, why it is necessary to define and name an MBRL study subgenre, rather than use the existing framework of online/offline MBRL. \nI appreciate the informal and visual presentation of the studied setting using the example of John, a telecommunication engineer. However, I would appreciate a formal statement and assumptions behind the studied iterated batch RL setting. Especially that it has been studied in the literature under different names. This critique extends further, in my impression the paper lacks mathematical rigor necessary in the setting of such a problem like acrobot (authors showing) being extremely sensitive to: the formulation of observation vector (sincos vs pure angles), initial condition (instability and chaotic behavior), employed model (probabilistic / multivariate / stochastic /deterministic).  \n\n### 2. Disputable significance of the reported improvement over the state-of-the-art in a single benchmark environment\nOne of the main selling points of the paper is the 10% improvement for the resource-limited acrobot (in terms of Mean Asymptotic Reward after the fixed number of system access steps. Basing the paper contribution on an improvement like that for a  single benchmark can be misleading. First, the issue of ensuring a sufficient number of runs with different random seeds seems to be addressed by averaging the result from 10 different runs (standard in RL literature) and reporting confidence intervals. However, the 10% improvement may be the result of a simple hyperparameter optimization.  It is now known that exhaustive hyperparameter optimization in MBRL may result in significant performance gains, and even solving for exploitative solutions, see B. Zhang et. al. _“On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning”_ AISTATS 2021.  Therefore, it is not unexpected that an exhaustive hyper-parameter search could significantly boost the MAR/MRCP scores of the other evaluated algorithms and change the final leaderboard. Table 3 given in the appendix indicates that the authors did not, in fact, perform an exhaustive hyperparameter search. Another concern is related to the fact that the DynaZero benchmark algorithm achieves higher MAR, and when hyper-tuned could become also the clear leader considering the other utilized metrics.\n\n### 3. Not clear motivation for choosing the Acrobot system as the studied benchmark\nAs for the motivation of choosing the Acrobot system as the benchmark, it is not clear at all. Page 8 starts with the sentence _“Acrobot is a small but relatively difficult and fascinating system, so it is an ideal benchmark for\ncontinuous-reward engineering systems”_. But I am not convinced, there is only a single other reference given providing the previous benchmark result K´egl et al. (2021). I would like to see more support for the claim that the Acrobot is the ideal system to study in the studied setting with adequate references.  A relation to classical optimal control theory would be a convincing example, and it could also serve as a baseline. \n\nNonetheless, even if the Acrobot system is an ideal benchmark it seems to be simplistic when compared to the usual continuous robotics RL benchmark environments studied in literature like Hopper or Half-Cheetah. I see significant differences between the Acrobot and the usual RL benchmark environments - the Acrobot does not require to deal with nonlinear forces generated by friction and contacts with the scene & other objects like in the other continuous RL benchmark envs. \n\n### Other remarks:\n\n* What is the ‘heating explorer’ used in the abstract,\n* Please clarify the terminology used for the studied setting, I found in the paper and related literature several names, I assume, for the same problem:  iterated batch RL/ growing batch RL/ micro-data RL / micro-data model-based RL. I know that the field is new, but this naming discrepancy can become confusing in the long run.\n* Please clarify what does precisely NaN reported in Table 2 is supposed to mean ?\n* p.7, fig.3, l.5: I do not see how to interpret the notation with brackets T^{(i^*)}[1,2]?\n", "summary_of_the_paper": "The authors study model-based reinforcement learning applied in the scenario of iterated batch reinforcement learning. There exist other namings of the studied in the paper scenario of iterated batch reinforcement learning, I found out that there exist several other synonymous names of such setting: growing batch RL/ micro-data RL / micro-data model-based RL. The authors motivate the problem informally using several bullet points listed in the introduction.\n\nThe main finding claimed by the authors is that the Dyna-style MBRL approach is an appropriate choice for the studied setting by enhancing it with decision-time planning based on a model-free policy and dynamically tuning exploration. \n\nThe algorithms are tested in the problem of resource-limited Acrobot; a detailed ablation study is provided for algorithm features (called here interchangeable algorithmic bricks for iterative batch RL. The performed ablation study helped to find the combination of a neural model, a bootstrapping DQN guide, and a heating explorer, which led to a 10% jump in state-of-the-art on the resource-limited Acrobot system. \n \n", "summary_of_the_review": "I am on the fence regarding the significance of the paper results. On one hand, it adds new ideas and establishes an interesting line of research. Eventually has a significant potential of being useful in the research on MBRL. \n\nHowever, the presentation of the paper is not mature enough. I think the paper has a few significant drawbacks including: lacks a proper mathematical formalization of the studied problem, and validation within a varied set of benchmarks and different scenarios. I explained my concerns in three paragraphs above. Therefore, I am leaning towards its rejection. An improved version of the paper may have a major impact.\n\nMore adequate score is 4.5", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "b7SF4KqN2b8", "reviewer": "Reviewer_H38y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The iterated batch RL motivation is interesting\n2. The scope of ablations is above average\n\nWeaknesses\n1. While I find the first paragraph or two of the introduction to be enjoyable, the informal, chatty approach to writing the rest of the paper detracts from the content. As an example: \"Our main contribution is a Dyna-style GUIDE&EXPLORE strategy (Fig 2). The gist is to learn a guide policy ξ using a model-free RL technique on the model p and on the traces collected on the real system T\". This is the main contribution! What is a guide, what is Dyna-style, why are we talking about a \"gist\"? Ordinarily I would avoid commenting on writing style in a review, but it must be addressed.\n\n2. Further, if we are to understand the above comment as the main contribution, the issue is that \"the guide\" is a concept, not a new concept at that (just a renaming), and this paper doesn't clearly state a hypothesis for any particular implementation.  The paper spends a section addressing terminology in the description of the guide we are told that sometimes the guide is trained on data from the model of the system (mostly in fact), but also sometimes (\"when possible\") trained with off policy data.\n\n3. Again in the \"terminology section\" the following appears: \"In that sense, through the data from the **real system**, planning is part of training the guide\". It appears that here the paper refers to the OpenAI Gym as the **real system**, the paper should clarify that a better experimental setup would include a real, real system, a simulator of the real system, and a learned model of the real system. Others have done physical acrobot experiments (c.f. https://arxiv.org/pdf/1210.0888.pdf, http://youtu.be/FeCwtvrD76I)\n\n4. Why do NaNs appear in some of the tables?\n\n5. What is the contribution? I see the list at the end of the introduction, but none are testable/falsifiable statements.", "summary_of_the_paper": "The paper describes meta-method and metrics for solving iterated batch reinforcement learning problems. The paper concludes with experiments on an \"acrobot\" simulation. ", "summary_of_the_review": "This paper explored a variety of approaches for iterated batch reinforcement learning. There is no obvious contribution other than the demonstration that experimentally both planning and exploration are important to improving the performance of Acrobot policies. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0nWdLHQTBsk", "reviewer": "Reviewer_EmNA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper empirically shows that the proposed changes involving exploration, bootstrapped returns, and other additions improve performance of dyna algorithms in Acrobot.  However, this seems like a limited result as currently presented.  Why were these extensions not studies in other environments, especially more realistic ones like those described in the introduction?  The paper spends a lot of space describing hypothetical industrial applications but never presents domains of that kind.  In addition, the changes proposed here, especially the automated exploration techniques are not theoretically investigated.  Are there any guarantees about the convergence of Dyna+these extensions in the tabular case?  \n\nThe main contribution of the paper are the empirical results on Acrobot.  But in the main paper, only a few algorithms are actually shown in Table 2 and Figure 5 (for instance where are the XXX variants?).  And the overall results in Figure 5 show virtually no difference in the asymptotic behavior of the different variants and only a small difference in convergence speed, which seems within the margin of error for the two planning variants and even comparing to current state of the art baseline (purple) there seems to be overlap by the error bars shown. I also question the use of a single percentage (70% here) in calculating the MRCP.  Ideally one would calculate convergence rate for several values, say 70%, 80%, and 90%, to make sure that convergence is proceeding at a fast pace throughout the learning process.  With only a single domain studied and marginal gains I think more studies are needed here.\n\nThe paper also does not provide any theoretical analysis of how the proposed changes will affect convergence.  Does the new “heating” exploration guarantee the optimal policy is preserved?  Is it guaranteed to converge in the limit?  What is the expected sample complexity of using this approach?  \n\nAnd what about the approach described on Page 6 where the planner “dynamically chooses the right amount of randomization” for exploration?  How is that actually implemented?  Is the randomization part of the available action space?  What kind of convergence guarantees are available there?  Page 9 claims that the paper shows “allowing the planner to choose the right amount is a robust and safe approach” but I do not really see evidence of that nor a concrete algorithmic description of how that procedure operates.\nLack of theory and concrete description of the hyper-parameter-less exploration\n\nOrganizationally, the paper seems unfocussed. The first page is spent on hypothetical business cases that the paper never returns to.  The related work section lists many different dyna-style algorithms but it is hard to find most of them in the actual algorithm results. I also found the pseudocode blocks quite hard to follow when comparing the different algorithm variants.  A better way to convey this information would be using a matrix where each row has an algorithm name and the columns describe the exploration type, the planner used, etc.\n", "summary_of_the_paper": "The paper presents extensions and modular additions to dyna-style algorithms, especially involving bootstrapping and improved exploration.  These changes are studies on the classical, but difficult, Acrobot domain and it is shown empirically that the changes improve convergence speed and asymptotic performance. ", "summary_of_the_review": "The new algorithm is interesting and seems to make some improvements over some of the existing approaches, but the gains seem marginal, some variants re not shown in the main results, and the reported statistics could use improvement.  there is also no theoretical justification or proofs provided for these changes,", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies dyna-style MBRL in a resource-limited setting. It is evaluated on an acrobat task where it shows very promising results.

The reviewers appreciated the extensive replies, but they did not fundamentally change their opinion. In particular:
- Lack of formal problem statement and definitions
- The experiment on a single task (and that being a non-standard version) isn't sufficient to demonstrate the general merits of the method

While the ideas are very promising, the paper cannot be published in its current form. We'd hence like to highly encourage the authors to revise the paper and to re-submit at a different venue.
Decision: Reject