Year: 2021

Abstract:
Iterated batch reinforcement learning (RL) is a growing subfield fueled by the demand from systems engineers for intelligent control solutions that they can apply within their technical and organizational constraints. Model-based RL (MBRL) suits this scenario well for its sample efficiency and modularity. Recent MBRL techniques combine efficient neural system models with classical planning (like model predictive control; MPC). In this paper we add two components to this classical setup. The first is a Dyna-style policy learned on the system model using model-free techniques. We call it the guide since it guides the planner. The second component is the explorer, a strategy to expand the limited knowledge of the guide during planning. Through a rigorous ablation study we show that exploration is crucial for optimal performance. We apply this approach with a DQN guide and a heating explorer to improve the state of the art of the resource-limited Acrobot benchmark system by about 10%.

heading: 1 INTRODUCTION
text: John is a telecommunication engineer. His day job is to operate a mobile antenna. He has about forty knobs to turn, in principle every five minutes, based on about a hundred external and internal system observables. His goal is to keep some performance indicators within operational limits while optimizing some others. In the evenings John dreams about using reinforcement learning (RL) to help him with his job. He knows that he cannot put an untrusted model-free agent on the antenna control (failures are very costly), but he manages to convince his boss to run live tests a couple of days every month.
John’s case is arguably on the R&D table of a lot of engineering companies today. AI adoption is slow, partly because these companies have little experience with AI, but partly also because the algorithms we develop fail to address the constraints and operational requirements of these systems. What are the common attributes of these systems?
• They are physical, not getting faster with time, producing tiny data compared to what model-free RL (MFRL) algorithms require for training.
• System access is limited to a small number of relatively short live tests, each producing logs that can be used to evaluate the current policy and can be fed into the training of the next.
• They are relatively small-dimensional, and system observables were designed to support human control decisions, so there is no need to filter them or to learn representations (with the exception when the engineer uses complex images, e.g., a driver).
• Rewards are non-sparse, performance indicators come continually. Delays are possible but usually not long.
• Safe operation even during live tests is crucial. Simulators and digital twins are more and more available, but some systems are too complex to simulate from first principles.
The RL setup that fits this scenario is neither pure batch (offline; Levine et al. (2020)) as interacting with the system is possible during short periods of time nor pure online as the policy can only be
updated offline between two interaction periods. We refer to it as iterated batch RL (Lange et al. (2012) call it growing batch). Furthermore, we are interested in model-based RL (Deisenroth & Rasmussen, 2011; Chua et al., 2018; Moerland et al., 2021) because i) it is sample efficient (Chua et al., 2018; Wang et al., 2019), ii) it works well on small-dimensional systems with dense rewards, and iii) the system model (data-driven simulator) itself is an object of interest because it can ease the adoption of data-driven algorithms by systems engineers.
Given a robust system model, simple model predictive control (MPC) agents using random shooting (RS; Richards (2005); Rao (2010)) or the cross entropy method (CEM; de Boer et al. (2004)) have been shown to perform remarkably well on many benchmark systems (Nagabandi et al., 2018; Chua et al., 2018; Wang et al., 2019; Hafner et al., 2019; Kégl et al., 2021) and real-life domains such as robotics (Yang et al., 2020). On the other hand, implementing successfully the seemingly elegant Dyna-style approach (Sutton, 1991; Kurutach et al., 2018; Clavera et al., 2018; Luo et al., 2019), when we learn model-free agents on the system model and apply them on the real system, remains challenging especially on systems that require planning with long horizon. Our main finding is that the Dyna-style approach can be an excellent choice for iterated batch RL by adding to it a decisiontime planning guided by the model-free policy and a dynamically self-tuning exploration. The key is that exploration and bootstrapping with a value function estimate affords us shorter rollouts so we can reduce the accumulation of errors that plagues the Dyna-style approach with long horizons (Janner et al., 2019). We also innovate on the experimental framework (metrics, statistically rigorous measurements), so we can profit from the modularity of the Dyna-style approach, tuning ingredients (model, the MFRL guide policy, exploration, planning, bootstrapping) independently. This modular approach makes engineering easier (as opposed to monolithic approaches like AlphaZero (Silver et al., 2017)), which is an important aspect if we want to give the methodology to non-expert systems engineers.
We demonstrate our GUIDE&EXPLORE approach on the small-dimensional but difficult Acrobot system which is known to be especially tough for Dyna-style techniques (Wang et al., 2019). We show in a rigorous ablation study how each ingredient adds a small but significant improvement, achieving a 10% total margin over the current state of the art (Wang et al., 2019; Wang & Ba, 2020; Kégl et al., 2021). DYNAZERO (pure Dyna-style with an AlphaZero guide) also matches the asymptotic performance with a larger sample complexity (3x) and computational (5x) price tag.

heading: 1.1 SUMMARY OF CONTRIBUTIONS

heading: 2 RELATED WORK
text: The MBRL subfield has seen a proliferation of powerful methods, but most of them miss the specific requirements (solving problems irrelevant in this scenario like representation learning or sparse rewards) and missing others (limited and costly system access; data taking and experimentation through campaigns, live tests; safety) (Hamrick, 2019).
The Dyna framework developed by Sutton (1991) consists in training an agent from both real experience and from simulations from a system model learned from the real data. Its efficient use of system access makes it a natural candidate for iterated batch RL. The well-known limitation of this approach is the agent overfitting the imperfect system model (Grill et al., 2020). A first solution is to use short rollouts on the model to reduce error accumulation as done in Model-Based Policy Optimization (MBPO; Janner et al. (2019)). Another solution is to rely on ensembling techniques for the model. Kurutach et al. (2018)’s ME-TRPO is based on an ensemble of models and Trust Region Policy Optimization (TRPO; Schulman et al. (2015)). At each TRPO step a transition is sampled from a randomly picked model of the ensemble, preventing the policy from overfitting one model. An ensemble is also used in Model-Based Meta-Policy Optimization (MP-MPO; Clavera et al. (2018)) where each model of the ensemble is seen as a different task used to meta-learn a policy. The meta-policy is then able to quickly adapt to any of the dynamics model and is more robust to model inaccuracies. Instead of learning the model and then the policy from the model, Stochastic Lower Bound Optimization (SLBO; Luo et al. (2019)) alternates between model and policy updates. In our ITERATEDMBRL skeleton (Fig 1), this strategy would couple the LEARN and MFRL steps, which we do not study in this paper. We note that according to the results shown in Fig 1.(a) and Table 1 in Wang et al. (2019), ME-TRPO, SLBO and MB-MPO alone (pure Dyna, without planning) are clearly suboptimal on the Acrobot system1, similar to the performance obtained by our pure Dyna DQN (Section 4). Finally, Yu et al. (2020) and Kidambi et al. (2020) use a Dyna-style approach in the context of pure batch RL where no further data collection and therefore no further model updates are assumed.
The idea of using a guide and a value function when planning is not novel (Silver et al., 2017; Schrittwieser et al., 2020; Wang & Ba, 2020; Argenson & Dulac-Arnold, 2020). We were greatly inspired by these elements in our objective of building smarter agents as they can make the search more efficient and thus lead to a better performance. POPLIN-A (Wang & Ba, 2020) relies on behavior cloning (using only real experience, unlike our Dyna-style approach that mainly uses the model), but their guide is similar to our approach. During the planning, they add random noise to the actions recommended by a deterministic policy network and update the noise distribution using a CEM strategy. In a similar way our GUIDE&EXPLORE strategy also adds a carefully controled amount of noise to the recommended actions. Our results highlight the importance of a well-calibrated exploration, which also contributes to the understanding of POPLIN-A. Argenson & Dulac-Arnold (2020) and Lowrey et al. (2019) both found that bootstrapping with a value estimate improves the performance of simple guided MPC strategies. The popular AlphaZero (Silver et al., 2017) and MuZero (Schrittwieser et al., 2020) algorithms also rely on a guide and a value function for their Monte Carlo Tree Search (MCTS). We implement a Dyna-style version of AlphaZero, which we call DYNAZERO. The principal issue of MuZero (Schrittwieser et al., 2020) in our micro-data iterated batch RL context is that it does not control the number of system access steps: it needs to simulate a lot from the real environment to establish the targets for the value function. In these two algorithms the guide is updated from the results obtained during the search that it guided, a procedure known as Dual Policy Iteration (Sun et al., 2018). We prefer experiencing with Dyna-style approaches first to leverage popular MFRL algorithms and defer the study of Dual Policy Iteration to future work.
Our results show that planning is an important ingredient, a claim already made by Hamrick et al. (2021). They use MuZero to run their ablation study while we prefer using an explicit model for practical reasons explained in the introduction. Besides planning we also study the importance of exploration.
1The conversion from Wang et al. (2019): our mean reward per episode of 200 steps = their return/200+1.

heading: 3 THE FRAMEWORK FOR RESOURCE-LIMITED ITERATIVE BATCH RL

heading: 3.1 THE FORMAL SETUP
text: Let TT = ( (s1,a1), . . . , (sT ,aT ) ) be a system trace consisting of T steps of observable-action pairs (st,at): given an observable st of the system state at time t, an action at was taken, leading to a new system state observed as st+1. The observable vector st = (s1t , . . . , s ds t ) contains ds numerical or categorical variables, measured on the system at time t. The action vector at contains da numerical or categorical action variables, typically set by a control function at = π(st) of the current observable st (or by a stochastic policy at ∼ π(st); we will also use the notation π : st ; at). The performance of the policy is measured by the reward rt which we assume to be a function of the observables st. Given a trace TT and a reward rt obtained at each step t, we define the mean reward as R(TT ) = 1T ∑T t=1 rt.
2 The system model p : (st, at) ; st+1 can be a deterministic point predictor or a probabilistic (generative) model that, besides the point prediction E { p ( st+1|(st, at) )} , also provides information on the uncertainty of the prediction and/or to model the randomness of the system (Deisenroth & Rasmussen, 2011; Chua et al., 2018).

heading: 3.2 A NOTE ON TERMINOLOGY
text: By model we will consistently refer to the learned transition or system model p (never to any policy). Rollout is the procedure of obtaining a trace T from an initial state s1 by alternating a model or real system p and a policy π (Fig 1). We decided to rename what Silver et al. (2017) calls the prior policy to guide since prior clashes with Bayesian terminology (as, e.g., Grill et al. (2020); Hamrick et al. (2021) also note), and guide expresses well that the role of this policy is to guide the search/planning. Sometimes the guide is also called the reactive policy (Sun et al., 2018) since it is typically an explicit function or conditional distribution ξ : s ; a that can be executed or drawn from rapidly. We will call the (often implicit) policy π : s ; a resulting from the guided plan/search the actor (sometimes also called the non-reactive policy since it takes time to simulate from the model, before each action). We will learn the guide ξ using MFRL mainly on the model p, but, when possible, off-policy (since π 6= ξ) data (system trace T ) obtained by acting on the real system will also be used. In that sense, through the data from the real system, planning is part of training the guide. However, since from the point of view of the MFRL guide, the model p is the world, we will not use the term planning to refer to the rollouts of the model p and the guide ξ when training the guide on p, rather we will reserve planning to the guided search procedure that results in acting on the real system.

heading: 3.3 EXPERIMENTAL SETUP: THE ITERATED BATCH MBRL
text: For rigorously studying and comparing algorithms and algorithmic ingredients, we need to fix not only the simulation environment but also the experimental setup. We parameterize the iterated batch RL loop (Fig 1) by four parameters:
• the number of episodes N , • the number of system access steps T per episode, • the planning horizon L, and • the number of generated rollouts n at each planning step.
N and T are usually set by hard organizational constraints (numberN and length T of live tests) that are part of the experimental setup. Our main goal is to measure the performance of our algorithms at a given (and challengingly small) number of system access stepsN ×T . Planning happens in silico, and so n and L (and so the total number of calls N × T × n × L to the simulator/system model p) are softer constraints determined by the (physical) time between two steps and the computational resources available for the planning.
In benchmark studies, such as this paper, we argue that fixing N , T , n, and L is important for making the problem well defined (taking some of the usual algorithmic choices out of the input of
2We use the mean reward (as opposed to the total reward, a.k.a return), since it is invariant to episode length and its unit is more meaningful to systems engineers.
the optimizer), affording meaningful comparison across papers and steady progress of algorithms. As in all benchmark designs, the goal is to make the problem challenging but not unsolvable. That said, we are aware that these choices may change the task and the research priorities implicitly but significantly (for example, a longer horizon L will be more challenging for the model but may make the planning easier), so if the MBRL community can agree, it would make sense to carefully design several settings (quadruples N − T − n− L) on the same environment. Our main operational cost is system access step so we are looking for any-time algorithms that achieve the best possible performance at any episode τ . Hence, in the MBRL iteration (Fig 1), we use the same traces T (τ), rolled out in each iteration (Line 7), to i) update the model p (Line 4) and the actor policy (Line 5) and ii) to measure the performance of the techniques (Section 3.5).

heading: 3.4 MODEL-BASED ACTOR POLICIES: GUIDE AND EXPLORE
text: Our main contribution is a Dyna-style GUIDE&EXPLORE strategy (Fig 2). The gist is to learn a guide policy ξ using a model-free RL technique on the model p and on the traces collected on the real system T . It is known that the guide ξ, executed as an actor π = ξ on the real system, does not work (we also confirm it in Section 4), partly because ξ overfits the model (Fig 5 in Kurutach et al. (2018); Grill et al. (2020)), partly because the goal of π is not only to exploit the traces T = ∪Nτ=1T (τ) collected so far and model p = LEARN(T ), but also to collect data to self-improve p and ξ/π in the next episode τ . This second reason is particular in our iterated batch setup: in pure batch RL, exploration is not an issue. We explore implicitly because of the randomness of planning (HETEROGENEOUSRS in Fig 3), but it turns out that it needs to be augmented by explicit exploration strategies. To show this, we experiment with two strategies, i) HEATING: modulating
the temperature of the guide distribution ξ(a|s), and ii) EPSGREEDY: choosing a random action with probability ε (Fig 4). The novelty of our approach is that, instead of constant T and ε, tuned as hyperparameters, we use a set of temperatures [Ti]ni=1 and probabilities [εi] n i=1 to further diversify the search and to let the planner to dynamically choose the right amount of randomization when selecting the best trace in Line 4 in Fig 3. Finally, similarly to Lowrey et al. (2019); Argenson & Dulac-Arnold (2020), we found that bootstrapping the planning with the learned value function at the end of each rollout trace (BOOTSTRAP in Fig 3) is crucial for optimizing the performance with a short horizon.
The main competitor of our GUIDE&EXPLORE actor is an actor that delegates all planning and exploration to the Monte-Carlo tree search of Silver et al. (2017)’s ALPHAZERO (Fig 4).

heading: 3.5 METRICS
text: We use two rigorously defined and measured metrics (Kégl et al., 2021) to assess the performance of the different algorithmic combinations. MAR measures the asymptotic performance after the learning has converged, and MRCP measures the convergence pace. Both can be averaged over seeds, and MAR is also an average over episodes, so we can detect statistically significant differences even when they are tiny, leading to a proper support for experimental development.
MEAN ASYMPTOTIC REWARD (MAR). Our measure of asymptotic performance, the mean asymptotic reward, is the mean reward MR(τ) = R ( T (τ)T ) in the second half of the episodes (after
convergence; we set N in such a way that the algorithms converge after less than N/2 episodes) MAR = 2N ∑N τ=N/2 MR(τ).
MEAN REWARD CONVERGENCE PACE (MRCP(r̄)). To assess the speed of convergence, we define the mean reward convergence pace MRCP(r̄) as the number of steps needed to achieve mean reward r̄, smoothed over a window of size 5: MRCP(r̄) = T×arg minτ ( 1 5 ∑τ+2 τ ′=τ−2 MR(τ) > r̄ ) .
The unit of MRCP(r̄) is system access steps, not episodes, first to make it invariant to episode length, and second because in micro-data RL the unit of cost is a system access step. For Acrobot, we use r̄ = 1.8 in our experiments, which is roughly 70% of the best achievable mean reward.
HEATINGEXPLORE
(
π(0), ξ, n
)
[s]:
1 for i← 1 to n: 2 ρi(a|s) = ξ(a|s)1/Ti∑ a′ ξ(a
′|s)1/Ti 3 return [ρi]ni=1
EPSGREEDYEXPLORE ( π(0), ξ, n ) [s]:
1 for i← 1 to n: 2 ρi(a|s) = { arg maxa ξ(a|s) with probability (1− εi), π(0)(a) with probability εi. 3 return [ρi]ni=1
Figure 4: Exploration strategies. HEATINGEXPLORE heats the guide action distribution ξ(a|s) to n different temperatures, and EPSGREEDYEXPLORE changes the best action to a random action π(0) ; a with different probabilities. The temperatures [Ti]ni=1 and probabilities [εi] n i=1 are hyperparameters.

heading: 4 EXPERIMENTS

heading: 4.1 THE ACROBOT BENCHMARK ENVIRONMENT
text: Acrobot is an underactuated double pendulum with four observables st = [θ1, θ2, θ̇1, θ̇2] which are usually augmented to six by taking sine and cosine of the angles (Brockman et al., 2016); θ1 is the angle to the vertical axis of the upper link; θ2 is the angle of the lower link relative to the upper link, both being clipped to [−π, π]; θ̇1 and θ̇2 are the corresponding angular momenta. For the starting position s1 of each episode, all four state variables are sampled uniformly from an approximately hanging and stationary position sj1 ∈ [−0.1, 0.1]. The action is a discrete torque on the lower link a ∈ {−1, 0, 1}. The reward is the height of the tip of the lower link over the hanging position r(s) = 2− cos θ1 − cos(θ1 + θ2) ∈ [0, 4].3
3We chose this rather than the sparse variable-episode-length version r(s) = I {2− cos θ1 − cos(θ1 + θ2) > 3} (Sutton, 1996) since it corresponds better to the continuous aspect of engineering systems.
Acrobot is a small but relatively difficult and fascinating system, so it is an ideal benchmark for continuous-reward engineering systems. Similarly to Kégl et al. (2021), we set the number of episodes to N = 100, the number of steps per episode to T = 200, the number of planning rollouts to n = 100, and the horizon to L = 10. With these settings, we can identify four distinctively different regimes (see the attached videos): i) the random uniform policy π(0) achieves MAR ≈ 0.1 − 0.2 (Acrobot keeps approximately hanging), ii) reasonable models with random shooting or pure Dyna-style controllers achieve MAR ≈ 1.4−1.6 (Acrobot gains energy but moves its limb quite uncontrollably), iii) random shooting n = 100, L = 10 with good models such as PETS (Chua et al., 2018; Wang et al., 2019) or DARMDN (Kégl et al., 2021) keep the limb up and manage to have its tip above horizon on average MAR ≈ 2.0−2.1 (previous state of the art), and iv) in our experiments we could achieve a quasi perfect policy (Acrobot moves up like a gymnast and stays balanced at the top) MAR ≈ 2.7− 2.8 using random shooting with n = 100 K, L = 20 on the real system, giving us a target and a possibly large margin of improvement. Acrobot is also an ideal benchmark for making our point since it turned out quite challenging for Dyna-style techniques in Wang et al. (2019)’s benchmarks (MAR ≈ 1.6− 1.7).

heading: 4.2 MODELS, GUIDES, AND ACTORS
text: Following Kégl et al. (2021), we tried different system models (Fig 1/Line 4) from the family of Deep Autoregressive Mixture Density Networks (DARMDN) and decided to use DARMDN(1)det, an autoregressive network trained probabilistically (the output of the net for each observable dimension is a single Gaussian learned by minimizing the negative log-likelihood loss) and sampled deterministically (the mean of the Gaussian). In Kégl et al. (2021) this was the best model using random shooting; Table 4 in the Appendix confirms this using our best actor.
In principle, any MFRL technique working with discrete action space and providing a value function and a policy can be used as a guide ξ when applying ITERATEDMBRL to the Acrobot system (Fig 2/Line 1). We experimented with Deep Q Networks (DQN; Mnih et al. (2015)) and PPO (Schulman et al., 2017). For the DQN, following Janner et al. (2019), short rollouts starting from real observations are performed on the model to sample transitions which are then placed in an experience replay buffer, along with the real transitions observed during the rollouts (Fig 1/Line 7). The DQN is then updated by sampling batches from this buffer. For DQN alone (without planning), an ε-greedy strategy is used for its training (ε decaying as more training samples are used) and at decision time (ε = 0.05).
For actors (Fig 1/Line 5, Fig 2), we start from the simple DQN(ε = 0.05) or PPO guide which is interacting with the system without planning. We then add different ingredients to these simple agents to improve the performance. Adding (n, L) to the name of the agent means that we use the agent to guide a planning of n rollouts with horizon L. It is important to note here that planning without exploration using the greedy guide is, in our case, equivalent to no planning since both the model p and the guides ξ are deterministic. XXX-HEATINGEXPLORE and XXX-EPSGREEDYEXPLORE refer to the additional use of the associated exploration strategies (Fig 1/Line 2, Fig 4). When a fixed ε is used for the exploration strategy, we add it as an explicit parameter, e.g., EPSGREEDYEXPLORE(ε). No parameters means that a different ε or temperature is used for each of the n rollouts. By default the value estimate is the TOTALREWARD (Fig 3). We add BOOTSTRAP when we use bootstrapping by the value function estimate. Finally, we also apply AlphaZero (Silver et al., 2017) in a Dyna-style fashion (Fig 2) which we refer to as DYNAZERO.
Appendix A contains detailed information on the various algorithmic choices.

heading: 4.3 RESULTS
text: Table 2 and Figure 5 present our the results with the main actors. We see that planning with exploration and bootstrapping gives the best results among all the DQN actors. The approach beats the previous state of the art: both RSACTOR(n = 100, L = 10) from Kégl et al. (2021) and POPLIN-A (Wang & Ba (2020); Table 1) by 10% in terms of MAR. PPO is slightly suboptimal compared to DQN. DYNAZERO matches the best MAR but converges three times slower and takes about five times more computational time, mainly because, unlike random shooting, MCTS cannot easily be parallelized.
Table 2 and the detailed ablation study in Appendix B show that all ingredients add to the performance. Although DQN alone (DQN(ε = 0.05)) is far better than a random policy, it is clearly not
sufficient to obtain a reasonable actor. Using it as a planning guide (DQN(n = 100, L = 10)- EPSGREEDYEXPLORE(ε = 0.05)) significantly improves the performance but remains worse than random shooting RSACTOR(n = 100, L = 10). To propel the Dyna-style actor above simple MPC, we need to fine tune exploration. We found that allowing the planner to choose the right amount is a robust and safe approach. The final improvement was attained by adding value bootstrapping to the heated explorer/planner; bootstrapping does not improve the ε-greedy actor, possibly because the value estimate of the guide ξ is off on random actions, unlike in the heating exploration when the explorer “softly” increases the probability of actions, given by the guide. As a reference, we also include a simple but costly RSACTOR(n = 100 K, L = 10). It would not be officially accepted in our benchmark as we restrict n to 100 and L to 10, but it can serve as a target to reach.

heading: 5 CONCLUSION
text: In this paper we show that an offline Dyna-style approach can be successfully applied on the Acrobot system where previously Dyna-style algorithms were failing. Our empirical results exhibit the importance of using a planning guided by a policy with the correct amount of exploration. We thus propose a strategy to automatically fine-tune exploration and add bootstrapping with a value function estimate to further improves the performance. This combination leads to an improvement over the previous state of the art by 10% while respecting the same resource constraints. Future work includes modelling the uncertainties of the value estimates so as to use them for better exploration.

heading: 6 REPRODUCIBILITY STATEMENT
text: All the code to produce our results will be made publicly available after publication. Details for the implementations of ITERATEDMBRL, the different exploration strategies, the system models, and the different MFRL agents are also provided in Section A.

heading: B ABLATION STUDY
text: B.1 ACTORS
We show in this section that a guided planning and exploration are needed to obtain the best result. Remove either planning or exploration and you will obtain suboptimal performance. To support this claim we ran an ablation study with EPSGREEDYEXPLORE. EPSGREEDYEXPLORE makes it easy to control the degree of exploration through the ε parameter. Setting ε to 0 corresponds to no exploration and is equivalent to using the guide greedily without planning (n = 1 and L = 1) as our model is used deterministically when sampled from. Setting ε to 1 corresponds to full exploration and is equivalent to the purely random RSACTOR(n = 100, L = 10). Figure 6 shows the results obtained by DQN(ε = 0.05), DQN with a forced ε-greedy scheme at decision time with ε = 0.4 and no planning (DQN(ε = 0.4)), and DQN with planning and fixed ε values (DQN(n = 100, L = 10)-EPSGREEDYEXPLORE(ε) for ε ∈ {0.0001, 0.01, 0.05, 0.1, 0.2, 0.4, 0.8, 0.99, 0.9999}). All the results are also put in Table 3.
First, we see that planning definitely helps when comparing the performance of DQN(ε = 0.4) which uses no planning but forces the exploration with ε = 0.4 and the performance of DQN(n = 100, L = 10)-EPSGREEDYEXPLORE(ε = 0.4) which uses planning and the same fixed value of ε. Second, as expected, the closer ε is to 0 the closer the performance is to DQN(ε = 0.05), and the closer ε is to 1 the closer the performance is to RSACTOR(n = 100, L = 10). With a well-chosen ε between these two extremes, say ε = 0.4, we obtain a better performance than either extremes.
We can thus claim that planning is required and planning with a correct amount of exploration. Our EPSGREEDYEXPLORE or HEATEDEXPLORE exploration strategies, used with multiple ε or temperatures values allows for the automatic and dynamic selection of the good amount of exploration.
B.2 MODELS
In Table 4 we compare the performances of models from Kégl et al. (2021) when applied with our best agent DQN(n = 100, L = 10)-HEATINGEXPLORE-BOOTSTRAP. Most models belong to the family of Deep (Autoregressive) Mixture Density Networks with D components, D{AR}MDN(D){det}, trained by minimizing the negative log-likelihood. The ‘det’ suffix means that the model is sampled from deterministically, returning the mean of the predicted distribution. DARNNs are Deep Autoregressive Neural Networks point-estimating a mean by minimizing the mean squared error and estimating a constant variance using the residual errors. These models are homoscedastic with a variance independent of the input. The reader can also refer to Kégl et al. (2021) for a complete description of these models. In the paper we use the second best model DARMDN(1)det over the non-autoregressive DMDN(1)det because we find autoregressive models to be easier to use in practice on real-life projects due to their ability to model dependence between the observable dimensions and to model different feature types. We also note that, as observed in Kégl et al. (2021) with RSACTOR(n = 100, L = 10), the deterministic versions perform slightly better than the stochastic ones, in general.

heading: C RSACTOR PERFORMANCE ON THE REAL SYSTEM
text: We present the results one can obtain on the real system with an RSACTOR and different values of the planning horizon L and the number of generated rollouts n in Fig 7. For the considered planning horizons a larger number of generated rollouts lead to a better performance. We also observed in our simulations that for the Acrobot to stay balanced, it was necessary (although not always sufficient) to have a reward larger than 2.6. We see from Fig 7 that this can be achieved with a simple agent such as RSACTOR but at the price of a very large number of generated rollouts. The goal is therefore to design a smarter agent that can come as close as possible to this performance with a limited budget.

Review summaries:
The authors study model-based reinforcement learning applied in the scenario of iterated batch reinforcement learning. There exist other namings of the studied in the paper scenario of iterated batch reinforcement learning, I found out that there exist several other synonymous names of such setting: growing batch RL/ micro-data RL / micro-data model-based RL. The authors motivate the problem informally using several bullet points listed in the introduction.  The main finding claimed by the authors is that the Dyna-style MBRL approach is an appropriate choice for the studied setting by enhancing it with decision-time planning based on a model-free policy and dynamically tuning exploration.   The algorithms are tested in the problem of resource-limited Acrobot; a detailed ablation study is provided for algorithm features (called here interchangeable algorithmic bricks for iterative batch RL. The performed ablation study helped to find the combination of a neural model, a bootstrapping DQN guide, and a heating explorer, which led to a 10% jump in state-of-the-art on the resource-limited Acrobot system.