The guide and the explorer: smart agents for resource-limited iterated batch reinforcement learning

1 INTRODUCTION . John is a telecommunication engineer . His day job is to operate a mobile antenna . He has about forty knobs to turn , in principle every five minutes , based on about a hundred external and internal system observables . His goal is to keep some performance indicators within operational limits while optimizing some others . In the evenings John dreams about using reinforcement learning ( RL ) to help him with his job . He knows that he can not put an untrusted model-free agent on the antenna control ( failures are very costly ) , but he manages to convince his boss to run live tests a couple of days every month . John ’ s case is arguably on the R & D table of a lot of engineering companies today . AI adoption is slow , partly because these companies have little experience with AI , but partly also because the algorithms we develop fail to address the constraints and operational requirements of these systems . What are the common attributes of these systems ? • They are physical , not getting faster with time , producing tiny data compared to what model-free RL ( MFRL ) algorithms require for training . • System access is limited to a small number of relatively short live tests , each producing logs that can be used to evaluate the current policy and can be fed into the training of the next . • They are relatively small-dimensional , and system observables were designed to support human control decisions , so there is no need to filter them or to learn representations ( with the exception when the engineer uses complex images , e.g. , a driver ) . • Rewards are non-sparse , performance indicators come continually . Delays are possible but usually not long . • Safe operation even during live tests is crucial . Simulators and digital twins are more and more available , but some systems are too complex to simulate from first principles . The RL setup that fits this scenario is neither pure batch ( offline ; Levine et al . ( 2020 ) ) as interacting with the system is possible during short periods of time nor pure online as the policy can only be updated offline between two interaction periods . We refer to it as iterated batch RL ( Lange et al . ( 2012 ) call it growing batch ) . Furthermore , we are interested in model-based RL ( Deisenroth & Rasmussen , 2011 ; Chua et al. , 2018 ; Moerland et al. , 2021 ) because i ) it is sample efficient ( Chua et al. , 2018 ; Wang et al. , 2019 ) , ii ) it works well on small-dimensional systems with dense rewards , and iii ) the system model ( data-driven simulator ) itself is an object of interest because it can ease the adoption of data-driven algorithms by systems engineers . Given a robust system model , simple model predictive control ( MPC ) agents using random shooting ( RS ; Richards ( 2005 ) ; Rao ( 2010 ) ) or the cross entropy method ( CEM ; de Boer et al . ( 2004 ) ) have been shown to perform remarkably well on many benchmark systems ( Nagabandi et al. , 2018 ; Chua et al. , 2018 ; Wang et al. , 2019 ; Hafner et al. , 2019 ; Kégl et al. , 2021 ) and real-life domains such as robotics ( Yang et al. , 2020 ) . On the other hand , implementing successfully the seemingly elegant Dyna-style approach ( Sutton , 1991 ; Kurutach et al. , 2018 ; Clavera et al. , 2018 ; Luo et al. , 2019 ) , when we learn model-free agents on the system model and apply them on the real system , remains challenging especially on systems that require planning with long horizon . Our main finding is that the Dyna-style approach can be an excellent choice for iterated batch RL by adding to it a decisiontime planning guided by the model-free policy and a dynamically self-tuning exploration . The key is that exploration and bootstrapping with a value function estimate affords us shorter rollouts so we can reduce the accumulation of errors that plagues the Dyna-style approach with long horizons ( Janner et al. , 2019 ) . We also innovate on the experimental framework ( metrics , statistically rigorous measurements ) , so we can profit from the modularity of the Dyna-style approach , tuning ingredients ( model , the MFRL guide policy , exploration , planning , bootstrapping ) independently . This modular approach makes engineering easier ( as opposed to monolithic approaches like AlphaZero ( Silver et al. , 2017 ) ) , which is an important aspect if we want to give the methodology to non-expert systems engineers . We demonstrate our GUIDE & EXPLORE approach on the small-dimensional but difficult Acrobot system which is known to be especially tough for Dyna-style techniques ( Wang et al. , 2019 ) . We show in a rigorous ablation study how each ingredient adds a small but significant improvement , achieving a 10 % total margin over the current state of the art ( Wang et al. , 2019 ; Wang & Ba , 2020 ; Kégl et al. , 2021 ) . DYNAZERO ( pure Dyna-style with an AlphaZero guide ) also matches the asymptotic performance with a larger sample complexity ( 3x ) and computational ( 5x ) price tag . 1.1 SUMMARY OF CONTRIBUTIONS . 2 RELATED WORK . The MBRL subfield has seen a proliferation of powerful methods , but most of them miss the specific requirements ( solving problems irrelevant in this scenario like representation learning or sparse rewards ) and missing others ( limited and costly system access ; data taking and experimentation through campaigns , live tests ; safety ) ( Hamrick , 2019 ) . The Dyna framework developed by Sutton ( 1991 ) consists in training an agent from both real experience and from simulations from a system model learned from the real data . Its efficient use of system access makes it a natural candidate for iterated batch RL . The well-known limitation of this approach is the agent overfitting the imperfect system model ( Grill et al. , 2020 ) . A first solution is to use short rollouts on the model to reduce error accumulation as done in Model-Based Policy Optimization ( MBPO ; Janner et al . ( 2019 ) ) . Another solution is to rely on ensembling techniques for the model . Kurutach et al . ( 2018 ) ’ s ME-TRPO is based on an ensemble of models and Trust Region Policy Optimization ( TRPO ; Schulman et al . ( 2015 ) ) . At each TRPO step a transition is sampled from a randomly picked model of the ensemble , preventing the policy from overfitting one model . An ensemble is also used in Model-Based Meta-Policy Optimization ( MP-MPO ; Clavera et al . ( 2018 ) ) where each model of the ensemble is seen as a different task used to meta-learn a policy . The meta-policy is then able to quickly adapt to any of the dynamics model and is more robust to model inaccuracies . Instead of learning the model and then the policy from the model , Stochastic Lower Bound Optimization ( SLBO ; Luo et al . ( 2019 ) ) alternates between model and policy updates . In our ITERATEDMBRL skeleton ( Fig 1 ) , this strategy would couple the LEARN and MFRL steps , which we do not study in this paper . We note that according to the results shown in Fig 1 . ( a ) and Table 1 in Wang et al . ( 2019 ) , ME-TRPO , SLBO and MB-MPO alone ( pure Dyna , without planning ) are clearly suboptimal on the Acrobot system1 , similar to the performance obtained by our pure Dyna DQN ( Section 4 ) . Finally , Yu et al . ( 2020 ) and Kidambi et al . ( 2020 ) use a Dyna-style approach in the context of pure batch RL where no further data collection and therefore no further model updates are assumed . The idea of using a guide and a value function when planning is not novel ( Silver et al. , 2017 ; Schrittwieser et al. , 2020 ; Wang & Ba , 2020 ; Argenson & Dulac-Arnold , 2020 ) . We were greatly inspired by these elements in our objective of building smarter agents as they can make the search more efficient and thus lead to a better performance . POPLIN-A ( Wang & Ba , 2020 ) relies on behavior cloning ( using only real experience , unlike our Dyna-style approach that mainly uses the model ) , but their guide is similar to our approach . During the planning , they add random noise to the actions recommended by a deterministic policy network and update the noise distribution using a CEM strategy . In a similar way our GUIDE & EXPLORE strategy also adds a carefully controled amount of noise to the recommended actions . Our results highlight the importance of a well-calibrated exploration , which also contributes to the understanding of POPLIN-A . Argenson & Dulac-Arnold ( 2020 ) and Lowrey et al . ( 2019 ) both found that bootstrapping with a value estimate improves the performance of simple guided MPC strategies . The popular AlphaZero ( Silver et al. , 2017 ) and MuZero ( Schrittwieser et al. , 2020 ) algorithms also rely on a guide and a value function for their Monte Carlo Tree Search ( MCTS ) . We implement a Dyna-style version of AlphaZero , which we call DYNAZERO . The principal issue of MuZero ( Schrittwieser et al. , 2020 ) in our micro-data iterated batch RL context is that it does not control the number of system access steps : it needs to simulate a lot from the real environment to establish the targets for the value function . In these two algorithms the guide is updated from the results obtained during the search that it guided , a procedure known as Dual Policy Iteration ( Sun et al. , 2018 ) . We prefer experiencing with Dyna-style approaches first to leverage popular MFRL algorithms and defer the study of Dual Policy Iteration to future work . Our results show that planning is an important ingredient , a claim already made by Hamrick et al . ( 2021 ) . They use MuZero to run their ablation study while we prefer using an explicit model for practical reasons explained in the introduction . Besides planning we also study the importance of exploration . 1The conversion from Wang et al . ( 2019 ) : our mean reward per episode of 200 steps = their return/200+1 . 3 THE FRAMEWORK FOR RESOURCE-LIMITED ITERATIVE BATCH RL . 3.1 THE FORMAL SETUP . Let TT = ( ( s1 , a1 ) , . . . , ( sT , aT ) ) be a system trace consisting of T steps of observable-action pairs ( st , at ) : given an observable st of the system state at time t , an action at was taken , leading to a new system state observed as st+1 . The observable vector st = ( s1t , . . . , s ds t ) contains ds numerical or categorical variables , measured on the system at time t. The action vector at contains da numerical or categorical action variables , typically set by a control function at = π ( st ) of the current observable st ( or by a stochastic policy at ∼ π ( st ) ; we will also use the notation π : st ; at ) . The performance of the policy is measured by the reward rt which we assume to be a function of the observables st . Given a trace TT and a reward rt obtained at each step t , we define the mean reward as R ( TT ) = 1T ∑T t=1 rt . 2 The system model p : ( st , at ) ; st+1 can be a deterministic point predictor or a probabilistic ( generative ) model that , besides the point prediction E { p ( st+1| ( st , at ) ) } , also provides information on the uncertainty of the prediction and/or to model the randomness of the system ( Deisenroth & Rasmussen , 2011 ; Chua et al. , 2018 ) .

Summary: The authors study model-based reinforcement learning applied in the scenario of iterated batch reinforcement learning. There exist other namings of the studied in the paper scenario of iterated batch reinforcement learning, I found out that there exist several other synonymous names of such setting: growing batch RL/ micro-data RL / micro-data model-based RL. The authors motivate the problem informally using several bullet points listed in the introduction.  The main finding claimed by the authors is that the Dyna-style MBRL approach is an appropriate choice for the studied setting by enhancing it with decision-time planning based on a model-free policy and dynamically tuning exploration.   The algorithms are tested in the problem of resource-limited Acrobot; a detailed ablation study is provided for algorithm features (called here interchangeable algorithmic bricks for iterative batch RL. The performed ablation study helped to find the combination of a neural model, a bootstrapping DQN guide, and a heating explorer, which led to a 10% jump in state-of-the-art on the resource-limited Acrobot system.