Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization

1 INTRODUCTION . Reinforcement learning ( RL ) algorithms have recently demonstrated impressive success in learning behaviors for a variety of sequential decision-making tasks ( Barth-Maron et al. , 2018 ; Hessel et al. , 2018 ; Nachum et al. , 2019 ) . Virtually all of these demonstrations have relied on highly-frequent online access to the environment , with the RL algorithms often interleaving each update to the policy with additional experience collection of that policy acting in the environment . However , in many real-world applications of RL , such as health ( Murphy et al. , 2001 ) , education ( Mandel et al. , 2014 ) , dialog agents ( Jaques et al. , 2019 ) , and robotics ( Gu et al. , 2017a ; Kalashnikov et al. , 2018 ) , the deployment of a new data-collection policy may be associated with a number of costs and risks . If we can learn tasks with a small number of data collection policies , we can substantially reduce them . Based on this idea , we propose a novel measure of RL algorithm performance , namely deployment efficiency , which counts the number of changes in the data-collection policy during learning , as illustrated in Figure 1 . This concept may be seen in contrast to sample efficiency or data efficiency ( Precup et al. , 2001 ; Degris et al. , 2012 ; Gu et al. , 2017b ; Haarnoja et al. , 2018 ; Lillicrap et al. , 2016 ; Nachum et al. , 2018 ) , which measures the amount of environment interactions incurred during training , without regard to how many distinct policies were deployed to perform those interactions . Even when the ∗Equal contribution . 1Codes and pre-trained models are available at https : //github.com/matsuolab/BREMEN . data efficiency is high , the deployment efficiency could be low , since many on-policy and off-policy algorithms alternate data collection with each policy update ( Schulman et al. , 2015 ; Lillicrap et al. , 2016 ; Gu et al. , 2016 ; Haarnoja et al. , 2018 ) . Such dependence on high-frequency policy deployments is best illustrated in the recent works in offline RL ( Fujimoto et al. , 2019 ; Jaques et al. , 2019 ; Kumar et al. , 2019 ; Levine et al. , 2020 ; Wu et al. , 2019 ) , where baseline off-policy algorithms exhibited poor performance when trained on a static dataset . These offline RL works , however , limit their study to a single deployment , which is enough for achieving high performance with data collected from a sub-optimal behavior policy , but often not from a random policy . In contrast to those prior works , we aim to learn successful policies from scratch in a manner that is both sample and deployment-efficient . Many existing model-free offline RL algorithms ( Levine et al. , 2020 ) are tuned and evaluated on massive datasets ( e.g. , one million transitions ) . In order to develop an algorithm that is both sample and deployment-efficient , each iteration of the algorithm between successive deployments has to work effectively on much smaller dataset sizes . We believe model-based RL is better suited to this setting due to its higher demonstrated sample efficiency than model-free RL ( Kurutach et al. , 2018 ; Nagabandi et al. , 2018 ) . Although the combination of model-based RL and offline or limiteddeployment settings seems straight-forward , we find this naïve approach leads to poor performance . This problem can be attributed to extrapolation errors ( Fujimoto et al. , 2019 ) similar to those observed in model-free methods . Specifically , the learned policy may choose sequences of actions which lead it to regions of the state space where the dynamics model can not predict properly , due to poor coverage of the dataset . This can lead the policy to exploit approximation errors of the dynamics model and be disastrous for learning . In model-free settings , similar data distribution shift problems are typically remedied by regularizing policy updates explicitly with a divergence from the observed data distribution ( Jaques et al. , 2019 ; Kumar et al. , 2019 ; Wu et al. , 2019 ) , which , however , can overly limit policies ’ expressivity ( Sohn et al. , 2020 ) . In order to better approach these problems arising in limited deployment settings , we propose Behavior-Regularized Model-ENsemble ( BREMEN ) , which learns an ensemble of dynamics models in conjunction with a policy using imaginary rollouts while implicitly regularizing the learned policy via appropriate parameter initialization and conservative trust-region learning updates . We evaluate BREMEN on standard offline RL benchmarks of high-dimensional continuous control tasks , where only a single static dataset is used . In this fixed-batch setting , our experiments show that BREMEN can not only achieve performance competitive with state-of-the-art when using standard dataset sizes but also learn with 10-20 times smaller datasets , which previous methods are unable to attain . Enabled by such stable and sample-efficient offline learning , we show that BREMEN can learn successful policies with only 5-10 deployments in the online setting , significantly outperforming existing off-policy and offline RL algorithms in deployment efficiency while keeping sample efficiency . 2 PRELIMINARIES . We consider a Markov Decision Process ( MDP ) setting , characterized by the tuple M = ( S , A , p , r , γ ) , where S is the state space , A is the action space , p ( s′|s , a ) is the transition probability distribution or dynamics , r ( s ) is the reward function and γ ∈ ( 0 , 1 ) is the discount factor . A policy π is a function that determines the agent behavior , mapping from states to probability distributions over actions . The goal is to obtain the optimal policy π∗ , which maximizes the expectation of discounted sum of rewards . The transition probability p ( s′|s , a ) is usually unknown , and estimated with a parameterized dynamics model fφ ( e.g . a neural network ) in model-based RL . For simplicity , we assume that the reward function r ( s ) is known , and the reward can be computed for any arbitrary state , but we may extend to the unknown setting and predict it using a parameterized function . On-policy vs Off-policy , Online vs Offline At a high-level , most RL algorithms alternate many times between collecting a batch of transitions ( deployments ) and optimizing the policy ( learning ) . If the algorithms discard data after each policy update , they are on-policy ( Schulman et al. , 2015 ; 2017 ) , while if they accumulate data in a buffer D , i.e . experience replay ( Lin , 1992 ) , they are off-policy ( Mnih et al. , 2015 ; Lillicrap et al. , 2016 ; Gu et al. , 2016 ; 2017b ; Haarnoja et al. , 2018 ; Fujimoto et al. , 2019 ; Fakoor et al. , 2019 ) because not all the data in buffer comes from the current policy . However , we consider all these algorithms to be online RL algorithms , since they involve many deployments during learning , ranging from hundreds to millions . On the other hand , in pure offline RL , one does not assume direct interaction and learns a policy from only a fixed dataset , which effectively corresponds to a single deployment allowed for learning . Classically , interpolating these two extremes were semi-batch RL algorithms ( Lange et al. , 2012 ; Singh et al. , 1995 ) , which improve the policy through repetitions of collecting a large batch of transitions D = { ( s , a , s′ , r ) } and performing many or full policy updates . While these semi-batch RL also realize good deployment efficiency , they have not been extensively studied with neural network function approximators or in off-policy settings with experience replay for scalable sample-efficient learning . In our work , we aim to have both high deployment efficiency and sample efficiency by developing an algorithm that can solve the tasks with minimal policy deployments as well as transition samples . 3 DEPLOYMENT EFFICIENCY . Deploying a new policy for data collection can be associated with a number of costs and risks for many real-world applications like medicine , dialogue systems , or robotic control ( Murphy et al. , 2001 ; Mandel et al. , 2014 ; Gu et al. , 2017a ; Kalashnikov et al. , 2018 ; Nachum et al. , 2019 ; Jaques et al. , 2019 ) . While there are abundant works on safety for RL ( Chow et al. , 2015 ; Eysenbach et al. , 2018 ; Chow et al. , 2018 ; Ray et al. , 2019 ; Chow et al. , 2019 ) , they often do not provide guarantees in practice when combined with neural networks and stochastic optimization . It is therefore necessary to verify each policy before deployment ( e.g . measuring the variance of rewards or checking out-of-bounds actions ) . Due to such costs associated with each deployment , it is desirable to minimize the number of distinct deployments needed during the learning process . Even ignoring safety considerations , frequent updates to a deployed policy can exacerbate communication bottlenecks in large-scale distributed RL systems , which are becoming more prevalent ( Nair et al. , 2015 ; Espeholt et al. , 2018 ; 2019 ) . We additionally discuss on the importance of the deployment efficiency in real-world applications . See Appendix C. In order to focus research on these practical bottlenecks , we propose a novel measure of RL algorithms , namely , deployment efficiency , which counts how many times the data-collection policy has been changed during improvement from random policy to solve the task . For example , if an RL algorithm operates by using its learned policy to collect transitions from the environment I times , each time collecting a batch of B new transitions , then the number of deployments is I , while the total number of samples collected is I × B . The lower I is , the more deployment-efficient the algorithm is ; in contrast , sample efficiency looks at I × B. Online RL algorithms , whether they are on-policy or off-policy , typically update the policy and acquire new transitions by deploying the newly updated policy at every iteration . This corresponds to performing hundreds to millions of deployments during learning on standard benchmarks ( Haarnoja et al. , 2018 ) , which is severely deployment inefficient . On the other hand , offline RL literature only studies the case of 1 deployment . A deployment-efficient algorithm would stand in the middle of these two extremes and ideally learn a successful policy from scratch while deploying only a few distinct policies , as illustrated in Figure 1 . Recent deep RL literature seldom emphasizes deployment efficiency , with few exceptions in specific applications ( Kalashnikov et al. , 2018 ) where such a learning procedure is necessary . Deploymentinefficient algorithms will fail in scenarios where the deployment of each new policy is exorbitantly expensive , such as safety-critical robotics or user-facing products . Although current state-of-the-art algorithms on continuous control have substantially improved sample or data efficiency , they have not optimized for deployment efficiency . For example , SAC ( Haarnoja et al. , 2018 ) , an efficient model-free off-policy algorithm , performs half a million to one million policy deployments during learning on MuJoCo ( Todorov et al. , 2012 ) benchmarks . ME-TRPO ( Kurutach et al. , 2018 ) , a model-based algorithm , performs a much lower 100-300 policy deployments , although this is still relatively high for practical settings.2 In our work , we demonstrate successful learning on standard benchmark environments with only 5-10 deployments .

Summary: This paper proposes a new approach to learning control policies with improved data efficiency and fewer number of data collection sessions (with each session using a different policy). Further, the authors proposed a new concept of “deployment efficiency”, with a new “deployment” referring to using a new policy to interact with the real environment, for example, for data collection. The new approach belongs to the family of model-based online reinforcement learning algorithms and seems to primarily augment a prior approach, called ME-TRPO, by using a helper policy trained by behavior cloning data collected after the most recent deployment of learner policy. The experiment results validate that the proposed approach achieves better data efficiency and deployment efficiency compared to prior approaches.