Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimality

Deployment efficiency is an important criterion for many real-world applications of reinforcement learning ( RL ) . Despite the community ’ s increasing interest , there lacks a formal theoretical formulation for the problem . In this paper , we propose such a formulation for deployment-efficient RL ( DE-RL ) from an “ optimization with constraints ” perspective : we are interested in exploring an MDP and obtaining a near-optimal policy within minimal deployment complexity , whereas in each deployment the policy can sample a large batch of data . Using finite-horizon linear MDPs as a concrete structural model , we reveal the fundamental limit in achieving deployment efficiency by establishing information-theoretic lower bounds , and provide algorithms that achieve the optimal deployment efficiency . Moreover , our formulation for DE-RL is flexible and can serve as a building block for other practically relevant settings ; we give “ Safe DE-RL ” and “ Sample-Efficient DE-RL ” as two examples , which may be worth future investigation . 1 INTRODUCTION . In many real-world applications , deploying a new policy to replace the previous one is costly , while generating a large batch of samples with an already deployed policy can be relatively fast and cheap . For example , in recommendation systems ( Afsar et al. , 2021 ) , education software ( Bennane et al. , 2013 ) , and healthcare ( Yu et al. , 2019 ) , the new recommendation , teaching , or medical treatment strategy must pass several internal tests to ensure safety and practicality before being deployed , which can be time-consuming . On the other hand , the algorithm may be able to collect a large amount of samples in a short period of time if the system serves a large population of users . Besides , in robotics applications ( Kober et al. , 2013 ) , deploying a new policy usually involves operations on the hardware level , which requires non-negligible physical labor and long waiting periods , while sampling trajectories is relatively less laborious . However , deployment efficiency was neglected in most of existing RL literatures . Even for those few works considering this important criterion ( Bai et al. , 2020 ; Gao et al. , 2021 ; Matsushima et al. , 2021 ) , either their settings or methods have limitations in the scenarios described above , or a formal mathematical formulation is missing . We defer a detailed discussion of these related works to Section 1.1 . In order to close the gap between existing RL settings and real-world applications requiring high deployment efficiency , our first contribution is to provide a formal definition and tractable objective for Deployment-Efficient Reinforcement Learning ( DE-RL ) via an “ optimization with constraints ” perspective . Roughly speaking , we are interested in minimizing the number of deploymentsK under two constraints : ( a ) after deploying K times , the algorithm can return a near-optimal policy , and ( b ) the number of trajectories collected in each deployment , denoted as N , is at the same level across K deployments , and it can be large but should still be polynomial in standard parameters . Similar to the notion of sample complexity in online RL , we will refer to K as deployment complexity . ˚Work done during the internship at Microsoft Research Asia . To provide a more quantitative understanding , we instantiate our DE-RL framework in finite-horizon linear MDPs1 ( Jin et al. , 2019 ) and develop the essential theory . The main questions we address are : Q1 : What is the optimum of the deployment efficiency in our DE-RL setting ? . Q2 : Can we achieve the optimal deployment efficiency in our DE-RL setting ? When answering these questions , we separately study algorithms with or without being constrained to deploy deterministic policies each time . While deploying more general forms of policies can be practical ( e.g. , randomized experiments on a population of users can be viewed as deploying a mixture of deterministic policies ) , most previous theoretical works in related settings exclusively focused on upper and lower bounds for algorithms using deterministic policies ( Jin et al. , 2019 ; Wang et al. , 2020b ; Gao et al. , 2021 ) . As we will show , the origin of the difficulty in optimizing deployment efficiency and the principle in algorithm design to achieve optimal deployment efficiency are generally different in these two settings , and therefore , we believe both of them are of independent interests . As our second contribution , in Section 3 , we answer Q1 by providing information-theoretic lower bounds for the required number of deployments under the constraints of ( a ) and ( b ) in Def 2.1 . We establish ΩpdHq and rΩpHq lower bounds for algorithms with and without the constraints of deploying deterministic policies , respectively . Contrary to the impression given by previous empirical works ( Matsushima et al. , 2021 ) , even if we can deploy unrestricted policies , the minimal number of deployments can not be reduced to a constant without additional assumptions , which sheds light on the fundamental limitation in achieving deployment efficiency . Besides , in the line of work on “ horizon-free RL ” ( e.g. , Wang et al. , 2020a ) , it is shown that RL problem is not significantly harder than bandits ( i.e. , when H “ 1 ) when we consider sample complexity . In contrast , the H dependence in our lower bound reveals some fundamental hardness that is specific to long-horizon RL , particularly in the deployment-efficient setting . 2 Such hardness results were originally conjectured by Jiang & Agarwal ( 2018 ) , but no hardness has been shown in sample-complexity settings . After identifying the limitation of deployment efficiency , as our third contribution , we address Q2 by proposing novel algorithms whose deployment efficiency match the lower bounds . In Section 4.1 , we propose an algorithm deploying deterministic policies , which is based on Least-Square Value Iteration with reward bonus ( Jin et al. , 2019 ) and a layer-by-layer exploration strategy , and can return an ε-optimal policy within OpdHq deployments . As part of its analysis , we prove Lemma 4.2 as a technical contribution , which can be regarded as a finite-sample version of the well-known “ Elliptical Potential Lemma ” ( Carpentier et al. , 2020 ) and may be of independent interest . Moreover , our analysis based on Lemma 4.2 can be applied to the reward-free setting ( Jin et al. , 2020 ; Wang et al. , 2020b ) and achieve the same optimal deployment efficiency . In Section 4.2 , we focus on algorithms which can deploy arbitrary policies . They are much more challenging because it requires us to find a provably exploratory stochastic policy without interacting with the environment . To our knowledge , Agarwal et al . ( 2020b ) is the only work tackling a similar problem , but their algorithm is model-based which relies on a strong assumption about the realizability of the true dynamics and a sampling oracle that allows the agent to sample data from the model , and how to solve the problem in linear MDPs without a model class is still an open problem . To overcome this challenge , we propose a model-free layer-by-layer exploration algorithm based on a novel covariance matrix estimation technique , and prove that it requires ΘpHq deployments to return an ε-optimal policy , which only differs from the lower bound rΩpHq by a logarithmic factor . Although the per-deployment sample complexity of our algorithm has dependence on a “ reachability coefficient ” ( see Def . 4.3 ) , similar quantities also appear in related works ( Zanette et al. , 2020 ; Agarwal et al. , 2020b ; Modi et al. , 2021 ) and we conjecture that it is unavoidable and leave the investigation to future work . Finally , thanks to the flexibility of our “ optimization with constraints ” perspective , our DE-RL setting can serve as a building block for more advanced and practically relevant settings where optimizing the number of deployments is an important consideration . In Appendix F , we propose two potentially interesting settings : “ Safe DE-RL ” and “ Sample-Efficient DE-RL ” , by introducing constraints regarding safety and sample efficiency , respectively . 1Although we focus on linear MDPs , the core idea can be extended to more general settings such as RL with general function approximation ( Kong et al. , 2021 ) . 2Although ( Wang et al. , 2020a ) considered stationary MDP , as shown in our Corollary 3.3 , the lower bounds of deployment complexity is still related to H . 1.1 CLOSELY RELATED WORKS . We defer the detailed discussion of previous literatures about pure online RL and pure offline RL to Appendix A , and mainly focus on those literatures which considered deployment efficiency and more related to us in this section . To our knowledge , the term “ deployment efficiency ” was first coined by Matsushima et al . ( 2021 ) , but they did not provide a concrete mathematical formulation that is amendable to theoretical investigation . In existing theoretical works , low switching cost is a concept closely related to deployment efficiency , and has been studied in both bandit ( Esfandiari et al. , 2020 ; Han et al. , 2020 ; Gu et al. , 2021 ; Ruan et al. , 2021 ) and RL settings ( Bai et al. , 2020 ; Gao et al. , 2021 ; Kong et al. , 2021 ) . Another related concept is concurrent RL , as proposed by Guo & Brunskill ( 2015 ) . We highlight the difference with them in two-folds from problem setting and techniques . As for the problem setting , existing literature on low switching cost mainly focuses on sub-linear regret guarantees , which does not directly implies a near-optimal policy after a number of policy deployments3 . Besides , low switching-cost RL algorithms ( Bai et al. , 2020 ; Gao et al. , 2021 ; Kong et al. , 2021 ) rely on adaptive switching strategies ( i.e. , the interval between policy switching is not fixed ) , which can be difficult to implement in practical scenarios . For example , in recommendation or education systems , once deployed , a policy usually needs to interact with the population of users for a fair amount of time and generate a lot of data . Moreover , since policy preparation is time-consuming ( which is what motivates our work to begin with ) , it is practically difficult if not impossible to change the policy immediately once collecting enough data for policy update , and it will be a significant overhead compared to a short policy switch interval . Therefore , in applications we target at , it is more reasonable to assume that the sample size in each deployment ( i.e. , between policy switching ) has the same order of magnitude and is large enough so that the overhead of policy preparation can be ignored . More importantly , on the technical side , previous theoretical works on low switching cost mostly use deterministic policies in each deployment , which is easier to analyze . This issue also applies to the work of Guo & Brunskill ( 2015 ) on concurrent PAC RL . However , if the agent can deploy stochastic ( and possibly non-Markov ) policies ( e.g. , a mixture of deterministic policies ) , then intuitively— and as reflected in our lower bounds—exploration can be done much more deployment-efficiently , and we provide a stochastic policy algorithm that achieves an rOpHq deployment complexity and overcomes the ΩpdHq lower bounds for deterministic policy algorithms ( Gao et al. , 2021 ) .

Summary: The authors consider the problem of deployment complexity in reinforcement learning problems, where we want to reduce the number of cycles an agent is typically deployed for interacting with the environment. In the context of a linear MDP, the authors prove an information theoretic lower bound. Further, they also propose algorithms which achieve this optimal deployment efficiency.