Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning

Recent theoretical work studies sample-efficient reinforcement learning ( RL ) extensively in two settings : learning interactively in the environment ( online RL ) , or learning from an offline dataset ( offline RL ) . However , existing algorithms and theories for learning near-optimal policies in these two settings are rather different and disconnected . Towards bridging this gap , this paper initiates the theoretical study of policy finetuning , that is , online RL where the learner has additional access to a “ reference policy ” µ close to the optimal policy π ? in a certain sense . We consider the policy finetuning problem in episodic Markov Decision Processes ( MDPs ) with S states , A actions , and horizon length H . We first design a sharp offline reduction algorithm—which simply executes µ and runs offline policy optimization on the collected dataset—that finds an ε near-optimal policy within Õ ( H3SC ? /ε2 ) episodes , where C ? is the single-policy concentrability coefficient between µ and π ? . This offline result is the first that matches the sample complexity lower bound in this setting , and resolves a recent open question in offline RL . We then establish an Ω ( H3Smin { C ? , A } /ε2 ) sample complexity lower bound for any policy finetuning algorithm , including those that can adaptively explore the environment . This implies that—perhaps surprisingly—the optimal policy finetuning algorithm is either offline reduction or a purely online RL algorithm that does not use µ . Finally , we design a new hybrid offline/online algorithm for policy finetuning that achieves better sample complexity than both vanilla offline reduction and purely online RL algorithms , in a relaxed setting where µ only satisfies concentrability partially up to a certain time step . Overall , our results offer a quantitative understanding on the benefit of a good reference policy , and make a step towards bridging offline and online RL . 1 Introduction . Reinforcement learning ( RL ) —where agents learn to play sequentially in an environment to maximize a cumulative reward function—has achieved great recent success in many artificial intelligence challenges such as video games playing [ 38 , 52 ] , large-scale strategy games ( e.g . GO ) [ 44 , 45 ] , robotic manipulation [ 3 , 32 ] , behavior learning in social scenarios [ 8 ] , and more . In many such challenging domains , achieving human-like or superhuman performance requires training the RL agent with millions of samples ( steps of acting or game playing ) or more . Understanding and improving the sample efficiency of RL algorithms has been a central topic of research . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Sample-efficient RL has been studied in a rich body of theoretical work in two main settings : online RL , in which the learner has interactive access to the environment and can execute any policy ; and offline RL , in which the learner only has access to an “ offline ” dataset collected by executing some ( one or many ) policies within the environment , and is not allowed to further access the environment . These two settings share some common learning goals such as the sample complexity ( number of episodes of playing ) for finding the optimal policy . However , existing algorithms and theories in the online and offline setting seem rather different and disconnected—In online RL , state-of-the-art sample-efficient algorithms typically explore the entire environment , e.g . by using optimism to encourage visitation to unseen states and actions [ 9 , 27 , 19 , 41 , 21 , 5 , 22 , 12 , 23 , 53 ] . In contrast , offline RL does not allow interactive exploration , and sample-efficient policy optimization algorithms typically focus on optimizing an unbiased ( or downward biased ) estimator of the value function [ 39 , 48 , 4 , 40 , 10 , 56 , 35 , 58 , 25 , 42 ] . It is therefore of interest to ask whether these two types of algorithms and theories can be connected in any way . Further , on the empirical end , insights and patterns from offline RL often help as well in designing online RL algorithms and improving the sample efficiency in the real world . For example , there are online RL algorithms that alternate between data collection steps using a fixed policy , and policy improvement steps by learning on the collected dataset [ 20 ] . The replay buffer in value-based algorithms can also be seen as a local form of offline ( off-policy ) policy optimization and are often be used in conjunction with optimistic exploration techniques [ 38 , 18 , 49 ] . The prevalence of these algorithms also offers practical motivations for us to look for a more unified understanding of online and offline RL in theory . These reasonings motivate us to ask the following question : Can we bridge sample-efficient offline and online RL from a theoretical perspective ? This paper proposes policy finetuning , a new RL setting that investigates the benefit of a good initial policy in reinforcement learning , and encapsulates challenges of both online and offline RL . In the policy finetuning problem , the learner is given interactive access to the environment and asked to learn a near-optimal policy , but in addition has access to a reference policy µ that is good in certain aspects . This setting offers great flexibility for the algorithm design : For example , the algorithm is allowed to either simply collect data from µ and run any offline policy optimization algorithm on the collected dataset . It is also allowed to play any other policy interactively , including those that adaptively explores the environment . The policy finetuning problem offers a common playground for both offline and online types of algorithms , and has a unified performance metric ( sample complexity for finding the near-optimal policy ) for comparing their performance . We study the policy finetuning problem theoretically in finite-horizon Markov Decision Processes ( MDPs ) with H time steps , S states , and A actions . We summarize our contributions as follows . • We begin by considering offline reduction algorithms which simply collect data using the reference policy µ and run an offline policy optimization algorithm on the collected dataset . This setting equivalent to offline RL with behavior policy µ , and thus our result translates to a same result for offline RL as well . We design an algorithm PEVI-ADV that is able to find an ε-optimal policy ( for small ε ) within Õ ( H3SC ? /ε2 ) episodes of play , where C ? is the single-policy concentrability coefficient between µ and some optimal policy π ? ( Section 3 ) . This improves over the best existing offline result by anH2 factor in the same setting and matches the lower bound ( up to log factors ) , thereby resolving the recent open question of [ 42 ] on tight offline RL under single-policy concentrability . • Under the same assumption on µ , we establish an Ω ( H3Smin { C ? , A } /ε2 ) sample complexity lower bound for any policy finetuning algorithm , including those that adaptively explores the environment ( Section 4 ) . This implies that the optimal policy finetuning algorithm is either offline reduction via PEVI-ADV , or a “ purely ” online RL algorithm from scratch ( such as UCBVI ) , depending on whether C ? ≤ A . This comes rather surprising , as it rules out possibilities of combining online exploration and knowledge of µ to further improve the sample complexity over the aforementioned two baselines . • Finally , we consider policy finetuning in a more challenging setting where µ only satisfies concentrability up to a certain time step . We design a “ hybrid offline/online ” algorithm HOOVI that combines online exploration and offline data collection , and show that it achieves better sample complexity than both vanilla offline reduction and purely online algorithms in certain cases ( Section 5 ) . This gives a positive example on when such hybrid algorithm designs are beneficial . 1.1 Related work . Sample-efficient online RL There is a long line of work on establishing provably sample-efficient online RL algorithms . A major portion of these works is concerned with the tabular setting with finitely many states and actions [ 9 , 27 , 19 , 5 , 11 , 2 , 22 , 63 ] . For episodic MDPs with inhomogeneous transition functions with S states , andA actions , and horizon lengthH , the optimal sample complexity for finding the ε near-optimal policy is Õ ( H3SA/ε2 ) , achieved by various algorithms such as UCBVI of Azar et al . [ 5 ] and UCB-Advantage of Zhang et al . [ 63 ] . Our paper adapts the reference-advantage decomposition technique of Zhang et al . [ 63 ] to designing sharp offline algorithms . Online RL with with large state/action spaces are also studied by using function approximation in conjunction with structural assumptions on the MDP [ 23 , 61 , 62 , 1 , 41 , 21 , 47 , 53 , 57 , 14 , 24 ] . Offline RL Offline/batch RL studies the case where the agent only has access to an offline dataset obtained by executing a behavior policy in the environment . Sample-efficient learning results in offline RL typically work by assuming either sup-concentrability assumptions [ 39 , 48 , 4 , 40 , 15 , 51 , 10 , 56 ] ) or lower bounded exploration constants [ 58 , 59 ] to ensure the sufficient coverage of offline data over all ( relevant ) states and actions . However , such strong coverage assumptions can often fail to hold in practice [ 16 ] . More recent works address this by using either policy constraint/regularization [ 16 , 35 , 29 , 55 ] , or the pessimism principle to optimize conservatively on the offline data [ 30 , 60 , 28 , 25 , 59 , 42 ] . The policy-constraint/regularization-based approaches prevent the policy to visit states and actions that has no or low coverage from the offline data . Our proposed offline RL algorithm PEVI-ADV ( Algorithm 1 ) is inspired by the pessimistic value iteration algorithms of [ 25 , 42 ] and achieves an improved sample complexity over these work under the same single-policy concentrability assumption on the behavior policy . Bridging online and offline RL Kalashnikov et al . [ 26 ] observed empirically that the performance of policies trained purely from offline data can be improved considerably by a small amount of additional online fine-tuning . A recent line of work studied low switching cost RL [ 6 , 63 , 17 , 54 ] — which forbits online RL algorithms from switching its policy too often—as an interpolation between the online and offline settings . The same problem is also studied empirically as deployment-efficient RL [ 36 , 46 ] . While we also attempt to bridge online and offline RL , our work differs from this line in that our policy finetuning setting allows a direct comparison between “ fully offline ” and “ fully online ” algorithms , whereas the low switching cost setting prohibits fully online algorithms . 2 Preliminaries . Markov Decision Processes In this paper , we consider episodic Markov decision processes ( MDPs ) with time-inhomogeneous transitions , specified by M = ( S , A , H , P , r ) , where S is the state space , A is the action space , H is the horizon length , P = { Ph } Hh=1 where Ph ( ·|s , a ) ∈ ∆S is the transition probabilities at step h , and r = { rh : S ×A → [ 0 , 1 ] } Hh=1 are the deterministic1 reward functions at time step h ∈ [ H ] . Without loss of generality , we assume that the initial state s1 is deterministic2 . Policies , value functions , visitation distributions A policy π = { πh ( ·|s ) } h∈ [ H ] , s∈S consists of distributions πh ( ·|s ) ∈ ∆A . We use Eπ [ · ] to denote the expectation with respect to the random trajectory induced by π in the MDP M , that is , ( s1 , a1 , r1 , s2 , a2 , r2 , . . . , sH , aH , rH ) , where ah = πh ( sh ) , rh = rh ( sh , ah ) , sh+1 ∼ Ph ( ·|sh , ah ) . For each policy π , let V πh : S → R and Qπh : S ×A → R denote its value functions and Q functions at each time step h ∈ [ H ] , that is , V πh ( s ) : = Eπ [ H∑ h′=h rh′ ( sh′ , ah′ ) ∣∣∣∣sh = s ] , Qπh ( s , a ) : = Eπ [ H∑ h′=h rh′ ( sh′ , ah′ ) ∣∣∣∣sh = s , ah = a ] . The operators Ph and Vh are defined as [ PhVh+1 ] ( s , a ) : = E [ Vh+1 ( s′ ) |sh = s , ah = a ] and [ VhVh+1 ] ( s , a ) : = Var [ Vh+1 ( s′ ) |sh = s , ah = a ] for any value function Vh+1 at time step h + 1 . 1While we assume deterministic rewards for simplicity , our results can be straightforwardly generalized to stochastic rewards , as the major difficulty is in learning the transitions rather than learning the rewards . 2Any MDP with stochastic s1 is equivalent to an MDP with deterministic by creating a dummy initial state s0 and increasing the horizon by 1 . We also use P̂h and V̂h to denote empirical versions of these operators building on estimated models ( which will be clear in the context ) . We use π ? : = arg maxπ V π 1 ( s1 ) to denote any optimal policy , and V ? h : = V π ? h and Q ? h : = Q π ? h to denote the value function and Q function of π ? at all h ∈ [ H ] . Throughout this paper , our learning goal is to find an near-optimal policy π̂ such that V ? 1 ( s1 ) − V π̂1 ( s1 ) ≤ ε . Finally , we let dπh denote the state ( -action ) visitation distributions of π at time step h ∈ [ H ] : dπh ( s ) : = P ( sh = s|π ) , and dπh ( s , a ) : = P ( sh = s , ah = a|π ) . Miscellaneous We use standard O ( · ) and Ω ( · ) notation : A = O ( B ) is defined as A ≤ CB for some absolute constant C > 0 ( and similarly for Ω ) . The tilded notation A = Õ ( B ) denotes A ≤ CL ·B where L is a poly-logarithmic factor of problem parameters .

Summary: This paper provides a theoretical framework for policy finetuning in episodic finite-horizon MDPs, where the agent can interact with the environment online and also has access to offline data from a reference policy. The authors develop/provides three main results:  (1) A fully offline algorithm based on finite-horizon VI-LCB (Rashidinejad et al.'21) and improving its near-optimal sample complexity by the order of O(H^2) in polylogarithmic time (H being the horizon length). This is done by (a) replacing Hoeffding bonus in VI-LCB with Bernstein bonus and (b) adopting the reference-advantage decomposition technique of (Zhang et al'20)  (2) Under concentrability assumption (boundedness of state-action importance weights w.r.t some optimal policy), the authors argue that there is no finetuning algorithm with better sample complexity than the extreme cases(purely online or the proposed offline algorithm) for some MDP and reference policy. This establishes a lower bound for finetuning.  (3) A partial concentrability assumption (reference policy is only guaranteed to satisfy concentrability up to a certain step) and a two-satge hybrid algorithm based on Nash-VI algorithm(Liu, et.al'20) and truncated version of their proposed offline algorithm.