Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning

Recent theoretical work studies sample-efficient reinforcement learning ( RL ) extensively in two settings : learning interactively in the environment ( online RL ) , or learning from an offline dataset ( offline RL ) . However , existing algorithms and theories for learning near-optimal policies in these two settings are rather different and disconnected . Towards bridging this gap , this paper initiates the theoretical study of policy finetuning , that is , online RL where the learner has additional access to a “ reference policy ” µ close to the optimal policy π ? in a certain sense . We consider the policy finetuning problem in episodic Markov Decision Processes ( MDPs ) with S states , A actions , and horizon length H . We first design a sharp offline reduction algorithm—which simply executes µ and runs offline policy optimization on the collected dataset—that finds an ε near-optimal policy within Õ ( H3SC ? /ε2 ) episodes , where C ? is the single-policy concentrability coefficient between µ and π ? . This offline result is the first that matches the sample complexity lower bound in this setting , and resolves a recent open question in offline RL . We then establish an Ω ( H3Smin { C ? , A } /ε2 ) sample complexity lower bound for any policy finetuning algorithm , including those that can adaptively explore the environment . This implies that—perhaps surprisingly—the optimal policy finetuning algorithm is either offline reduction or a purely online RL algorithm that does not use µ . Finally , we design a new hybrid offline/online algorithm for policy finetuning that achieves better sample complexity than both vanilla offline reduction and purely online RL algorithms , in a relaxed setting where µ only satisfies concentrability partially up to a certain time step . Overall , our results offer a quantitative understanding on the benefit of a good reference policy , and make a step towards bridging offline and online RL . 1 Introduction . Reinforcement learning ( RL ) —where agents learn to play sequentially in an environment to maximize a cumulative reward function—has achieved great recent success in many artificial intelligence challenges such as video games playing [ 38 , 52 ] , large-scale strategy games ( e.g . GO ) [ 44 , 45 ] , robotic manipulation [ 3 , 32 ] , behavior learning in social scenarios [ 8 ] , and more . In many such challenging domains , achieving human-like or superhuman performance requires training the RL agent with millions of samples ( steps of acting or game playing ) or more . Understanding and improving the sample efficiency of RL algorithms has been a central topic of research . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Sample-efficient RL has been studied in a rich body of theoretical work in two main settings : online RL , in which the learner has interactive access to the environment and can execute any policy ; and offline RL , in which the learner only has access to an “ offline ” dataset collected by executing some ( one or many ) policies within the environment , and is not allowed to further access the environment . These two settings share some common learning goals such as the sample complexity ( number of episodes of playing ) for finding the optimal policy . However , existing algorithms and theories in the online and offline setting seem rather different and disconnected—In online RL , state-of-the-art sample-efficient algorithms typically explore the entire environment , e.g . by using optimism to encourage visitation to unseen states and actions [ 9 , 27 , 19 , 41 , 21 , 5 , 22 , 12 , 23 , 53 ] . In contrast , offline RL does not allow interactive exploration , and sample-efficient policy optimization algorithms typically focus on optimizing an unbiased ( or downward biased ) estimator of the value function [ 39 , 48 , 4 , 40 , 10 , 56 , 35 , 58 , 25 , 42 ] . It is therefore of interest to ask whether these two types of algorithms and theories can be connected in any way . Further , on the empirical end , insights and patterns from offline RL often help as well in designing online RL algorithms and improving the sample efficiency in the real world . For example , there are online RL algorithms that alternate between data collection steps using a fixed policy , and policy improvement steps by learning on the collected dataset [ 20 ] . The replay buffer in value-based algorithms can also be seen as a local form of offline ( off-policy ) policy optimization and are often be used in conjunction with optimistic exploration techniques [ 38 , 18 , 49 ] . The prevalence of these algorithms also offers practical motivations for us to look for a more unified understanding of online and offline RL in theory . These reasonings motivate us to ask the following question : Can we bridge sample-efficient offline and online RL from a theoretical perspective ? This paper proposes policy finetuning , a new RL setting that investigates the benefit of a good initial policy in reinforcement learning , and encapsulates challenges of both online and offline RL . In the policy finetuning problem , the learner is given interactive access to the environment and asked to learn a near-optimal policy , but in addition has access to a reference policy µ that is good in certain aspects . This setting offers great flexibility for the algorithm design : For example , the algorithm is allowed to either simply collect data from µ and run any offline policy optimization algorithm on the collected dataset . It is also allowed to play any other policy interactively , including those that adaptively explores the environment . The policy finetuning problem offers a common playground for both offline and online types of algorithms , and has a unified performance metric ( sample complexity for finding the near-optimal policy ) for comparing their performance . We study the policy finetuning problem theoretically in finite-horizon Markov Decision Processes ( MDPs ) with H time steps , S states , and A actions . We summarize our contributions as follows . • We begin by considering offline reduction algorithms which simply collect data using the reference policy µ and run an offline policy optimization algorithm on the collected dataset . This setting equivalent to offline RL with behavior policy µ , and thus our result translates to a same result for offline RL as well . We design an algorithm PEVI-ADV that is able to find an ε-optimal policy ( for small ε ) within Õ ( H3SC ? /ε2 ) episodes of play , where C ? is the single-policy concentrability coefficient between µ and some optimal policy π ? ( Section 3 ) . This improves over the best existing offline result by anH2 factor in the same setting and matches the lower bound ( up to log factors ) , thereby resolving the recent open question of [ 42 ] on tight offline RL under single-policy concentrability . • Under the same assumption on µ , we establish an Ω ( H3Smin { C ? , A } /ε2 ) sample complexity lower bound for any policy finetuning algorithm , including those that adaptively explores the environment ( Section 4 ) . This implies that the optimal policy finetuning algorithm is either offline reduction via PEVI-ADV , or a “ purely ” online RL algorithm from scratch ( such as UCBVI ) , depending on whether C ? ≤ A . This comes rather surprising , as it rules out possibilities of combining online exploration and knowledge of µ to further improve the sample complexity over the aforementioned two baselines . • Finally , we consider policy finetuning in a more challenging setting where µ only satisfies concentrability up to a certain time step . We design a “ hybrid offline/online ” algorithm HOOVI that combines online exploration and offline data collection , and show that it achieves better sample complexity than both vanilla offline reduction and purely online algorithms in certain cases ( Section 5 ) . This gives a positive example on when such hybrid algorithm designs are beneficial . 1.1 Related work . Sample-efficient online RL There is a long line of work on establishing provably sample-efficient online RL algorithms . A major portion of these works is concerned with the tabular setting with finitely many states and actions [ 9 , 27 , 19 , 5 , 11 , 2 , 22 , 63 ] . For episodic MDPs with inhomogeneous transition functions with S states , andA actions , and horizon lengthH , the optimal sample complexity for finding the ε near-optimal policy is Õ ( H3SA/ε2 ) , achieved by various algorithms such as UCBVI of Azar et al . [ 5 ] and UCB-Advantage of Zhang et al . [ 63 ] . Our paper adapts the reference-advantage decomposition technique of Zhang et al . [ 63 ] to designing sharp offline algorithms . Online RL with with large state/action spaces are also studied by using function approximation in conjunction with structural assumptions on the MDP [ 23 , 61 , 62 , 1 , 41 , 21 , 47 , 53 , 57 , 14 , 24 ] . Offline RL Offline/batch RL studies the case where the agent only has access to an offline dataset obtained by executing a behavior policy in the environment . Sample-efficient learning results in offline RL typically work by assuming either sup-concentrability assumptions [ 39 , 48 , 4 , 40 , 15 , 51 , 10 , 56 ] ) or lower bounded exploration constants [ 58 , 59 ] to ensure the sufficient coverage of offline data over all ( relevant ) states and actions . However , such strong coverage assumptions can often fail to hold in practice [ 16 ] . More recent works address this by using either policy constraint/regularization [ 16 , 35 , 29 , 55 ] , or the pessimism principle to optimize conservatively on the offline data [ 30 , 60 , 28 , 25 , 59 , 42 ] . The policy-constraint/regularization-based approaches prevent the policy to visit states and actions that has no or low coverage from the offline data . Our proposed offline RL algorithm PEVI-ADV ( Algorithm 1 ) is inspired by the pessimistic value iteration algorithms of [ 25 , 42 ] and achieves an improved sample complexity over these work under the same single-policy concentrability assumption on the behavior policy . Bridging online and offline RL Kalashnikov et al . [ 26 ] observed empirically that the performance of policies trained purely from offline data can be improved considerably by a small amount of additional online fine-tuning . A recent line of work studied low switching cost RL [ 6 , 63 , 17 , 54 ] — which forbits online RL algorithms from switching its policy too often—as an interpolation between the online and offline settings . The same problem is also studied empirically as deployment-efficient RL [ 36 , 46 ] . While we also attempt to bridge online and offline RL , our work differs from this line in that our policy finetuning setting allows a direct comparison between “ fully offline ” and “ fully online ” algorithms , whereas the low switching cost setting prohibits fully online algorithms . 2 Preliminaries . Markov Decision Processes In this paper , we consider episodic Markov decision processes ( MDPs ) with time-inhomogeneous transitions , specified by M = ( S , A , H , P , r ) , where S is the state space , A is the action space , H is the horizon length , P = { Ph } Hh=1 where Ph ( ·|s , a ) ∈ ∆S is the transition probabilities at step h , and r = { rh : S ×A → [ 0 , 1 ] } Hh=1 are the deterministic1 reward functions at time step h ∈ [ H ] . Without loss of generality , we assume that the initial state s1 is deterministic2 . Policies , value functions , visitation distributions A policy π = { πh ( ·|s ) } h∈ [ H ] , s∈S consists of distributions πh ( ·|s ) ∈ ∆A . We use Eπ [ · ] to denote the expectation with respect to the random trajectory induced by π in the MDP M , that is , ( s1 , a1 , r1 , s2 , a2 , r2 , . . . , sH , aH , rH ) , where ah = πh ( sh ) , rh = rh ( sh , ah ) , sh+1 ∼ Ph ( ·|sh , ah ) . For each policy π , let V πh : S → R and Qπh : S ×A → R denote its value functions and Q functions at each time step h ∈ [ H ] , that is , V πh ( s ) : = Eπ [ H∑ h′=h rh′ ( sh′ , ah′ ) ∣∣∣∣sh = s ] , Qπh ( s , a ) : = Eπ [ H∑ h′=h rh′ ( sh′ , ah′ ) ∣∣∣∣sh = s , ah = a ] . The operators Ph and Vh are defined as [ PhVh+1 ] ( s , a ) : = E [ Vh+1 ( s′ ) |sh = s , ah = a ] and [ VhVh+1 ] ( s , a ) : = Var [ Vh+1 ( s′ ) |sh = s , ah = a ] for any value function Vh+1 at time step h + 1 . 1While we assume deterministic rewards for simplicity , our results can be straightforwardly generalized to stochastic rewards , as the major difficulty is in learning the transitions rather than learning the rewards . 2Any MDP with stochastic s1 is equivalent to an MDP with deterministic by creating a dummy initial state s0 and increasing the horizon by 1 . We also use P̂h and V̂h to denote empirical versions of these operators building on estimated models ( which will be clear in the context ) . We use π ? : = arg maxπ V π 1 ( s1 ) to denote any optimal policy , and V ? h : = V π ? h and Q ? h : = Q π ? h to denote the value function and Q function of π ? at all h ∈ [ H ] . Throughout this paper , our learning goal is to find an near-optimal policy π̂ such that V ? 1 ( s1 ) − V π̂1 ( s1 ) ≤ ε . Finally , we let dπh denote the state ( -action ) visitation distributions of π at time step h ∈ [ H ] : dπh ( s ) : = P ( sh = s|π ) , and dπh ( s , a ) : = P ( sh = s , ah = a|π ) . Miscellaneous We use standard O ( · ) and Ω ( · ) notation : A = O ( B ) is defined as A ≤ CB for some absolute constant C > 0 ( and similarly for Ω ) . The tilded notation A = Õ ( B ) denotes A ≤ CL ·B where L is a poly-logarithmic factor of problem parameters .

Summary: The paper attempts to bridge the gap in theory for online and offline RL. The paper introduces a theoretical study of policy fine-tuning, an online RL setup where the learner has access to policy close to the optimal policy in a certain metric. Then the authors introduce a shart offline reduction algorithm that collects data using this policy and then runs offline RL training on the collected data and prove a converge bound for this algorithm. Then authors establish a lower bound on any policy fine-tuning method. Finally, the authors design a new hybrid offline/online algorithm.