Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning

1 Introduction . The problem of learning a near-optimal policy is a core challenge in reinforcement learning ( RL ) . In many settings , it is beneficial to be able to learn a good policy using only a pre-collected set of data , without further exploration with the environment ; this problem is known as offline or batch policy learning . The offline setting has unique challenges due to the incomplete information about the Markov decision process ( MDP ) encoded in the available dataset . For example , due to maximization bias , a naive offline algorithm can return a policy with a severely overestimated value . In order to avoid such undesirable behavior , researchers have introduced the idea of pessimism under uncertainty , and there is now a growing literature ( e.g. , Liu et al . ( 2020 ) ; Jin et al . ( 2020b ) ; Buckman et al . ( 2020 ) ; Kumar et al . ( 2019 ) ; Kidambi et al . ( 2020 ) ; Yu et al . ( 2020 ) ) on different ways in which pessimism can be incorporated . See Appendix B for additional references and discussion of this body of work . At a high level , incorporating pessimism prevents algorithms from settling down on uncertain policies whose value might be misleadingly high under the current dataset due to statistical errors . By using pessimism , uncertain policies are penalized in such a way that only those policies robust to statistical errors are returned . The principle can be implemented in at least two different ways : ( a ) by penalizing policies that are far from the one that generated the dataset ; or ( b ) by penalizing the value functions of policies not well covered by the dataset . In this paper , we take the latter avenue . ⇤This work was fully completed while Andrea Zanette was a PhD candidate at Stanford University . Future updates of this work will be available at https : //arxiv.org/abs/2108.08812 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1.1 Overview and our contributions . Implementing pessimism with function approximation is challenging for several reasons . First , uncertainty must be estimated with particular care . On one hand , underestimating it can fail to correct the coverage problem . On the other hand , overestimating it leads to policies that are too conservative and thus underperform . Second , the incorporation of pessimism may introduce complex , higher order perturbations into the value function class handled by the algorithm . Similar issues can arise when adding optimistic bonuses in the exploration . The increased complexity of the function class often requires additional assumptions on the model , because the new class needs to interact “ nicely ” with the Bellman operator . Prior art on pessimism with function approximation has by-passed this problem by making strong model assumptions , such as low-rank transitions Jin et al . ( 2020b ) or algorithm-specific assumptions Liu et al . ( 2020 ) . Actor-critic methods : Most past theoretical work on offline reinforcement learning on finding with high probability the policy with the highest performance has focused on algorithms that are either model or value-based2 Liu et al . ( 2020 ) ; Jin et al . ( 2020b ) ; Buckman et al . ( 2020 ) ; Kidambi et al . ( 2020 ) ; Yu et al . ( 2020 ) ; these often incorporate pessimism into the estimates of the policy performance . Actor-critic methods are a hybrid class of methods that mitigate some deficiencies of methods that are either purely policy or purely value-based Konda and Tsitsiklis ( 2000 , 2003 ) ; Heess et al . ( 2015 ) ; Haarnoja et al . ( 2017 , 2018 ) ; in modern RL , they are widely used in practice ( e.g. , Levine et al . ( 2020 ) ; Wu et al . ( 2019 , 2021 ) ; Kumar et al . ( 2019 , 2020 ) ) . An actor-critic method generally consists of an actor that changes the policy in order to maximize its value as estimated by the critic . Given their popularity , it is natural to ask the following question : do actor-critic methods provably offer any advantage in offline RL ? The main contribution of this paper is to give a positive answer to this question : by separating the policy optimization from the policy evaluation , both tasks become simpler to design and the pessimism principle can be incorporated more naturally . Contributions : More specifically , we study the problem of policy learning using linear function approximation in the offline setting . We assume that we are given a batch data set D , in which each sample consists of a quadruple . The first two components are the state-action pair , corresponding to the state in which a given action was taken , and the last two components correspond to a noisy observation of the reward , and a successor state drawn from the appropriate transition function . Our theory allows for a very general dependence structure among the the state-action pairs in these samples ; when the data set is ordered according to how the samples were collected ( which need not be related to a trajectory ) , we allow the state-action pair at any given instant to depend on all past samples . This set-up allows from data collected from arbitrary policies , mixtures of policies , generative models or even in adversarial manner . Given such a data set , our objective is to find the policy that performs best in the face of uncertainty . In particular , we need to account for the fact that the optimal policy ⇡⇤ for the underlying MDP may not be well covered by the dataset D , in which case the associated uncertainty would be prohibitive . In order to achieve this goal , we design an actor-critic procedure that iteratively optimizes a lower bound on the value of the optimal policy . Suppose that we are interested in optimizing the value function at some given initial s1 . Our strategy works as follows : for any given policy ⇡ , we construct a family M ( ⇡ ) of “ statistically plausible ” MDPs , and use them to define a simple second-order cone program . By solving this convex program , we obtain value function estimate V ⇡ M ( s1 ) = argminM2M ( ⇡ ) V ⇡ M ( s1 ) that—for an appropriately constructed family M ( ⇡ ) —is guaranteed to be a lower bound on the true value function of ⇡ in the unknown MDP that generated the dataset . Given a procedure for producing such lower bounds , it is then natural to maximize these lower bounds over some family ⇧ of policies . This combination leads to the saddle-point problem max ⇡2⇧ min M2M ( ⇡ ) V ⇡ M ( s1 ) . ( 1 ) Note that actor-critic methods fit naturally in this framework : the critic provides a pessimistic evaluation of any given policy ⇡ , and the actor solves the outer maximization problem over policies . 2Exceptions to this include importance-sampling based approaches to selecting among a finite set of policies ( e.g . Mandel et al . ( 2014 ) ; Thomas et al . ( 2015 , 2019 ) ) ; however , such approaches have focused on operating without a Markov assumption and inherently provide much looser guarantees than the ones we and others consider for the Markov setting . This decoupling lends itself to a computationally tractable implementation , along with an analysis of the procedure . In particular , we show that the actor ’ s sequence of estimated policies enjoys online learning-style guarantees with respect to a sequence of pessimistic MDPs implicitly identified by the critic . The way in which we introduce pessimism is a second key component of the algorithmic framework . In particular , in line with our previous paper Zanette et al . ( 2020b ) , we do so without enlarging the prescribed classes of functions and policies . We do so by a direct perturbation of the value functions examined by the critic ; there is no addition of pessimistic bonuses or absorbing states . Since the class of value functions is not altered , this method has two main advantages . First , there are no additional model assumptions compared to the standard—that is non-pessimistic—version of the actor-critic method . Second , the complexity of the underlying classes is not increased , thereby allowing us to construct tight confidence intervals and estimation error bounds that are minimax optimal up to logarithmic factors . The remainder of this paper is organized as follows . We begin in Section 2 with background on MDPS , and then introduce the modeling assumptions that underlie the analysis of this paper . In Section 3 , we introduce the algorithm studied in this paper , namely the Pessimistic Actor Critic for Learning without Exploration ( for short , PACLE ) algorithm . Section 4 provides statements of our main results and discussion of their consequences , including an upper bound on the PACLE algorithm in Theorem 1 , and a minimax lower bound in Theorem 2 . In Section A , we provide an outline of the proof of Theorem 1 , with various technical details as well as the proof of Theorem 2 deferred to the appendices . We conclude with a discussion in Section 5 . 1.2 Notation . For the reader ’ s convenience , we summarize here some notation used throughout the paper . We let Bd ( r ) = { x 2 Rd | kxk2 r } denote the Euclidean ball of radius r 2 R in dimension d ; we simply write B when there is no possibility of confusion . For a vector x 2 Rd , we use [ x ] i to denote its ith component . We use the eO notation to denote an upper bound that holds up to constants and log factors in the input parameters ( 1 , d , H ) . The notation . means an upper bound that holds up to a constant , with an analogous definition for & . 2 Background and problem formulation . We begin by providing some background , before introducing the assumptions that underlie our problem formulation . 2.1 Markov decision processes . In this paper , we focus on finite-horizon Markov decision processes , for which we provide a very brief introduction here . See the books Puterman ( 1994 ) ; Bertsekas and Tsitsiklis ( 1996 ) ; Bertsekas ( 1995 ) ; Sutton and Barto ( 2018 ) for more background and detail . A finite-horizon MDP is specified by a positive integer H , and events take place over a sequence of stages indexed by the time step h 2 [ H ] def= { 1 , . . . , H } . The underlying dynamics involve a state space S , and are controlled by actions that take values in some action set A . In this paper , we allow the state space to be arbitrary ( continous or discrete ) , whereas our analysis applies to discrete action spaces . For each time step h 2 [ H ] , there is a reward function rh : S ⇥ A ! R , and for every time step h and state-action pair ( s , a ) , there is a transition function Ph ( · | s , a ) . When at horizon h , if the agent takes action a in state s , it receives a random reward drawn from a distribution Rh ( s , a ) with mean rh ( s , a ) , and it then transitions randomly to a next state s+ drawn from the transition function Ph ( · | s , a ) . A policy ⇡h at stage h is a mapping from the state space S to the action space A . Given a full policy ⇡ = ( ⇡1 , . . . , ⇡H ) , the state-action value function at time step h is given by Q⇡ h ( s , a ) = rh ( s , a ) + ES ` ⇠⇡| ( s , a ) HX ` =h+1 r ` ( S ` , ⇡ ` ( S ` ) ) , ( 2 ) where the expectation is over the trajectories induced by ⇡ upon starting from the pair ( s , a ) . When we omit the starting state-action pair ( s , a ) , the expectation is intended to start from a fixed state denoted by s1 . The value function associated to ⇡ is V ⇡h ( s ) = Q ⇡ h ( s , ⇡h ( s ) ) . For a given policy ⇡ , we define the Bellman evaluation operator T ⇡ h ( Qh+1 ) ( s , a ) = rh ( s , a ) + ES0⇠Ph ( s , a ) EA0⇠⇡Qh+1 ( S0 , A0 ) . Under some regularity conditions Puterman ( 1994 ) ; Shreve and Bertsekas ( 1978 ) , there always exists an optimal policy ⇡ ? whose value and action-value functions are defined as V ? h ( s ) = V ⇡ ? h ( s ) = sup ⇡ V ⇡ h ( s ) , and Q ? h ( s , a ) = Q⇡ ? h ( s , a ) = sup ⇡ Q⇡ h ( s , a ) .

Summary: This paper presents actor-critic methods for offline RL. Specific aspects of the problem addressed by the paper include being able to achieve pessimism without adding in pessimism based bonuses or through defining absorbing states. This allows the approach to work with the original value function class without modifications. The paper also presents distinctions between standard assumptions used in both online/offline RL literature relating to low rank MDPs, restricted closeness, and linear Q^\pi assumption.