The Importance of Pessimism in Fixed-Dataset Policy Optimization

We study worst-case guarantees on the expected return of fixed-dataset policy optimization algorithms . Our core contribution is a unified conceptual and mathematical framework for the study of algorithms in this regime . This analysis reveals that for naı̈ve approaches , the possibility of erroneous value overestimation leads to a difficult-to-satisfy requirement : in order to guarantee that we select a policy which is near-optimal , we may need the dataset to be informative of the value of every policy . To avoid this , algorithms can follow the pessimism principle , which states that we should choose the policy which acts optimally in the worst possible world . We show why pessimistic algorithms can achieve good performance even when the dataset is not informative of every policy , and derive families of algorithms which follow this principle . These theoretical findings are validated by experiments on a tabular gridworld , and deep learning experiments on four MinAtar environments . 1 INTRODUCTION . We consider fixed-dataset policy optimization ( FDPO ) , in which a dataset of transitions from an environment is used to find a policy with high return.1 We compare FDPO algorithms by their worst-case performance , expressed as high-probability guarantees on the suboptimality of the learned policy . It is perhaps obvious that in order to maximize worst-case performance , a good FDPO algorithm should select a policy with high worst-case value . We call this the pessimism principle of exploitation , as it is analogous to the widely-known optimism principle ( Lattimore & Szepesvári , 2020 ) of exploration.2 Our main contribution is a theoretical justification of the pessimism principle in FDPO , based on a bound that characterizes the suboptimality incurred by an FDPO algorithm . We further demonstrate how this bound may be used to derive principled algorithms . Note that the core novelty of our work is not the idea of pessimism , which is an intuitive concept that appears in a variety of contexts ; rather , our contribution is a set of theoretical results rigorously explaining how pessimism is important in the specific setting of FDPO . An example conveying the intuition behind our results can be found in Appendix G.1 . We first analyze a family of non-pessimistic naı̈ve FDPO algorithms , which estimate the environment from the dataset via maximum likelihood and then apply standard dynamic programming techniques . We prove a bound which shows that the worst-case suboptimality of these algorithms is guaranteed to be small when the dataset contains enough data that we are certain about the value of every possible policy . This is caused by the outsized impact of value overestimation errors on suboptimality , sometimes called the optimizer ’ s curse ( Smith & Winkler , 2006 ) . It is a fundamental consequence of ignoring the disconnect between the true environment and the picture painted by our limited observations . Importantly , it is not reliant on errors introduced by function approximation . 1We use the term fixed-dataset policy optimization to emphasize the computational procedure ; this setting has also been referred to as batch RL ( Ernst et al. , 2005 ; Lange et al. , 2012 ) and more recently , offline RL ( Levine et al. , 2020 ) . We emphasize that this is a well-studied setting , and we are simply choosing to refer to it by a more descriptive name . 2The optimism principle states that we should select a policy with high best-case value . We contrast these findings with an analysis of pessimistic FDPO algorithms , which select a policy that maximizes some notion of worst-case expected return . We show that these algorithms do not require datasets which inform us about the value of every policy to achieve small suboptimality , due to the critical role that pessimism plays in preventing overestimation . Our analysis naturally leads to two families of principled pessimistic FDPO algorithms . We prove their improved suboptimality guarantees , and confirm our claims with experiments on a gridworld . Finally , we extend one of our pessimistic algorithms to the deep learning setting . Recently , several deep-learning-based algorithms for fixed-dataset policy optimization have been proposed ( Agarwal et al. , 2019 ; Fujimoto et al. , 2019 ; Kumar et al. , 2019 ; Laroche et al. , 2019 ; Jaques et al. , 2019 ; Kidambi et al. , 2020 ; Yu et al. , 2020 ; Wu et al. , 2019 ; Wang et al. , 2020 ; Kumar et al. , 2020 ; Liu et al. , 2020 ) . Our work is complementary to these results , as our contributions are conceptual , rather than algorithmic . Our primary goal is to theoretically unify existing approaches and motivate the design of pessimistic algorithms more broadly . Using experiments in the MinAtar game suite ( Young & Tian , 2019 ) , we provide empirical validation for the predictions of our analysis . The problem of fixed-dataset policy optimization is closely related to the problem of reinforcement learning , and as such , there is a large body of work which contains ideas related to those discussed in this paper . We discuss these works in detail in Appendix E . 2 BACKGROUND . We anticipate most readers will be familiar with the concepts and notation , which is fairly standard in the reinforcement learning literature . In the interest of space , we relegate a full presentation to Appendix A . Here , we briefly give an informal overview of the background necessary to understand the main results . We represent the environment as a Markov Decision Process ( MDP ) , denoted M : = 〈S , A , R , P , γ , ρ〉 . We assume without loss of generality that R ( 〈s , a〉 ) ∈ [ 0 , 1 ] , and denote its expectation as r ( 〈s , a〉 ) . ρ represents the start-state distribution . Policies π can act in the environment , represented by action matrix Aπ , which maps each state to the probability of each state-action when following π . Value functions v assign some real value to each state . We use vπM to denote the value function which assigns the sum of discounted rewards in the environment when following policy π . A dataset D contains transitions sampled from the environment . From a dataset , we can compute the empirical reward and transition functions , rD and PD , and the empirical policy , π̂D . An important concept for our analysis is the value uncertainty function , denoted µπD , δ , which returns a high-probability upper-bound to the error of a value function derived from datasetD . Certain value uncertainty functions are decomposable by states or state-actions , meaning they can be written as the weighted sum of more local uncertainties . See Appendix B for more detail . Our goal is to analyze the suboptimality of a specific class of FDPO algorithms , called value-based FDPO algorithms , which have a straightforward structure : they use a fixed-dataset policy evaluation ( FDPE ) algorithm to assign a value to each policy , and then select the policy with the maximum value . Furthermore , we consider FDPE algorithms whose solutions satisfy a fixed-point equation . Thus , a fixed-point equation defines a FDPE objective , which in turn defines a value-based FDPO objective ; we call the set of all algorithms that implement these objectives the family of algorithms defined by the fixed-point equation . 3 OVER/UNDER DECOMPOSITION OF SUBOPTIMALITY . Our first theoretical contribution is a simple but informative bound on the suboptimality of any value-based FDPO algorithm . Next , in Section 4 , we make this concrete by defining the family of naı̈ve algorithms and invoking this bound . This bound is insightful because it distinguishes the impact of errors of value overestimation from errors of value underestimation , defined as : Definition 1 . Consider any fixed-dataset policy evaluation algorithm E on any dataset D and any policy π. Denote vπD : = E ( D , π ) . We define the underestimation error as Eρ [ vπM − vπD ] and overestimation error as Eρ [ vπD − vπM ] . The following lemma shows how these quantities can be used to bound suboptimality . Lemma 1 ( Value-based FDPO suboptimality bound ) . Consider any value-based fixed-dataset policy optimization algorithmOVB , with fixed-dataset policy evaluation subroutine E . For any policy π and dataset D , denote vπD : = E ( D , π ) . The suboptimality of OVB is bounded by SUBOPT ( OVB ( D ) ) ≤ inf π ( Eρ [ v π∗M M − v π M ] + Eρ [ vπM − vπD ] ) + sup π ( Eρ [ vπD − vπM ] ) Proof . See Appendix C.1 . This bound is tight ; see Appendix C.2 . The bound highlights the potentially outsized impact of overestimation on the suboptimality of a FDPO algorithm . To see this , we consider each of its terms in isolation : SUBOPT ( OVB ( D ) ) ≤ inf π ( ( A1 ) ︷ ︸︸ ︷ Eρ [ v π∗M M − v π M ] + ( A2 ) ︷ ︸︸ ︷ Eρ [ vπM − vπD ] ) ︸ ︷︷ ︸ ( A ) + sup π ( ( B1 ) ︷ ︸︸ ︷ Eρ [ vπD − vπM ] ) ︸ ︷︷ ︸ ( B ) The term labeled ( A ) reflects the degree to which the dataset informs us of a near-optimal policy . For any policy π , ( A1 ) captures the suboptimality of that policy , and ( A2 ) captures its underestimation error . Since ( A ) takes an infimum , this term will be small whenever there is at least one reasonable policy whose value is not very underestimated . On the other hand , the term labeled ( B ) corresponds to the largest overestimation error on any policy . Because it consists of a supremum over all policies , it will be small only when no policies are overestimated at all . Even a single overestimation can lead to significant suboptimality . We see from these two terms that errors of overestimation and underestimation have differing impacts on suboptimality , suggesting that algorithms should be designed with this asymmetry in mind . We will see in Section 5 how this may be done . But first , let us further understand why this is necessary by studying in more depth a family of algorithms which treats its errors of overestimation and underestimation equivalently . 4 NAÏVE ALGORITHMS . The goal of this section is to paint a high-level picture of the worst-case suboptimality guarantees of a specific family of non-pessimistic approaches , which we call naı̈ve FDPO algorithms . Informally , the naı̈ve approach is to take the limited dataset of observations at face value , treating it as though it paints a fully accurate picture of the environment . Naı̈ve algorithms construct a maximum-likelihood MDP from the dataset , then use standard dynamic programming approaches on this empirical MDP . Definition 2 . A naı̈ve algorithm is any algorithm in the family defined by the fixed-point function fnaı̈ve ( v π ) : = Aπ ( rD + γPDv π ) . Various FDPE and FDPO algorithms from this family could be described ; in this work , we do not study these implementations in detail , although we do give pseudocode for some implementations in Appendix D.1 . One example of a naı̈ve FDPO algorithm which can be found in the literature is certainty equivalence ( Jiang , 2019a ) . The core ideas behind naı̈ve algorithms can also be found in the function approximation literature , for example in FQI ( Ernst et al. , 2005 ; Jiang , 2019b ) . Additionally , when available data is held fixed , nearly all existing deep reinforcement learning algorithms are transformed into naı̈ve value-based FDPO algorithms . For example , DQN ( Mnih et al. , 2015 ) with a fixed replay buffer is a naı̈ve value-based FDPO algorithm . Theorem 1 ( Naı̈ve FDPO suboptimality bound ) . Consider any naı̈ve value-based fixed-dataset policy optimization algorithm OVBnaı̈ve . Let µ be any value uncertainty function . With probability at least 1− δ , the suboptimality of OVBnaı̈ve is bounded with probability at least 1− δ by SUBOPT ( OVBnaı̈ve ( D ) ) ≤ inf π ( Eρ [ v π∗M M − v π M ] + Eρ [ µπD , δ ] ) + sup π Eρ [ µπD , δ ] Proof . This result follows directly from Lemma 1 and Lemma 3 . The infimum term is small whenever there is some reasonably good policy with low value uncertainty . In practice , this condition can typically be satisfied , for example by including expert demonstrations in the dataset . On the other hand , the supremum term will only be small if we have low value uncertainty for all policies – a much more challenging requirement . This explains the behavior of pathological examples , e.g . in Appendix G.1 , where performance is poor despite access to virtually unlimited amounts of data from a near-optimal policy . Such a dataset ensures that the first term will be small by reducing value uncertainty of the near-optimal data collection policy , but does little to reduce the value uncertainty of any other policy , leading the second term to be large . However , although pathological examples exist , it is clear that this bound will not be tight on all environments . It is reasonable to ask : is it likely that this bound will be tight on real-world examples ? We argue that it likely will be . We identify two properties that most real-world tasks share : ( 1 ) The set of policies is pyramidal : there are an enormous number of bad policies , many mediocre policies , a few good policies , etc . ( 2 ) Due to the size of the state space and cost of data collection , most policies have high value uncertainty . Given that these assumptions hold , naı̈ve algorithms will perform as poorly on most real-world environments as they do on pathological examples . Consider : there are many more policies than there is data , so there will be many policies with high value uncertainty ; naı̈ve algorithms will likely overestimate several of these policies , and erroneously select one ; since good policies are rare , the selected policy will likely be bad . It follows that running naı̈ve algorithms on real-world problems will typically yield suboptimality close to our worst-case bound . And , indeed , on deep RL benchmarks , which are selected due to their similarity to real-world settings , overestimation has been widely observed , typically correlated with poor performance ( Bellemare et al. , 2016 ; Van Hasselt et al. , 2016 ; Fujimoto et al. , 2019 ) .

Summary: The message of this paper is that naive policy evaluations common in current (deep) RL algorithms, can lead to a dangerous overestimation of the value function. This overestimation of the value function can then lead to policy improvements with poor theoretical guarantees. To combat overestimation, the authors propose to penalize state-action pairs that are rarely visited. As an easier to implement alternative, and closer to existing algorithms in the literature, the authors also study another penalty term that penalizes deviation from the data generating policy. The authors show on a numerical example that the more principled penalty term that depends on visitation counts is better performing, and that the proximal penalty term only yields minor improvements over imitation learning (i.e. returning the data generating policy).