Addressing Extrapolation Error in Deep Offline Reinforcement Learning

1 INTRODUCTION . Agents are , fundamentally , entities which map observations to actions and can be trained with reinforcement learning ( RL ) in either an online or offline fashion . When trained online , an agent learns through trial and error by interacting with its environment . Online RL has had considerable success recently : on Atari ( Mnih et al. , 2015 ) , the game of GO ( Silver et al. , 2017 ) , video games like StarCraft II , and Dota 2 , ( Vinyals et al. , 2019 ; Berner et al. , 2019 ) , and robotics ( Andrychowicz et al. , 2020 ) . However , the requirement of extensive environmental interaction combined with a need for exploratory behavior makes these algorithms unsuitable and potentially unsafe for many real world applications . In contrast , in the offline setting ( Fu et al. , 2020 ; Fujimoto et al. , 2018 ; Gulcehre et al. , 2020 ; Levine et al. , 2020 ) , also known as batch RL ( Ernst et al. , 2005 ; Lange et al. , 2012 ) , agents learn from a fixed dataset which is assumed to have been logged by other ( possibly unknown ) agents . See also Fig . 1 for an illustration of these two settings . Learning purely from logged data allows these algorithms to be more widely applicable , including in problems such as healthcare and self-driving cars , where repeated interaction with the environment is costly and potentially unsafe or unethical , and where logged historical data is abundant . However these algorithms tend to behave considerably worse than their online counterpart . Although similar in principle , there are some important differences between the two regimes . While it is useful for online agents to explore unknown regions of the state space so as to gain knowledge about the environment and better their chances of finding a good policy ( Schmidhuber , 1991 ) , this is not the case for the offline setting . Choosing actions not well-represented in the dataset for offline methods would force the agent to rely on function approximators ’ extrapolation ability . This can lead to substantial errors during training , as well as during deployment of the agent . During training , the extrapolation errors are exacerbated by bootstrapping and the use of max operators ( e.g . in Q-learning ) where evaluating the loss entails taking the maximum over noisy and possibly overestimated values of the different possible actions . This can result in a propagation of the erroneous values , leading to extreme over-estimation of the value function and potentially unbounded error ; see ( Fujimoto et al. , 2019b ) and our remark in Appendix A . As we empirically show in Section 4.2 , extrapolation errors are a different source of overestimation compared to those considered by standard methods such as Double DQN ( Hasselt , 2010 ) , and hence can not be addressed by those approaches . In addition to extrapolation errors during training , a further degradation in performance can result from the use of greedy policies at test time which maximize over value estimates extrapolated to under-represented actions . We propose a coherent set of techniques that work well together to combat extrapolation error and overestimation : Behavior value estimation . First , we address extrapolation errors during training time . Instead of Qπ ∗ , we estimate the value of the behavioral policy QπB , thereby avoid the max-operator during training . To improve upon the behavioral policy , we conduct what amounts to a single step of policy improvement by employing a greedy policy at test time . Surprisingly , this technique with only one round of improvement allows us to perform significantly better than the behavioral policies and often outperform existing offline RL algorithms . Ranking regularization . We introduce a max-margin based regularizer that encourages the value function , represented as a deep neural network , to rank actions present in the observed rewarding episodes higher than any other actions . Intuitively , this regularizer pushes down the value of all unobserved state-action pairs , thereby minimizing the chance of a greedy policy selecting actions under-represented in the dataset . Employing the regularizer during training will minimize the impact of the max-operator used by the greedy policy at test time , i.e . this approach addresses extrapolation errors both at training and ( indirectly ) at test time . Reparametrization of Q-values . While behavior value estimation typically performs well , particularly when combined with ranking regularization , it only allows for one iteration of policy improvement . When more data is available , and hence we can trust our function approximator to capture more of the structure of the state space and as a result generalize better , we can rely on Q-learning which permits multiple policy improvement iterations . However this exacerbates the overestimation issue . We propose , in addition to the ranking loss , a simple reparametrization of the value function to disentangle the scale from the relative ranks of the actions . This reparametrization allows us to introduce a regularization term on the scale of the value function alone , which reduces over-estimation . To evaluate our proposed method , we introduce new datasets based on bsuite environments ( Osband et al. , 2019 ) , as well as the partially observable DeepMind Lab environments ( Beattie et al. , 2016 ) . We further evaluate our method as well as baselines on the RL Unplugged ( RLU ) Atari dataset ( Gulcehre et al. , 2020 ) . We achieve a new state of the art ( SOTA ) performance on the RLU Atari dataset as well as outperform existing SOTA offline RL methods on our newly introduced datasets . Last but not least , we provide careful ablations and analyses that provide insights into our proposed method as well as other existing offline RL algorithms . Related work . Early examples of offline/batch RL include least-squares temporal difference methods ( Bradtke & Barto , 1996 ; Lagoudakis & Parr , 2003 ) and fitted Q iteration ( Ernst et al. , 2005 ; Riedmiller , 2005 ) . Recently , Agarwal et al . ( 2019a ) , Fujimoto et al . ( 2019b ) , Kumar et al . ( 2019 ) , Siegel et al . ( 2020 ) , Wang et al . ( 2020 ) and Ghasemipour et al . ( 2020 ) have proposed offline-RL algorithms and shown that they outperform off-the-shelf off-policy RL methods . There also exist methods explicitly addressing the issues stemming from extrapolation error ( Fujimoto et al. , 2019b ) . 2 BACKGROUND AND PROBLEM STATEMENT . We consider , in this work , Markov Decision Processes ( MDPs ) defined by ( S , A , P , R , ρ0 , γ ) where S is the set of all possible states and A all possible actions . An agent starts in some state s0 ∼ ρ0 ( · ) where ρ0 ( · ) is a distribution over S and takes actions according to its policy a ∼ π ( ·|s ) , a ∈ A , when in state s. Then it observes a new state s′ and reward r according to the transition distribution P ( s′|s , a ) and reward function r ( s , a ) . The state action value function Qπ describes the expected discounted return starting from state s and action a and following π afterwards : Qπ ( s , a ) = E [ ∑ t=0 γtr ( st , at ) ] , s0 = s , a0 = a , st ∼ P ( ·|st−1 , at−1 ) , at ∼ π ( ·|st ) , ( 1 ) and V π ( s ) = Ea∼π ( ·|s ) Qπ ( s , a ) is the state value function . The optimal policy π∗ , which we aim to discover through RL , is one that maximizes the expected cumulative discounted rewards , or expected returns such that Qπ ∗ ( s , a ) ≥ Qπ ( s , a ) ∀s , a , π . For notational simplicity , we denote the policy used to generate an offline dataset as πB1 . In the same vein , for a state s in an offline dataset , we write GB ( s ) to denote an empirical estimate of V πB ( s ) , computed by summing future discounted rewards over the trajectory that s is part of . Approaches to RL can be broadly categorized as either on-policy or off-policy algorithms . Whereas on-policy algorithms update their current policy based on data generated by that same policy , offpolicy approaches can take advantage of data generated by other policies . Algorithms in the mold of fitted Q-iteration make up many of the most popular approaches to deep off-policy RL ( Mnih et al. , 2015 ; Lillicrap et al. , 2015 ; Haarnoja et al. , 2018 ) . This class of algorithms learns a Q function by minimizing the Temporal Difference ( TD ) error . To increase stability and sample efficiency , the use of experience replay is also typically employed . For example , DQN ( Mnih et al. , 2015 ) minimizes the following loss function : Lθ′ ( θ ) = E ( s , a , r , s′ ) ∼D ( Qθ ( s , a ) − ( r + γmax a′ Qθ′ ( s ′ , a′ ) ) ) 2 , ( 2 ) where D represents experience replay , i.e . a dataset generated by some behavior policy . Typically , for off-policy algorithms the behavior policy is periodically updated to remain close to the policy being optimized . A deterministic policy can be derived by being greedy with respect to Q̂ , i.e . by defining π ( s ) = arg maxaQ ( s , a ) . In cases where maximization is nontrivial ( e.g . continuous action spaces ) , we typically adopt a separate policy π and optimize losses similar to : Lθ′ ( θ ) = E ( s , a , r , s′ ) ∼D ( Qθ ( s , a ) − ( r+ γEa′∼π ( ·|s′ ) [ Qθ′ ( s′ , a′ ) ] ) ) 2 . In this case , π is optimized separately in order to maximize Ea∼π ( ·|s ) [ Q ( s , a ) ] , sometimes subject to other constraints ( Lillicrap et al. , 2015 ; Haarnoja et al. , 2018 ) . Various extensions have been proposed for this class of algorithms , including but not limited to : distributional critics ( Bellemare et al. , 2017 ) , prioritized replays ( Schaul et al. , 2015 ) , and n-step returns ( Kapturowski et al. , 2019 ; Barth-Maron et al. , 2018 ; Hessel et al. , 2017 ) . In the offline RL setting ( see Figure 1 , right ) , agents learn from fixed datasets generated via other processes , thus rendering off-policy RL algorithms particularly pertinent . Many existing offline RL algorithms adopt variants of Equation ( 2 ) to learn value functions ; e.g . Agarwal et al . ( 2019b ) . Offline RL , however , is different from off-policy learning in the online setting . The dataset used is finite and fixed , and does not track the policy being learned . When a policy moves towards a part of the state space not covered by the behavior policy ( s ) , for example , one can not effectively learn the value function . We will explore this in more detail in the next subsection .

Summary: This paper focuses on the problem of Q value over-estimation in offline reinforcement learning and proposes three approaches (tricks) to help solve this problem. (1) estimate Q value of behavior policy avoiding max-operator in Q learning and take greedy action according to the behavior value estimation. (2) introduce ranking loss to push down the value estimation of all unobserved state-action pairs to avoid over-estimation. (3) use tanh operator to bound the range of Q value estimation, and learn a scale parameter with regularization term. The experimental results on several domains (Atari, Bsuite, Deepmind Lab) with discrete action space show performance better than existing algorithms.