Addressing Extrapolation Error in Deep Offline Reinforcement Learning

1 INTRODUCTION . Agents are , fundamentally , entities which map observations to actions and can be trained with reinforcement learning ( RL ) in either an online or offline fashion . When trained online , an agent learns through trial and error by interacting with its environment . Online RL has had considerable success recently : on Atari ( Mnih et al. , 2015 ) , the game of GO ( Silver et al. , 2017 ) , video games like StarCraft II , and Dota 2 , ( Vinyals et al. , 2019 ; Berner et al. , 2019 ) , and robotics ( Andrychowicz et al. , 2020 ) . However , the requirement of extensive environmental interaction combined with a need for exploratory behavior makes these algorithms unsuitable and potentially unsafe for many real world applications . In contrast , in the offline setting ( Fu et al. , 2020 ; Fujimoto et al. , 2018 ; Gulcehre et al. , 2020 ; Levine et al. , 2020 ) , also known as batch RL ( Ernst et al. , 2005 ; Lange et al. , 2012 ) , agents learn from a fixed dataset which is assumed to have been logged by other ( possibly unknown ) agents . See also Fig . 1 for an illustration of these two settings . Learning purely from logged data allows these algorithms to be more widely applicable , including in problems such as healthcare and self-driving cars , where repeated interaction with the environment is costly and potentially unsafe or unethical , and where logged historical data is abundant . However these algorithms tend to behave considerably worse than their online counterpart . Although similar in principle , there are some important differences between the two regimes . While it is useful for online agents to explore unknown regions of the state space so as to gain knowledge about the environment and better their chances of finding a good policy ( Schmidhuber , 1991 ) , this is not the case for the offline setting . Choosing actions not well-represented in the dataset for offline methods would force the agent to rely on function approximators ’ extrapolation ability . This can lead to substantial errors during training , as well as during deployment of the agent . During training , the extrapolation errors are exacerbated by bootstrapping and the use of max operators ( e.g . in Q-learning ) where evaluating the loss entails taking the maximum over noisy and possibly overestimated values of the different possible actions . This can result in a propagation of the erroneous values , leading to extreme over-estimation of the value function and potentially unbounded error ; see ( Fujimoto et al. , 2019b ) and our remark in Appendix A . As we empirically show in Section 4.2 , extrapolation errors are a different source of overestimation compared to those considered by standard methods such as Double DQN ( Hasselt , 2010 ) , and hence can not be addressed by those approaches . In addition to extrapolation errors during training , a further degradation in performance can result from the use of greedy policies at test time which maximize over value estimates extrapolated to under-represented actions . We propose a coherent set of techniques that work well together to combat extrapolation error and overestimation : Behavior value estimation . First , we address extrapolation errors during training time . Instead of Qπ ∗ , we estimate the value of the behavioral policy QπB , thereby avoid the max-operator during training . To improve upon the behavioral policy , we conduct what amounts to a single step of policy improvement by employing a greedy policy at test time . Surprisingly , this technique with only one round of improvement allows us to perform significantly better than the behavioral policies and often outperform existing offline RL algorithms . Ranking regularization . We introduce a max-margin based regularizer that encourages the value function , represented as a deep neural network , to rank actions present in the observed rewarding episodes higher than any other actions . Intuitively , this regularizer pushes down the value of all unobserved state-action pairs , thereby minimizing the chance of a greedy policy selecting actions under-represented in the dataset . Employing the regularizer during training will minimize the impact of the max-operator used by the greedy policy at test time , i.e . this approach addresses extrapolation errors both at training and ( indirectly ) at test time . Reparametrization of Q-values . While behavior value estimation typically performs well , particularly when combined with ranking regularization , it only allows for one iteration of policy improvement . When more data is available , and hence we can trust our function approximator to capture more of the structure of the state space and as a result generalize better , we can rely on Q-learning which permits multiple policy improvement iterations . However this exacerbates the overestimation issue . We propose , in addition to the ranking loss , a simple reparametrization of the value function to disentangle the scale from the relative ranks of the actions . This reparametrization allows us to introduce a regularization term on the scale of the value function alone , which reduces over-estimation . To evaluate our proposed method , we introduce new datasets based on bsuite environments ( Osband et al. , 2019 ) , as well as the partially observable DeepMind Lab environments ( Beattie et al. , 2016 ) . We further evaluate our method as well as baselines on the RL Unplugged ( RLU ) Atari dataset ( Gulcehre et al. , 2020 ) . We achieve a new state of the art ( SOTA ) performance on the RLU Atari dataset as well as outperform existing SOTA offline RL methods on our newly introduced datasets . Last but not least , we provide careful ablations and analyses that provide insights into our proposed method as well as other existing offline RL algorithms . Related work . Early examples of offline/batch RL include least-squares temporal difference methods ( Bradtke & Barto , 1996 ; Lagoudakis & Parr , 2003 ) and fitted Q iteration ( Ernst et al. , 2005 ; Riedmiller , 2005 ) . Recently , Agarwal et al . ( 2019a ) , Fujimoto et al . ( 2019b ) , Kumar et al . ( 2019 ) , Siegel et al . ( 2020 ) , Wang et al . ( 2020 ) and Ghasemipour et al . ( 2020 ) have proposed offline-RL algorithms and shown that they outperform off-the-shelf off-policy RL methods . There also exist methods explicitly addressing the issues stemming from extrapolation error ( Fujimoto et al. , 2019b ) . 2 BACKGROUND AND PROBLEM STATEMENT . We consider , in this work , Markov Decision Processes ( MDPs ) defined by ( S , A , P , R , ρ0 , γ ) where S is the set of all possible states and A all possible actions . An agent starts in some state s0 ∼ ρ0 ( · ) where ρ0 ( · ) is a distribution over S and takes actions according to its policy a ∼ π ( ·|s ) , a ∈ A , when in state s. Then it observes a new state s′ and reward r according to the transition distribution P ( s′|s , a ) and reward function r ( s , a ) . The state action value function Qπ describes the expected discounted return starting from state s and action a and following π afterwards : Qπ ( s , a ) = E [ ∑ t=0 γtr ( st , at ) ] , s0 = s , a0 = a , st ∼ P ( ·|st−1 , at−1 ) , at ∼ π ( ·|st ) , ( 1 ) and V π ( s ) = Ea∼π ( ·|s ) Qπ ( s , a ) is the state value function . The optimal policy π∗ , which we aim to discover through RL , is one that maximizes the expected cumulative discounted rewards , or expected returns such that Qπ ∗ ( s , a ) ≥ Qπ ( s , a ) ∀s , a , π . For notational simplicity , we denote the policy used to generate an offline dataset as πB1 . In the same vein , for a state s in an offline dataset , we write GB ( s ) to denote an empirical estimate of V πB ( s ) , computed by summing future discounted rewards over the trajectory that s is part of . Approaches to RL can be broadly categorized as either on-policy or off-policy algorithms . Whereas on-policy algorithms update their current policy based on data generated by that same policy , offpolicy approaches can take advantage of data generated by other policies . Algorithms in the mold of fitted Q-iteration make up many of the most popular approaches to deep off-policy RL ( Mnih et al. , 2015 ; Lillicrap et al. , 2015 ; Haarnoja et al. , 2018 ) . This class of algorithms learns a Q function by minimizing the Temporal Difference ( TD ) error . To increase stability and sample efficiency , the use of experience replay is also typically employed . For example , DQN ( Mnih et al. , 2015 ) minimizes the following loss function : Lθ′ ( θ ) = E ( s , a , r , s′ ) ∼D ( Qθ ( s , a ) − ( r + γmax a′ Qθ′ ( s ′ , a′ ) ) ) 2 , ( 2 ) where D represents experience replay , i.e . a dataset generated by some behavior policy . Typically , for off-policy algorithms the behavior policy is periodically updated to remain close to the policy being optimized . A deterministic policy can be derived by being greedy with respect to Q̂ , i.e . by defining π ( s ) = arg maxaQ ( s , a ) . In cases where maximization is nontrivial ( e.g . continuous action spaces ) , we typically adopt a separate policy π and optimize losses similar to : Lθ′ ( θ ) = E ( s , a , r , s′ ) ∼D ( Qθ ( s , a ) − ( r+ γEa′∼π ( ·|s′ ) [ Qθ′ ( s′ , a′ ) ] ) ) 2 . In this case , π is optimized separately in order to maximize Ea∼π ( ·|s ) [ Q ( s , a ) ] , sometimes subject to other constraints ( Lillicrap et al. , 2015 ; Haarnoja et al. , 2018 ) . Various extensions have been proposed for this class of algorithms , including but not limited to : distributional critics ( Bellemare et al. , 2017 ) , prioritized replays ( Schaul et al. , 2015 ) , and n-step returns ( Kapturowski et al. , 2019 ; Barth-Maron et al. , 2018 ; Hessel et al. , 2017 ) . In the offline RL setting ( see Figure 1 , right ) , agents learn from fixed datasets generated via other processes , thus rendering off-policy RL algorithms particularly pertinent . Many existing offline RL algorithms adopt variants of Equation ( 2 ) to learn value functions ; e.g . Agarwal et al . ( 2019b ) . Offline RL , however , is different from off-policy learning in the online setting . The dataset used is finite and fixed , and does not track the policy being learned . When a policy moves towards a part of the state space not covered by the behavior policy ( s ) , for example , one can not effectively learn the value function . We will explore this in more detail in the next subsection .

Summary: The paper deals with offline aka batch RL for discrete actions. Three techniques ((i) behavior value estimation, (ii) ranking regularization, and (iii) reparametrization of the value function), which can be combined with each other, are presented. These techniques are compared with other methods in different experiments. Furthermore a new benchmark is being introduced.  It is claimed that in this new benchmark, the new techniques outperform state-of-the-art methods. Furthermore it is claimed that the presented method „behavior value estimation“, although it is only a one-step greedy optimization is typically already sufficient for dramatic gains.