Behavior Regularized Offline Reinforcement Learning

1 INTRODUCTION . Offline reinforcement learning ( RL ) describes the setting in which a learner has access to only a fixed dataset of experience . In contrast to online RL , additional interactions with the environment during learning are not permitted . This setting is of particular interest for applications in which deploying a policy is costly or there is a safety concern with updating the policy online ( Li et al. , 2015 ) . For example , for recommendation systems ( Li et al. , 2011 ; Covington et al. , 2016 ) or health applications ( Murphy et al. , 2001 ) , deploying a new policy may only be done at a low frequency after extensive testing and evaluation . In these cases , the offline dataset is often very large , potentially encompassing years of logged experience . Nevertheless , the inability to interact with the environment directly poses a challenge to modern RL algorithms . Issues with RL algorithms in the offline setting typically arise in cases where state and actions spaces are large or continuous , necessitating the use of function approximation . While off-policy ( deep ) RL algorithms such as DQN ( Mnih et al. , 2013 ) , DDPG ( Lillicrap et al. , 2015 ) , and SAC ( Haarnoja et al. , 2018 ) may be run directly on offline datasets to learn a policy , the performance of these algorithms has been shown to be sensitive to the experience dataset distribution , even in the online setting when using a replay buffer ( Van Hasselt et al. , 2018 ; Fu et al. , 2019 ) . Moreover , Fujimoto et al . ( 2018a ) and Kumar et al . ( 2019 ) empirically confirm that in the offline setting , DDPG fails to learn a good policy , even when the dataset is collected by a single behavior policy , with or without noise added to the behavior policy . These failure cases are hypothesized to be caused by erroneous generalization of the state-action value function ( Q-value function ) learned with function approximators , as suggested by Sutton ( 1995 ) ; Baird ( 1995 ) ; Tsitsiklis & Van Roy ( 1997 ) ; Van Hasselt et al . ( 2018 ) . To remedy this issue , two types of approaches have been proposed recently : 1 ) Agarwal et al . ( 2019 ) proposes to apply a random ensemble of Q-value targets to stabilize the learned Q-function , 2 ) Fujimoto et al . ( 2018a ) ; Kumar et al . ( 2019 ) ; Jaques et al . ( 2019 ) ; Laroche & Trichelair ( 2017 ) propose to regularize the learned policy towards the behavior policy based on the intuition that unseen state-action pairs are more likely to receive overestimated Q-values . These proposed remedies have been shown to improve upon DQN or DDPG at performing policy improvement based on offline data . Still , each proposal makes several modifications to the building components of baseline off-policy RL algorithms , and each modification may be implemented in various ways . So a natural question to ask is , which of the design choices in these offline RL algorithms are necessary to achieve good performance ? For example , to estimate the target Q-value when minimizing the Bellman error , Fujimoto et al . ( 2018a ) uses a soft combination of two target Q-values , which is different from TD3 ( Fujimoto et al. , 2018b ) , where the minimum of two target Q-values is used . This soft combination is maintained by Kumar et al . ( 2019 ) , while further increasing the number of Q-networks from two to four . As another example , when regularizing towards the behavior policy , Jaques et al . ( 2019 ) uses Kullback-Leibler ( KL ) divergence with a fixed regularization weight while Kumar et al . ( 2019 ) proposes to use Maximum Mean Discrepancy ( MMD ) with an adaptively trained regularization weight . Are these design choices crucial to success in offline settings ? Or are they simply the result of multiple , human-directed iterations of research ? In this work , we aim at evaluating the importance of different algorithmic building components as well as comparing different design choices in offline RL approaches . We focus on behavior regularized approaches applied to continuous action domains , encompassing many of the recently demonstrated successes ( Fujimoto et al. , 2018a ; Kumar et al. , 2019 ) . We introduce behavior regularized actor critic ( BRAC ) , a general algorithmic framework which covers existing approaches while enabling us to compare the performance of different variants in a modular way . We find that many simple variants of the behavior regularized approach can yield good performance , while previously suggested sophisticated techniques such as weighted Q-ensembles and adaptive regularization weights are not crucial . Experimental ablations reveal further insights into how different design choices affect the performance and robustness of the behavior regularized approach in the offline RL setting . 2 BACKGROUND . 2.1 MARKOV DECISION PROCESSES . We consider the standard fully-observed Markov Decision Process ( MDP ) setting ( Puterman , 1990 ) . An MDP can be represented asM = ( S , A , P , R , γ ) where S is the state space , A is the action space , P ( ·|s , a ) is the transition probability distribution function , R ( s , a ) is the reward function and γ is the discount factor . The goal is to find a policy π ( ·|s ) that maximizes the cumulative discounted reward starting from any state s ∈ S. Let Pπ ( ·|s ) denote the induced transition distribution for policy π . For later convenience , we also introduce the notion of multi-step transition distributions as Pπt , where P π t ( ·|s ) denotes the distribution over the state space after rolling out Pπ for t steps starting from state s. For example , Pπ0 ( ·|s ) is the Dirac delta function at s and Pπ1 ( ·|s ) = Pπ ( ·|s ) . We use Rπ ( s ) to denote the expected reward at state s when following policy π , i.e . Rπ ( s ) = Ea∼π ( ·|s ) [ R ( s , a ) ] . The state value function ( a.k.a . value function ) is defined by V π ( s ) = ∑∞ t=0 γ tEst∼Pπt ( s ) [ R π ( st ) ] . The action-value function ( a.k.a . Q-function ) can be written as Qπ ( s , a ) = R ( s , a ) + γEs′∼P ( ·|s , a ) [ V π ( s′ ) ] . The optimal policy is defined as the policy π∗ that maximizes V π ∗ ( s ) at all states s ∈ S. In the commonly used actor critic paradigm , one optimizes a policy πθ ( ·|s ) by alternatively learning a Q-value functionQψ to minimize Bellman errors over single step transitions ( s , a , r , s′ ) , Ea′∼πθ ( ·|s′ ) [ ( r + γQ̄ ( s′ , a′ ) −Qψ ( s , a ) ) 2 ] , where Q̄ denotes a target Q function ; e.g. , it is common to use a slowly-updated target parameter set ψ′ to determine the target Q function as Qψ′ ( s′ , a′ ) . Then , the policy is updated to maximize the Q-values , Ea∼π ( ·|s ) [ Qψ ( s , a ) ] . 2.2 OFFLINE REINFORCEMENT LEARNING . Offline RL ( also known as batch RL ( Lange et al. , 2012 ) ) considers the problem of learning a policy π from a fixed dataset D consisting of single-step transitions ( s , a , r , s′ ) . Slightly abusing the notion of “ behavior ” , we define the behavior policy πb ( a|s ) as the conditional distribution p ( a|s ) observed in the dataset distributionD . Under this definition , such a behavior policy πb is always well-defined even if the dataset was collected by multiple , distinct behavior policies . Because we do not assume direct access to πb , it is common in previous work to approximate this behavior policy with max-likelihood over D : π̂b : = argmax π̂ E ( s , a , r , s′ ) ∼D [ log π̂ ( a|s ) ] . ( 1 ) We denote the learned policy as π̂b and refer to it as the “ cloned policy ” to distinguish it from the true behavior policy . In this work , we focus on the offline RL problem for complex continuous domains . We briefly review two recently proposed approaches , BEAR ( Kumar et al. , 2019 ) and BCQ ( Fujimoto et al. , 2018a ) . BEAR Motivated by the hypothesis that deep RL algorithms generalize poorly to actions outside the support of the behavior policy , Kumar et al . ( 2019 ) propose BEAR , which learns a policy to maximize Q-values while penalizing it from diverging from behavior policy support . BEAR measures divergence from the behavior policy using kernel MMD ( Gretton et al. , 2007 ) : MMD2k ( π ( ·|s ) , πb ( ·|s ) ) = E x , x′∼π ( ·|s ) [ K ( x , x′ ) ] − 2E x∼π ( ·|s ) y∼πb ( ·|s ) [ K ( x , y ) ] + E y , y′∼πb ( ·|s ) [ K ( y , y′ ) ] , ( 2 ) whereK is a kernel function . Furthermore , to avoid overestimation in the Q-values , the target Q-value function Q̄ is calculated as , Q̄ ( s′ , a′ ) : = 0.75 · min j=1 , ... , k Qψ′j ( s ′ , a′ ) + 0.25 · max j=1 , ... , k Qψ′j ( s ′ , a′ ) , ( 3 ) where ψ′j is denotes a soft-updated ensemble of target Q functions . In BEAR ’ s implementation , this ensemble is of size k = 4 . BEAR also penalizes target Q-values by an ensemble variance term . However , their empirical results show that there is no clear benefit to doing so , thus we omit this term . BCQ BCQ enforces π to be close to πb with a specific parameterization of π : πθ ( a|s ) : = argmax ai+ξθ ( s , ai ) Qψ ( s , ai + ξθ ( s , ai ) ) for ai ∼ πb ( a|s ) , i = 1 , . . . , N , ( 4 ) where ξθ is a function approximator with bounded ouptput in [ −Φ , Φ ] where Φ is a hyperparameter . N is an additional hyperparameter used during evaluation to compute πθ and during training for Q-value updates . The target Q-value function Q̄ is calculated as in Equation 3 but with k = 2 . 3 BEHAVIOR REGULARIZED ACTOR CRITIC . Encouraging the learned policy to be close to the behavior policy is a common theme in previous approaches to offline RL . To evaluate the effect of different behavior policy regularizers , we introduce behavior regularized actor critic ( BRAC ) , an algorithmic framework which generalizes existing approaches while providing more implementation options . There are two common ways to incorporate regularization to a specific policy : through a penalty in the value function or as a penalty solely on the policy . We begin by introducing the former , value penalty ( vp ) . Similar to SAC ( Haarnoja et al. , 2018 ) which adds an entropy term to the target Q-value calculation , we add a term to the target Q-value calculation that regularizes the learned policy π towards the behavior policy πb . Specifically , we define the penalized value function as V πD ( s ) = ∑∞ t=0 γtEst∼Pπt ( s ) [ R π ( st ) − αD ( π ( ·|st ) , πb ( ·|st ) ) ] , ( 5 ) where D is a divergence function between distributions over actions ( e.g. , MMD or KL divergence ) . Following the typical actor critic framework , the Q-value objective is given by , min Qψ E ( s , a , r , s′ ) ∼D a′∼πθ ( ·|s′ ) [ ( r + γ ( Q̄ ( s′ , a′ ) − αD̂ ( πθ ( ·|s′ ) , πb ( ·|s′ ) ) ) −Qψ ( s , a ) ) 2 ] , ( 6 ) where Q̄ again denotes a target Q function and D̂ denotes a sample-based estimate of the divergence function D. The policy learning objective can be written as , max πθ E ( s , a , r , s′ ) ∼D [ Ea′′∼πθ ( ·|s ) [ Qψ ( s , a ′′ ) ] − αD̂ ( πθ ( ·|s ) , πb ( ·|s ) ) ] . ( 7 ) Accordingly , one performs alternating gradient updates based on ( 6 ) and ( 7 ) . This algorithm is equivalent to SAC when using a single-sample estimate of the entropy for D̂ ; i.e. , D̂ ( πθ ( ·|s′ ) , πb ( ·|s′ ) ) : = log π ( a′|s′ ) for a′ ∼ π ( ·|s′ ) . The second way to add the regularizer is to only regularize the policy during policy optimization . That is , we use the same objectives in Equations 6 and 7 , but use α = 0 in the Q update while using a non-zero α in the policy update . We call this variant policy regularization ( pr ) . This proposal is similar to the regularization employed in A3C ( Mnih et al. , 2016 ) , if one uses the entropy of πθ to compute D̂ . In addition to the choice of value penalty or policy regularization , the choice of D and how to perform sample estimation of D̂ is a key design choice of BRAC : Kernel MMD We can compute a sample based estimate of kernel MMD ( Equation 2 ) by drawing samples from both πθ and πb . Because we do not have access to multiple samples from πb , this requires a pre-estimated cloned policy π̂b . KL Divergence With KL Divergence , the behavior regularizer can be written as DKL ( πθ ( ·|s ) , πb ( ·|s ) ) = Ea∼πθ ( ·|s ) [ log πθ ( a|s ) − log πb ( a|s ) ] . Directly estimating DKL via samples requires having access to the density of both πθ and πb ; as in MMD , the cloned π̂b can be used in place of πb . Alternatively , we can avoid estimating πb explicitly , by using the dual form of the KL-divergence . Specifically , any f -divergence ( Csiszár , 1964 ) has a dual form ( Nowozin et al. , 2016 ) given by , Df ( p , q ) = Ex∼p [ f ( q ( x ) /p ( x ) ) ] = max g : X 7→dom ( f∗ ) Ex∼q [ g ( x ) ] − Ex∼p [ f∗ ( g ( x ) ) ] , where f∗ is the Fenchel dual of f . In this case , one no longer needs to estimate a cloned policy π̂b but instead needs to learn a discriminator function g with minimax optimization as in Nowozin et al . ( 2016 ) . This sample based dual estimation can be applied to any f -divergence . In the case of a KL-divergence , f ( x ) = − log x and f∗ ( t ) = − log ( −t ) − 1 . Wasserstein Distance One may also use the Wassertein distance as the divergence D. For samplebased estimation , one may use its dual form , W ( p , q ) = sup g : ||g||L≤1 Ex∼p [ g ( x ) ] − Ex∼q [ g ( x ) ] and maintain a discriminator g as in Gulrajani et al . ( 2017 ) . Now we discuss how existing approaches can be instantiated under the framework of BRAC . BEAR To re-create BEAR with BRAC , one uses policy regularization with the sample-based kernel MMD for D̂ and uses a min-max ensemble estimate for Q̄ ( Equation 3 ) . Furthermore , BEAR adaptively trains the regularization weight α as a Lagriagian multiplier : it sets a threshold > 0 for the kernel MMD distance and increases α if the current average divergence is above the threshold and decreases α if below the threshold . BCQ The BCQ algorithm does not use any regularizers ( i.e . α = 0 for both value and policy objectives ) . Still , the algorithm may be realized by BRAC if one restricts the policy optimization in Equation 7 to be over parameterized policies based on Equation 4 . KL-Control There has been a rich set of work which investigates regularizing the learned policy through KL-divergence with respect to another policy , e.g . Abdolmaleki et al . ( 2018 ) ; Kakade ( 2002 ) ; Peters et al . ( 2010 ) ; Schulman et al . ( 2015 ) ; Nachum et al . ( 2017 ) . Notably , Jaques et al . ( 2019 ) apply this idea to offline RL in discrete action domains by introducing a KL value penalty in the Q-value definition . It is clear that BRAC can realize this algorithm as well . To summarize , one can instantiate the behavior regularized actor critic framework with different design choices , including how to estimate the target Q value , which divergence to use , whether to learn α adaptively , whether to use a value penalty in the Q function objective ( 6 ) or just use policy regularization in ( 7 ) and so on . In the next section , we empirically evaluate a set of these different design choices to provide insights into what actually matters when approaching the offline RL problem .

Summary: This paper proposes a unifying framework, BRAC, which summarizes the idea and evaluates the effectiveness of recently proposed offline reinforcement learning algorithms, specifically BEAR, BCQ, and KL control. The authors generalize existing offline RL approaches to an actor-critic algorithm that regularizes the learned policy such that it can stay close to the behavior policy. Based on prior work, the authors state that there are two variants of regularizations to the behavior policy, value penalty (vp) and policy regularization (pr) and three choices of divergence functions along with their sample estimate that measures the distance between the learned policy and the behavior policy. The paper empirically investigates the effectiveness of each regularization scheme as well as each divergence function and conclude that vp is slightly more effective than pr while all divergence functions have similar performances.