Why Should I Trust You, Bellman? Evaluating the Bellman Objective with Off-Policy Data

1 INTRODUCTION . In reinforcement learning ( RL ) , value functions are a measure of performance of a target policy . Value functions are an important quantity in RL as they can be used to inform decision-making . Consequently , many modern reinforcement learning algorithms rely on a value function in some capacity ( Gu et al. , 2016 ; Schulman et al. , 2017 ; Fujimoto et al. , 2018 ; Badia et al. , 2020 ) . The Bellman equation is a fundamental relationship in RL which relates the value of a state-action pair to the state-action pair that follows , and is uniquely satisfied over all state-action pairs by the true value function . The existence of the Bellman equation suggests a straightforward approach for approximate value function learning , where a function is trained to minimize the Bellman error ( the difference of both sides of the equation ) . The Bellman equation has played a prominent role in many historically significant approaches ( Schweitzer & Seidmann , 1985 ; Baird , 1995 ; Bradtke & Barto , 1996 ; Antos et al. , 2008 ; Sutton et al. , 2009 ) , as well as the more modern family of deep RL algorithms ( Mnih et al. , 2015 ; Lillicrap et al. , 2015 ; Gu et al. , 2016 ; Hessel et al. , 2017 ) . In this work , we examine the relationship between the Bellman equation and the accuracy of value functions . We do so through off-policy evaluation ( OPE ) , which presents the task of learning the value function of a target policy with data gathered from a separate and possibly unknown behavior policy . OPE , which is a subcomponent of virtually any off-policy RL algorithm , is an ideal setting for evaluating value functions as it provides a clear metric of performance ( value prediction error ) and provides consistency across trials ( fixed dataset and target policy ) . Our main thesis is that since the Bellman equation is meant to consider the entire MDP and all possible state-action pairs , when it is instead estimated over a finite dataset , there is likely to be some breakdown in its relationship to value prediction . This work aims to better understand that breakdown through theoretical analysis and empirical study . Our key discoveries are under offpolicy , function approximations , and finite data assumptions : Bellman error is a poor metric for value error . We find that given two arbitrary value functions , comparing their Bellman error is insufficient to determine which value function is more accurate . This problem is highlighted by experiments which show that value functions trained to minimize Bellman error directly ( Baird , 1995 ) have lower Bellman error but higher value error , than value functions trained by iterative methods ( Ernst et al. , 2005 ) . We find that this non-correspondence in relative ordering over error terms holds even when evaluated over on-policy data ( Figure 2 ) , and only worsens further with off-policy datasets ( Figure 3 ) . Bellman error is a poor objective for learning off-policy . A natural consequence of the Bellman error being a weak metric for value error , is that the Bellman error makes for a poor off-policy objective . Our experiments show that value functions trained by different algorithms exhibit different behaviors . As such , Bellman error can not be used as a metric for arbitrary value functions . However , we find that when comparing value functions trained by the same algorithm , Bellman error can be used as an accurate measure for value error , but only if the error terms are evaluated with on-policy data ( Table 1 ) . This means that Bellman error is only a meaningful objective when used on-policy . Iterative methods rely on generalization for successful training . Iterative methods , such as many deep RL algorithms ( Mnih et al. , 2015 ; Lillicrap et al. , 2015 ) , use a slightly different objective than Bellman error , where the target is assumed to be fixed . This means the objective is non-stationary and evolves during learning . Similar to the Bellman error objective , we find that examining the FQE loss alone is insufficient to determine the accuracy of the value function . However , we remark that we can compare two functions if we take the loss with respect to the same fixed target . This means that if the frozen target is accurate , then the distance to that target is a good proxy for value error . This exposes the reliance of iterative methods to generalization which occurs during training . Our work highlights problems with using Bellman error as a signal , or objective , in the off-policy setting , and aims to provide practitioners a better understanding of Bellman equation-based loss functions , the role of generalization in RL , and the learning dynamics of value functions . Our findings point to an underappreciation of the importance of finite data in widely used objectives and we encourage the community to place a higher emphasis on practical settings . 2 BACKGROUND . Reinforcement learning ( RL ) is an optimization framework for tasks of sequential nature ( Sutton & Barto , 1998 ) . Typically , tasks are defined as a Markov decision process ( S , A , R , p , d0 , γ ) , with state space S , action spaceA , reward functionR , transition dynamics p , initial state distribution d0 , and discount factor γ ∈ [ 0 , 1 ) . Actions are selected according to a policy π . The performance of a policy is measured by its discounted return Eπ [ ∑∞ t γ tr ( st , at ) ] . Offpolicy evaluation ( OPE ) is the task of approximating the value function Qπ ( s , a ) = Eπ [ ∑∞ t γ tr ( st , at ) |s0 = s , a0 = a ] of a target policy , given samples from an arbitrary dataset . A fundamental relationship regarding value functions is the Bellman equation ( Bellman , 1957 ) : Qπ ( s , a ) = Er , s′∼p , a′∼π [ r + γQπ ( s′ , a′ ) ] , ( 1 ) which relates the value of the current state-action pair to an expectation over the next state-action pair . Given an approximate value function Q ( distinguished from the true value function Qπ by dropping the π superscript ) of a target policy π , we denote the Bellman error ( s , a ) : ( s , a ) : = Q ( s , a ) − Er , s′∼p , a′∼π [ r + γQ ( s′ , a′ ) ] . ( 2 ) In policy evaluation , the main objective of interest is value error of a state-action pair ∆ ( s , a ) : ∆ ( s , a ) : = Q ( s , a ) −Qπ ( s , a ) , ( 3 ) where Qπ , the true value function , is intractable without access to the underlying MDP . A standard result is if the Bellman equation converges to the fixed point then the value function must be the true value function . We can re-frame this result in terms of Bellman errors and value errors . Proposition 1 If the Bellman error ( s , a ) = 0 for all state-action pairs ( s , a ) ∈ S × A , then the value error ∆ ( s , a ) = 0 for all state-action pairs ( s , a ) ∈ S ×A . In instances where we can not compute the Bellman error exactly , such as from samples in a nondeterministic environment , we can instead use temporal difference ( TD ) learning , where the TD error δ ( i ) is a sample-based approximation to Bellman error which can be computed over a transition i : = ( s , a , r , s′ ) , δ ( i ) : = Q ( s , a ) − ( r + γQ ( s′ , a′ ) ) , where a′ is sampled from the policy π . Note that the expected TD error is simply the Bellman error ( s , a ) = Er , s′ , a′ [ δ ( i ) ] , where the two values are identical if the environment and policy are deterministic . In this work we focus on two algorithms based on the Bellman equation , which will update an approximate value function Q , using samples from a finite dataset D. Bellman residual minimization ( BRM ) ( Baird , 1995 ) directly minimizes the Bellman error over samples from the dataset D : LBRM ( Q ) : = 1 |D| ∑ ( s , a , r , s′ ) ∼D , a′∼π ( Q ( s , a ) − ( r + γQ ( s′ , a′ ) ) ) 2 . ( 4 ) Fitted Q-Evaluation ( FQE ) ( Ernst et al. , 2005 ; Le et al. , 2019 ) is an iterative method for minimizing Bellman error : LFQE ( Q ) : = 1 |D| ∑ ( s , a , r , s′ ) ∼D , a′∼π ( Q ( s , a ) − ( r + γQ̄ ( s′ , a′ ) ) ) 2 . ( 5 ) The key distinction between the two algorithms is that BRM directly updates both Q ( s , a ) and Q ( s′ , a′ ) , while FQE only considers Q ( s , a ) . This is because FQE uses Q̄ ( s′ , a′ ) , a target value function which is updated Q̄← Q after a fixed number of time steps ( possibly including every time step ) , meaning that only the left side of the Bellman equation is directly updated . 3 EXPERIMENTAL DESIGN . Our goal is to thoroughly evaluate the relationship between Bellman error ( a measurable proxy ) and value error ( an unmeasurable true objective ) in the case of off-policy evaluation with finite samples . In this section we outline the experimental choices used in our empirical evaluation . Comprehensive experimental details ( i.e . hyperparameters , architecture , etc . ) can be found in the Appendix E. Setting . We consider the setting of off-policy evaluation ( OPE ) , as it allows to directly compare value functions over a clear metric , value error . Our experiments consider a variety of continuousaction tasks through the MuJoCo simulator ( Todorov et al. , 2012 ; Brockman et al. , 2016 ) , as it is deterministic and high-dimensional . Determinism in the dynamics is desirable as it , alongside a deterministic policy , makes the Bellman error and TD error identical . This allows us to compute the Bellman error exactly and ignore the double sampling issue for residual gradient methods ( Baird , 1995 ) . Value functions are trained to evaluate an expert deterministic target policy from a fully trained TD3 agent ( Fujimoto et al. , 2018 ) , using a standard discount factor γ = 0.99 . Algorithms . Our experiments are based on Bellman residual minimization ( BRM ) ( Baird , 1995 ) and Fitted Q-Evaluation ( FQE ) ( Ernst et al. , 2005 ; Le et al. , 2019 ) . We use these algorithms due to their popularity in the literature , and to highlight differences in methods which minimize Bellman error directly or indirectly . Network architecture and hyperparameters are the same between algorithms and are selected to match state-of-the-art deep RL methods ( Fujimoto et al. , 2018 ; Haarnoja et al. , 2018a ) for the MuJoCo domain . FQE is implemented using a target network updated with Polyak averaging . In every experiment , algorithms are trained for 1 million time steps and 10 seeds . Training Datasets . Each dataset is collected by using noisy versions of the target policy . This allows us to rank the distribution shift of each dataset . Each noise level corresponds to both the probability of selecting a uniformly random action , as well as the standard deviation of Gaussian noise added to the actions ( noting that actions are in the range [ −1 , 1 ] ) . We use uniformly random actions to ensure that not all actions are centered around the target policy , and Gaussian noise to ensure that every action is distinct from actions selected by the target policy . Metrics . We use the mean squared Bellman error , as it is the most common objective on the Bellman error ( Baird , 1995 ; Sutton & Barto , 1998 ) . For better interpretability , we use the absolute value error , normalized by dividing by a constant term equal to the average true value function Qπ sampled onpolicy . As an example , this means that 0.1 value error roughly corresponds to a percent difference of 10 % . Some experiments are repeated in Appendix D with variations of these metrics .

Summary: This paper studies the Bellman equation commonly used in reinforcement learning (RL) algorithms. The typical motivation behind using the Bellman equation to design RL objectives is that uniformly driving the Bellman error to 0 implies that the true value function has been learned, i.e., the value error is 0. However, as this paper demonstrates both theoretically and empirically, a number of issues may arise before this point in practical regimes. For example, Bellman error and value error may not be well correlated, as shown via experiments evaluating minimizing the Bellman error directly versus fitted Q-evaluation. Furthermore, when evaluated using off-policy data, Bellman error may not even be well correlated with value error for different runs of the same algorithm. These findings may be useful for a more complete understanding of when and why RL algorithms break down, especially in paradigms such as offline RL.