On the Estimation Bias in Double Q-Learning

1 Introduction . Value-based reinforcement learning with neural networks as function approximators has become a widely-used paradigm and shown great promise in solving complicated decision-making problems in various real-world applications , including robotics control ( Lillicrap et al. , 2016 ) , molecular structure design ( Zhou et al. , 2019 ) , and recommendation systems ( Chen et al. , 2018 ) . Towards understanding the foundation of these successes , investigating algorithmic properties of deep-learning-based value function approximation has attracted a growth of attention in recent years ( Van Hasselt et al. , 2018 ; Fu et al. , 2019 ; Achiam et al. , 2019 ; Dong et al. , 2020 ) . One of the phenomena of interest is that Q-learning ( Watkins , 1989 ) is known to suffer from overestimation issues , since it takes a maximum operator over a set of estimated action-values . Comparing with underestimated values , overestimation errors are more likely to be propagated through greedy action selections , which leads to an overestimation bias in value prediction ( Thrun and Schwartz , 1993 ) . This overoptimistic behavior of decision making has also been investigated in the literature of management science ( Smith and Winkler , 2006 ) and economics ( Thaler , 1988 ) . In deep Q-learning algorithms , one major source of value estimation errors comes from the optmization procedure . Although a deep neural network may have a sufficient expressiveness power to represent an accurate value function , the back-end optimization is hard to solve . As a result of computational considerations , stochastic gradient descent is almost the default choice for training deep Q-networks . As pointed out by Riedmiller ( 2005 ) and Van Hasselt et al . ( 2018 ) , a mini-batch gradient update may have unpredictable effects on state-action pairs outside the training batch . The high variance of gradient estimation by such stochastic methods would lead to an unavoidable approximation error in value prediction , which can not be eliminated by simply increasing sample size and †Work done while Zhizhou was an undergraduate at Tsinghua University . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . network capacity . Through the maximum operator in the Q-learning paradigm , such approximation error would propagate and accumulate to form an overestimation bias . In practice , even if most benchmark environments are nearly deterministic ( Brockman et al. , 2016 ) , a dramatic overestimation can be observed ( Van Hasselt et al. , 2016 ) . Double Q-learning ( Van Hasselt , 2010 ) is a classical method to reduce the risk of overestimation , which is a specific variant of the double estimator ( Stone , 1974 ) in the Q-learning paradigm . Instead of taking the greedy maximum values , it uses a second value function to construct an independent action-value evaluation as a cross validation . With proper assumptions , double Q-learning was proved to slightly underestimate rather than overestimate the maximum expected values ( Van Hasselt , 2010 ) . This technique has become a default implementation for stabilizing deep Q-learning algorithms ( Hessel et al. , 2018 ) . In continuous control domains , a famous variant named clipped double Qlearning ( Fujimoto et al. , 2018 ) also shows great success in reducing the accumulation of errors in actor-critic methods ( Haarnoja et al. , 2018 ; Kalashnikov et al. , 2018 ) . To understand algorithmic properties of double Q-learning and its variants , most prior work focus on the characterization of one-step estimation bias , i.e. , the expected deviation from target values in a single step of Bellman operation ( Lan et al. , 2020 ; Chen et al. , 2021 ) . In this paper , we present a different perspective on how these one-step errors accumulate in stationary solutions . We first review a widely-used analytical model introduced by Thrun and Schwartz ( 1993 ) and reveal a fact that , due to the perturbation of approximation error , both double Q-learning and clipped double Q-learning have multiple approximate fixed points in this model . This result raises a concern that double Q-learning may easily get stuck in some local stationary regions and become inefficient in searching for the optimal policy . Motivated by this finding , we propose a novel value estimator , named doubly bounded estimator , that utilizes an abstracted dynamic programming as a lower bound estimation to rule out the potential non-optimal fixed points . The proposed method is easy to be combined with other existing techniques such as clipped double Q-learning . We extensively evaluate our approach on a variety of Atari benchmark tasks , and demonstrate significant improvement over baseline algorithms in terms of sample efficiency and convergence performance . 2 Background . Markov Decision Process ( MDP ; Bellman , 1957 ) is a classical framework to formalize an agentenvironment interaction system which can be defined as a tupleM = 〈S , A , P , R , γ〉 . We use S and A to denote the state and action space , respectively . P ( s′|s , a ) and R ( s , a ) denote the transition and reward functions , which are initially unknown to the agent . γ is the discount factor . The goal of reinforcement learning is to construct a policy π : S → A maximizing cumulative rewards V π ( s ) = E [ ∞∑ t=0 γtR ( st , π ( st ) ) ∣∣∣∣ s0 = s , st+1 ∼ P ( ·|st , π ( st ) ) ] . Another quantity of interest can be defined through the Bellman equation Qπ ( s , a ) = R ( s , a ) + γEs′∼P ( ·|s , a ) [ V π ( s′ ) ] . The optimal value function Q∗ corresponds to the unique solution of the Bellman optimality equation , Q∗ ( s , a ) = R ( s , a ) + γ Es′∼P ( ·|s , a ) [ maxa′∈AQ∗ ( s′ , a′ ) ] . Q-learning algorithms are based on the Bellman optimality operator T stated as follows : ( T Q ) ( s , a ) = R ( s , a ) + γ E s′∼P ( ·|s , a ) [ max a′∈A Q ( s′ , a′ ) ] . ( 1 ) By iterating this operator , value iteration is proved to converge to the optimal value function Q∗ . To extend Q-learning methods to real-world applications , function approximation is indispensable to deal with a high-dimensional state space . Deep Q-learning ( Mnih et al. , 2015 ) considers a sample-based objective function and deploys an iterative optimization framework θt+1 = arg min θ∈Θ E ( s , a , r , s′ ) ∼D [ ( r + γ max a′∈A Qθt ( s ′ , a′ ) −Qθ ( s , a ) ) 2 ] , ( 2 ) in which Θ denotes the parameter space of the value network , and θ0 ∈ Θ is initialized by some predetermined method . ( s , a , r , s′ ) is sampled from a data distribution D which is changing during exploration . With infinite samples and a sufficiently rich function class , the update rule stated in Eq . ( 2 ) is asymptotically equivalent to applying the Bellman optimality operator T , but the underlying optimization is usually inefficient in practice . In deep Q-learning , Eq . ( 2 ) is optimized by mini-batch gradient descent and thus its value estimation suffers from unavoidable approximation errors . 3 On the Effects of Underestimation Bias in Double Q-Learning . In this section , we will first revisit a common analytical model used by previous work for studying estimation bias ( Thrun and Schwartz , 1993 ; Lan et al. , 2020 ) , in which double Q-learning is known to have underestimation bias . Based on this analytical model , we show that its underestimation bias could make double Q-learning have multiple fixed-point solutions under an approximate Bellman optimality operator . This result suggests that double Q-learning may have extra non-optimal stationary solutions under the effects of the approximation error . 3.1 Modeling Approximation Error in Q-Learning . In Q-learning with function approximation , the ground truth Bellman optimality operator T is approximated by a regression problem through Bellman error minimization ( see Eq . ( 1 ) and Eq . ( 2 ) ) , which may suffer from unavoidable approximation errors . Following Thrun and Schwartz ( 1993 ) and Lan et al . ( 2020 ) , we formalize underlying approximation errors as a set of random noises e ( t ) ( s , a ) on the regression outcomes : Q ( t+1 ) ( s , a ) = ( T Q ( t ) ) ( s , a ) + e ( t ) ( s , a ) . ( 3 ) In this model , double Q-learning ( Van Hasselt , 2010 ) can be modeled by two estimator instances { Q ( t ) i } i∈ { 1,2 } with separated noise terms { e ( t ) i } i∈ { 1,2 } . For simplification , we introduce a policy function π ( t ) ( s ) = arg maxaQ ( t ) 1 ( s , a ) to override the state value function as follows : V ( t ) ( s ) = Q ( t ) 2 ( s , π ( t ) ( s ) = arg maxa∈AQ ( t ) 1 ( s , a ) ) , ∀i ∈ { 1 , 2 } , Q ( t+1 ) i ( s , a ) = R ( s , a ) + γEs′ [ V ( t ) ( s′ ) ] ︸ ︷︷ ︸ target value + e ( t ) i ( s , a ) ︸ ︷︷ ︸ approximation error . ( 4 ) A minor difference of Eq . ( 4 ) from the definition of double Q-learning given by Van Hasselt ( 2010 ) is the usage of a unified target value V ( t ) ( s′ ) for both two estimators . This simplification does not affect the derived implications , and is also implemented by advanced variants of double Q-learning ( Fujimoto et al. , 2018 ; Lan et al. , 2020 ) . To establish a unified framework for analysis , we use a stochastic operator T̃ to denote the Q-iteration procedure Q ( t+1 ) ← T̃ Q ( t ) , e.g. , the updating rules stated as Eq . ( 3 ) and Eq . ( 4 ) . We call such an operator T̃ as a stochastic Bellman operator , since it approximates the ground truth Bellman optimality operator T and carries some noises due to approximation errors . Note that , as shown in Eq . ( 4 ) , the target value can be constructed only using the state-value function V ( t ) . We can define the stationary point of state-values V ( t ) as the fixed point of a stochastic Bellman operator T̃ . Definition 1 ( Approximate Fixed Points ) . Let T̃ denote a stochastic Bellman operator , such as what are stated in Eq . ( 3 ) and Eq . ( 4 ) . A state-value function V is regarded as an approximate fixed point under a stochastic Bellman operator T̃ if it satisfies E [ T̃ V ] = V , where T̃ V denotes the output state-value function while applying the Bellman operator T̃ on V . Remark . In prior work ( Thrun and Schwartz , 1993 ) , value estimation bias is defined by expected one-step deviation with respect to the ground truth Bellman operator , i.e. , E [ ( T̃ V ( t ) ) ( s ) ] − ( T V ( t ) ) ( s ) . The approximate fixed points stated in Definition 1 characterizes the accumulation of estimation biases in stationary solutions . In Appendix A.2 , we will prove the existence of such fixed points as the following statement . Proposition 1 . Assume the probability density functions of the noise terms { e ( s , a ) } are continuous . The stochastic Bellman operators defined by Eq . ( 3 ) and Eq . ( 4 ) must have approximate fixed points in arbitrary MDPs .

Summary: This paper studies the effect of the underestimation induced by Double Q-Learning, a well-known method to solve the overestimation problem of Q-Learning. The paper shows that Double Q-Learning might incur in suboptimal fixed points under an approximated setting, e.g. Double DQN, and proposes a method, i.e, Doubly Bounded Q-Learning, to curb this problem. The method consists of computing the target of the Q-Learning update as a maximum operator between the application of the Bellman operator on an approximation of the value function, and the application of the Bellman operator on the optimal value function computed on a discrete abstraction of the original MDP. The method is evaluated on several Atari games.