On the Estimation Bias in Double Q-Learning

1 Introduction . Value-based reinforcement learning with neural networks as function approximators has become a widely-used paradigm and shown great promise in solving complicated decision-making problems in various real-world applications , including robotics control ( Lillicrap et al. , 2016 ) , molecular structure design ( Zhou et al. , 2019 ) , and recommendation systems ( Chen et al. , 2018 ) . Towards understanding the foundation of these successes , investigating algorithmic properties of deep-learning-based value function approximation has attracted a growth of attention in recent years ( Van Hasselt et al. , 2018 ; Fu et al. , 2019 ; Achiam et al. , 2019 ; Dong et al. , 2020 ) . One of the phenomena of interest is that Q-learning ( Watkins , 1989 ) is known to suffer from overestimation issues , since it takes a maximum operator over a set of estimated action-values . Comparing with underestimated values , overestimation errors are more likely to be propagated through greedy action selections , which leads to an overestimation bias in value prediction ( Thrun and Schwartz , 1993 ) . This overoptimistic behavior of decision making has also been investigated in the literature of management science ( Smith and Winkler , 2006 ) and economics ( Thaler , 1988 ) . In deep Q-learning algorithms , one major source of value estimation errors comes from the optmization procedure . Although a deep neural network may have a sufficient expressiveness power to represent an accurate value function , the back-end optimization is hard to solve . As a result of computational considerations , stochastic gradient descent is almost the default choice for training deep Q-networks . As pointed out by Riedmiller ( 2005 ) and Van Hasselt et al . ( 2018 ) , a mini-batch gradient update may have unpredictable effects on state-action pairs outside the training batch . The high variance of gradient estimation by such stochastic methods would lead to an unavoidable approximation error in value prediction , which can not be eliminated by simply increasing sample size and †Work done while Zhizhou was an undergraduate at Tsinghua University . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . network capacity . Through the maximum operator in the Q-learning paradigm , such approximation error would propagate and accumulate to form an overestimation bias . In practice , even if most benchmark environments are nearly deterministic ( Brockman et al. , 2016 ) , a dramatic overestimation can be observed ( Van Hasselt et al. , 2016 ) . Double Q-learning ( Van Hasselt , 2010 ) is a classical method to reduce the risk of overestimation , which is a specific variant of the double estimator ( Stone , 1974 ) in the Q-learning paradigm . Instead of taking the greedy maximum values , it uses a second value function to construct an independent action-value evaluation as a cross validation . With proper assumptions , double Q-learning was proved to slightly underestimate rather than overestimate the maximum expected values ( Van Hasselt , 2010 ) . This technique has become a default implementation for stabilizing deep Q-learning algorithms ( Hessel et al. , 2018 ) . In continuous control domains , a famous variant named clipped double Qlearning ( Fujimoto et al. , 2018 ) also shows great success in reducing the accumulation of errors in actor-critic methods ( Haarnoja et al. , 2018 ; Kalashnikov et al. , 2018 ) . To understand algorithmic properties of double Q-learning and its variants , most prior work focus on the characterization of one-step estimation bias , i.e. , the expected deviation from target values in a single step of Bellman operation ( Lan et al. , 2020 ; Chen et al. , 2021 ) . In this paper , we present a different perspective on how these one-step errors accumulate in stationary solutions . We first review a widely-used analytical model introduced by Thrun and Schwartz ( 1993 ) and reveal a fact that , due to the perturbation of approximation error , both double Q-learning and clipped double Q-learning have multiple approximate fixed points in this model . This result raises a concern that double Q-learning may easily get stuck in some local stationary regions and become inefficient in searching for the optimal policy . Motivated by this finding , we propose a novel value estimator , named doubly bounded estimator , that utilizes an abstracted dynamic programming as a lower bound estimation to rule out the potential non-optimal fixed points . The proposed method is easy to be combined with other existing techniques such as clipped double Q-learning . We extensively evaluate our approach on a variety of Atari benchmark tasks , and demonstrate significant improvement over baseline algorithms in terms of sample efficiency and convergence performance . 2 Background . Markov Decision Process ( MDP ; Bellman , 1957 ) is a classical framework to formalize an agentenvironment interaction system which can be defined as a tupleM = 〈S , A , P , R , γ〉 . We use S and A to denote the state and action space , respectively . P ( s′|s , a ) and R ( s , a ) denote the transition and reward functions , which are initially unknown to the agent . γ is the discount factor . The goal of reinforcement learning is to construct a policy π : S → A maximizing cumulative rewards V π ( s ) = E [ ∞∑ t=0 γtR ( st , π ( st ) ) ∣∣∣∣ s0 = s , st+1 ∼ P ( ·|st , π ( st ) ) ] . Another quantity of interest can be defined through the Bellman equation Qπ ( s , a ) = R ( s , a ) + γEs′∼P ( ·|s , a ) [ V π ( s′ ) ] . The optimal value function Q∗ corresponds to the unique solution of the Bellman optimality equation , Q∗ ( s , a ) = R ( s , a ) + γ Es′∼P ( ·|s , a ) [ maxa′∈AQ∗ ( s′ , a′ ) ] . Q-learning algorithms are based on the Bellman optimality operator T stated as follows : ( T Q ) ( s , a ) = R ( s , a ) + γ E s′∼P ( ·|s , a ) [ max a′∈A Q ( s′ , a′ ) ] . ( 1 ) By iterating this operator , value iteration is proved to converge to the optimal value function Q∗ . To extend Q-learning methods to real-world applications , function approximation is indispensable to deal with a high-dimensional state space . Deep Q-learning ( Mnih et al. , 2015 ) considers a sample-based objective function and deploys an iterative optimization framework θt+1 = arg min θ∈Θ E ( s , a , r , s′ ) ∼D [ ( r + γ max a′∈A Qθt ( s ′ , a′ ) −Qθ ( s , a ) ) 2 ] , ( 2 ) in which Θ denotes the parameter space of the value network , and θ0 ∈ Θ is initialized by some predetermined method . ( s , a , r , s′ ) is sampled from a data distribution D which is changing during exploration . With infinite samples and a sufficiently rich function class , the update rule stated in Eq . ( 2 ) is asymptotically equivalent to applying the Bellman optimality operator T , but the underlying optimization is usually inefficient in practice . In deep Q-learning , Eq . ( 2 ) is optimized by mini-batch gradient descent and thus its value estimation suffers from unavoidable approximation errors . 3 On the Effects of Underestimation Bias in Double Q-Learning . In this section , we will first revisit a common analytical model used by previous work for studying estimation bias ( Thrun and Schwartz , 1993 ; Lan et al. , 2020 ) , in which double Q-learning is known to have underestimation bias . Based on this analytical model , we show that its underestimation bias could make double Q-learning have multiple fixed-point solutions under an approximate Bellman optimality operator . This result suggests that double Q-learning may have extra non-optimal stationary solutions under the effects of the approximation error . 3.1 Modeling Approximation Error in Q-Learning . In Q-learning with function approximation , the ground truth Bellman optimality operator T is approximated by a regression problem through Bellman error minimization ( see Eq . ( 1 ) and Eq . ( 2 ) ) , which may suffer from unavoidable approximation errors . Following Thrun and Schwartz ( 1993 ) and Lan et al . ( 2020 ) , we formalize underlying approximation errors as a set of random noises e ( t ) ( s , a ) on the regression outcomes : Q ( t+1 ) ( s , a ) = ( T Q ( t ) ) ( s , a ) + e ( t ) ( s , a ) . ( 3 ) In this model , double Q-learning ( Van Hasselt , 2010 ) can be modeled by two estimator instances { Q ( t ) i } i∈ { 1,2 } with separated noise terms { e ( t ) i } i∈ { 1,2 } . For simplification , we introduce a policy function π ( t ) ( s ) = arg maxaQ ( t ) 1 ( s , a ) to override the state value function as follows : V ( t ) ( s ) = Q ( t ) 2 ( s , π ( t ) ( s ) = arg maxa∈AQ ( t ) 1 ( s , a ) ) , ∀i ∈ { 1 , 2 } , Q ( t+1 ) i ( s , a ) = R ( s , a ) + γEs′ [ V ( t ) ( s′ ) ] ︸ ︷︷ ︸ target value + e ( t ) i ( s , a ) ︸ ︷︷ ︸ approximation error . ( 4 ) A minor difference of Eq . ( 4 ) from the definition of double Q-learning given by Van Hasselt ( 2010 ) is the usage of a unified target value V ( t ) ( s′ ) for both two estimators . This simplification does not affect the derived implications , and is also implemented by advanced variants of double Q-learning ( Fujimoto et al. , 2018 ; Lan et al. , 2020 ) . To establish a unified framework for analysis , we use a stochastic operator T̃ to denote the Q-iteration procedure Q ( t+1 ) ← T̃ Q ( t ) , e.g. , the updating rules stated as Eq . ( 3 ) and Eq . ( 4 ) . We call such an operator T̃ as a stochastic Bellman operator , since it approximates the ground truth Bellman optimality operator T and carries some noises due to approximation errors . Note that , as shown in Eq . ( 4 ) , the target value can be constructed only using the state-value function V ( t ) . We can define the stationary point of state-values V ( t ) as the fixed point of a stochastic Bellman operator T̃ . Definition 1 ( Approximate Fixed Points ) . Let T̃ denote a stochastic Bellman operator , such as what are stated in Eq . ( 3 ) and Eq . ( 4 ) . A state-value function V is regarded as an approximate fixed point under a stochastic Bellman operator T̃ if it satisfies E [ T̃ V ] = V , where T̃ V denotes the output state-value function while applying the Bellman operator T̃ on V . Remark . In prior work ( Thrun and Schwartz , 1993 ) , value estimation bias is defined by expected one-step deviation with respect to the ground truth Bellman operator , i.e. , E [ ( T̃ V ( t ) ) ( s ) ] − ( T V ( t ) ) ( s ) . The approximate fixed points stated in Definition 1 characterizes the accumulation of estimation biases in stationary solutions . In Appendix A.2 , we will prove the existence of such fixed points as the following statement . Proposition 1 . Assume the probability density functions of the noise terms { e ( s , a ) } are continuous . The stochastic Bellman operators defined by Eq . ( 3 ) and Eq . ( 4 ) must have approximate fixed points in arbitrary MDPs .

Summary: This paper proposes a new problem of double Q-learning, that is, the multiple fixed points caused by underestimation bias. This problem may lead to the non-optimal model after value iterations. Meanwhile, the authors provide many mathematical approvals. To solve it, the authors integrate another value estimator to escape the saddle points and approve its effectiveness with experiments.