Maxmin Q-learning: Controlling the Estimation Bias of Q-learning

1 INTRODUCTION . Q-learning ( Watkins , 1989 ) is one of the most popular reinforcement learning algorithms . One of the reasons for this widespread adoption is the simplicity of the update . On each step , the agent updates its action value estimates towards the observed reward and the estimated value of the maximal action in the next state . This target represents the highest value the agent thinks it could obtain from the current state and action , given the observed reward . Unfortunately , this simple update rule has been shown to suffer from overestimation bias ( Thrun & Schwartz , 1993 ; van Hasselt , 2010 ) . The agent updates with the maximum over action values might be large because an action ’ s value actually is high , or it can be misleadingly high simply because of the stochasticity or errors in the estimator . With many actions , there is a higher probability that one of the estimates is large simply due to stochasticity and the agent will overestimate the value . This issue is particularly problematic under function approximation , and can significant impede the quality of the learned policy ( Thrun & Schwartz , 1993 ; Szita & Lőrincz , 2008 ; Strehl et al. , 2009 ) or even lead to failures of Q-learning ( Thrun & Schwartz , 1993 ) . More recently , experiments across several domains suggest that this overestimation problem is common ( Hado van Hasselt et al. , 2016 ) . Double Q-learning ( van Hasselt , 2010 ) is introduced to instead ensure underestimation bias . The idea is to maintain two unbiased independent estimators of the action values . The expected action value of estimator one is selected for the maximal action from estimator two , which is guaranteed not to overestimate the true maximum action value . Double DQN ( Hado van Hasselt et al. , 2016 ) , the extension of this idea to Q-learning with neural networks , has been shown to significantly improve performance over Q-learning . However , this is not a complete answer to this problem , because trading overestimation bias for underestimation bias is not always desirable , as we show in our experiments . 1Code is available at https : //github.com/qlan3/Explorer Several other methods have been introduced to reduce overestimation bias , without fully moving towards underestimation . Weighted Double Q-learning ( Zhang et al. , 2017 ) uses a weighted combination of the Double Q-learning estimate , which likely has underestimation bias , and the Q-learning estimate , which likely has overestimation bias . Bias-corrected Q-Learning ( Lee et al. , 2013 ) reduces the overestimation bias through a bias correction term . Ensemble Q-learning and Averaged Q-learning ( Anschel et al. , 2017 ) take averages of multiple action values , to both reduce the overestimation bias and the estimation variance . However , with a finite number of actionvalue functions , the average operation in these two algorithms will never completely remove the overestimation bias , as the average of several overestimation biases is always positive . Further , these strategies do not guide how strongly we should correct for overestimation bias , nor how to determine—or control—the level of bias . The overestimation bias also appears in the actor-critic setting ( Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) . For example , Fujimoto et al . ( 2018 ) propose the Twin Delayed Deep Deterministic policy gradient algorithm ( TD3 ) which reduces the overestimation bias by taking the minimum value between two critics . However , they do not provide a rigorous theoretical analysis for the effect of applying the minimum operator . There is also no theoretical guide for choosing the number of estimators such that the overestimation bias can be reduced to 0 . In this paper , we study the effects of overestimation and underestimation bias on learning performance , and use them to motivate a generalization of Q-learning called Maxmin Q-learning . Maxmin Q-learning directly mitigates the overestimation bias by using a minimization over multiple action-value estimates . Moreover , it is able to control the estimation bias varying from positive to negative which helps improve learning efficiency as we will show in next sections . We prove that , theoretically , with an appropriate number of action-value estimators , we are able to acquire an unbiased estimator with a lower approximation variance than Q-learning . We empirically verify our claims on several benchmarks . We study the convergence properties of our algorithm within a novel Generalized Q-learning framework , which is suitable for studying several of the recently proposed Q-learning variants . We also combine deep neural networks with Maxmin Q-learning ( Maxmin DQN ) and demonstrate its effectiveness in several benchmark domains . 2 PROBLEM SETTING . We formalize the problem as a Markov Decision Process ( MDP ) , ( S , A , P , r , γ ) , where S is the state space , A is the action space , P : S×A×S → [ 0 , 1 ] is the transition probabilities , r : S×A×S → R is the reward mapping , and γ ∈ [ 0 , 1 ] is the discount factor . At each time step t , the agent observes a state St ∈ S and takes an action At ∈ A and then transitions to a new state St+1 ∈ S according to the transition probabilities P and receives a scalar reward Rt+1 = r ( St , At , St+1 ) ∈ R. The goal of the agent is to find a policy π : S × A → [ 0 , 1 ] that maximizes the expected return starting from some initial state . Q-learning is an off-policy algorithm which attempts to learn the state-action valuesQ : S×A → R for the optimal policy . It tries to solve for Q∗ ( s , a ) = E [ Rt+1 + max a′∈A Q∗ ( St+1 , a ′ ) ∣∣∣ St = s , At = a ] The optimal policy is to act greedily with respect to these action values : from each s select a from arg maxa∈AQ ∗ ( s , a ) . The update rule for an approximation Q for a sampled transition st , at , rt+1 , st+1 is : Q ( st , at ) ← Q ( st , at ) + α ( Y Qt −Q ( st , at ) ) for Y Q t def = rt+1 + γ max a′∈A Q ( st+1 , a ′ ) ( 1 ) where α is the step-size . The transition can be generated off-policy , from any behaviour that sufficiently covers the state space . This algorithm is known to converge in the tabular setting ( Tsitsiklis , 1994 ) , with some limited results for the function approximation setting ( Melo & Ribeiro , 2007 ) . 3 UNDERSTANDING WHEN OVERESTIMATION BIAS HELPS AND HURTS . In this section , we briefly discuss the estimation bias issue , and empirically show that both overestimation and underestimation bias may improve learning performance , depending on the environment . This motivates our Maxmin Q-learning algorithm described in the next section , which allows us to flexibly control the estimation bias and reduce the estimation variance . The overestimation bias occurs since the target maxa′∈AQ ( st+1 , a′ ) is used in the Q-learning update . Because Q is an approximation , it is probable that the approximation is higher than the true value for one or more of the actions . The maximum over these estimators , then , is likely to be skewed towards an overestimate . For example , even unbiased estimates Q ( st+1 , a′ ) for all a′ , will vary due to stochasticity . Q ( st+1 , a′ ) = Q∗ ( st+1 , a′ ) + ea′ , and for some actions , ea′ will be positive . As a result , E [ maxa′∈AQ ( st+1 , a′ ) ] ≥ maxa′∈A E [ Q ( st+1 , a′ ) ] = maxa′∈AQ∗ ( st+1 , a′ ) . This overestimation bias , however , may not always be detrimental . And , further , in some cases , erring towards an underestimation bias can be harmful . Overestimation bias can help encourage exploration for overestimated actions , whereas underestimation bias might discourage exploration . In particular , we expect more overestimation bias in highly stochastic areas of the world ; if those highly stochastic areas correspond to high-value regions , then encouraging exploration there might be beneficial . An underestimation bias might actually prevent an agent from learning that a region is high-value . Alternatively , if highly stochastic areas also have low values , overestimation bias might cause an agent to over-explore a low-value region . We show this effect in the simple MDP , shown in Figure 1 . The MDP for state A has only two actions : Left and Right . It has a deterministic neutral reward for both the Left action and the Right action . The Left action transitions to state B where there are eight actions transitions to a terminate state with a highly stochastic reward . The mean of this stochastic reward is µ . By selecting µ > 0 , the stochastic region becomes high-value , and we expect overestimation bias to help and underestimation bias to hurt . By selecting µ < 0 , the stochastic region becomes low-value , and we expect overestimation bias to hurt and underestimation bias to help . We test Q-learning , Double Q-learning and our new algorithm Maxmin Q-learning in this environment . Maxmin Q-learning ( described fully in the next section ) uses N estimates of the action values in the targets . For N = 1 , it corresponds to Q-learning ; otherwise , it progresses from overestimation bias at N = 1 towards underestimation bias with increasing N . In the experiment , we used a discount factor γ = 1 ; a replay buffer with size 100 ; an -greedy behaviour with = 0.1 ; tabular action-values , initialized with a Gaussian distributionN ( 0 , 0.01 ) ; and a step-size of 0.01 for all algorithms . The results in Figure 2 verify our hypotheses for when overestimation and underestimation bias help and hurt . Double Q-learning underestimates too much for µ = +1 , and converges to a suboptimal policy . Q-learning learns the optimal policy the fastest , though for all values of N = 2 , 4 , 6 , 8 , Maxmin Q-learning does progress towards the optimal policy . All methods get to the optimal policy for µ = −1 , but now Double Q-learning reaches the optimal policy the fastest , and followed by Maxmin Q-learning with larger N . 4 MAXMIN Q-LEARNING . In this section , we develop Maxmin Q-learning , a simple generalization of Q-learning designed to control the estimation bias , as well as reduce the estimation variance of action values . The idea is to maintain N estimates of the action values , Qi , and use the minimum of these estimates in the Q-learning target : maxa′ mini∈ { 1 , ... , N } Qi ( s′ , a′ ) . For N = 1 , the update is simply Q-learning , and so likely has overestimation bias . As N increase , the overestimation decreases ; for some N > 1 , this maxmin estimator switches from an overestimate , in expectation , to an underestimate . We characterize the relationship between N and the expected estimation bias below in Theorem 1 . Note that Maxmin Q-learning uses a different mechanism to reduce overestimation bias than Double Qlearning ; Maxmin Q-learning with N = 2 is not Double Q-learning . The full algorithm is summarized in Algorithm 1 , and is a simple modification of Q-learning with experience replay . We use random subsamples of the observed data for each of the N estimators , to make them nearly independent . To do this training online , we keep a replay buffer . On each step , a random estimator i is chosen and updated using a mini-batch from the buffer . Multiple such updates can be performed on each step , just like in experience replay , meaning multiple estimators can be updated per step using different random mini-batches . In our experiments , to better match DQN , we simply do one update per step . Finally , it is also straightforward to incorporate target networks to get Maxmin DQN , by maintaining a target network for each estimator . We now characterize the relation between the number of action-value functions used in Maxmin Q-learning and the estimation bias of action values . For compactness , we write Qisa instead of Qi ( s , a ) . Each Qisa has random approximation error e i sa Qisa = Q ∗ sa + e i sa . We assume that eisa is a uniform random variable U ( −τ , τ ) for some τ > 0 . The uniform random assumption was used by Thrun & Schwartz ( 1993 ) to demonstrate bias in Q-learning , and reflects that non-negligible positive and negative eisa are possible . Notice that for N estimators with nsa samples , the τ will be proportional to some function of nsa/N , because the data will be shared amongst the N estimators . For the general theorem , we use a generic τ , and in the following corollary provide a specific form for τ in terms of N and nsa . Recall that M is the number of actions applicable at state s′ . Define the estimation bias ZMN for transition s , a , r , s′ to be ZMN def = ( r + γmax a′ Qmins′a′ ) − ( r + γmax a′ Q∗s′a′ ) = γ ( max a′ Qmins′a′ −max a′ Q∗s′a′ ) Algorithm 1 : Maxmin Q-learning Input : step-size α , exploration parameter > 0 , number of action-value functions N Initialize N action-value functions { Q1 , . . . , QN } randomly Initialize empty replay buffer D Observe initial state s while Agent is interacting with the Environment do Qmin ( s , a ) ← mink∈ { 1 , ... , N } Qk ( s , a ) , ∀a ∈ A Choose action a by -greedy based on Qmin Take action a , observe r , s′ Store transition ( s , a , r , s′ ) in D Select a subset S from { 1 , . . . , N } ( e.g. , randomly select one i to update ) for i ∈ S do Sample random mini-batch of transitions ( sD , aD , rD , s′D ) from D Get update target : YMQ ← rD + γmaxa′∈AQmin ( s′D , a′ ) Update action-value Qi : Qi ( sD , aD ) ← Qi ( sD , aD ) + α [ YMQ −Qi ( sD , aD ) ] end s← s′ end where Qminsa def = min i∈ { 1 , ... , N } Qisa = Q ∗ sa + min i∈ { 1 , ... , N } eisa We now show how the expected estimation bias E [ ZMN ] and the variance of Qminsa are related to the number of action-value functions N in Maxmin Q-learning . Theorem 1 Under the conditions stated above , ( i ) the expected estimation bias is E [ ZMN ] = γτ [ 1− 2tMN ] where tMN = M ( M − 1 ) · · · 1 ( M + 1N ) ( M − 1 + 1 N ) · · · ( 1 + 1 N ) . E [ ZMN ] decreases as N increases : E [ ZM , N=1 ] = γτ M−1M+1 and E [ ZM , N→∞ ] = −γτ . ( ii ) V ar [ Qminsa ] = 4Nτ2 ( N + 1 ) 2 ( N + 2 ) . V ar [ Qminsa ] decreases as N increases : V ar [ Q min sa ] = τ2 3 for N=1 and V ar [ Q min sa ] = 0 for N →∞ . Theorem 1 is a generalization of the first lemma in Thrun & Schwartz ( 1993 ) ; we provide the proof in Appendix A as well as a visualization of the expected bias for varying M and N . This theorem shows that the average estimation bias E [ ZMN ] , decreases as N increases . Thus , we can control the bias by changing the number of estimators in Maxmin Q-learning . Specifically , the average estimation bias can be reduced from positive to negative as N increases . Notice that E [ ZMN ] = 0 when tMN = 12 . This suggests that by choosing N such that tMN ≈ 1 2 , we can reduce the bias to near 0 . Furthermore , V ar [ Qminsa ] decreases asN increases . This indicates that we can control the estimation variance of target action value throughN . We show just this in the following Corollary . The subtlety is that with increasing N , each estimator will receive less data . The fair comparison is to compare the variance of a single estimator that uses all of the data , as compared to the maxmin estimator which shares the samples across N estimators . We show that there is an N such that the variance is lower , which arises largely due to the fact that the variance of each estimator decreases linearly in n , but the τ parameter for each estimator only decreases at a square root rate in the number of samples . Corollary 1 Assuming the nsa samples are evenly allocated amongst the N estimators , then τ =√ 3σ2N/nsa where σ2 is the variance of samples for ( s , a ) and , for Qsa the estimator that uses all nsa samples for a single estimate , V ar [ Qminsa ] = 12N2 ( N + 1 ) 2 ( N + 2 ) V ar [ Qsa ] . Under this uniform random noise assumption , for N ≥ 8 , V ar [ Qminsa ] < V ar [ Qsa ] .

Summary: The paper tackles the problem of bias in target Q-values when performing Q-learning.  The paper proposes a technique for computing target Q-values, by first taking the min over an ensemble of learned Q-values and then taking the max over actions.  The paper provides some theoretical properties of this technique: (1) the bias of the estimator can be somewhat controlled by the size of the ensemble; (2) performing Q-learning with these target values is convergent. Experimental results show that the proposed technique can provide performance improvement on a number of tasks.