Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedback

1 Introduction . Thanks to recent advances in function approximation methods using deep neural networks [ 20 ] , Q-learning [ 35 ] has been widely used to solve reinforcement learning ( RL ) problems in a variety of applications , e.g. , robotic control [ 23 , 13 ] , path planning [ 15 , 24 ] and production scheduling [ 34 , 21 ] . Despite the great success , it is well recognized that Q-learning may suffer from the notorious overestimation bias [ 29 , 33 , 32 , 10 , 37 ] , which would significantly impede the learning efficiency . Recent work [ 9 , 11 ] indicates that this problem also persists in the actor-critic setting . To address this issue , the ensemble method [ 16 , 1 , 26 , 7 ] has emerged as a promising solution in which multiple Q-function approximators are used to get better estimation of the action values . Needless to say , the ensemble size , i.e. , the number of Q-function approximators used in the target , has intrinsic impact on Q-learning . Notably , it is shown in [ 6 , 17 ] that while a large ensemble size could completely remove the overestimation bias , it may go to the other extreme and result in underestimation bias and unstable training , which is clearly not desirable . Therefore , instead of simply increasing the ensemble 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . size to mitigate the overestimation issue , a fundamental question to ask is : “ Is it possible to determine the right ensemble size on the fly so as to minimize the estimation bias ? ” Some existing ensemble methods [ 2 , 19 , 17 ] adopt a trial-and-error strategy to search for the ensemble size , which would be time-consuming and require a lot of human engineering for different RL tasks . The approximation error of the Q-function during the learning process plays a nontrivial role in the selection of the ensemble size , since it directly impacts the Q-target estimation accuracy . This however remains not well understood . In particular , the fact that the approximation error is time-varying , due to the iterative nature of Q-learning [ 36 , 5 ] , gives rise to the question that whether a fixed ensemble size should be used in the learning process . To answer this question , we show in Section 2.2 that using a fixed ensemble size is likely to lead to either overestimation or underestimation bias , and the bias may shift between overestimation and underestimation because of the time-varying approximation error , calling for an adaptive ensemble size so as to drive the bias close to zero based on the underlying learning dynamics . Thus motivated , in this work we study effective ensemble size adaptation to minimize the estimation bias that hinges heavily on the time-varying approximation errors during the learning process . To this end , we first characterize the relationship among the ensemble size , the function approximation errors , and the estimation bias , by deriving an upper bound and a lower bound on the estimation bias . Our findings reveal that the ensemble size should be selected adaptively in a way to cope with the impact of the time-varying approximation errors . Building upon the theoretic results , we cast the estimation bias minimization as an adaptive control problem where the approximation error during the learning process is treated as the control object , and the ensemble size is adapted based on the feedback of the control output , i.e. , the value of the approximation error from the last iteration . The key idea in this approach is inspired from the classic Model Identification Adaptive Control ( MIAC ) framework [ 3 , 25 ] , where at each step the current system identification of the control object is fed back to adjust the controller , and consequently a new control signal is devised following the updated control law . One main contribution of this work lies in the development of AdaEQ , a generalized ensemble method for the ensemble size adaptation , aiming to minimize the estimation bias during the learning process . Specifically , the approximation error in each iteration is quantified by comparing the difference between the Q-estimates and the Monte Carlo return using the current learned policy over a testing trajectory [ 29 , 17 ] . Inspired by MIAC , the approximation error serves as the feedback to adapt the ensemble size . Besides , we introduce a ‘ tolerance ’ parameter in the adaptation mechanism to balance the control tendency towards positive or negative bias during the learning process . In this way , AdaEQ can encompass other existing ensemble methods as special cases , including Maxmin [ 17 ] , by properly setting this hyperparameter . A salient feature of the feedback-adaptation mechanism is that it can be used effectively in conjunction with both standard Q-learning [ 22 ] and actor-critic methods [ 28 , 11 ] . Experimental results on the continuous-control MuJoCo benchmark [ 30 ] show that AdaEQ is robust to the initial ensemble size in different environments , and achieves higher average return , thanks to keeping the estimation bias close to zero , when compared to the state-of-the-art ensemble methods such as REDQ [ 6 ] and Average-DQN [ 2 ] . Related Work . Bias-corrected Q-learning [ 18 ] introduces the bias correction term to reduce the overestimation bias . Double Q-learning is proposed in [ 12 , 33 ] to address the overestimation issue in vanilla Q-learning , by leveraging two independent Q-function approximators to estimate the maximum Q-function value in the target . S-DQN and S-DDQN use the softmax operator instead of the max operator to further reduce the overestimation bias [ 27 ] . Self-correcting Q-learning aims to balance the underestimation in double Q-learning and overestimation in classic Q learning by introducing a new self-correcting estimator [ 38 ] . Weighted Q-learning proposes a new estimator based on the weighted average of the sample means , and conducts the empirical analysis in the discrete action space [ 8 ] . Weighted Double Q-learning [ 37 ] uses the Q-approximator together with the double Q-approximator to balance the overestimation and underestimation bias . Nevertheless , acquiring independent approximators is often intractable for large-scale tasks . To resolve this issue , the Twin-Delayed Deep Deterministic policy gradient algorithm ( TD3 ) [ 9 ] and Soft Actor-Critic ( SAC ) [ 11 ] have been devised to take the minimum over two approximators in the target network . Along a different avenue , the ensemble-based methods generalize double Q-learning to correct the overestimation bias by increasing the number of Q-function approximators . Particularly , AverageDQN [ 2 ] takes the average of multiple approximators in the target to reduce the overestimation error , and Random Ensemble Mixture ( REM ) [ 1 ] estimates the target value using the random convex combination of the approximators . It is worth noting that both Average-DQN and REM can not completely eliminate the overestimation bias . Most recently , Maxmin Q-learning [ 17 ] defines a proxy Q-function by choosing the minimum Q-value for each action among all approximators . Similar to Maxmin , Random Ensembled Q-learning ( REDQ ) [ 6 ] formulates the proxy Q-function by choosing only a subset of the ensemble . Nevertheless , both Maxmin and REDQ use a fixed ensemble size . In this study , we introduce an adaptation mechanism for the ensemble size to drive the estimation bias to be close to zero , thereby mitigating the possible overestimation and underestimation issues . 2 Impact of Ensemble Size on Estimation Bias . 2.1 Ensemble Q-learning . As is standard , we consider a Markov decision process ( MDP ) defined by the tuple 〈S , A , P , r , γ〉 , where S and A denote the state space and the action space , respectively . P ( s′|s , a ) : S ×A× S → [ 0 , 1 ] denotes the probability transition function from current state s to the next state s′ by taking action a ∈ A , and r ( s , a ) : S ×A → R is the corresponding reward . γ ∈ ( 0 , 1 ] is the discount factor . At each step t , the agent observes the state st , takes an action at following a policy π : S → A , receives the reward rt , and evolves to a new state st+1 . The objective is to find an optimal policy π∗ to maximize the discounted return R = ∑∞ t=0 γ trt . By definition , Q-function is the expected return when choosing action a in state s and following with the policy π : Qπ = E [ ∑∞ t=0 γ trt ( st , at ) |s0 = s , a0 = a ] . Q-learning is an off-policy value-based method that aims at learning the optimal Q-function Q∗ : S ×A → R , where the optimal Q-function is a fixed point of the Bellman optimality equation [ 4 ] : T Q∗ ( s , a ) = r ( s , a ) + γEs′∼P ( s′|s , a ) [ maxa′∈AQ∗ ( s′ , a′ ) ] . ( 1 ) Given a transition sample ( s , a , r , s′ ) , the Bellman operator can be employed to update the Q-function as follows : Q ( s , a ) ← ( 1− α ) Q ( s , a ) + αy , y : = r + γmaxa′∈AQ ( s′ , a′ ) . ( 2 ) where α is the step size and y is the target . Under some conditions , Q-learning can converge to the optimal fixed-point solution asymptotically [ 31 ] . In deep Q-learning , the Q-function is approximated by a neural network , and it has been shown [ 33 ] that the approximation error , amplified by the max operator in the target , results in the overestimation phenomena . One promising approach to address this issue is the ensemble Q-learning method , which is the main subject of this study . The Ensemble Method . Specifically , the ensemble method maintains N separate approximators Q1 , Q2 , · · · , QN of the Q-function , based on which a subset of these approximators is used to devise a proxy Q-function . For example , in Average-DQN [ 2 ] , the proxy Q-function is obtained by computing the average value over all N approximators to reduce the overestimation bias : Qave ( · ) = 1N ∑N i=1Q i ( · ) . However , the average operation can not completely eliminate the overestimation bias , since the average of the overestimation bias is still positive . To tackle this challenge , Maxmin [ 17 ] and REDQ [ 6 ] take the ‘ min ’ operation over a subsetM ( size M ) of the ensemble : Qproxy ( · ) = mini∈MQi ( · ) . ( 3 ) The target value in the ensemble-based Q-learning is then computed as y = r +maxa′∈AQproxy . It is worth noting that in the existing studies , the in-target ensemble size M , pre-determined for a given environment , remain fixed in the learning process . 2.2 An Illustrative Example . It is known that the determination of the optimal ensemble size is highly nontrivial , and a poor choice of the ensemble size would degrade the performance of ensemble Q-learning significantly [ 17 ] . As mentioned earlier , it is unclear a priori if a fixed ensemble size should be used in the learning process . ( a ) Function approximation . ( b ) Five function approximators . ( c ) Ensemble via ‘ min ’ operator . ( d ) Estimation error . Figure 2 : Illustration of estimation bias in the ensemble method . ( a ) Each approximator is fitted to the noisy values ( green dots ) at the sampled states independently . ( b ) Five Q-function approximators are obtained for both actions ( green lines and blue lines ) . ( c ) Apply the min operator over M ( M = 3 ) randomly selected approximators to obtain a proxy approximator for each action . ( d ) The estimation error is obtained by comparing the underlying true value ( purple line in ( a ) ) and the target value using the proxy approximator . ( a ) Estimation bias vs. τ . ( b ) Estimation bias vs. numbers of actions . Figure 3 : Illustration of overestimation and underestimation phenomena for different ensemble sizes . In what follows , we use an example to illustrate the potential pitfalls in the ensemble methods by examining the sensitivity of the estimation bias to the ensemble size [ 6 , 17 ] . Along the same line as in [ 33 ] , we consider an example with a real-valued continuous state space . In this example , there are two discrete actions available at each state and the optimal action values depend only on the state , i.e. , in each state both actions result in the same optimal value Q∗ ( s , · ) , which is assumed to be Q∗ ( s , · ) = sin ( s ) . Figure 2 demonstrates how the ensemble method is carried out in four stages : ( I ) For each Q-function approximator Qi , i = 1 , 2 , · · · , 5 , we first generate 10 noisy action-value samples independently ( green dots in Figure 2 ( a ) ) . Let ei ( s , a ) denote the approximation error of Qi : Qi ( s , a ) = Q∗ ( s , a ) + ei ( s , a ) , with ei ( s , a ) ∼ U ( −τi , τi ) , ( 4 ) where τi ∼ U ( 0 , τ ) models the approximation error distribution for the i-th approximator . Note that the assumption on the uniform error distribution is commonly used to indicate that both positive and negative approximation error are possible in Q-function approximators [ 29 ] [ 17 ] [ 6 ] . ( II ) Next , Figure 2 ( b ) illustrates the ensemble ( N = 5 ) of approximators for two actions , where each approximator is a 6-degree polynomial that fits the noisy values at sampled states . ( III ) Following the same ensemble approach in [ 6 ] [ 17 ] , we randomly choose M approximators from the ensemble and take the minimum over them to obtain a proxy approximator for each action , resulting in the dashed lines in Figure 2 ( c ) . ( IV ) Finally , the maximum action value of the proxy approximator is used as the target to update the current approximators . To evaluate the target value estimation error , Figure 2 ( d ) depicts the difference between the obtained target value and the underlying true value when using different ensemble size M . As in [ 33 ] , we utilize the average estimation error ( i.e. , estimation bias ) to quantify the performance of current approximators . For example , when the ensemble size M = 2 , the red line is above zero for most states , implying the overestimation tendency in the target . Clearly , Figure 2 ( d ) indicates that the estimation bias is highly dependent on the ensemble size , and even a change of M can lead the shift from overestimation to underestimation . Since the Q-function approximation error of each approximator changes over time in the training process [ 5 ] ( examples for this phenomenon can be found in Appendix B.3 ) , we next analyze the impact of the ensemble size on the estimation bias under different approximation error distributions . As shown in Figure 3 ( a ) , with a fixed ensemble size M , the estimation bias may shift between positive and negative and be ‘ dramatically ’ large for some error distributions . In light of this observation , departing from using a fixed size , we advocate to adapt the in-target ensemble size , e.g. , set M = 4 when the noise parameter τ > 1.5 and M = 3 otherwise . The estimation bias resulted by this adaptation mechanism is much closer to zero . Besides , Figure 3 ( b ) characterizes the estimation bias under different action spaces , which is also important considering that different tasks normally have different action spaces and the number of available actions may vary in different states even for the same task . The adaptive ensemble approach is clearly more robust in our setting . In a nutshell , both Figure 3 ( a ) and 3 ( b ) suggest that a fixed ensemble size would not work well to minimize the estimation bias during learning for different tasks . This phenomenon has also been observed in the empirical results [ 17 ] . In stark contrast , adaptively changing the ensemble size based on the approximation error indeed can help to reduce the estimation bias in different settings .

Summary: Some Reinforcement Learning algorithms like Q-learning are subject to a bias known as the maximization or overestimation bias. This is due to the fact that value iteration approximates the value of the next state as the max over actions of Q-values. As a consequence, if the Q-values are approximate, the maximum over such approximations tends to be skewed towards high values. Ensembling many independent estimates of the Q-values helps mitigatin the overestimation bias, but often at the risk of achieving the opposite effect, i.e. an underestimation bias, if the ensemble is too large. The point of this paper is to propose and develop a method to estimate the current bias, and use that to estimate to dynamically choose the size of the ensemble so as to find a trade-off between over- and underestimation that minimizes the estimation bias. The method is demonstrated on a simple toy task and then on control tasks with continuous action spaces simulated with MuJoCo physics engine.