Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedback

1 Introduction . Thanks to recent advances in function approximation methods using deep neural networks [ 20 ] , Q-learning [ 35 ] has been widely used to solve reinforcement learning ( RL ) problems in a variety of applications , e.g. , robotic control [ 23 , 13 ] , path planning [ 15 , 24 ] and production scheduling [ 34 , 21 ] . Despite the great success , it is well recognized that Q-learning may suffer from the notorious overestimation bias [ 29 , 33 , 32 , 10 , 37 ] , which would significantly impede the learning efficiency . Recent work [ 9 , 11 ] indicates that this problem also persists in the actor-critic setting . To address this issue , the ensemble method [ 16 , 1 , 26 , 7 ] has emerged as a promising solution in which multiple Q-function approximators are used to get better estimation of the action values . Needless to say , the ensemble size , i.e. , the number of Q-function approximators used in the target , has intrinsic impact on Q-learning . Notably , it is shown in [ 6 , 17 ] that while a large ensemble size could completely remove the overestimation bias , it may go to the other extreme and result in underestimation bias and unstable training , which is clearly not desirable . Therefore , instead of simply increasing the ensemble 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . size to mitigate the overestimation issue , a fundamental question to ask is : “ Is it possible to determine the right ensemble size on the fly so as to minimize the estimation bias ? ” Some existing ensemble methods [ 2 , 19 , 17 ] adopt a trial-and-error strategy to search for the ensemble size , which would be time-consuming and require a lot of human engineering for different RL tasks . The approximation error of the Q-function during the learning process plays a nontrivial role in the selection of the ensemble size , since it directly impacts the Q-target estimation accuracy . This however remains not well understood . In particular , the fact that the approximation error is time-varying , due to the iterative nature of Q-learning [ 36 , 5 ] , gives rise to the question that whether a fixed ensemble size should be used in the learning process . To answer this question , we show in Section 2.2 that using a fixed ensemble size is likely to lead to either overestimation or underestimation bias , and the bias may shift between overestimation and underestimation because of the time-varying approximation error , calling for an adaptive ensemble size so as to drive the bias close to zero based on the underlying learning dynamics . Thus motivated , in this work we study effective ensemble size adaptation to minimize the estimation bias that hinges heavily on the time-varying approximation errors during the learning process . To this end , we first characterize the relationship among the ensemble size , the function approximation errors , and the estimation bias , by deriving an upper bound and a lower bound on the estimation bias . Our findings reveal that the ensemble size should be selected adaptively in a way to cope with the impact of the time-varying approximation errors . Building upon the theoretic results , we cast the estimation bias minimization as an adaptive control problem where the approximation error during the learning process is treated as the control object , and the ensemble size is adapted based on the feedback of the control output , i.e. , the value of the approximation error from the last iteration . The key idea in this approach is inspired from the classic Model Identification Adaptive Control ( MIAC ) framework [ 3 , 25 ] , where at each step the current system identification of the control object is fed back to adjust the controller , and consequently a new control signal is devised following the updated control law . One main contribution of this work lies in the development of AdaEQ , a generalized ensemble method for the ensemble size adaptation , aiming to minimize the estimation bias during the learning process . Specifically , the approximation error in each iteration is quantified by comparing the difference between the Q-estimates and the Monte Carlo return using the current learned policy over a testing trajectory [ 29 , 17 ] . Inspired by MIAC , the approximation error serves as the feedback to adapt the ensemble size . Besides , we introduce a ‘ tolerance ’ parameter in the adaptation mechanism to balance the control tendency towards positive or negative bias during the learning process . In this way , AdaEQ can encompass other existing ensemble methods as special cases , including Maxmin [ 17 ] , by properly setting this hyperparameter . A salient feature of the feedback-adaptation mechanism is that it can be used effectively in conjunction with both standard Q-learning [ 22 ] and actor-critic methods [ 28 , 11 ] . Experimental results on the continuous-control MuJoCo benchmark [ 30 ] show that AdaEQ is robust to the initial ensemble size in different environments , and achieves higher average return , thanks to keeping the estimation bias close to zero , when compared to the state-of-the-art ensemble methods such as REDQ [ 6 ] and Average-DQN [ 2 ] . Related Work . Bias-corrected Q-learning [ 18 ] introduces the bias correction term to reduce the overestimation bias . Double Q-learning is proposed in [ 12 , 33 ] to address the overestimation issue in vanilla Q-learning , by leveraging two independent Q-function approximators to estimate the maximum Q-function value in the target . S-DQN and S-DDQN use the softmax operator instead of the max operator to further reduce the overestimation bias [ 27 ] . Self-correcting Q-learning aims to balance the underestimation in double Q-learning and overestimation in classic Q learning by introducing a new self-correcting estimator [ 38 ] . Weighted Q-learning proposes a new estimator based on the weighted average of the sample means , and conducts the empirical analysis in the discrete action space [ 8 ] . Weighted Double Q-learning [ 37 ] uses the Q-approximator together with the double Q-approximator to balance the overestimation and underestimation bias . Nevertheless , acquiring independent approximators is often intractable for large-scale tasks . To resolve this issue , the Twin-Delayed Deep Deterministic policy gradient algorithm ( TD3 ) [ 9 ] and Soft Actor-Critic ( SAC ) [ 11 ] have been devised to take the minimum over two approximators in the target network . Along a different avenue , the ensemble-based methods generalize double Q-learning to correct the overestimation bias by increasing the number of Q-function approximators . Particularly , AverageDQN [ 2 ] takes the average of multiple approximators in the target to reduce the overestimation error , and Random Ensemble Mixture ( REM ) [ 1 ] estimates the target value using the random convex combination of the approximators . It is worth noting that both Average-DQN and REM can not completely eliminate the overestimation bias . Most recently , Maxmin Q-learning [ 17 ] defines a proxy Q-function by choosing the minimum Q-value for each action among all approximators . Similar to Maxmin , Random Ensembled Q-learning ( REDQ ) [ 6 ] formulates the proxy Q-function by choosing only a subset of the ensemble . Nevertheless , both Maxmin and REDQ use a fixed ensemble size . In this study , we introduce an adaptation mechanism for the ensemble size to drive the estimation bias to be close to zero , thereby mitigating the possible overestimation and underestimation issues . 2 Impact of Ensemble Size on Estimation Bias . 2.1 Ensemble Q-learning . As is standard , we consider a Markov decision process ( MDP ) defined by the tuple 〈S , A , P , r , γ〉 , where S and A denote the state space and the action space , respectively . P ( s′|s , a ) : S ×A× S → [ 0 , 1 ] denotes the probability transition function from current state s to the next state s′ by taking action a ∈ A , and r ( s , a ) : S ×A → R is the corresponding reward . γ ∈ ( 0 , 1 ] is the discount factor . At each step t , the agent observes the state st , takes an action at following a policy π : S → A , receives the reward rt , and evolves to a new state st+1 . The objective is to find an optimal policy π∗ to maximize the discounted return R = ∑∞ t=0 γ trt . By definition , Q-function is the expected return when choosing action a in state s and following with the policy π : Qπ = E [ ∑∞ t=0 γ trt ( st , at ) |s0 = s , a0 = a ] . Q-learning is an off-policy value-based method that aims at learning the optimal Q-function Q∗ : S ×A → R , where the optimal Q-function is a fixed point of the Bellman optimality equation [ 4 ] : T Q∗ ( s , a ) = r ( s , a ) + γEs′∼P ( s′|s , a ) [ maxa′∈AQ∗ ( s′ , a′ ) ] . ( 1 ) Given a transition sample ( s , a , r , s′ ) , the Bellman operator can be employed to update the Q-function as follows : Q ( s , a ) ← ( 1− α ) Q ( s , a ) + αy , y : = r + γmaxa′∈AQ ( s′ , a′ ) . ( 2 ) where α is the step size and y is the target . Under some conditions , Q-learning can converge to the optimal fixed-point solution asymptotically [ 31 ] . In deep Q-learning , the Q-function is approximated by a neural network , and it has been shown [ 33 ] that the approximation error , amplified by the max operator in the target , results in the overestimation phenomena . One promising approach to address this issue is the ensemble Q-learning method , which is the main subject of this study . The Ensemble Method . Specifically , the ensemble method maintains N separate approximators Q1 , Q2 , · · · , QN of the Q-function , based on which a subset of these approximators is used to devise a proxy Q-function . For example , in Average-DQN [ 2 ] , the proxy Q-function is obtained by computing the average value over all N approximators to reduce the overestimation bias : Qave ( · ) = 1N ∑N i=1Q i ( · ) . However , the average operation can not completely eliminate the overestimation bias , since the average of the overestimation bias is still positive . To tackle this challenge , Maxmin [ 17 ] and REDQ [ 6 ] take the ‘ min ’ operation over a subsetM ( size M ) of the ensemble : Qproxy ( · ) = mini∈MQi ( · ) . ( 3 ) The target value in the ensemble-based Q-learning is then computed as y = r +maxa′∈AQproxy . It is worth noting that in the existing studies , the in-target ensemble size M , pre-determined for a given environment , remain fixed in the learning process . 2.2 An Illustrative Example . It is known that the determination of the optimal ensemble size is highly nontrivial , and a poor choice of the ensemble size would degrade the performance of ensemble Q-learning significantly [ 17 ] . As mentioned earlier , it is unclear a priori if a fixed ensemble size should be used in the learning process . ( a ) Function approximation . ( b ) Five function approximators . ( c ) Ensemble via ‘ min ’ operator . ( d ) Estimation error . Figure 2 : Illustration of estimation bias in the ensemble method . ( a ) Each approximator is fitted to the noisy values ( green dots ) at the sampled states independently . ( b ) Five Q-function approximators are obtained for both actions ( green lines and blue lines ) . ( c ) Apply the min operator over M ( M = 3 ) randomly selected approximators to obtain a proxy approximator for each action . ( d ) The estimation error is obtained by comparing the underlying true value ( purple line in ( a ) ) and the target value using the proxy approximator . ( a ) Estimation bias vs. τ . ( b ) Estimation bias vs. numbers of actions . Figure 3 : Illustration of overestimation and underestimation phenomena for different ensemble sizes . In what follows , we use an example to illustrate the potential pitfalls in the ensemble methods by examining the sensitivity of the estimation bias to the ensemble size [ 6 , 17 ] . Along the same line as in [ 33 ] , we consider an example with a real-valued continuous state space . In this example , there are two discrete actions available at each state and the optimal action values depend only on the state , i.e. , in each state both actions result in the same optimal value Q∗ ( s , · ) , which is assumed to be Q∗ ( s , · ) = sin ( s ) . Figure 2 demonstrates how the ensemble method is carried out in four stages : ( I ) For each Q-function approximator Qi , i = 1 , 2 , · · · , 5 , we first generate 10 noisy action-value samples independently ( green dots in Figure 2 ( a ) ) . Let ei ( s , a ) denote the approximation error of Qi : Qi ( s , a ) = Q∗ ( s , a ) + ei ( s , a ) , with ei ( s , a ) ∼ U ( −τi , τi ) , ( 4 ) where τi ∼ U ( 0 , τ ) models the approximation error distribution for the i-th approximator . Note that the assumption on the uniform error distribution is commonly used to indicate that both positive and negative approximation error are possible in Q-function approximators [ 29 ] [ 17 ] [ 6 ] . ( II ) Next , Figure 2 ( b ) illustrates the ensemble ( N = 5 ) of approximators for two actions , where each approximator is a 6-degree polynomial that fits the noisy values at sampled states . ( III ) Following the same ensemble approach in [ 6 ] [ 17 ] , we randomly choose M approximators from the ensemble and take the minimum over them to obtain a proxy approximator for each action , resulting in the dashed lines in Figure 2 ( c ) . ( IV ) Finally , the maximum action value of the proxy approximator is used as the target to update the current approximators . To evaluate the target value estimation error , Figure 2 ( d ) depicts the difference between the obtained target value and the underlying true value when using different ensemble size M . As in [ 33 ] , we utilize the average estimation error ( i.e. , estimation bias ) to quantify the performance of current approximators . For example , when the ensemble size M = 2 , the red line is above zero for most states , implying the overestimation tendency in the target . Clearly , Figure 2 ( d ) indicates that the estimation bias is highly dependent on the ensemble size , and even a change of M can lead the shift from overestimation to underestimation . Since the Q-function approximation error of each approximator changes over time in the training process [ 5 ] ( examples for this phenomenon can be found in Appendix B.3 ) , we next analyze the impact of the ensemble size on the estimation bias under different approximation error distributions . As shown in Figure 3 ( a ) , with a fixed ensemble size M , the estimation bias may shift between positive and negative and be ‘ dramatically ’ large for some error distributions . In light of this observation , departing from using a fixed size , we advocate to adapt the in-target ensemble size , e.g. , set M = 4 when the noise parameter τ > 1.5 and M = 3 otherwise . The estimation bias resulted by this adaptation mechanism is much closer to zero . Besides , Figure 3 ( b ) characterizes the estimation bias under different action spaces , which is also important considering that different tasks normally have different action spaces and the number of available actions may vary in different states even for the same task . The adaptive ensemble approach is clearly more robust in our setting . In a nutshell , both Figure 3 ( a ) and 3 ( b ) suggest that a fixed ensemble size would not work well to minimize the estimation bias during learning for different tasks . This phenomenon has also been observed in the empirical results [ 17 ] . In stark contrast , adaptively changing the ensemble size based on the approximation error indeed can help to reduce the estimation bias in different settings .

Summary: This paper proposes an ensemble Q-learning which can adaptively adjust the ensemble size based on estimation error. The authors find that the ensemble min(Q_1, Q_2, ..., Q_M) may suffer from overestimation or underestimation with different M, so they propose to adapt M based on approximation error, which can minimize the estimation bias correspondingly.   The paper analyzes both in theoretically and empirically, show better performance than the existing ensemble q-learning methods on a few MuJoCo benchmark tasks.