Regularized Softmax Deep Multi-Agent Q-Learning

1 Introduction . In recent years , multi-agent reinforcement learning ( MARL ) has achieved significant progress [ 4 , 20 ] under the popular centralized training with decentralized execution ( CTDE ) paradigm [ 23 , 15 , 7 ] . In CTDE , the agents must learn decentralized policies so that at execution time they can act based on only local observations , but the training itself is centralized , with access to global information . A critical challenge in this setting is how to represent and learn the joint action-value function [ 28 ] . In learning the value function , overestimation is an important challenge that stems from the max operator [ 34 ] typically used in the bootstrapping target . Specifically , the max operator in Q-learning [ 39 ] approximates the maximum expected value with the maximum estimated value . This can lead to overestimation as E [ maxiXi ] ≥ maxi E [ Xi ] due to noise [ 34 , 11 ] , where Xi is a random variable representing the Q-value of action i given a state . This overestimation error can accumulate during learning , lead to sub-optimal policy updates and behaviors , and hurt the performance of both valuebased [ 37 , 2 , 32 , 16 ] and actor-critic algorithms [ 8 ] , and has been widely studied in the single-agent domain . However , overestimation can be even more severe in the multi-agent setting . For example , suppose there are n agents , each agent has K actions , and the Q-value for each action given a state is independently drawn from a uniform distribution U ( 0 , 1 ) . Then , maxi E [ Xi ] is 12 while E [ maxiXi ] is K n Kn+1 , which quickly approaches 1 ( the maximum value of Xi ) as the size of the joint action space ∗Work done while at University of Oxford . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . increases exponentially with the number of agents . Nonetheless , this problem has received much less attention in MARL . QMIX [ 29 ] is a popular CTDE deep multi-agent Q-learning algorithm for cooperative MARL . It combines the agent-wise utility functionsQa into the joint action-value functionQtot , via a monotonic mixing network to ensure consistent value factorization . Due to its superior performance , there have been many recent efforts to improve QMIX ’ s representation capability [ 31 , 41 , 27 , 38 ] . However , the overestimation problem of the joint-action Q-function Qtot can be exacerbated in QMIX due not only to overestimation in agents ’ Qa but also the non-linear monotonic mixing network . Despite this , the role of overestimation in limiting its performance has been largely overlooked . In this paper , we show empirically that overestimation in deep multi-agent Q-learning is more severe than previously acknowledged and can lead to divergent learning behavior in practice . In particular , consider double estimators , which can successfully reduce overestimation bias in the single-agent domain [ 37 , 8 ] . Although QMIX applies Double DQN [ 37 ] to estimate the value function ( as mentioned in Appendix D.3 in [ 28 ] and open-source PyMARL [ 30 ] implementations ) , we find that it is ineffective in deep multi-agent Q-learning . As shown in Figure 1 ( same experimental setup as in Sec- tion 5 ) , value estimates of the joint-action Q-function can increase without bound in tasks from the multi-agent particle framework [ 18 ] , yielding catastrophic performance degradation . Our experiments show that surprisingly , even applying Clipped Double Q-learning ( a key technique from a stateof-the-art TD3 [ 8 ] algorithm ) to the multi-agent setting does not resolve the severe overestimation bias in the joint-action Q-function . Therefore , alleviating overestimation in MARL is a particularly important and challenging problem . To tackle this issue , we propose a novel update scheme that penalizes large joint-action Q-values . Our key idea is to introduce a regularizer in the Bellman loss . A direct penalty on the magnitude of joint-action Q-values can result in a large estimation bias and hurt performance . Instead , to better trade off learning efficiency and stability , we introduce a baseline into the penalty , thereby constraining the joint-action Q-values to not deviate too much from this baseline . Specifically , we use the discounted return as the baseline . By regularizing towards a baseline , we stabilize learning and effectively avoid the unbounded growth in our value estimates . However , regularization is not enough to fully avoid overestimation bias in the joint-action Qfunction due to the max operator in the target ’ s value estimate [ 34 ] . To this end , we propose to employ a softmax operator , which has been shown to efficiently improve value estimates in the single-agent setting [ 32 , 26 , 25 ] . Unfortunately , a direct application of the softmax operator is often too computationally expensive in the multi-agent case , due to the exponentially-sized joint action space . We therefore propose a novel method that provides an efficient and reliable approximation to the softmax operator based on a joint action subspace , where the gap between our approximation and its direct computation converges to 0 at an exponential rate with respect to its inverse temperature parameter . The computational complexity of our approximation scales only linearly in the number of agents , as opposed to exponentially for the original softmax operator . We show that our softmax operator can further improve the value estimates in our experiments . We refer to our method as RES ( Regularized Softmax ) deep multi-agent Q-learning , which utilizes the discounted return-based regularization and our approximate softmax operator . To validate RES , we first prove that it can reduce the overestimation bias of QMIX . Next , we conduct extensive experiments in the multi-agent particle tasks [ 18 ] , and show that RES simultaneously enables stable learning , avoids severe overestimation when applied to QMIX , and achieves state-ofthe-art performance . RES is not tied to QMIX and can significantly improve the performance and stability of other deep multi-agent Q-learning algorithms , e.g. , Weighted-QMIX [ 27 ] and QPLEX [ 38 ] , demonstrating its versatility . Finally , to demonstrate its ability to scale to more complex scenarios , we evaluate it on a set of challenging StarCraft II micromanagement tasks [ 30 ] . Results show that RES-QMIX provides a consistent improvement over QMIX in all scenarios tested . 2 Background . Decentralized partially observable Markov decision process ( Dec-POMDP ) . A fully cooperative multi-agent task can be formulated as a Dec-POMDP [ 22 ] represented by a tuple 〈A , S , U , P , r , Z , O , γ〉 , where A ≡ { 1 , ... , n } denotes the finite set of agents , s ∈ S is the global state , and γ ∈ [ 0 , 1 ) is the discount factor . The action space for an agent is U with size K. At each timestep , each agent a ∈ A receives an observation z ∈ Z from the observation function O ( s , a ) due to partial observability , and chooses an action ua ∈ U , which forms a joint action u ∈ U ≡ Un . This leads to a transition to the next state s′ ∼ P ( s′|s , u ) and a joint reward r ( s , u ) . We assume that the reward function is bounded [ 35 ] , i.e. , |r ( s , u ) | ≤ Rmax . Each agent has an action-observation history τa ∈ T ≡ ( Z × U ) ∗ , based on which it constructs a policy πa ( ua|τa ) . The goal is to find an optimal joint policy π = 〈π1 , ... , πn〉 , whose joint action-value function is Qπ ( st , ut ) = E [ ∑∞ i=0 γ irt+i ] . Deep multi-agent Q-learning . Deep multi-agent Q-learning [ 33 , 29 , 31 , 38 ] extends deep Qlearning [ 39 ] , a popular value-based method for learning optimal action values , to the multi-agent setting . Given transitions ( s , u , r , s′ ) sampled from the experience replay buffer B , its objective is to minimize the mean squared error loss L ( θ ) on the temporal-difference ( TD ) error δ = y−Qtot ( s , u ) , where y = r + γmaxu′ Q̄tot ( s′ , u′ ) is the target value , and Q̄tot is the target network for the jointaction Q-function that is periodically copied from Qtot . Parameters of Qtot are denoted by θ that are updated by θ′ = θ − α∇θL ( θ ) , where α is the learning rate . To mitigate the overestimation bias from the max operator in the target ’ s value estimate [ 34 ] , Double DQN [ 37 , 11 ] estimates the target value as y = r + γQ̄tot ( s′ , arg maxu′ Qtot ( s′ , u′ ) ) which decouples action selection and evaluation . Centralized training with decentralized execution ( CTDE ) . In CTDE [ 23 , 15 ] , agents are trained in a centralized way with access to the overall action-observation history and global state during training , but during execution have access only to their own local action-observation histories . The individual-global-max ( IGM ) property [ 31 ] is a popular concept to realize efficient CTDE as in Eq . ( 1 ) , where Qtot and Qa denote the joint-action Q-function and agent-wise utilities respectively . arg max u Qtot ( s , u ) = ( arg max u1 Q1 ( s , u1 ) , · · · , arg max un Qn ( s , un ) ) . ( 1 ) The IGM property enables efficient decentralized execution , which ensures the consistency between greedy action selection in the local and joint Q-values . QMIX [ 29 ] is a popular CTDE method that combines Qa into Qtot via a non-linear monotonic function fs that is state-dependent as in Eq . ( 2 ) . To satisfy the IGM property , fs is constrained to be monotonic in Qa , i.e. , ∂fs∂Qa ≥ 0 , ∀a ∈ A , which is achieved by enforcing non-negative weights in fs . Qtot ( s , u ) = fs ( Q1 ( s , u1 ) , · · · , Qn ( s , un ) ) . ( 2 ) 3 Overestimation in QMIX In this section , we show empirically that the overestimation problem for deep multi-agent Q-learning can be more severe in practice than previously acknowledged . In particular , we demonstrate that state-ofthe-art methods for tackling this issue in the single-agent domain can fail in our multi-agent setting . We investigate the behavior of QMIX in the predator-prey task from the multi-agent particle environments [ 18 ] , where 3 slower predators need to coordinate to capture a faster prey to solve the task . A detailed description of the task is included in Appendix E.1.1 . We consider a fully cooperative setting where the prey is pre-trained by MADDPG [ 18 ] to avoid capture by the predators , and the predators need to learn to cooperate with each other in order to surround and capture the prey . Figures 2 and 3 ( a ) show the performance of QMIX and its estimated values of the joint-action Q-function during training respectively . QMIX suffers from catastrophic performance degradation and a severe overestimation bias , where its value estimates grow without bound . Figures 3 ( b ) and 3 ( c ) illustrate the mean agent-wise utilities Qa and their gradients ∂fs/∂Qa over agents respectively . We see that for QMIX , the gradients ∂fs/∂Qa increase rapidly and continuously during learning . Figure 3 ( d ) shows the learned weight ( matrix ) of the monotonic mixing network fs . The weights similarly grow larger during training and further amplify the overestimation in each Qa when computing Qtot , leading to a severely overestimated joint-action Q-function that continues to grow without bound . To reduce overestimation , Ackermann et al . [ 1 ] extend the state-of-the-art TD3 algorithm [ 8 ] , which addresses overestimation in the single-agent case , to the multi-agent setting . We obtain QMIX ( CDQ ) by applying the key Clipped Double Q-learning technique from TD3 [ 8 ] to per-agent utilities in QMIX for value estimation.2 As shown in Figure 3 ( b ) , QMIX ( CDQ ) slows the increase of agent-wise utilities . However , Figure 3 ( a ) shows that it still does not eliminate the severe overestimation bias in the joint-action Q-function , and subsequently suffers a performance degradation as shown in Figure 2 . This is due to the gradients of the monotonic mixing network , ∂fs/∂Qa , which still continuously increase as shown in Figure 3 ( c ) , leading to the large and increasing value estimates in Figure 3 ( a ) . Another way to avoid overly large value estimates is to limit the gradients themselves . We propose Gradient Regularization ( GradReg ) to prevent the gradients ∂fs/∂Qa from growing too large by regularizing the gradient using a quadratic penalty . Specifically , the loss function of QMIX ( GradReg ) is defined as LGradReg ( θ ) = E ( s , u , r , s′ ) ∼B [ δ2 + λ ( ∂fs/∂Qa ) 2 ] , where δ is the TD error defined in Section 2 and λ is the regularization coefficient . However , although QMIX ( GradReg ) with a tuned coefficient for the regularizer prevents the gradients from growing overly large during the early stage of learning ( Figure 3 ( c ) ) , the agent-wise utility functions grow larger instead as shown in Figure 3 ( b ) . This then still leads to a particularly large value estimate as shown in Figure 3 ( a ) . Finally , a direct combination of both CDQ and GradReg also fails to avoid the problem . From these experiments , we see that even applying the state-of-the-art TD3 [ 8 ] method from the single-agent literature can fail in our multi-agent setting . In addition , it is also insufficient to tackle the problem by regularizing the magnitude of the gradients ∂fs/∂Qa . These results show that mitigating overestimation in MARL is particularly important and challenging , and requires novel solutions . 4 Regularized Softmax ( RES ) Deep Multi-Agent Q-Learning Our analysis shows that preventing gradients from being large as in QMIX ( GradReg ) and/or utilizing clipped double estimators as in QMIX ( CDQ ) are not sufficient to prevent overly large value estimates . It is important to note that these methods take an indirect approach to reduce the overestimation of the joint-action Q-value , and as shown do not solve the severe overestimation problem . Therefore , we propose a novel way to directly penalize large joint-action Q-values to avoid value estimate explosion . Directly penalizing large joint-action Q-values can push their values towards 0 , yielding a large estimation bias . Instead , we introduce a baseline into the penalty to better trade off learning efficiency and stability . Specifically , the new learning objective penalizes joint-action Q-values deviating from a baseline b ( s , u ) , by adding a regularizer to the loss : λ ( Qtot ( s , u ) − b ( s , u ) ) 2 , where we use the mean squared error loss and λ is the regularization coefficient . A potential choice for the baseline is theN -step return , i.e. , b ( s , u ) = ∑N−1 t=0 γ trt+γ N maxu′ Qtot ( sN , u ′ ) . Figure 4 ( a ) shows the value estimates of Regularized ( RE ) -QMIX ( b = N -step return ) using the N -step return as the baseline ( with best N , λ ) . However , as it still involves Qtot , the value estimation can still grow extremely large despite being discounted , yielding estimates that almost coincide with those of QMIX . 2Applying CDQ on Qtot leads to larger value estimates than applying it on Qa ( details are in Appendix A.2 ) . Therefore , we propose to use the discounted return starting from state s as the baseline to regularize learning , i.e. , b ( s , u ) = Rt ( s , u ) = ∑∞ k=0 γ krt+k for the t-th timestep . The regularized learning objective directly penalizes large joint-action Q-values and allows the value estimates to remain grounded by real data . As shown in Figure 4 ( a ) , RE-QMIX effectively avoids particularly large value estimations and stabilizes learning . However , while the regularized learning objective is effective in resolving value explosion , it can still overestimate due to the use of the max operator when computing target value estimates [ 34 ] . Figure 4 ( b ) shows the estimated and true values of REQMIX ( with best λ ) . The estimated values are computed by averaging over 100 states sampled from the replay buffer at each timestep , and we estimate true values by averaging the discounted returns which are obtained by following the greedy policy with respect to the current Qtot starting from the sampled states . As shown in Fig- ure 4 ( b ) , although RE-QMIX avoids unbounded growth of its value estimates , it does not fully avoid the overestimation bias , which still estimates the target value according to the max operator [ 34 ] . To further mitigate overestimation bias in the joint-action Q-function , we adopt the softmax operator , which has shown great potential in reducing overestimation bias in place of the max operator in single-agent domains [ 32 , 26 , 25 ] . Specifically , the softmax operator is defined in Eq . ( 3 ) , where β ≥ 0 is the inverse temperature parameter . smβ , U ( Qtot ( s , · ) ) = ∑ u∈U eβQtot ( s , u ) ∑ u′∈U e βQtot ( s , u′ ) Qtot ( s , u ) . ( 3 ) When β approaches∞ or 0 , softmax reduces to the max or mean operator respectively . Unfortunately , computing Eq . ( 3 ) in the multi-agent case can be computationally intractable as the size of the joint action space grows exponentially with the number of agents . In addition , as the action space in the multi-agent case is much larger than that in the single-agent case , some joint-action Q-value estimates Qtot ( s , u ) can be unreliable due to a lack of sufficient training . Thus , directly taking them all into consideration for computing the softmax operator in Eq . ( 3 ) can result in inaccurate value estimates . We propose a novel method to approximate the computation of the softmax operator efficiently and reliably . Specifically , we first obtain the maximal joint action û = arg maxuQtot ( s , u ) with respect to Qtot according to the individual-global-max ( IGM ) property discussed in Section 2 . Then , for each agent a , we considerK joint actions , by changing only agent a ’ s action while keeping the other agents ’ actions û−a fixed . We denote the resulting action set of agent a by Ua = { ( ua , û−a ) |ua ∈ U } . Finally , we form a joint action subspace Û = U1 ∪ · · · ∪ Un , where each Ua contributes K actions , and use Û in Eq . ( 3 ) for computing the approximate softmax operator . In Theorem 1 , we show that the gap between our approximation in Û and its direct computation in the joint action space U converges to 0 at an exponential rate with respect to β . The proof can be found in Appendix B.1 . Theorem 1 . Let u∗ and u′ denote the optimal joint actions in U and U − Û with respect to Qtot , respectively . The difference between our approximate softmax operator and its direct computation in the whole action space satisfies : ∀s ∈ S , |smβ , Û ( Qtot ( s , · ) ) − smβ , U ( Qtot ( s , · ) ) | ≤ 2Rmax 1−γ |U−Û | |U−Û |+exp ( β ( Qtot ( s , u∗ ) −Qtot ( s , u′ ) ) ) , where |U − Û | denotes the size of the set U − Û . Discussion . Since its computational complexity is linear instead of exponential in the number of agents , our approximation is feasible to compute even when the number of agents grows large , as opposed to the original softmax operator . According to the IGM property , Û consists of joint actions that are close to the maximal joint action û . Thus , it it more likely to contain joint actions with more accurate and reliable value estimates , especially when using -greedy exploration , since they are more likely to be sampled and therefore trained . Due to this reduced reliance on unreliable joint-action Q-values , our approximate softmax operator actually outperforms its counterpart using a direct computation in the joint action space U , as well as a random sampling scheme , as shown in Section 5.1.3 . The full algorithm for our approximate softmax is in Appendix B.2 . As shown in Figure 4 ( b ) , RES ( Regularized Softmax ) -QMIX fully addresses the overestimation bias and achieves better value estimates than RE-QMIX.3 The loss of Regularized Softmax ( RES ) Deep Multi-Agent Q-Learning LRES ( θ ) is defined as : E ( s , u , r , s′ ) ∼B [ δ2smβ + λ ( Qtot ( s , u ) −Rt ( s , u ) ) 2 ] , ( 4 ) where δsmβ = r+ γsmβ , Û ( Q̄tot ( s ′ , · ) ) −Qtot ( s , u ) , and λ denotes the coefficient of the regularizer . To better understand the effectiveness of our method , we provide a theoretical justification that connects the loss function of RES and a new Bellman operator in Theorem 2 . Theorem 2 . Given the same sample distribution , the update of the RES method is equivalent to the update using L ( θ ) = E ( s , u , r , s′ ) ∼B [ ( y −Qtot ( s , u ) ) 2 ] with learning rate ( λ+ 1 ) α , which estimates the target value according to y = r+γsmβ , Û ( Q̄tot ( s ′ , · ) ) λ+1 + λRt ( s , u ) λ+1 . Its proof is in Appendix C. In the special case where λ = 0 , it is equivalent to changing the max operator to our approximate softmax in the target value estimation . When λ > 0 , it can be thought of as learning with a weighted combination of the 1-step TD target with our approximate softmax and the discounted return , which allows the bootstrapping target to remain grounded by real data.4 Let T be the value estimation operator which estimates the value of the next state s′ . From Theorem 2 , we have TQMIX = maxu′ Q̄tot ( s′ , u′ ) , TRE-QMIX = maxu′ Q̄tot ( s ′ , u′ ) +λRt+1 ( s ′ ) λ+1 , and TRES-QMIX = smβ , Û ( Q̄tot ( s ′ , · ) ) +λRt+1 ( s′ ) λ+1 . We now analyze the relationship between the value estimation bias of these operators in Theorem 3 ( the proof is in Appendix D ) , and show that RES-QMIX can reduce the overestimation bias when built upon QMIX . It is also worth noting that since RES is general , it can be readily applied to other deep multi-agent Q-learning algorithms , as investigated in Section 5.1.4 . Theorem 3 . Let B ( T ) = E [ T ( s′ ) ] −maxu′ Q∗tot ( s′ , u′ ) be the bias of value estimates of T and the true optimal joint-action Q-function Q∗tot . Given the same assumptions as in [ 37 ] for the joint-action Q-function , where there exists some V ∗tot ( s ′ ) = Q∗tot ( s ′ , u′ ) for different joint actions , ∑ u′ ( Q̄tot ( s ′ , u′ ) − V ∗tot ( s′ ) ) = 0 , and 1|U | ∑ u′ ( Q̄tot ( s ′ , u′ ) − V ∗tot ( s′ ) ) 2 = C ( C > 0 ) with Q̄tot an arbitrary joint-action Q-function , then B ( TRES-QMIX ) ≤ B ( TRE-QMIX ) ≤ B ( TQMIX ) . From the proof of Theorem 3 and Figure 4 ( a ) , we find that RE-QMIX can reduce the overestimation bias of QMIX following the same standard assumptions in [ 37 , 32 ] . In addition , Theorem 3 also shows that the bias of value estimates of RES-QMIX is no larger than that of RE-QMIX , which validates our RES method .

Summary: Overestimation in Q-learning is an important problem. However, there has been less attention on the overestimation problem in multi-agent setting compared to the single-agent setting. The authors show that Q-learning in multi-agent setting suffers from overestimation even more than the previously acknowledged. This paper proposes a regularized update scheme that tackles this issue. Also, the new softmax operator, Regularized Softmax (RES) is introduced. Empirical results on various domains, including Starcraft II tasks, are given to support the claim.