Regularized Softmax Deep Multi-Agent Q-Learning

1 Introduction . In recent years , multi-agent reinforcement learning ( MARL ) has achieved significant progress [ 4 , 20 ] under the popular centralized training with decentralized execution ( CTDE ) paradigm [ 23 , 15 , 7 ] . In CTDE , the agents must learn decentralized policies so that at execution time they can act based on only local observations , but the training itself is centralized , with access to global information . A critical challenge in this setting is how to represent and learn the joint action-value function [ 28 ] . In learning the value function , overestimation is an important challenge that stems from the max operator [ 34 ] typically used in the bootstrapping target . Specifically , the max operator in Q-learning [ 39 ] approximates the maximum expected value with the maximum estimated value . This can lead to overestimation as E [ maxiXi ] ≥ maxi E [ Xi ] due to noise [ 34 , 11 ] , where Xi is a random variable representing the Q-value of action i given a state . This overestimation error can accumulate during learning , lead to sub-optimal policy updates and behaviors , and hurt the performance of both valuebased [ 37 , 2 , 32 , 16 ] and actor-critic algorithms [ 8 ] , and has been widely studied in the single-agent domain . However , overestimation can be even more severe in the multi-agent setting . For example , suppose there are n agents , each agent has K actions , and the Q-value for each action given a state is independently drawn from a uniform distribution U ( 0 , 1 ) . Then , maxi E [ Xi ] is 12 while E [ maxiXi ] is K n Kn+1 , which quickly approaches 1 ( the maximum value of Xi ) as the size of the joint action space ∗Work done while at University of Oxford . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . increases exponentially with the number of agents . Nonetheless , this problem has received much less attention in MARL . QMIX [ 29 ] is a popular CTDE deep multi-agent Q-learning algorithm for cooperative MARL . It combines the agent-wise utility functionsQa into the joint action-value functionQtot , via a monotonic mixing network to ensure consistent value factorization . Due to its superior performance , there have been many recent efforts to improve QMIX ’ s representation capability [ 31 , 41 , 27 , 38 ] . However , the overestimation problem of the joint-action Q-function Qtot can be exacerbated in QMIX due not only to overestimation in agents ’ Qa but also the non-linear monotonic mixing network . Despite this , the role of overestimation in limiting its performance has been largely overlooked . In this paper , we show empirically that overestimation in deep multi-agent Q-learning is more severe than previously acknowledged and can lead to divergent learning behavior in practice . In particular , consider double estimators , which can successfully reduce overestimation bias in the single-agent domain [ 37 , 8 ] . Although QMIX applies Double DQN [ 37 ] to estimate the value function ( as mentioned in Appendix D.3 in [ 28 ] and open-source PyMARL [ 30 ] implementations ) , we find that it is ineffective in deep multi-agent Q-learning . As shown in Figure 1 ( same experimental setup as in Sec- tion 5 ) , value estimates of the joint-action Q-function can increase without bound in tasks from the multi-agent particle framework [ 18 ] , yielding catastrophic performance degradation . Our experiments show that surprisingly , even applying Clipped Double Q-learning ( a key technique from a stateof-the-art TD3 [ 8 ] algorithm ) to the multi-agent setting does not resolve the severe overestimation bias in the joint-action Q-function . Therefore , alleviating overestimation in MARL is a particularly important and challenging problem . To tackle this issue , we propose a novel update scheme that penalizes large joint-action Q-values . Our key idea is to introduce a regularizer in the Bellman loss . A direct penalty on the magnitude of joint-action Q-values can result in a large estimation bias and hurt performance . Instead , to better trade off learning efficiency and stability , we introduce a baseline into the penalty , thereby constraining the joint-action Q-values to not deviate too much from this baseline . Specifically , we use the discounted return as the baseline . By regularizing towards a baseline , we stabilize learning and effectively avoid the unbounded growth in our value estimates . However , regularization is not enough to fully avoid overestimation bias in the joint-action Qfunction due to the max operator in the target ’ s value estimate [ 34 ] . To this end , we propose to employ a softmax operator , which has been shown to efficiently improve value estimates in the single-agent setting [ 32 , 26 , 25 ] . Unfortunately , a direct application of the softmax operator is often too computationally expensive in the multi-agent case , due to the exponentially-sized joint action space . We therefore propose a novel method that provides an efficient and reliable approximation to the softmax operator based on a joint action subspace , where the gap between our approximation and its direct computation converges to 0 at an exponential rate with respect to its inverse temperature parameter . The computational complexity of our approximation scales only linearly in the number of agents , as opposed to exponentially for the original softmax operator . We show that our softmax operator can further improve the value estimates in our experiments . We refer to our method as RES ( Regularized Softmax ) deep multi-agent Q-learning , which utilizes the discounted return-based regularization and our approximate softmax operator . To validate RES , we first prove that it can reduce the overestimation bias of QMIX . Next , we conduct extensive experiments in the multi-agent particle tasks [ 18 ] , and show that RES simultaneously enables stable learning , avoids severe overestimation when applied to QMIX , and achieves state-ofthe-art performance . RES is not tied to QMIX and can significantly improve the performance and stability of other deep multi-agent Q-learning algorithms , e.g. , Weighted-QMIX [ 27 ] and QPLEX [ 38 ] , demonstrating its versatility . Finally , to demonstrate its ability to scale to more complex scenarios , we evaluate it on a set of challenging StarCraft II micromanagement tasks [ 30 ] . Results show that RES-QMIX provides a consistent improvement over QMIX in all scenarios tested . 2 Background . Decentralized partially observable Markov decision process ( Dec-POMDP ) . A fully cooperative multi-agent task can be formulated as a Dec-POMDP [ 22 ] represented by a tuple 〈A , S , U , P , r , Z , O , γ〉 , where A ≡ { 1 , ... , n } denotes the finite set of agents , s ∈ S is the global state , and γ ∈ [ 0 , 1 ) is the discount factor . The action space for an agent is U with size K. At each timestep , each agent a ∈ A receives an observation z ∈ Z from the observation function O ( s , a ) due to partial observability , and chooses an action ua ∈ U , which forms a joint action u ∈ U ≡ Un . This leads to a transition to the next state s′ ∼ P ( s′|s , u ) and a joint reward r ( s , u ) . We assume that the reward function is bounded [ 35 ] , i.e. , |r ( s , u ) | ≤ Rmax . Each agent has an action-observation history τa ∈ T ≡ ( Z × U ) ∗ , based on which it constructs a policy πa ( ua|τa ) . The goal is to find an optimal joint policy π = 〈π1 , ... , πn〉 , whose joint action-value function is Qπ ( st , ut ) = E [ ∑∞ i=0 γ irt+i ] . Deep multi-agent Q-learning . Deep multi-agent Q-learning [ 33 , 29 , 31 , 38 ] extends deep Qlearning [ 39 ] , a popular value-based method for learning optimal action values , to the multi-agent setting . Given transitions ( s , u , r , s′ ) sampled from the experience replay buffer B , its objective is to minimize the mean squared error loss L ( θ ) on the temporal-difference ( TD ) error δ = y−Qtot ( s , u ) , where y = r + γmaxu′ Q̄tot ( s′ , u′ ) is the target value , and Q̄tot is the target network for the jointaction Q-function that is periodically copied from Qtot . Parameters of Qtot are denoted by θ that are updated by θ′ = θ − α∇θL ( θ ) , where α is the learning rate . To mitigate the overestimation bias from the max operator in the target ’ s value estimate [ 34 ] , Double DQN [ 37 , 11 ] estimates the target value as y = r + γQ̄tot ( s′ , arg maxu′ Qtot ( s′ , u′ ) ) which decouples action selection and evaluation . Centralized training with decentralized execution ( CTDE ) . In CTDE [ 23 , 15 ] , agents are trained in a centralized way with access to the overall action-observation history and global state during training , but during execution have access only to their own local action-observation histories . The individual-global-max ( IGM ) property [ 31 ] is a popular concept to realize efficient CTDE as in Eq . ( 1 ) , where Qtot and Qa denote the joint-action Q-function and agent-wise utilities respectively . arg max u Qtot ( s , u ) = ( arg max u1 Q1 ( s , u1 ) , · · · , arg max un Qn ( s , un ) ) . ( 1 ) The IGM property enables efficient decentralized execution , which ensures the consistency between greedy action selection in the local and joint Q-values . QMIX [ 29 ] is a popular CTDE method that combines Qa into Qtot via a non-linear monotonic function fs that is state-dependent as in Eq . ( 2 ) . To satisfy the IGM property , fs is constrained to be monotonic in Qa , i.e. , ∂fs∂Qa ≥ 0 , ∀a ∈ A , which is achieved by enforcing non-negative weights in fs . Qtot ( s , u ) = fs ( Q1 ( s , u1 ) , · · · , Qn ( s , un ) ) . ( 2 ) 3 Overestimation in QMIX In this section , we show empirically that the overestimation problem for deep multi-agent Q-learning can be more severe in practice than previously acknowledged . In particular , we demonstrate that state-ofthe-art methods for tackling this issue in the single-agent domain can fail in our multi-agent setting . We investigate the behavior of QMIX in the predator-prey task from the multi-agent particle environments [ 18 ] , where 3 slower predators need to coordinate to capture a faster prey to solve the task . A detailed description of the task is included in Appendix E.1.1 . We consider a fully cooperative setting where the prey is pre-trained by MADDPG [ 18 ] to avoid capture by the predators , and the predators need to learn to cooperate with each other in order to surround and capture the prey . Figures 2 and 3 ( a ) show the performance of QMIX and its estimated values of the joint-action Q-function during training respectively . QMIX suffers from catastrophic performance degradation and a severe overestimation bias , where its value estimates grow without bound . Figures 3 ( b ) and 3 ( c ) illustrate the mean agent-wise utilities Qa and their gradients ∂fs/∂Qa over agents respectively . We see that for QMIX , the gradients ∂fs/∂Qa increase rapidly and continuously during learning . Figure 3 ( d ) shows the learned weight ( matrix ) of the monotonic mixing network fs . The weights similarly grow larger during training and further amplify the overestimation in each Qa when computing Qtot , leading to a severely overestimated joint-action Q-function that continues to grow without bound . To reduce overestimation , Ackermann et al . [ 1 ] extend the state-of-the-art TD3 algorithm [ 8 ] , which addresses overestimation in the single-agent case , to the multi-agent setting . We obtain QMIX ( CDQ ) by applying the key Clipped Double Q-learning technique from TD3 [ 8 ] to per-agent utilities in QMIX for value estimation.2 As shown in Figure 3 ( b ) , QMIX ( CDQ ) slows the increase of agent-wise utilities . However , Figure 3 ( a ) shows that it still does not eliminate the severe overestimation bias in the joint-action Q-function , and subsequently suffers a performance degradation as shown in Figure 2 . This is due to the gradients of the monotonic mixing network , ∂fs/∂Qa , which still continuously increase as shown in Figure 3 ( c ) , leading to the large and increasing value estimates in Figure 3 ( a ) . Another way to avoid overly large value estimates is to limit the gradients themselves . We propose Gradient Regularization ( GradReg ) to prevent the gradients ∂fs/∂Qa from growing too large by regularizing the gradient using a quadratic penalty . Specifically , the loss function of QMIX ( GradReg ) is defined as LGradReg ( θ ) = E ( s , u , r , s′ ) ∼B [ δ2 + λ ( ∂fs/∂Qa ) 2 ] , where δ is the TD error defined in Section 2 and λ is the regularization coefficient . However , although QMIX ( GradReg ) with a tuned coefficient for the regularizer prevents the gradients from growing overly large during the early stage of learning ( Figure 3 ( c ) ) , the agent-wise utility functions grow larger instead as shown in Figure 3 ( b ) . This then still leads to a particularly large value estimate as shown in Figure 3 ( a ) . Finally , a direct combination of both CDQ and GradReg also fails to avoid the problem . From these experiments , we see that even applying the state-of-the-art TD3 [ 8 ] method from the single-agent literature can fail in our multi-agent setting . In addition , it is also insufficient to tackle the problem by regularizing the magnitude of the gradients ∂fs/∂Qa . These results show that mitigating overestimation in MARL is particularly important and challenging , and requires novel solutions . 4 Regularized Softmax ( RES ) Deep Multi-Agent Q-Learning Our analysis shows that preventing gradients from being large as in QMIX ( GradReg ) and/or utilizing clipped double estimators as in QMIX ( CDQ ) are not sufficient to prevent overly large value estimates . It is important to note that these methods take an indirect approach to reduce the overestimation of the joint-action Q-value , and as shown do not solve the severe overestimation problem . Therefore , we propose a novel way to directly penalize large joint-action Q-values to avoid value estimate explosion . Directly penalizing large joint-action Q-values can push their values towards 0 , yielding a large estimation bias . Instead , we introduce a baseline into the penalty to better trade off learning efficiency and stability . Specifically , the new learning objective penalizes joint-action Q-values deviating from a baseline b ( s , u ) , by adding a regularizer to the loss : λ ( Qtot ( s , u ) − b ( s , u ) ) 2 , where we use the mean squared error loss and λ is the regularization coefficient . A potential choice for the baseline is theN -step return , i.e. , b ( s , u ) = ∑N−1 t=0 γ trt+γ N maxu′ Qtot ( sN , u ′ ) . Figure 4 ( a ) shows the value estimates of Regularized ( RE ) -QMIX ( b = N -step return ) using the N -step return as the baseline ( with best N , λ ) . However , as it still involves Qtot , the value estimation can still grow extremely large despite being discounted , yielding estimates that almost coincide with those of QMIX . 2Applying CDQ on Qtot leads to larger value estimates than applying it on Qa ( details are in Appendix A.2 ) . Therefore , we propose to use the discounted return starting from state s as the baseline to regularize learning , i.e. , b ( s , u ) = Rt ( s , u ) = ∑∞ k=0 γ krt+k for the t-th timestep . The regularized learning objective directly penalizes large joint-action Q-values and allows the value estimates to remain grounded by real data . As shown in Figure 4 ( a ) , RE-QMIX effectively avoids particularly large value estimations and stabilizes learning . However , while the regularized learning objective is effective in resolving value explosion , it can still overestimate due to the use of the max operator when computing target value estimates [ 34 ] . Figure 4 ( b ) shows the estimated and true values of REQMIX ( with best λ ) . The estimated values are computed by averaging over 100 states sampled from the replay buffer at each timestep , and we estimate true values by averaging the discounted returns which are obtained by following the greedy policy with respect to the current Qtot starting from the sampled states . As shown in Fig- ure 4 ( b ) , although RE-QMIX avoids unbounded growth of its value estimates , it does not fully avoid the overestimation bias , which still estimates the target value according to the max operator [ 34 ] . To further mitigate overestimation bias in the joint-action Q-function , we adopt the softmax operator , which has shown great potential in reducing overestimation bias in place of the max operator in single-agent domains [ 32 , 26 , 25 ] . Specifically , the softmax operator is defined in Eq . ( 3 ) , where β ≥ 0 is the inverse temperature parameter . smβ , U ( Qtot ( s , · ) ) = ∑ u∈U eβQtot ( s , u ) ∑ u′∈U e βQtot ( s , u′ ) Qtot ( s , u ) . ( 3 ) When β approaches∞ or 0 , softmax reduces to the max or mean operator respectively . Unfortunately , computing Eq . ( 3 ) in the multi-agent case can be computationally intractable as the size of the joint action space grows exponentially with the number of agents . In addition , as the action space in the multi-agent case is much larger than that in the single-agent case , some joint-action Q-value estimates Qtot ( s , u ) can be unreliable due to a lack of sufficient training . Thus , directly taking them all into consideration for computing the softmax operator in Eq . ( 3 ) can result in inaccurate value estimates . We propose a novel method to approximate the computation of the softmax operator efficiently and reliably . Specifically , we first obtain the maximal joint action û = arg maxuQtot ( s , u ) with respect to Qtot according to the individual-global-max ( IGM ) property discussed in Section 2 . Then , for each agent a , we considerK joint actions , by changing only agent a ’ s action while keeping the other agents ’ actions û−a fixed . We denote the resulting action set of agent a by Ua = { ( ua , û−a ) |ua ∈ U } . Finally , we form a joint action subspace Û = U1 ∪ · · · ∪ Un , where each Ua contributes K actions , and use Û in Eq . ( 3 ) for computing the approximate softmax operator . In Theorem 1 , we show that the gap between our approximation in Û and its direct computation in the joint action space U converges to 0 at an exponential rate with respect to β . The proof can be found in Appendix B.1 . Theorem 1 . Let u∗ and u′ denote the optimal joint actions in U and U − Û with respect to Qtot , respectively . The difference between our approximate softmax operator and its direct computation in the whole action space satisfies : ∀s ∈ S , |smβ , Û ( Qtot ( s , · ) ) − smβ , U ( Qtot ( s , · ) ) | ≤ 2Rmax 1−γ |U−Û | |U−Û |+exp ( β ( Qtot ( s , u∗ ) −Qtot ( s , u′ ) ) ) , where |U − Û | denotes the size of the set U − Û . Discussion . Since its computational complexity is linear instead of exponential in the number of agents , our approximation is feasible to compute even when the number of agents grows large , as opposed to the original softmax operator . According to the IGM property , Û consists of joint actions that are close to the maximal joint action û . Thus , it it more likely to contain joint actions with more accurate and reliable value estimates , especially when using -greedy exploration , since they are more likely to be sampled and therefore trained . Due to this reduced reliance on unreliable joint-action Q-values , our approximate softmax operator actually outperforms its counterpart using a direct computation in the joint action space U , as well as a random sampling scheme , as shown in Section 5.1.3 . The full algorithm for our approximate softmax is in Appendix B.2 . As shown in Figure 4 ( b ) , RES ( Regularized Softmax ) -QMIX fully addresses the overestimation bias and achieves better value estimates than RE-QMIX.3 The loss of Regularized Softmax ( RES ) Deep Multi-Agent Q-Learning LRES ( θ ) is defined as : E ( s , u , r , s′ ) ∼B [ δ2smβ + λ ( Qtot ( s , u ) −Rt ( s , u ) ) 2 ] , ( 4 ) where δsmβ = r+ γsmβ , Û ( Q̄tot ( s ′ , · ) ) −Qtot ( s , u ) , and λ denotes the coefficient of the regularizer . To better understand the effectiveness of our method , we provide a theoretical justification that connects the loss function of RES and a new Bellman operator in Theorem 2 . Theorem 2 . Given the same sample distribution , the update of the RES method is equivalent to the update using L ( θ ) = E ( s , u , r , s′ ) ∼B [ ( y −Qtot ( s , u ) ) 2 ] with learning rate ( λ+ 1 ) α , which estimates the target value according to y = r+γsmβ , Û ( Q̄tot ( s ′ , · ) ) λ+1 + λRt ( s , u ) λ+1 . Its proof is in Appendix C. In the special case where λ = 0 , it is equivalent to changing the max operator to our approximate softmax in the target value estimation . When λ > 0 , it can be thought of as learning with a weighted combination of the 1-step TD target with our approximate softmax and the discounted return , which allows the bootstrapping target to remain grounded by real data.4 Let T be the value estimation operator which estimates the value of the next state s′ . From Theorem 2 , we have TQMIX = maxu′ Q̄tot ( s′ , u′ ) , TRE-QMIX = maxu′ Q̄tot ( s ′ , u′ ) +λRt+1 ( s ′ ) λ+1 , and TRES-QMIX = smβ , Û ( Q̄tot ( s ′ , · ) ) +λRt+1 ( s′ ) λ+1 . We now analyze the relationship between the value estimation bias of these operators in Theorem 3 ( the proof is in Appendix D ) , and show that RES-QMIX can reduce the overestimation bias when built upon QMIX . It is also worth noting that since RES is general , it can be readily applied to other deep multi-agent Q-learning algorithms , as investigated in Section 5.1.4 . Theorem 3 . Let B ( T ) = E [ T ( s′ ) ] −maxu′ Q∗tot ( s′ , u′ ) be the bias of value estimates of T and the true optimal joint-action Q-function Q∗tot . Given the same assumptions as in [ 37 ] for the joint-action Q-function , where there exists some V ∗tot ( s ′ ) = Q∗tot ( s ′ , u′ ) for different joint actions , ∑ u′ ( Q̄tot ( s ′ , u′ ) − V ∗tot ( s′ ) ) = 0 , and 1|U | ∑ u′ ( Q̄tot ( s ′ , u′ ) − V ∗tot ( s′ ) ) 2 = C ( C > 0 ) with Q̄tot an arbitrary joint-action Q-function , then B ( TRES-QMIX ) ≤ B ( TRE-QMIX ) ≤ B ( TQMIX ) . From the proof of Theorem 3 and Figure 4 ( a ) , we find that RE-QMIX can reduce the overestimation bias of QMIX following the same standard assumptions in [ 37 , 32 ] . In addition , Theorem 3 also shows that the bias of value estimates of RES-QMIX is no larger than that of RE-QMIX , which validates our RES method .

Summary: The paper addresses overestimation in return estimator in the context of multi-agent reinforcement learning, while claiming the overestimation could be more severe in multi-agent than in single-agent setting. The authors propose an update scheme to penalize large joint-action Q values that deviate from a baseline through the use of regularizer term in the Bellman update; and include an approximated softmax operator to further improve the overestimation issue. The approximated softmax operator is introduced to reduce computational complexity of employing softmax in multi-agent scenario.