SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} _ {\mathrm {e g o}} = - \left(\log \pi^ {\mathrm {e g o}} \left(\boldsymbol {u} ^ {\star} \mid s\right) + \lambda \log \pi^ {\mathrm {e g o}} \left(\hat {\boldsymbol {u}} ^ {\star} \mid s\right)\right), \tag {9}
$$
where the exploration coefficient  $\lambda \in [0,1]$  is annealed linearly during the exploration phase. The choice of hyperparameter  $\lambda$  requires a compromise between exploration and exploitation. Note that the dual self-awareness framework and the anti ego exploration method are NOT two separate contributions. The latter is customized for the former, and it is only used to solve the convergence difficulties caused by the explicit sampling procedure in DAVE. By incorporating the anti-ego exploration module, the dual self-awareness framework with low  $M$  can also fully explore the action space and significantly avoid becoming stuck in bad local optima. The whole workflow of the anti-ego exploration mechanism is shown in Appendix B.
Our proposed DAVE can be applied to most existing value decomposition methods, as it simply adds an extra ego policy network to the conventional value decomposition framework. We believe that the IGM-free DAVE framework can utilize richer function classes to approximate joint state-action value functions and thus be able to solve the non-monotonic multi-agent cooperation problems.
In this section, we apply the DAVE framework to standard value decomposition methods and evaluate its performance on both didactic problems and complex Dec-POMDPs. First, we will compare the performance of the DAVE variant with QPLEX, Weighted QMIX, QTRAN, FACMAC-nonmonotonic [25], MAVEN, and the vanilla QMIX. Note that most baselines are directly related to IGM. Then the contribution of different modules in the framework and the impact of hyperparameters will be investigated. Except for the new hyperparameters introduced in the DAVE variant, all other hyperparameters are consistent with the original method. Unless otherwise stated, DAVE refers to the DAVE variant of QMIX, which enhances representational capacity while ensuring fast convergence. For the baseline methods, we obtain their implementation from the released code. The details of the implementation and all environments are provided in Appendix C and Appendix D, respectively.
5.1 Single-State Matrix Game
Certain types of tasks in multi-agent cooperation problems cannot be handled under the IGM assumptions, such as the two matrix games we will introduce next. These two matrix games capture very simple cooperative multi-agent tasks and are shown in Figure 3. These payoff matrices of the two-player three-action matrix games describe scenarios that IGM-based value decomposition methods cannot address effectively. The variable  $k$  is related to difficulty. In
matrix game I with  $k \in [0,8)$ , agents need to choose the same action to get the reward and the optimal joint action is  $(A,A)$ . Furthermore, the closer  $k$  is to 8, the more challenging it becomes to jump out of the local optimum. Similarly,  $k \geq 0$  and  $(A,C)$  is the optimal joint action in matrix
(a) Matrix game I.
(b) Matrix game II.
Figure 3: Payoffs of the two matrix games.
Figure 4: The learning curves of DAVE and other baselines on the matrix games. Note that the ordinates are non-uniform.
game II. Nevertheless, as  $k$  increases, so does the risk of selecting the optimal action  $(A, C)$ , which leads agents in the IGM-based method to prefer the suboptimal joint action  $(B, B)$  corresponding to the maximum expected return. Therefore, both matrix games are challenging for traditional value factorization methods. To disregard the influence of exploration capabilities, we implement various algorithms under uniform visitation, which means that all state-action pairs are explored equally. We also evaluate the performance of popular policy-based MARL algorithms. Results in Appendix D.2 demonstrate that although they do not explicitly introduce IGM, policy-based methods still perform poorly on these two simple tasks.
The results of all algorithms on the two matrix games with varying  $k$  are presented in Figure 4. We limit the number of interactions between agents and the environment to 400 across all environments. All baselines find it challenging to discover the optimal joint action with such limited data. Compared to other algorithms, DAVE can quickly converge to the optimal solution in all situations and exhibits robust performance, showing little variation as task difficulty increases. CW-QMIX can converge to the optimal joint solution in all cases except matrix game I with  $k = 7.5$ . QPLEX performs better in matrix game II than in matrix game I, likely due to its sensitivity to the penalty of non-cooperation. In addition, as the task difficulty increases, both CW-QMIX and QPLEX show a significant drop in performance. Although QTRAN can converge to the optimal solution in matrix game I with  $k = 0$ , it may require more interactive samples for further learning. The remaining algorithms can converge to the optimal joint action  $(A, C)$  or the suboptimal joint action  $(C, A)$  in matrix game II with  $k = 0$ , but in the more difficult matrix game II can only converge to the joint action  $(B, B)$ . We add additional action information to the mixing network of QMIX, similar to that in DAVE, but QMIX still cannot converge to the optimal solution. In summary, DAVE can easily solve the above single-state matrix games and outperforms other IGM-based variants and the basic algorithm QMIX by a significant margin.
5.2 Multi-Step Matrix Game
Next, we evaluate the performance of each algorithm in the multi-step matrix game. The environment involves two agents, each with two actions. The initial state is non-monotonic, and agents terminate immediately after receiving a zero reward. If the game is not terminated after the first step, agents need to continue to repeat the last joint action to obtain a non-zero reward. According to Figure 5, agents repeat either the upper-left joint action or the lower-right joint action eight times, resulting in two distinct terminal states. The optimal policy is
for agents to repeat the upper-left joint action in the initial state and choose the lower-right joint action in the terminal state, which yields the highest return of 13. Local observations of agents in this
Figure 5: Left: Illustrations of the multi-step matrix game. Right: Performance over time on the multi-step matrix game.
Figure 6: Performance comparison with baselines in different scenarios.
environment include the current step count and the last joint action. The protracted decision-making process requires agents to fully explore the state-action space to find the optimal policy. Therefore, this task can also test the exploration capacity of the algorithm.
Figure 5 shows the performance of DAVE and other baselines on the multi-step matrix game. Only DAVE successfully learned the optimal policy, while most other algorithms were trapped in the suboptimal policy with payoff 10. QPLEX can occasionally converge to the optimal policy, but in most cases it still cannot learn the optimal solution. Besides, to examine the contribution of the anti-ego exploration to the DAVE framework, we implemented DAVE without exploration in this environment. The experimental results show that DAVE without anti-ego exploration still performs better than other baselines, but is significantly inferior to DAVE. Therefore, for some complex tasks, the anti-ego exploration mechanism can assist the agent in exploring novel state-action pairs and avoiding being restricted to suboptimal solutions.
5.3 StarCraft II
SMAC [29] is the most popular experimental platform for multi-agent micromanagement, based on the famous real-time strategy game StarCraft II. SMAC offers a diverse range of mini-scenarios, including homogeneous, heterogeneous, symmetric, and asymmetric problems. The goal of each task is to direct agents to defeat enemy units controlled by built-in heuristic AI. Agents can only access information within their own visual range and share a reward function. So we assess the ability of DAVE to solve complex multi-agent cooperation problems in this environment.
The version of StarCraft II used in this paper is 4.10 and performance is not comparable between versions. We test the proposed method in different scenarios. In addition to the above-mentioned methods, baselines also include IGM-Free QMIX that simply does not take the absolute value of the weight of the mixing network and keeps others unchanged. Both IGM-Free QMIX and FACMAC-nonmonotonic are IGM-free methods, but do not have convergence guarantees. The performance comparison between above two IGM-free methods and DAVE that is also IGM-free can reflect the necessity of our proposed ego policy network. In addition, to balance exploration and exploitation, we disable the anti-ego exploration mechanism in some easy scenarios and use it only in hard scenarios.
The experimental results are shown in Figure 6. Undoubtedly, IGM-Free QMIX and FACMAC-nonmonotonic fail to solve almost all tasks. Without the IGM assumption, QMIX is unable to find the correct update direction. Therefore, the assistance of the ego policy network in the DAVE framework is essential for the convergence of IGM-free algorithms. QPLEX performs best in some easy scenarios but fails to solve the task in most hard scenarios. Weighted QMIX performs well in the MMM2 scenario but has mediocre performance in most other scenarios. Although DAVE is not the best in easy maps, it still shows competitive performance. In hard scenarios, such as  $5m\_ vs\_ 6m$ ,  $8m\_ vs\_ 9m$  and  $10m\_ vs\_ 11m$ , DAVE performs significantly better than other algorithms. We believe
DAVE QPLEX OW-QMIX CW-QMIX MAVEN QTRAN FACMAC (Non-Mono) IGM-Free QMIX QMIX
Figure 7: Influence of the sample size  $M$  for DAVE.
Figure 8: Results of DAVE with different  $\lambda_{\mathrm{init}}$ . The action space for the three scenarios gradually increases from left to right.
that it may be that the characteristics in these scenarios are far from the IGM assumptions, resulting in low sample efficiency of conventional value decomposition methods. Moreover, because DAVE is IGM-free, it can guide the agent to learn the excellent policy quickly without restriction. Especially in the  $6h\_vs\_8z$  scenario, only DAVE got a non-zero median test win ratio. In summary, even if the IGM principle is completely abandoned, DAVE can still successfully solve complex tasks and even outperforms existing IGM-based value decomposition algorithms in certain challenging scenarios. MORE experimental results on other complex environments such as SMACv2 [6] and Multi-Agent MuJoCo [25] can be found in Appendix D. These results demonstrate the excellent generalization of DAVE to different algorithms and environments.
Finally, we evaluate the performance of DAVE under different hyperparameter settings in SMAC scenarios. We focus on the number of samples  $M$  and the initial value of the exploration coefficient  $\lambda_{\mathrm{init}}$ . According to Appendix A, increasing the number of samples  $M$  improves the probability of drawing the optimal joint action. Nonetheless, due to time and space complexity constraints, it is crucial to find an appropriate value for the hyperparameter  $M$ . Then we test DAVE with different  $M$  values in three scenarios and present results in Figure 7. The results indicate that a larger  $M$  leads to a faster learning speed for DAVE, which aligns with our intuition and proof. DAVE with  $M = 1$  fails in all scenarios. When  $M$  increases to 10, the performance of DAVE is already considerable. However, as  $M$  increases further, the performance enhancement it provides diminishes.
The equilibrium between exploration and exploitation is directly related to the initial value of the exploration coefficient  $\lambda_{\mathrm{init}}$ . The final value  $\lambda_{\mathrm{end}}$  and the annealing period also affect the exploration, but only  $\lambda_{\mathrm{init}}$  is discussed in this paper.  $\lambda_{\mathrm{end}}$  is generally set to 0, so DAVE is only affected by anti-ego exploration in the early stages of training. Figure 8 shows the learning curve of DAVE under different  $\lambda_{\mathrm{init}}$ .  $\lambda_{\mathrm{init}} = 0$  means that DAVE does not perform anti-ego exploration. The addition of anti-ego exploration in easy scenarios will prevent the algorithm from exploiting better in the early stages, which will significantly slow down the training speed. Conversely, DAVE with high  $\lambda_{\mathrm{init}}$  values can achieve superior performance in hard scenarios. So a moderately high  $\lambda_{\mathrm{init}}$  is beneficial for DAVE to visit key state-action pairs and converge to the optimal solution. In summary, appropriate values of  $M$  and  $\lambda_{\mathrm{init}}$  are indispensable for DAVE.
It would be valuable to explore the performance of DAVE with a monotonic mixing network. As shown in Figure 9 and Figure 10, DAVE with a monotonic mixing network still cannot converge to the global optimal solution in matrix games. Since the ego policy is updated by an explicit search procedure, DAVE with a monotonic mixing network has more chances to converge to the optimal solution than vanilla QMIX. However, limited by fewer function families that can be represented, the performance of DAVE with a monotonic mixed network is much worse than that of DAVE and is unstable. In addition, in the complex SMAC environment, the performance of DAVE with a monotonic mixing network is similar to or even better than that of QMIX, but worse than that of
Figure 9: The learning curves of vanilla DAVE and DAVE with a monotonic mixing network on the matrix games.
Figure 10: The learning curves of vanilla DAVE and DAVE with a monotonic mixing network on SMAC.
vanilla DAVE. This is also substantial proof that the unconstrained factored mixing network indeed enhances performance.
Inspired by the concept of dual self-awareness in psychology, we propose the IGM-free value decomposition framework DAVE in this paper. By dividing the agent network into two parts, ego policy and alter ego value function, DAVE is the first value decomposition method that completely abandons the IGM assumption. We find that the larger the number of samples in the explicit search procedure, the better the algorithm converges to the global optimum, but this will lead to high computational complexity. Therefore, we propose the anti-ego exploration mechanism, which can effectively prevent DAVE from falling into local optima, especially in problems with large action spaces. Experimental results in some didactic problems and complex environments show that DAVE outperforms existing IGM-based value decomposition methods. We believe that DAVE provides a new perspective on addressing the problems posed by the IGM assumption in value decomposition methods. In our future research, we intend to concentrate on how to choose the exploration coefficients in an adaptive manner. Further study of the issue would be of interest.
This project was supported by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No. XDA27050100.
[1] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Jozefowicz, Scott Gray, Catherine Olsson, Jakub W. Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. ArXiv, abs/1912.06680, 2019.
[34] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech M. Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-decomposition networks for cooperative multi-agent learning. In Adaptive Agents and Multi-Agent Systems, 2017.
[35] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 16:285-286, 2005.
[36] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX: duplex dueling multiagent q-learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[37] Jianhong Wang, Wangkun Xu, Yunjie Gu, Wenbin Song, and Tim C. Green. Multi-agent reinforcement learning for active voltage control on power distribution networks. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 3271-3284, 2021.
[38] Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. Influence-based multi-agent exploration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[39] Zhiwei Xu, Yunpeng Bai, Dapeng Li, Bin Zhang, and Guoliang Fan. SIDE: state inference for partially observable cooperative multi-agent reinforcement learning. In Piotr Faliszewski, Viviana Mascardi, Catherine Pelachaud, and Matthew E. Taylor, editors, 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, New Zealand, May 9-13, 2022, pages 1400-1408. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2022.
[40] Zhiwei Xu, Yunpeng Bai, Bin Zhang, Dapeng Li, and Guoliang Fan. HAVEN: hierarchical cooperative multi-agent reinforcement learning with dual coordination mechanism. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 11735-11743. AAAI Press, 2023.
[41] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611-24624, 2022.
[42] Bin Zhang, Lijuan Li, Zhiwei Xu, Dapeng Li, and Guoliang Fan. Inducing stackelberg equilibrium through spatio-temporal sequential decision-making in multi-agent reinforcement learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China, pages 353-361. ijcai.org, 2023.
Proposition A.1. As long as the ego policy assigns non-zero probabilities to all actions, this method will approach the objective described by Equation (1) as the number of samples  $M$  increases.
Proof. Let  $(u^a)^*$  denote the individual actions corresponding to the global optimal joint state-action value function under state  $s$ , and the optimal joint action is  $\pmb{u}^{*} = \{(u^{1})^{*},\dots ,(u^{n})^{*}\}$ . Then the probability that the sampling procedure draws  $\pmb{u}^{*}$  is expressed as:
$$
\begin{array}{l} p \left(\boldsymbol {u} ^ {*}\right) = 1 - \left(1 - \pi^ {\text {e g o}} \left(\boldsymbol {u} ^ {*} \mid s\right)\right) ^ {M} \\ = 1 - \left(1 - \prod_ {i = 1} ^ {n} \pi_ {a} ^ {\mathrm {e g o}} \left(\left(u ^ {a}\right) ^ {*} \mid \tau^ {a}\right)\right) ^ {M}, \\ \end{array}
$$
where  $\pi_a^{\mathrm{ego}}(\cdot |\cdot)\in (0,1)$  is true for any action. So the second term  $(1 - \pi^{\mathrm{ego}}(\pmb {u}^{*}|s))^{M}\in (0,1)$  in the equation decreases as  $M$  increases, indicating that  $p(\pmb {u}^{*})$  is positively correlated with the sample size  $M$ .
B Details of the Ego Policy Update
Figure 11: Diagram of the update of ego policy, where  $U^{\mathrm{ego}} \coloneqq \{ \pmb{u}_i^{\mathrm{ego}} \sim \pmb{\pi}^{\mathrm{ego}}(s) \}_{i=1}^M$  and  $\hat{\pmb{U}}^{\mathrm{ego}} \coloneqq \{ \hat{\pmb{u}}_i^{\mathrm{ego}} \sim \hat{\pmb{\pi}}^{\mathrm{ego}}(s) \}_{i=1}^M$ . Dots indicate different joint actions. The selected actions are  $\pmb{u}^\star = \underset{\pmb{u}^{\mathrm{ego}}} {\arg \max } Q_{\mathrm{tot}}^{\mathrm{alter}}(s, \pmb{u}^{\mathrm{ego}})$  and  $\hat{\pmb{u}}^\star = \underset{\hat{\pmb{u}}^{\mathrm{ego}}} {\arg \max } \mathcal{L}_{\mathrm{recon}}(s, \hat{\pmb{u}}^{\mathrm{ego}})$ .
The update process of ego policy is shown in Figure 11. During the initialization of neural networks, the state-action value corresponding to the optimal joint action may be underestimated, making it difficult for the ego policy to choose and then correct it, especially in problems with large action spaces. So we use the Softmin operator to generate an anti-ego policy so that low-probability actions are more likely to be selected. However, not all low-probability joint actions are unexplored and undervalued; some are sub-optimal actions that have been exhaustively explored (orange dots in Figure 11). In order to select only unexplored joint actions, we distinguish the above two through the auto-encoder. Figure 5 and Figure 8 demonstrate the contribution of the anti-ego exploration mechanism.
C.1 Algorithmic Description
The pseudo-code of DAVE is shown in Algorithm 1.
Algorithm 1 Training Procedure for DAVE
Hyperparameters: Sample size  $M$ , discount factor  $\gamma$ , exploration coefficients  $\lambda_{\mathrm{init}}$   
Initialize the parameters of the neural networks shown in Figure 2  
1: for each episode do  
2: Get the global state  $s_1$  and the local observations  $z_1 = \{z_1^1, z_1^2, \ldots, z_1^n\}$  of all agents  
3: for  $t \gets 1$  to  $T - 1$  do  
4: for  $a \gets 1$  to  $n$  do  
5: Select action  $u_t^a$  according to the ego policy  $\pi_a^{\mathrm{ego}}$   
6: end for  
7: Carry out the joint action  $u_t = \{u_t^1, \ldots, u_t^n\}$   
8: Get the global reward  $r_{t+1}$ , the next local observations  $z_{t+1}$ , and the next state  $s_{t+1}$   
9: end for  
10: Store the episode in the replay buffer  $\mathcal{D}$   
11: Sample a batch of episodes  $\mathcal{B} \sim \mathrm{Uniform}(\mathcal{D})$   
12: Sample and obtain the joint action set  $U^{\mathrm{ego}} := \{\pmb{u}_i^{\mathrm{ego}} \sim \pmb{\pi}^{\mathrm{ego}}(s)\}_{i=1}^{M}$  for each trajectory in  $\mathcal{B}$   
13: Update the parameters of the alter ego value function and the IGM-free mixing network according Equation (5)  
14: Obtain the anti-ego policies  $\hat{\pi}_a^{\mathrm{ego}}$  of each agent  
15: Sample and obtain  $\hat{U}^{\mathrm{ego}} := \{\hat{\pmb{u}}_i^{\mathrm{ego}} \sim \hat{\pi}^{\mathrm{ego}}(s)\}_{i=1}^{M}$  by sampling  $M$  times from the anti-ego policy for each state  $s$  in  $\mathcal{B}$   
16: Find the most novel joint action  $u^*$  for each state  $s$  in  $\mathcal{B}$  according Equation (8)  
17: Update the parameters of the ego policy according Equation (9)  
18: Update the parameters of the auto-encoder according Equation (6)  
19: Update the parameters of the target network periodically  
20: end for
Unless otherwise stated, the hyperparameter settings in different environments are shown in Table 1, which are exactly the same as the settings in PyMARL<sup>1</sup>. All experiments in this paper are run on Nvidia GeForce RTX 3090 graphics cards and Intel(R) Xeon(R) Platinum 8280 CPU. For baselines, exploration is performed during training using independent  $\epsilon$ -greedy action selection.  $\epsilon$  is annealing linearly from 1.0 to 0.05 and the annealing period of  $\epsilon$  is 50000 steps. All experiments in this paper are carried out with 6 random seeds in order to avoid the effect of any outliers.