SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Although containing two mixing networks, the total mixing-net size of GoMARL is smaller than the commonly used single monotonic mixing network [27], as verified in Section 5.1. This is mainly attributed to the input dimension reduction. The monotonic mixing network takes the global state  $s$  as input of all the hypernetworks. In contrast, we take specific group information (i.e., individual group-related info  $e$ , group state  $s^g$ , and state  $s$  is only used in one hypernetwork). This parameter reduction offsets the increase of an extra mixing network. Compared with flat value factorization methods learning only with the global state, our method allows fine-grained learning guided by more direct signals embedded in the policy gradients, facilitating intra- and inter-group cooperation.
Baselines. We compare GoMARL with prominent baselines to verify its effectiveness and efficiency. Hu et al. [8] fairly compared existing MARL methods without code-level optimizations and reported that QMIX [27] and QPLEX [38] are the top two value factorization methods. The authors also finetuned QMIX (denoted as Ft-QMIX in our paper), which attains higher win rates than the vanilla QMIX. VAST [24] learns value factorization for sub-teams based on apriori setting on group number. Therefore, we compare GoMARL with Ft-QMIX, QPLEX, and VAST to show its performance as a value factorization method. The baselines also include role-based methods ROMA [39] and the representative credit assignment method RIIT [8]. The latter combines effective modules of noticeable methods, and comparing them can further illustrate the superiority of GoMARL.
Experimental setup. All the methods are trained with 8 parallel runners for 10M steps and are evaluated every 10K steps with 32 episodes. We report the 1st, median, and 3rd quartile win rates across 5 random seeds. Please refer to Appendix B for experimental setup details.
5.1 Performance on SMAC
Methods are evaluated on six challenging SMAC maps. The chosen maps involve homogeneous and heterogeneous teams with asymmetric battles, allowing a holistic study of all methods.
Overall performance. The performance comparison of all the methods in the six Hard and Super Hard SMAC maps is shown in Figure 5. As the results show, each baseline method only achieves satisfactory performance on some of the tasks with specific properties they specialize in; e.g., RIIT performs well on MM2 but converges much slower in other tasks; QPLEX's leaning is not efficient in  $8\mathrm{m\_vs\_9m}$  and corridor. Ft-QMIX significantly outperforms the vanilla QMIX and has more efficient learning than other baselines. GoMARL has a similar performance as Ft-QMIX in  $8\mathrm{m\_vs\_9m}$  and corridor. However, the superiority of GoMARL can be clearly validated in all the other maps.
Parameter size for value mixing. Many methods use larger mixing networks with stronger expressiveness and fitting abilities to obtain superior performance. However, they often fail when compared to baselines that use a mixing net of the same size [8]. As shown in Table 1, our dual-hierarchical mixing architecture has fewer parameters when there are a large number of agents (when  $n > 5$ ). VAST is not compared since it utilizes a linear summation of local utilities. The results of GoMARL are the
Figure 5: Comparison of GoMARL against baseline algorithms on six SMAC maps.
average of the five runs since each run may learn a slightly different grouping with various group numbers. GoMARL outperforms other methods despite using fewer mixing parameters, highlighting its inherent superiority over methods relying on stronger mixing networks.
Table 1: Size comparison of all methods' mixing network(s)
Maps	(Ft-)QMIX	QPLEX	ROMA	RIIT	GoMARL
3s_vs_5z	21.601K	72.482K	13.281K	37.986K	26.530K
5m_vs_6m	31.521K	107.574K	25.377K	51.362K	31.554K
8m_vs_9m	53.313K	197.460K	63.393K	93.986K	51.427K
corridor	68.929K	303.808K	81.537K	122.882K	53.859K
MMM2	84.929K	342.248K	134.401K	177.282K	74.244K
3s5z_vs_3s6z	63.105K	243.156K	81.345K	118.466K	61.028K
Component analysis and ablation study. We next conduct detailed component studies to analyze how GoMARL improves efficiency and enhances performance. GoMARL contains three key components: (1) an automatic grouping mechanism that progressively divides the team into proper groups; (2) specialized agent networks that generate diversified policies while sharing all the parameters; and (3) sufficiently informative signals integrated into the gradients to promote efficient cooperation. We respectively study each module on three Super Hard maps to show how they influence performance.
(1.1) Ablation study of the grouping mechanism. We validate our grouping mechanism by comparing GoMARL with other intuitive alternatives. All methods utilize the same architecture as GoMARL to reflect how grouping itself influences performance. As shown in the top row of Figure 6, setting all agents as a group in corridor converges faster but has significant variances. It may be due to the hard exploration of efficient cooperation without grouping guidance. Another two intuitive groupings, each agent a group and an equal division into two groups  $\{\{a_1,a_2,a_3\} ,\{a_4,a_5,a_6\} \}$ , have minor variance. However, their learning efficiency is affected since inappropriate groupings fail to promote cooperative behaviors. MM2 contains heterogeneous agents; thus, a natural grouping is to keep the agents of the same type in one group. This natural grouping outperforms most alternatives but is inferior to our dynamic grouping adjustment. 3s5z_vs_3s6z is another scenario with heterogeneous agents, and the natural grouping of setting homogeneous agents as a group is also studied. It performs nearly the same as GoMARL because our mechanism learns the grouping precisely according to the agents' type in this scenario. We also analyze this map with a grouping containing three groups  $\{\{a_1,a_4,a_5\} ,\{a_2,a_6,a_7\} ,\{a_3,a_8\} \}$  (1s2z, 1s2z, 1s1z); however, this balanced grouping fails to form effective collaboration. We can see from these results that appropriate grouping facilitates efficient cooperation and accelerates learning. Our grouping mechanism automatically learns adaptive grouping in different tasks and assists GoMARL with superior performance and efficiency.
(1.2) Learned grouping analysis. To demonstrate whether the learned grouping makes sense, we further visualize the trained strategy in a corridor battle, as illustrated in Figure 7. Six allied Zealots
Figure 6: Ablations of the grouping mechanism (top row) and the group-related signals (bottom).
Figure 7: The learnt groups  $\{\{a_1, a_2\}, \{a_3, a_4, a_5, a_6\}\}$  is explicable to the corridor combat situation.
fight twenty-four Zerglings on this Super Hard map. The massive disparity in unit numbers between the two sides implies that the whole team must refrain from launching an attack together. The only winning strategy is to sacrifice a small number of agents who leave the team and attract the attention of most enemies. Taking this opportunity, our large force eliminates the rest of the enemies. The surviving agents then use the same tactic to keep attracting several enemies and kill them together. In this battle, Agent 1 and 2 sacrifice themselves to attract most enemies and bring enough time for the team to eliminate the remaining enemies. Other agents fight as a subgroup and successfully kill all the surviving enemies. GoMARL learns a double-group setting in this battle, where Agent 1 and 2 are in the same group while the others are set in another group. This grouping is explicable in light of the combat situation, and this reasonable grouping guidance contributes to our superior performance.
(2) Component study of the specialized agents. We transplant our specialized agent network (SAN) into other baselines to verify module effectiveness. ROMA is not included since its actors are produced by its learned roles, and the replacement will invalidate the method. As shown in Figure 8, all methods are improved when equipped with SAN. However, even with SAN, all baselines fail to surpass GoMARL.
As shown in the learning curves in Appendix C, although dynamic grouping may reduce learning efficiency in the early stage of training, the learned grouping will significantly accelerate learning in the later stage. Therefore, the automatic grouping module and SAN are both crucial to GoMARL.
(3) Ablation study of the informative group-related signals. Our method models individual group-related information  $e$  and group state  $s^g$  to integrate them into the gradients. The former adjusts local utilities for cooperative behaviors and realizes policy diversity, while the latter fosters efficient group cooperation. We ablate them respectively to validate their effectiveness. Concretely, we replace the group state  $s^g$  with the full state  $s$  and remove the group-related information  $e$  to degenerate the specialized agent networks into vanilla agent networks. As shown in the bottom row of Figure 6, both signals greatly improve learning efficiency. On the one hand, the agent info  $e$  carrying group-related information is trained to encourage inter-group diversity, enabling extensive cooperative exploration and learning acceleration. On the other hand, the fused  $s^g$  summarizes informative group-related status that facilitates efficient group-wise coordination. Although the global state  $s$  provides complete
Figure 8: Method performance with and without SAN in 3s5z_vs_3s6z after training 8M steps.
, although dynamic grouping may reduce learning. Fixed grouping will significantly accelerate learning in a long module and SAN are both crucial to GoMARL.
Figure 9: Performance comparison with baselines in three Google Research Football scenarios.
Figure 10: The learned groups  $\{\{a_1, a_2\}, \{a_3, a_4\}\}$  is explicable to a counterattack situation.
knowledge, it is burdensome for agents to extract effective guidance that promotes policy learning. Therefore, utilizing our specific group-wise signals  $e$  and  $s^g$  is more efficient than the global state  $s$ .
5.2 Performance on Google Research Football
We also test GoMARL on three challenging Google Research Football (GRF) offensive scenarios. Agents in GRF need to coordinate timing and positions for organizing offense to seize fleeting opportunities, and only scoring leads to rewards. Therefore, the GRF tasks are more complicated than the SMAC battles, and the comparison in GRF is a secondary proof of our method's effectiveness.
Performance. Figure 9 shows the performance of all methods in GRF. GoMARL, Ft-QMIX, and QPLEX all perform well in pass_and_shoot_with_keeper. However, the advantages of our method become increasingly evident in the other two environments requiring more coordination. Only GoMARL achieves over  $50\%$  of the score reward in all scenarios. The superior performance with significant efficiency in the second testbed further demonstrates the transferability of our method.
Visualization. The trained strategy in counterattack_easy is visualized to validate if the learned group division makes sense. GoMARL learned a reasonable grouping  $\mathcal{G} = \{\{a_1, a_2\}, \{a_3, a_4\}\}$  for this complex goal, in which the first group brought the ball into the penalty area through smooth coordination, while the second group created two shoots and the final goal through skillful cooperation, as shown in Figure 10. Appendix D provides a detailed discussion of this visualization.
Grouping is essential to the efficient cooperation of multi-agent systems. Instead of utilizing apriori knowledge, this paper proposes an automatic grouping mechanism that gradually learns reasonable grouping as training proceeds. Based on the dynamic group division, we further model informative group-related signals to achieve fine-grained value factorization and encourage policy specialization, promoting efficient intra- and inter-group coordination. With these novelties, our method GoMARL achieves impressive performance with high efficiency on the SMAC and GRF benchmarks.
The automatic grouping mechanism of GoMARL is currently grounded in the value decomposition process, rendering it inapplicable to policy-based methods. Our future research will delve into automatic grouping based on policy gradients to enhance the learning efficiency of the policy-based methods. Furthermore, the current version of GoMARL is only designed for one-stage and one-level grouping, and it possesses the potential to achieve finer granularity in group division. We will investigate multi-stage and multi-level grouping (i.e., group-within-a-group) in the future to pursue better performance and higher learning efficiency in more complex environments.
This work is supported in part by the National Key Research and Development Program of China under Grant No. 2020AAA0103401; in part by the Natural Science Foundation of China under Grant 62076238, Grant 62222606, and Grant 61902402; and in part by the China Computer Federation (CCF)-Tencent Open Fund.
[1] Noelie Bonjean, Wafa Mefteh, Marie-Pierre Gleizes, Christine Maurel, and Frédéric Migeon. Adelfe 2.0. Handbook on Agent-Oriented Design Processes, pages 19-63, 2014.
[2] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, pages 2974-2982, 2018.
[3] Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: A survey. Artificial Intelligence Review, pages 1-49, 2021.
[4] Jayesh K. Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multi-Agent Systems, pages 66-83, 2017.
[5] Tarun Gupta, Anuj Mahajan, Bei Peng, Wendelin Böhmer, and Shimon Whiteson. Uneven: Universal value exploration for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 3930-3941. PMLR, 2021.
[6] David Ha, Andrew Dai, and Quoc V. Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016.
[7] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E. Taylor. A survey and critique of multiagent deep reinforcement learning. In International Conference on Autonomous Agents and Multi-Agent Systems, volume 33, pages 750-797, 2019.
[8] Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih-wei Liao. Riit: Rethinking the importance of implementation tricks in multi-agent reinforcement learning. arXiv preprint arXiv:2102.03479, 2021.
[9] Maximilian Hüttenrauch, Adrian Šošić, and Gerhard Neumann. Guided deep reinforcement learning for swarm systems. arXiv preprint arXiv:1709.06011, 2017.
[10] Shariq Iqbal, Robby Costales, and Fei Sha. Alma: Hierarchical learning for composite multiagent tasks. In Advances in Neural Information Processing Systems, volume 35, pages 7155-7166, 2022.
[11] Shariq Iqbal, Christian A. Schroeder de Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4596-4606. PMLR, 2021.
[12] Shariq Iqbal and Fei Sha. Actor-attention-critic for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 2961–2970. PMLR, 2019.
[13] Raphaël Jeanson, Penelope F. Kukuk, and Jennifer H. Fewell. Emergence of division of labour in halictine bees: Contributions of social interactions and behavioural variance. Animal Behaviour, 70(5):1183-1193, 2005.
[14] Jiechuan Jiang and Zongqing Lu. The emergence of individuality. In International Conference on Machine Learning, pages 4992-5001. PMLR, 2021.
[15] Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82-94, 2016.
[49] Yifan Zang, Jinmin He, Kai Li, Haobo Fu, Qiang Fu, and Junliang Xing. Sequential cooperative multi-agent reinforcement learning. In International Conference on Autonomous Agents and Multi-Agent Systems, pages 485-493, 2023.
[50] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pages 321-384, 2021.
[51] Meng Zhou, Ziyu Liu, Pengwei Sui, Yixuan Li, and Yuk Ying Chung. Learning implicit credit assignment for cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pages 11853-11864, 2020.
A The Insight of the Group Shifting Scheme
The proposed automatic grouping mechanism dynamically adjusts the group division as the training proceeds. In the beginning, all agents belong to one group. As introduced in Section 4.1, we examine the learned  $\mathbf{w}_1^i$  of each agent every  $c$  timesteps to check if the grouping needs adjustment. If there are agents who contribute little to their group, it indicates that these agents do not belong to their current group. All these selected agents are assigned to the following group until they appropriately contribute to where they belong. A new group is built for selected agents in the last group.
The proposed scheme ensures that agents who belong to the same group will not be misassigned to different groups after training. For the example in the right side of Figure 1, agent  $a_{i}$  is first selected after  $\alpha_{1} \cdot c$  timesteps' training, and the initial grouping  $\mathcal{G} = \{a_{1}, a_{2}, \dots, a_{n}\}$  changes to  $\mathcal{G}' = \{\{a_{1}, \dots, a_{i-1}, a_{i+1}, \dots, a_{n}\}, \{a_{i}\}\}$ . Subsequently, after another training period, at timestep  $\alpha_{2} \cdot c$ , two agents  $a_{j}$  and  $a_{k}$  in the first group are selected simultaneously to be moved out of the first group. At this point, they are automatically placed in the second group, i.e., the group to which agent  $a_{i}$  belongs, and  $\mathcal{G}'' = \{\{A_{-i,j,k}\}, \{a_{i}, a_{j}, a_{k}\}\}$ . Instead of placing  $a_{j}$  and  $a_{k}$  in a brand new group, our group shifting scheme ensures that they have the opportunity to train with  $a_{i}$  together in one group, determining if  $a_{i}, a_{j}$ , and  $a_{k}$  (or two of them) are supposed to be in a group.
Later, agent  $a_{l}$  in the first group and agent  $a_{i}, a_{k}$  in the second group are chosen at timestep  $\alpha_{3} \cdot c$ . The selection of  $a_{i}$  and  $a_{k}$  indicates that agent  $a_{j}$  cannot cooperate well with them. Therefore,  $a_{i}$  and  $a_{k}$  are set in a new group (they should not return to the first group since they have been proved inappropriate for the first group), while  $a_{j}$  stays alone in the second group. Agent  $a_{l}$  from the first group is assigned automatically to the second group  $\{a_{j}\}$ . If  $a_{l}$  is supposed to be with  $a_{i}$  and  $a_{k}$ , it will be selected afterward. Because  $a_{j}$  fails to form efficient cooperation with  $\{a_{i}, a_{k}\}$ , and if  $a_{j}$  also cannot cooperate well with  $a_{l}$ , then  $a_{l}$  will be selected later on and set in the third group  $\{a_{i}, a_{k}\}$ .
The group shifting scheme of GoMARL assigns the selected agents to their following group and ensures that each group is trained for a fixed period of cooperative attempts. As analyzed above, each agent can be grouped with appropriate agents that can efficiently work together after training.
B Experiment Details
B.1 Detailed Experimental Setup
We compare all the methods in six Hard and Super Hard StarCraft II micromanagement tasks (SMAC) [29] and three challenging Google Research Football (GRF) [16] scenarios. When resources are available, parallel training can facilitate faster convergence of methods [8] and is very common in RL and MARL communities [39, 45, 11]. GoMARL and all the baseline methods in our paper are trained with 8 parallel runners for 10M steps in both testbeds. Therefore, the timestep is not comparable to some papers that conduct experiments with only one "episode" runner. We use one NVIDIA Titan V GPU for training. We evaluate each method every 10K steps with 32 episodes and report the 1st, median, and 3rd quartile win rates across 5 random seeds. The detailed setting of GoMARL's hyperparameters is shown in our source code<sup>1</sup>.
Many algorithms introduce implementation tricks when they are built. These code-level optimizations were studied in depth in [31, 48, 8] and were shown to have a significant impact on algorithm performance. To ensure fair comparisons, our experiments are based on the PyMARL2 [8] framework proposed to compare algorithms fairly. Please refer to PyMARL2's open-source implementation for further training details and fair comparison settings. In addition, some methods implement specific parameter tuning in diverse scenarios, and is unfair for comparison. Therefore, we fixed parameters of all methods for all scenarios in our experiments.
Lastly, adding vanilla-QMIX in our experiments is not intended to unfairly compare GoMARL and QMIX. After all, GoMARL is already better than the finetuned version of QMIX (denoted as Ft-QMIX), and there is no need to compare it with vanilla-QMIX to show our superiority. We added vanilla-QMIX's learning curves in Figure 5 only to avoid potential misunderstanding caused by the performance difference between Ft-QMIX in our paper and that of vanilla QMIX in other papers.
<https://github.com/zyfsjycc/GoMARL>
B.2 Detailed Information about SMAC Tasks
In each SMAC micromanagement problem, a group of units controlled by decentralized agents cooperates to defeat the enemy agent system controlled by handcrafted heuristics. Each agent's partial observation comprises the attributes (such as health, location, unit_type) of all units shown up in its view range. The global state information includes all agents' positions and health, and allied units' last actions and cooldown, which is only available to agents during centralized training. The agents' discrete action space consists of attack[enemy_id], move[direction], stop, and no-op for the dead agents only. A particular unit, Medivac, has no action attack[enemy_id] but has action heal[enemy_id]. Agents can only attack enemies within their shooting range. Proper micromanagement requires agents to maximize the damage to the enemies and take as little damage as possible in combat, so they need to cooperate with each other or even sacrifice themselves. We follow the default setup of SMAC in our experiments, and more settings, including rewards and observation information, can be acquired from the original paper [29] or open-source implementation.
Based on the performance of baseline algorithms, the tasks in SMAC are broadly grouped into three categories: Easy, Hard, and Super Hard. The key to winning some Hard or Super Hard battles is mastering specific micro techniques, such as focusing fire, kiting, avoiding overkill, etc. Cetera. The battles can be symmetric or asymmetric, and the group of agents can be homogeneous or heterogeneous. Here we provide some characteristics of the scenarios to help gain insights into the good or poor performance of the methods: