SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- 3s_vs_5z is a Hard asymmetric battle between two homogeneous teams. The allied Stalkers have to master the kiting technique and disperse in the area to kill the Zealots that chase them one after another. This map faces the delayed reward problem; however, it is not very strict about micro-cooperation between agents because of agents' scattering.
- The asymmetric scenarios  $5\mathrm{m\_vs\_6m}$  and  $8\mathrm{m\_vs\_9m}$  are two Hard maps offering a substantial challenge. The allied agents must learn to focus fire without overkill and to correctly position themselves with considerable precision to overcome the enemy team with more agents. The  $5\mathrm{m\_vs\_6m}$  battle is relatively more difficult than  $8\mathrm{m\_vs\_9m}$  due to fewer agents, so the enemy force with one more agent has a greater advantage.
- corridor is a Super Hard map that needs extensive exploration. Six allied Zealots fight twenty-four Zerglings on this task. The massive disparity in unit numbers between the two sides implies that the whole team cannot launch an attack together. The only winning strategy is to sacrifice a small number of agents who leave the team and attract the attention of most enemies, and the large force eliminates the rest of the enemies. The surviving agents then keep repeating the same tactic till the whole team kills all the enemies.
- MMM2 is a representative Super Hard asymmetric battle between two heterogeneous teams with three kinds of units. One Medivac, two Marauders, and seven Marines have to battle against a team with one more Marine. Marauder has greater attack damage and health than Marine but with a longer cooldown. Medivac has no damage but can heal other agents.
- 3s5z_vs_3s6z is a Super Hard map that requires breaking the bottleneck of exploration, where three Stalkers and five Zealots battle against three Stalkers and six Zealots. This map requires more cooperation between agents and is the most challenging task in SMAC.
B.3 Detailed Information about GRF Academy Scenarios
Google Research Football (GRF) Academy includes several scenarios which can be commonly found in football games. Agents need to coordinate timing and positions for organizing offense to seize fleeting opportunities, and only scoring leads to rewards. Each agent's partial observation contains the absolute positions and moving direction of the ego-agent, relative positions and moving directions of other agents, and the ball. The global state information includes the absolute positions and directions of all agents and the ball. Agents have a discrete action space of 19, including moving in eight directions, three kinds of ball pass, two kinds of dribble, two kinds of sprint, shot, sliding, stop-moving and do-nothing. Proper cooperation requires agents to pass and shoot effectively with a good direction at proper timing, thus making GRF scenarios much more complicated than SMAC's. Other details can be acquired from the original paper [16] or the open-source implementation.
Figure A11: Method improvement with the proposed SAN in the Super Hard map 3s5z_vs_3s6z.
Here we provide an introduction of the three scenarios we utilized to help gain insights into the good or poor performance of all the methods in our paper:
- academy_pass_and_shoot_with_keeper includes two of our players trying to score from the edge of the box. One of our agent is on the side with the ball and next to a defender. The other agent is at the center, unmarked, and facing the opponent keeper.
- academy_3_vs_1_with_keeper contains three of our agents and two opponent players (a defender and a keeper). Our agents try to score from the edge of the penalty area. One stands in the middle, while the others are located on both sides of the area. Initially, the agent at the center keeps the ball and directly faces the defender.
- academyCounterattack_easy contains four of our agents and two opponent players (a defender and a keeper). Agents are initialized far from the penalty area and stand evenly in an arc centered on the goal. The second agent from the top initially keeps the ball and has to pass it to a teammate at the appropriate time to avoid interception.
C Performance of Methods with and without the Specialized Agent Network
The proposed specialized agent network (SAN) is highly transferable. To further validate its effectiveness, we perform the specialized agent network on other baseline methods in Section 5.1 to see if they can perform better. ROMA is not included in this study since ROMA's agent networks are produced by its learned roles, and the replacement will invalidate the main idea of ROMA. As shown in Figure 8, our specialized agent networks significantly improve methods' performance and learning efficiency in the Super Hard SMAC task 3s5z_vs_3s6z. Compared to agent networks utilizing the vanilla parameter-sharing mechanism that limits policy diversity, our specialized agent network adjusts the local utility with the individual group-related information and enables various styles of behaviors related to the group status, encouraging extensive exploration and accelerating learning. The complete learning curves are shown in Figure A11.
As Figure A11 illustrates, the performance of all methods is further improved when equipped with our specialized agent network. Specifically, the learning efficiency of Ft-QMIX is boosted. The variance of RIIT is markedly reduced, and the win rate is increased by about  $10\%$ . The improvement of VAST is the most obvious; both learning efficiency and win rate are enhanced, and the variance is very clearly reduced. QPLEX's improvement is not very obvious; however, it obtains a slightly higher learning speed and achieves the highest win rate among all the baseline methods.
Most importantly, even equipped with our specialized agent network, all baseline algorithms fail to surpass GoMARL in terms of learning efficiency and the final win rate. GoMARL with vanilla agent networks looks much inferior to GoMARL with the specialized agent network, so it is worth questioning whether the dynamic grouping module is ineffective. The dynamic group learning may
(d)
Figure A12: A visualization of learned policies of academy counterattack_easy. The yellow arrows show the motion of the ball, and the red arrow illustrates the scoring shoot.
reduce efficiency to a certain extent in the early stage of training compared to QPLEX, however, in the middle and late training stages, the learned grouping will have a significant effect when equipped with the proposed specialized agent network. Therefore, the two modules of GoMARL, the automatic grouping module and the specialized agent network, are both crucial to the performance of GoMARL.
D Visualizations of the Learned Policies in GRF
The trained strategy in Google Research Football is visualized to validate if the learned grouping makes sense. In Section 5.2, we visualize a match of academy counterattack easy with a more complex strategy to prove the rationality of the grouping GoMARL learned. Here, we provide a detailed discussion of the visualization in Figure A12 (a zoom-in version of Figure 10 for clarity).
As shown in Figure A12(a), Agent 2 holds the ball at the beginning and faces an opponent rushing toward him. Agent 2 passes the ball to Agent 1 to prevent the ball from being stolen. Subsequently, in (b), Agent 1 carries the ball and tries to break through. However, the opponent goalkeeper blocks his attacking route, and Agent 1 continues passing after a short carry. Agent 3 makes a good run and catches the ball smoothly but shoots quickly in Figure A12(c) since the opponent is close. The goalkeeper easily saves this hasty attack. Agent 4 in (d), who learns excellent coordination with Agent 3, stops the ball and immediately adds another shot to create the goal.
GoMARL's automatic grouping module learns a reasonable grouping  $\mathcal{G} = \{\{a_1, a_2\}, \{a_3, a_4\}\}$  for this complex goal, in which the first group successfully brings the ball into the penalty area through smooth coordination, while the second group creates the final goal through skillful cooperation.