SlowGuess/NeurIPS_2023_OCR huggingface dataset:
to minimize the damage inflicted upon them while maximizing their effectiveness in dealing damage to the enemy. Consequently, the Marines face higher learning difficulties compared to the Marauders. Reflecting this distinction, the average sampling percentage of the Marine unit class surpasses that of the Marauder unit class.
E Discussion Related to Policy-Based Methods
E.1 Performance Comparison with Policy-Based Method COMA
In this section, we compare DIFFER with the classic policy-based method COMA[3] on four SMAC scenarios as shown in Fig. 13. We can see that DIFFER performs better than COMA on all four scenarios, which strongly highlight the strengths and advantages of our proposed DIFFER framework.
(a) 2c_vs_64zg
(b) MMM2
Figure 13: Performance comparison of QMIX-DIFFER (ours) and a classic policy-based method COMA on SMAC scenarios, highlighting the performance improvement of our method towards COMA.
(c) 3s5z_vs_3s6z
(d)  $27\mathrm{m\_vs\_30m}$
E.2 Potential Application to Policy-Based Approaches
Our method, DIFFER, is specifically designed to address MARL problems where individual rewards are not available, and teams only have access to a shared team reward. In such scenarios, calculating individual action-values becomes a challenge. However, DIFFER provides a solution by decomposing individual trajectories, enabling the calculation and update of individual action-values. For policy-based methods that approximate individual action-values using individual critics, such as IPPO[42] and MADDPG[43], the traditional approach involves treating the team reward as an individual reward and using it to update individual critics. In this context, the decomposition of team experiences provided by DIFFER may not be directly applicable or necessary since the individual critics can approximate action-values using the team reward. On the other hand, for policy-based methods that do not require the approximation of individual action-values, such as MAPPO[4], our DIFFER method might not offer significant enhancements. In such cases, where there is no need to decompose team trajectories for individual action-values training, the contribution of DIFFER may not be as pronounced.
F Environment Introduction
In this section, we provide a brief overview of the multi-agent environment discussed in this paper.
F.1 SMAC
The Starcraft Multi-Agent Challenge (SMAC)[29] (as shown in Fig. 14 (a)) is a prominent cooperative multi-agent environment known for its partial observability. In this paper, we utilize the default environment setting of SMAC, specifically version SC2.4.10. SMAC offers a variety of scenarios, each featuring a confrontation between two teams of units. One team is controlled by a trained model, while the other team is governed by a built-in strategy. The scenarios differ in terms of the initial unit positions, the number and types of units in each team, as well as the characteristics of the map terrain. In the SMAC environment, a team is deemed victorious when all units belonging to the
(a) SMAC
(b) GRF
(c) MAMujoco
Figure 14: Visualisation of three experimental environments.
opposing team are eliminated. The primary objective of the policy is to maximize the win rate across all scenarios. To facilitate training, the environment provides a shaped reward function that takes into account factors such as hit-point damage inflicted and received by agents, the number of units killed, and the outcome of the battle. Learning a coordinated micromanagement strategy in diverse maps poses a significant challenge for the agents in this environment. Such a strategy aims to maximize the damage inflicted upon enemies while minimizing the damage sustained. Overall, the SMAC environment demands agents to acquire sophisticated coordination and decision-making skills in order to excel in diverse scenarios and effectively apply micromanagement strategies.
F.2 GRF
Google Research Football (GRF) environment [30] (shown in Fig. 14 (b)) presents a challenging multi-agent reinforcement learning (MARL) setting where a team of agents must learn to pass the ball amongst themselves and overcome their opponents' defense to score goals. Similar to the SMAC environment, the opposing team in GRF is controlled by a built-in strategy. In GRF, each agent has 19 different actions at their disposal, including standard move actions in eight directions, as well as various ball-kicking techniques such as short and long passes, shooting, and high passes that are difficult for opponents to intercept. The agents' observations encompass information about their own position, movement direction, the positions of other agents, and the ball. An episode in GRF terminates either after a fixed number of steps or when one of the teams successfully scores a goal. The winning team receives a reward of  $+100$ , while the opposing team is rewarded with -1. In this study, we specifically focus on two official scenarios: academy_3_vs_1_with_keeper and academycounterattack-hard.
F.3 MAMujoco
The Multi-Agent Mujoco (MAMujoco) environment[31] (shown in Fig. 14 (c)) is a highly versatile and realistic simulation platform designed for multi-agent robotic control tasks. It is based on the Mujoco physics engine and offers continuous action spaces, allowing for smooth and precise control of robotic agents. Each agent in the environment represents a specific component or a complete robot, enabling the simulation of complex multi-robot systems. All of the tasks mentioned in this work are configured according to their default configuration. We set maximum observation distance to  $k = 0$  for Humanoid - v2 and  $k = 1$  for manyagent swimsmer.
We also provide the description of agent types in each scenario in Table 2. It aims to enhance the reader's understanding of the distinct performance improvements brought about by DIFFER.
Table 2: The agent type distribution in each scenario.
Env.	Scenario Name	#Agent	#Type	Type Distribution
SMAC	27m_vs_30m	27	1	27
2c_vs_64zg	2	1	2
MMM2	10	3	1-2-7
3s5z_vs_3s6z	8	2	3-5
GRF	academy_3_vs_1_with_keeper	3	2	1-2
academy counterattack-hard	4	2	2-2
MAMujoco	Humanoid-v2_0	2	2	1-1
manyagent swimsimmer	20	10	2-2-2-2-2-2-2-2-2