SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Name	Description	Value
α	Learning rate	0.0005
Type of optimizer	RMSProp
RMSProp param	0.99
ε for optimizer	RMSProp param	0.00001
How many episodes to update target networks	200
Reduce global norm of gradients	10
Batch size	32
Capacity of replay buffer	5000
γ	Discount factor	0.99
M	Sample size	100
λinit	Starting value for exploration coefficient annealing	0.5
λend	Ending value for exploration coefficient annealing	0
Annealing period for the multi-step matrix game	25k
Annealing period for easy scenarios in SMAC	500k
Annealing period for hard scenarios in SMAC	1.5m
Table 1: Hyperparameter settings.
$^{1}$ https://github.com/oxwhirl/pymarl.
D Additional Experiments and Details
D.1 Comparison between IGM-Based and IGM-Free Mixing Networks
To verify that IGM-based mixing network restricts the set of global state-action value functions that can be represented, we synthesize an artificial dataset to represent the reward functions of two agents in a single state matrix game. Each agent has 101 actions, which means  $u^i \in \{0,1,\dots,100\}$ . In order to only measure the influence of whether the mixing network follows the IGM assumption, we fix the individual action-value function as follows:
$$
Q _ {i} \left(u ^ {i}\right) = \frac {u ^ {i}}{1 0 0}, \quad \forall i \in \{1, 2 \}.
$$
The reward value  $R$  for a joint action is given as follows:
$$
R \left(u ^ {1}, u ^ {2}\right) = \sin \left(2 \pi Q _ {1} \left(u ^ {1}\right)\right) + \exp \left(Q _ {2} \left(u ^ {2}\right)\right).
$$
The ground-truth joint value function is shown in Figure 12. Next we calculate the joint action value  $Q_{\mathrm{tot}}$  for the selected  $\pmb{u}$  based on the given  $Q_{i}$  and  $R$ . Most value decomposition approaches, such as QMIX and QPLEX, directly constrain the weights of the neural network to be non-negative, so we use the mixing network in QMIX as the IGM-based model to be trained. Meanwhile, the variant that removes the absolute function is viewed as the IGM-free model. It should also be noted that we use the global state and the actions of all agents as input to the hypernetwork that generates weights to enhance the expressiveness of the mixing network. Figure 12 represents the loss curve and Figure 13 illustrates the joint action-value function estimated by the two models.
Figure 12: Left: Contours for ground-truth  $Q_{\mathrm{tot}}$ , where  $x$ -axis and  $y$ -axis mean each agent's actions. Right: The loss curves for two models during training.
(a) 200 step
(b) 400 step
(c) 600 step
(d) 800 step
(e) 1000 step
(f) 200 step
(g) 400 step
(h) 600 step
Figure 13: Contours for estimated  $Q_{\mathrm{tot}}$ . Colored values represent the values of  $Q_{\mathrm{tot}}$  for IGM-based ((a)-(e)) and IGM-free ((f)-(j)) mixing networks.
(i) 800 step
(j) 1000 step
From the results, it can be concluded that when the joint value function  $Q_{\mathrm{tot}}$  is not a simple monotone function with respect to  $Q_{i}$ , the representation ability of the IGM-based mixing network is obviously limited, and the convergence speed is also slower than IGM-free mixing network. However, it is not possible to simply replace the IGM-based mixing network in the existing work with an IGM-free one, because this will cause each agent to fail to find the individual action corresponding to the optimal joint action during decentralized execution. Of course, this is why we propose the DAVE framework.
D.2 Policy-Based MARL Methods on Single-State Matrix Games
DAVE is not a standard actor-critic framework in the strict sense. The ego policy in DAVE is updated in a supervised learning manner, which maximizes the probability of the joint action corresponding to the optimal joint action value in the sample. Thus DAVE can explicitly search for optimal joint actions and easily solve non-monotonic problems. FACMAC still has to rely on the monotonic mixing network to update the critic; otherwise, the performance will deteriorate.
Moreover, the existing policy-based method that does not use IGM for the critic cannot solve the non-monotonic task in our paper. We test two representative policy-based algorithms, MAPPO and MADDPG, in two didactic problems. We present the experimental results over 100 trials of 10000 episodes in Table 2 and Table 3, which demonstrate that although they do not explicitly introduce IGM, they still perform poorly.
k=0	k=6	k=7.5
-12	0	8	-12	0	6	8	-12	0	7.5	8
MADDPG	35%	45%	20%	27%	34%	31%	8%	27%	35%	34%	4%
MAPPO	0%	100%	0%	0%	0%	100%	0%	0%	0%	100%	0%
Table 2: Proportion of different convergence results in Matrix Game I.
k=0	k=25	k=100
0	2	10	-25	0	2	10	-100	0	2	10
MADDPG	51%	0%	49%	3%	45%	45%	7%	1%	20%	76%	3%
MAPPO	0%	0%	100%	0%	0%	100%	0%	0%	0%	100%	0%
Table 3: Proportion of different convergence results in Matrix Game II.
We carry out MADDPG and MAPPO in a policy-based setting. The above two algorithms cannot perfectly solve the single-state matrix problems even when more samples can be accessed. As the difficulty of the problem increases, the proportion of MADDPG converging to the optimal solution gradually decreases. MAPPO converges steadily to a sub-optimal solution.
D.3 Additional Explanations for Results in Multi-Step Matrix Game
In MAVEN's official code, the implementation of the multi-step matrix problem does not match the description in its original paper, which is reflected in the terminal state corresponding to the bad branch (see Figure 14). This does not change the non-monotonicity of the initial state, but reduces the expected return for agents to choose the bad branch. In other words, the above mistakes will make it EAS-IER for agents to learn the optimal policy. We rectified this error and retested all baselines for their real performance. Results in Figure 5 show that MAVEN performed worse than in its original paper.
(a) Multi-step matrix game in MAVEN's paper.
(b) Multi-step matrix game in MAVEN's code.
Figure 14: The difference between the multi-step matrix game environment in the original paper of MAVEN and in the actual code of MAVEN.
D.4 Additional Introduction for SMAC Environment
The single-state matrix games and the multi-step matrix game are elaborated in the main body of the paper. Here we only elaborate on the SMAC environment. The action space of each agent is discrete. The version of StarCraft II used in this paper is the latest 4.10. For a fair comparison, all algorithms are run on the same version of StarCraft and SMAC. Table 4 shows the details of the scenarios used in this paper.
Name	Ally Units	Enemy Units	Type
2s_vs_1sc	2 Stalkers	1 Spine Crawler	Homogeneous
Asymmetric
3s5z	3 Stalkers	3 Stalkers	Heterogeneous
Symmetric
5 Zealots	5 Zealots
2c_vs_64zg	2 Colossi	64 Zerglings	Homogeneous
Asymmetric
5m_vs_6m	5 Marines	6 Marines	Homogeneous
Asymmetric
3s_vs_5z	3 Stalkers	5 Zealots	Homogeneous
Asymmetric
8m_vs_9m	8 Marines	9 Marines	Homogeneous
Asymmetric
10m_vs_11m	10 Marines	11 Marines	Homogeneous
Asymmetric
MMM2	1 Medivac	1 Medivac	Heterogeneous
Asymmetric
2 Marauders	3 Marauder	Macro tactics
7 Marines	8 Marines
6h_vs_8z	6 Hydralisks	8 Zealots	Homogeneous
Asymmetric
Table 4: Maps in different scenarios.
D.5 Additional Experiments on SMACv2
Although SMAC is a popular multi-agent benchmark in recent years, it suffers from a lack of stochasticity. To remedy this significant deficiency, a more challenging SMACv2 environment is proposed. In SMACv2, team composition is randomly generated, which means that unit types are not fixed. Meanwhile, the start positions of all agents are also random. Finally, the sight range and attack range of the units are changed to the values from SC2 instead of fixed values. These three major changes make the tasks in SMACv2 more challenging.
To assess the generalizability of our proposed algorithm, we evaluate the performance of DAVE in several representative scenarios of SMACv2. Some of the scenarios are symmetrical and others are asymmetrical. To ensure consistency with the original paper on SMACv2, we use the hyperparameter settings from PyMARL2² [15]. To demonstrate that the DAVE framework is applicable to any existing value decomposition method and can transform the original algorithm into an IGM-free version with guaranteed convergence, we compare the performance of two classic algorithms, QMIX and QPLEX, with their variants respectively. Specifically, IGM-Free variants are obtained by simply removing all absolute functions from the original methods. We conduct the experiments using StarCraft II version 4.10 and present the results in Figure 15 and Figure 16.
The experimental results demonstrate that DAVE-QMIX outperforms QMIX in most scenarios, particularly in the *protoss_5.vs_5* and *terraran_5.vs_6* scenarios. Notably, both DAVE-QMIX and IGM-Free QMIX are IGM-free methods. However, while IGM-Free QMIX is unable to guarantee individual actions consistent with the optimal joint action, its performance is significantly worse than DAVE-QMIX and QMIX, and even fails in most scenarios. Similarly, DAVE-QPLEX outperforms both vanilla QPLEX and IGM-Free QPLEX, despite the weight of its transformation module no longer being restricted to non-negative values. In summary, DAVE variants perform significantly
$^{2}$ https://github.com/benellis3/pymarl2.
Figure 15: Comparisons of median win rate for variants of QMIX on SMACv2.
Figure 16: Comparisons of median win rate for variants of QPLEX on SMACv2.
better than original methods in scenarios that do not meet the IGM assumptions, and also exhibit competitive performance in other scenarios, regardless of the environment complexity.
In addition to evaluating different variants of the same algorithm, we compare the performance of different baselines designed to solve the IGM problem. Since these baselines are all based on QMIX, we show their learning curves and that of DAVE-QMIX in Figure 17. It is evident from the results that the complex mechanisms introduced by other algorithms cause them to perform worse than the vanilla QMIX and DAVE-QMIX. The experimental results on SMACv2 demonstrate the excellent generalization of DAVE to different algorithms and environments.
D.6 Additional Experiments on Multi-Agent MuJoCo
In order to test the performance of the same algorithm following IGM and not following IGM in a complex environment, we also carry out vanilla QMIX, IGM-Free QMIX and DAVE on Multi-Agent MuJoCo (MA-MuJoCo). It is worth noting that the latter two do not require the IGM assumption at all, which means that they can fit richer function classes but it is more difficult to find the optimal joint action. Since most of the conventional value decomposition methods are only applicable to environments where the action space is discrete, we discretize the action space of the MA-MuJoCo environment reasonably. Each joint in the robot is considered as an agent. We use uniform quantization to discretize continuous actions and modify the action space of each agent to  $\mathcal{U} = \left\{\frac{2j}{K - 1} -1\right\}_{j = 0}^{K - 1}$  instead of the original  $\mathcal{U} = [-1,1]$ .  $K$  is a hyperparameter and in this paper  $K = 31$ . In addition, MA-MuJoCo is a partially observable environment, and each agent (joint) can only observe the information of neighbor agents (joints). Figure 18 shows three classic scenarios. Numbers in brackets indicate the number of agents (joints) in each task.
It can be seen from Figure 19 that even without the IGM assumption, DAVE can still perform better than vanilla QMIX in most scenarios, especially in the Hopper and Humanoid scenarios. IGM-Free
Figure 17: Median win rates on a range of SMACv2 mini-games. Note that only DAVE-QMIX is IGM-free.
(a) Hopper (3)
(b) HalfCheetah (6)
(c) Humanoid (17)
Figure 18: Illustration of benchmark tasks in Multi-Agent MuJoCo.
QMIX fails in all tasks. In summary, in the case of the same loss function related to reinforcement learning, QMIX with the dual self-awareness framework performs better than vanilla QMIX in all cases. Even in scenarios with large action spaces like Humanoid, which contains  $31^{17} \approx 2.26 \times 10^{25}$  joint actions, DAVE still outperforms QMIX. Like the results in the main text, the experimental results in MA-MuJoCo further prove that DAVE is not only suitable for didactic examples, but also capable of complex scenarios.
Figure 19: Median episode return on different MA-MuJoCo tasks. Note that only QMIX follows the IGM assumption.
D.7 Additional Notes on Model Size
We give the relative size of neural network models for some value decomposition variants over QMIX in Table 5. The model size here refers to the number of parameters each algorithm needs to train in the MMM2 scenario.
The number of parameters in the value decomposition framework mainly depends on the mixing network structure. The DAVE framework only has three more fully connected layers (an additional ego model) than QMIX, so DAVE has the smallest change compared to QMIX among the variants. Nevertheless, DAVE still performs better than other baselines related to IGM. The most important thing is that DAVE can quickly converge to the optimal solution in the matrix game, while QMIX cannot.
Algo	QMIX	DAVE	CW-QMIX	OW-QMIX	MAVEN	QPLEX
Relative Size	100%	132%	373%	373%	167%	142%
Table 5: The relative size of neural network models related to reinforcement learning for some value decomposition variants over QMIX.
E More Related Work
By improving the joint exploration of all agents, UneVEn [10] addresses the issue that stems from the monotonicity constraint in representing the joint-action value function. However, in some complex environments such as SMAC, its additional complex mechanism leads to significantly slower convergence. TESSERACT [22] views the Q-function as a tensor and utilizes low-rank tensor approximations to model agent interactions relevant to the task. DCG [2] factors the joint value function of all agents into payoffs between pairs of agents based on a coordination graph, but it requires additional domain knowledge and violates CTDE principles. QTRAN++ [31] is an improved version of QTRAN, however it still does not completely remove absolute functions. STEP [42] can solve non-monotonic problems but it focuses on how to find Stackelberg equilibrium via asynchronous action coordination. Besides, some value decomposition methods [33] from the distributional perspective replace the original IGM principle with the distribution-style IGM to deal with highly stochastic environments. The DAVE framework we proposed does not change the loss function of the underlying value decomposition algorithm, and only requires simple modifications to make it IGM-free. The DAVE-based value decomposition method can outperform the original algorithm in scenarios where the IGM assumption is not satisfied, and it can also have competitive performance in other scenarios.
There has been a lot of excellent work on exploration in the field of multi-agent reinforcement learning. UneVEn uses novel action-selection schemes during exploration to overcome relative overgeneralization. EITI/EDTI [38] measure the amount and value of one agent's influence on the other agents' exploration processes to improve exploration. Dimakopoulou and Roy [4] present several seed sampling schemes for efficient coordinated exploration in concurrent reinforcement learning. CMAE [18] uses target policies to maximize returns and exploration policies to collect unexplored data, which is similar to our work. However, CMAE focuses on solving the sparse-reward task, whereas anti-ego exploration in DAVE alleviates the suboptimal convergence caused by limited search. In addition, the anti-ego exploration mechanism also leverages the characteristics of the auto-encoder to further narrow the state-action space that needs to be explored.