Dual Self-Awareness Value Decomposition Framework without Individual Global Max for Cooperative MARL
Authors: Zhiwei Xu
Bin Zhang
Dapeng Li
Guangchong Zhou
Zeren Zhang
Guoliang Fan
Venue: NeurIPS 2023 poster (2023)
Keywords: Multi-Agent Reinforcement Learning
Individual Global Max
TL;DR: We propose a dual self-awareness value decomposition framework with a novel anti-ego exploration mechanism to remove the Individual Global Max (IGM) assumption completely.

Abstract: Value decomposition methods have gained popularity in the field of cooperative multi-agent reinforcement learning. However, almost all existing methods follow the principle of Individual Global Max (IGM) or its variants, which limits their problem-solving capabilities. To address this, we propose a dual self-awareness value decomposition framework, inspired by the notion of dual self-awareness in psychology, that entirely rejects the IGM premise. Each agent consists of an ego policy for action selection and an alter ego value function to solve the credit assignment problem. The value function factorization can ignore the IGM assumption by utilizing an explicit search procedure. On the basis of the above, we also suggest a novel anti-ego exploration mechanism to avoid the algorithm becoming stuck in a local optimum. As the first fully IGM-free value decomposition method, our proposed framework achieves desirable performance in various cooperative tasks.

Dual Self-Awareness Value Decomposition Framework without Individual Global Max for Cooperative MARL
Zhiwei Xu, Bin Zhang, Dapeng Li, Guangchong Zhou, Zeren Zhang, Guoliang Fan Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences (xuzhiwei2019, guoliang. fan @aaa ac. cn
Abstract
Value decomposition methods have gained popularity in the field of cooperative multi-agent reinforcement learning. However, almost all existing methods follow the principle of Individual Global Max (IGM) or its variants, which limits their groblem-solvingg capabilities. To address this, we propose dual self-awareness value decomposition framework, inspired by the notion of dual self-awareness in psychology, that entirely rejects the IGM premise. Each agent consists of an ego policy for action selection and alter ego value function solve the credit assign- ment problem. The value function factorization can ignore the IGM assumption by utilizing an explicit search procedure. On the basis of the above, we also suggest a novel anti-ego exploration mechanism to avoid the algorithm becoming stuck in a local optimum. As the first fully IGM-free value decomposition method, our proposed framework achieves desirable performance in various cooperative tasks.
Introduction
Multi-agent reinforcement learning (MARL) has recently drawn increasing attention. MARL al- gorithms have been successfully applied in diverse fields, including game AI [T] and energy net- works (37), to solve practical problems. The fully cooperative multi-agent task is the most com- mon scenario, where all agents must cooperate achieve the same goal. Several cooperative MARL algorithms have been proposed, with impressive results on complex tasks. Some algo- enable agents to collaborate through communication. However, these approaches incur additional bandwidth overhead. Most communication-fiee MARL algorithms follow the con- ventional paradigm of centralized training with decentralized execution (CTDE) 20). These works can be roughly categorized into multi-agent Actor-Critte algorithm
rithms 7 17
[8] 16 and value decomposi- tion method [34 27 19 40 39]. Due the high sample efficiency in off-policy settings, value
decomposition methods can outperform other MARI algorithms.
However, due to the nature the CTDE paradigm, the value decomposition method can only operate under the assumption of Individual Global Max (IGM) [30]. The IGM principle links the optimal joint action to the optimal individual one, allowing the agent find the action corresponding to the maximal Q value during decentralized execution. Nevertheless, this assumption is not applicable in most real- real-world scenarios. And most value decomposition approaches [27] directly constrain the weights of the neural network to be non-negative, which restricts the set of global state-action value functions that can represented. We confirm this point through experiments in Appendix [D.]
Removing the IGM assumption requires agents to be able to find actions corresponding to the maximal joint Q value during execution. Therefore, addition to the evaluation network, agents also need an additional network to learn how to search for the best actions in a decentralized manner. Similarly, there are concepts of ego and alter ego psychology. The ego usually refers to the conscious part of the individual, and Freud considered the ego to be the executive of the personality (9]. Some people
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

believe that an alter ego pertains to a dif- ferent version of oneself from the authentic self 124 12 Others define the alter ego in more detail as the evaluation of the self by others 3 As a result, both ego and alter ego influence a person's personality and behaviour, as depicted in Figure [ Besides, Duval and Wicklund (5] put forth the dual self-awareness hypothesis, which posits that individuals pos- sess two states of consciousness: public self- awareness and private self- ooff-wwreessss Private self-consciousness refers to an individual's in- ternal understanding of himself, whereas public self-awarenes enables individuals recognize that they are evaluated by others. Enlightened by these psychological concepts, we propose a novel MARL algorithm, Dual self- Awrrnness Value dEcomposition (DAVE). Each agent in DAVE maintains two neural network models: the alter ego model and the ego model. The former participates in the evaluation of the group to solve the global credit assignment problem, and the latter is not directly optimized by the joint state-action value. The ego policy can help the alter ego value function to find the individual action corresponding to the optimal joint action through explicit search procedures so that the value decomposition framework can completely eliminate the constraints of the IGM assumption. Furthermore, to prevent the ego policy based on the above framework from potentially getting stuck in a bad local optimum, we also propose an anti-ego exploration method. This method can effectively avoid the underestimation of the optimal action that may be caused by the limited search procedure.
Human Society
Evaluate
Individuaiity
Alter Ego
Ego
In psychology, it is commonly believed that the ego and alter ego determine an individual's behavior jointly.
Figure I:
We first evaluate the performance of DAVE in several didactic problems demonstrate that DAVE can resolve the issues caused by IGM in conventional value decomposition methods. Then we run DAVE in various complex scenarios including StarCraft II, and compare it to other popular baselines. Our experimental results show that DAVE can still achieve competitive performance despite the absence of IGM. To the best of our knowledge, DAVE is the first multi-agent value decomposition method that completely abandons the IGM assumption.
Related Work
Recently, value decomposition methods have made significant progress in problems involving the decentralized partially observable Markov decision process (Dec-POMDP) [23). By calculating joint state-action value function based on the utility functions all agents, value decomposition methods can effectively alleviate the notorious credit assignment issue. The first proposed method, VDN 34. simply sums up the utility of all agents to evaluate joint actions. This undoubtedly limits the range of joint value functions that can be approximated. QMIX (27 incorporates hypernetwrks [T]] into the value decomposition method and employs the mixing network to fit the joint value function based on the individual ones, thereby expressing richer classes of joint value function. To enable the agent to choose actions with the greedy policy during decentralized training, QMIX assumes monotonicity and restricts the weight parameters of the mixing network output by hypernetworks So the monotonic mixing network still cannot represent some class of value functions. Various value decomposition variants have subsequently been proposed. Officially proposing IGM, QTRAN [30] indicated that the value decomposition method must satisfy the assumption that the optimal joint actions across agents are equivalent to the collection of individual optimal actions of each agent. Unlike VDN and QMIX, QTRAN is not restricted by structural constraints such as non-negative weights, but performs poorly in complex environments such as StarCraft II. MAVEN (2 enhances its performance by improving its exploration capabilities. Weighted QMIX (28) takes into account weighted projection that places greater emphasis on better joint actions. QPLEX (36) takes duplex dueling network architecture to factorize the joint action-value function and proposes the advantage-ber based IGM. Nevertheless, the aforementioned methods only loosen the restrictions by proposing IGM variants. More importantly, existing policy- based MARL methods [1 25] do not explicitly introduce IGM, but they also cannot solve some non-monon non-monotonic problems. So how to remove the IGM assumption is still an open and challenging problem. Our proposed DAVE is completely IGM-free and can be applied to existing IGM-based value decomposition methods. More related work will be discussed in Appendix E
non-negative.

---

3 Preliminaries
3.1 Dec-POMDPss
In this paper, we concentrate on the Dec-POMDD problems. It is an important mathematical model for multi-agent collaborative sequential decision- makinn problem, in which agents act based on different pieces of information about the environment and maximize the global shared return. Dec-POMDPss can be formulated as a tuple S,,, Z, P. P,O,r, a E A (I,...,,?) nn represents the agent in the task. E 8 is the global state of the environment. The joint action set U is the Cartesian product of action sets 2t2 of all agents and u E 2 denotes the joint action. Given the state s and the joint action u, the state transition function P S U x S - 00. 1] returns the probability of the next potential states. All agents receive their observations x = {2 ..... 2") obtained by the observation function O S x - Z. r : S U + R denotes the global shared reward function and is the discount factor. Besides, recent work frequently allows agents to choose actions based on historical trajectory 7t rather than local observation 2 in order alleviate the partially observable problem.
In Dec-POMDP, all agents attempt to maximize the discount return R² Ei 7'r and the joint state-action value function is defined as:
Qsa(s, u) = Ex [R u),
where π = [TI,...,, Tr} is the joint policy of all agents.
3.2 IGM-based Value Decomposition Methods
The goal of value decomposition method is to enable the application of value-based reinforcement learning methods in the CTDE paradigm. This approach decomposes the joint state-action value function Qie into the utility function Qa of each agent. In VDN, the joint state-action value function is equal to the sum of all utility functions. And then QMIX maps utility functions to the joint value function using monotonous mixing network and achieve better performance.
Nonetheless, additivity and monotonicity still impede the expressive capacity of the neural network model. which may result in the algorithm being unable to handle even for some simple matrix game problems. To this end, QTRAN proposes the Individual Global Max (IGM) principle, which can be represented as:
arg max 82"
Qsor(s, u) = arg max (T", (T", Va E A.
The IGM principle requires consistency between local optimal actions and global ones. Most value decomposition methods including VDN and QMIX must follow IGM and its variants. This because only when the IGM principle holds, the individual action corresponding to the maximum individual action value during decentralized execution is a component of the optimal joint action. Otherwise, the agent cannot select an appropriate action based solely on the individual action-value function.
4 Methodology
In this section, we introduce the implementation of the dual self self-awareness value decomposition (DAVE) framework in detail. DAVE incorporates an additional ego policy through supervised learning in comparison to the conventional value decomposition method, allowing the agent to identify the optimal individual action without relying on the IGM assumption. Moreover, DAVE can applied to most IGM-based value decomposition methods and turn them into IGM-free ones.
4.1 Dual Self-Awareness Framework
Almost all value decomposition methods follow the IGM assumption, which means that the optimal joint policy is consistent with optimal individual policies. According to IGM, the agent only needs to follow the greedy policy during decentralized execution to cater to the optimal joint policy. Once the IGM assumption is violated, agents cannot select actions that maximize the joint state-action value based solely on local observations. Consequently, the objective the IGM-frre value decomposition method is as follows:
arg max Qrae (s, u),
(I)
77
3

---

Q..
Qt (r)), (4))
-)
IGM-Free Mixing
Softmax
Sample
Qo (rt:
[(r))
#1 (rt,
Que (r)
Q (17,0))
U
Softmin
argmaa
ft (T!,
" 42 (ri, (Ti.')
Encoder
GRU
GRU
Agent
Agent
(2f (6
Anti- Ego Exploration
Alter Ego Model
(at (st)at-))
Ego Model
Figure 2: Middle: The overall dual self-awareness value decomposition framework. Right: Agent network structure. Left: The setup of the anti-ego exploration.
where u= = {u- u") and u ~ Ta (T)) Without the IGM assumption, Equation T is NP-hard because it cannot be solved and verified in polynomial time [14] Therefore, in our proposed dual self-awareness framework, each agent has an additional policy network to assist the value function network to find the action corresponding dual self-awareness theory in psychology, we named these two networks contained in each agent as ego policy model and alter ego value function model, respectively. Figure 2 depicts the dual self-awareness framework. Note that the IGM-free mixing network refers to the mixing network obtained by removing structural constraints. For instance, the IGM-free QMIX variant does not impose non-negative constraint on the parameters of the mixing network.
the optimal joint policy. Inspired by the
In DAVE, the alter ego value function model of each agent a identical the agent network in other value decomposition methods and generates individual state-action value Qallie(7" based on the current historical trajectory n". To obtain the joint state-action value Qullee (s. u) for carrying out the joint action in the state s, the value function of each agent is fed into the IGM-free mixing network. However, the alter ego model does not determine the actions chosen by the agent, either during centralized training or decentralized execution. What determines the action of each agent is the ego policy 000 The joint ego policy can be expressed as T 8) = IT== Ta (u" 7%). When executing, the agent interacts with the environment by sampling actions from the categorical distribution output by the ego policy. And during training, we replace the exact maximization of Qs, u) over the total joint action space with maximization over set of samples U from T80 So we modify the objective of the cooperative multi-agent value decomposition problem from Equation T to
arg max Quller (s, 2 x150
s.t.
E U
(2)
where U = {u 020 7 (s) state & and M the number of samples. Therefore, the dual self-awareness framework substitutes the global optimal state-action value with the local optimal one found in the small space U which is obtained by sampling M times from T As long as the ego policy assigns non-zero probabilities all available actions, this method will converge to the objective described by Equation (T) as the number of samples M increases. The detailed proof can be found in Appendix A Nevertheless, the computational cost directly proportional to M. So the choice of M must strike a balance between precision and computational complexity.
Next, we introduce the update process of the alter ego value function and the ego policy in the dual self-awareness framework. According to the modified objective of the cooperative multi-agent value decomposition problem illustrated by Equation (2). the ego policy needs to be able to predict the individual action that corresponds to the maximal joint state-action value. Therefore, we update the ego policy through supervised learning increase the probability of the optimal joint action obtained through the sampling procedure. We define u* = arg max Quilierss (s, ==0) as the joint action
corresponding to the maximal joint state-action value in the samples, where 000 8 The loss function for the joint ego policy can be written as:
Lago log n 2 s).
(3)
The objective function is the negative lig-likeldhood of the probability of the joint action ut and can be further expressed as Σ" log ego @2 7²) associated with individual ego policies.

---

Additionally, the alter ego value function Qtherr is constantly changing during the training process. To prevent the ego policy कh from becoming too deterministic and getting trapped in local optimal solutions, we will introduce a novel exploration method in the next subsection.
Conventional value decomposition methods can be trained end-to-end by minimizing the following loss:
C = (yhot Quo (8t, uz))²²
where yoot = + maxu.:: Qio (st+1, ux+1) and Que is the target joint value function. The loss function described above cannot be directly applied to our proposed dual self-awareness framework because of the max operator. If we also traverse the samples of the joint action in U to select the local maximum target joint state-action value function, which may result high variance. Inspired by the Expected SARSA 35 we use the expected value instead of the maximal one. The update rule of Qalter considers the likelihood of each action under the current policy and can eliminate the variance, which can be expressed as:
Catter = (ylu Cullter Quller uz))
(5)
where y Te BB (eEeeel[(Quut [(QtNer) (8c+1 Ue+1 )}. However, this method is still computationally
complex. So we reuse the action sample set U and replace Egy [[Qaleer ) (x++11,44+)) with M En (Q (Quller)
ut " where u E - in state Se+1- The modified update rule for Qaker reduces the variance and does not increase the order of the computational complexity compared to the original update method shown in Equation ( Each hypernetwork takes the global state and the actions of all agents as input and generates the weights of the mixing network. It should also be noted that we use the global state and the actions of all agents as input to the hypernetwork that generates weights to enhance the expressiveness of the mixing network.
In summary, we have constructed the fundamental framework of DAVE. Although DAVE consists of a value network and policy network, DISTINCT from the Actor-Critic framework or other policy-based methods such as MAPPO A1], MADDPG [2] and FACMAC [25]. DAVE requires an explicit sampling procedure to update the network model. Besides, the actions sampled from the ego policies remain independent and identically distributed (IID) (26). so the ego policy is trained through supervised learning rather than policy gradient used in policy-based methods. Most intuitively, conventional policy-based methods cannot solve simple non- monooonte matrix games but DAVE can easily find the optimal solution, as described in Section 5.1 and Appendix
4.2 Anti-Ego Exploration
with an insufficient sample size M may lead to a local optimum. This is because the state-action value corresponding to the optimal joint action may be grossly underestimated during initialization, making it challenging for the ego policy to choose the optimal joint action. Consequently, the underestimated Qalter will have no opportunity to be corrected, creating vieious circle. Therefore, we developed an anti-ego exploration method based on ego policies. Its core idea is to assess the occurrence frequeney of state-action pairs using the reconstruction model based on the auto-encoder [33. The auto-encoder model often requires the encoder to map the input data to low-dimensional space, and then reconstructs the original input data based on the obtained oin-einenio representation using the decoder. Our method leverages the fact that the auto-encoder cannot effectively encode novel data and accurately reconstruct it. The more significant the difference between the original iatteacction pair (s, u) and the reconstructed output (s u²), the less frequently the agents carry out the joint action u in state 8. Simultaneously, the difference between the reconstructed state-action pairs and the original ones is also the objective function for updating the auto-encoder, which can be expressed as:
As stated previously, updating the ego policy network solely based on Equation (3)
Lrecom (8, u) = MSF (8, + (F((E,(1A)))
where MSE(-) and CE(-) represent the mean square error and the cross loss function respectively, So we need to encourage agents to choose joint actions with a higher Crecon- Obviously, it is impractical to traverse all available joint actions u. Then we look for the most novel actions within the limited set of actions obtained via the sampling procedure. In order to get state-action pairs that are as uncommon possible, generate an anti-ego policy ago based on πt as shown
5

---

in Figure 2
The relationship between the ego and anti-ego policy of each agent is as follows:
5g0 (T") = Softmax (f Za
(T") Soflmin(f((7))
where f() denotes the neural network model of ego policy. Equation demonstrates that actions with lower probabilities in a have higher probabilities in ao Of course, actions with low probabilities Ta ego may be non-optimal actions that have been fully explored, so we filter out such
actions through the reconstruction model introduced above. 1 == {#: カ (8)) Ti=] is the
joint action set obtained by sampling M times from the anti-ego policy in state s. We can identify the most novel joint action ù²" in state by the following equation:
ú = arg max ( t00
(8)
where ' E 0 Therefore, the objective function in Equation 3 increase the probability of the joint action it" The final objective function of the ego policy 8800 can be written as:
needs to be modified to
Logo - (log n """"""" () Alog N """"""" |s)),
where the exploration coefficient [0, 1] is annealed linearly during the exploration phase. The choice hyperparameter requires a compromise between exploration and exploitation. Note that the dual self-awareness framework and the anti ego exploration method are NOT two separate contributions. The latter is customized for the former, and it is only used to solve the convergence difficulties caused by the explicit sampling procedure in DAVE. By incorporating the anti-ego exploration module, the dual self-awareness framework with low M can also fully explore the action space and significantly avoid becoming stuck in bad local optima. The whole workflow of the anti-ego exploration mechanism shown in Appendix
Our proposed DAVE can be applied to most existing value decomposition methods, it simply adds an extra ego policy network to the conventional value decomposition framework. We believe that the IGM-free DAVE framework can utilize richer function classes to approximate joint state-action value functions and thus be able to solve the non-monotonic multi-agent cooperation problems.
Experiment
In this section, we apply the DAVE framework standard value decomposition methods and evaluate its performance on both didactic problems and complex Dec-POMDPs. First, we will compare the performance of the DAVE variant with QPLEX, Weighted QMIX, QTRAN, FACMAC- nonmonotonic (25), MAVEN, and the vanilla QMIX. Note that most baselines are directly related to IGM. Then the contribution of different modules in the framework and the impact of hyperparameters will be investigated. Except for the new hyperparameters introduced in the DAVE variant, all other hyperparameters are consistent with the original method. Unless otherwise stated, DAVE refers the DAVE variant of QMIX, which enhances representational capacity while ensuring fast convergence. For the baseline methods, we obtain their implementation from the released code. The details of the implementation and all environments are provided in Appendix and Appendix D respectively.
5.1 Single-State Matrix Game
Certain types of tasks in multi-agent cooperation problems cannot be handled under the IGM assump- tions, such as the two matrix games we will introduce next. These two matrix games capture very simple cooperative multi-agent tasks and are shown Fig- ure 3 These payoff matrices of the two-player three- action matrix games describe scenarios that IGM- based value decomposition methods cannot address effectively. The variable k related to difficulty. In matrix game I with & E '0, 8), agents need to choose the same action to get the reward and the optimal joint action (A, A). Furthermore, the closer l: to 8, the more challenging it becomes to jump out of the local optimum. Similarly, k > 0 and (A, C') is the optimal joint action in matrix
A
B
A
8
-12
-12
A
-k
-12
-12
-
(a) Matrix game I.
(b) Matrix game IT.
Figure Payoffs of the two matrix games.

---

DAVE
OPLEX
ONI-DMXX
CV-OMMX
QTRAN
FACZAGG (Non-Mono)
MAVEN
QMX
Calode
300
Epioca
130
Spraad
Masta Game with k-0
Matrix Game with 25
Morts Gome I with c-100
Figure 4:
The learning curves of DAVE and other baselines on the matrix games. Note that the ordinates are non-uniform.
game II. Nevertheless, as l increases, so does the risk of selecting the optimal action (A, C), which leads agents in the IGM-based method prefer the suboptimal joint action (B, B) corresponding to the maximum expected return. Therefore, both matrix games are challenging for traditional value factorization methods. To disregard the influence of exploration capabilities, we implement various algorithms under uniform visitation, which means that all state-action pairs are explored equally. We also evaluate the performance of popular policy-based MARL algorithms. Results in Appendix [D.2 demonstrate that although they do not explicitly introduce IGM, policy-based methods still perform poorly on these two simple tasks.
The results of all algorithms on the two matrix games with varying & are presented in Figure we limit the number of interactions between agents and the environment 400 across all environments. All baselines find it challenging to discover the optimal joint action with such limited data. Compared other algorithms, DAVE can quickly converge the optimal solution in all situations and exhibits robust performance, showing little variation as task difficulty increases. CW-QMIX can converge to the optimal joint solution in all cases except matrix game with 7.5. QPLEX performs better in matrix game II than in matrix game I, likely due to its sensitivity to the penalty of non-cooperation. In addition, as the task difficulty increases, both CW-QMIX and QPLEX show a significant drop in performance. Although QTRAN can converge to the optimal solution in matrix game I with 0, it may require more interactive samples for further learning. The remaining algorithms can converge to the optimal joint action (A, C*) the suboptimal joint action (C, A) in matrix game II with & = 0, but in the more difficult matrix game II can only converge to the joint action (B, B). We add additional action information the mixing network of QMIX, similar to that in DAVE, but QMIX still cannot converge to the optimal solution. In summary, DAVE can easily solve the above single-state matrix games and outperforms other IGM-based variants and the basic algorithm QMIX by a significant margin.
5.2 Multi-Step Matrix Game
Next, we evaluate the performance of each algorithm in the multi-step matrix game, The environment involves two agents, each with two actions. The initial state is monotonic, and agents terminate immedi- ately after receiving zero reward. If the game is not terminated after the first step, agents need continue to repeat the last joint action to obtain a non-zero reward. According to Figure 5] agents repeat either the upper-left joint action or the lower-right joint action eight times, resulting two dis- tinct terminal states. The optimal policy is for agents to repeat the upper-left joint action in the initial state and choose the lower-right joint action in the terminal state, which yields the highest return of 13. Local observations of agents in this
Nut: Step Maria Gamee
Og
O
0
DAVS
GRIEX
UNVEN
EXCMANO (Mas Masc)
Figure 5: Left: Illustrations of the multi-step matrix game. Right: Performance over time the multi-step matrix game.
7

---

DAVE
GPLEX
OW-OMIX
CW-OMI
MAVEN
OTRAN
FACMAC (Non-Mond)
IGM-Froo OMIX
GMIX
3602
2o Glog
Tymt
Em vs. 6m
25. V5 5z
8m vs. Dm
Mallin Teatt0
0.4
011 Text)
0.4
8.4
20-
Maddan Tww.
Tyst)
20
Figure 6: Performance comparison with baselines in different scenarios.
environment include the current step count and the last joint action. The protracted decision-making process requires agents to fully explore the state-action space to find the optimal policy. Therefore, this task can also test the exploration capacity of the algorithm.
Figure 5) shows the performance of DAVE and other baselines on the multi-step matrix game, Only DAVE successfully learned the optimal policy, while most other algorithms were trapped in the suboptimal policy with payoff 10. QPLEX can occasionally converge to the optimal policy, but in most cases it still cannot learn the optimal solution. Besides, to examine the contribution of the anti-ego exploration to the DAVE framework, we implemented DAVE without exploration this environment. The experimental results show that DAVE without anti-ego exploration still performs hetter than other baselines, but is significantly inferior to DAVE. Therefore, for some complex tasks, the anti-ego exploration mechanism can assist the agent in exploring novel state-action pairs and avoiding being restricted to suboptimal solutions.
5.3 StarCraft II
SMAC (29 the most popular experimental platform for multi-agent micromanagement, based on the famous real-time strategy game StarCraft II. SMAC offers diverse range of mini-scenarios, including homogeneous, heterogeneous, symmetric, and asymmetric problems. The goal of each task is to direct agents to defeat enemy units controlled by built-in heuristic AI. Agents information within their own visual range and share a reward function. So we assess the ability of DAVE to solve complex multi-agent cooperation problems in this environment.
only access
The version of StarCraft used this paper 4.10 and performance is not comparable between versions. We test the proposed method in different scenarios. In addition to the above- mentonedt methods, baselines also include IGM-Free QMIX that simply does not take the absolute value of the weight of the mixing network and keeps others unchanged. Both IGM-Free QMIX and FACMAC- nonmonotonic are IGM-free methods, but do not have convergence guarantees. The performance comparison between above two IGM-free methods and DAVE that is also IGM-free can reflect the necessity of our proposed ego policy network. In addition, to balance exploration and exploitation, we disable the anti-ego exploration mechanism in some easy scenarios and use it only in hard scenarios.
The experimental results are shown in Figure Undoubtedly, IGM-Free QMIX and FACMAC- nonmonotonic fail to solve almost all tasks. Without the IGM assumption, QMIX is unable to find the correct update direction. Therefore, the assistance of the ego policy network in the DAVE framework is essential for the convergence of IGM-free algorithms. QPLEX performs best in some easy scenarios but fails to solve the task most hard scenarios. Weighted QMIX performs well in the MMM2 scenario but has mediocre performance in most other scenarios. Although DAVE is not the best easy maps, still shows competitive performance. In hard scenarios, such as 5m vs ss_6m, 8m 9m and 10m vs 1mm, DAVE performs significantly better than other algorithms. We believe
8

---

M+5
M-10
M-50
M-100 MAMN
v 1ec
8m
Figure 7: Influence of the sample size M for DAVE.
Int=c.00
A_Int=1.0
vs. Tec
bi vs. 6m
MVN2
MAO
Figure 8: Results of DAVE with different Xinit- The action space for the three scenarios gradually increases from left to right.
that it may be that the characteristics these scenarios are far from the IGM assumptions, resulting in low sample efficiency of conventional value decomposition methods. Moreover, because DAVE IGM-free, it can guide the agent to learn the excellent policy quickly without restriction. Especially in the 6h_vy_8: scenario, only DAVE got non-zero median win ratio. In summary, even if the IGM principle is completely abandoned, DAVE can still successfully solve complex tasks and even outperforms existing (GM--ased value decomposition algorithms in certain challenging scenarios. MORE experimental results on other complex environments such as SMACv2 [6] and Multii-Aent These results demonstrate the excellent generalization of DAVE to different algorithms and environments.
MuJoCo [25] can be found in Appendix D
5.4 Ablation Study
Finally, we evaluate the performance of DAVE under different hyperparameter settings in SMAC scenarios. We focus the number of samples M and the initial value of the exploration coefficient Ximit According to Appendix A increasing the number of samples M improves the probability of drawing the optimal joint action. Nonetheless, due to time and space complexity constraints, is crucial to find an appropriate value for the hyperparameter M. Then we test DAVE with different M values in three scenarios and present results Figure The results indicate that larger M leads to faster learning speed for DAVE, which aligns with our intuition and proof. DAVE with M = 1 fails in all scenarios. When M increases to 10, the performance DAVE is already considerable. However, as M increases further, the performance enhancement it provides diminishes.
The equilibrium between exploration and exploitation is directly related the initial value of the exploration coefficient Aimi The final value Aend and the annealing period also affect the exploration, but only Amii is discussed in this paper. Aend is generally set to 0, so DAVE is only affected by anti-ego exploration in the early stages of training. Figure [8 shows the learning curve of DAVE under different Amni- Amat 0 means that DAVE does not perform anti-ego exploration. The addition of anti-ego exploration in easy scenarios will prevent the algorithm from exploiting better in the early stages, which will significantly slow down the training speed. Conversely, DAVE with high Ain values can achieve superior performance in hard scenarios. So moderately high Amii is beneficial for DAVE to visit key state-action pairs and converge to the optimal solution. summary, appropriate values of M and Ainit are indispensable for DAVE.
would be valuable explore the performance of DAVE with monotonic mixing network. As shown in Figure and Figure 10 DAVE with a monotonic mixing network still cannot converge to and the global optimal solution in matrix games. Since the ego policy is updated by an explicit search procedure, DAVE with a monotonic mixing network has more chances to converge to the optimal solution than vanilla QMIX. However, limited by fewer function families that can represented, the performance of DAVE with monotonic mixed network is much worse than that of DAVE and unstable. In addition, in the complex SMAC environment, the performance of DAVE with monotonic mixing network is similar to even better than that of QMIX, but worse than that of

---

DAVE
DAVE-Monn
QMIX
k-7.5
100
130
203
2-0
Figure 9: The learning curves of vanilla DAVE and DAVE with a monotonic mixing network on the matrix games.
DAVE
DAVE-Mono
QMX
10mm
5 0
0
5
0.
Tos1
12
18
044
23
Teml
12
1.5
04
28
Trans
12
Figure 10: The learning curves of vanilla DAVE and DAVE with monotonic mixing network SMAC.
vanilla DAVE. This is also substantial proof that the unconstrained factored mixing network indeed enhances performance.
Conclusion and Discussion
Inspired by the concept of dual self-awareness in psychology, we propose the IGM-free value decomposition framework DAVE in this paper. By dividing the agent network into two parts, ego policy and alter ego value function, DAVE is the first value decomposition method that completely abandons the IGM assumption. We find that the larger the number of samples in the explicit search procedure, the better the algorithm converges to the global optimum, but this will lead to high computational complexity. Therefore, we propose the anti-ego exploration mechanism, which can effectively prevent DAVE from falling into local optima, especially in problems with large action spaces, Experimental results in some didactic problems and complex environments show that DAVE outperforms existing IGM-based value decomposition methods. We believe that DAVE provides a new perspective on addressing the problems posed by the IGM assumption value decomposition methods. In our future research, we intend to concentrate on how to choose the exploration coefficients an adaptive manner. Further study of the issue would be of interest.
Acknowledgments and Disclosure of Funding
This project was supported by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No. XDA27050100.
References
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub W. Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, llya Sutskever. Jie Tang. Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. ArXiv, abs/1912.06680, 2019.
10

---

[2] Wendelin Boehmer. Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. In Proceedings of the 37th International Conference Machine Learning. ICML 2020, 13-18 July 2020, Viruual Event, volume 119 of Proceedings of Machine Learning Rexearch, pages 980-991. PMLR, 2020.
[3] Martin Buber and Walter Amoll Kaufmann. I and thou.
[4] Maria Dimakopoulou and Benjamin Van Roy. Coordinated exploration in concurrent reinforcement lcarning. In Jemnifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan. Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1270-1278. PMLR, 2018.
[5] Shelley Duval and Robert A. Wicklund. A theory of objective self awareness. 1972.
[6] Benjamin Ellis, Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Nicolaus Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. ArXiv, abs/2212.07489, 2022.
[7] Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson, Learning to communicate with deep multi-agent reinforcement learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg. Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2137-2145, 2016.
[8] Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Sheila A. Mcllraith and Kilian Q. Weinberger, editors. Proceedings of the Thirty-Second AAAI Conference Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18) 18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 2974 2982. AAAI Press, 2018.
[9] Sigmund Freud. The standard edition of the complete psychological works of sigmund freud. 1953.
[10] Tarun Gupta, Anuj Mahajan, Bei Peng, Wendelin Boehmer, and Shimon Whiteson. Uneven: Universal value exploration for multi-agent reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 3930 3941. PMLR, 2021.
[11] David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, Frunce, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[12] John Ryan Haule. Jung in the 21st century volume two: Synchronicity and science. 2010.
[13] Geoffrey F. Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313:504- - 507, 2006.
[14] Dorit S. Hochbaum. Approximation algorithms for np-hard problems. 1996.
[15] Jian Hu, Siyang Jiang. Seth Austin Harding. Haibin Wu, and Shih wei Liao. Revisiting the monotonicity constraint in cooperative multi-agent reinforcement learning. 2021.
[16] Shariq Iqbal and Fei Sha. Actor-attention-critte for multi-agent reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov. editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages
PMLR, 2019.
[17] Dapeng Li, Zhiwei Xu, Bin Zhang, and Guoliang Fan. From explicit communication to tacit cooperation: A novel paradigm for cooperative marl. arXiv preprint arXiv:2304.144 14656. 2023.
[18] lou-Jen Liu, Unnat Jain, Raymond A. Yeh, and Alexander G. Schwing. Cooperative exploration for multi-agent deep reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedingy of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Viruual Event, volume 139 of Proceedings of Machine Learning Research, pages 6826 6836. PMLR, 2021.
[19] Zhihao Liu, Zhiwei Xu, and Guoliang Fan. Hierarchical multi-agent reinforcement learning with intrinsic reward rectification. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE, 2023.

---

[20] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel. and Igor Mordatch. Multi-agent actor-critic environments. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio,
for mixed cooperative-compeliiee Hanna M. Wallach, Rob Fergus. S. V. N. Vishwanathan, and Roman Garnett, editors, Advunces Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6379 -390,, 2017.
[21] Anuj Mahajan. Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. MAVEN: multi-agent varia- tional exploration. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alcch-BBuc Emily B. Fox, and Roman Gameett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Syxtems 2019, NewrIPS 2019, December 8-14, 2019. Vancouver, BC, Canada, pages 7611-7622. 2019.
[22] Anuj Mahajan, Mikayel Samvelyan, Lei Mao, Viktor Makoviychuk, Animesh Garg, Jean Kossaifi, Shi- mon Whiteson, Yuke Zhu, and Animashree Anandkumar. Tesseract: Tensorised actors for multi-agen agent reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 7301-7312. PMLR, 2021.
[23] F. Oliehoek and Chris Amato. A concise introduction to decentralized pomdps. In SpringerBriefs in Intelligent Systems, 2016.
[24] David L. Pedersen. Cameral analysis: A method of treating the psychoneuroses using hypnosis. 1994.
[25] Bei Peng. Tabish Rashid, Christian Schröder de Witt, Pierre-Alexandre Kamienny, Philip H. S. Torr, Wendelin Boehmer, and Shimon Whiteson. FACMAC: factored multi-agent centralised policy gradients. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 12208-1222 12221, 2021.
[26] Phillip E. Pfcifer. A brief primer on probability distributions. Economics Educator: Courses, 2008
[27] Tabish Rashid, Mikayel Samvelyan, Christian Schrüder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässsan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4292-4301. PMLR, 2018
[28] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted QMIX: expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florim, Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020.
[29] Mikayel Samvelyan. Tabish Rashid, C. S. D. Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner. Chia-Man Hung. Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. ArXiv, Abs/1992.04043, 2019.
[30] Kyunghwan Son, Dacwoo Kim, Wan Ju Kang, David Hostallero, and Yung Yi. QTRAN: learning to factorize with transformation for cooperative multi-agent reinforeement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. ICML 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5887-5896. PMLR, 2019.
[31] Kyunghwan Son, Sungsoo Ahn, Roben Delos Reyes, Jinwoo Shin, and Yung Yi. Qtran++: Improved value transformation for cooperative aulti-agent reinforcement learning. arXiv: Learning, 2020.
[32] Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with back- propagation. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors. Advances in Neural Information Processing Systems 29; Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2244-2252, 2016.
[33] Wei-Fang Sun, Cheng- Kuang Lee, and Chun- Yi Lee. DFAC framework: Factorizing the value function via quantile mixture for multi-agent distributional q-learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021. Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9945 9954. PMLR, 2021.
12

---

[34] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech M. Czamecki, Vinícius Flores Zambakdi, Max Jaderberg. Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value- decomposition networks for cooperative multi-agent learning. Adaptive Agents and Multi-Agent Systems, 2017.
[35] Richard Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 16:285-2220 286, 2005.
[36] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang and Chongjie Zhang. QPLEX: duplex dueling multi- agent q-learning 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[37] Jianhong Wang. Wangkun Xu, Yunjie Gu, Wenbin Song, and Tim C. Green. Multi-agent reinforcement learning for active voltage control on power distribution networks, In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 3271-3284, 2021.
[38] Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. influence-based multi-agent exploration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Erhiopia, April 26-30, 2020. OpenReview.nct, 2020.
[39] Zhiwei Xu, Yumpeng Bai, Dapeng Li, Bin Zhang, and Guoliang Fan. SIDE: state inference for partially observable cooperative multi-agent reinforcement learning. In Piotr Faliszewski, Viviana Mascardi, Catherine Pelachaud, and Matthew F. Taylor, editors, 2 International Conference Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, New Zealand, May 9-13, 2022. pages 1400-14006 International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2022.
[40] Zhiwei Xu, Yunpeng Bai, Bin Zhang. Dapeng Li, and Guoliang Fan. HAVEN: hierarchical cooperative multi-agent reinforcement learning with dual coordination mechanism. Brian Williams, Yiling Chen, and Jennifer Neville, editors, Thirty- Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty- Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposiun on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 11735-11 11733. AAAI Press, 2023.
[41] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness ppo cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:2461 1-24624, 2022.
[42] Bin Zhang, Lijuan Li, Zhiwei Xu, Dapeng Li, and Guoliang Fan. Inducing stackelberg equilibrium through spatio-temporal sequential decision vocision-making in multi-agent reinforcement learning. In Proceedings of the Secood International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China, pages 353-361. ijeai.org, 2023.
13

---

A Proof
Proposition A.1. As long as the ego policy assigns non-zero probabilities all actions, this method will approach the objective described by Equation as the number of samples M increases.
Proof. Let (u²)* denote the individual actions corresponding to the global optimal joint state-action value function under state 8, and the optimal joint action is u² ((u")"..... (u*)"). Then the probability that the sampling procedure draws uu" is expressed as:
p(u²) (1 π 20 """ (u'Is) 1s))
II & (("))))7)))
where (-1-) T (0, is true for any action. So the second term (1 π (u (s))s (0, in the equation decreases as M increases, indicating that p(u") is positively correlated with the sample size M.
B Details of the Ego Policy Update
C (u)
High
Joint Action
Anti-Ego Exploration
Subset Too
Sort
Select
Encoder Decoder
Pollcy Net
Maximize
(u x
Learning
Anti-Ego Policy
Softmin
Logits f(r)
Softmax
Ego Pollcy
Sample
Sample |
Select ts
Joint Actions
Explored High Que
Joint Action
Unexplored Low Q [Maybe Underestimated)
Space
T
Explored & Low Q...
Low
High
Figure 11: Diagram of the update of ego policy, where U := Ü e00
{u 2go
M and
{ * (M Dots indicate different joint actions. The selected actions ut arg max Qaller (s, u =M) and ú arg max Lrcom "
u
The update process of ego policy is shown in Figure During the initialization of neural networks, the state-action value corresponding to the optimal joint action may be underestimated, making difficult for the ego policy to choose and then correct it, especially in problems with large action spaces. So use the Softmin operator to generate an anti-ego policy so that low-probability actions are more likely to be selected. However, not all low-probability joint actions are unexplored and undervalued; some are sub-optimal actions that have been exhaustively explored (orange dots in Figure [1). In order to select only unexplored joint actions, we distinguish the above two through the auto-encoder. Figure and Figure demonstrate the contribution of the anti-ego exploration mechanism.
C Implementation Details
C.1 Algorithmic Description
The pseudo-code of DAVE is shown in Algorithm
14

---

Algorithm Training Procedure for DAVE
Hyperparameters: Sample size M, discount factor exploration coefficients Annit Initialize the parameters of the neural networks shown in Figure
1: for each episode do
2: Get the global state $1 and the local observations 21 2i...... ................................ 21 . of agents
3: for + 1 to T do
4: for a f-
do Select action uz according to the ego policy cgo
5:
6:
end for
7:
Carry out the joint action u, {ut,... ........ uf
8:
Get the global reward 11 the next local observations 21 and the next state 81+-1
9:
end for
10:
Store the episode in the replay buffer D Sample batch of episodes B ~ Uniform(D)
11:
12:
Sample and obtain the joint action set U := {u, ~ (s))
[=1 for each trajectory in
13:
Update the parameters of the alter ego value function and the IGM-free mixing network according Equation (5)
14:
Obtain the anti-ego policies # of each agent
Sample and obtain Ú == (ft 0g0 n 000 (s) by sampling M times from the anti-ego
policy for each state in B
16:
Find the most novel joint action u" for each state s in B according Equation (
17:
Update the parameters of the ego policy according Equation (
18:
Update the parameters of the auto-encoder according Equation (6)
19:
Update the parameters of the target network periodically
20:
end for
C.2 Hyperparameters
Unless otherwise stated, the hyperparameter settings in different environments are shown in Table] [] which are exactly the same as the settings in PyMARL" All experiments in this paper are run on Nvidia GeForce RTX 3090 graphics cards and Intel(R) Xcon(R) Platinum 8280 CPU. For baselines, exploration is performed during training using independent e-greedy action selection. is annealing linearly from 1.0 to 0.05 and the annealing period of is 50000 steps. All experiments in this paper are carried out with 6 random seeds order avoid the effect of any outliers.
Name
Description
Value
Learning rate
0.0005
Type of optimizer
RMSProp
RMSProp param
0.99
e for optimizer
RMSProp param
0.00001
How many episodes to update target networks
200
Reduce global norm of gradients
10
Batch size
32
Capacity of replay buffer
5000
Discount factor
0.99
M
Sample size
100
Ajnit
Starting value for exploration coefficient annealing
0.5
Xend
Ending value for exploration coefficient annealing
0
Annealing period for the multi-step matrix game
25k
Annealing period for easy scenarios in SMAC Annealing period for hard scenarios in SMAC
500k
1.5m
Table 1: Hyperparameter settings.
https:/ggthib.cco//wwwwhhnngggg
15

---

D Additional Experiments and Details
D.1 Comparison between IGM-Based and IGM-Free Mixing Networks
To verify that IGM-based mixing network restricts the set of global state-action value functions that can be represented, synthesize artificial dataset represent the reward functions of two agents in single state matrix game. Each agent has 101 actions, which means u' E {0. 10....... 100}. In order only measure the influence of whether the mixing network follows the IGM assumption, fix the individual action-value function as follows:
u²
Qx(n') 100' ViE 21.
The reward value R for a joint action is given as follows:
R(u", u²) = sin (2 + cxp (Q2(u²)).
The ground-trutie joint value function is shown in Figure Next we calculate the joint action value Que for the selected u based on the given Q: and R. Most value decomposition approaches, such as QMIX and QPLEX, directly constrain the weights of the neural network be non-negative, so we use the mixing network in QMIX as the IGM-bbsed model to be trained. Meanwhile, the variant that removes the absolute function is viewed as the IGM-free model. It should also noted that we use the global state and the actions of all agents input to the hypernetwork that generates weights to enhance the expressiveness of the mixing network. Figure [2 represents the loss curve and Figure [3 illustrates the joint action-value function estimated by the two models.
Vodel Loes
GM Basod
13W res
389
(S0ap)
Figure 12: Left: Contours for ground-truth Qui, where x-axis and y-axis mean each agent's actions. Right: The loss curves for two models during training.
(a) 200 step
(b) 400 step
(c) 600 step
(d) 800 step
(c) 1000 step
(f) 200 step
(g) 400 step
(h) 600 step
800 step
(i) 1000 step
Figure 13: Contours for estimated Qua Colored values represent the values of Qua for (GM-based
((a)-ee) and IGM-free ((f-(j) mixing networks.
From the results, can be concluded that when the joint value function Qui not a simple monotone function with respect to Q., the representation ability of the IGM-based mixing network is obviously limited, and the convergence speed is also slower than IGM-free mixing network. However, possible to simply replace the IGM-based mixing network in the existing work with an IGM-free one, because this will cause each agent to fail find the individual action corresponding the optimal joint action during decentralized exccution. Of course, this is why propose the DAVE framework.
not
16

---

D.2 Policy-Based MARL Methods on Single-State Matrix Games
DAVE is not standard actor-critic framework in the strict sense. The ego policy in DAVE is updated in supervised learning manner, which maximizes the probability of the joint action corresponding to the optimal joint action value in the sample. Thus DAVE can explicitly search for optimal joint actions and easily solve non-monotonic problems. FACMAC still has to rely on the monotonic mixing network to update the critic; otherwise, the performance will deteriorate.
Moreover, the existing policy-based method that does not use IGM for the critic cannot solve the non-monotonic task in our paper. We test two representative policy-based algorithms, MAPPO and MADDPG, two didactic problems. We present the experimental results over 100 trials of 10000 episodes in Table 2 and Table 3 which demonstrate that although they do not explicitly introduce IGM, they still perform poorly.
k=0
k=6
k=7.5
-12
0
-12
0
8
-12
0
7.5
8
MADDPG
35%
45%
20%
27%
34%
31 %
8%
27%
35%
34%
4%
MАPPO
0%
100%
0%
0%
0%
100%
0%
0%
0%
100%
0%
Table 2: Proportion of different convergence results in Matrix Game I.
k=0
k=25
k=100
0
2
10
-25
2
10
-100
2
10
MADDPG MAPPO
51%
0%
49%
3%
45%
45%
7%
1%
20%
76%
3%
0%
0%
100%
0%
0%
100%
0%
0%
0%
100%
0%
Table 3: Proportion of different convergence results in Matrix Game II.
We carry out MADDPG and MAPPO in a policy-based setting, The above two algorithms cannot perfectly solve the single-state matrix problems even when more samples can be accessed. As the difficulty of the problem increases, the proportion of MADDPG converging to the optimal solution gradually decreases. MAPPO converges steadily to a sub-optimal solution.
D.3 Additional Explanations for Results in Multi-Step Matrix Game
In MAVEN's official code, the implementation of the multi-step matrix problem does not match the de- scription in its original paper, which is reflected in the terminal state corresponding to the bad branch (see Figure 14). This does not change the non-monotonicity of the initial state, but reduces the ex- pected return for agents to choose the bad branch. In other words, the above mistakes will make it EAS- IER for agents to learn the optimal policy. We rectified this error and retested all baselines for their real performance. Results in Figure shows that MAVEN performed worse than in its original paper.
0
0g
1
8 Times
Tomms
A Tiress
Intraal Stase
Initee State
4
Terminal
(a) Multi-step matrix game in MAVEN's paper.
(b) Multi-step matrix game in MAVEN's code.
Figure 14: The difference between the multi-step matrix game environment in the original paper of MAVEN and in the actual code of MAVEN.
17

---

D.4 Additional Introduction for SMAC Environment
The single-state matrix games and the multi-step matrix game are elaborated in the main body of the paper. Here we only elaborate on the SMAC environment. The action space of each agent is discrete. The version of StarCraft used in this paper is the latest 4.10. For a fair comparison, all algorithms the same version of StarCraft and SMAC. Table the details of the scenarios used in this paper.
Name
Ally Units
Enemy Units
Type
Homogeneous
2s_ se_s__sss Isc
2 Stalkers
Spine Crawler
Asymmetric
3 Stalkers
Stalkers
Heterogeneous
3s5z
Zealots
5 Zealots
Symmetrie
Homogeneous
2c_ _e_vs_64gg
Colossi
64 Zerglings
Asymmetric
Homogeneous
5m_vs_6m
5 Marines
Marines
Asymmetric
Homogeneous
3s_vs_5
3 Stalkers
5 Zealots
Asymmetric
Homogeneous
8m_vs_ vm_
8 Marines
9 Marines
Asymmetric
Homogeneous
10m_vs_11e 11m
10 Marines
11 Marines
Asymmetric
I Medivac
Medivac
Heterogeneous
MMM2
2 Marauders
Marauder
Asymmetric
7 Marines
Marines
Macro tactics
Homogeneous
6h vs 8z
6 Hydralisks
8 Zealots
Asymmetric
Table 4: Maps in different scenarios.
D.5 Additional Experiments on SMACv2
Although SMAC is popular multi-agent benchmark in recent years, it suffers from a lack of stochasticity. To remedy this significant deficiency, a more challenging SMACv2 environment is proposed. In SMACv2, team composition is randomly generated, which means that unit types are not fixed. Meanwhile, the start positions of all agents are also random. Finally, the sight range and attack range of the units are changed to the values from SC2 instead of fixed values. These three major changes make the tasks SMACv2 more challenging.
To assess the generalizability of our proposed algorithm, we evaluate the performance of DAVE in several representative scenarios of SMACv2. Some of the scenarios are symmetrical and others are asymmetrical. To ensure consistency with the original paper on SMACv2, we use the hyperparameter settings from PYMARL 15 To demonstrate that the DAVE framework is applicable to any existing value decomposition method and can transform the original algorithm into an IGM-free version with guaranteed convergence, we compare the performance of two classic algorithms, QMIX and QPLEX, with their variants respectively. Specifically, IGM-Free variants are obtained by simply removing all absolute functions from the original methods. We conduct the experiments using StarCraft version 4.10 and present the results in Figure 15 and Figure 16
The experimental results demonstrate that DAVE- DAVE-QMIX outperforms QMIX in most scenarios. particularly in the protoss_ 5 vs__ 5 and terran_ 5_ vs 6 scenarios. Notably, both DAVE-QMIX and IGM-Free QMIX are IGM-free methods. However, while IGM-Free QMIX is unable guarantee individual actions consistent with the optimal joint action, its performance is significantly worse than DAVE- -MIX and QMIX, and even fails in most scenarios. Similarly, DAVE- EPEEEX outperforms both vanilla QPLEX and IGM-Free QPLEX, despite the weight of its transformation module no longer being restricted to non-negative values. In summary, DAVE variants perform significantly
18

---

DAVE-QMIX
CMIX
13M- QMIX
protoss 5 us.
Siman 5 5.v8.5
5.
Tgsb)
T
protoss 5 vs
23
Sorran 5 vs.
20ng vs
Mean Weg
Timl
Figure 15: Comparisons of median win rate for variants of QMIX SMACv2.
DAVE-OPLEX
IGMM-Free OPLEX
protoss
Sarran vs.
2ang
protoss
torran
Mechn Teet:00
Figure 16: Comparisons of median win rate for variants of QPLEX on SMACv2.
better than original methods in scenarios that not meet the IGM assumptions, and also exhibit competitive performance in other scenarios, regardless of the environment complexity.
In addition evaluating different variants of the same algorithm, compare the performance of different baselines designed to solve the IGM problem. Since these baselines are all based on QMIX, we show their learning curves and that of DAVE-QMIX in Figure 17 It is evident from the results that the complex mechanisms introduced by other algorithms cause them to perform worse than the vanilla QMIX and DAVE-QN -MEI.. The experimental results on SMACv2 demonstrate the excellent generalization of DAVE to different algorithms and environments.
D.6 Additional Experiments on Multi-Agent MuJoCo
In order to test the performance of the same algorithm following IGM and not following IGM in complex environment, we also carry out vanilla QMIX, IGM-Free QMIX and DAVE on Multi- Agent MuJoCo (MA-MuJoCo). It is worth noting that the latter two do not require the IGM assumption all, which means that they can fit richer function classes but it is more difficult to find the optimal joint action. Since most of the conventional value decomposition methods are only applicable to environments where the action space is discrete, we discretize the action space of the MA-MuloCo environment reasonably. Each joint in the robot is considered as an agent. We use uniform quantization to discretize continuous actions and modify the action space of each agent to K-1
2: 4 = { k-1 K- } j=
instead of the original 4 = [-1, K is a hyperparameter and in this paper
K = 31. In addition, MA-MuloCo is a partially observable environment, and each agent (joint) can only observe the information of neighbor agents (joints). Figure [8] shows three classic scenarios. Numbers in brackets indicate the number of agents (joints) in each task.
can be seen from Figure 19 that even without the IGM assumption, DAVE can still perform better than vamilla QMIX in most scenarios, especially in the Hopper and Humanoid scenarios. IGM-Free
19

---

DAVE-QMIX
OW-QMIX
CW-QMIX
CTTAN
QMIX
protoss 5 us.
Sammin vs.
5.
198
T(9b
Tysl
protoss 5 vs
Sarian vs.
z0ng vs
Mean Weethe
Tomb
Timl
Figure 17: Median win rates range of SMACv2 mini-games. Note that only DAVE-QMIX is IGMe-ree.
(a) Hopper (3)
(b) HalfCheetah (6)
(c) Humanoid (17)
Figure 18: Illustration benchmark tasks in Multi-Agent MuJoCo.
QMIX fails all tasks. In summary, in the case of the loss function related to reinforcement learning, QMIX with the dual self-awareness framework performs better than vanilla QMIX in all cases. Even in scenarios with large action spaces like Humanoid, which contains 17 2.26 10 joint actions, DAVE still outperforms QMIX. Like the results in the main text, the experimental results in MA-MuJoCA further prove that DAVE is not only suitable for didactic examples, but also capable of complex scenarios.
DAVE
CMIX
IGM-Frre QMIX
HarCheatah
Humanoid
5
Colock
Figure 19: Median episode return on different MA-MuloCo tasks. Note that only QMIX follows the IGM assumption.
D.7 Additional Notes on Model Size
We give the relative size of neural network models for some value decomposition variants over QMIX in Table The model size here refers to the number parameters each algorithm needs to train in the MMM2 scenario.
The number of parameters in the value decomposition framework mainly depends on the mixing network structure. The DAVE framework only has three more fully connected layers (an additional ego model) than QMIX, so DAVE has the smallest change compared to QMIX among the variants. Nevertheless, DAVE still performs better than other baselines related to IGM. The most important thing is that DAVE can quickly converge to the optimal solution the matrix game, while QMIX cannot.
20

---

Algo
QMIX
DAVE
CW-QMIX
OW-QMIX
MAVEN
QPLEX
Relative Size
100%
132%
373%
373%
167%
142%
Table 5: The relative size of neural network models related to reinforcement learning for some value decomposition variants over QMIX.
E More Related Work
By improving the joint exploration of all agents, UneVEn [0]] addresses the issue that stems from the monotonicity constraint in representing the joint-action value function. However, in some complex environments such SMAC, its additional complex mechanism leads to significantly slower convergence. TESSERACT [22] views the Q-function as a tensor and utilizes low-rank tensor approximations to model agent interactions relevant the task. DCG [2] factors the joint value function of all agents into payoff's between pairs of agents based on a coordination graph, but requires additional domain knowledge and violates CTDE principles. QTRAN+ [1 is an improved version of QTRAN, however it still does not completely remove absolute functions. STEP 42 can solve non- non-monotonic problems but focuses on how to find Stackelberg equilibrium via asynchronous action coordination. Besides, some value decomposition methods 33 from the distributional perspective replace the original IGM principle with the distribution-stye IGM to deal with highly stochastic environments. The DAVE framework we proposed does not change the loss function of the underlying value decomposition algorithm, and only requires simple modifications to make it IGM-freee The DAVE-based value decomposition method can outperform the original algorithm in scenarios where the IGM assumption not satisfied, and it can also have competitive performance in other scenarios.
There has been lot of excellent work on exploration in the field of multi multi-agent agent reinforcement learning. Vnn uses novel action-selection schemes during exploration to overcome relative overgeneralization EITI/EDTI [3 measure the amount and value of one agent's s influence on the other agents' exploration processes to improve exploration. Dimakopoulou and Roy 4 present several seed sampling schemes for efficient coordinated exploration concurrent reinforcement learning. CMAE [18] uses target policies to maximize returns and exploration policies to collect unexplored data, which similar to our work. However, CMAE focuses on solving the sparse-reward task, whereas anti-ego exploration in DAVE alleviates the suboptimal convergence caused by limited search. In addition, the anti-ego exploration mechanism also leverages the characteristics of the auto-encoder to further narrow the acteon space that needs to be explored.
21


Reviews:
[{"review_id": "5MKHrCvwW2", "reviewer": "Reviewer_Yoei", "summary": "The paper proposes a novel value decomposition framework for MARL based on the notion of dual self-awareness in psychology. The framework, called DAVE, consists of two neural network models for each agent: the alter ego value function model and the ego policy model. The former participates in the evaluation of the group to solve the global credit assignment problem, and the latter helps the former to find the optimal individual action through an explicit search procedure. The paper claims that DAVE is the first value decomposition method that completely abandons the Individual Global Max (IGM) assumption, which requires consistency between local and global optimal actions. The paper also introduces an anti-ego exploration method to prevent the ego policy from getting stuck in a bad local optimum. The paper evaluates the performance of DAVE in both simple and complex environments, such as StarCraft II and Multi-Agent MuJoCo, and compares it to other popular baselines. The paper shows that DAVE can solve non-monotonic problems that challenge existing value decomposition methods, and can achieve competitive performance despite the absence of IGM. The paper argues that DAVE provides a new perspective on addressing the problems posed by the IGM assumption in value decomposition methods.", "questions": "Can you provide more context into what is needed to tune exploration coefficients dynamically?\nHow well can this method generalize to other variants of the MARL problem (e.g. mixed incentive, social dilemmas)? \nCan DAVE provably increase the solution concept reached by agents in social dilemmas / game theoretic settings?\nHow do you balance the trade-off between exploration and exploitation in the anti-ego exploration method?", "limitations": "The main limitation here is the computational cost of large values of M. This does not prevent strong performance in the environments studied in this work, but may be expensive in more complex and/or real-world environments. There are also other small limitations, e.g. tuning exploration coefficients.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Improving the efficacy of CTDE and value decomposition-based approaches to cooperative MARL is an important topic. The paper's investigation of the role of the IGM principle in CTDE approaches is novel and should be of interest to the broader MARL and RL communities. The method is supported by empirical results in both small-scale, well-principled environments and larger-scale, complex environments; and is comprehensive in its comparison to baselines. The paper is well-written and clear.", "weaknesses": "- The related work in the main paper is a bit sparse given the depth of work in this area. Moving more of the discussion from Appendix E into the Related Work section would improve the contextualization of this work.\n- Given the importance of sample size M, it would have been nice to see a greater exploration of the performance/computational cost trade-off. For example, how does M scale in more realistic environments? Are there environments where an unreasonably high value of M is needed? In the MMM2 example in Fig. 7 it looks like the learning signal is just beginning.\n- There is limited discussion of limitations, future work, and societal impact.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and inspiring comments! We sincerely thank you for taking the time to read our paper and the supplementary material carefully! We hope we can address your concerns below.\n\n**Q1**: Can you provide more context into what is needed to tune exploration coefficients dynamically?\n\n**A1**: We propose an anti-ego exploration mechanism, which is able to select the action least likely to be selected by the ego policy in the current state $s$ (implemented by softmin) and the action least seen before (implemented by autoencoder). Through these two filterings, *relatively* unexplored actions can be effectively selected as long as the number of samples $M$ is large enough. In order to allow the agent to choose these unexplored actions in real interactions, we use NLL loss to increase the probability of selecting these actions, as shown in Equation 8. We use the exploration coefficient $\\lambda$ to control the strength of exploration. We emphasize \"relative\" because almost all actions have been fully explored in the later stages of training, but there will always be actions that are *relatively* unexplored compared to others. Therefore, $\\lambda$ must decrease over time and must eventually be 0, otherwise it will be unfavorable for exploitation in the later stage of training. Through some experiments, we found that the annealing period of $\\lambda$ should account for 20%-40% of the entire training period. That is to say, anti-ego exploration is only performed during the first 20%-40% of the training period. A comparison of DAVE with different $\\lambda$ initial values is shown in Figure 8.\n\n**Q2**: How well can this method generalize to other variants of the MARL problem (e.g. mixed incentive, social dilemmas)?\n\n**A2**: We focus on how to completely abandon the IGM principle, which is specific to value decomposition methods. The value decomposition method is only applicable to decentralized partially observable Markov decision process (Dec-POMDP) tasks, in which all agents share a reward function. Therefore DAVE cannot be directly applied to other variants of the MARL problem where agents have their own reward functions. In order to enable DAVE to quickly generalize to problems with mixed incentives or social dilemmas, there is a simple way to artificially design a global reward function related to all individual reward functions. In this way, DAVE is also able to converge to the optimal solution, just like the result of the matrix game in our paper.\n\n**Q3**: Can DAVE provably increase the solution concept reached by agents in social dilemmas / game theoretic settings?\n\n**A3**: Global reward game, such as Dec-POMDP task, can be formed as Markov game with a global reward. Its aim is learning a stationary joint policy so that no agent tends to unilaterally change its policy to maximize cumulative global rewards and a Markov equilibrium is reached. For example, for the two single-state matrix games in our paper, their corresponding Markov equilibrium are (A,A) and (A,C). As shown in Figure 4, DAVE can always quickly converge to the equilibrium solution while other IGM-based algorithms cannot. For other problems such as social dilemmas, we can try to convert the original problem into a Markov game with a global reward according to the method mentioned in **A2** and then solve it through DAVE.\n\n**Q4**: How do you balance the trade-off between exploration and exploitation in the anti-ego exploration method?\n\n**A4**: First of all, in order to allow the agent to exploit better, the exploration coefficient $\\lambda$ is generally set to $\\lambda\\in[0,1)$ in Equation 8. The two terms in Equation 8 denote exploitation and exploration, respectively. Second, as mentioned in **A1**, the exploration in the later stage of training is less meaningful, so $\\lambda$ should anneal over time and eventually become 0.\n\n**Q5**: It would have been nice to see a greater exploration of the performance/computational cost trade-off. For example, how does $M$ scale in more realistic environments? Are there environments where an unreasonably high value of $M$ is needed?\n\n**A5**: Thank you for your suggestion, we will add the experimental results in more realistic environments in the next version of the paper. According to our current version of the paper, we believe that $M=100$ is sufficient and affordable for several currently popular multi-agent benchmarks. It can be seen from Figure 7 that in most environments, when $M$ is $10$ or even $5$, the performance of DAVE does not drop or drops very little compared to $M=100$. Furthermore, the results for *Humanoid* task shown in Figure 16(c) also illustrate that $M=100$ is sufficient even for tasks with a larger number of agents and actions.\n\n**Q6**: Moving more of the discussion from Appendix E into the Related Work section would improve the contextualization of this work. And there is limited discussion of limitations, future work, and societal impact.\n\n**A6**: Thank you for your suggestion! We will correct it in the next version of the paper.\n\nWe really appreciate your comments and they really help us improve our paper! And we also appreciate it if you have any further comments.\n"}, {"review_id": "VYEgBUAPXa", "reviewer": "Reviewer_uEFU", "summary": "This paper presents DAVE, an IGM-free value decomposition method, to enhance coordination ability in MARL. Drawing inspiration from the concept of dual self-awareness in psychology, DAVE consists of two components: an ego policy responsible for executing actions, and an alter ego value function involved in credit assignment and value estimation. By incorporating an explicit search procedure, DAVE eliminates the need for the IGM assumption and updates the actor using supervised signals instead of policy gradients. Additionally, the authors propose a novel anti-ego exploration mechanism to prevent the algorithm from being trapped in local optima. The method demonstrates impressive performance across a range of cooperative tasks, highlighting its effectiveness.", "questions": "1. Considering that [2] states, \"On SMAC, FACMAC-nonmonotonic performs similarly to FACMAC on easy maps, but exhibits significantly worse performance on harder maps,\" it would be worth discussing why the authors chose to compare DAVE with FACMAC-nonmonotonic instead of FACMAC (fairness may not be an issue).\n2. There appears to be a notable discrepancy between the performance curves reported in Figure 6 of the current paper and Figure 7 in [2], despite both versions using SMAC 4.10. Could you explain the reasons behind this divergence? \n3. While the elimination of the IGM assumption is touted as a significant contribution of DAVE, it would be valuable to explore the performance of DAVE with a monotonic mixing network (or others with IGM assumption). Does the unconstrained factored mixing network indeed enhance performance?\n\n[1] Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2019, May). Exploration by random network distillation. In ICLR, 2019.\n\n[2] Peng, B., Rashid, T., Schroeder de Witt, C., Kamienny, P. A., Torr, P., Böhmer, W., & Whiteson, S. (2021). Facmac: Factored multi-agent centralised policy gradients. *In NeurIPS, 2021*.\n\n[3] Simmons-Edler, R., Eisner, B., Mitchell, E., Seung, S., & Lee, D. (2019). Q-learning for continuous actions with cross-entropy guided policies. *arXiv preprint arXiv:1903.10605*.\n\n[4] Neumann, S., Lim, S., Joseph, A., Pan, Y., White, A., & White, M. (2018). Greedy Actor-Critic: A New Conditional Cross-Entropy Method for Policy Improvement. In ICLR, 2023.\n\n[5] Wang, Y., Han, B., Wang, T., Dong, H., & Zhang, C. (2020). Off-policy multi-agent decomposed policy gradients. *arXiv preprint arXiv:2007.12322*.\n\n[6] Ackermann, Johannes, et al. \"Reducing overestimation bias in multi-agent domains using double centralized critics.\" *arXiv preprint arXiv:1910.01465* (2019).\n\n[7] Wang, Tonghan et al. “Influence-Based Multi-Agent Exploration.” *ArXiv* abs/1910.05512 (2019)", "limitations": "As mentioned in Weakness.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. By employing NLL instead of policy gradients to update actors, DAVE effectively addresses the challenges associated with using unconstrained (without IGM assumption/monotonicity) factored critic.\n2. The utilization of AE loss in Anti-Ego Exploration to assess the novelty of joint-action is intriguing, albeit bearing some similarities to RND [1].\n3. The paper is well written and easy to follow. The integration of psychological concepts with algorithm design is a unique aspect that captured my interest. Furthermore, each component of the overall framework is thoroughly elucidated.\n4. DAVE demonstrates impressive performance in both matrix games and SMAC, with clear discussions regarding its limitations.", "weaknesses": "1. While the authors claim that DAVE is the first method to completely eliminate the IGM assumption in value decomposition approaches, it's worth noting that FACMAC-nonmonotonic [2] also achieves full IGM-free. Although DAVE may outperform FACMAC-nonmonotonic, the contribution of DAVE might have been overestimated. \n2. The discussion on related works is insufficient, and it would be beneficial to include references to value decomposition actor critic methods [2][5].\n3. Minor 1: The idea of applying NLL loss bears similarities to previous works that combine Cross-Entropy Methods with Policy Improvement [3][4] (discussion can be included).\n4. Minor 2: It appears that exploration has a significant impact on the performance of DAVE, making it a crucial component that cannot be disregarded, unlike simple epsilon-greedy exploration. It would be advisable to include comparisons between DAVE and other MARL exploration methods, such as EITI/EDTI[6] and CMAE[7], in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and inspiring comments! We hope we can address your concerns below.\n\n**Q1**: Why did the authors choose to compare DAVE with FACMAC-nonmonotonic instead of FACMAC?\n\n**A1**: FACMAC is an IGM-based method. Because in the canonical implementation of FACMAC, the mixing network is a non-linear monotonic function, as in QMIX. And FACMAC focuses on continuous action space tasks rather than non-monotonic tasks. However, DAVE focuses on non-monotonic tasks, so the baselines we selected are methods related to relaxing the IGM assumption, except for the basic algorithm QMIX. The comparison of the performance of the IGM-based method and its DAVE variant on different tasks is most indispensable, so we compare DAVE-QMIX with QMIX, and DAVE-QPLEX with QPLEX in the appendix. The DAVE variant outperforms or equals its base algorithm in almost all scenarios. Thanks for your suggestion. We will add the comparison of DAVE-FACMAC and FACMAC in the next version of our paper.\n\n**Q2**: There appears to be a notable discrepancy between the performance curves reported in Figure 6 of the current paper and Figure 7 in the origin paper of FACMAC.\n\n**A2**: Figure 6 in our paper only contains 2 tasks that are also found in Figure 7 in the origin paper of FACMAC, namely *MMM2* and *2c_vs_64zg*. At the same time, only two algorithms are shared, namely QMIX and QPLEX. In our paper, the final winning rates of these two algorithms in these two scenarios are similar to those in Figure 7 in [1]. The difference in the training process may be due to the relatively large variance of the median.\n\n**Q3**: It would be valuable to explore the performance of DAVE with a monotonic mixing network.\n\n**A3**: Thank you very much for the reminder! This is indeed a valuable experiment. We added some experimental results, mainly the performance of DAVE with a monotonic mixing network on different domains. As shown in Figure 18 and 19 in the PDF we added in the global response, DAVE with a monotonic mixing network still cannot converge to the global optimal solution in matrix games. In addition, in the complex SMAC environment, the performance of DAVE with a monotonic mixing network is similar to or even better than that of QMIX, but worse than that of vanilla DAVE. This is also substantial proof that the unconstrained factored mixing network indeed enhances performance.\n\n**Q4**: Although DAVE may outperform FACMAC-nonmonotonic, the contribution of DAVE might have been overestimated.\n\n**A4**: Thank you for your correction. We will declare our contribution more rigorously. Before that, we declared that DAVE was the first method to completely eliminate the IGM assumption because we thought that FACMAC-nonmonotonic simply changed the mixing network of FACMAC from monotonic to non-monotonic. The resulting consequences caused by FACMAC-nonmonotonic have not been considered in depth. This also leads to the poor performance of FACMAC-nonmonotonic in most tasks.\n\n**Q5**: It would be beneficial to include some references and discussions.\n\n**A5**: Thank you for your suggestion. More discussion of related work can be found in Appendix E. And we will supplement the relevant literature and experiments you mentioned.\n\nWe really appreciate your comments and they really help us improve our paper! And we also appreciate it if you have any further comments.\n\n**Reference**\n\n[1] Peng, B., Rashid, T., Schroeder de Witt, C., Kamienny, P. A., Torr, P., Böhmer, W., & Whiteson, S. (2021). Facmac: Factored multi-agent centralised policy gradients. *In NeurIPS, 2021*."}, {"review_id": "39KlT68D61", "reviewer": "Reviewer_NJQv", "summary": "This paper proposes a novel MARL algorithm, Dual self-Awareness Value dEcomposition (DAVE), which avoids the IGM constrain obeyed by most of the previous researches. The algorithm introduces three different policies including Ego Policy, Alter Ego Policy and Anti-ego Policy, where the Ego Policy tries to fit the optimal actions of the Alter Ego Policy and the Anti-ego Policy serves as an exploration strategy. Empirical results show that DAVE outperforms the baseline methods on several cooperative tasks.", "questions": "1. The optimization of the $Q_{\\text{tot} }^\\text{{alter}}$ network heavily relies on the action sample set $U^{\\text{ego}}$ due to the computation of the target. However, unlike common MARL algorithms where the policy receives gradients through backpropagation, in DAVE, the Ego policy and $Q_{\\text{tot}}^\\text{{alter}}$ network are trained individually. Although *Proposition A.1* guarantees that objectively existing $u^*$ will be sampled when $M$ is sufficiently large, it does not ensure convergence to the optimal point in the optimization process. \nThen, there are two different models need to optimize and they are highly related, how do you ensure the convergence of them? \n\n2. There are concerns regarding whether Equation (8) truly returns \"the most novel joint action.\" As mentioned in *1.b)* in the *Weakness* part, reaching state $s$ can be challenging. However, the action set $\\hat{U}^{\\text{ego}}$ is sampled from the Anti-Ego Policy. When $M$ is limited, the suboptimal-Q actions are hard to be sampled  both in the true interaction process (due to the hardness of reaching $s$) and in the Anti-Ego Policy sampling process (as their Q-values are not sufficiently low).\n\n3. \n    *a*) Most figures in the *Experiment Section* display the median performance. Can you replace them with the mean performance as it is more often used in other works?\n\n    *b*) The results on SMAC of DAVE is based on QMIX as stated in the paper. To facilitate better comparison, it is suggested to use a more distinguishable color for the QMIX curve.\n\n    *c*) If i've identified the QMIX curve correctly, DAVE exhibits significant superiority over QMIX in Figure 6. However, in the Appendix (Figure 13, 15, 17), DAVE shows only a slight advantage over QMIX, can you analyze the reason for this?\n\n4. line 312: \"The experimental results show that DAVE without anti-ego exploration still performs better than other baselines.\" Actually, I do not get this conclusion. In contrast, the green curve for \"DAVE w/o Exploration\" demonstrates a significant drop compared to the QMIX baseline. Does this mean that the exploration part plays the most important role in DAVE but not the relaxation of the IGM principle?\n\n5. It is unclear why the curve for *M=100* does not outperform the others in Figure 7 (2s_vs_1sc, 5m_vs_6m). Further explanation is required to address this discrepancy.\n\n", "limitations": "Limitations: This work is mostly aimed to solve the cooperative Multi-agent tasks.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed DAVE algorithm avoids the IGM principle, which typically requires consistency between local and global optimal actions. This deviation enables DAVE to leverage the expressive capabilities of neural networks, particularly the mixing network component.\n2. The Anti-Ego policy, a key component as the DAVE's exploration strategy, adopts the idea of count-based exploration techniques such as RND [1] and \"Ensemble\" in model-based RL. It quantifies the familiarity of $(s,a)$ pairs by evaluating the reconstruction loss of an auto-encoder.\n3. Experimental results on Matrix Games show that DAVE does avoid local optima. Additionally, experiments conducted on SMAC, MA-MuJoCo indicate that DAVE-QMIX performs comparably to QPLEX and other baseline algorithms.\n\n\n[1] Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018). Exploration by random network distillation. arXiv preprint arXiv:1810.12894.", "weaknesses": "1. **Methodology**: \n\n    *a*) The Ego Policy only approximates the global optimal actions $u^{*}$ by NLL loss, which means the equivalence between the Ego Policy and the global optimum is not strictly guaranteed while it is hold strictly by IGM principle.\n\n    *b*) The basic idea of the exploration strategy is not quite novel. Simultaneously, while sampling from the Anti-Ego Policy guarantees the selection of low-Q actions, it does not guarantee that suboptimal-Q actions have been explored enough in the environment when the $M$ is not large enough. This is because reaching state $s$ is not guaranteed. As a result, the Q-values associated with these actions are prone to overestimation or underestimation, and the exploration policy tends to overlook this aspect of the action space.\n\n2. **Experiments**: The results on SMAC and MA-MuJoCo are a bit confusing to me and i have listed my concerns in the *Qustions*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and constructive comments! We are glad that you have read our paper and the supplementary material carefully! We hope we can address your concerns below.\n\n**Q1**: The Ego policy and $Q_\\text{tot}^\\text{alter}$ network are trained individually. How to ensure the convergence of these two highly related models?\n\n**A1**: The training of these two models is complementary rather than independent. After evaluation by $Q_\\text{tot}^\\text{alter}$ network, the action $\\boldsymbol{u}^\\ast$ with high $Q_\\text{tot}^\\text{alter}$ will be used to train the ego policy. On the other hand, the new trajectory obtained by the ego policy interacting with the environment will help the $Q_\\text{tot}^\\text{alter}$ network to fit the real return. The above-mentioned update paradigm of DAVE is similar to Best-of-N sampling, also known as Rejection Sampling [1], which is an effective method in the recently popular large language model field.\n\n**Q2**: For the anti-ego exploration mechanism, it does not guarantee that suboptimal-Q actions have been explored enough in the environment when the $M$ is not large enough. Does Equation (8) truly return \"the most novel joint action\"?\n\n**A2**: Your analysis is very detailed and accurate. When $M$ is limited, it does affect the performance of the dual self-awareness framework and anti-ego exploration mechanism in DAVE. We illustrate this in both Figure 7 and Section 6. However, for most of the existing multi-agent benchmarks and real tasks, $M=100$ or even $10$ can make DAVE's performance close to or better than the existing IGM-based method. The resulting computational cost is entirely affordable.\n\n**Q3**: Can you replace the median performance in the Experiment Section with the mean one?\n\n**A3**: Thanks for your reminder. We will replace the aggregate function from median to mean.\n\n**Q4**: It is suggested to use a more distinguishable color for the QMIX curve in figures.\n\n**A4**: Thank you for your advice. We will correct it in the next version of our paper.\n\n**Q5**: DAVE exhibits significant superiority over QMIX in Figure 6 but shows only a slight advantage over QMIX in the appendix.\n\n**A5**: There are two main reasons. The first and most important reason is the illusion brought by the different ranges of the X-axis. In the appendix, we run all experiments for 5M or 6M steps. But in Figure 6, most tasks are run for 2M steps. This leads to the fact that DAVE's advantage over QMIX *appears negligible* in the appendix. This is a visual deception. For example, in the *terran_5_vs_6* task in Figure 15, DAVE can achieve 15% winning rate at 2M steps, while QMIX needs to achieve it at 3M. \n\nThe second reason is that SMACv2 is more challenging than SMAC. Since the position, number, and category of agents are constantly changing in each episode, there may be situations where our agents are at a complete disadvantage. So there is no guarantee that the winning rate can theoretically reach 100%. Therefore, DAVE's performance can only be closer to a certain winning rate value than QMIX, but cannot exceed this value.\n\n**Q6**: I cannot draw such a conclusion that DAVE without anti-ego exploration still performs better than other baselines.\n\n**A6**: In Figure 5 Right, although the median of DAVE w/o Exploration has always been 10, its upper quantile reaches 12 first compared to other baselines. If we change the aggregation function from median to mean, the result will be more obvious. As shown in Figure 20 in the PDF we added, although the mean test return of DAVE w/o Exploration is lower than that of DAVE, it is still higher than other baselines. Furthermore, in the single-state matrix game, we implement various algorithms under uniform visitation to disregard the influence of exploration capabilities, as described in lines 269 to 270. Therefore,  in Figure 4, DAVE is equivalent to DAVE w/o Exploration.\n\n**Q7**: It is unclear why the curve for *M=100* does not outperform the others in Figure 7.\n\n**A7**: This discrepancy is due to the randomness of sampling. In *2s_vs_1c* scenario, the performance of DAVE with $M=100$ has been stable at 100% win rate while other performances are unstable. In addition, in *MMM2* map, the performance of DAVE with $M=100$ is also significantly better than others. In most scenarios, the difference in performance is not obvious, but this also shows that in common multi-agent tasks, $M=10$ or even $M=5$ is enough.\n\n**Q8**: The equivalence between the Ego Policy and the global optimum is not strictly guaranteed while it is held strictly by the IGM principle.\n\n**A8**: The IGM-based method cannot guarantee its strict convergence to the global optimum due to the limited family of functions. However, DAVE has a better chance of converging to the global optimal solution than the IGM-based method. This is why IGM-based methods tend to fall into local optimal solutions in matrix games, while DAVE can quickly converge to global optimal solutions.\n\nWe really appreciate your comments and they really help us improve our paper. And we also appreciate it if you have any further comments or improve your score.\n\n**Reference**\n\n[1] Touvron, Hugo et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. 2023.\n\n"}, {"review_id": "CFxUFHEGJE", "reviewer": "Reviewer_rpvy", "summary": "\nThis paper proposes a different approach for multi-agent RL algorithms that does not depend on individual global max (IGM), but instead builds a new framework, DAVE,  based on dual self-awareness. The algorithm is shown to perform favorably in several testing cases.\n", "questions": "Question: from figure 2, it appears that the alter ego model calculates the q value function and is similar in concept to a critic network. The ego model also computes the policy which is the same as an actor network. How is the dual self-awareness model different from an actor-critic model? The description in lines 193-194 appears to limit the self-awareness model to be a special case of a general actor-critic model. ", "limitations": "\nThe proposed model “dual self-awareness” requires a specific sampling procedure to get the actor policy which renders this framework as a special case of actor-critic algorithms.  \n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well organized and the sections’ structure serve their purpose in illustrating this new framework. The experimental results also corroborate the authors’ claims and show that the proposed framework/algorithm, DAVE, performs better than other MARL algorithms. \n", "weaknesses": "The main weakness in this paper is the lack of justification of differences between DAVE and the actor-critic models. In addition, since this paper proposes a new algorithm, it would be nice to have the pseudocode be in the main text. Overall, a better analysis of the sampling procedure is needed to justify the new DAVE framework.\n \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the sincere comments! We thank you for pointing out the many strengths of our paper. We hope we can address your concerns below.\n\n**Q1**: How is the dual self-awareness model different from an actor-critic model?\n\n**A1**: We apologize for misleading you into thinking that DAVE is a special case of actor-critic algorithms. We bolded \"DISTINCT\" in lines 193-194 to mean that DAVE is *completely* different from other actor-critic and policy-based algorithms. We then pointed out the reasons as follows:\n\n> The actions sampled from the ego policies remain independent and identically distributed (IID), so the ego policy is trained through *supervised learning* rather than *policy gradient* used in policy-based methods.\n\nAll actors in the actor-critic framework are updated in the policy gradient manner, so the ego policies cannot be regarded as the actor model in the actor-critic framework. The ego policy model in DAVE is more like Best-of-N sampling, or rejection sampling [1] in the recently popular large language model field. We do not use policy gradient to update the ego policy because it is not effective for non-monotonic tasks, as shown in Appendix D.2. Details of the ego policy update can be found in Appendix B.\n\nThank you so much for your helpful comments. And we also appreciate it if you have any further comments or improve your score.\n\n**Reference**\n\n[1] Touvron, Hugo et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. 2023."}, {"review_id": "lDdkrMJSZa", "reviewer": "Reviewer_BzQX", "summary": "The paper focuses on value decomposition without IGM constraints by proposing Dual self-Awareness Value dEcomposition (DAVE) framework. DAVE is inspired by dual self-awareness studied in psychology and uses an ego model, i.e., a policy for each agent for actual action selection and an alter ego model, i.e., a value function to address credit assignment without the IGM constraint. To avoid premature convergence to poor local optima, an anti-ego exploration method is proposed. DAVE is evaluated in several domains and compared with many state-of-the-art MARL approaches.", "questions": "I do not fully understand the purpose of the alter ego models. Since the policies are trained to directly maximize the joint-Q function, why introducing a „detour“ to the mixing network through the individual Q-functions? Wouldn’t simply feeding the resulting joint action (e.g., as one hot vector) be sufficient to compute the TD-target for Eq. 4?", "limitations": "Some limitations like the additional compute through sampling actions are mentioned in the text.\n\nPotential negative societal impact is not discussed at all.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper proposes an interesting approach inspired by knowledge from psychology.\n\nThe evaluation is structured well by starting with small and tractable problems first and scaling up. Many state-of-the-art MARL algorithms are used as baseline for sufficient comparison.", "weaknesses": "The paper strongly focuses on the omission of IGM, which is motivated as a limitation of MARL. However, DAVE introduces additional complexity that at least doubles the computational effort compared to the baselines, e.g., two networks are used per agents and the exploration mechanism adds more parameters on top of that (note that despite doubled parameters, DAVE does neither perform twice as well nor learn twice as fast to fully compensate for the additional effort). Therefore, the fairness of comparison is questionable. Furthermore, I am not sure about the purpose of the individual Q-functions, since they are not used to maximize the joint Q function as usual. The ego model/policy is used instead thus individual Q-functions could be completely omitted as far as I understood the text.\n\nThe initial motivation of dual self-awareness in the introduction implies that ego and alter ego models have a symmetric relationship. However, the ego model represents a policy while the alter ego model represents a value function, which is sometimes called \"alter ego model\" and sometimes \"alter ego value function model\", which is confusing to read.\n\nThe maximization of the joint Q function through sampling, reminds me of FACMAC, where agents select their actions in a similar way (albeit for continuous action spaces though).\n\nDespite the nice structure of the experimental presentation, the SMAC results are not particularly overwhelming because SMAC is a widely solved benchmark, where most baselines already achieve high win rates. Furthermore, the results are not in line with previous work, e.g.\n\n- MAVEN reaches an average win rate of 40% at 5 million steps in 6h_vs_8z in the original paper, while it remains with 0% win rate in this paper with the same about of steps.\n- QMIX reaches 100% win rate in 10m_vs_11m and 80% in 3s_vs_5z in the SMAC paper, while it is notably lower in this paper.\n\nThe paper heavily depends on the appendix indicating its lack of self-containment. The paper's content or the approach need more simplification to give a sense of completeness of the presentation.\n\nFigure labels and numbers are only readable on a sufficient zoom scale which is problematic when printed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and interesting comments! We are glad that you have read our paper carefully. We hope we can address your concerns below.\n\n**Q1**: Wouldn’t simply feeding the resulting joint action (e.g., as one-hot vector) be sufficient to compute the TD-target for Eq. 4?\n\n**A1**: We think this is a valuable question. We do not employ a monolithic critic whose input is the global state and the joint actions of all agents because our method can scale better to tasks with a larger number of agents and actions. For example, as shown in Figure 16(c), the *Humanoid* task involves 17 agents, each with 31 actions. Then the dimension of the sparse one-hot vector corresponding to the joint action is $17\\times31=527$. However, if each agent has an alter ego model to output an individual Q-function as in DAVE, it can avoid generating the sparse high-dimensional one-hot vector mentioned above. In addition, we also tested the performance of DAVE when introducing sparse one-hot vectors corresponding to joint actions in complex environments. The performance did not change significantly but the algorithm lost scalability. So the value decomposition mechanism in DAVE is not a detour. We apologize for not explicitly mentioning this point in the main text and will incorporate it in the next version of our paper.\n\n**Q2**: Despite doubled parameters, DAVE does neither perform twice as well nor learn twice as fast to fully compensate for the additional effort.\n\n**A2**: We give the relative size of neural network models for some value decomposition variants over QMIX. The model size here refers to the number of parameters each algorithm needs to train in the *MMM2* scenario.\n\n| Algo          | QMIX | DAVE | CW-QMIX | OW-QMIX | MAVEN | QPLEX |\n| ------------- | :--: | :--: | :-----: | :-----: | :---: | :---: |\n| Relative Size | 100% | 132% |  373%   |  373%   | 167%  | 142%  |\n\nThe number of parameters in the value decomposition framework mainly depends on the mixing network structure. The DAVE framework only has three more fully connected layers (an additional ego model) than QMIX, so DAVE has the smallest change compared to QMIX among the variants. Nevertheless, DAVE still performs better than other baselines related to IGM. The most important thing is that DAVE can quickly converge to the optimal solution in the matrix game, while QMIX cannot.\n\n**Q3**: The SMAC results are not particularly overwhelming because SMAC is a widely solved benchmark, where most baselines already achieve high win rates.\n\n**A3**: Our concern is that our IGM-free framework can achieve similar or better performance than IGM-based methods, especially in non-monotonic problems. In addition, we also tested the performance of DAVE and some baselines in other more challenging environments, including SMACv2 and MA-MuJoCo, which are shown in the appendix.\n\n**Q4**: The results are not in line with previous work.\n\n**A4**: We think there are two reasons. The first and most important point is that the version of StarCraft we use is 4.10 instead of 4.6.2 used in [1,2]. Performance is *not* always comparable between versions. Secondly, there are some small typos in the original paper of MAVEN. The X-axis in Figure 4(b) in [2] is strange, the \"0m\" scale is not at the origin and there are two \"4m\" scales. Also we point out some other typos about MAVEN in Appendix D.3. All experiments in this paper are carried out with 6 random seeds in order to avoid the effect of any outliers.\n\n**Q5**: The agents in FACMAC select their actions in a similar way as DAVE (albeit for continuous action spaces though).\n\n**A5**: The motivation of DAVE is completely different from that of FACMAC. FACMAC is IGM-based and focuses on continuous action space tasks, while DAVE is IGM-free and focuses on non-monotonic tasks. Although FACMAC proposes its IGM-free variant FACMAC-nonmonotonic, it performs worse than IGM-based methods in most of all scenarios, even in simple matrix games.\n\nWe will try to simplify the paper's content while keeping the experiments sufficient and change the minimum text size in figures to 9pt. We really appreciate your comments and they really help us improve our paper. And we also appreciate it if you have any further comments or improve your score.\n\n**Reference**\n\n[1] Samvelyan, Mikayel et al. The StarCraft Multi-Agent Challenge. 2019.\n\n[2] Mahajan, Anuj et al. MAVEN: Multi-Agent Variational Exploration. 2019."}]

Author rebuttal:
Dear Reviewers,

Thank you for your time and effort in reviewing our manuscript. We are delighted to receive your comments and suggestions, which have been valuable in improving the quality of our research.

As part of the rebuttal process, we have submitted a PDF as an additional auxiliary material, containing supplementary information that we believe is important for your review. We would like to remind you to please take a moment to review the attached file for further details. The PDF contains the following content:

1. The learning curves of vanilla DAVE and DAVE with a monotonic mixing network on the matrix games and SMAC.

2. Performance over time on the multi-step matrix game. As suggested by Reviewer NJQv, we change the aggregation function from *median* to *mean* in Figure 5.

We have also addressed each comment and provided responses inline with the reviewers' comments. We believe that these responses fully address the concerns raised by each reviewer and hope that you find them satisfactory.

We look forward to hearing your feedback and appreciate the time you have taken to review our manuscript. Thank you for your consideration.

Meta-review: The paper introduces DAVE, a novel value decomposition framework for multi-agent reinforcement learning inspired by the dual self-awareness concept in psychology. DAVE does not use the Individual Global Max (IGM) assumption, offering a fresh approach to addressing issues linked to the IGM assumption in value decomposition. 

The authors have addressed the reviewers' main concerns. The mitigation of the IGM assumption/monotonicity is an important challenge for the MARL community. While the lower scoring reviewers did not change their scores, they did not respond to the authors' as well as further attempts to understand if the authors have addressed their concerns. I find most of the reviewers' concerns to be simple to address. I therefore recommend acceptance of the paper, given that they will address the reviewers' comments in the final version of their paper.
Decision: Accept (poster)