Title: Dual Self-Awareness Value Decomposition Framework without Individual Global Max for Cooperative MARL
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multi-Agent Reinforcement Learning

Individual Global Max
Abstract: Value decomposition methods have gained popularity in the field of cooperative multi-agent reinforcement learning. However, almost all existing methods follow the principle of Individual Global Max (IGM) or its variants, which limits their problem-solving capabilities. To address this, we propose a dual self-awareness value decomposition framework, inspired by the notion of dual self-awareness in psychology, that entirely rejects the IGM premise. Each agent consists of an ego policy for action selection and an alter ego value function to solve the credit assignment problem. The value function factorization can ignore the IGM assumption by utilizing an explicit search procedure. On the basis of the above, we also suggest a novel anti-ego exploration mechanism to avoid the algorithm becoming stuck in a local optimum. As the first fully IGM-free value decomposition method, our proposed framework achieves desirable performance in various cooperative tasks.

Dual Self-Awareness Value Decomposition Framework without Individual Global Max for Cooperative MARL

Abstract

Value decomposition methods have gained popularity in the field of cooperative multi-agent reinforcement learning. However, almost all existing methods follow the principle of Individual Global Max (IGM) or its variants, which limits their problem-solving capabilities. To address this, we propose a dual self-awareness value decomposition framework, inspired by the notion of dual self-awareness in psychology, that entirely rejects the IGM premise. Each agent consists of an ego policy for action selection and an alter ego value function to solve the credit assignment problem. The value function factorization can ignore the IGM assumption by utilizing an explicit search procedure. On the basis of the above, we also suggest a novel anti-ego exploration mechanism to avoid the algorithm becoming stuck in a local optimum. As the first fully IGM-free value decomposition method, our proposed framework achieves desirable performance in various cooperative tasks.

1 Introduction

Multi-agent reinforcement learning (MARL) has recently drawn increasing attention. MARL algorithms have been successfully applied in diverse fields, including game AI [1] and energy networks [37], to solve practical problems. The fully cooperative multi-agent task is the most common scenario, where all agents must cooperate to achieve the same goal. Several cooperative MARL algorithms have been proposed, with impressive results on complex tasks. Some algorithms [7, 32, 17] enable agents to collaborate through communication. However, these approaches incur additional bandwidth overhead. Most communication-free MARL algorithms follow the conventional paradigm of centralized training with decentralized execution (CTDE) [20]. These works can be roughly categorized into multi-agent Actor-Critic algorithm [8, 16] and value decomposition method [34, 27, 19, 40, 39]. Due to the high sample efficiency in off-policy settings, value decomposition methods can outperform other MARL algorithms.

However, due to the nature of the CTDE paradigm, the value decomposition method can only operate under the assumption of Individual Global Max (IGM) [30]. The IGM principle links the optimal joint action to the optimal individual one, allowing the agent to find the action corresponding to the maximal Q value during decentralized execution. Nevertheless, this assumption is not applicable in most real-world scenarios. And most value decomposition approaches [27] directly constrain the weights of the neural network to be non-negative, which restricts the set of global state-action value functions that can be represented. We confirm this point through experiments in Appendix D.1.

Removing the IGM assumption requires agents to be able to find actions corresponding to the maximal joint Q value during execution. Therefore, in addition to the evaluation network, agents also need an additional network to learn how to search for the best actions in a decentralized manner. Similarly, there are concepts of ego and alter ego in psychology. The ego usually refers to the conscious part of the individual, and Freud considered the ego to be the executive of the personality [9]. Some people

believe that an alter ego pertains to a different version of oneself from the authentic self [24, 12]. Others define the alter ego in more detail as the evaluation of the self by others [3]. As a result, both ego and alter ego influence a person's personality and behaviour, as depicted in Figure 1. Besides, Duval and Wicklund [5] put forth the dual self-awareness hypothesis, which posits that individuals possess two states of consciousness: public selfawareness and private self-awareness. Private self-consciousness refers to an individual's in-

Ego

Individuality

Evaluate

Act

ternal understanding of himself, whereas public self-awareness enables individuals to recognize that they are evaluated by others. Enlightened by these psychological concepts, we propose a novel MARL algorithm, D ual selfA wareness V alue d E composition ( DAVE ). Each agent in DAVE maintains two neural network models: the alter ego model and the ego model. The former participates in the evaluation of the group to solve the global credit assignment problem, and the latter is not directly optimized by the joint state-action value. The ego policy can help the alter ego value function to find the individual action corresponding to the optimal joint action through explicit search procedures so that the value decomposition framework can completely eliminate the constraints of the IGM assumption. Furthermore, to prevent the ego policy based on the above framework from potentially getting stuck in a bad local optimum, we also propose an anti-ego exploration method. This method can effectively avoid the underestimation of the optimal action that may be caused by the limited search procedure.

We first evaluate the performance of DAVE in several didactic problems to demonstrate that DAVE can resolve the issues caused by IGM in conventional value decomposition methods. Then we run DAVE in various complex scenarios including StarCraft II, and compare it to other popular baselines. Our experimental results show that DAVE can still achieve competitive performance despite the absence of IGM. To the best of our knowledge, DAVE is the first multi-agent value decomposition method that completely abandons the IGM assumption.

2 Related Work

Recently, value decomposition methods have made significant progress in problems involving the decentralized partially observable Markov decision process (Dec-POMDP) [23]. By calculating a joint state-action value function based on the utility functions of all agents, value decomposition methods can effectively alleviate the notorious credit assignment issue. The first proposed method, VDN [34], simply sums up the utility of all agents to evaluate joint actions. This undoubtedly limits the range of joint value functions that can be approximated. QMIX [27] incorporates hypernetworks [11] into the value decomposition method and employs the mixing network to fit the joint value function based on the individual ones, thereby expressing richer classes of joint value function. To enable the agent to choose actions with the greedy policy during decentralized training, QMIX assumes monotonicity and restricts the weight parameters of the mixing network output by hypernetworks to non-negative. So the monotonic mixing network still cannot represent some class of value functions. Various value decomposition variants have subsequently been proposed. Officially proposing IGM, QTRAN [30] indicated that the value decomposition method must satisfy the assumption that the optimal joint actions across agents are equivalent to the collection of individual optimal actions of each agent. Unlike VDN and QMIX, QTRAN is not restricted by structural constraints such as non-negative weights, but it performs poorly in complex environments such as StarCraft II. MA VEN [21] enhances its performance by improving its exploration capabilities. Weighted QMIX [28] takes into account a weighted projection that places greater emphasis on better joint actions. QPLEX [36] takes a duplex dueling network architecture to factorize the joint action-value function and proposes the advantage-based IGM. Nevertheless, the aforementioned methods only loosen the restrictions by proposing IGM variants. More importantly, existing policy-based MARL methods [41, 20, 25] do not explicitly introduce IGM, but they also cannot solve some non-monotonic problems. So how to remove the IGM assumption is still an open and challenging problem. Our proposed DAVE is completely IGM-free and can be applied to existing IGM-based value decomposition methods. More related work will be discussed in Appendix E.

3 Preliminaries

3.1 Dec-POMDPs

In this paper, we concentrate on the Dec-POMDP problems. It is an important mathematical model for multi-agent collaborative sequential decision-making problem, in which agents act based on different pieces of information about the environment and maximize the global shared return. Dec-POMDPs can be formulated as a tuple ⟨A , S , U , Z , P, O, r, γ ⟩ . a ∈ A := { 1 , . . . , n } represents the agent in the task. s ∈ S is the global state of the environment. The joint action set U is the Cartesian product of action sets U a of all agents and u ∈ U denotes the joint action. Given the state s and the joint action u , the state transition function P : S × U × S → [0 , 1] returns the probability of the next potential states. All agents receive their observations z = { z 1 , ..., z n } obtained by the observation function O : S × A → Z . r : S × U → R denotes the global shared reward function and γ is the discount factor. Besides, recent work frequently allows agents to choose actions based on historical trajectory τ a rather than local observation z a in order to alleviate the partially observable problem.

In Dec-POMDP, all agents attempt to maximize the discount return R t = ∑ ∞ t =0 γ t r t and the joint state-action value function is defined as:

where π = { π 1 , . . . , π n } is the joint policy of all agents.

3.2 IGM-based Value Decomposition Methods

The goal of value decomposition method is to enable the application of value-based reinforcement learning methods in the CTDE paradigm. This approach decomposes the joint state-action value function Q tot into the utility function Q a of each agent. In VDN, the joint state-action value function is equal to the sum of all utility functions. And then QMIX maps utility functions to the joint value function using a monotonous mixing network and achieve better performance.

Nonetheless, additivity and monotonicity still impede the expressive capacity of the neural network model, which may result in the algorithm being unable to handle even for some simple matrix game problems. To this end, QTRAN proposes the Individual Global Max (IGM) principle, which can be represented as:

The IGM principle requires consistency between local optimal actions and global ones. Most value decomposition methods including VDN and QMIX must follow IGM and its variants. This is because only when the IGM principle holds, the individual action corresponding to the maximum individual action value during decentralized execution is a component of the optimal joint action. Otherwise, the agent cannot select an appropriate action based solely on the individual action-value function.

4 Methodology

In this section, we introduce the implementation of the dual self-awareness value decomposition (DAVE) framework in detail. DAVE incorporates an additional ego policy through supervised learning in comparison to the conventional value decomposition method, allowing the agent to identify the optimal individual action without relying on the IGM assumption. Moreover, DAVE can be applied to most IGM-based value decomposition methods and turn them into IGM-free ones.

4.1 Dual Self-Awareness Framework

Almost all value decomposition methods follow the IGM assumption, which means that the optimal joint policy is consistent with optimal individual policies. According to IGM, the agent only needs to follow the greedy policy during decentralized execution to cater to the optimal joint policy. Once the IGM assumption is violated, agents cannot select actions that maximize the joint state-action value based solely on local observations. Consequently, the objective of the IGM-free value decomposition method is as follows:

IGM-Free Mixing

Network

Agent 1

Agent n

…

MLP

GRU

MLP

Softmax

MLP

GRU

MLP

Softmin

…

…

Encoder

Decoder

Sample

argmax

where u = { u 1 , . . . , u n } and u a ∼ π a ( τ a ) . Without the IGM assumption, Equation (1) is NP-hard because it cannot be solved and verified in polynomial time [14]. Therefore, in our proposed dual self-awareness framework, each agent has an additional policy network to assist the value function network to find the action corresponding to the optimal joint policy. Inspired by the dual self-awareness theory in psychology, we named these two networks contained in each agent as ego policy model and alter ego value function model , respectively. Figure 2 depicts the dual self-awareness framework. Note that the IGM-free mixing network refers to the mixing network obtained by removing structural constraints. For instance, the IGM-free QMIX variant does not impose a non-negative constraint on the parameters of the mixing network.

In DAVE, the alter ego value function model of each agent a is identical to the agent network in other value decomposition methods and generates individual state-action value Q alter a ( τ a , · ) based on the current historical trajectory τ a . To obtain the joint state-action value Q alter tot ( s, u ) for carrying out the joint action u in the state s , the value function of each agent is fed into the IGM-free mixing network. However, the alter ego model does not determine the actions chosen by the agent, either during centralized training or decentralized execution. What determines the action of each agent is the ego policy π ego a . The joint ego policy can be expressed as π ego ( u | s ) = ∏ n a =1 π ego a ( u a | τ a ) . When executing, the agent interacts with the environment by sampling actions from the categorical distribution output by the ego policy. And during training, we replace the exact maximization of Q tot ( s, u ) over the total joint action space with maximization over a set of samples U ego from π ego . So we modify the objective of the cooperative multi-agent value decomposition problem from Equation (1) to

where U ego := { u ego i ∼ π ego ( s ) } i M =1 in state s and M is the number of samples. Therefore, the dual self-awareness framework substitutes the global optimal state-action value with the local optimal one found in the small space U ego , which is obtained by sampling M times from π ego . As long as the ego policy assigns non-zero probabilities to all available actions, this method will converge to the objective described by Equation (1) as the number of samples M increases. The detailed proof can be found in Appendix A. Nevertheless, the computational cost is directly proportional to M . So the choice of M must strike a balance between precision and computational complexity.

Next, we introduce the update process of the alter ego value function and the ego policy in the dual self-awareness framework. According to the modified objective of the cooperative multi-agent value decomposition problem illustrated by Equation (2), the ego policy needs to be able to predict the individual action that corresponds to the maximal joint state-action value. Therefore, we update the ego policy through supervised learning to increase the probability of the optimal joint action obtained through the sampling procedure. We define u ⋆ = arg max u ego Q alter tot ( s, u ego ) as the joint action corresponding to the maximal joint state-action value in the samples, where u ego ∈ U ego . The loss function for the joint ego policy can be written as:

The objective function is the negative log-likelihood of the probability of the joint action u ⋆ and can be further expressed as -∑ n a =1 log π ego a ( u a⋆ | τ a ) associated with individual ego policies.

Additionally, the alter ego value function Q alter is constantly changing during the training process. To prevent the ego policy π ego from becoming too deterministic and getting trapped in local optimal solutions, we will introduce a novel exploration method in the next subsection.

Conventional value decomposition methods can be trained end-to-end by minimizing the following loss:

where y tot = r t + γ max u t +1 Q -tot ( s t +1 , u t +1 ) and Q -tot is the target joint value function. The loss function described above cannot be directly applied to our proposed dual self-awareness framework because of the max operator. If we also traverse the samples of the joint action in U ego to select the local maximum target joint state-action value function, which may result in high variance. Inspired by the Expected SARSA [35], we use the expected value instead of the maximal one. The update rule of Q alter tot considers the likelihood of each action under the current policy and can eliminate the variance, which can be expressed as:

where y = r t + γ E π ego [( Q alter tot ) -( s t +1 , u t +1 )] . However, this method is still computationally complex. So we reuse the action sample set U ego and replace E π ego [( Q alter tot ) -( s t +1 , u t +1 )] with 1 M ∑ M i =1 ( Q alter tot ) -( s t +1 , u ego i ) , where u ego i ∈ U ego in state s t +1 . The modified update rule for Q alter tot reduces the variance and does not increase the order of the computational complexity compared to the original update method shown in Equation (4). Each hypernetwork takes the global state and the actions of all agents as input and generates the weights of the mixing network. It should also be noted that we use the global state and the actions of all agents as input to the hypernetwork that generates weights to enhance the expressiveness of the mixing network.

In summary, we have constructed the fundamental framework of DAVE. Although DAVE consists of a value network and a policy network, it is DISTINCT from the Actor-Critic framework or other policy-based methods such as MAPPO [41], MADDPG [20] and FACMAC [25]. DAVE requires an explicit sampling procedure to update the network model. Besides, the actions sampled from the ego policies remain independent and identically distributed (IID) [26], so the ego policy is trained through supervised learning rather than policy gradient used in policy-based methods. Most intuitively, conventional policy-based methods cannot solve simple non-monotonic matrix games but DAVE can easily find the optimal solution, as described in Section 5.1 and Appendix D.2.

4.2 Anti-Ego Exploration

As stated previously, updating the ego policy network solely based on Equation (3) with an insufficient sample size M may lead to a local optimum. This is because the state-action value corresponding to the optimal joint action may be grossly underestimated during initialization, making it challenging for the ego policy to choose the optimal joint action. Consequently, the underestimated Q alter tot will have no opportunity to be corrected, creating a vicious circle. Therefore, we developed an anti-ego exploration method based on ego policies. Its core idea is to assess the occurrence frequency of state-action pairs using the reconstruction model based on the auto-encoder [13]. The auto-encoder model often requires the encoder to map the input data to a low-dimensional space, and then reconstructs the original input data based on the obtained low-dimensional representation using the decoder. Our method leverages the fact that the auto-encoder cannot effectively encode novel data and accurately reconstruct it. The more significant the difference between the original state-action pair ( s, u ) and the reconstructed output ( s ′ , u ′ ) , the less frequently the agents carry out the joint action u in state s . Simultaneously, the difference between the reconstructed state-action pairs and the original ones is also the objective function for updating the auto-encoder, which can be expressed as:

where MSE( · ) and CE( · ) represent the mean square error and the cross-entropy loss function respectively. So we need to encourage agents to choose joint actions with a higher L recon. Obviously, it is impractical to traverse all available joint actions u . Then we look for the most novel actions within the limited set of actions obtained via the sampling procedure. In order to get state-action pairs that are as uncommon as possible, we generate an anti-ego policy ˆ π ego based on π ego , as shown

in Figure 2. The relationship between the ego and anti-ego policy of each agent is as follows:

where f ( · ) denotes the neural network model of ego policy. Equation (7) demonstrates that actions with lower probabilities in π ego a have higher probabilities in ˆ π ego a . Of course, actions with low probabilities in π ego a may be non-optimal actions that have been fully explored, so we filter out such actions through the reconstruction model introduced above. ˆ U ego := { ˆ u ego i ∼ ˆ π ego ( s ) } i M =1 is the joint action set obtained by sampling M times from the anti-ego policy in state s . We can identify the most novel joint action ˆ u ⋆ in state s by the following equation:

where ˆ u ego ∈ ˆ U ego . Therefore, the objective function in Equation (3) needs to be modified to increase the probability of the joint action ˆ u ⋆ . The final objective function of the ego policy π ego can be written as:

where the exploration coefficient λ ∈ [0 , 1] is annealed linearly during the exploration phase. The choice of hyperparameter λ requires a compromise between exploration and exploitation. Note that the dual self-awareness framework and the anti ego exploration method are NOT two separate contributions. The latter is customized for the former, and it is only used to solve the convergence difficulties caused by the explicit sampling procedure in DAVE. By incorporating the anti-ego exploration module, the dual self-awareness framework with low M can also fully explore the action space and significantly avoid becoming stuck in bad local optima. The whole workflow of the anti-ego exploration mechanism is shown in Appendix B.

Our proposed DAVE can be applied to most existing value decomposition methods, as it simply adds an extra ego policy network to the conventional value decomposition framework. We believe that the IGM-free DAVE framework can utilize richer function classes to approximate joint state-action value functions and thus be able to solve the non-monotonic multi-agent cooperation problems.

5 Experiment

In this section, we apply the DAVE framework to standard value decomposition methods and evaluate its performance on both didactic problems and complex Dec-POMDPs. First, we will compare the performance of the DAVE variant with QPLEX, Weighted QMIX, QTRAN, FACMACnonmonotonic [25], MAVEN, and the vanilla QMIX. Note that most baselines are directly related to IGM. Then the contribution of different modules in the framework and the impact of hyperparameters will be investigated. Except for the new hyperparameters introduced in the DAVE variant, all other hyperparameters are consistent with the original method. Unless otherwise stated, DA VE refers to the DAVE variant of QMIX, which enhances representational capacity while ensuring fast convergence. For the baseline methods, we obtain their implementation from the released code. The details of the implementation and all environments are provided in Appendix C and Appendix D, respectively.

5.1 Single-State Matrix Game

Certain types of tasks in multi-agent cooperation problems cannot be handled under the IGM assumptions, such as the two matrix games we will introduce next. These two matrix games capture very simple cooperative multi-agent tasks and are shown in Figure 3. These payoff matrices of the two-player threeaction matrix games describe scenarios that IGMbased value decomposition methods cannot address effectively. The variable k is related to difficulty. In

A

B

C

A

8

-12

-12

B

-12

0

C

-12

0

(a) Matrix game I.

A

B

C

A

0

10

B

0

2

0

C

8

0

(b) Matrix game II.

matrix game I with k ∈ [0 , 8) , agents need to choose the same action to get the reward and the optimal joint action is ( A,A ) . Furthermore, the closer k is to 8, the more challenging it becomes to jump out of the local optimum. Similarly, k ≥ 0 and ( A,C ) is the optimal joint action in matrix

0

100

200

300

400

Episode

-12

0

8

Median Test Return

Matrix Game I with k=0

0

100

200

300

400

Episode

-12

0

6

8

Median Test Return

Matrix Game I with k=6

0

100

200

300

400

Episode

-12

0

7.5

8

Median Test Return

Matrix Game I with k=7.5

0

100

200

300

400

Episode

0

2

8

10

Median Test Return

Matrix Game II with k=0

0

100

200

300

400

Episode

-25

0

2

8

10

Median Test Return

Matrix Game II with k=25

0

100

200

300

400

Episode

-100

0

2

8

10

Median Test Return

Matrix Game II with k=100

DAVE

QPLEX

OW-QMIX

CW-QMIX

QTRAN

FACMAC (Non-Mono)

MAVEN

game II. Nevertheless, as k increases, so does the risk of selecting the optimal action ( A,C ) , which leads agents in the IGM-based method to prefer the suboptimal joint action ( B,B ) corresponding to the maximum expected return. Therefore, both matrix games are challenging for traditional value factorization methods. To disregard the influence of exploration capabilities, we implement various algorithms under uniform visitation, which means that all state-action pairs are explored equally. We also evaluate the performance of popular policy-based MARL algorithms. Results in Appendix D.2 demonstrate that although they do not explicitly introduce IGM, policy-based methods still perform poorly on these two simple tasks.

The results of all algorithms on the two matrix games with varying k are presented in Figure 4. we limit the number of interactions between agents and the environment to 400 across all environments. All baselines find it challenging to discover the optimal joint action with such limited data. Compared to other algorithms, DA VE can quickly converge to the optimal solution in all situations and exhibits robust performance, showing little variation as task difficulty increases. CW-QMIX can converge to the optimal joint solution in all cases except matrix game I with k = 7 . 5 . QPLEX performs better in matrix game II than in matrix game I, likely due to its sensitivity to the penalty of non-cooperation. In addition, as the task difficulty increases, both CW-QMIX and QPLEX show a significant drop in performance. Although QTRAN can converge to the optimal solution in matrix game I with k = 0 , it may require more interactive samples for further learning. The remaining algorithms can converge to the optimal joint action ( A,C ) or the suboptimal joint action ( C, A ) in matrix game II with k = 0 , but in the more difficult matrix game II can only converge to the joint action ( B,B ) . We add additional action information to the mixing network of QMIX, similar to that in DAVE, but QMIX still cannot converge to the optimal solution. In summary, DAVE can easily solve the above single-state matrix games and outperforms other IGM-based variants and the basic algorithm QMIX by a significant margin.

5.2 Multi-Step Matrix Game

Next, we evaluate the performance of each algorithm in the multi-step matrix game. The environment involves two agents, each with two actions. The initial state is nonmonotonic, and agents terminate immediately after receiving a zero reward. If the game is not terminated after the first step, agents need to continue to repeat the last joint action to obtain a non-zero reward. According to Figure 5, agents repeat either the upper-left joint action or the lower-right joint action eight times, resulting in two distinct terminal states. The optimal policy is

Initial State

Terminal

Terminal

8 Times

8 Times

1

0

0

1

1

1

1

1

0

0

0

1

1

0

0

0

1

1

1

4

0

20

40

60

80

100

0

2

4

6

8

10

12

14

Mean Test Return

Multi-Step Matrix Game

DAVE

DAVE w/o Exploration

QPLEX

FACMAC (Non-Mono)

OW-QMIX

CW-QMIX

QTRAN

MAVEN

Step (K)

for agents to repeat the upper-left joint action in the initial state and choose the lower-right joint action in the terminal state, which yields the highest return of 13. Local observations of agents in this

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

2s_vs_1sc

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

3s5z

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

2c_vs_64zg

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

5m_vs_6m

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

3s_vs_5z

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

8m_vs_9m

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

10m_vs_11m

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

MMM2

0.0

1.0

2.0

3.0

4.0

5.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

6h_vs_8z

DAVE

QPLEX

OW-QMIX

CW-QMIX

MAVEN

QTRAN

FACMAC (Non-Mono)

IGM-Free QMIX

environment include the current step count and the last joint action. The protracted decision-making process requires agents to fully explore the state-action space to find the optimal policy. Therefore, this task can also test the exploration capacity of the algorithm.

Figure 5 shows the performance of DAVE and other baselines on the multi-step matrix game. Only DAVE successfully learned the optimal policy, while most other algorithms were trapped in the suboptimal policy with payoff 10. QPLEX can occasionally converge to the optimal policy, but in most cases it still cannot learn the optimal solution. Besides, to examine the contribution of the anti-ego exploration to the DAVE framework, we implemented DAVE without exploration in this environment. The experimental results show that DAVE without anti-ego exploration still performs better than other baselines, but is significantly inferior to DA VE. Therefore, for some complex tasks, the anti-ego exploration mechanism can assist the agent in exploring novel state-action pairs and avoiding being restricted to suboptimal solutions.

5.3 StarCraft II

SMAC [29] is the most popular experimental platform for multi-agent micromanagement, based on the famous real-time strategy game StarCraft II. SMAC offers a diverse range of mini-scenarios, including homogeneous, heterogeneous, symmetric, and asymmetric problems. The goal of each task is to direct agents to defeat enemy units controlled by built-in heuristic AI. Agents can only access information within their own visual range and share a reward function. So we assess the ability of DAVE to solve complex multi-agent cooperation problems in this environment.

The version of StarCraft II used in this paper is 4.10 and performance is not comparable between versions. We test the proposed method in different scenarios. In addition to the above-mentioned methods, baselines also include IGM-Free QMIX that simply does not take the absolute value of the weight of the mixing network and keeps others unchanged. Both IGM-Free QMIX and FACMACnonmonotonic are IGM-free methods, but do not have convergence guarantees. The performance comparison between above two IGM-free methods and DAVE that is also IGM-free can reflect the necessity of our proposed ego policy network. In addition, to balance exploration and exploitation, we disable the anti-ego exploration mechanism in some easy scenarios and use it only in hard scenarios.

The experimental results are shown in Figure 6. Undoubtedly, IGM-Free QMIX and FACMACnonmonotonic fail to solve almost all tasks. Without the IGM assumption, QMIX is unable to find the correct update direction. Therefore, the assistance of the ego policy network in the DA VE framework is essential for the convergence of IGM-free algorithms. QPLEX performs best in some easy scenarios but fails to solve the task in most hard scenarios. Weighted QMIX performs well in the MMM2 scenario but has mediocre performance in most other scenarios. Although DAVE is not the best in easy maps, it still shows competitive performance. In hard scenarios, such as 5m_vs_6m , 8m_vs_9m and 10m_vs_11m , DAVE performs significantly better than other algorithms. We believe

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

2s_vs_1sc

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

5m_vs_6m

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

MMM2

M=1

M=5

M=10

M=50

M=100

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

2s_vs_1sc

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

5m_vs_6m

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

MMM2

λ_init=0.0

λ_init=0.5

λ_init=1.0

that it may be that the characteristics in these scenarios are far from the IGM assumptions, resulting in low sample efficiency of conventional value decomposition methods. Moreover, because DAVE is IGM-free, it can guide the agent to learn the excellent policy quickly without restriction. Especially in the 6h_vs_8z scenario, only DAVE got a non-zero median test win ratio. In summary, even if the IGM principle is completely abandoned, DAVE can still successfully solve complex tasks and even outperforms existing IGM-based value decomposition algorithms in certain challenging scenarios. MORE experimental results on other complex environments such as SMACv2 [6] and Multi-Agent MuJoCo [25] can be found in Appendix D. These results demonstrate the excellent generalization of DAVE to different algorithms and environments.

5.4 Ablation Study

Finally, we evaluate the performance of DAVE under different hyperparameter settings in SMAC scenarios. We focus on the number of samples M and the initial value of the exploration coefficient λ init. According to Appendix A, increasing the number of samples M improves the probability of drawing the optimal joint action. Nonetheless, due to time and space complexity constraints, it is crucial to find an appropriate value for the hyperparameter M . Then we test DAVE with different M values in three scenarios and present results in Figure 7. The results indicate that a larger M leads to a faster learning speed for DA VE, which aligns with our intuition and proof. DA VE with M = 1 fails in all scenarios. When M increases to 10, the performance of DAVE is already considerable. However, as M increases further, the performance enhancement it provides diminishes.

The equilibrium between exploration and exploitation is directly related to the initial value of the exploration coefficient λ init. The final value λ end and the annealing period also affect the exploration, but only λ init is discussed in this paper. λ end is generally set to 0, so DA VE is only affected by anti-ego exploration in the early stages of training. Figure 8 shows the learning curve of DA VE under different λ init . λ init = 0 means that DAVE does not perform anti-ego exploration. The addition of anti-ego exploration in easy scenarios will prevent the algorithm from exploiting better in the early stages, which will significantly slow down the training speed. Conversely, DAVE with high λ init values can achieve superior performance in hard scenarios. So a moderately high λ init is beneficial for DA VE to visit key state-action pairs and converge to the optimal solution. In summary, appropriate values of M and λ init are indispensable for DAVE.

It would be valuable to explore the performance of DAVE with a monotonic mixing network. As shown in Figure 9 and Figure 10, DAVE with a monotonic mixing network still cannot converge to the global optimal solution in matrix games. Since the ego policy is updated by an explicit search procedure, DAVE with a monotonic mixing network has more chances to converge to the optimal solution than vanilla QMIX. However, limited by fewer function families that can be represented, the performance of DAVE with a monotonic mixed network is much worse than that of DAVE and is unstable. In addition, in the complex SMAC environment, the performance of DAVE with a monotonic mixing network is similar to or even better than that of QMIX, but worse than that of

0

100

200

300

400

Episode

-12

0

8

Median Test Return

Matrix Game I with k=0

0

100

200

300

400

Episode

-12

0

6

8

Median Test Return

Matrix Game I with k=6

0

100

200

300

400

Episode

-12

0

7.5

8

Median Test Return

Matrix Game I with k=7.5

0

100

200

300

400

Episode

0

2

8

10

Median Test Return

Matrix Game II with k=0

0

100

200

300

400

Episode

-25

0

2

8

10

Median Test Return

Matrix Game II with k=25

0

100

200

300

400

Episode

-100

0

2

8

10

Median Test Return

Matrix Game II with k=100

DAVE

DAVE-Mono

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

5m_vs_6m

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

8m_vs_9m

0.0

0.4

0.8

1.2

1.6

2.0

T (mil)

0

20

40

60

80

100

Median Test Win (%)

10m_vs_11m

DAVE

DAVE-Mono

vanilla DAVE. This is also substantial proof that the unconstrained factored mixing network indeed enhances performance.

6 Conclusion and Discussion

Inspired by the concept of dual self-awareness in psychology, we propose the IGM-free value decomposition framework DAVE in this paper. By dividing the agent network into two parts, ego policy and alter ego value function, DAVE is the first value decomposition method that completely abandons the IGM assumption. We find that the larger the number of samples in the explicit search procedure, the better the algorithm converges to the global optimum, but this will lead to high computational complexity. Therefore, we propose the anti-ego exploration mechanism, which can effectively prevent DAVE from falling into local optima, especially in problems with large action spaces. Experimental results in some didactic problems and complex environments show that DAVE outperforms existing IGM-based value decomposition methods. We believe that DAVE provides a new perspective on addressing the problems posed by the IGM assumption in value decomposition methods. In our future research, we intend to concentrate on how to choose the exploration coefficients in an adaptive manner. Further study of the issue would be of interest.

Acknowledgments and Disclosure of Funding

This project was supported by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No. XDA27050100.

References

Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub W. Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. ArXiv , abs/1912.06680, 2019.

Wendelin Boehmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 980-991. PMLR, 2020.

Martin Buber and Walter Arnold Kaufmann. I and thou.

Maria Dimakopoulou and Benjamin Van Roy. Coordinated exploration in concurrent reinforcement learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 1270-1278. PMLR, 2018.

Shelley Duval and Robert A. Wicklund. A theory of objective self awareness. 1972.

Benjamin Ellis, S. Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Nicolaus Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. ArXiv , abs/2212.07489, 2022.

Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages 2137-2145, 2016.

Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 , pages 2974-2982. AAAI Press, 2018.

Sigmund Freud. The standard edition of the complete psychological works of sigmund freud. 1953.

Tarun Gupta, Anuj Mahajan, Bei Peng, Wendelin Boehmer, and Shimon Whiteson. Uneven: Universal value exploration for multi-agent reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 3930-3941. PMLR, 2021.

David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017.

John Ryan Haule. Jung in the 21st century volume two: Synchronicity and science. 2010.

Geoffrey E. Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. Science , 313:504 - 507, 2006.

Dorit S. Hochbaum. Approximation algorithms for np-hard problems. 1996.

Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih wei Liao. Revisiting the monotonicity constraint in cooperative multi-agent reinforcement learning. 2021.

Shariq Iqbal and Fei Sha. Actor-attention-critic for multi-agent reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 2961-2970. PMLR, 2019.

Dapeng Li, Zhiwei Xu, Bin Zhang, and Guoliang Fan. From explicit communication to tacit cooperation: A novel paradigm for cooperative marl. arXiv preprint arXiv:2304.14656 , 2023.

Iou-Jen Liu, Unnat Jain, Raymond A. Yeh, and Alexander G. Schwing. Cooperative exploration for multi-agent deep reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 6826-6836. PMLR, 2021.

Zhihao Liu, Zhiwei Xu, and Guoliang Fan. Hierarchical multi-agent reinforcement learning with intrinsic reward rectification. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5. IEEE, 2023.

Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 6379-6390, 2017.

Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. MAVEN: multi-agent variational exploration. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 7611-7622, 2019.

Anuj Mahajan, Mikayel Samvelyan, Lei Mao, Viktor Makoviychuk, Animesh Garg, Jean Kossaifi, Shimon Whiteson, Yuke Zhu, and Animashree Anandkumar. Tesseract: Tensorised actors for multi-agent reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 7301-7312. PMLR, 2021.

F. Oliehoek and Chris Amato. A concise introduction to decentralized pomdps. In SpringerBriefs in Intelligent Systems , 2016.

David L. Pedersen. Cameral analysis: A method of treating the psychoneuroses using hypnosis. 1994.

Bei Peng, Tabish Rashid, Christian Schröder de Witt, Pierre-Alexandre Kamienny, Philip H. S. Torr, Wendelin Boehmer, and Shimon Whiteson. FACMAC: factored multi-agent centralised policy gradients. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 12208-12221, 2021.

Phillip E. Pfeifer. A brief primer on probability distributions. Economics Educator: Courses , 2008.

Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 4292-4301. PMLR, 2018.

Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted QMIX: expanding monotonic value function factorisation for deep multi-agent reinforcement learning. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Mikayel Samvelyan, Tabish Rashid, C. S. D. Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. ArXiv , abs/1902.04043, 2019.

Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Hostallero, and Yung Yi. QTRAN: learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 5887-5896. PMLR, 2019.

Kyunghwan Son, Sungsoo Ahn, Roben Delos Reyes, Jinwoo Shin, and Yung Yi. Qtran++: Improved value transformation for cooperative multi-agent reinforcement learning. arXiv: Learning , 2020.

Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages 2244-2252, 2016.

Wei-Fang Sun, Cheng-Kuang Lee, and Chun-Yi Lee. DFAC framework: Factorizing the value function via quantile mixture for multi-agent distributional q-learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 9945-9954. PMLR, 2021.

Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech M. Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Valuedecomposition networks for cooperative multi-agent learning. In Adaptive Agents and Multi-Agent Systems , 2017.

Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks , 16:285-286, 2005.

Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX: duplex dueling multiagent q-learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Jianhong Wang, Wangkun Xu, Yunjie Gu, Wenbin Song, and Tim C. Green. Multi-agent reinforcement learning for active voltage control on power distribution networks. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 3271-3284, 2021.

Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. Influence-based multi-agent exploration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

Zhiwei Xu, Yunpeng Bai, Dapeng Li, Bin Zhang, and Guoliang Fan. SIDE: state inference for partially observable cooperative multi-agent reinforcement learning. In Piotr Faliszewski, Viviana Mascardi, Catherine Pelachaud, and Matthew E. Taylor, editors, 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, New Zealand, May 9-13, 2022 , pages 1400-1408. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2022.

Zhiwei Xu, Yunpeng Bai, Bin Zhang, Dapeng Li, and Guoliang Fan. HAVEN: hierarchical cooperative multi-agent reinforcement learning with dual coordination mechanism. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, ThirtyFifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023 , pages 11735-11743. AAAI Press, 2023.

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems , 35:24611-24624, 2022.

Bin Zhang, Lijuan Li, Zhiwei Xu, Dapeng Li, and Guoliang Fan. Inducing stackelberg equilibrium through spatio-temporal sequential decision-making in multi-agent reinforcement learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China , pages 353-361. ijcai.org, 2023.

A Proof

Proposition A.1. As long as the ego policy assigns non-zero probabilities to all actions, this method will approach the objective described by Equation (1) as the number of samples M increases.

Proof. Let ( u a ) ∗ denote the individual actions corresponding to the global optimal joint state-action value function under state s , and the optimal joint action is u ∗ = { ( u 1 ) ∗ , . . . , ( u n ) ∗ } . Then the probability that the sampling procedure draws u ∗ is expressed as:

where π ego a ( ·|· ) ∈ (0 , 1) is true for any action. So the second term (1 -π ego ( u ∗ | s )) M ∈ (0 , 1) in the equation decreases as M increases, indicating that p ( u ∗ ) is positively correlated with the sample size M .

B Details of the Ego Policy Update

Policy

Net

Softmax

Softmin

Logits

Anti-Ego

Policy

Ego

Policy

Encoder

Decoder

Joint Action

Space

Joint Action

Sample

Sample

Select

Select

Maximize

&

Low

High

Low

High

Explored & High

Unexplored & Low

(Maybe Underestimated)

Explored & Low

Joint Actions

Sort

Anti-Ego Exploration

Supervised Learning

The update process of ego policy is shown in Figure 11. During the initialization of neural networks, the state-action value corresponding to the optimal joint action may be underestimated, making it difficult for the ego policy to choose and then correct it, especially in problems with large action spaces. So we use the Softmin operator to generate an anti-ego policy so that low-probability actions are more likely to be selected. However, not all low-probability joint actions are unexplored and undervalued; some are sub-optimal actions that have been exhaustively explored (orange dots in Figure 11). In order to select only unexplored joint actions, we distinguish the above two through the auto-encoder. Figure 5 and Figure 8 demonstrate the contribution of the anti-ego exploration mechanism.

C Implementation Details

C.1 Algorithmic Description

The pseudo-code of DAVE is shown in Algorithm 1.

C.2 Hyperparameters

Unless otherwise stated, the hyperparameter settings in different environments are shown in Table 1, which are exactly the same as the settings in PyMARL 1 . All experiments in this paper are run on Nvidia GeForce RTX 3090 graphics cards and Intel(R) Xeon(R) Platinum 8280 CPU. For baselines, exploration is performed during training using independent ϵ -greedy action selection. ϵ is annealing linearly from 1.0 to 0.05 and the annealing period of ϵ is 50000 steps. All experiments in this paper are carried out with 6 random seeds in order to avoid the effect of any outliers.

D Additional Experiments and Details

D.1 Comparison between IGM-Based and IGM-Free Mixing Networks

To verify that IGM-based mixing network restricts the set of global state-action value functions that can be represented, we synthesize an artificial dataset to represent the reward functions of two agents in a single state matrix game. Each agent has 101 actions, which means u i ∈ { 0 , 1 , . . . , 100 } . In order to only measure the influence of whether the mixing network follows the IGM assumption, we fix the individual action-value function as follows:

The reward value R for a joint action is given as follows:

The ground-truth joint value function is shown in Figure 12. Next we calculate the joint action value Q tot for the selected u based on the given Q i and R . Most value decomposition approaches, such as QMIX and QPLEX, directly constrain the weights of the neural network to be non-negative, so we use the mixing network in QMIX as the IGM-based model to be trained. Meanwhile, the variant that removes the absolute function is viewed as the IGM-free model. It should also be noted that we use the global state and the actions of all agents as input to the hypernetwork that generates weights to enhance the expressiveness of the mixing network. Figure 12 represents the loss curve and Figure 13 illustrates the joint action-value function estimated by the two models.

0

20

40

60

80

100

0

20

40

60

80

100











































0

20

40

60

80

100

0

20

40

60

80

100

(a) 200 step

0

20

40

60

80

100

0

20

40

60

80

100

(b) 400 step

0

20

40

60

80

100

0

20

40

60

80

100

(c) 600 step

0

20

40

60

80

100

0

20

40

60

80

100

(d) 800 step

0

20

40

60

80

100

0

20

40

60

80

100

(e) 1000 step

0

20

40

60

80

100

0

20

40

60

80

100

(f) 200 step

0

20

40

60

80

100

0

20

40

60

80

100

(g) 400 step

0

20

40

60

80

100

0

20

40

60

80

100

(h) 600 step

0

20

40

60

80

100

0

20

40

60

80

100

(i) 800 step

0

20

40

60

80

100

0

20

40

60

80

100

(j) 1000 step

From the results, it can be concluded that when the joint value function Q tot is not a simple monotone function with respect to Q i , the representation ability of the IGM-based mixing network is obviously limited, and the convergence speed is also slower than IGM-free mixing network. However, it is not possible to simply replace the IGM-based mixing network in the existing work with an IGM-free one, because this will cause each agent to fail to find the individual action corresponding to the optimal joint action during decentralized execution. Of course, this is why we propose the DA VE framework.

D.2 Policy-Based MARL Methods on Single-State Matrix Games

DAVE is not a standard actor-critic framework in the strict sense. The ego policy in DA VE is updated in a supervised learning manner, which maximizes the probability of the joint action corresponding to the optimal joint action value in the sample. Thus DAVE can explicitly search for optimal joint actions and easily solve non-monotonic problems. FACMAC still has to rely on the monotonic mixing network to update the critic; otherwise, the performance will deteriorate.

Moreover, the existing policy-based method that does not use IGM for the critic cannot solve the non-monotonic task in our paper. We test two representative policy-based algorithms, MAPPO and MADDPG, in two didactic problems. We present the experimental results over 100 trials of 10000 episodes in Table 2 and Table 3, which demonstrate that although they do not explicitly introduce IGM, they still perform poorly.

We carry out MADDPG and MAPPO in a policy-based setting. The above two algorithms cannot perfectly solve the single-state matrix problems even when more samples can be accessed. As the difficulty of the problem increases, the proportion of MADDPG converging to the optimal solution gradually decreases. MAPPO converges steadily to a sub-optimal solution.

D.3 Additional Explanations for Results in Multi-Step Matrix Game

In MAVEN's official code, the implementation of the multi-step matrix problem does not match the description in its original paper, which is reflected in the terminal state corresponding to the bad branch (see Figure 14). This does not change the non-monotonicity of the initial state, but reduces the expected return for agents to choose the bad branch. In other words, the above mistakes will make it EASIER for agents to learn the optimal policy. We rectified this error and retested all baselines for their real performance. Results in Figure 5 shows that MAVEN performed worse than in its original paper.

Initial State

Terminal

Terminal

8 Times

8 Times

1

0

0

1

1

1

1

1

0

0

0

1

1

0

0

0

1

1

1

4

Initial State

Terminal

Terminal

8 Times

8 Times

1

0

0

1

0

0

0

1

0

0

0

1

1

0

0

0

1

1

1

4

D.4 Additional Introduction for SMAC Environment

The single-state matrix games and the multi-step matrix game are elaborated in the main body of the paper. Here we only elaborate on the SMAC environment. The action space of each agent is discrete. The version of StarCraft II used in this paper is the latest 4.10. For a fair comparison, all algorithms are run on the same version of StarCraft and SMAC. Table 4 shows the details of the scenarios used in this paper.

D.5 Additional Experiments on SMACv2

Although SMAC is a popular multi-agent benchmark in recent years, it suffers from a lack of stochasticity. To remedy this significant deficiency, a more challenging SMACv2 environment is proposed. In SMACv2, team composition is randomly generated, which means that unit types are not fixed. Meanwhile, the start positions of all agents are also random. Finally, the sight range and attack range of the units are changed to the values from SC2 instead of fixed values. These three major changes make the tasks in SMACv2 more challenging.

To assess the generalizability of our proposed algorithm, we evaluate the performance of DAVE in several representative scenarios of SMACv2. Some of the scenarios are symmetrical and others are asymmetrical. To ensure consistency with the original paper on SMACv2, we use the hyperparameter settings from PyMARL2 2 [15]. To demonstrate that the DA VE framework is applicable to any existing value decomposition method and can transform the original algorithm into an IGM-free version with guaranteed convergence, we compare the performance of two classic algorithms, QMIX and QPLEX, with their variants respectively. Specifically, IGM-Free variants are obtained by simply removing all absolute functions from the original methods. We conduct the experiments using StarCraft II version 4.10 and present the results in Figure 15 and Figure 16.

The experimental results demonstrate that DAVE-QMIX outperforms QMIX in most scenarios, particularly in the protoss_5_vs_5 and terran_5_vs_6 scenarios. Notably, both DAVE-QMIX and IGM-Free QMIX are IGM-free methods. However, while IGM-Free QMIX is unable to guarantee individual actions consistent with the optimal joint action, its performance is significantly worse than DAVE-QMIX and QMIX, and even fails in most scenarios. Similarly, DAVE-QPLEX outperforms both vanilla QPLEX and IGM-Free QPLEX, despite the weight of its transformation module no longer being restricted to non-negative values. In summary, DAVE variants perform significantly

0

1

2

3

4

5

T (mil)

0

20

40

60

80

Median Test Win (%)

protoss_5_vs_5

0

1

2

3

4

5

T (mil)

0

20

40

60

80

Median Test Win (%)

terran_5_vs_5

0

1

2

3

4

5

T (mil)

0

15

30

45

60

Median Test Win (%)

zerg_5_vs_5

0

1

2

3

4

5

T (mil)

0

5

10

15

20

Median Test Win (%)

protoss_5_vs_6

0

1

2

3

4

5

T (mil)

0

5

10

15

20

25

Median Test Win (%)

terran_5_vs_6

0

1

2

3

4

5

T (mil)

0

5

10

15

20

Median Test Win (%)

zerg_5_vs_6

IGM-Free QMIX

0

1

2

3

4

5

T (mil)

0

20

40

60

80

Median Test Win (%)

protoss_5_vs_5

0

1

2

3

4

5

T (mil)

0

20

40

60

80

Median Test Win (%)

terran_5_vs_5

0

1

2

3

4

5

T (mil)

0

15

30

45

60

Median Test Win (%)

zerg_5_vs_5

0

1

2

3

4

5

T (mil)

0

5

10

15

20

Median Test Win (%)

protoss_5_vs_6

0

1

2

3

4

5

T (mil)

0

10

20

30

Median Test Win (%)

terran_5_vs_6

0

1

2

3

4

5

T (mil)

0

5

10

15

20

25

Median Test Win (%)

zerg_5_vs_6

DAVE-QPLEX

QPLEX

IGM-Free QPLEX

better than original methods in scenarios that do not meet the IGM assumptions, and also exhibit competitive performance in other scenarios, regardless of the environment complexity.

In addition to evaluating different variants of the same algorithm, we compare the performance of different baselines designed to solve the IGM problem. Since these baselines are all based on QMIX, we show their learning curves and that of DAVE-QMIX in Figure 17. It is evident from the results that the complex mechanisms introduced by other algorithms cause them to perform worse than the vanilla QMIX and DAVE-QMIX. The experimental results on SMACv2 demonstrate the excellent generalization of DAVE to different algorithms and environments.

D.6 Additional Experiments on Multi-Agent MuJoCo

In order to test the performance of the same algorithm following IGM and not following IGM in a complex environment, we also carry out vanilla QMIX, IGM-Free QMIX and DAVE on MultiAgent MuJoCo (MA-MuJoCo). It is worth noting that the latter two do not require the IGM assumption at all, which means that they can fit richer function classes but it is more difficult to find the optimal joint action. Since most of the conventional value decomposition methods are only applicable to environments where the action space is discrete, we discretize the action space of the MA-MuJoCo environment reasonably. Each joint in the robot is considered as an agent. We use uniform quantization to discretize continuous actions and modify the action space of each agent to

K = 31 . In addition, MA-MuJoCo is a partially observable environment, and each agent (joint) can only observe the information of neighbor agents (joints). Figure 18 shows three classic scenarios. Numbers in brackets indicate the number of agents (joints) in each task.

It can be seen from Figure 19 that even without the IGM assumption, DAVE can still perform better than vanilla QMIX in most scenarios, especially in the Hopper and Humanoid scenarios. IGM-Free

0

1

2

3

4

5

T (mil)

0

20

40

60

80

Median Test Win (%)

protoss_5_vs_5

0

1

2

3

4

5

T (mil)

0

20

40

60

80

Median Test Win (%)

terran_5_vs_5

0

1

2

3

4

5

T (mil)

0

15

30

45

60

Median Test Win (%)

zerg_5_vs_5

0

1

2

3

4

5

T (mil)

0

5

10

15

20

Median Test Win (%)

protoss_5_vs_6

0

1

2

3

4

5

T (mil)

0

5

10

15

20

25

Median Test Win (%)

terran_5_vs_6

0

1

2

3

4

5

T (mil)

0

5

10

15

20

Median Test Win (%)

zerg_5_vs_6

OW-QMIX

CW-QMIX

QTRAN

(b) HalfCheetah (6)

QMIX fails in all tasks. In summary, in the case of the same loss function related to reinforcement learning, QMIX with the dual self-awareness framework performs better than vanilla QMIX in all cases. Even in scenarios with large action spaces like Humanoid , which contains 31 17 ≈ 2 . 26 × 10 25 joint actions, DAVE still outperforms QMIX. Like the results in the main text, the experimental results in MA-MuJoCo further prove that DAVE is not only suitable for didactic examples, but also capable of complex scenarios.

0

1

2

3

4

5

6

T (mil)

0

10

20

30

Hopper

0

1

2

3

4

5

6

T (mil)

-10

0

10

20

30

HalfCheetah

0

1

2

3

4

5

6

T (mil)

0

5

10

15

Humanoid

DAVE

IGM-Free QMIX

D.7 Additional Notes on Model Size

We give the relative size of neural network models for some value decomposition variants over QMIX in Table 5. The model size here refers to the number of parameters each algorithm needs to train in the MMM2 scenario.

The number of parameters in the value decomposition framework mainly depends on the mixing network structure. The DAVE framework only has three more fully connected layers (an additional ego model) than QMIX, so DAVE has the smallest change compared to QMIX among the variants. Nevertheless, DAVE still performs better than other baselines related to IGM. The most important thing is that DA VE can quickly converge to the optimal solution in the matrix game, while QMIX cannot.

E More Related Work

By improving the joint exploration of all agents, UneVEn [10] addresses the issue that stems from the monotonicity constraint in representing the joint-action value function. However, in some complex environments such as SMAC, its additional complex mechanism leads to significantly slower convergence. TESSERACT [22] views the Q-function as a tensor and utilizes low-rank tensor approximations to model agent interactions relevant to the task. DCG [2] factors the joint value function of all agents into payoffs between pairs of agents based on a coordination graph, but it requires additional domain knowledge and violates CTDE principles. QTRAN++ [31] is an improved version of QTRAN, however it still does not completely remove absolute functions. STEP [42] can solve non-monotonic problems but it focuses on how to find Stackelberg equilibrium via asynchronous action coordination. Besides, some value decomposition methods [33] from the distributional perspective replace the original IGM principle with the distribution-style IGM to deal with highly stochastic environments. The DAVE framework we proposed does not change the loss function of the underlying value decomposition algorithm, and only requires simple modifications to make it IGM-free. The DAVE-based value decomposition method can outperform the original algorithm in scenarios where the IGM assumption is not satisfied, and it can also have competitive performance in other scenarios.

There has been a lot of excellent work on exploration in the field of multi-agent reinforcement learning. UneVEn uses novel action-selection schemes during exploration to overcome relative overgeneralization. EITI/EDTI [38] measure the amount and value of one agent's influence on the other agents' exploration processes to improve exploration. Dimakopoulou and Roy [4] present several seed sampling schemes for efficient coordinated exploration in concurrent reinforcement learning. CMAE [18] uses target policies to maximize returns and exploration policies to collect unexplored data, which is similar to our work. However, CMAE focuses on solving the sparse-reward task, whereas anti-ego exploration in DA VE alleviates the suboptimal convergence caused by limited search. In addition, the anti-ego exploration mechanism also leverages the characteristics of the auto-encoder to further narrow the state-action space that needs to be explored.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on value decomposition without IGM constraints by proposing Dual self-Awareness Value dEcomposition (DAVE) framework. DAVE is inspired by dual self-awareness studied in psychology and uses an ego model, i.e., a policy for each agent for actual action selection and an alter ego model, i.e., a value function to address credit assignment without the IGM constraint. To avoid premature convergence to poor local optima, an anti-ego exploration method is proposed. DAVE is evaluated in several domains and compared with many state-of-the-art MARL approaches.\n\nSTRENGTHS:\nThe paper proposes an interesting approach inspired by knowledge from psychology.\n\nThe evaluation is structured well by starting with small and tractable problems first and scaling up. Many state-of-the-art MARL algorithms are used as baseline for sufficient comparison.\n\nWEAKNESSES:\nThe paper strongly focuses on the omission of IGM, which is motivated as a limitation of MARL. However, DAVE introduces additional complexity that at least doubles the computational effort compared to the baselines, e.g., two networks are used per agents and the exploration mechanism adds more parameters on top of that (note that despite doubled parameters, DAVE does neither perform twice as well nor learn twice as fast to fully compensate for the additional effort). Therefore, the fairness of comparison is questionable. Furthermore, I am not sure about the purpose of the individual Q-functions, since they are not used to maximize the joint Q function as usual. The ego model/policy is used instead thus individual Q-functions could be completely omitted as far as I understood the text.\n\nThe initial motivation of dual self-awareness in the introduction implies that ego and alter ego models have a symmetric relationship. However, the ego model represents a policy while the alter ego model represents a value function, which is sometimes called \"alter ego model\" and sometimes \"alter ego value function model\", which is confusing to read.\n\nThe maximization of the joint Q function through sampling, reminds me of FACMAC, where agents select their actions in a similar way (albeit for continuous action spaces though).\n\nDespite the nice structure of the experimental presentation, the SMAC results are not particularly overwhelming because SMAC is a widely solved benchmark, where most baselines already achieve high win rates. Furthermore, the results are not in line with previous work, e.g.\n\n- MAVEN reaches an average win rate of 40% at 5 million steps in 6h_vs_8z in the original paper, while it remains with 0% win rate in this paper with the same about of steps.\n- QMIX reaches 100% win rate in 10m_vs_11m and 80% in 3s_vs_5z in the SMAC paper, while it is notably lower in this paper.\n\nThe paper heavily depends on the appendix indicating its lack of self-containment. The paper's content or the approach need more simplification to give a sense of completeness of the presentation.\n\nFigure labels and numbers are only readable on a sufficient zoom scale which is problematic when printed.\n\nQUESTIONS:\nI do not fully understand the purpose of the alter ego models. Since the policies are trained to directly maximize the joint-Q function, why introducing a „detour“ to the mixing network through the individual Q-functions? Wouldn’t simply feeding the resulting joint action (e.g., as one hot vector) be sufficient to compute the TD-target for Eq. 4?\n\nLIMITATIONS:\nSome limitations like the additional compute through sampling actions are mentioned in the text.\n\nPotential negative societal impact is not discussed at all.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-24"}

comments: SUMMARY:
This paper proposes a different approach for multi-agent RL algorithms that does not depend on individual global max (IGM), but instead builds a new framework, DAVE,  based on dual self-awareness. The algorithm is shown to perform favorably in several testing cases.

STRENGTHS:
The paper is well organized and the sections’ structure serve their purpose in illustrating this new framework. The experimental results also corroborate the authors’ claims and show that the proposed framework/algorithm, DAVE, performs better than other MARL algorithms.

WEAKNESSES:
The main weakness in this paper is the lack of justification of differences between DAVE and the actor-critic models. In addition, since this paper proposes a new algorithm, it would be nice to have the pseudocode be in the main text. Overall, a better analysis of the sampling procedure is needed to justify the new DAVE framework.

QUESTIONS:
Question: from figure 2, it appears that the alter ego model calculates the q value function and is similar in concept to a critic network. The ego model also computes the policy which is the same as an actor network. How is the dual self-awareness model different from an actor-critic model? The description in lines 193-194 appears to limit the self-awareness model to be a special case of a general actor-critic model.

LIMITATIONS:
The proposed model “dual self-awareness” requires a specific sampling procedure to get the actor policy which renders this framework as a special case of actor-critic algorithms.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel MARL algorithm, Dual self-Awareness Value dEcomposition (DAVE), which avoids the IGM constrain obeyed by most of the previous researches. The algorithm introduces three different policies including Ego Policy, Alter Ego Policy and Anti-ego Policy, where the Ego Policy tries to fit the optimal actions of the Alter Ego Policy and the Anti-ego Policy serves as an exploration strategy. Empirical results show that DAVE outperforms the baseline methods on several cooperative tasks.\n\nSTRENGTHS:\n1. The proposed DAVE algorithm avoids the IGM principle, which typically requires consistency between local and global optimal actions. This deviation enables DAVE to leverage the expressive capabilities of neural networks, particularly the mixing network component.\n2. The Anti-Ego policy, a key component as the DAVE's exploration strategy, adopts the idea of count-based exploration techniques such as RND [1] and \"Ensemble\" in model-based RL. It quantifies the familiarity of $(s,a)$ pairs by evaluating the reconstruction loss of an auto-encoder.\n3. Experimental results on Matrix Games show that DAVE does avoid local optima. Additionally, experiments conducted on SMAC, MA-MuJoCo indicate that DAVE-QMIX performs comparably to QPLEX and other baseline algorithms.\n\n\n[1] Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018). Exploration by random network distillation. arXiv preprint arXiv:1810.12894.\n\nWEAKNESSES:\n1. **Methodology**: \n\n    *a*) The Ego Policy only approximates the global optimal actions $u^{*}$ by NLL loss, which means the equivalence between the Ego Policy and the global optimum is not strictly guaranteed while it is hold strictly by IGM principle.\n\n    *b*) The basic idea of the exploration strategy is not quite novel. Simultaneously, while sampling from the Anti-Ego Policy guarantees the selection of low-Q actions, it does not guarantee that suboptimal-Q actions have been explored enough in the environment when the $M$ is not large enough. This is because reaching state $s$ is not guaranteed. As a result, the Q-values associated with these actions are prone to overestimation or underestimation, and the exploration policy tends to overlook this aspect of the action space.\n\n2. **Experiments**: The results on SMAC and MA-MuJoCo are a bit confusing to me and i have listed my concerns in the *Qustions*.\n\nQUESTIONS:\n1. The optimization of the $Q_{\\text{tot} }^\\text{{alter}}$ network heavily relies on the action sample set $U^{\\text{ego}}$ due to the computation of the target. However, unlike common MARL algorithms where the policy receives gradients through backpropagation, in DAVE, the Ego policy and $Q_{\\text{tot}}^\\text{{alter}}$ network are trained individually. Although *Proposition A.1* guarantees that objectively existing $u^*$ will be sampled when $M$ is sufficiently large, it does not ensure convergence to the optimal point in the optimization process. \nThen, there are two different models need to optimize and they are highly related, how do you ensure the convergence of them? \n\n2. There are concerns regarding whether Equation (8) truly returns \"the most novel joint action.\" As mentioned in *1.b)* in the *Weakness* part, reaching state $s$ can be challenging. However, the action set $\\hat{U}^{\\text{ego}}$ is sampled from the Anti-Ego Policy. When $M$ is limited, the suboptimal-Q actions are hard to be sampled  both in the true interaction process (due to the hardness of reaching $s$) and in the Anti-Ego Policy sampling process (as their Q-values are not sufficiently low).\n\n3. \n    *a*) Most figures in the *Experiment Section* display the median performance. Can you replace them with the mean performance as it is more often used in other works?\n\n    *b*) The results on SMAC of DAVE is based on QMIX as stated in the paper. To facilitate better comparison, it is suggested to use a more distinguishable color for the QMIX curve.\n\n    *c*) If i've identified the QMIX curve correctly, DAVE exhibits significant superiority over QMIX in Figure 6. However, in the Appendix (Figure 13, 15, 17), DAVE shows only a slight advantage over QMIX, can you analyze the reason for this?\n\n4. line 312: \"The experimental results show that DAVE without anti-ego exploration still performs better than other baselines.\" Actually, I do not get this conclusion. In contrast, the green curve for \"DAVE w/o Exploration\" demonstrates a significant drop compared to the QMIX baseline. Does this mean that the exploration part plays the most important role in DAVE but not the relaxation of the IGM principle?\n\n5. It is unclear why the curve for *M=100* does not outperform the others in Figure 7 (2s_vs_1sc, 5m_vs_6m). Further explanation is required to address this discrepancy.\n\nLIMITATIONS:\nLimitations: This work is mostly aimed to solve the cooperative Multi-agent tasks.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents DAVE, an IGM-free value decomposition method, to enhance coordination ability in MARL. Drawing inspiration from the concept of dual self-awareness in psychology, DAVE consists of two components: an ego policy responsible for executing actions, and an alter ego value function involved in credit assignment and value estimation. By incorporating an explicit search procedure, DAVE eliminates the need for the IGM assumption and updates the actor using supervised signals instead of policy gradients. Additionally, the authors propose a novel anti-ego exploration mechanism to prevent the algorithm from being trapped in local optima. The method demonstrates impressive performance across a range of cooperative tasks, highlighting its effectiveness.\n\nSTRENGTHS:\n1. By employing NLL instead of policy gradients to update actors, DAVE effectively addresses the challenges associated with using unconstrained (without IGM assumption/monotonicity) factored critic.\n2. The utilization of AE loss in Anti-Ego Exploration to assess the novelty of joint-action is intriguing, albeit bearing some similarities to RND [1].\n3. The paper is well written and easy to follow. The integration of psychological concepts with algorithm design is a unique aspect that captured my interest. Furthermore, each component of the overall framework is thoroughly elucidated.\n4. DAVE demonstrates impressive performance in both matrix games and SMAC, with clear discussions regarding its limitations.\n\nWEAKNESSES:\n1. While the authors claim that DAVE is the first method to completely eliminate the IGM assumption in value decomposition approaches, it's worth noting that FACMAC-nonmonotonic [2] also achieves full IGM-free. Although DAVE may outperform FACMAC-nonmonotonic, the contribution of DAVE might have been overestimated. \n2. The discussion on related works is insufficient, and it would be beneficial to include references to value decomposition actor critic methods [2][5].\n3. Minor 1: The idea of applying NLL loss bears similarities to previous works that combine Cross-Entropy Methods with Policy Improvement [3][4] (discussion can be included).\n4. Minor 2: It appears that exploration has a significant impact on the performance of DAVE, making it a crucial component that cannot be disregarded, unlike simple epsilon-greedy exploration. It would be advisable to include comparisons between DAVE and other MARL exploration methods, such as EITI/EDTI[6] and CMAE[7], in the appendix.\n\nQUESTIONS:\n1. Considering that [2] states, \"On SMAC, FACMAC-nonmonotonic performs similarly to FACMAC on easy maps, but exhibits significantly worse performance on harder maps,\" it would be worth discussing why the authors chose to compare DAVE with FACMAC-nonmonotonic instead of FACMAC (fairness may not be an issue).\n2. There appears to be a notable discrepancy between the performance curves reported in Figure 6 of the current paper and Figure 7 in [2], despite both versions using SMAC 4.10. Could you explain the reasons behind this divergence? \n3. While the elimination of the IGM assumption is touted as a significant contribution of DAVE, it would be valuable to explore the performance of DAVE with a monotonic mixing network (or others with IGM assumption). Does the unconstrained factored mixing network indeed enhance performance?\n\n[1] Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2019, May). Exploration by random network distillation. In ICLR, 2019.\n\n[2] Peng, B., Rashid, T., Schroeder de Witt, C., Kamienny, P. A., Torr, P., Böhmer, W., & Whiteson, S. (2021). Facmac: Factored multi-agent centralised policy gradients. *In NeurIPS, 2021*.\n\n[3] Simmons-Edler, R., Eisner, B., Mitchell, E., Seung, S., & Lee, D. (2019). Q-learning for continuous actions with cross-entropy guided policies. *arXiv preprint arXiv:1903.10605*.\n\n[4] Neumann, S., Lim, S., Joseph, A., Pan, Y., White, A., & White, M. (2018). Greedy Actor-Critic: A New Conditional Cross-Entropy Method for Policy Improvement. In ICLR, 2023.\n\n[5] Wang, Y., Han, B., Wang, T., Dong, H., & Zhang, C. (2020). Off-policy multi-agent decomposed policy gradients. *arXiv preprint arXiv:2007.12322*.\n\n[6] Ackermann, Johannes, et al. \"Reducing overestimation bias in multi-agent domains using double centralized critics.\" *arXiv preprint arXiv:1910.01465* (2019).\n\n[7] Wang, Tonghan et al. “Influence-Based Multi-Agent Exploration.” *ArXiv* abs/1910.05512 (2019)\n\nLIMITATIONS:\nAs mentioned in Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel value decomposition framework for MARL based on the notion of dual self-awareness in psychology. The framework, called DAVE, consists of two neural network models for each agent: the alter ego value function model and the ego policy model. The former participates in the evaluation of the group to solve the global credit assignment problem, and the latter helps the former to find the optimal individual action through an explicit search procedure. The paper claims that DAVE is the first value decomposition method that completely abandons the Individual Global Max (IGM) assumption, which requires consistency between local and global optimal actions. The paper also introduces an anti-ego exploration method to prevent the ego policy from getting stuck in a bad local optimum. The paper evaluates the performance of DAVE in both simple and complex environments, such as StarCraft II and Multi-Agent MuJoCo, and compares it to other popular baselines. The paper shows that DAVE can solve non-monotonic problems that challenge existing value decomposition methods, and can achieve competitive performance despite the absence of IGM. The paper argues that DAVE provides a new perspective on addressing the problems posed by the IGM assumption in value decomposition methods.\n\nSTRENGTHS:\nImproving the efficacy of CTDE and value decomposition-based approaches to cooperative MARL is an important topic. The paper's investigation of the role of the IGM principle in CTDE approaches is novel and should be of interest to the broader MARL and RL communities. The method is supported by empirical results in both small-scale, well-principled environments and larger-scale, complex environments; and is comprehensive in its comparison to baselines. The paper is well-written and clear.\n\nWEAKNESSES:\n- The related work in the main paper is a bit sparse given the depth of work in this area. Moving more of the discussion from Appendix E into the Related Work section would improve the contextualization of this work.\n- Given the importance of sample size M, it would have been nice to see a greater exploration of the performance/computational cost trade-off. For example, how does M scale in more realistic environments? Are there environments where an unreasonably high value of M is needed? In the MMM2 example in Fig. 7 it looks like the learning signal is just beginning.\n- There is limited discussion of limitations, future work, and societal impact.\n\nQUESTIONS:\nCan you provide more context into what is needed to tune exploration coefficients dynamically?\nHow well can this method generalize to other variants of the MARL problem (e.g. mixed incentive, social dilemmas)? \nCan DAVE provably increase the solution concept reached by agents in social dilemmas / game theoretic settings?\nHow do you balance the trade-off between exploration and exploitation in the anti-ego exploration method?\n\nLIMITATIONS:\nThe main limitation here is the computational cost of large values of M. This does not prevent strong performance in the environments studied in this work, but may be expensive in more complex and/or real-world environments. There are also other small limitations, e.g. tuning exploration coefficients.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}