SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhiwei Xu, Bin Zhang, Dapeng Li, Guangchong Zhou, Zeren Zhang, Guoliang Fan  
Institute of Automation, Chinese Academy of Sciences  
School of Artificial Intelligence, University of Chinese Academy of Sciences  
{xuzhiwei2019, guoliang.fan}@ia.ac.cn
Value decomposition methods have gained popularity in the field of cooperative multi-agent reinforcement learning. However, almost all existing methods follow the principle of Individual Global Max (IGM) or its variants, which limits their problem-solving capabilities. To address this, we propose a dual self-awareness value decomposition framework, inspired by the notion of dual self-awareness in psychology, that entirely rejects the IGM premise. Each agent consists of an ego policy for action selection and an alter ego value function to solve the credit assignment problem. The value function factorization can ignore the IGM assumption by utilizing an explicit search procedure. On the basis of the above, we also suggest a novel anti-ego exploration mechanism to avoid the algorithm becoming stuck in a local optimum. As the first fully IGM-free value decomposition method, our proposed framework achieves desirable performance in various cooperative tasks.
Multi-agent reinforcement learning (MARL) has recently drawn increasing attention. MARL algorithms have been successfully applied in diverse fields, including game AI [1] and energy networks [37], to solve practical problems. The fully cooperative multi-agent task is the most common scenario, where all agents must cooperate to achieve the same goal. Several cooperative MARL algorithms have been proposed, with impressive results on complex tasks. Some algorithms [7, 32, 17] enable agents to collaborate through communication. However, these approaches incur additional bandwidth overhead. Most communication-free MARL algorithms follow the conventional paradigm of centralized training with decentralized execution (CTDE) [20]. These works can be roughly categorized into multi-agent Actor-Critic algorithm [8, 16] and value decomposition method [34, 27, 19, 40, 39]. Due to the high sample efficiency in off-policy settings, value decomposition methods can outperform other MARL algorithms.
However, due to the nature of the CTDE paradigm, the value decomposition method can only operate under the assumption of Individual Global Max (IGM) [30]. The IGM principle links the optimal joint action to the optimal individual one, allowing the agent to find the action corresponding to the maximal Q value during decentralized execution. Nevertheless, this assumption is not applicable in most real-world scenarios. And most value decomposition approaches [27] directly constrain the weights of the neural network to be non-negative, which restricts the set of global state-action value functions that can be represented. We confirm this point through experiments in Appendix D.1.
Removing the IGM assumption requires agents to be able to find actions corresponding to the maximal joint Q value during execution. Therefore, in addition to the evaluation network, agents also need an additional network to learn how to search for the best actions in a decentralized manner. Similarly, there are concepts of ego and alter ego in psychology. The ego usually refers to the conscious part of the individual, and Freud considered the ego to be the executive of the personality [9]. Some people
believe that an alter ego pertains to a different version of oneself from the authentic self [24, 12]. Others define the alter ego in more detail as the evaluation of the self by others [3]. As a result, both ego and alter ego influence a person's personality and behaviour, as depicted in Figure 1. Besides, Duval and Wicklund [5] put forth the dual self-awareness hypothesis, which posits that individuals possess two states of consciousness: public self-awareness and private self-awareness. Private self-consciousness refers to an individual's in
Figure 1: In psychology, it is commonly believed that the ego and alter ego determine an individual's behavior jointly.
ternal understanding of himself, whereas public self-awareness enables individuals to recognize that they are evaluated by others. Enlightened by these psychological concepts, we propose a novel MARL algorithm, Dual self-Awareness Value dEcomposition (DAVE). Each agent in DAVE maintains two neural network models: the alter ego model and the ego model. The former participates in the evaluation of the group to solve the global credit assignment problem, and the latter is not directly optimized by the joint state-action value. The ego policy can help the alter ego value function to find the individual action corresponding to the optimal joint action through explicit search procedures so that the value decomposition framework can completely eliminate the constraints of the IGM assumption. Furthermore, to prevent the ego policy based on the above framework from potentially getting stuck in a bad local optimum, we also propose an anti-ego exploration method. This method can effectively avoid the underestimation of the optimal action that may be caused by the limited search procedure.
We first evaluate the performance of DAVE in several didactic problems to demonstrate that DAVE can resolve the issues caused by IGM in conventional value decomposition methods. Then we run DAVE in various complex scenarios including StarCraft II, and compare it to other popular baselines. Our experimental results show that DAVE can still achieve competitive performance despite the absence of IGM. To the best of our knowledge, DAVE is the first multi-agent value decomposition method that completely abandons the IGM assumption.
Recently, value decomposition methods have made significant progress in problems involving the decentralized partially observable Markov decision process (Dec-POMDP) [23]. By calculating a joint state-action value function based on the utility functions of all agents, value decomposition methods can effectively alleviate the notorious credit assignment issue. The first proposed method, VDN [34], simply sums up the utility of all agents to evaluate joint actions. This undoubtedly limits the range of joint value functions that can be approximated. QMIX [27] incorporates hypernetworks [11] into the value decomposition method and employs the mixing network to fit the joint value function based on the individual ones, thereby expressing richer classes of joint value function. To enable the agent to choose actions with the greedy policy during decentralized training, QMIX assumes monotonicity and restricts the weight parameters of the mixing network output by hypernetworks to non-negative. So the monotonic mixing network still cannot represent some class of value functions. Various value decomposition variants have subsequently been proposed. Officially proposing IGM, QTRAN [30] indicated that the value decomposition method must satisfy the assumption that the optimal joint actions across agents are equivalent to the collection of individual optimal actions of each agent. Unlike VDN and QMIX, QTRAN is not restricted by structural constraints such as non-negative weights, but it performs poorly in complex environments such as StarCraft II. MAVEN [21] enhances its performance by improving its exploration capabilities. Weighted QMIX [28] takes into account a weighted projection that places greater emphasis on better joint actions. QPLEX [36] takes a duplex dueling network architecture to factorize the joint action-value function and proposes the advantage-based IGM. Nevertheless, the aforementioned methods only loosen the restrictions by proposing IGM variants. More importantly, existing policy-based MARL methods [41, 20, 25] do not explicitly introduce IGM, but they also cannot solve some non-monotonic problems. So how to remove the IGM assumption is still an open and challenging problem. Our proposed DAVE is completely IGM-free and can be applied to existing IGM-based value decomposition methods. More related work will be discussed in Appendix E.
3.1 Dec-POMDPs
In this paper, we concentrate on the Dec-POMDP problems. It is an important mathematical model for multi-agent collaborative sequential decision-making problem, in which agents act based on different pieces of information about the environment and maximize the global shared return. Dec-POMDPs can be formulated as a tuple  $\langle \mathcal{A},\mathcal{S},\mathcal{U},\mathcal{Z},P,O,r,\gamma\rangle$ .  $a\in \mathcal{A} := \{1,\dots,n\}$  represents the agent in the task.  $s\in S$  is the global state of the environment. The joint action set  $\mathcal{U}$  is the Cartesian product of action sets  $\mathcal{U}^a$  of all agents and  $\pmb {u}\in \mathcal{U}$  denotes the joint action. Given the state  $s$  and the joint action  $\pmb{u}$ , the state transition function  $P:S\times \mathcal{U}\times \mathcal{S}\rightarrow [0,1]$  returns the probability of the next potential states. All agents receive their observations  $z = \{z^{1},\dots,z^{n}\}$  obtained by the observation function  $O:S\times \mathcal{A}\to \mathcal{Z}$ .  $r:S\times \mathcal{U}\to \mathbb{R}$  denotes the global shared reward function and  $\gamma$  is the discount factor. Besides, recent work frequently allows agents to choose actions based on historical trajectory  $\tau^a$  rather than local observation  $z^a$  in order to alleviate the partially observable problem.
In Dec-POMDP, all agents attempt to maximize the discount return  $R_{t} = \sum_{t=0}^{\infty} \gamma^{t} r_{t}$  and the joint state-action value function is defined as:
$$
Q _ {\text {t o t}} (s, \boldsymbol {u}) = \mathbb {E} _ {\boldsymbol {\pi}} \left[ R _ {t} | s, \boldsymbol {u} \right],
$$
where  $\pi = \{\pi_1,\dots ,\pi_n\}$  is the joint policy of all agents.
3.2 IGM-based Value Decomposition Methods
The goal of value decomposition method is to enable the application of value-based reinforcement learning methods in the CTDE paradigm. This approach decomposes the joint state-action value function  $Q_{\mathrm{tot}}$  into the utility function  $Q_{a}$  of each agent. In VDN, the joint state-action value function is equal to the sum of all utility functions. And then QMIX maps utility functions to the joint value function using a monotonous mixing network and achieve better performance.
Nonetheless, additivity and monotonicity still impede the expressive capacity of the neural network model, which may result in the algorithm being unable to handle even for some simple matrix game problems. To this end, QTRAN proposes the Individual Global Max (IGM) principle, which can be represented as:
$$
\arg \max  _ {u ^ {a}} Q _ {\mathrm {t o t}} (s, \boldsymbol {u}) = \arg \max  _ {u ^ {a}} Q _ {a} \left(\tau^ {a}, u ^ {a}\right), \quad \forall a \in \mathcal {A}.
$$
The IGM principle requires consistency between local optimal actions and global ones. Most value decomposition methods including VDN and QMIX must follow IGM and its variants. This is because only when the IGM principle holds, the individual action corresponding to the maximum individual action value during decentralized execution is a component of the optimal joint action. Otherwise, the agent cannot select an appropriate action based solely on the individual action-value function.
In this section, we introduce the implementation of the dual self-awareness value decomposition (DAVE) framework in detail. DAVE incorporates an additional ego policy through supervised learning in comparison to the conventional value decomposition method, allowing the agent to identify the optimal individual action without relying on the IGM assumption. Moreover, DAVE can be applied to most IGM-based value decomposition methods and turn them into IGM-free ones.
4.1 Dual Self-Awareness Framework
Almost all value decomposition methods follow the IGM assumption, which means that the optimal joint policy is consistent with optimal individual policies. According to IGM, the agent only needs to follow the greedy policy during decentralized execution to cater to the optimal joint policy. Once the IGM assumption is violated, agents cannot select actions that maximize the joint state-action value based solely on local observations. Consequently, the objective of the IGM-free value decomposition method is as follows:
$$
\underset {\pi} {\arg \max } Q _ {\text {t o t}} (s, \boldsymbol {u}), \tag {1}
$$
Figure 2: Middle: The overall dual self-awareness value decomposition framework. Right: Agent network structure. Left: The setup of the anti-ego exploration.
where  $\pmb{u} = \{u^{1},\dots,u^{n}\}$  and  $u^a\sim \pi_a(\tau^a)$ . Without the IGM assumption, Equation (1) is NP-hard because it cannot be solved and verified in polynomial time [14]. Therefore, in our proposed dual self-awareness framework, each agent has an additional policy network to assist the value function network to find the action corresponding to the optimal joint policy. Inspired by the dual self-awareness theory in psychology, we named these two networks contained in each agent as ego policy model and alter ego value function model, respectively. Figure 2 depicts the dual self-awareness framework. Note that the IGM-free mixing network refers to the mixing network obtained by removing structural constraints. For instance, the IGM-free QMIX variant does not impose a non-negative constraint on the parameters of the mixing network.
In DAVE, the alter ego value function model of each agent  $a$  is identical to the agent network in other value decomposition methods and generates individual state-action value  $Q_{a}^{\mathrm{alter}}(\tau^{a},\cdot)$  based on the current historical trajectory  $\tau^a$ . To obtain the joint state-action value  $Q_{\mathrm{tot}}^{\mathrm{alter}}(s,\pmb {u})$  for carrying out the joint action  $\pmb{u}$  in the state  $s$ , the value function of each agent is fed into the IGM-free mixing network. However, the alter ego model does not determine the actions chosen by the agent, either during centralized training or decentralized execution. What determines the action of each agent is the ego policy  $\pi_a^{\mathrm{ego}}$ . The joint ego policy can be expressed as  $\pi^{\mathrm{ego}}(\pmb {u}\mid s) = \prod_{a = 1}^{n}\pi_{a}^{\mathrm{ego}}(u^{a}\mid \tau^{a})$ . When executing, the agent interacts with the environment by sampling actions from the categorical distribution output by the ego policy. And during training, we replace the exact maximization of  $Q_{\mathrm{tot}}(s,\pmb {u})$  over the total joint action space with maximization over a set of samples  $U^{\mathrm{ego}}$  from  $\pi^{\mathrm{ego}}$ . So we modify the objective of the cooperative multi-agent value decomposition problem from Equation (1) to
$$
\underset {\pi^ {\mathrm {e g o}}} {\arg \max } Q _ {\text {t o t}} ^ {\text {a l t e r}} (s, \boldsymbol {u} ^ {\text {e g o}}), \quad \text {s . t .} \quad \boldsymbol {u} ^ {\text {e g o}} \in \boldsymbol {U} ^ {\text {e g o}}, \tag {2}
$$
where  $U^{\mathrm{ego}} \coloneqq \{\pmb{u}_i^{\mathrm{ego}} \sim \pmb{\pi}^{\mathrm{ego}}(s)\}_{i=1}^M$  in state  $s$  and  $M$  is the number of samples. Therefore, the dual self-awareness framework substitutes the global optimal state-action value with the local optimal one found in the small space  $U^{\mathrm{ego}}$ , which is obtained by sampling  $M$  times from  $\pmb{\pi}^{\mathrm{ego}}$ . As long as the ego policy assigns non-zero probabilities to all available actions, this method will converge to the objective described by Equation (1) as the number of samples  $M$  increases. The detailed proof can be found in Appendix A. Nevertheless, the computational cost is directly proportional to  $M$ . So the choice of  $M$  must strike a balance between precision and computational complexity.
Next, we introduce the update process of the alter ego value function and the ego policy in the dual self-awareness framework. According to the modified objective of the cooperative multi-agent value decomposition problem illustrated by Equation (2), the ego policy needs to be able to predict the individual action that corresponds to the maximal joint state-action value. Therefore, we update the ego policy through supervised learning to increase the probability of the optimal joint action obtained through the sampling procedure. We define  $\pmb{u}^{\star} = \underset {\pmb{u}^{\mathrm{ego}}}{\arg \max}\mathcal{Q}_{\mathrm{tot}}^{\mathrm{alter}}(s,\pmb{u}^{\mathrm{ego}})$  as the joint action corresponding to the maximal joint state-action value in the samples, where  $\pmb{u}^{\mathrm{ego}}\in U^{\mathrm{ego}}$  . The loss function for the joint ego policy can be written as:
$$
\mathcal {L} _ {\text {e g o}} = - \log \pi^ {\text {e g o}} \left(\boldsymbol {u} ^ {\star} \mid s\right). \tag {3}
$$
The objective function is the negative log-likelihood of the probability of the joint action  $\pmb{u}^{\star}$  and can be further expressed as  $-\sum_{a=1}^{n} \log \pi_a^{\mathrm{ego}}(u^{a\star} \mid \tau^a)$  associated with individual ego policies.
Additionally, the alter ego value function  $Q^{\mathrm{alter}}$  is constantly changing during the training process. To prevent the ego policy  $\pi^{\mathrm{ego}}$  from becoming too deterministic and getting trapped in local optimal solutions, we will introduce a novel exploration method in the next subsection.
Conventional value decomposition methods can be trained end-to-end by minimizing the following loss:
$$
\mathcal {L} = \left(y _ {\text {t o t}} - Q _ {\text {t o t}} \left(s _ {t}, \boldsymbol {u} _ {t}\right)\right) ^ {2}, \tag {4}
$$
where  $y_{\mathrm{tot}} = r_t + \gamma \max_{\boldsymbol{u}_{t+1}} Q_{\mathrm{tot}}^- (s_{t+1}, \boldsymbol{u}_{t+1})$  and  $Q_{\mathrm{tot}}^-$  is the target joint value function. The loss function described above cannot be directly applied to our proposed dual self-awareness framework because of the max operator. If we also traverse the samples of the joint action in  $U^{\mathrm{ego}}$  to select the local maximum target joint state-action value function, which may result in high variance. Inspired by the Expected SARSA [35], we use the expected value instead of the maximal one. The update rule of  $Q_{\mathrm{tot}}^{\mathrm{alter}}$  considers the likelihood of each action under the current policy and can eliminate the variance, which can be expressed as:
$$
\mathcal {L} _ {\text {a l t e r}} = \left(y _ {\text {t o t}} ^ {\text {a l t e r}} - Q _ {\text {t o t}} ^ {\text {a l t e r}} \left(s _ {t}, \boldsymbol {u} _ {t}\right)\right) ^ {2}, \tag {5}
$$
where  $y = r_{t} + \gamma \mathbb{E}_{\pi^{\mathrm{ego}}}[(Q_{\mathrm{tot}}^{\mathrm{alter}})^{-}(s_{t+1}, \boldsymbol{u}_{t+1})]$ . However, this method is still computationally complex. So we reuse the action sample set  $\boldsymbol{U}^{\mathrm{ego}}$  and replace  $\mathbb{E}_{\pi^{\mathrm{ego}}}[(Q_{\mathrm{tot}}^{\mathrm{alter}})^{-}(s_{t+1}, \boldsymbol{u}_{t+1})]$  with  $\frac{1}{M} \sum_{i=1}^{M} (Q_{\mathrm{tot}}^{\mathrm{alter}})^{-}(s_{t+1}, \boldsymbol{u}_{i}^{\mathrm{ego}})$ , where  $\boldsymbol{u}_{i}^{\mathrm{ego}} \in \boldsymbol{U}^{\mathrm{ego}}$  in state  $s_{t+1}$ . The modified update rule for  $Q_{\mathrm{tot}}^{\mathrm{alter}}$  reduces the variance and does not increase the order of the computational complexity compared to the original update method shown in Equation (4). Each hypernetwork takes the global state and the actions of all agents as input and generates the weights of the mixing network. It should also be noted that we use the global state and the actions of all agents as input to the hypernetwork that generates weights to enhance the expressiveness of the mixing network.
In summary, we have constructed the fundamental framework of DAVE. Although DAVE consists of a value network and a policy network, it is DISTINCT from the Actor-Critic framework or other policy-based methods such as MAPPO [41], MADDPG [20] and FACMAC [25]. DAVE requires an explicit sampling procedure to update the network model. Besides, the actions sampled from the ego policies remain independent and identically distributed (IID) [26], so the ego policy is trained through supervised learning rather than policy gradient used in policy-based methods. Most intuitively, conventional policy-based methods cannot solve simple non-monotonic matrix games but DAVE can easily find the optimal solution, as described in Section 5.1 and Appendix D.2.
4.2 Anti-Ego Exploration
As stated previously, updating the ego policy network solely based on Equation (3) with an insufficient sample size  $M$  may lead to a local optimum. This is because the state-action value corresponding to the optimal joint action may be grossly underestimated during initialization, making it challenging for the ego policy to choose the optimal joint action. Consequently, the underestimated  $Q_{\mathrm{tot}}^{\mathrm{alter}}$  will have no opportunity to be corrected, creating a vicious circle. Therefore, we developed an anti-ego exploration method based on ego policies. Its core idea is to assess the occurrence frequency of state-action pairs using the reconstruction model based on the auto-encoder [13]. The auto-encoder model often requires the encoder to map the input data to a low-dimensional space, and then reconstructs the original input data based on the obtained low-dimensional representation using the decoder. Our method leverages the fact that the auto-encoder cannot effectively encode novel data and accurately reconstruct it. The more significant the difference between the original state-action pair  $(s, u)$  and the reconstructed output  $(s', u')$ , the less frequently the agents carry out the joint action  $u$  in state  $s$ . Simultaneously, the difference between the reconstructed state-action pairs and the original ones is also the objective function for updating the auto-encoder, which can be expressed as:
$$
\mathcal {L} _ {\text {r e c o n}} (s, \boldsymbol {u}) = \operatorname {M S E} (s, s ^ {\prime}) + \sum_ {a = 1} ^ {n} \operatorname {C E} \left(u ^ {a}, \left(u ^ {a}\right) ^ {\prime}\right), \tag {6}
$$
where  $\mathrm{MSE}(\cdot)$  and  $\mathrm{CE}(\cdot)$  represent the mean square error and the cross-entropy loss function respectively. So we need to encourage agents to choose joint actions with a higher  $\mathcal{L}_{\mathrm{recon}}$ . Obviously, it is impractical to traverse all available joint actions  $\pmb{u}$ . Then we look for the most novel actions within the limited set of actions obtained via the sampling procedure. In order to get state-action pairs that are as uncommon as possible, we generate an anti-ego policy  $\hat{\pi}^{\mathrm{ego}}$  based on  $\pi^{\mathrm{ego}}$ , as shown
in Figure 2. The relationship between the ego and anti-ego policy of each agent is as follows:
$$
\pi_ {a} ^ {\mathrm {e g o}} \left(\tau^ {a}\right) = \operatorname {S o f t m a x} \left(f \left(\tau^ {a}\right)\right),
$$
$$
\hat {\pi} _ {a} ^ {\mathrm {e g o}} \left(\tau^ {a}\right) = \operatorname {S o f t m i n} \left(f \left(\tau^ {a}\right)\right), \tag {7}
$$
where  $f(\cdot)$  denotes the neural network model of ego policy. Equation (7) demonstrates that actions with lower probabilities in  $\pi_{a}^{\mathrm{ego}}$  have higher probabilities in  $\hat{\pi}_{a}^{\mathrm{ego}}$ . Of course, actions with low probabilities in  $\pi_{a}^{\mathrm{ego}}$  may be non-optimal actions that have been fully explored, so we filter out such actions through the reconstruction model introduced above.  $\hat{U}^{\mathrm{ego}} := \{\hat{u}_i^{\mathrm{ego}} \sim \hat{\pi}^{\mathrm{ego}}(s)\}_{i=1}^{M}$  is the joint action set obtained by sampling  $M$  times from the anti-ego policy in state  $s$ . We can identify the most novel joint action  $\hat{\pmb{u}}^{\star}$  in state  $s$  by the following equation:
$$
\hat {\boldsymbol {u}} ^ {\star} = \underset {\hat {\boldsymbol {u}} ^ {\mathrm {e g o}}} {\arg \max } \mathcal {L} _ {\text {r e c o n}} (s, \hat {\boldsymbol {u}} ^ {\mathrm {e g o}}), \tag {8}
$$
where  $\hat{\pmb{u}}^{\mathrm{ego}}\in \hat{\pmb{U}}^{\mathrm{ego}}$ . Therefore, the objective function in Equation (3) needs to be modified to increase the probability of the joint action  $\hat{\pmb{u}}^{\star}$ . The final objective function of the ego policy  $\pi^{\mathrm{ego}}$  can be written as: