Consciousness-Aware Multi-Agent Reinforcement Learning
Authors: Jianzhun Shao
Hongchang Zhang
Yun Qu
Chang Liu
Shuncheng He
Yuhang Jiang
Xiangyang Ji
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-agent reinforcement learning

Abstract: In cooperative multi-agent reinforcement learning, centralized training with decentralized execution (CTDE) shows great promise for a trade-off between independent Q-learning and joint action learning. However, vanilla CTDE methods assumed a fixed number of agents could hardly adapt to real-world scenarios where dynamic team compositions typically suffer from the dilemma of dramatic partial observability variance. Specifically, agents with extensive sight ranges are prone to be affected by trivial environmental substrates, dubbed the “attention distraction” issue; ones with limited observability can hardly sense their teammates, hindering the quality of cooperation. In this paper, we propose a Consciousness-Aware Multi-Agent reinforcement learning (CAMA) approach, which roots in a divide-and-conquer strategy to facilitate stable and sustainable teamwork. Concretely, CAMA targets dividing the input entities with controlled observability masks by an Entity Dividing Module (EDM) according to their execution relevance for consciousness learning. To tackle the attention distraction issue, the highly related entities are fed to a Consciousness Enhancement Module (CEM) for consciousness-aware representation extraction via action prediction with an inverse model. For better out-of-sight-range cooperation, the lowly related ones are compressed to brief messages by a Consciousness Replenishment Module (CRM) with a conditional mutual information estimator. Our CAMA outperforms the SOTA methods significantly on the challenging StarCraftII, MPE, and Traffic Junction benchmarks.

Under review as a conference paper at ICLR 2023
CONCENTRATED ATTENTION FOR MULTI-AGENT RE- INFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In cooperative multi-agent reinforcement learning, centralized training with decen- tralized execution (CTDE) shows great promise for a trade-off between independent Q-learning and joint action learning. However, vanilla CTDE methods assumed a fixed number of agents could hardly adapt to real-world scenarios where dynamic team compositions typically suffer from the dilemma of dramatic partial observabil- ity variance. Specifically, agents with extensive sight ranges are prone to be affected by trivial environmental substrates, dubbed the "attention distraction" issue; ones with limited observability can hardly sense their teammates, hindering the quality cooperation. In this paper, we propose a Concentrated Attention for Multi-Agent reinforcement learning (CAMA) approach, which roots in divide-and-conquer strategy to facilitate stable and sustainable teamwork. Concretely, CAMA targets dividing the input entities with controlled observability masks by an Entity Divid- ing Module (EDM) according to their contributions for attention weights. To tackle the attention distraction issue, the highly contributed entities are fed an Attention Enhancement Module (AEM) for execution-related representation extraction via action prediction with an inverse model. For better out-of-sight-rang cooperation, the lowly contributed ones are compressed to brief messages by Attention Re- plenishment Module (ARM) with conditional mutual information estimator. Our CAMA outperforms the SOTA methods significantly the challenging StarCraftII, MPE, and Traffic Junction benchmarks.
INTRODUCTION
Cooperative multi-agent deep reinforce- ment learning (MARL) has gained in-
Agents
Sight Range
Global Coach
creasing attention in many areas such as
games (Berner et al. 2019 Samvelyan et al., 2019, Kurach et al. 2019), so-
cial science (Jauus al.) 2019), sensor
networks (Zhang & Lesser 2013). and
autonomous vehicle control (Xu al. 2018). With practical agent cooperation and scalable deployment capability, cen- tralized training with decentralized execu- tion (CTDE) (Rashid et al. 2018 Gupta et al. 2017) has been widely adopted for MARL. Current CTDE methods usually assume a fixed number of agents such QMIX (Rashid et 2018). MADDPG (Lowe et al. 2017), QPLEX (Wang al.) 2020a), etc. To adapt to complicated and
(8)
(b)
(c)
Figure The dynamic sight range dilemma. (a) Agents
hardly cooperate beyond their sight ranges. (b) Agents with large sight ranges may perform worse due to "attention distraction". (c) A sketch of our CAMA.
dynamic real-world scenarios with dynamic team compositions (i.e., the team size varies), researchers
extend these methods by introducing the attention mechanism (Vaswani et al. 2017). which usually
requires splitting the state of the environment into a series of entities (Yang et al. 2020 Agarwal
al. 2019
Igbal et al. 2021).
However, attention based methods can hardly handle the varying partial observability (e.g., the varying sight range of each agent) in multi-agent systems, Fig, [ With severe partial observability, agents usually lose the sight of teammates, leading to the poor coordination quality. We use a demo in Sec. 5.1 to verify the phenomenon. With slight partial observability (large sight ranges with

---

Under review as a conference paper at ICLR 2023
near perfect information), these methods exhibit apparent performance degradation that more trivial entities may distract the agents' attention and interfere with their decision making. See Sec. [4. for a detailed analysis. Therefore, maintaining agents' attention on potential cooperators and execution- related entities to adapt to the partial observability variation is crucial for MARL in challenging environments.
In this paper, we propose a Concentrated Attention for Multi-Agent reinforcement learning (CAMA) approach, which roots in divide-anddcooquee strategy to facilitate stable and sustainable teamwork via attention learning. Specifically, we first use an Entity Dividing Module (EDM) to divide the raw entities into two parts for each agent according to its attention weights. For the attention distraction issue in settings with large sight ranges, an Attention Enhancement Module (AEM) is applied on the entities with high attention weights for execution-related representation extraction via action prediction with an inverse model. For out-of-sight-range coordination in low sight ranges, an Attention Replenishment Module (ARM) with novel conditional mutual information estimator is applied to compress the information in entities with low attention weights. With the above three modules, agents' attention can be properly concentrated on execution of local actions and potential teamwork to deal with dynamic partial observability.
We evaluate our method on three commonly used benchmarks: StarCraftII (SC2) (Samvelyan et al. 2019). Multi-agent Particle Environment (MPE) (Lowe et al. (017). and Traffic Junction (Sukhbaatar et al. 2016). The proposed CAMA outperforms SOTA methods significantly on all conducted experi- ments and exhibits remarkable robustness to sight range variation and dynamic team composition.
2 RELATED WORK
As a popular paradigm for single reward MARL, CTDE is a trade-off between independent Q- learning Tan 1993) and joint action learning CClaus & Boutilier 1998). Centralized training makes agents cooperate better while decentralized execution benefits the flexible deployment capability. A series of works concentrate on distributing the team reward to all agents by value function factorization (Sunehag et al. 2018; Rashid et al. (018). deriving and extending the Individual-Global Max (IGM) principle for policy optimality analysis (Son al. 2019 Wang al. 2020a Rashid
et al. 2020; Wan et al., (02T). D). To avoid constraints of IGM, some works then delve into applying centralized critics on local policies using the actor-critic paradigm (Lowe et al. [2177: Foerster et al.
2018 Zhou et al., 2020). Although CTDEs have achieved great progresses in recent years, with fixed
sizes of agents, they are typically impeded by the dynamic team composition issue (Schroeder del 2021) in real-world applications.
Witt et al. 2019 Liu et al.
Dynamic Team Composition. When the agent number varies in each episode, the attention mecha- nism (Vaswani et al. (017)) is commonly adopted to handle the issue (Jiang et al. 2018 Agarwal et al.,
2019 Yang et al. 2020: Hu et al. 2020; Iqbal et al.
202 Some works develop a set of curricula
to adapt to the increasing team sizes (Baker al. 2019: Long al. 2020; Wang et al.; 2020c)
with negliggli computational costs for training on different team sizes, Iqbal et al. (2021) add
auxiliary Q-learning tasks to increase the multi-agent system's 's robustness by randomly masking
part of agents' observability, which increases the types of situations encountered by agents. Although these methods adapt different team sizes well, they still suffer obvious attention distraction when
the sight ranges of agents is large, and are prone to fail in some situations where agents with limited observability must cooperate beyond their sight ranges (Liu et al. 202T)
Communication Mechanism is a feasible solution to enhance agents' cooperation. Recently some works regard the relationship between agents as a proximity-basee or fully-connected graph and assume the information can propagate among the graph edges (Foerster et al. 2016, SSttle et al. 2020 Agarwal et al. 2019, Zhang et al., 2018, Sukhhaataar al. [2066] et al., 2020
|020||aa Mao et al. 2020a). These methods usually let agents communicate with all neighbors or the whole team, which brings high communication costs. Moreover, the communication mode is sensitive to team sizes. Some works assume the existence of centralized coach to integrate information and send messages to all agents (Liu et al. 2021: Mao et al.) 2020b. Niu et al., 2021). which requires centralized execution, with relatively low communication costs. The communication messages are usually trained by backpropagation of RL loss, sometimes with constraint from mutual information objective to
reduce the communication bandwidth (Wang al. 2020b), help action decision Yuuan al. 2022)
2

---

Under review as a conference paper at ICLR 2023
or predict future trajectories (Liu et al. 2021). Unlike these methods, communication from centralized coach to apply attention replenishment for agents' better coordination.
3 BACKGROUND
MARL Symbols. We model a fully collaborative multi-agent task with agents decen- tralised partially observable Markov decision process (Dec-POMDP) (Olichock et al. 2016) G= (S, P.r, Z, O, 7), where E S is the environment's state. At time step each agent i LE [I...., Ra. chooses an action
A, which makes up the joint action a E A A"". P(s a+) S x + [0, 1] is the environment's 's state transition distribution. All agents share
the same reward function r(s. a) 5 x A - RR. The discount factor is denoted by E [0. 1). Each
agent i has its local observations of E O drawn from the observation function Z(s, i) : S x 1 -
A IJ(A)), where p' T (O x A)' denotes the action-observation history of agent i, and / is the number of state- action pairs in p'. p is the communication message and n' has no dependence on x" in CTDE. The agents' joint policy π induces St, ar), where R. = Σi= 7 r*+++ is the discounted accumulated team reward. The goal of MARL is to find the optimal joint policy n
and chooses an action by its stochastic policy >'aa x²) r X
histories of all agents. X" E denotes the additional
a joint action- -vauue function: Q" (st. a+) E E 11000 100.. 1
such that Q" (s, a) 2 Q" (sssa)). for all and (s, a) E S x A.
Value Function Factorization. During execution in CTDE, each agent chooses actions by its local Q function Q' (²')) induced by local observation o'. During training, since all agents share a common team reward, global Q function Quo is calculated from all local Q' conditioned on the global state by a M Mizer module as: Qto a) = (p', (p',a')) a.)...,QQ'' Q"(p", (pp,,"") a,"),,, s). To guarantee the global optimality from the local optimality, a Mixer should satisfy the IGM principle (Son et al.] 2019): arg max, (p, a) (arg max,' Q² (p², a²),.., arg max." Q"(p", (p"","" aa").
Multi-Heaa Attention (MHA) MARL. Under the condition of dynamic teams, the observation vector for each agent may have varying sizes during episode, and therefore traditional methods which only accept fixed-size input. In contrast, we use the "entity-wise input" to represent the observation and the multi-head attention (MHA) module to embed the entities with dynamic number into a fixed length vector for each agent. To feed MHA, the raw state s of the environment is commonly expressed as series of entities e', i.e., s {e'), i n. with the same vector length, where ne is the maximum number of entities. The entities include agents we can or can control, and other substrates in the multi-agent scenario (e.g. obstacles). We consider a meximmity-based observation function ((s i) for agent i, i.e., each agent has sight range SR, and [ss", i) := {e' [d(i.j) <= SR).j [1, ne]. where d(i..) is the Euclidean (or Manhattan) distance between entity and Let X' R xd be the entity input, in which each row is an entity. Let Ia S I == (1,..., n be the set of indices that selects which entities of the input X' are used to compute queries such that Xi, E Rd
hardly apply
xd (usually i E I, means entity i is an controllable agent, and la = {1, (l,... }). The attention head is as follows:
AH [L, M: W W WV) softmax mask (QKT vE MM) V E ER Q = - W K = wWW.KK. V XWV, M E (0, (0.1)n.x0 Jn.xn, Wº. WK WV Rdxx
The mask()) .M) operation takes two matrices with the same size input, and fills the entries of with ooo where M equals O. If we set the values in the positions of unseen entities to 0, this operation blocks the information from certain entities after softmax, uphold the partial observability for local agents. Wº, WK and W are learnable parameters. Then we can define the mulit-head attention module by concatenating n attention heads together:
MHA(I, X,MM = concat (AH (I,x,M;WW, W w.) .j (1 (I....I)), (2)
where information not blocked can be integrated across all entities.
3

---

Under review as a conference paper at ICLR 2023
Erdiy x
Maxer
Q
Mack M
Mask M,
AEM
CDM
Enily
Attention
Enhancereent Module
Agent i
Cup
Invalid
MHA
of
(eoot Af
Asin Weihts
[one]]
AFM
Agent 1
Altertion
FC
FC
Vaild
Replerishment
Module
f
Ni-1
F6
EDM
Emiky Dividing Module
Agents
MHHA
EDM
Others
Entlyy
Enity.
Mask
Lows
Figure 2: Network structure of our proposed CAMA. Entities and the observation mask are first fed into the Entity Dividing Module (EDM) get the Attention Enhancement embedding f and Attention Replenishment embedding f which is trained by inverse model (IM) loss and mutual information (MI) objective, respectively. For RL training, local Q' is generated by f¹, f. and its inservation-actioo history with a GRU, and further fed into the mixing network for Quat
5Rm3, WR=719 71%
5R=9, WR=53%
SRme, WR-51%
AI M 45
A5
Figure 3: The attention distraction issue illustration with the agents' average attention heatmap of REFIL on StarCCraftll map symmeersymetrrc" (the hardest scenario "8sz_vs_8a" vs__s=") during one episode on all entities. The y axis denotes 8 agents (A1-A8), while the axis with additional 8 enemies (E1-E8) records 16 entities. The sum of each row is normalized to 1. "SR" is each agent's sight range, and "WR" the winning rate against preset AI.
4 METHOD
4.1 INTUITION AND OVERVIEW
Intuition. The performance of traditional MARL methods is highly affected by the partial observ- ability of the environment. We use the agents in a game (e.g., StarCraftII) for example. When the sight range is small, the agents can hardly find the teammates and support them, leading to poor team coordination (a demo in Sec. 5. verifies this hypothesis). However, it is counterintuitive that with increasing sight ranges, the agents' performance typically degrades (this phenomenon is detailed in Sec. S.I. We argue that the agents' attention is easily distracted by unrelated scenarios, causing the atlention distraction issue. To reveal the importance of agents' attention on their performance, train the SOTA algorithm REFIL (Iqbal al. 202T) different sight ranges (3,9, and oo) and visualize the agents' attention weights on all entities (i.e., the value of the matrix QK T in MHA module) in Fig. 3 can be seen that when more entities are visible, the agents' attentions get more dispersed so that more difficulties need to be overcome to win the preset AI. Existing dynamic team MARL methods commonly face such problem (see Appendix E).
To deal with the dilemma of the dynamic partial observability in MARL, we resort to a divide- and-conquer learning strategy, dubbed CAMA, for stable performance and sustainable teamwork. Concretely, in low sight ranges we need to improve agents' attention realizing the potential coop- erators by -iggt----agte information, while large sight ranges, agents' attention concentration on exccution-related entities should be kept.
Framework and RL Training. CAMA mainly consists of three components, including an Entity- Dividing Module (EDM) for dynamic partial observability control, an Attention Enhancement Module (AEM) for attention concentration on execution-relatee Module (ARM) for agents' out of sight range coordination, Fig. 2 Specifically, for each agent i, EDM divides and embeds the raw entities as f and f to feed AEM and ARM respectively. For AEM, an inverse model is applied resolve the attention distraction issue. For ARM, a coach with global
entities, and an Attention Replenishment
4

---

Under review as a conference paper at ICLR 2023
sights is introduced to generate communication message Ç for agents' coordination. Following Yuan et al. (2022), we generate a Q((locat) from ft and agent i's observation-action history p² (which is the output of gated recurrent unit (GRU) cell), and Q'(slobutt from Ç² Q' is computed by their summation, i.e., Q' Q'(iocal) + Qi(olobat), All local Q's are fed into a mixing network (Yang et al. 2020 to calculate a Qtot, The RL loss can be formulated as follows:
[(
+ max Stat (P)+1, (po) )
for E Bl
(3)
where Qtot is the target network, and D the replay buffer.
4.2 ENTITY DIVIDING MODULE
An EDM divides raw entities into an attention enhancement part and an attention replenishment part by their ranking of attention weights. For the former one, we wish to constrain the maximum of observed entities to avoid the attention distraction. And the latter one should contain enough out- of-sight-range information for team coordination. We first deal with the attention enhancement part. Recall that in MHA module, . {0, x is binary mask applied on the entity embeddings which is generated by the environment. To uphold each agent's 'sts partial observability, a more sparse mask M, is introduced to replace M which satisfies the following constraints:
|, S ane, -M, ① -M -M,
where ne is the maximum number of entities, a E (0, 1] is a hyper- and O means the llement-wise multiplication operation. The negation of M is defined as -M := 1 M, where 1 is an all matrix with the same shape as M. The left side of Eq. 4 ensures that the percent of observable entities is less than a, while the right side makes the agent observe the entities available in the original mask M only. We can assign a low value for a (e.g.. 0.4) to limit each agent's visible entities in complicated environments. To get M,, we define (W) operation get the indices of the top i values in each row of W. W. can be calculated as follows:
M, = M O M, = M, [others] 0, = ==B[ann, B|an, [anm](QQ]]] (QKT),
(5)
where [-] means to round down the value, and Q. K are matrices of queries and keys in MHA module, respectively. Under Eq. (5). each agent only remains the sight on at most |ane] entities with the highest attention weight, and improves its attention concentration. We prove in Appendix A that M, obtained by Eq. (5) is adequate for Eq. (44. After getting Ms. we can compute f' for the attention enhancement part, which the output of MHA(L, X, M....
Since the attention replenishment part should contain all the information not involved in the former one for better out-of-sight sight--raaege coordination, we transmit it with f-i which the embedding of the complement entities of M, on with the same MHA module as MHA(I, -MM).
4.3 ATTENTION ENHANCEMENT FOR LOCAL AGENT
When knowing what will happen when a specific action is taken, the learned agents can hardly be distracted. Thereby, we aim to concentrate agents' attention on exccution- distilled from the high-dimensional state space. Specifically, we resort to the inverse model (Pathak et al. 2017). two-layer MLP, that uses the local observation of and of to predict the agent i's action a'. In the prediction module, the local observation of and of are first fed into the same EDM to get the features ft and Si+1- Then, the probability of each action can be predicted as p(at) IM The learning loss can be defined as:
information
LiM CE(p(at), of).
6
where CE means the cross entropy. fs will contain the necessary information for predicting a' by optimizing Eq. (6). which encourages EDMs to discard irrelevant distracting information for the consciousness enhancement embedding f With the auxiliary representation learning task, the learned embedding I can be used to calculate each agent's local Q function ("(local).
5

---

Under review as a conference paper at ICLR 2023
4.4 ATTENTION REPLENISHMENT BY GLOBAL COACH
To equip agents with the ability of out-of-sight-range coordination, we use a centralized coach equipped with global states to generate a message Ç from f and compute a Q'(gtobal) for each agent i each time step (Liu et al. 202T; Niu et al. 2021 Mao et al., (000b). The message plays the role of attention replenishment when agents facing difficulty to cooperate through their local observations. The learning objective for ( should contain the information unknown to agent i, and not distract the agent's 's attention. Therefore, we maximize:
I(C": - - (1- 3)](( I(C's(
where is the global state, and Z(:;-) means the mutual information. f indicates the replenishment information for agent We leave the derivation of Eq. (7) Appendix B Maximizing I((": (") let Ç* be the summary of f- By feeding (* to agent the agent can sense the information beyond its sight range, and further alleviate the difficulties of cooperation caused by partial observability. Minimizing ;(5*;s) s) compresses the information Ç has, which can be regarded as information bottleneck constraint on Ç* /Wang et al. 2020bj. We use hyper--arammttrr /3 E [0, 1| to control the compression degree of Ç*. Combining the two terms, we (1) discard the information in f", which is already known to agent i, and (2) compress the sophisticated f - into a brief message, which can hardly distract the agent while promotes coordination.
We then separately optimize the two mutual information term in the right side of Eq. 7 Directly maximizing I(("; s) is difficult, but there exist some tools estimating its differentiable lower bound, e.g. infoNCE (Oord et al. (0188) and MINE (Belghazi et al. 2018). We choose the CatGen formula- tion (Fischer 2020) of the former one and maximize the following lower bound of :({:s))
p(('|s)
INCE(S:; E [log
(8)
1 K R P((1s()))
where is the sample number of a mini-batch.
Then we move minimizing f') There are existing tools that minimize the upper bound of the mutual information between two random variables, such as CLUB (Cheng et al. 2020) and LIOut (Poole al., 2019). But these methods can not be directly applied to the conditional mutual information paradigm. Therefore, we extend the CLUB estimator to the conditional form and present Conditional-CLA (CC) estimator differentiable upper bound of f
Ioo(S'.'i1 Bes. P(C' [- Dog
Theorem 4.1. For three random variables Ç², f and f-;
Icc(S' 2
(10)
The equality holds and only if f is independent of the joint distribution of f
The proof of Thm. T.m. [.I can be referred in Appendix C Accordingly, we can minimize the conditional mutual information via minimizing Icg. In practice, assuming we have the conditional distribution
(f we can sample pairs [(Ck fir ))
)) and get an unbiased estimation of Ice:
1 K
Icc-
log pcp F
log p(t(,, k=1
(11)
K2
Since the second part of Eq. faster counterpart:
requires O 2 computational complexity, we use the following
K
Ice
log pesp(h P Si.
1
Σ log p Pl fE");
(12)
K
where ua()) is mapping from A E {1,.., K) to its random permutation. Since Ice and Ice are both unbiased estimators, B[fccl B[īcel Icc. In practice, we assume ff distribution. We use a neural network with input concat(f" f f-1 to calculate its mean and variance and optimize it with the reparameterization trick (Kingma & Welling 2013). Since con
Gaussian

---

Under review as a conference paper at ICLR 2023
has involved all the information in the global state 8, use the same distribution to represent p(( in INCE and therefore optimizing Las = Icc - (1 3)INCE. If we need a coach with stronger capability by e.g., introducing the memory module, we will get the prior distribution P(('lp) instead of p(5") [ )) therefore we can not estimate Ice directly. In Appendix D we introduce a variational distribution to estimate ( when it is unknown.
EXPERIMENTS
5.1
DILEMMA IN SIGHT RANGES
30
Oatten REPIL
Move direction
MALC
CAJSA
2.6
2-4
Agents
Apple
of
Obstacles
12:35
@Hom
Finta
Timl
(a)
(b)
(c)
(d)
Figure 4: (a) A sketch of environment "Resource Collection" (b) A sketch of the demo "Catch Apple".
(c) Task sovling rate "Catch Apple". Qatten and REFIL are CTDE, and MAIC and CAMA have a
centralized coach. (d) Visualization of messages received (grouped by the next action agents take).
Coordination in Low Sight Ranges. We first show the defect of CTDE methods with a demo. In a 11 11 grid-world, an apple is uniformly generated one grid of the 3 grids in the map center, with obstacles of length 4 at its right&left or up&down, Fig. 4b) 2 controllable agents are initialized at random places. They can move towards 4 directions, one grid at time. The goal that two agents should arrive the apple simultaneously, rewarded 10. Any agent touching the apple or team reaching the episode time limit will end the game. The reward of one agent touching apple for time penalty is 1..1. The agents can only see the entities in the 3 3 grids, each centered on its position. Agents can not overlap, which means the only way to achieve the team goal is the two agents moving towards the apple from the opposite sides of the obstacles, at which time they can not see each other. We train two CTDE methods: Qatten (Yang et al.) (020) and REFIL (Iqbal et al. 2021), and two communication- bbasdd methods: global coach, see Sec. 5.c. details). The target achieving rate during training are shown in F Cc.. Only methods with communication can solve the task while CTDE methods fail completely.
The reward
CAMA and MAIC Yuan et al. 2022) (with
To validate the effect of communication, we use t-SNE ( (Van der Maaten & Hinton| 2008) to visualize the messages Ç from 160 testing episodes of method in 3D-space. We show the results of the last time step of each episode in Fig. d). Each point is a message and the color means the agent's s action after it receives the message. The results show that agents' actions have obvious correlation with messages received, which shows the importance of communication in out-of-sight coordination.
52w033 5
HACK
511-10
Ordere
LIM LMi SR = 0.5
SR = 1
100
260.47+37.45 $39,88+21.4 1,44
343.73+52.26 393. 193.14423.01
280.36427.96 993.93432.1
385.25+25.61 458, 188.13446.93
18
30
Table 1: Test returns on sight ranges (SR) on RC. "/" means the corresponding loss is added.
1198
Figure 5: Test returns of 3 sight ranges (SR) on RC.
Attention Distraction in Large Sight Ranges. We use the Resource Collection (RC) task (Fig. (a)) to show the attention concentration effect of CAMA. In this task, group of 3-8 agents coordinate to collection resources from various places and bring them home. is slightly modified from Liu et al. (202)) make it suitable for both CTDE and communication-based settings. Please refer to
7

---

Under review as a conference paper at ICLR 2023
Appendix G.2 for detailed description. We test the performance of the following methods three
sight ranges in Fig. | (1) 2 CTDE methods: Qatten (Yagg et al.
2020). REFIL (Igbal et al. (021).
(2) 1 graph communication method: EMP (Agarwal et | 2019 methods: MAGIC (Niu et al., 2021), Lated__atten (Mao et al. 2020b), COPA (Liu et al. 2021).
(3) centralized communication
and our CAMA. MAGIC and Gated_ Gated_Qatten
modified make them suitable for the entity-wiss input setting. The results show that CAMA outperforms all baselines in all sight range settings, And
only CAMA performs better in SR 1.5 than SR 1.0 and SR 0.5, while others suffer from attention distraction and perform worse in larger sight range.
5.2 COMPONENT ANALYSIS
Entity Dividing Module. In EDM, the parameter a plays the role balancing the observability between AEM and ARM. We explore how to choose in different sight ranges in Fig. 6 To make the error bars (std) more clear, points with the same a are slightly offset on the X axis. Note that a = 1.0 means constraint observability function, i.e., deleting EDM. We find that the sight range increases, the agents can see more entities, and therefore a lower a can help attention concentration.
8
11-TJ
21-TJ
RC-SR=1
RC-SR=0.5
0.1
30.89+2.54
95.5445.73
389.47441.8
366215.35
0.3
29.9+2.99
99.64:4.45
398.74468.33
379.8+36.
0.5
29.8943.21
95.8543.49
458.13246.93
385.25 61
SRad.
0.7
27.58+1.23
91.32+0.74
373.19+25.43
396:63.19
5A=1
0.9
27.2940.74
91.2844.23
354.06420.24
369.4 169.4252.52
SR-1.
Table 2: The effect of 3 on different tasks: 1 1-lane and 2-lane Traffic Junction, Resource Collection with sight range (SR) and 0.5. From left to right, the task gets harder.
Figure 6: The Effect of a on differ- ent sight ranges (SR).
Attention Enhancement. We analyze the contribution each loss Table LyM (AEM) is important when SR is large, which is reasonable since large SR brings attention distraction issue and requires agents to focus attention on execution-related entities.
Method
MAIC&NDQ
IMAC
COPA
CAMA I(C)f"") ((():))
Apents
Map SR
1 {Z(((aa"(s)
LI((":0')
1 I(Ch:p42,
Sight Range
23.3524.45
19.6740.43
17.3842.92
30.1842.94
1-lane
21.81#3.04
20.75+0.37
20,39
29.89=3.21
66.81+12.33
65.7446.59
73. 16+2.37
96.28=7.64
2-lane
64.6248.81
58.542.27
59,2344.68
95.8543.49
Table 3: Results Trafficc Junction. SR = 0 or means the agents
Figure 7: Traffic Junc- themselves only or can observe the 3 x grids around them. "1" means MI tion teaser (SR=1). maximization and "I" means minimization.
Attention Replenishment. Table shows that CM1 (ARM) can bring obvious improvement under both SR conditions. To check whether the improvement comes from the communication mechanism or mutual information objective, we turn to a grid-world based traffic junction environment (TJ). We the map from (Sukhbaatar al. (2016), which crossroads where generated from a Poisson distribution at one of the four entrances, and aiming for one of the other three exits (Fig.7) 7).. Unlike the original simplified setting that the routes are fixed so that agents only choose to accelerate or brake, we use more difficult setting that agents can move towards four directions freely, which is harder and more realistic. Please refer to Appendix G.2 for environment details. We compare four MI-based message generators under our entity-wise setting: MAIC&NDQ (Yuan et al.) 2022 Wang et al. 2019), IMAC (Wang et al., (020b), COPA (Liu et al.. 202T) and our CAMA. To make fair comparison on MI objective, we only implement the mutual information part of each method and keep the others same to ours, e.g., the structure of neural networks and communication mechanism. Since MAIC and NDQ regard them as one method. The test return results are shown in Table which exhibit the superior performance of our MI objective in limited sight range.
are continuously
similar MI objectives under the centralized coach setting,
8

---

Under review as a conference paper at ICLR 2023
We check the effect of information compression degree 3 in Las with different environment difficulties in Table 2 and find that in simple tasks such as 1-lane TJ, the coach can handle f without obvious information compression. While in hard tasks, e.g., RC with SR 0.5, a large B can simplify communication messages, so that not to distract the agents and bring higher performance.
5.3 HARD TASK SMAC
We test CAMA's performance on the hard SMAC (Samvelyan et al. 2019) tasks with dynamic teams in this section. We use the setting from et al. (022)) and lgbal et al. (2021) that the start of each episode a total of 3-8 agents are randomly divided into 2-4 groups and initialized at different places on the edge of circle with the radius 9, and enemies are divided into 1-2 groups, Fig. aa). Agents have the sight range 9. Agents must learn to find teammates first before fighting against enemies with more quantities. We show the results of test win rate (TWR) on 3 maps in Fig. B)). Our method remarkably exceeds the current SOTA methods on all maps.
12 Slanwry
01
AAgani Btudde
(b)
Figure 8: (a) An initialization teaser on SC2. (b) TWR comparisons on the 3 SC2 maps.
Sight Range Testing. In Fig. 9 (),, with the varying SR comparisons in (3, 6, ) our CAMA
reports superior TWR over SOTA methods on all SR settings and exhibits impressive robustness.
Dynamic Team Composi- tion. In Fig. 9b), we test the same model on differ- ent team sizes and plot the logarithm of the rel- ative winning rate of the corresponding agent num- ber against the total agents, where CAMA shows out- standing robustness to agent variation with nearly sta- tionary performance curve.
100-
Caleen
Agoothea
Agent nun
Agent
66
30
Figure 9: TWR comparisons with different SRs (a) and dynamic team composition (b). The x axis is the agent number which varies from 3-8. The y axis log(w/@), where the winning rate and agents' average winning rate. The black dotted line denotes w.
6 CONCLUSION
In this paper, we explore the dilemma of the partial observability in MARL. With severe partial observability, agents usually can sense the teammates and show poor team coordination. While with large sight range, agents are troubled by the attention distraction issue and exhibit apparent performance degradation. To tackle such sight range dilemma, we propose a Concentrated Attention for Multi-Agent reinforcement learning (CAMA) approach. First an Entity Dividing Module is used to divide raw entities for the local agents and the global coach separately. The suitable division percent varies due to the agents partial observability. For local agents, an Attention Enhancement Module improves their attention on execution-relle related entities under the condition of large sight ranges. For the global coach, the messages generated by novel conditional mutual information estimator replenish the information required for team coordination in all sight ranges. We evaluate our method on three commonly used MARL benchmarks: StarCraftII, MPE, and Traffic Junction. With raised agents' conseiousness, the proposed CAMA reports significantly superior performance compared with SOTA methods and exhibits remarkable robustness sight range variation and dynamic team composition.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Akshat Agarwal, Sumit Kumar, and Katia Sycara. Learning transferable cooperative behavior in multi-agent teams. arXiv preprint (rxxiv/906.0/2000 2019.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint 2019.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference machine learning, pp. 531-540. PMLR, 2018.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1:XX:::912.006680 2019.
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound mutual information. In International conference on machine learning, pp. 1779-1788. PMLR, 2020.
Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. In AAAI Conference on Artificial Intelligence (AAAI), 1998.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning, pp. 1538-1546. PMLR, 2019.
Ian Fischer. The conditional entropy bottleneck. Entropy, 22(9):999, 2020.
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep agent reinforcement learning. In Advances neural information processing systems (NeurIPS), 2016.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2017.
Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet:
Universal multi-agent rl via policy decoupling with transformers. In International Conference on Learning Representations (ICIR), 2020.
Shariq Iqbal, Christian A Schroeder De Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2021.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference Machine Learning (ICML), 2019.
Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforcement learning. arXiv preprint
2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.61 14, 2013.
arXiv preprint
Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. arXiv preprint errxiv/9977 1800, 2019.
10

---

Under review as a conference paper at ICLR 2023
Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, and Animashree Anandkumar. Coach- player multi-agent reinforcement learning for dynamic team composition. arXiv preprint S0N:::200500869, 2021.
Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. Multi-agent game abstraction via graph attention neural network. In AAAI Conference on Artificial Intelligence (AAAI), 2020.
Qian Long, Zihan Zhou, Abhibav Gupta, Fei Fang, Yi Wu, and Xiaolong Wang. Evolutionary popula- tion curriculum for scaling multi-agent reinforcement learning. arXiv preprint 20Xi::2003.10423, 2020.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Picter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cnoperative-oompettivi environments. In Advances in neural information processing systems (NeurIPS), 2017.
Hangyu Mao, Wulong Liu, Jianye Hao, Jun Luo, Dong Li, Zhengchao Zhang, Jun Wang, and Zhen Xiao. Neighborhood cognition consistent multi-agent reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7219-7226, 2020a.
Hangyu Mao, Zhengchao Zhang. Zhen Xiao, Zhibo Gong, and Yan Ni. Learning agent communication under limited bandwidth by message pruning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3142-5149, 2020b.
Yaru Niu, Rohan R Paleja, and Matthew C Gombolay. Multi-agent graph-attention communication and teaming. In AAMAS, pp. 964-973, 2021.
Frans A Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.0338, 2018.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driver exploration by self-supervised prediction. In International Conference on Machine Learning (ICML), 2017.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference Machine Learning, pp. 5171-5180.
PMLR, 2019.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv:1803. 10000.114880 1485, 2018.
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation. arXiv e-prints, pp. arXiv-2006, 2020.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung. Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: 190...40220,,4, 2019.
Christian Schroeder de Witt, Jakob Foerster, Gregory Farquhar, Philip Torr, Wendelin Boehmer, and Shimon Whiteson. Multi-agent common knowledge reinforcement learning. Advances in Neural Information Processing Systems, 32:9927-9939, 2019.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. arXiv preprint 18 2018.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Farl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning, In International Conference on Machine Learning (ICML), 2019.
Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation. In Advances in neural information processing systems (NeurIPS), 2016.
11

---

Under review as a conference paper at ICLR 2023
Peter Sunehag, Guy Lever, Audrumas Gruslys, Wojeiech Marian Czarnecki, Vinícius Flores Zam- baldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning based on team reward. In Interna- tional Conference on Autonomous Agents and MultiAgent Systems (AAMAS), 2018.
Wesley Suttle, Zhuoran Yang, Kaiqing Zhang, Zhaoran Wang, Tamer Başar, and Ji Liu. A multi-agent off-policy actor-critic algorithm for distributed reinforcement learning. PaAC-SaaprsO,nLine, 53(2): 1549-1554, 2020.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In International Conference on Machine Learning (ICML), 1993.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems (NeurIPS), 2017.
Lipeng Wan, Zeyang Liu, Xingyu Chen, Han Wang, and Xuguang Lan. Greedy-based value representation for optimal coordination in multi-agent reinforcement learning, arXiv preprint arXiv::112.00445, 2021.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint 2xxi:20088.00062 2020a.
Rundong Wang, Xu He, Runsheng Yu, Wei Qiu, Bo An, and Zinovi Rabinovich. Learning efficient multi-agent communication: An information bottleneck approach. In International Conference on Machine Learning, pp. 9908-9918. PMLR, 2020b.
Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly decomposable value functions via communication minimization. arXiv preprint arXiv:1910.05366 2019.
Weixun Wang, Tianpei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. From few to more: Large-scale dynamic multiagent curriculum learning. In AAAI Conference on Artificial Intelligence (AAAI), 2020c.
Zhao Xu, Yang Lyu, Quan Pan, Jinwen Hu, Chunhui Zhao, and Shuai Liu. Multi-vehicle flocking control with deep deterministic policy gradient method. In International Conference on Control and Automation (ICCA), 2018.
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03939, 2020.
Lei Yuan, Jianhao Wang, Fuxiang Zhang, Chenghe Wang, Zongzhang Zhang, Yang Yu, and Chongjie Zhang. Multi-agent incentive communication via decentralized teammate modeling. 2022.
Chongjie Zhang and Victor Lesser. Coordinating multi-agent reinforcement learning with limited communication. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2013.
Kaiqing Zhang, Zhuoran Yang. Han Liu, Tong Zhang, and Tamer Basar. Fully decentralized multi- agent reinforcement learning with networked agents. In International Conference Machine Learning (ICML), 2018.
Meng Zhou, Ziyu Liu, Pengwei Sui, Yixuan Li, and Yuk Ying Chung. Learning implicit credit assignment for multi-agent actor-critic, arXiv preprint arXiv:2007.02529 2020.
12

---

Under review as a conference paper at ICLR 2023
A PROOF OF THE MASK GENERATOR
We now prove that M, got by Eq. ( is sufficient for Eq. (4)
We first show that M2 in Eq. (5) satisfies ||M I am,. Recall the definition of M, it is indicator matrix, that at each row, only the positions top [am,] values of the corresponding row in the attention weight matrix are 1, and others are O. Therefore, at each row of M,, there are at most [ 1s, which means the sum of the absolute value in each row no larger than [one]- So ||M [ane] I afge Then show ||Mall 2o Σ ang- Since the observability mask M is also a 0,1 mask, Msllc = M @ Mylla S MyMlls & Offe-
Then show that -M, - M,, which equals to (M,, O M) -M = MM. Recall that M and M are both 0,1 matrices with the same shape. For any position in the matrices, we can use Table to conclude all the situations:
M
My (M&&MM)s M
-M
0
1
1
0)
0
1
0
0
Table 4: Logical table of the mask.
We can find that at any situation (M, O M) C -M -M, therefore -M, o - = -MM In summary, M, got by Eq. (5) is sufficient for Eq.
B DERIVATION OF MUTUAL INFORMATION OBJECTIVE
We now give the derivation of Eq 7
(
-H(S') - -(8))))-((()))))))))
=(1 (1-)))((())))++++ ((())
(13)
=(1 -
H((1)s)) - =(1 B)I((';8) I(C
//plit into I' and f
C PROOF OF THEOREM
For three random variables ,
fr and f
IcolS", /' 2 /')")))))))
(14)
The equality holds if and only if f independent of the joint distribution of f
Proof. Let A be the gap between Icc(St, and M()+))))))
A --cloa /" -(C,
(ec.s..-=((ogp(((1(1 - Be. (-
(15)
-E - - Bo
- ( (,,
Since log(:) is concave function, A > 0 due Jensen's Inequality.
13

---

Under review as a conference paper at ICLR 2023
D GENERATE MESSAGES WITH MEMORY
In Sec. 4.4, we use ( to generate the message S². It means we have the true prior distribution p( f), which can be expediently applied to estimate Icc. However, if we wish to enhance the coach's ability, e.g., introducing memory structure (similar to the GRU cell in local agents) into the message generator, the prior distribution turns into p(C'lp) and therefore we can not obtain p( (+) directly to estimate Icc- Similar the idea in Cheng et al. (2020), propose a variational term " " estimate p(S')
TocolS' 5 R (lo ((5" [ - RR Bet Dogg ,-))
(16)
Theorem gives sufficient condition ensure ( be an upper bound of I(S', f ): Theorem 1. Denote
- =
K
(17)
then Is (S' 2 I(S', (l The equality holds when I and the join distribution S are independent.
Proof. Let Å be the gap between Loco(s f and f(f-))))))) We have:
(Bgp -
(18)
((5" Ee Ee p(
if
Bey flog
(((
[
-E [log M(S" )p( (p(s))(())))))))))))))))))))))
Ecs flog
((
,', Egg pls' pls'fi..
= (LL(p(F))(s""
" - KK(p(('
Theorem reveals that Co an MI upper bound if the variational joint distribution (( is more "closer" to p(C f³, f-) than to ((( Let dg be the parameterization of q. In addition to the Eq. 16 that optimizes I,cc, we should minimize the KL divergence between p(S', f f and (5 f
min KL(P( ,
-min Bet flog no (,
(19)
min Re Re.r.-- . log (C (C)))
= max Re log ('
14

---

Under review as a conference paper at ICLR 2023
Therefore, with sample pairs ((St fhi fo () we can maximize the log- ligelikei of qa:
K
max
Σ
log 2 (Shlf
(20)
K
With enough optimization times of Eq. 20 Loce is guaranteed to be an MI upper bound.
We test the effect of coach with memory in the Resource Collection environment with SR 0.5 and SR = 1.0, Table We find that although equip the coach with the memory module improves the average performance, it brings large variance that the method becomes unstable. Therefore, to keep stable performance of our method, we still use the coach with MLP in the main paper.
Coach Style
Known Prior
SR=0.5
SR=1.0
MLP RNN
pf f
385.25425.61 419.21=177.99
458.13+46.90 533.54191.78
p))
Table 5: Comparison on the style of coach.
E HEAT MAPS OF ATTENTION WEIGHTS
E. AVERAGE HEAT MAPS OF MORE METHIOSS
As mentioned in Sec. 4.1
we visualize the attention weights of four methods in Fig. TO Only our
CAMA pay more attention to the agents themselves when the sight range is large, and therefore keeps a high performance.
WR-34%
(a) Qatten
3R-J. 51%
SR-3, WR-31%
SR WR-21%
(b) COPA
SR-9 WR+53%
5R.
43
(c) REFIL
SR=9. WR=71%
Sic WR=65%
(d) CAMA (Ours)
Figure 10: Attention Weights of Qatten, COPA, REFIL and our CAMA.
15

---

Under review as a conference paper at ICLR 2023
E.2
HEAT MAPS AT SINGLE TIME STEPS
CAMA,SR=3,WR-719%
CAMA,SR= WR-666
REFIL,SR-3,WR-719
REFIL,SR= WR=51%
Figure 11: Attention Weights on single time steps of REFIL and CAMA.
We also visualize the heat maps of our method and REFIL at single time steps. We choose the first attention head of each saved model, and the color bar's range attention weight [0,0.35]. We visualize in Fig. I1 both methods' attention weights on all agents from = 0 to 24, at which time all the agents are usually alive. We find that the large sight range distracts the attention of REFIL, while our method CAMA keeps attention concentration.
F HYPERPARAMETERS
We summarize the hyperparameters in Table. 6
G ENVIRONMENT DETAILS
G.1 RESOURCE COLLECTION
On map of [-1,1], agents are initialized at random places with team size sampled from [3,8]. They need to collect resources from resource points and transport the goods home. The locations of resource points and home are randomly sampled from the whole map at the start of each episode.
16

---

Under review as a conference paper at ICLR 2023
Name
Description
Value
Discounted factor
0.99
anneal time
Time-steps for to anneal from 6, e-. y. is the probability for
500000
agents choosing random actions.
Start
f
Final E
0.05
Nene
The number of parallel environments
8
'D]
Replay buffer size
5000
Thead
Number of heads in multi-head attention
Datta
Dimension of Attention embedding in local agent
128
mma
Dimension of Attention embedding mixer
64
Bran
Dimension of RNN cells Learning rate
64
Ir
0.0005
Default attention concentration rate
0.8
B
Default communication message compression rate a value in RMSprop
0.5
ORMS
0.99
value in RMSprop
0.00001
ubotch
Batch size Time interval for updating the target network
32
ttorget
200
A
Weight for LyM
0.005
X2
Weight for LMI
0.1
Gmaz
Clipping value for all gradients
10
Table 6: Hyper-parameters.
The radius of the home and the resource location & agent is 0.1 and 0.05, respectively. There are kinds of resources, and each agent has own ability bf uniformly sampled from {0.1, 0.5, 0.9) to collect each kind of resource Each agent can accelerate towards 4 directions or apply no forces at each time-step. Each agent has its maximal speed uniformly sampled from (0.3, 0.5, 0.7) and the accelcration is fixed to 3.0. Every time the agent i collects resource the team will get reward 10 * b9. When an agent brings the resource home, the team will get a reward An agent can only carry one resource starts to collect the next one. The episode limit is 145. The number of the agents for training is uniformly sampled from (2.3,4.5), while for testing it is sampled from (6,7.8). Each agent has sight range 5 SR. Entities including other agents and resource points that exceed agent i's $ R are invisible agent i.
time, which means agent needs to bring the collected resource home before
G.2 TRAFFIC JUNCTION
The simulated traffic junction environment from Sukhbaatar et al. (2016) has been a conventional and useful testbed for testing the performance of multi-agent communication algorithms Singh et al. (2018); Das et al. (2019): Liu et al.] (2020). Despite its great success, cars in the original traffic junction environment can only move along pre-assigned routes on one or more road junctions and so the action space for each car only consists of two actions, i.c. gas and brake, which restricts its ability simulate real-world environments and test communication algorithms. Moreover, the original observation is not fit for the need of entity-wise input.
So, we modify the original traffic junction environment to a more flexible version. In stead pre- assigned routes, a random selected navigation target is assigned to each active car and the aim of each active car is navigating its own target and avoiding collision which occurs when two cars are on same location. Along with the modified setting, the observation for each active car is altered to entity-wise form which contains the position cars in limited visibility (e.g. x 3 for sight_range range I) and the action space is more flexible with five actions i.e. forward, buck, left, right and wait. Besides, the rewards consists of a linear time penalty -0.017, where 7 is the number of active time steps for the since the last resurrection, collision penalty Tcollision Manhattan distance from the target between the previous and the current step We choose two different difficulty levels following the settings in Singh al. (20T8), illustrated in Fig 12 Moreover,
10 and the difference
17

---

Under review as a conference paper at ICLR 2023
Map
Nmaz
max_steps
Parrive start
Parrine end
entrances
targets
I-lane
20
0.1
0.3
4
2-lane
10
40
0.05
0.2
8
8
Table 7: Detailed environment parameters in two versions of Traffic Junction Environment. mas_sleps the length of each episode. entrances largels indicate the number of choices of entrance target and in the hard version the cars are restricted to keep to the right when entering or exiting, so there are only 8 entrances or targets.
the total number of cars is fixed at maz and the cars will be turned inactive when reaching the target or colliding with others and new cars get added to the environment with probability Parrise at every time-step which varies during training consistent with the curriculum learning in [Singh et al. (2018) The detailed environment parameters settings are indicated in Table
(a) 1-lane
(b) 2-lane
Figure 12: Traffic Junction Environment. Cars are navigating to their own assigned target and avoiding collision. There are two difficulty levels.
H HARDWARE
We ran experiments on GPU servers, with each one having 8*RTX3090TI GPUS and 2* AMD EPYC 7H12 CPUs. Each experiment (one seed) takes 12-24 hours one GPU.
I GENERALIZABILITY
We test the generalizability of our method in the environment Resource Collection. We train each method for 10" time steps the environment with 2-5 agents. Every 10 time steps, the model is evaluated on the environment with 6-8 agents for 160 episodes, and we plot the test return curves in Fig. 13 The results show that our method can generalize well larger team sizes than training, which probably due to the ARM module with the message generator that passes the global information.
18

---

Under review as a conference paper at ICLR 2023
Qatten
REFIL
EMP
COPA
CAMA
SR=0.5
SR=1.0
600
600
400
Return
Return
300
300
200
100
0
2
2
T (mil)
T (mil)
Figure 13: Generalizability the task Resource Collection. Each method is trained on the agent
number 2-5, and tested on agent number 6-8 every 50k time steps to plot the curve.
19


Reviews:
[{"review_id": "q6T_j_TU_E", "reviewer": "Reviewer_g4aW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This papers propose a “consciousness-aware” method to coordinate between multiple RL agents.  The authors state that individual may have attention distraction, and CAMA is good to address the issue to achieve better performance.", "summary_of_the_review": "The paper is well structured and the proposed method and modules appear to be sound.  However, it is really hard to assess the impact of this work based on just a couple of examples.  It is not clear about its data complexity and limitations.  It is also good to thoroughly justify the use of the term “consciousness” and explain why “coordinated” or “supervised” can do a justified job.", "strength_and_weaknesses": "Strengths:\n1. The problem is well stated, and it is a realistic problem to tackle.\n2. The proposed method and its modules are well organized and proved.\n3. The empirical study justifies the claim of SOTA against other methods.\n\nWeaknesses:\n1. The name CAMA uses the term consciousness but it is not clear what this term means in this context.  Consciousness is one of the most misunderstood terms. And when it is used, which parts of the system are unconscious?  \n2. Figures are well drawn and can be presented and explained in greater details. \n3. It is not clear if CAMA requires higher volume of training data? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ixEFMoDNl93", "reviewer": "Reviewer_xWNF", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors provide a method for \"consciousness-aware\" reinforcement learning that consists on several modules that provide attention and cooperation between agents to provide a solution for the varying partial observability. In particular, these modules are defined as \"consciousness-aware\" in an analogy to actions that a \"consciousness-aware\" human being do. However, we can not say that this system  possess awareness, nor phenomenal consciousness, just because of that. Hence, the title of the paper is misleading. It is a good paper but several terms must be changed. I would talk about attention or access consciousness but not awareness. ", "summary_of_the_review": "The main issue of this paper is their use of the words consciousness and awareness. Consciousness is a property that we cannot assign freely to machines without accepting lots of assumptions that can not be validated with the scientific method like multiple realizability and functionalism. Those can be reviewed in a philosophy of mind manual or even studied from a neuroscientific point of view. Even, Roger Penrose provides strong arguments against the emergence of consciousness in machines. Why is this algorithm in particular conscious, or aware, about itself? If it would, it need to catch qualia, and this assumption does not hold for a simple information processing algorithm such as the one present here. One thing is to talk about attention layers, which is valid, and another is just putting the title \"Consciousness-aware multi-agent reinforcement learning\". I can not accept the paper and need to recommend a strong reject, although technically is not that bad, just because of this. This seems to be a great multi-agent reinforcement learning algorithm from an engineering perspective but it is not aware of itself. Please review the literature about philosophy of mind to learn more about this. We can hardly assume that this algorithm would simulate features about access-consciousness, but not awareness which is the same as phenomenal consciousness. Consequently, the title can not include \"consciousness\" nor the name of the modules. \n\nOutside of the discussion about consciousness:\n\nMultiple terms such as varying partial observability vs slight partial observability need to be better defined. \nI find a lack of cohesion between multiple parts of the paper, sections are written in a very straight-forward style. They need to be better introduced. \n\"Consciousness-learning\" does not make sense. \nThe related work section deliberately ignores work on consciousness, which is OK if the paper does not talk about consciousness.\nIt would be interesting to describe an example of observation function.\nFormalize the concept of dynamic team. \n\"mulit\" -> multi\nThe experiment section is repeated just with 6 different seeds. I recommend to repeat it using 25 different seeds or more to make the results  statistically more solid. ", "strength_and_weaknesses": "Strengths: Rigorous, style. \nWeaknesses: Mainly the biggest is not knowing what is consciousness.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RENoHvOZCM", "reviewer": "Reviewer_UtXQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an architecture which divides entities into two groups, one of locally relevant entities passed directly to the agent's utility network and another of less relevant entities which are passed into a global coach which compresses global information. This architecture produces improved performance on multi-agent tasks with varying quantities of agents.", "summary_of_the_review": "While the results presented are impressive, the proposed architecture is complex and could use more justification for the individual components. It's not clear to me that the improved results are a result of addressing the problems laid out in the introduction or some other emergent properties of the architecture.", "strength_and_weaknesses": "# Strengths\n* Strong results on a variety of domains. The consistent performance as the number of agents increases shown in figure 9bc is especially impressive. it would be interesting to see if this extends beyond the number of agents seen during training.\n\n# Weaknesses\n* The paper is rife with excessive anthropomorphizing language, down to the title (\"consciousness\"). I would recommend that the authors tone this down.\n* The \"consciousness replenishment module\" is conceptually similar to the coach in COPA [1], though it is presented as a novel contribution with significant exposition dedicated to it. It should be described more in relation to the existing work to emphasize what the differences are.\n* Figure 3 is averaging attention across time steps and (presumably) attention heads, so it would be expected to be dispersed over the whole set of agents and entities (since many varying interactions may happen over the course of an episode). If one were to plot attention at a single time step from a single head and it still looked this dispersed, it would be a more effective argument for what the authors are describing. The reason it looks less dispersed in the SR=3 case is likely that agents spend many time steps in this setting only being able to observe themselves.\n* A thorough ablation study is needed to understand the significance of each component. For example, the loss functions that are unique to each module could be ablated.\n\n# Questions\n* From Figure 2, it seems like the entities in the complement of the selected mask are still used in computing the agent utilities, so why are they described as \"out-of-sight-range\"?\n\n[1] Liu, Bo, et al. \"Coach-player multi-agent reinforcement learning for dynamic team composition.\" International Conference on Machine Learning. PMLR, 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "41_flvreKX7", "reviewer": "Reviewer_zvuV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a solution for improving performance in multi-agent reinforcement learning. The term \"consciousness\" in this work essentially refers to attention based mechanisms and to the fact that the authors adopt and approach that resides in the idea of dividing the general problem into sub-parts in order to tackle it. The work is evaluated by means of a series of experiments with testbeds, namely an apple picking task with obstacles, a resource collection task, a traffic junction navigation task and the StarCraft environment. \n\nThe paper is evaluated using a variety of tasks and this is definitely commendable. However, the evaluation does not show the actual contributions of the three key components/mechanism of the proposed solution, namely the entity dividing module, the consciousness enhancement for the local agent and the consciousness replacement.\n\nThe main contribution of the work is essentially in putting together a series of existing mechanisms in order to achieve better performance in the tasks under consideration. \n\n\n", "summary_of_the_review": "The contribution of this work is limited - the proposed solution is essentially a composition of existing components. This is not a problem per se, but the design choices and the actual evidence supporting the need for these components is not satisfactory. The evaluation is not satisfactory since it does not really show the contributions in terms of performance of the different mechanisms. The theoretical foundations of this work are also not completely convincing. The authors include some theoretical results that are not key for the actual contribution of this work. For this reason, the reviewer cannot recommend this work for publication.", "strength_and_weaknesses": "This paper considers the difficult problem of coordination of teams of agents for the execution of a complex task. The main mechanism used by the authors is the \"classic\" divide-and-conquer strategy. The goal is to deal with the problem of \"attention distraction\" in partially observed environments.\n\nThe use of term \"consciousness-aware\" is probably out of place, since the mapping to the concept of consciousness is actually very weak. Moreover, \"consciousness\" itself is about awareness, so the use of the term \"consciousness-aware\" might also be questionable. Some expressions should be really removed from the text such as \"...CAMA empowers agents with purposeful consciousness\". However, the reviewer believes that this is not the most important concern related to this work. One of the major weakness of this work is the fact that it looks like a composition of existing \"components\", which has somehow renamed and adapted. This would be fine, but the actual need for each of them is not supported by sufficient evidence.\n\nThe design of the different components is not completely new. They are essentially mechanisms that are mapped to existing work  in the area, which are correctly cited by the authors. \n\nA key problem that is essentially not tackled in a general way is the decomposition of the general problem into subproblems. The presentation of these mechanisms is very high level in the paper. It is difficult to understand how this is performed in general (and I would say also in the environments considered by the authors).\n\nThe authors present some theoretical results about the \"consciousness replenishment mechanism by local coach\", but they appear rather disjoint from the main text. It is not completely clear why they are relevant for this work. In particular, the relevance of Theorem 4.1 to this specific work is not apparent. In fact, the theorem considers a property of a system of 3 random variables. How is this mapped to the situations and environments under consideration?\n\nThe reviewer would like to suggest the authors to improve their evaluation by showing the actual contributions of the different mechanisms separately (and in composition) from the point of view of the actual performance obtained in the system. Ideally, a discussion about the generalizability of the approach would really contribute to make this work stronger.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject