Action Semantics Network: Considering the Effects of Actions in Multiagent Systems
Authors: Weixun Wang
Tianpei Yang
Yong Liu
Jianye Hao
Xiaotian Hao
Yujing Hu
Yingfeng Chen
Changjie Fan
Yang Gao
Venue:  (2020)
Keywords: multiagent coordination
multiagent learning

Abstract: In multiagent systems (MASs), each agent makes individual decisions but all of them contribute globally to the system evolution. Learning in MASs is difficult since each agent's selection of actions must take place in the presence of other co-learning agents. Moreover, the environmental stochasticity and uncertainties increase exponentially with the increase in the number of agents. Previous works borrow various multiagent coordination mechanisms into deep learning architecture to facilitate multiagent coordination. However, none of them explicitly consider action semantics between agents that different actions have different influences on other agents. In this paper, we propose a novel network architecture, named Action Semantics Network (ASN), that explicitly represents such action semantics between agents. ASN characterizes different actions' influence on other agents using neural networks based on the action semantics between them. ASN can be easily combined with existing deep reinforcement learning (DRL) algorithms to boost their performance. Experimental results on StarCraft II micromanagement and Neural MMO show ASN significantly improves the performance of state-of-the-art DRL approaches compared with several network architectures.

Published as a conference paper at ICLR 2020
ACTION SEMANTICS NETWORK: CONSIDERING THE EFFECTS OF ACTIONS IN MULTIAGENT SYSTEMS
Weixun -aang'Tianpel Yang'; Yong Liu , Jianye Hao H Xiaotian Hao', Yujing Hu², Yingfeng Chen', Changjie Fan', Yang Gao2
(wxwang, ipyang) @tju.edu.cn, lucasliunju@ wmaal.mmammmmmmm (jianye.hao, xiaotianhao) @tju.edu.cn, (huyujing, chenyingfengl, fanchangjie) @ @coor.. www.nnetteeccomm gaoy @ @nju.edu.cn
"College of Intelligence and Computing, Tianjin University
Nanjing University
3 TTiajinn Key Lab of Machine Learning, Tianjin University
" "Noah's ' Ark Lab, Huawei
Nettaase Fuxi Al Lab
ABSTRACT
In multiagent systems (MASs), each agent makes individual decisions but all of them contribute globally to the system evolution. Learning in MASs is difficult since each agent's selection of actions must take place in the presence of other co-learning agents. Moreover, the environmental stochasticity and uncertainties increase exponentially with the increase in the number of agents. Previous work- borrow various multiagent coordination mechanisms into deep learning archi- tecture to facilitate multiagent coordination. However, none of them explicitly consider action semantics between agents that different actions have different in- fluence on other agents. In this paper, we propose novel network architecture, named Action Semanties Network (ASN), that explicitly represents such action semantics between agents. ASN characterizes different actions' influence on other agents using neural networks based on the action semantics between them. ASN can be easily combined with existing deep reinforcement learning (DRL) algo- rithms to boost their performance. Experimental results on StarCraft II microman- agement and Neural MMO show ASN significantly improves the performance of state-of-the f--thee-rt art DRL approaches compared with several network architectures.
INTRODUCTION
Deep reinforcement learning (DRL) (Sutton & Barto 2018) has achieved a lot of success at finding
optimal policies to address single-agent complex tasks (Mnih et al. 2015 Lillierap et al. 2016
Silver et al. 2017). However, there also exist a lot of challenges in multiagent systems (MASs) since
agents' behaviors are influenced by each other and the environment exhibits more stochasticity and
uncertainties (Claus & Boutilier 1998 Hu & Wellman 1998 Bu et al. 2008 Hauwere et al. 2016).
Recently, a number of deep multiagent reinforcement learning (MARL) approaches have been pro- posed to address complex multiagent problems, e.g., coordination of robot swarm systems (Sosic al. 2017) and autonomous cars (Oh et al., 2013). One major class of works incorporates various multaggnt coordination mechanisms into deep multiagent learning architecture {Lowe et al.) [017]]
Foerster et al., 2018: Yang et al., 2018 Palmer al. 2018). Lowe et al.) (2017) proposed a cen- tralized actor-critic architecture to address the partial observability in MASS. They also incorporate the idea of joint action learner (JAL) (Littman| 1994) to facilitate multiagent coordination. L.tee,,
Foerster et al. (2018) proposed Counterfactual MultiAgent Policy Gradients (COMA) motivated from the difference reward mechanism (Wolpert & Tumer 2001) to address the challenges multi- agent credit assignment. Recently, Yang et al. (2018) proposed applying mean-fiell theory (Stanley) 197I) to solve large-scale multiagent learning problems. More recently, [Plmer et al. (2018) extend- ed the idea of leniency (Potter & Jong, 1994, Panait et al., (2088) to deep MARL and proposed the retroactive temperature decay schedule to address stochastic rewards problems. However, all these
"Equal contribution, corresponding author

---

Published as a conference paper at ICLR 2020
works ignore the natural property of the action influence between agents, which aim to exploit to facilitate multiagent coordination.
Another class of works focus specific network structure design to address multiagent learning problems (Sunchag et al., 2018 Rashid et al. 2018) Sukhbaatar et al. 2016 Singh et al. 2019). Sunehag et al. (2018) designed
value-decomposition network (VDN) to learn optimal linear value decomposition from the team reward signal based on the assumption that the joint action- value function for the system can be additívely decomposed into value functions across agents. Later, Rashid et al. (2018) relaxed the linear assumption in VDN by assuming that the Q-values of individual agents and the global one are also monotonic, and proposed QMIX employing network as a complex non- non-lineat linear combination of per-agen values. Recently, Zambaldi et al. (2019) proposed the relational deep RI. to learn environmental entities relations. However, they considered the entity relations on the pixel-level of raw visual data, which ignores the natural property of the influence of actions between agents. Tacchetti et al.] (2019) proposed novel network architecture called Relational Forward Model (RFM) for predictive modeling in multiagent learning. RFM takes semantic description of the state of an environment as input, and outputs either action prediction for each agent or prediction of the cumulative reward of an episode. However, RFM does not consider from the perspective of the influence of each action on other agents. OpenAT designed network structures to address multiagent learning problems in famous Multiplayer Online Battle Arena (MOBA), Dota2. They used a scaled-up version of PPO (Schulman al. (2017)) )) adopted the attention mechanism compute the weight of choosing the larget unit, with some of information selected from all information as input. However, this selection not considered from the influence of each action on other agents. There are also a number of
that estimates joint action-value
works designing network structures for multiagent communication (Sukhbaatar et al. 2016; (Singh ct al. 2019).
However, none of the above works explicitly leverage the fact that an agent's 's different actions may have different impacts on other agents, which a natural property in MASs and should be considered in the decision-making process. In multiagent settings, each agent's action set can be naturally divided into two types: one type containing actions that affect environmental information or its private properties and the other type containing actions that directly influence other agents (i.e., their private properties). Intuitively, the estimation of performing actions with different types should be evaluated separately by explicitly considering different information. We refer to the property that different actions may have different impacts on other agents as action semantics. We can leverage the action semanties information to improve agent's ' policy/Q network design toward more efficient multiagent learning.
To this end, propose novel network architecture, named Action Semantics Network (ASN) to characterize such action semantics for more efficient multiagent coordination. The main contribu- tions of this paper can be summarized as follows: 1) to the best of our knowledge, we are the first to explicitly consider action semantics and design novel network to extract it to facilitate learn- ing in MASs; 2) ASN can be easily combined with existing DRL algorithms boost its learning performance; 3) experimental results"] on StarCraft II micromanagement (Samvelyan et al., 2019) and Neural MMO (Suarez et al. (219 show our ASN leads to better performance compared with state-of-the-aar approaches in terms of both convergence speed and final performance.
2 BACKGROUND
Stochastic games (SGs) (Littman 1994) are natural multiagent extension of Markov decision pro- cesses (MDPs), which models the dynamic interactions among multiple agents. Considering the fact that agents may not have access to the complete environmental information, we follow previous work's settings and model the multiagent learning problems as partially observable stochastic games (POSGs) (Hansen al. 2004).
A partially observable stochastic game (POSG) is defined as tuple S,S,A' A.,...RR......
AA"A
R". O¹ ..... O"), where N' is the set of agents; S is the set of states; A is the set of actions available to agent (the joint action space A = A' x A2 x ... x A²); T is the transition function
"More details can be found at htttss:/siites .ooglg..9ggg..o www..oww/.w..ee.eeee rasn the source code is put on http:: ccm/ MAS- anssannoy/AAa ASN
2

---

Published as a conference paper at ICLR 2020
that defines transition probabilities between global states: S x A S - [0. 1]: R2 is the reward function for agent i: S A - R and Oi is the set of observations for agent 2.
Note that a state E S describes the environmental information and the possible configurations of all agents, while each agent draws a private observation o' correlated with the state: S + O² c.g., an agent's observation includes the agent's 's private information and the relative distance between itself and other agents. Formally, an observation of agent at step t can be constructed as follows:
2.r-1 of oztt! ,tt ")"",, ), where Of Linv is the observed environmental information, my is the private property of agent i (e.g., in robotics, mi includes agent i's location, the battery power and the healthy status of each component) and the rest are the observations of agent& i on other agents (e.g., in robotics, o, pi--- includes the relative location, the exterior of agent that agent observes). An policy πş: O' x AA - [0; 1] specifies the probability distribution over the action space of agent i. The goal of agent is to learn policy тr that maximizes the expected return with a discount factor y/: J Ex, E 7/rl]
mi: Of 5.1 ,.... of of = =of ,
3 THE ACTION SEMANTICS NETWORK ARCHITECTURE
3.1 MOTIVATION
In MASs, multiple agents interact with the environment simultaneously which increases the envi- ronmental stochasticity and uncertainties, making it difficult to learn consistent globally optimal policy for each agent. A number of Deep Multiagent Reinforcement Learning (MARL) approaches have been proposed to address such complex problems in MASS by either incorporating various mul- tiagent coordination mechanisms into deep multiagent learning architecture (Foerster et al. 2018 Yang et al. 2018; 0018: Palmer et al. 2018) or designing specialized network structures to facilitate mul- liagent learning (Sunehag al. 2018 Rashid al. 2018 Sukhbaatar et al.) 2016). However, none of them explicitly consider extracting action semantics, which we believe is critical factor that we can leverage to facilitate coordination in multiagent settings. Specifically, each agent's ' action set can be naturally classified into two types: one type containing actions that directly affect environmental information or its private properties and the other type of actions directly influence other agents. Therefore, if agent's 's action directly influences one of the other agents, the value of performing this action should be explicitly dependent more on the agent's 's observation for the environment and the information of the agent to be influenced by this action, while any additional information (c.g., part of the agent's 's observation for other agents) is irrelevant and may add noise. We refer to the property that different actions may have different impacts other agents as action semantics.
However, previous works usually use all available information for estimating the value of all actions, which can be quite inefficient. To this end, we propose a new network architecture called Action Semantics Network (ASN) that explicitly considers action semantics between agents to improve the estimation accuracy over different actions. Instead of inputting an agent's 's total observation into one network, ASN consists of several sub-modules that take different parts of the agent's 's observation as input according to the semantics of actions. In this way, ASN can effectively avoid the negative influence of the irrelevant information, and thus provide a more accurate estimation of performing each action. Besides, ASN is general and can be incorporated into existing deep MARL frameworks to improve the performance of existing DRL algorithms. In the next section, we will describe the ASN structure in detail.
3.2 ASN
Considering the semantic difference of different actions, we classify an agent's action set A'' of agent i into two subsets: AA, and Abur- AA,, contains actions that affect the environmental information or its private properties and do not influence other agents directly, e.g., moving to different destinations would only affect its own location information. A'ut corresponds to those actions that directly influence some of other agents, e.g., attack agent in competitive settings communicate with agent j in cooperative settings.
Following the above classification, the proposed network architecture. ASN, explicitly considers the different influence of an agent's 's actions on other agents by dividing the network into different nub-modules, each of which takes different parts of the agent's observation as input according to
3

---

Published as a conference paper at ICLR 2020
the semantics of actions (shown in Figure Considering agent and agents in its neigh- borhood, ASN decouples agent i's network into n sub-modules as follows. The first one shown in Figure 1 (left side O2.4') contains network O2E which is used to generate the observation emhedding e' given the full observation of of agent as input, and a network F22 (embedding to action) which generates the values of all action in A output. The rest of n sub-modules (O2A4H, E j. & i) are used to estimate the values of those actions in Abue related with each influenced agent, composed of determining the observation embeddings related with each influenced agent, denoted as e Each of n sub-modules O2 only takes part of agent i's observation related with one neighbor agent j. og as input.
networks (O2B*), j E N. & i) which are responsible for
For value- based RL methods, at each step t, the evaluation of executing each action aj E Ain is Q(o), af) = fa(e), af), where fa(e), aj) one of the outputs of the E2A network corresponding to a; To evalu- ate the performance of executing an action a, E Abut on another agent j. ASN com- bines these two embeddings et and e us- ing a pairwise interaction function M (e.g.. inner product):
((o,,) or PC 1of
(ea)()))))) 0 a) syoap(s ol) 0(0..) a Raa
E2A
interaction function
024 i
02,4 fm
02A
02F
Qloi, (/)) MMee, e
of
Loo m ot,
Lenv
i,1
im
O
then agent i selects the action of arg max at}} with certain explo- af
Figure 1: ASN of agent i contains n sub-modules: 02A',02At each of which takes different parts of the agent's observation input.
02444-1 024 ... 02.4m.
ration e.
Similarly, if the policy directly opti- mized through policy-based RL method- s, the probability of choosing each action is proportional to the output of each sub-module: (aa]o;)) x (x)() n (aa (of) &
exp(MMee, )) Then agent i selects an action following n':
o(a))o)) exp(sa(e),at) (o)(
x(af%lof) lo.) exp(M(ei),, e
(=((of)
where zdd (o;) is the partition function that normalizes the distribution. Note that we only consider the case that action a²x directly influences one particular agent j. In general, there may exist mul- tiple actions directly influencing one particular agent and how to extend our ASN will be introduced in Section 3.3 Multi-action ASN).
3.3 ASN-MARL
Next, we describe how ASN can be incorporated into existing deep MARL,, which can be classified into two paradigms: Independent Learner (IL) (Mnih et al.] 2015 Schulman et al. 2017 and Joint Action Learner (JAL) (Lowe et al. 2017: Rashid et al. 2018 Foerster et al. 201 8). IL applies single-agent learning algorithm to a multiagent domain to treat other agents as part of the environ- ment. In contrast, JALs observe the actions of other agents, and optimize the policy for each joint action. Following the above two paradigms, we propose two classes of ASN- MARL: ASN-IL and ASN-JAL. For ASN-IL, we focus on the case of combing ASN with PPO (Schulman et al.) 2017]. a popular single-agent policy-based RL. The way ASN combines with other single-agent RL similar. In contrast, ASN-JAL describes existing deep MARL approaches combined with ASN, e.g., QMIX (Rashid et al.) 2018) and VDN (Sunchag et al. 2018).
ASN-PPO For each agent equipped with a policy network parameterized by 0', ASN-PPO replaces the vanilla policy network architecture with ASN and optimizes the policy following PPO.
Generally, IL ignores the existence of other agents, thus, for each agent i, the expected return J(0)) is optimized using the policy gradient theorem: I(O)( E, y log ne. (allo))A.oof at)].
4

---

Published as a conference paper at ICLR 2020
0244
02A41
0244
02
0224 L
LG
02g4
028
Weggh Sharin
111
02
02
(a) Multi-action ASN
(b) Homogeneous ASN
(c) Mixed ASN
Figure 3: Different variants of ASN. Here we only present the right part ASN (excluding the left
part 02A' of ASN) as different variants.
where A, the advantage function at timestep t. PPO uses constraints and advantage estimation to reformulate the optimization problem
max EE, (o) #+
(3)
Tos (afla,)
where re(0") is the probability ratio
Tnd (atlo)) Oodd is the policy parameters before the update.
Then in ASN-PPO, (0') can be rewritten follows by substituting Equation 2
exp(falee 12000) (ot (3a))
if of E A if of E About
(+()
2 (at:0") exp(Mie), (01)))))
(4)
exp(M(ci, ",
2dd (ot:00))
Lastly, ASN-PPO maximizes the objective (Equation 3 following PPO during each iteration.
ASN-QMIX The way ASN com- bines with deep MARL algorithms is similar and we QMIX (Rashid et al. 2018) as an example to present. Figure 2 illustrates the ASN-QMIX network structure, where for each agent ASN-QMIX replaces the vanilla Q-network architecture with ASN. At each step t, the individu- al Q-function Q(oi, a.) is first calcu- lated following Section 3.2 and then input into the mixing network. The mixing network mixes the output of all agents' networks monotonically and produces the joint action-value function Qtoe (s;, ). The weights of the mixing network are restricted to be non-negative and produced by separate hypernetworks, each of which takes state 8, as input and generates the weights of one layer of the mixing network. Final-
(ue(S R2)
Rea (5.
Mixing Network St
02402
Qi (ol. ae)
Qx(o up
$
Agent
028
az( (x(of.a))
200
mg
oz
of
of
Figure 2: QMIX-ASN contains one mixing network and networks of all agents, and each agent network follows the ASN architecture.
ly, ASN- QMIX is trained to minimize the loss: L(Ø) = B [(yt Quor (s, a; 8)) where B is the batch size of transitions, yf rt - maxa Qtoe a'; )) and A are the parameters of the target network as in DQN (Mnih et al. 2015).
The general case in MASs that an agent may have multiple actions which can directly influence another agent, e.g., router can send packages with different size to one of its neighbors, soldier can select different weapons to attack enemies and cause different damages. To address this, we extend the basic ASN to generalized version, named Multi-action ASN (shown in Figure a)), that takes g² as input, and produces a number of embeddings ei _ where is the number of actions that directly influences agent j. After that, multi-action ASN calculates the estimation of performing each action, which uses a pairwise interaction function M to combine the two embeddings e (t. and e following Equation (1)
Multi-action ASN
5

---

Published as a conference paper at ICLR 2020
Parameter-sharing between sub-modules Parameter-sharing (PS) mechanism is widely used in MARL. If agents are homogeneous, their policy networks can be trained more efficiently using PS which greatly reduces the training complexity (Gupta et al. 2017). Recent work (Rashid et al. 2018) also incorporates PS on heterogeneous agents by adding extra information to identity agent type. Following previous work, here we incorporate PS to enable parameter-sharing between different sub-modules of ASN. The basic ASN (Figure I for agent contains a number of sub-modules O2A45, each of which takes a² as input. In this way, if an action ai A A'out has a direct impact on any of another agent J, the number of uub-mouuees is equal to the number of other agents. The training of basic ASN is inefficient since the number of sub-modules is increasing with the increase in the number of agents. If the other agents that agent i can directly influence are homogeneous, the sub-module parameters can be shared across those agents. Thus, in a homogeneous MAS, all influencing agents can share one sub-module (shown in 3 in a MAS that contains several types of agents, each type of agents can share one sub-module (Mixed ASN in Figure 3 (c)). Note that the basic ASN can be seen as the simplest case that designs a sub-module for each influencing agent without PS.
4 SIMULATIONS
We evaluate the performance of ASN compared with different network structures including the vanilla network (i.e., aggregate all information and input into one single network), the dueling net- work (Wang al.] 2016), the attention network that expects to learn which information should be focused on more automatically (i.e., adds an additional hidden layer to compute the weights of the input and then generate an element-wise product to input into the next layer) and entity-attentin network (i.e., instead of computing attention weight for each dimension of the input, the weight is computed for each entity/agent) under various DRL approaches. Other network architectures as we mentioned before are not comparable here since they are orthogonal to our ASN. Our test domains include StarCraft micromanagement (Samvelyan et al. 2019) and Massively Multiplayer Online Role- Games (Neural MMO) (Suarez et al. 2019). and parameter settings are in the appendix.
The details of neural network structures
4.1 STARCRAFT II
StarCraft II is a real-time strategic game with one or more humans competing against each other or a built-in game AI. Here we focus on decentralized multiagent control that each of the learning agents controls an individual army entity. At each step, each agent observes the local game state which consists of the following information for all units in its field of view: relative distance between other units, the position and unit type (detailed in the appendix) and selects one of the following actions: move north, south, east or west, attack one of its enemies, stop and the null action. Agents belonging to the same side receive the same joint reward at each time step that equals to the total damage on the enemy units. Agents also receive joint reward of 10 points after killing each enemy, and 200 points after killing all enemies. The game ends when all agents on one side die or the time exceeds fixed period. Note that previous works (Foerster al. 2018 Rashid et al., 2018 (Samelyyan et al.] 2019) reduce the learning complexity by manually adding a rule that forbids each agent to select an invalid action, e.g., attack an opponent that beyond the attack range and move beyond the grid border. We relax this setting since it requires prior knowledge, which is hard obtain in the real world. We are interested in evaluating whether these rules can be learned automatically through end-to-end training as well. Thus, the following results are based on the setting that each agent can select an action that causes an invalid effect, and in result, the agent will standstill at the current time step. We also evaluate ASN following previous settings (adding the manual rule in StarCraft II that forbidding the invalid actions) and ASN still achieves better performance which can be found in the appendix.
In StarCraft 8m map (8 Marines vs 8 Marines), each agent is homogeneous to each other, so we adopt homogeneous ASN to evaluate whether it can efficiently characterize action semantics between two agents. Figure 4 a). (b) and (c) show the performance of ASN on an 8m map compared with vanilla. dueling, attention and entity-attention networks under different DRL algorithms (IQL, QMIX, VDN). We can see that ASN performs best among all of the network structures in terms of both convergence rate and average win rates. By taking different observation information as the
6

---

Published as a conference paper at ICLR 2020
0.8
0.8
0.8
0.6
Win rates
Win rates
0.6
0.6
Win
0.4
0.4
0.4
0.2
0.2
0.2
ABN
0.0
Enitty Alteetiee
0.0
Ameettioon
0.0
Entry Attortion
0.00
0.25
0,50 0.75 1.00 1.25
0.00
0.25
0.50 0.75
1.00 1.25 Step
0.00
0.25 0.50 0.75 1.00
1.25 x10*
Step (a) IQL
X10*
x10*
Step
(b) QMIX
(c) VDN
Figure 4: Win rates of various methods on the StarCraft II 8m map.
input of different sub-modules, ASN enables an agent to learn the right timing to attack different opponents to maximize its total damage on opponents. In contrast, existing network architectures simply input all information into one network, thus an agent cannot distinguish the difference of effects that different actions may have on the opponents and may choose the suboptimal opponent to attack, thus resulting in lower performance than ASN. Attention network performs better than vanilla and dueling when combined with IQL., while both them show very similar performance with the vanilla network when combined with QMIX and VDN. However, antity-attention performs worst since it is hard to figure out the useful information for each entity when input all information into one network initially. Since the performance difference of other network architecture is marginal, we only present results of ASN-QMIX compared with the vanilla network under QMIX (denoted as vanilla- (MMl) in the following sections.
Next, we consider a more complex scenario: StarCraft II 2S3Z (2 Stalkers and 3 Zealots vs 2 Stalkers and 3 Zealots) which contains two heterogeneous groups, each agent inside one group are homogeneous and can evaluate the performance of Mixed ASN compared with vanilla-QMIX. From Figure 5)a) we can observe that Mixed ISN-QMIX perform better than vanilla- QMIX. The reason is that ASN efficiently identifies action semantics between each type of two agents, thus selects more proper attack options each time and achieves better performance last vanilla-QMIX.
1.0
0.8
0.6
Wia
Vanille-QMIn
ASN- QMIX
0.0
Stop
(a) 2x.3z
Is ASN still effective on large-scale scenarios? We further test on large-scale agent space 15m map. Figure 5 (b) depicts the dynamics of the aver- age win rates of ASN-QMIX and Aanill-QMMIX We can see that ASN-QMIX quickly learns the average win rates of approximately 80 %, while vanilla-QMIX fails, with the average win rates of approximately only 20 %. From Figure 4 (b) and (b) we can find that with the increase of the agent number, the margin be- comes larger between two methods. Intuitively, ASN enables an agent to explicitly consider more numbers of other agents' information with a larger agent size. However, for the vanilla network, it is more difficult to identify the action influence on other agents from larger amount of mixed information, which results in lower average win rates than ASN. An interesting observation for vanilla- QMIX is that they will leam to run away to avoid all being killed, and testing videos can be found in our anonymous @e
0.8
Vanill-
Win rates
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
X31"
(b) 15m
Figure 5: Win rates of ASN-QMIX and vanilla-QMIX on different SC II maps.
Table I: PCT of choosing valid action for ASN- RENI and vanilla-QMIX.
ASN
Vanilla
PCT
71.9 + 0.15% 44.3 + 0.11%
Can ASN recognize the influence of different actions? Table I presents the average percent- ages of choosing valid action for ASN-QMIX and vanilla-QMIX 15m map. Note that
7

---

Published as a conference paper at ICLR 2020
Vanila
-0.16
-0.17
-0.18
Opporwet
Opponent
-0.34
-0.36
Opponent Opponent
-0.19
-0.38
-044
-0.20
-0.40
-044
-0.8
-0.22
-0.8
ulack sarge
Within
4
Bayyond wftick
10
14
-0.21
-0.23
--00.--02--0100 0.0
-0.46
-0.44
-0.40.3-022-0111
Distance
HP Difference
HP Difference
(a) Scenario
(b) Vanilla-QMa on scenario 2
(c) ASN- QMIX scenario 2
Figure 6: The attack action's Q-values of ASN and vanilla under different circumstances.
1.0
1.0
Win rates
Win rates
0.5
0.5
Allertun Enilyy Allastes
Alizzkn Eallyy Albetion
0.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
Step
Step
(a) -1 L-paddings
(b) 1- 1-paddings
Figure 7: Win rates on SC II 8m map when replacing O-paddings with 1-paddings and 1-paddings.
remove the manually added rule (which prevents selecting any invalid action), and agents would probably select the invalid action and standstill, which increases the learning difficulties. We can that ASN-QMIX achieves average percentage of approximately 71.9% for choosing valid action. However, vanilla- -MIX only achieves an average percentage of approximately 44.3%. This phenomenon confirms that ASN effectively exploits action semantics between agents and enables agents to learn which action can be chosen at each time step, facilitating more robust learning, even in large-scale MASs.
Can ASN effectively improve the estimation accuracy of actions? We investigate whether ASN can efficiently characterize the action semantics and facilitate multiagent coordination. To make the analysis more clear, we test the model learned on a 15m map on two illustrating scenarios: 1) the one-on-one combat scenario that the distance between two agents is dynamically changing: 2) the dynamically
one Marine vs two Marines scenario that the HPs (Hit Points) of two opponents different. Figure 6]a) shows the dynamics of the attack action's Q-value with the distance change of the ASN agent and its opponent. We can observe that the Q-value of the action that the ASN agent attacking its opponent decreases as the distance of the agent and its opponent increases, and stabilizes when the distance exceeds the attack range. However, the vanilla agent keeps the Q- value of the attack action nearly unchanged. This indicates that ASN can automatically learn the information of when an action is valid and behave appropriately, while the vanilla agent has rely on manually added rules to avoid choosing invalid actions. Figure (b) and (c) shows the dynamics of the attack action's Q-value of ASN agent and vanilla agent with The HPs difference of two opponents changing (i.e., the HP difference equals to the HP of opponent I minus the HP of opponent 2). We can see that the ASN agent holds higher Q-value of attacking opponent when opponent I's HP is lower than opponent 2 and vice versa. The symmetric curve of ASN is due to the fact that the state description of two opponents is very similar in this scenario. However, the vanilla agent always keeps higher attack action's 's Q-value selects to attack Opponent 1. These results indicate that ASN can effectively exploit the action semantics between agents and improves the estimation accuracy on different actions, thus facilitates robust learning among agents.
Opponent than Opponent 2, which means it always
Does ASN exploits the O-padding information? When one of the army units dies some units are beyond the range of vision, one common practice is to use O-paddings as the input for the obser- vation of the died army unit. In this section, we provide ablation study whether ASN design
8

---

Published as a conference paper at ICLR 2020
125
125
Avg Reward
100
Avg Reward
Avg Reward
100
100
75
75
60
50
VasiEa
50
Wentla
40
Vanih
ASN-M
25
ABN
Atterdicc
A58-0
Easry Atsaatnn
ASN-4
20
Atrestoon
0.0
0.2
0.4
0.6
0.8
1.0
0
2
0.0
0.2
0.4
0.6
0.8
1.0
Step
X10
Step
K10
Step
x10
(a) PPO
(b) ACKTR
(c) A2C
Figure 9: Average rewards of various methods on Neural MMO.
exploits the O-padding information. Figure 7 shows the win rates of various network architectures combined with QMIX when using 1-paddings and -1-paddings as the input for the observation of the died army unit. We can see that ASN still performs best among all network architectures in terms of both convergence speed and final win rates. This indicates that ASN effectively extracts the action semantics between agents, instead of benefiting from the particular settings of O-paddings.
4.2 NEURAL MMO
The Neural MMO (Suarez et al., 2019) is
massively multiagent environment that defines combat systems for a large number of agents. Figure [ illustrates
Damage Range
Mage
simple Neural MMO
scene with two groups of agents on a 10x 10 tile. Each group contains 3 agents, each of which starts at any of the tiles, with HP = 100. At each step, each agent loses unit of HP, observes local game state (detailed in the ap- pendix) and decides on an action, i.e., moves one tile (up, right, left, down and stop) or makes an attack using any of three attack options (shown the left part in Figure 8 "Melee" with the at- tack distance is 2, the amount of damage is 5; "Range" with the attack distance is 4, the amount of damage 2: "Mage" with the attack distance is 10, the amount of damage 1). Each action that causes an invalid effect (e.g., attack an opponent that beyond the attack range and move beyond the grid border) would make the agent standstill. Each agent gets penalty of -0.1 if the attack fails. The game ends when all agents in one group die, and agents belonging to the same group receive joint reward, which is the difference of the total HPs between itself and its opposite side.
Range
Melce
Figure
An illustration of Neural MMO that contains two armies (red and green).
In Neural MMO, agent can attack one of its opponent using one three different attack options, which can be used to eval- uate whether multi-action ASN can efficiently identify the mul-
3.0
tiple action semantics between agents. Here we adopt two kinds
2.5
of multi-action ASN: ASN-MI that shares parameters of the first neural network layer across three attack actions on one enemy (as shown in Figure a))); and ASN-M that does not share. Fig- ure a). (b) and (c) present the performance of multi-action AS- on Neural MMO compared with vanilla, attention and entity- attention networks under different IL methods (PPO, ACKTR (Wu et al. 2017) and A2C (Mnih et al. 2016)). We can observe that ASN performs best under all three IL approaches in terms of average rewards. This is because ASN can learn to choose appropriate actions against other agents at different time steps to maximize the damage on others. However, the vanilla network just mixes all information together which makes it difficult to identify and take advantage of the tion semantics between agents, thus it achieves lower performance than ASN. Since the information is mixed initially, although the attention and entity-attention networks try learn which information
2.0
1.5
1.0
Alardion
Figure 10: The average damage of choosing each attack when distance dij S 2 under A2C.

---

Published as a conference paper at ICLR 2020
should be focused more, it is hard to distinguish which part of the information is more useful, thus achieving lower performance than ASN.
Can ASN recognize the best actions from multiple ones? We further investigate whether ASN can efficiently exploit different action semantics between agents and enable an agent to identify the best attack option (i.e., attack that causes the most damage) with the distance between the agent and its opponent changing. Figure 10 shows the average attack damage of each attack option in Neural MMO when the distance between agent and its opponent is less than equal (d:j S 2). The best attack option is "Melee" within this distance range since it causes the maximum damage among three attacks. We can see that both ASN-MI agent and ASN-M cause higher total damage than other methods, and ASN-MI agent causes the highest total damage on average. However, the attention network only causes average total damage of approximately 1.5, the entity-attention and vanilla network only cause average total damage of approximately 1.0 due to the lower probability of selecting the best attack action "Melee". This because two kinds of ASN have larger probability to select the best attach option "Melee" than other two networks, thus causing larger total damage. Similar results on other distance ranges (dz,j I 4, dz.5 S 10) can be found the appendix that ASN always causes higher total damage than other networks.
CONCLUSION AND FUTURE WORK
We propose network architecture, ASN, facilitate more efficient multiagent learning by explicitly investigating the semantics of actions between agents. To the best of our knowledge, ASN is the first to explicitly characterize the action semantics in MASs, which can be easily combined with various multiagent DRL algorithms to boost the learning performance. ASN greatly improves the performance state-of-the-ar thrt DRL methods compared with number of network architectures. In this paper, we only consider the direct action influence between any of two agents. As future work, is worth investigating how model the action semantics among more than two agents. Another interesting direction is to consider the action semantics between agents in continuous action spaces.
6 ACKNOWLEDGEMENTS
This work is supported by the National Natural Science Foundation of China (Grant Nos.: 61702362, U1836214, 61432008) and Science and Technology Innovation 2030 "New Generation Artificial Intelligence" Major Project No.22018AAA0000005,
REFERENCES
Lucian Bu, Robert Babu, Bart De Schutter, al. A comprehensive survey of multiagent reinforce- ment learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):156-172, 2008
Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multia- gent systems. In Proceedings of the Fifteenth National Conference Artificial Intelligence and Tenth Innovative Applications of Artificial Intelligence Conference, pp. 746-752, 1998.
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the Thirty-Second AAAI Confer- ence on Artificial Intelligence, 2018.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In Proceedings of the 16th International Conference on Autonomous Age Agents and Multiagent Systems, Workshops, pp. 66-83, 2017.
Eric A Hansen, Daniel S Bernstein, and Shlomo Zilberstein. Dynamic programming for partially observable stochastic games. In Proceedings of the Nineteenth National Conference Artificial Intelligence, volume 4, pp. 709-715, 2004.
Famn-Michaell De Hauwere, Sam Devlin, Daniel Kudenko, and Ann Nowé. Context-sensitie reward shaping for sparse interaction multi-agent systems. Knowledge Eng. Review, 31(1):59-76, 2016.
10

---

Published as a conference paper at ICLR 2020
Junling Hu and Michael P. Wellman. Multiagent reinforcement an algorithm. In Proceedings of the Fifteenth International Conference on Machine Learning, pp- 242-250 1998.
rentiagentiinfooremmeetttt learning: Theoretical framework and
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tas- sa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proceedings of the 4th International Conference on Learning Reprexentations, 2016.
Michael Littman. Markov games framework for multi-agent reinforcement learning. In Proceedings of the Eleventh International Conference on Machine Learning, pp. 157-163, 1994.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed seaperative-compettivv environments. Advances in Neural Information Processing Systems, pp. 6379-6390 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International conference machine learning, pp. 1928-
Kwang-Kyo Oh, Myoung-Chul Park, and Hyo-Sung Ahn. A survey of multi-agent formation con- trol. Automatica, 53:424-440, 2015.
OpenAI. Openai five. https: http:///bboooopeeeaacccoo///apap five/
Gregory Palmer, Karl Tuyls, Daan Bloembergen, and Rahul Savani. Lenient multi-agent deep rein- forcement learning. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 443 443-451,1 2018.
Liviu Panait, Karl Tuyls, and Sean Luke. Theoretical advantages of lenient learners: An evolutionary game theoretic perspective. J. Mach. Learn. Res., 9:423-457, 2008.
Mitchell A. Potter and Kenneth A. De Jong. A cooperative coevolutionary approach to function optimization. In Proceedings of International Conference on Evolutionary Computation, pp. 249- 257, 1994.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent rein- forcement learning. In Proceedings of the 35th International Conference on Machine Learning, pp. 4292-4301, 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. pp. 21 21882188, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
06347, 2017.
David Silver, Julian Schritwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap. Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Individualized controlled continuous communication model for multiagent cooperative and competitive tasks. Proceedings of the 7th International Conference Learning Representations, 2019.
Adrian Sosic, Wasiur R. KhudaBukhsh, Abdelhak M. Zoubir, and Heinz Koeppl. Inverse reinforce- ment learning in swarm systems. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pp. 1413-1421, 2017.
11

---

Published as a conference paper at ICLR 2020
H Bugene Stanley. Phase transitions and critical phenomena. Clarendon Press, Oxford, 1971.
Joseph Suarez, Yilun Du, Phillip Isola, and Igor Mordatch. Neural mmo: A massively mul- tiagent game environment for training and evaluating intelligent agents. arXiv preprint arX- iv:1903.00784, 2019.
Sainbayar Sukhbaatar, Rob Fergus, al. Learning multiagent communication with backpropaga- tion. In Advances in Neural Information Processing Systems, pp.
2016.
Peter Sunchag, Guy Lever, Audrunas Gruslys. Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning based team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2085-2087, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Andrea Tacchetti, H. Francis Song, Pedro A. M. Mediano, Vinícius Flores Zambaldi, János Kramár, Neil C. Rabinowitz, Thore Graepel, Matthew Botvinick, and Peter W. Battaglia. Relational for- ward models for multi-agent learning. In Proceedings of the 7th International Conference Learning Representations, 2019.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning, pp. 1995-2003, 2016.
David H. Wolpert and Kagan Tumer. Optimal payoff functions for members of collectives. Advances in Complex Systems, 4(2-3):265- 280, 2001.
Yuhuai Wu, Elman Mansimov, Roger Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation. In Advances in neural information processing systems, pp. 5279-5288, 2017.
Yang, R Luo, M Li, M Zhou, W Zhang, and Wang. Mean field multi-agent reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 5571-5580, 2018.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. Deep re- inforcement learning with relational inductive biases. Proceedings of the 7th International Conference on Learning Representations, 2019.
12

---

Published as a conference paper at ICLR 2020
A APPENDIX
I. StarCraft II
State Description In StarCraft II, we follow the settings of previous works (Rashid et al. 2018 Samvelyan et al., 2019). The local observation of each agent drawn within their field of view, which encompasses the circular area of the map surrounding units and has radius equal to the sight range. Each agent receives as input a vector consisting of the following features for all units in its field of view (both allied and enemy): distance, relative x, relative y, and unit type. More com/MAS MAS anony/AS) or https: htpp::///ithbb
details can be found at
com/ www.////wwraaraaeam. smac
Network Structure
The vanilla network (Figure aaa of each agent contains two fully-connected hidden layers with 64 units and one GRU layer with 64 units, taking of as input. The output layer is a fully-connected layer outputs the Q-values of each action. The attention network (Figure IT b)) of each agent i contains two isolated fully-connected layers with 64 units, taking of as input and computing the standard attention value for each dimension of the input. The following hidden layer is GRU with 64 units. The output contains the Q-values of each action. The entity-attention network (Figure (T())) is similar to that in Figure 11 except that the attention weight is calculated on each of The dueling network (Figure dTj) is the same vanilla except for the output layer that outputs the advantages of each action and also the state value. Our homogeneous ASN (Figure
The details of different network structures for StarCraft II are shown in Figure 11
(T(e)) of each agent i contains two sub-modules, one is the O2 which contains two fully-connected layers with 32 units, taking of as input, following with a GRU layer with 32 units; the other is a parameter- sharing sub-module which contains two conneccod layers with 32 units, taking each of as input, following with GRU layer with 32 units; the output layer outputs the Q-values of each action.
Vanilla
Attention
Eniity Attention
hidden
hidlen units
#A hidlden units
Fully conneced
64 hidden units
hidden units
64 hidden units
Bieu
Relu
Attention
Attention Weight
64 hidden units
64 hidden units
Entity Attention
Relu
Ralu 64 hidclnn units
hidden units
K
Bnitity hidden units
of
of
(a) Vanilla
(b) Attention
(c)
Dueling
ASN
Product
hidden units hidden units
#m hidden units
Homogeneous:
Weight Sharing
32 hidden units
32 hidden units
64 hidden units
Relu
Relu
Relu
64 hidden units
32 hidden units
32 hidden units
Relu
Relu
Relu
64 hidden units
32 hidden units
32 hidden units
of
i,1
02
(d) Dueling
(e) Homogeneous ASN
Figure 11: Various network structures StartCraft II 8m map.
13

---

Published as a conference paper at ICLR 2020
Table Hyperparameter settings for StarCraft II.
Hyperparameter
Value
Batch-size
32
Replay memory size
5000
Discount factor(>)
0.99
Optimizer
RMSProp
Learning rate
5e 4
0.99
l 5
Gradient- norm--
10
Action-selector
e-greedy
e-start
1.0
e-finish
0.05
e-anneal-time
50000 step
larget-update-interval
200
0.9
Win rates
0.8
0.7
0.6
Vanilla
ASN
0.5
0.2
0.4
0.6
0.8
1.0
1.2
1.4 x10
Step
Figure 12: Win rates of ASN-QMIX and vanilla-QMIX under 5m StarCraft II map.
Parameter Settings
Here we provide the hyperparameters for StarCraft II shown in Table 2
Experimental results
The following results present the performance of ASN-QMIX and vanill-QQMIX under different StarCraft II maps with adding the manual rule (forbids the agent to choose the invalid actions).
II. Neural MMO
State Description
teams In 10x10 tile (where each tile be different kinds, e.g., rocks, grass), there of agents (green and red), each of which has 3 agents. At the beginning of each episode, each agent appears on any of the 10 tiles. The observation of an agent in the form of 43-dimensional vector, in which the first 8 dimensions are: time to live, HP,, remaining foods (set 0), remaining water (set 0), current position and y), the amount of damage suffered, frozen state (1 0); the rest of 35 dimensions are divided equally to describe the other 5 agents' information. The first 14 dimensions describe the information of 2 teammates, following with the description of opponents' information. Fach observed agent's information includes the relative position(x and y), whether it is a teammate(1 or 0), HP, remaining foods, remaining water, and the frozen state.
More details he found httttp////iiihib.cc////saaaaaaaa
14

---

Published as a conference paper at ICLR 2020
Each agent chooses action from set of 14 discrete actions: stop, move left, right, up down, and three different attacks against one of its opponent ("Melee" with the attack distance is 2, the amount of damage is 5; "Range" with the attack distance is 4, the amount of damage is 2; "Mage" with the attack distance 10, the amount of damage is 1).
Each agent gets penalty of -0.1 if the attack fails. They get -0.01 reward for each tick and -10 penalty for being killed. The game ends when a group of agents dies or the time exceeds fixed period, and agents belonging to the same group receive the same reward, which is the difference of the total number of HPs between itself and its opposite side.
Network Structure
The details of vanilla, attention, entity-attention networks for Neural MMO are shown Figure 13 a-c) which contains an actor network, and critic network. All actors are similar those for StarCraft II in Figure[[]] except that the GRU layer is excluded and the output is the logic probability of choosing each action. All critics are the same shown in Figure a)). Since in Neural MMO, each agent has multiple actions that have direct influence on each other agent, i.e., three kinds of attack options, we test two kinds of ASN variants: one (Figure 13 d)) is the Multi-action ASN we mentioned in the previous section that shares the first layer parameters among multiple actions; the other (Figure (3(e)) the basic homogeneous ASN that does not share the first layer parameters among multiple actions.
Parameter Settings
Here we provide the hyperparameters for Neural MMO shown in Table
Experimental results
The above results present the average attack damage of each attack option under the different dis- lance ranges between the agent and its opponent.
15

---

Published as a conference paper at ICLR 2020
Vanilla
Actor
Tanh
Critic
#A hidden units
1 hidden units
64 hidden units
64 hidden units
Tanh
Tanh
64 hidden units
64 hidden units
t
of
(a) Vanilla
Attention
Entity Attention
Actor
Tanh
Actor
Tanh
#A hidden units
#4 hidden units
Tanh
Tanh
Attention
Attention Weight
64 hidden
Tach
64 hidden units
Entity Attention
Tanh
Atanroo Weight
#Entity hidden units
(b) Attention
(c) Entity-
Multi-action ASN
Homogeneous
Actor
Weight Sharing
X
Inner Product
Inner Product
#A hidden units
Tanh
Tanh
Tanh
16 hidden units
16 hidden units
16 hidden units Tanh
16 hidden units
Tanh
32 hidden units
32 hidden units
1
of
1
00
(d) Multi-action ASN
Basic ASN
Homogeneous:
Actor
Weight Sharing
Inner Product
Inner Product
Inner Product
#A hidden units
Tanh
Tanh
Tanh
Tanh
16 hidden units
hidden units
hidden
16 hidden units
Tanh
Taeh
Tanh
Twnh
32 hidden units
16 hidden units
16 hidden units
16 hidden units
(e) Homogeneous ASN
Figure 13: Various network structures on Neural MMO.
16

---

Published as a conference paper at ICLR 2020
Table 3: Parameters of all algorithms.
(a) PPO
(b) A2C
Hyperparameter
Value
Hyperparameter
Value
Number of processes
Number of processes
5
Discount factor(7)
0.99
Discount factor(ly
0.99
Optimizer
Adam
Optimizer
RMSProp
Learning rate
7e 4
Learning rate
7e
le 5
a
0.99
Entropy term coefficient
le-2
e
1e -
Value loss coefficient
0.5
Gradient-norm-clip
0.5
Actor loss coefficient
Entropy term coefficient
le-2
Value loss coefficient
0.5
Actor loss coefficient
1
(c) ACKTR
Hyperparameter
Value
Number of processes
Discount factor(>)
0.99
Optimizer
KFACOptimizer
Learning rate
0.25
Momentum
0.9
Stat decay
0.99
KL clip
le-3
Damping
1e-2
Weight decay
0
Entropy term coefficient
1e-2
Value loss coefficient
0.5
Actor loss coefficient
17

---

Published as a conference paper at ICLR 2020
4.5
4.5
4.5
4.0
Meleo
4.0
4.0
Range
3.0
Mage
3.0
330
2.5
Total
2.5
2.5
2.0
2.0
2.0
2 1.5
1.5
8 1.5
1.0
1.0
1.0
0.5
0.5
0.5
0.0
ASH
ASA-3
Aderdon
Varia
0.0
ASN-M
Adarion
Varia
0.0
ASN-V
Adrtin
Vanaa
(a)
dis < 2
(b) dgg S
(c) des S 10
Figure 14: The average probabilities of choosing each attack under different distance diy for various network architectures combined with PPO in Neural MMO.
Meles Range
Avig Damage
Magie
Avig Damage
Avi Damage
Total
(a) dag
(b) dig
(c) dey S 10
Figure 15: The average probabilities of choosing each attack under different distance dij for various network architectures combined with A2C in Neural MMO.
Maice
Range
Avg Damage
Mage
Avg Damagg
Total
Avg
TAVEN
Adarion
(a) dij S 2
(b) dij 4
(c) di S 10
Figure 16: The average probabilities of choosing each attack under different distance diy for various network architectures combined with ACKTR in Neural MMO.
18


Reviews:
[{"review_id": "r1lZZbJMqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The motivation for this paper is straightforward. The author believes that the semantic information of actions should be explicitly considered in the multi-agent reinforcement learning problem, and the information needed to make different semantic actions should be different. To this end, the author divides the action semantics into two categories, one is actions that only affect the environment and itself, and the other is actions that affects the other agents. Making a decision corresponding to the former requires relying on all local observations of the agent, but making a decision corresponding to the latter requires only local observations related to the affected agent.\n\nTo this end, the author proposes a novel network structure ASN that can achieve the above objectives. Because it is a modification of the network structure, ASN can be combined with any type of multi-agent reinforcement learning algorithm to improve its convergence speed and final performance. The author conducted a rich experiment and verified the validity and superiority of the ASN structure from various angles.\n\nBut for this paper, I have doubts in the following aspects:\n1. In the experimental part, the author compares the ASN-based algorithm with QMIX and VDN in the StarCraft II environment, but the performance of the benchmark algorithm on the 8m and 2s3z maps is significantly lower than that of the original paper. Because the author did not use the RNN network when implementing these algorithms? If so, why not implement it with the RNN network? Can ASN also be combined with RNN?\n2. The basic idea of ASN is similar to the attention mechanism, and the experimental part also shows that the algorithm with attention module also achieved good results. I think the author should also compare with the MAAC algorithm (Actor-Attention-Critic for Multi-Agent Reinforcement Learning, ICML2019). There are two reasons for this: First, the MAAC algorithm is a SOTA MARL method based on the attention mechanism; secondly it belongs to the actorcritic algorithm. In the comparison part of the JAL algorithm, the author only considered the value-based methods VDN, QMIX, etc., without considering the policy gradient method or the actor-critic method.\n3. In the experimental part, the author also compares the proportion of ASN-based methods and benchmark algorithms that perform non-valid actions during training, such as to attack an agent outside the scope. However, in the network structure of ASN, if an agent is outside the field of view, it will not output the Q value or probability of the action corresponding to that agent. Does this mean that ASN is impossible to choose non-valid actions? So is the corresponding experiment meaningless?\n4. If an action can affect multiple other agents at the same time, can the ASN network handle it?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJgwr3-TtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a neural network architecture that provides an agent-agent based embeddings that are used for actions that directly affect specific agent. Proposed architectural choice exploits (implicitly assumed) independence of some actions wrt. observations of agents that it is not directly affecting. Authors perform experiments in minigame of SC2 (controlling small armies) and in NeuralMMO (analogous setup).\n\nIt is worth stressing, that apart from conducting typical analysis, authors also put forward more qualitative hypotheses, that are then verified by custom experiments. I highly value this way of conducting research, as it provides way more insights into the method than just pure return plots, often encountered in similar papers.\n\nMajor comments\n- Please be more explicit in the abstract, when talking about the experiments conducted. Claiming \"Experimental results on StarCraft II and Neural MMO show ASN significantly improves the performance of state-of-the-art DRL approaches compared with several network architectures.\" could be misinterpreted as providing a solution that can play SC2 better than state-of-the-art methods, while the actual result is to be able to micromanage small fights in this environment well, and thus it would be fairer to say \"on StarCraft II Minigame\" or more explicitely \"StarCraft II small scale fight simulation\" or anything else in this spirit. These are great research domains, there is no need to overclaim the results.\n- A proposed architecture assumes (to some extent) an independence of effects of action-agent pairs (action aimed at player i is conditionally independent from observation of agent j). There is still a way to represent it, but only by bypassing the proposed architecture, and expressing this behaviour directly in e^i. While this sort of decomposition makes sense for various heavily localised problems (such as micro-management of fights in games like SC2 or NeuroMMO) it looks like a potential limitation, and assumption, that is not articulated in the text. Have authors performed any analysis looking into this phenomenon?\n- It is unclear, how ASN compares or relates to the architecture used in OpenAI Five network (https://d4mucfpksywv.cloudfront.net/research-covers/openai-five/network-architecture.pdf) which arguably should be used as a baseline, given that it follows a similar idea of producing per-agent embedding, that influences part of action space, that is directly related to influencing other agents (targeting opponents). Despite not being peer reviewed, it is a well known, and clearly described method, which should at the very least be acknowledged as prior work.\n- Some equations are hard to parse, e.g.:\n-- page 5, sum_i=1^b [(y_t^tot - Q_tot(s,a; theta))^2]  <- \"i\" is never used in the equation (I guess it changed meaning, and is no longer agent identity, but rather batch index?)\n-- page 4, what does expectation over \"t\" mean? In the introduced notation, expectation is taken wrt. control policy pi_i, interpreted as a joint probability distribution over agent actions and environment state transition. Can this be unified?\n\nMinor comments\n- \"When one of the arm units dies\" -> \"When one of the army units dies\"\n- details of ASN-PPO could be safely moved to the Appendix, as it is a direct substitution regular policy updates with (2), rather than a separate contribution that requires in-depth description (and given that paper is quite long, it could use a bit of message \"sharpening\")\n\nI am happy to revisit (and increase) the rating assigned, once authors address the comments above, especially the relation to OpenAI Five architecture.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJg38MWCKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focused on the multi-agent systems, and proposed a new network architecture (aka ASN). In the new architecture, the action set is  manually split into two subsets, each of which contains the actions that affects other agents or not.  Besides the net architecture, the authors also discussed several important issues including “Multi-action ASN” and  parameter-sharing. In the experiments, the authors evaluate the proposed nets in Starcraft and Neural MMO, and shows superior performance compared with other methods. \n\nComment:\nThis paper is relatively well-written and clearly introduces the basic idea of the work. The authors focuses on an important issue for MAS, i.e. how to reduce model complexity when increasing the number of agents. In their work, the authors would like to decompose the action set into two smaller sub-sets based on the so called action semantics, such that the designed nets have simpler and more straightforward (efficient) structure. It is interesting and  somehow convincing. One question on this point how to split the action set properly? My concern is from more general cases, where boundary  to classify the action set would be not so clear. In this scenario, how to obtain an proper split. And what would happen if we choose the wrong subset? At the beginning, I though the split would be learned by some sub-nets, but after reading I found it seems to manually set it. This is the motivation why I concern this problem. \n\nAnother point I concern is about the parameter-sharing (PS) issue in Page 5. I understand the authors want to impose the PS trick to reduce the training complexity for the model.  I also agree with this, but there is several points, which confuse me somewhat. (1) I’m still confused the detail of PS. Does PS means the agents share the exactly same weights, same structure, some correlation or else? (2) I know the PS trick is popularly used in multi-task learning, in which the designers have to face the question “HOW MUCH common information should the individual networks share”. It means that PS trick might results in challenging model selection problem in practice. Hence I am curious how such issue is handled in the proposed model? I know it might not be the main contribution of this paper, but I think it is important to discuss, which might be missed in the paper. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors address the challenge of sample-efficient learning in multi-agent systems. They propose a model that distinguishes actions in terms of their semantics, specifically in terms of whether they influence the acting agent and environment or whether they influence other agents. This additional structure is shown to substantially benefit learning speed when composed with a range of state of the art multi-agent RL algorithms. During the rebuttal, technical questions were well addressed and the overall quality of the paper improved. The paper provides interesting novel insights on how the proposed structure improves learning.
Decision: Accept (Poster)