Learning Nearly Decomposable Value Functions Via Communication Minimization
Authors: Tonghan Wang*
Jianhao Wang*
Chongyi Zheng
Chongjie Zhang
Venue:  (2020)
Keywords: Multi-agent reinforcement learning
Nearly decomposable value function
Minimized communication
Multi-agent systems

Abstract: Reinforcement learning encounters major challenges in multi-agent settings, such as scalability and non-stationarity. Recently, value function factorization learning emerges as a promising way to address these challenges in collaborative multi-agent systems. However, existing methods have been focusing on learning fully decentralized value functions, which are not efficient for tasks requiring communication. To address this limitation, this paper presents a novel framework for learning nearly decomposable Q-functions (NDQ) via communication minimization, with which agents act on their own most of the time but occasionally send messages to other agents in order for effective coordination. This framework hybridizes value function factorization learning and communication learning by introducing two information-theoretic regularizers. These regularizers are maximizing mutual information between agents' action selection and communication messages while minimizing the entropy of messages between agents. We show how to optimize these regularizers in a way that is easily integrated with existing value function factorization methods such as QMIX. Finally, we demonstrate that, on the StarCraft unit micromanagement benchmark, our framework significantly outperforms baseline methods and allows us to cut off more than $80\%$ of communication without sacrificing the performance. The videos of our experiments are available at https://sites.google.com/view/ndq.

Published as a conference paper at ICLR 2020
LEARNING NEARLY DECOMPOSABLE VALUE FUNC- TIONS VIA COMMUNICATION MINIMIZATION
Tonghan Wang"; Jianhao Wang", Chongyi Zheng' & Chongjie Zhang!
Institute for Interdisciplinary Information Sciences, Tsinghua University, Beijing, China "Turing AI Institute of Nanjing, Nanjing, China
wangth18@maile ts tsinghua. edu. en, wjh720 eri ericegmai com
chongyeezhen@@@mall com, chongjle@tsinghua edu. en
ABSTRACT
Reinforcement learning encounters major challenges in multi-agent settings, such as scalability and non-stationarity. Recently, value function factorization learning emerges as promising way to address these challenges in collaborative multi- agent systems. However, existing methods have been focusing on learning fully decentralized value functions, which are not efficient for tasks requiring com- munication. To address this limitation, this paper presents novel framework for learning nearly decomposable Q-functions (NDQ) via communication mini- mization, with which agents act their own most of the time but occasionally send messages to other agents in order for effective coordination. This framework hybridizes value function factorization learning and communication learning by introducing two information-theone mizing mutual information between agents' action selection and communication messages while minimizing the entropy of messages between agents. We show how optimize these regularizers in way that is easily integrated with existing value function factorization methods such as QMIX. Finally, we demonstrate that, the StarCraft unit micromanagement benchmark, our framework significantly outperforms baseline methods and allows us to cut off more than 80% of commu- nication without sacrificing the performance. The videos of our experiments are available at htthttts:/s/stt www.sittessoosel.eoooeooooooooee google. hwmm/vieww// ndc
regularizers. These regularizers are maxi-
INTRODUCTION
Cooperative multi-agent reinforcement learning (MARL) finding applications in many real- world domains, such as autonomous vehicle teams (Cao et al. 2012), intelligent warehouse sys- tems (Nowé et al. 2012). and sensor networks (Zhang & Lesser 2011). To help address these problems, recent years have made a great progress in MARI methods (Lowe et al., 2017] Foer- ster et al. 2018 Rashid et al.) [2018 Jaques et al., 2019). Among these successes, the paradigm of centralized training with decentralized execution has attracted much attention for its scalability and ability to deal with non non-stationarity.
Value function decomposition methods provide promising way to exploit such paradigm. They learn decentralized Q function for each agent and use mixing network combine these local Q values into global action value. In previous works, VDN (Sunehag al. 2018). QMIX (Rashid al., 2018), and QTRAN (Son al., 2019) have progressively enlarged the family of functions that can be represented by the mixing network. Despite their increasing ability in terms of value factorization representation, existing methods have been focusing on learning full decomposition, where each agent acts upon its local observations. However, many multi-agent tasks in the real world are not fully decomposable agents sometimes require information from other agents in order to effectively coordinate their behaviors. This is because partial observability and stochasticity in multi-agent environment can exacerbate an agent's uncertainty of other agents' states and actions during decentralized execution, which may result in catastrophic miscoordination.
"Equal Contribution.

---

Published as a conference paper at ICLR 2020
To address this limitation, this paper presents scalable multi-agent learning framework for learning nearly decomposable Q-finctions (NDQ) via communication minimization, with which agents act on their own most of the time but occasionally send messages to other agents in order for effective coordination. This framework hybridizes value function factorization learning and communication learning by introducing between agents" action selection and communication messages. Messages are parameterized in stochastic embedding space. To optimize communication, we introduce an additional information- theoretic regularizer to minimize the entropy of messages between agents. With these two regular- izers, our framework implicitly learn when, what, and with whom to communicate and also ensure communication to be both expressive (i.e., effectively reducing the uncertainty of agents' action- value functions) and succincl (i.e., only sending useful and necessary information). To optimize these regularizers, we derive a variational lower bound objective, which is easily integrated with existing value function factorization methods such as QMIX.
information-theoreiic regularizer for maximizing mutual information
We demonstrate the effectiveness of our learning framework on StarCraft 1f" unit micromanage- ment benchmark used in Foerster et al. (2017; 2018); Rashid et al. (2018); Samvelyan et al. (2019). Empirical results show that NDQ significantly outperforms baseline methods and allows to cut off more than 80% communication without sacrificing the performance. We also observe that agents can effectively learn to coordinate their actions at the cost of sending one or two bits of messages even in complex StarCraft II tasks.
BACKGROUND
In our work, we consider a fully cooperative multi-agent task that can be modelled by a Dec- POMDP (Olichoek et al. 2016 G = (I, A, P. R.S. O, n, 7), where I = {1,22..., n is the finite set of agents. s E the true state of the environment from which each agent draws an individual partial observation o E f2 according to the observation function O(s, i). Each agent has an cetioonobservation history т; T = (52 x A)" At each timestep, each agent selects an action a; E A, forming a joint action a E A", resulting in a shared reward r = R(ss a) for each agent and the next state s' according to the transition function P(s' 5, a). The joint policy π induces a joint action-value function: Q7r (T, a) E Beg.se (0. 12==0 7 ""r frrrs0=s, a0=a TT) aT,,, action-observation history and y E [0, 1) is the discount factor.
where T is the joint
Learning the optimal action-value value function encounters challenges in multi-agent settings. On the one hand, to properly coordinate actions of agents, learning a centralized action-value function Qiot seems a good choice. However, such a function is difficult to learn when the number of agents is large. On the other hand, directly learning decentralized action-value function Q; for each agent alleviates the scalability problem (Tan [993] Tampuu et al., (017). Nevertheless, such independent learning method largely neglects interactions among agents, which often results in miscoordination and inferior performance.
In between, value function factorization method provides a promising way to attenuate such dilemma by representing Qtor as mixing of decentralized Q; conditioned local information. Such method has shown their effectiveness on complex task (Samvelyan et al. (019)).
However, current value function factorization methods have been mainly focusing on full decompo- sition. Such decomposition reduces the complexity of learning Qtot by first learning independent Q: and putting the burden of coordinating actions on the mixing networks whose input is all Q 's and output is Qio-- For many tasks with partial observability and stochastic dynamics, mixing networks are not sufficient to learn coordinated actions, regardless of how powerful its representation ability is. The reason is that full decomposition cuts off all dependencies among decentralized action-value functions and agents will be uncertain about states and actions of other agents. Such uncertainty will increase as time goes by and can result in severe miscoordination and arbitrarily worse performance during decentralized execution.
StarCraft and StarCraft II trademarks of Blizzard Entertainment
2

---

Published as a conference paper at ICLR 2020
Qut
Maing Network
N
QCCr ap.mm))
Messages miy from other agints
R Entropy
[(A MoTT,.MM-OO Me-oj)
Agent i
Agent
Information
MLP
my
message my
ORU
MLP
GRU,
message my
Messages Wh-yy other agents
H(
Entrapy
Figure 1: Schematics of our approach. The message encoder generates an embedding distribution that sampled and concatenated with the current local history to serve an input the local action-value function. Local action values are fed into mixing network to to get an estimation of the global action value.
METHODOLOGY
In this section, we propose to learn nearly decomposable Q-functions (NDQ) via communication minimization, a new framework to overcome the miscoordination issue of full factorization methods.
In our learning framework (Fig. D. individual action-value functions condition on local action- observation history and, certain timesteps, messages received from few other agents. Messages from agent to agent j are drawn from multivariate Gaussian distribution whose parameters are given by encoder fm (T. 0z), where is the local observation-action history of agent and 0g are parameters of the encoder fm- Formally, message maj ~ [[ [T 0z). I), where is identity covariance matrix and the reasons will be discussed
an identity matrix. Here we in the next section. m{-i) is used to denote the messages sent to j from agents other than i. We learn nearly decomposable structure via learning minimized communication. We thus expect the communication to have the following properties:
Expressiveness: The message passed to one agent should effectively reduce the uncertainty in its action-valie function.
ii) Succinctness: Agents are expected send messages short as possible to the agents who need it and only when necessary.
To learn such a communicating strategy, we draw inspiration from variational inference for its proven ability in learning structure from data and endow a stochastic latent message space, which we also refer to as "message embedding". We impose constraints, which will discussed in detail in the next section, on the latent message embedding to enable an agent to decide locally which bits in message should sent according to their utility in terms of helping other agents make decisions. Agent j will receive input message m." that has been selectively cut, which it conditions the local action-value function (T ag. m. All the individual values are then fed into mixing network such as that used by QMIX (Rashid et al. 2018).
Apart from the constraints on the message embedding, all the components (the individual action- value functions, the message encoder, and the mixing network) are trained in an end-to-end manner by minimizing the TD loss. Thus, our overall objective is to minimize
( (O) + (O.).
where (T) = [r + max, (T' a': 0 Qroe(T a; 0)}² (0 are the parameters of a periodically updated target network as in DQN) is the TD loss O are all parameters in the model,
3

---

Published as a conference paper at ICLR 2020
and À is weighting term. We will discuss how to define and optimize Cc(Oe) to regularize the message embedding in the next section.
3.1 MINIMIZED COMMUNICATION OBJECTIVE AND VARIATIONAL BOUND
Introducing latent variables facilitates the representation of the message, but it does mean that the messages can reduce uncertainty in the action-value functions of other agents. To make mes- sage expressive, we maximize the mutual information between message and agent's action selection. Formally, we maximize Ie, (A3; MyJ (Ty. M((0)) where A, is agent j's action selection, Ty is the random variable of the local action-obe action-observation observation history of agent J. My and M(-i) are random vari- ables of mij and mi f)z- However, if this is the only objective, the encoder can easily learn to cheat by giving messages under different histories representations in different regions the latent space, rendering cutting off useless messages difficult. A natural constraint to avoid such representations to minimize the entropy of the messages. Therefore, our objective for optimizing communication of agent is to maximize:
12 Je(O2) 1100->0 M M(-ns) BHo (M,))]
where / is scaling factor trading expressiveness and succinctness.
This objective appealing because it agrees exactly with the desiderata that we impose on the message embedding. However, optimizing this objective needs extra efforts because computation involving mutual information is intractable. By introducing variational approximator, a popular technique from variational toolkit (Alemi et al. 2017), can derive lower bound for the mutual information term in Eq. 2 (a detailed derivation can be found in Appendix[A]
Ie. (Ay: MrJTT, M(-))) [-CE [p(AA|IT]l
E
where T T2..,...... ,T) the joint local history sampled from the replay buffer D, de( ne(AşiTT; Mi is the variational posterior estimator with parameters ç, and CE is the cross en- tropy operator. We share & among agents to accelerate learning.
Next we discuss how to minimize the term Ho, (Mi)). Directly minimizing this can cause the variances of the Gaussian distributions collapse to To deal with this numeric issue, we use the unit covariance matrix and try to minimize H (M,,)) minimizing H (Ma)) because H (May (T,) is the entropy of a multivariate Gaussian random variable and thus is constant (og(det(2ree)) have:
H (MrjiT,) instead. This is equivalent to
where Σ is unit matrix in formulation). Then
p(mayIT.) dm.,dT.. p(mu))
(M,)) (M,,IT.) = p(muy/T.)PTA log
(4)
We use a similar technique as for the mutual information term by introducing an distribution r(mig) to get upper bound of Eq. 4
H(M,) (M,)) < f p(muslT.)p(Ts) log p(moylT:) dmydd r(mg)
(5)
ETT-D (DKI (p( (M.T
This bound holds for any distribution r(Ma)). To facilitate cutting off messages, we use unit Gaus-
sian distribution (0, I). Combining Eq. and we get a tractable variational lower bound of our objective in Eq. 2
J.(O.)
ET (T [-c8 [p (4 -DDr))
(6)
We optimize this bound to generate an expressive and succinct message embedding. Specifically, we minimize:
(o.)) = EYw.D.M [CE (A,IT,, + D

---

Published as a conference paper at ICLR 2020
Intuitively, the first term, which we call the expressiveness loss, ensures that communication aims to reduce the uncertainty in action-value functions of other agents. The second term, called the succinctness loss, forces messages to get close to the unit Gaussian distribution. Since we set the covariances of the latent message variables to the unit matrix, this term actually pushes the means the origin of the latent space. Using these two losses leads to an
of the message distributions embedding space where useless messages distribute near the origin, while messages that contain important information for the decision-making processes of other agents occupy other spaces.
Note that the loss shown in Eq. 7 is used to update the parameters in the message encoder. In the meantime, all components (the individual action-value functions, the message encoder, and the mixing network) are trained in an end-to-end manner. Thus, the message encoder is updated by two gradients: the gradient induced by (O,) and the gradient associated with the TD loss LTD(O).
3.2 CUTTING OFF MESSAGES
Our objective pushes messages which can not reduce the uncertainties in action- value functions of other agents close to the origin of the latent message space. This naturally gives us hint on how to drop meaningless messages we can order the message distributions according to their means and drop accordingly. Note that since we use a unit covariance matrix for the latent message distribution, bits in message are independent. Thus, messages with various lengths. In this way, our method learns not only when and who (agent i does not communicate with agent j when all bits of maj are dropped) to communicate, but also what to communicate (how many bits are sent and their values). More details are discussed in Appendix B
can make decisions in bit-by-bit fashion and send
Our framework adopts the centralized training with decentralized execution paradigm. During cen- tralized training, we assume the learning algorithm has access to all agents' individual observation- action histories and the global state s. During execution, agents communicate and act in a decentral- ized fashion based on the learned message encoder and action-value functions.
4 RELATED WORKS
Deep multi-agent reinforcement learning has witnessed vigorous progress in recent years. COMA (Foerster al., 2018), MADDPG (Lowe al.) 2017), and PR2 (Wen al.) 2019) explores multi-agent policy gradients and respectively address the problem of credit assignment, learning in mixed environments and recursive reasoning. Another line of research focuses on value-based multi-agent RI., among which value-function factorization is the most popular method. Three rep- resentative examples: VDN (Sunehag et al. 2018), QMIX (Rashid et al. 2018), and QTRAN (Son) et al. 2019) gradually increase the representation ability of the mixing network. In particular, QMIX (Rashid et al. 2018) stands out as a scalable and robust algorithm and achieves state-of- the-art results StarCraf unit micromanagement benchmark (Samvelyan al. 2019).
Communication is a hot topic in multi-agent reinforcement learning. End-to-end learning with dif- ferentiable communication channel is popular approach now. [Sukhbaatar al. (2016); Hoshen (017): Jiang & Lu (2018); (iighh et al. (2019): [as et al.] (2019) focus on learning decentralized communication protocol and address the problem of when and who communicate. Foerster al. 2016); Das et al. (2017); Lazaridou et al. (2017); Mordatch & Abbeel (2018) study the emergence of natural language in the context of mulü-agent learning. IC3Net (Singh et al. 2019) learns gate to control the agents to only communicate with their teammates in mixed multi agentt environment. Zhang & Lesser (2013); Kim al.) (2019) study action coordination under limited communication channel and thus are related to our works. The difference lies in that they do not explicitly minimize communication. Social influence (Jaques et al. 2019) and InfoBot (Goyya et al. 2019) penalize message that has influence on policies of other agents.
Work that is most related to this paper is TarMAC (Das et al. 2019), where attention mechanism is used to differentiate the importance of incoming messages. In comparison, we use variation infer- ence to decide the content of messages and whether a message should be sent under the guidance of global reward signals. We compare our method with TarMAC and a baseline combining TarMAC and QMIX in our experiments. Related works on the task of StarCraft II unit micromanagement are discussed in Appendix C.2
5

---

Published as a conference paper at ICLR 2020
Taan Romarraa
Out 2ah0
(a) Task sensor
GM
Spima
Stapa (9)
Hamagg Catheis
Performance comparison
(c) Performance vs message drop rate
Figure 2: (a) Task sensor: (b) Performance comparison sensor, (c) Performance comparison when different percentages of messages are dropped. We measure the drop rate of our method in two ways: count by the number of messages (NDQ) or count by the number of bits (NDQ (bits)). QMIX (5M) is the performance of QMIX after training for million time steps.
to
10.00.2.17 77]
Others
Cl 10
Others
0.14
0.14
0.12
0.12
0.08
0.06
0.00
0.00
10
5
0.0
0.26
0.02
xx
0.02
X
0.02
-10
-10
10
(a) Target 2 present, B=1
(b) Target 2 present, 8=10-3
(c) Target 2 present, B=100
Co A
B 10.00-2 10110.00.3.94
Ouhers
Others
Kalr9roodd
0.14
0.12
0.14
0.10
0.10
10
0.08
10
6.08
10
0.04
0.04
0.04
0.02
0.02
0.02
-10
-10
-10
10
10
10
(d) Target absent, 8=1
(e) Target absent, 3=10-3
(f) Target absent, 3=10-5
Figure? 3: Message distributions learned by our method on sensor under different values of 3. (Mes- sages cut by bit. if < 2.0). A mean of means that the corresponding bit is below the cutting threshold and is not sent. When 3 = 10-3 NDQ learns the minimized communication strategy that is effective.
EXPERIMENTAL RESULTS
In this section, we show experiments answer the following questions: (i) Is the miscoordina- tion problem of full value function factorization methods widespread? (ii) Can our method learn the minimized communication protocol required by task? (iii) Can the learned message distributions reduce uncertainties in value functions of other agents? (iv) How does our method differ from com- munication with attention mechanism? (v) How does / influence the communication protocol? We will first show three simple examples to clarify our idea from different perspectives and then provide performance analysis StarCraftII unit micromanagement benchmark. For evaluation, all exper- iments are carried out with 5 random seeds and results are shown with 95% confidence interval. Details of the NDQ network architecture are given in Appendix B.2 Videos of our experiments on StarCraft II are available onlind
httpps:/sst google wwwgge..comvviiieooooooooooooo.e wtte////////////eeeeeeeeeeeeeeee

---

Published as a conference paper at ICLR 2020
320
Q1
02
4m
Texe Wn Dane
73-
9
Out
bp
Ous (abi)
gMiS- TaMMAC
OMIX
Optimal
(a) Task hallway
3.3
2.5
2000
(b) Performance comparison
(c) Performance vs message drop rate
Figure 4: Results hallway. (a. b) Task hallway and performance comparison. (c) Similar to
Fig. 2
we show performance comparison when different percentages of messages are dropped.
64
4
ANd 1006
AND TARO SWAIDRR FO0
A mod0
jood 5.51
ARA 524
534
0.2
1.0
6.00
6.00
200
800
(a)
t=1
(b) t=2
(c) t=3
(d) t=4
Figure 5: Message embedding representations learned by our method on hallway. mean of means that the corresponding bit is below the cutting threshold (μ=3) and is not sent.
We compare NDQ with the following baselines: (i) QMIX (Rashid et al. 2018:; (ii) TarMAC (Das
et al. 2019). QMIX and TarMAC are state-of-the-art full value function factorization and attentional communication methods, respectively. (iii) QMIX+TarMAC. We introduce the attentional commu- nication mechanism into the value function factorization paradigm by integrating the communication component of TarMAC into QMIX.
5.1 DIDACTIC EXAMPLES
We first demonstrate idea on three didactic examples: sensor, hallway, and independent search.
Sensor network is a frequently used testbed in multi-agent learning field (Kumar al.
2011 Zhang
& Lesser 201T). We use 3-chain sensor configuration in the task sensor (Fig. 2(a) Each sensor controlled by one agent, and they are rewarded for successfully locating targets, which requires two sensors to scan the same area simultancously when the target appears. At each timestep, target
appears in area I with possibility 1, and locating induces a team reward of 20; target appears with probability 0.5 in area 2, and agents are rewarded 30 for locating it. Agents can observe whether target present in nearby areas and need to choose one of the five actions: scanning north, east, south, west, and noop. Every scan induces cost of -5.
In the optimal policy, when target appears, sensor should turn itself off while sensors and expected to scan area to get the reward. And when target is absent, sensors I and 2 need to cooperatively scan area while sensor takes noop.
Sensor is representative of class of tasks where the uncertainties about the true states cause policies learned by full value function factorization method to be sub-optimal sensor has to know whether the target is present in area 2 to make decision. However, the mixing network of QMIX cannot provide such information. As a result, QMIX converges to a sub-optimal policy, which gets a team reward of 12.5 step on average (see Fig. 2(b)
7

---

Published as a conference paper at ICLR 2020
(a) 3b_vs_Ihlm
(b) 38_vs_5z
(c) 1o2r_vs.4r
(d) 5z.vs_lul
(c) 1o10b_vs.1r
MMM
Figure Snapshots of the StarCraft scenarios that we consider.
We are particularly interested in whether our method can learn the minimized communication strat- egy. Fig. 3 shows the latent message space learned by NDQ. When 3 10-3 agent learns to send a bit to tell agent 1 whether target 2 appears. In the meantime, the latent message distribution between any other pair of agents close the standard Gaussian distribution and thus is dropped. This result indicates that NDQ has discovered the minimized conditional graph and can explain why our method can still perform optimally when 80% of the messages are cut off (Fig, 2 When 3 becomes too large (1.0), all the message bits are pushed below the cutting threshold (Fig. 3(a) and 3(d). When 3 is too small (10-5), NDQ pays more attention on reducing uncertainties in Q-functions rather than compressing messages. Correspondingly, both agent 3 and agent 2 send message to agent 1 (Fig. 3(c) and (().. which is a redundant communication strategy.
The second example, hallway (Fig. 4a)). is Dec-POMDP with two agents randomly starting states 01 to am and d to b,, respectively. Agents can observe their position and choose to move left, move right, or keep still each timestep. Agents will win and get reward of 10 if they arrive at state simultaneously. Otherwise, if any agent arrives at earlier than the other, the team will not be rewarded, and the episode will begin. The horizon is max(m, + 10 avoid infinite loop.
Hallway aims show that the miscoordination problem of full factorization methods can be severe in multi-step scenarios. We set m and to 4 and show comparison of performance in Fig. 4(b) The miscoordination problem causes QMIX to lose about half of the games. We are again particularly interested in the message embedding representations learned by NDQ. We show an episode in Fig, 5]
Two agents begin a4 and ba. respectively. They first move left silently (t = and = 2) until agent B arrives at bi. On arriving b1, it sends bit whose value is 5.24 to A. After sending this bit, B stays at by and sends this message repeatedly until it receives bit from A indicating that A has arrived at a1. They then move left together and win. This is the minimized communication strategy. Taking advantage of this strategy, NDQ can still win in 100% of episodes when 80% of the communicating bits dropped (Fig. 4(c)
The third task, independent search, aims to demonstrate that NDQ can learn not to communicate in scenarios where agents are independent. Task description and results analysis are deferred to Appendix C.T
5.2 MAXIMUM VALUE FUNCTION FACTORIZATION IN STARCRAFT II
To demonstrate that the miscoordination problem of full decomposition methods is widespread in multi-agent learning, we apply our method and baselines to the StarCraft II micromanagement benchmark introduced by (Samvelyan et al. (2019), which is deseribed in detail in Appendix C.2 We further increase the difficulty of action coordination by i) reducing the sight range of agents from to 2; ii) introducing challenging maps with complex terrain or highly random spawning po-
8

---

Published as a conference paper at ICLR 2020
30 1n
35 without
102r 4r without
100
100
100 Muan
100
OREE- TTiMAC
Test Win Rate 55
OHEE TTeMAAC
QREE- TTriAA
hatz
Text Rate
60
60
60
40
40
0
20
10
20
(H)
(M)
message
1o10b
message
message
1.00
100
1.0-
Une
CMEE
Wi Fate
Toxt Win Rate %
Rate
RCO 10
40
40
40
(M)
Figure 7: Learning curves of our method and baselines when no message is cut for NDQ and QMIX+TarMAC.
111m message
message
1o2r
message
160
160
1.00
04t OMAATaaMAC
04A
OMRATaMAA
Test Win Rasz 96
Teo, Rate
L/// Liplin M
Test Ratt
40
40
40
A
0
20
Steps
Steps
MS. 1ul with message
1o10b vs. with message cut
with message
1.00
1.00
1.00
80
Toxt Win Rasa N
80
80
hasa
Tost Rata
40
40
40
20
20
20
10
20
Steps (M)
Stops
Steps (M)
Figure 8: Performance of our method and QMIX+TarMAC when 80% of messages are cut off. We also plot the learning curves of QMIX for comparison.
sitions for units. We test method the six maps shown in Fig. 6 Detailed descriptions of these scenarios are provided in Appendix C.2
We use the same hyper- parameter setting for NDQ on all maps: f is set to 10 10-5, A is set to 0.1, and the length of message may is set to 3. For evaluation, we pause training every 100k environment steps and run 48 testing episodes. Other hyper-parameters for NDQ are described in Appendix B.2
5.2.1 PERFORMANCE COMPARISON
We show the performance of our method and baselines when no message is cut in Fig. 7 superior performance of NDQ against QMIX demonstrates that the miscoordination problem of full factorization methods is widespread, especially in scenarios with high stochasticity, such as 1o2r vs 4r, 3b vs 1him, and 1o10b vs 1r. Notably, our method also outperforms the attentional
The

---

Published as a conference paper at ICLR 2020
1h1m with 100%
with 100%
1c2r with 100%
100
10
100
100
10
0914
Test Win nate 55
60
60
60
40
40
0
20
10
(M)
(M)
Tul message
1o10b_ Ir 100% message
HMM 100% message
1.00
100
100-
Une 04AkTarMAC
une
Wi Fate
Toxt Win Rate
Rate
40
40
40
20
Figure 9:
Performance our method and QMIX+TarMAC when 100% messages are cut off. We also plot the learning curves of QMIX for comparison.
communication mechanism (QMIX+ by large margin. Since agents communicate in both of these two methods and the same TD error is used, these results highlight the role of the constraints that we impose
message embedding. TarMAC struggles in all the scenarios. We believe that this is because it does not deal with the issue of reward assignment.
5.2.2 MESSAGE CUT OFF
To demonstrate that method can learn nearly decomposable Q-functions in complex tasks, cut off 80% of messages according to the means of distributions when testing and show the results in The results indicate that we omit more than 80% of communication without significantly
Fig. 8 affecting performance. For comparison, we cut off messages in QMIX+TarMAC whose weights are 80% smallest and find that its performance drops significantly (Fig 8). These results indicate that our method is more robust in terms of message cutting off compared to the attentional communication methods.
We further drop all the messages and show the developments of testing performance in Fig, 9 As expected, the win rates of NDQ decrease dramatically, proving that the superiority of our method when 80% of messages are dropped comes from expressive and succinet communication protocols.
CLOSING REMARKS
In this paper, we presented novel multi-agent learning framework within the paradigm of cen- tralized training with decentralized execution. This framework fuses value function factorization learning and communication learning and efficiently learns nearly decomposable value functions for agents to act most of the time independently and communicate when it is necessary for coordi- regularizers to minimize overall communication
nation. We introduce two informationn-therrtiin while maximizing the message information for coordination. Empirical results in challenging Star- Craft II tasks show that our method significantly outperforms baseline methods and allows us to reduce communication by more than 80% without sacrificing the performance. We also observe that nearly minimal messages (c.g., with one or two bits) are learned to communicate between agents in order ensure effective coordination.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Alexander A Alemi, lan Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottleneck. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial informatics, 9(1): 2012.
Abhishek Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. Learning cooperative visual dialog agents with deep reinforcement learning. Proceedings of the IEEE International Conference Computer Vision, pp. 2951-2960, 2017.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning, pp. 1538-1546 2019.
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2137-2145, 2016.
Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising experience replay for deep multi-agent reinforcement learning. In Pruceedings of the 34th International Conference on Machine Bourning-Volume 70, pp. 1146-1155. JMLR. org, 2017.
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Sergey Levine, and Yoshua Bengio. Infobot: Transfer and exploration via the infor- mation bottleneck. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Yedid Hoshen. Vain: Attentional multi-agent predictive modeling, In Advances in Neural Informa- fion Processing Systems, pp. 2701-2711, 2017.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, Dj Strouse, Joel 7. Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pp. 3040-3049, 2019.
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. In Advances in Neural Information Processing Systems, pp. 7254-7264 2018.
Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning to schedule communication multi-agent reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Akshat Kumar, Shlomo Zilberstein, and Mare Toussaint. Scalable multiagent planning using prob- abilistic inference. In Twenty-Second International Joint Conference on Artificial Intelligence, 2011.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitee environments. In Advances in Neural Information Processing Systems, pp. 6379-6390, 2017.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In Thirfy-Secondl AAAI Conference on Artificial Intelligence, 2018.
11

---

Published as a conference paper at ICLR 2020
Ann Nowé, Peter Vranex, and Yann-Michad De Hauwere. Game theory and multi-agent reinforce- ment learning. In Reinforcement Learning, pp. 441 440- Springer, 2012.
Frans A Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume Springer, 2016.
Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang- Multiagent httpeectionnalyyyorriiiii nets: Emergence of human-level coordination in learning to play stareraft combat games. arXiv preprint orX3vv/703. 10069, 2017.
Tabish Rashid, Mikayel Samvelyan. Christian Schroeder Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent rein- forcement learning. In International Conference on Machine Learning, pp. 4292-43000 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung. Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: 04043,, 2019.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Interna- fional Conference on Machine Learning, pp. 5887-5896, 2019.
Sainbayar Sukhbaatar, Rob Fergus, al. Learning multiagent communication with backpropaga- 2016.
tion. In Advances in Neural Information Processing Systems, pp.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value-decommosition networks for cooperative mulli-agent learning based team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2085 2087. Inter- national Foundation for Autonomous Agents and Multiagent Systems, 2018
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4)co1 2017.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pp. 330-337, 1993.
Nicolas Usunier, Gabriel Synnaeve, Zeming Lin, and Soumith Chintala. Episodic exploration for deep deterministic policies: An application to starcraft micromanagement tasks. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Chongjie Zhang and Victor Lesser. Coordinated multi-agent reinforcement learning in networked distributed pomdps. In Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011.
Chongjie Zhang and Victor Lesser. Coordinating multi-agent reinforcement learning with limited communication. Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems, pp. 1101-1108. International Foundation for Autonomous Agents and Multiagent Systems, 2013.
12

---

Published as a conference paper at ICLR 2020
APPENDIX
A VARIATIONAL BOUND ON MUTUAL INFORMATION
In order to enable messages to effectively reduce the uncertainties in action-value functions of other agents, we propose to maximize the mutual information between Aj and Myy. We borrow ideas from the variational inference literature and derive a lower bound of this mutual information regularizer.
Theorem 1. lower bound of mutual information La (A3s MTTTT M(-a)))
ET [-CE [p(A] T (A, [T, Mt )]]
(8)
where T is the local action-observation history of agent j. and T = (TT, T2,..,,,,, ,T,) is the joint local history sampled from the replay buffer DD qs (A(ATTT, Mf is the variational posterior estimator with parameters &.
Proof.
(Agi Mij [Tg, M(n)s)
(9)
-f p(ag: Tj. m.) log
pla;, megly, my 05)
da dm
(10)
p(a TTi mi m(-s))))))))))))))) (y))p(mm)ss)))))) [Tj, m(s))))))))))))))))))))))))))( pla, Ty. m²,") play [j, m(-s))) da,dr,dm,,"
= -s play: Ty, m" log
(11)
where p(a,) Tj, m3") ) is determined by the message encoder fn and Markov Chain:
p(a,1T5 p(a)l)..m²")
(12)
f a asls m dd
(13)
/ mfr=sls, (aammmm()))laatraa (According to [as 1 m3"111 [T]
(14)
((r))(m;"())p(( dr-
(15)
P(Ta. m))
We introduce qe (aga)) [T,, m² as variational approximation to p(ayltj, m."). Since
DKL (pa(p(alt,) (a m,"") 2 0.
(16)
we have
/ playlTz. m3r) log togp(as17 (Ty, m daa
(17)
2 f playlTy; m log q (agalll m ydaj-
(18)
Thus, for the mutual information term:
lo (Aji Mol M(-)))
(19)
f plas, Ty, m)") log q (T m
p(a, Tg. m(-)ss) da, dr,
(00)
f plas, Ty, m log q (a. ,7, m. daa
(21)
/ Ilay, T,, m3 log Log T,,, m(-)) )da dr,
(22)
-s (Tp(a//)p_(T___._p ppplllm m log (a, Tú my ddm
(23)
13

---

Published as a conference paper at ICLR 2020
plaj, Ty m((ns) (i
(24)
/ (rr)((m)(>))((ll>>>))) dd,
(According to [ag 1 my I=] (25)
Ho. (A₃ [T M(-s)
(26)
|f |llo)TTooo ydas
(27)
He. (A, M(11))
(28)
-ETDD, M.
(AT M]n)]]
(29)
Ho. [T Mc a)))-
(30)
Because He. (A, MM(-))) 2 0, we get the lower bound in Theorem 1.
B IMPLEMENTATION DETAILS
B. DETAILS OF MESSAGE DROPPING
In our methods, not only the number of messages but also the length of messages are minimized. In other words, send messages with varying lengths. However, messages are feed into an action- value function approximator at the recipient side, which requires inputs to have the same length. To solve this problem, we send masks indicating which bits are dropped along with the messages. To save channel bandwidth, masks are regarded as binary numbers, so each of them only consumes negligible log-sen scale space compared the length of messages. For the unsent bits, we fill in Os before feeding the messages into the local utility functions.
B.2 NETWORK ARCHITECTURE, HYPERFARAMETERS, AND INFRASTRUCTURE
We base our implementation on the PyMARL framework (Samvelyan et al. 2019) and use its default network structure and hyper-parameter setting for QMIX. For the message encoder, we use fully connected network with one 64-dimensional hidden layer and ReLU activation. For the posterior estimator qE. adopt fully connected network with two 20-dimensional hidden layers with ReLU activation. We train our models on NVIDIA RTX 2080Ti GPUs using experience sampled from 16 parallel environments. To benchmark NDQ, train all algorithms for 20 million time steps each StarCraft II unit micromanagement task and use the default hyper-parameter settings for baselines.
C EXPERIMENTAL RESULTS
C.I DIDACTIC EXAMPLE: INDEPENDENT SEARCI
In independent search, two agents are finding landmarks two independent 5 rooms for 100 time steps (see Fig. TOp. An agent is rewarded when it is on the landmark in its room.
Independent search an example where agents are totally independent. This task aims demon- strate that our method can learn not to communicate in independent scenarios. We show team per- formance in Table NDQ can achieve the optimal performance when agents do not communicate with each other.
Table 1: The average team reward gained in an episode on the task independent-search.
Ours
QMIX
TarMAC
TarMAC QMIX
No message is cut
96.0
96.0
96.0
96.0
100% messages are cut
96.0
96.0
14

---

Published as a conference paper at ICLR 2020
Room
Room 2
Target
Agent
Target 2
Agent 1
Figure 10: Task Independent-search. Two agents are hoth reward- and transition-indepeddent
C.2 STARCRAFT II
StarCraft unit micromanagement has attracted lots of research interests for its high degree of control complexity and environmental stochasticity, [ssunier et al. (2017) and Peng al. (2017) study this problem from a centralized perspective. In order to facilitate decentralized control, we use the setup introduced by Samvelyan al. (2019).
We first describe the scenarios that we consider in detail. We consider combat scenarios where the enemy units are controlled by StarCraft II built-in AI (difficulty level is set medium), and each of the ally units is controlled by learning agent. The units of the two groups can be asymmetric, and the initial placement is random. At each time step, each agent chooses one action from the discrete action space consisting of the following actions: noop, move ddirection], attack (enemy_ vemmyiid. and stop. Under the control of these actions, agents move and attack in a continuous map. A global reward that is equal to the total damage dealt on the enemy units is given at each timestep. Killing each enemy unit and winning a combat induce extra bonuses of 10 and 200, respectively.
Ab_vs_1hlmm 3 Banelings try to kill Hydralisk assisted by Medivac. 3 Banelings together just blow up the Hydralisk. Therefore, they should not give the Hydralisk rest time during which the Medivac can restore its health. Banelings have to attack at the same time to get the winning reward. This scenario is designed to test whether our method can learn a communication protocol to coordinate actions.
3s vs 5z. Stalkers encounter 5 Zealots on a map. Zealots can cause high damage but are much slower so that Stalkers have to take advantage of technique called kiting Stalkers should alterna- tively attack the Zealots and flee for a distance.
1o2r_vs_4r: An Overseer has found 4 Reapers. Its ally units, 2 Roaches, need get there and kill the Reapers to win. At the beginning of an episode, the Overseer and Reapers spawn at a random point on the map while the Roaches are initialized another random point. Given that only the Overseer knows the position of the enemy, a learning algorithm has to learn to deliver this message the Roaches effectively win the combat.
5z vs lul: 5 Zealots try to kill powerful Ultralisk. A sophisticated micro-trick demanding right positioning and attack timing has to be learned win.
MMM: Symmetric teams consisting of 7 Marines, 2 Marauders, and Medivac spawn at two fixed points, and the enemy team is tasked to attack the ally team. To win the battle, agents have to learn to communicate their health to the Medivac.
1o10b vs Ir: In a map full of cliffs, an Overseer detects a Roach. The teammates of the Overseer, 10 Banelings, need to kill this Roach to get the winning reward. The Overseer and the Roach spawn at a random point while the Banelings spawn randomly on the map. In the minimized communication strategy, the Banelings can keep silent, and the Overseer needs to encode its position and send it to the Banelings. We use this task to test the performance of our method in complex scenarios.
15


Reviews:
[{"review_id": "SygnB7pe5S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors propose a new method for multi-agent reinforcement learning by using nearly decomposable value functions. The main idea is to have local (agent specific) value function and one global value function (for all agents). They also try minimizing the required communication need for multi-agent setup. To this end they deploy variational inference tools and support their method with an experimental study.\n\nI found the paper interesting and empirical evaluation is good. However, my knowledge in the field is quite limited.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxgLm1q9B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tackles the collaborative multi-agent RL problem as the problem of finding almost-decentralized value functions, where the loss tries to minimize communications between the agents. The core idea is around maximizing the mutual information between each massage and the existing knowledge of its receiver. This way, redundant messages are naturally removed. The authors then assert an entropy regularization to (almost) prevent the agents from *cheating*. The paper is in general well-written and motivated. There are however certain issues that should be addressed. [The second one is my main issue.]  \n\n1. The term *message* has been used repeatedly but not defined. You should define precisely what you mean by a message in the background section. In particular, it is not clear from the text what a message looks like mathematically. You may also consider giving intuitive examples of how different ways of designing a message alters the behaviour in a given context before start talking about how to optimise them. \n\n2. Section 3.1 needs a revisit. I assume by \"optimal action policy of agent j\" you mean \"optimal policy of agent j\". Formally, an optimal policy is greedy to the optimal value function and is deterministic unless there exist multiple actions with same optimal value at a given state. Therefore, the mutual information is not mathematically well-defined since most of the time the optimal policy is deterministic and does not induce a probability distribution. \n\n2.1 What is the optimal value function of an agent? The agents are not optimizing their own value function, so even if your complex model converges, it does not imply optimality of agent-level value functions (and they shouldn’t be locally optimal). If by that you mean the agent-level value functions *after* convergence of $Q_tot$, then you need to clarify it to avoid the confusion. Even so, it is still not clear how you conceive the agent-level policies from these value functions. \n\n2.2. Minor: As for the notation, I found $A_j$ quite strange for a policy; you may want to consider something like $\\pi_j$. \n\n3. If $f_m$ is learnt, then message encoding is not stationary at least in the training time. It may make the training potentially become unstable. Specifically, there is no condition to assure stability of your model. Suggestion: your entropy regularization might be sufficient to induce stability. More discussion on this (or a simple experiment to show if an instability exists and will be alleviated with regularization) would be quite helpful.  \n\n4. If I understand it correctly [and it wasn't clear from the text], the encoder $f_m$ is conditioned on local history, which induces that all the agents must share same history shape (tensor-wise), which in turn means that all the local agents has to have same local state shape (computationally, they should for example have same output/internal-state shape in their neural networks). This sounds like a limitation, and it may only be OK in domains where agents are homogenous. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJgpxkoRFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a framework for combining value function factorization and communication learning in a multi-agent setting by introducing two regularizers, one for maximizing mutual information between decentralized Q functions and communication messages and the other for minimizing the entropy of messages between agents. The authors also discuss a method for dropping non-informative messages. They illustrate their approach on sensor and hallway tasks and evaluate their method on the decentralized StarCraft II benchmark. The paper addresses an interesting problem, and the authors show that their approach gives good performance compared to alternative approaches even when a large percentage of communication is cut off between the agents.\n\nQuestions/Comments:\n- Implementation details (e.g., hyper-parameters, model size) are missing from the paper.\n- The results are average over only 3 seeds, is this enough to compare different algorithms?\n- How should beta should be determined? \n- The authors present results when 80% of messages are cut off. What is the performance of the model when all communication is cut off for comparison?\n- How does the approach work in competitive environments?\n- The experimental results section is not well organized. The authors mention five question on page 6, but it is not very clear with examples/set of experiments address which question.\n- There are many spelling/grammar errors in the paper.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper extends recent value function factorization methods for the case where limited agent communication is allowed. The work is interesting and well motivated. The reviewers brought up a number of mostly minor issues, such as unclear terms and missing implementation details. As far as I can see, the reviewers have addressed these issues successfully in their updated version. Hence, my recommendation is accept.
Decision: Accept (Poster)