Episodic Multi-agent Reinforcement Learning with Curiosity-driven Exploration
Authors: Lulu Zheng
Jiarui Chen
Jianhao Wang
Jiamin He
Yujing Hu
Yingfeng Chen
Changjie Fan
Yang Gao
Chongjie Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: multi-agent reinforcement learning
multi-agent exploration
episodic control

Abstract: Efficient exploration in deep cooperative multi-agent reinforcement learning (MARL) still remains challenging in complex coordination problems. In this paper, we introduce a novel Episodic Multi-agent reinforcement learning with Curiosity-driven exploration, called EMC. We leverage an insight of popular factorized MARL algorithms that the ``induced" individual Q-values, i.e., the individual utility functions  used for local execution,  are the embeddings of local action-observation histories, and can capture the interaction between agents due to reward backpropagation during centralized training. Therefore, we use prediction errors of individual Q-values as intrinsic rewards for coordinated exploration and utilize episodic memory to exploit explored informative experience to boost policy training. As the dynamics of an agent's individual Q-value function captures the novelty of states and the influence from other agents, our intrinsic reward can induce coordinated exploration to new or promising states. We illustrate the advantages of our method by didactic examples, and demonstrate its significant outperformance over state-of-the-art MARL baselines on challenging tasks in the StarCraft II micromanagement benchmark.

Episodic Multi-agent Reinforcement Learning with Curiosity-driven Exploration
Lulu Zheng" Jiarui Chen 22 Jianhao Wang's Jiamin He*t, Yujing Hu², Yingfeng Chen',
Changjic Fan", Yang Gao", Chongjie Zhang'
Institute for Interdisciplinary Information Sciences, Tsinghua University, China Department of Computer Science and Technology, Nanjing University, China Fuui AI Lab, NetEase, China
Department of Computing Science, University of Alberta, Canada
z1119@mails. tainghua. edu.
chenjiaruilsmmil.. wtw.::/mmmma......ee..... edu. cn
Sehis0maalls.tsinggiin edu.
jiamin120ualberta.com
(huyujing, deenyingfeng01, f @corp
netease ..o.r.
com
gaoy@nju edu. cn
Chongjie@tsinghuaaeduu
Abstract
Efficient exploration in deep cooperative multi-agent reinforcement learning (MARL) still remains challenging in complex coordination problems. In this paper, we introduce novel Episodic Multi-agent reinforcement learning with Curiosity-driven exploration, called EMC. We leverage an insight of popular factor- ized MARL algorithms that the "induced" individual Q-values, i.e., the individual utility functions used for local execution, are the embeddings of local action- observation histories, and can capture the interaction between agents due to reward backpropagation during centralized training. Therefore, we use prediction errors individual Q-values as intrinsic rewards for coordinated exploration and uti- lize episodic memory to exploit explored informative experience to boost policy training. As the dynamics an agent's individual Q-value function captures the novelty of states and the influence from other agents, intrinsic reward can induce coordinated exploration to new or promising states. We illustrate the advantages of our method by didactic examples, and demonstrate its significant outperformance over state-of-the-aa MARL baselines on challenging tasks in the StarCraft II micromanagement benchmark.
Introduction
Cooperative multi-agent reinforcement learning (MARL) has great promise to solve many real-world multi-agent problems, such as autonomous cars [ and robots [2). These complex applications post two major challenges for cooperative MARL: scalability, i.e., the joint-action space exponentially grows as the number of agents increases, and partial observability, which requires agents to make decentralized decisions based on their local cecton-observation histories due communication constraints. Luckily, a popular MARL paradigm, called centralized training with decentralized execution (CTDE) [3, is adopted to deal with these challenges. With this paradigm, agents' policies are trained with access to global information in a centralized way and executed only based on local
"Equal contribution. Work performed while visiting Tsinghua Univeristy.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

histories in a decentralized way. Based on the paradigm of CTDE, many deep MARL methods have been proposed, including VDN QMIX [5]. QTRAN 6 and QPLEX
core idea of these approaches is to use value factorization, which uses neural networks to represent the joint state-action value as function of individual utility functions, which can be referred to individial Q-values for terminological simplicity. For example, VDN learns a centralized but factorizable joint value function Qto represented as the summation of individual value function Qi- During execution, the decentralized policies can be easily derived for each agent by greedily selecting actions with respect its local value function Qi- By utilizing this factorization structure, an implicit multi-agent credit assignment is realized because Q, is represented as a latent embedding and is learned by neural network backpropagation from the total cemporal-differees global reward signal, rather than on a local reward signal specific to agent i. This value factorization technique enables value-based MARI, approaches, such as QMIX and QPLEX, to achieve state-of- the-art performance in challenging tasks such as the StarCraft unit micromanagement [8].
error on the single
Despite the current success, since only using simple e-greedy exploration strategy, these deep MARL approaches are found ineffective to solve complex coordination tasks that require coordinated and efficient exploration 77.. Exploration has been extensively studied in single-agent reinforcement learning and many advanced methods have been proposed, including pseudo-counts (9] [O curiosity [T] 12, and information gain [3]. However, these methods cannot adopted into MARL directly, due to the exponentially growing state space and partial observability. leaving agent exploration challenging. Recently, only a few works have tried address this problem. For instance, EDTI [4 uses influence-based methods to quantify the value of agents' interactions and coordinate exploration towards high-value interactions. This approach empirically shows promising results but, because of the need to explicitly estimate the influence among agents, it is not scalable when the number of agents increases. Another method, called MAVEN [15), introduces hierarchical control method with a shared latent variable encouraging committed, temporally extended exploration. However, since the latent variable still needs to explore in the space of joint behaviours [15. is not efficient in complex tasks with large state spaces.
In this paper, we propose novel multi-agent curiosity-drin -drivee exploration method. Curiosity is a type of intrinsic motivation for exploration, which usually uses prediction errors on different spaces (e.g., future observations [2]. actions [11] or learnable representation [16] as reward signal. Re- cently, curiosity-drive methods have achieved significant success in single-agent reinforcement learn- ing 122 [8) However, curiosity-dree -driven methods face a critical challenge MARL: in which space should we define curiosity? The straightforward method is to measure curiosity the global observa- tion [12 or joint histories in a centralized way. However, is inefficient to find structured interaction between agents, which seems too sparse compared with the exponentially growing state space when the number of agents increases. In contrast, if curiosity defined as the novelty of local observation histories during the decentralized execution, although scalable, it still fails to guide agents to coordi- nate due to partial observability. Therefore, we find a middle point of centralized curiosity and decen- tralized curiosity, i.c., utilizing the value factorization of the state-of- ooffthe-an art multi-agent Q-learning approaches and defining the prediction errors of individual Q-value functions as intrinsic rewards. The significance of this intrinsic reward is two- fold: 1) it provides a novelty measure of joint observation histories with scalability because in- dividual Q-values are latent embeddings (i.e., an effective state abstraction (19] of observation histories in factorized multi-agent Q-learning (c.g.. VDN or QPLEX); and 2) as shown in Figure [] it captures the influence from other agents due the implicit credit assignment from global reward signal during centralized train- ing 20). and biases exploration into promising states where strong interdependence may lie between agents. Therefore, with this novel intrinsic reward, our curiosity-driven method enables efficient, diverse, and coordinated exploration for deep multi-agent Q-learning with value factorization.
Global Curiosity
Global Information
Centralize Trai Training Qcocal
Mixing Gradients
Individual
Q-values
2
QN
Local
Loca actionn voservet in Notolee
Local action- abserce or Noorle
Local action- obsenstian Nitories
Curiosity
Figure 1: CTDE Framework
Besides efficient exploration, another challenge for deep MARL approaches is how to make the best use of experiences collected by the exploration strategy. Prioritized experience replay based on TD errors shows effectiieness in single-agent deep reinforcement learning. However, does
2

---

not carry this promise in factorized multi Q-learning, since the projection error induced by value factorization is also fused into the TD error and severally degrades the effectiveness of the TD error as measure of the usefulness of experiences. To efficiently use promising exploratory experience trajectories, we augment factorized multi-agent reinforcement learning with episodic memory [21 [22] This memory stores and regularly updates the best returns for explored states. We use the results in the episodic memory to regularize the TD loss, which allows fast latching onto past successful experience trajectories collected by curiosity exploration and greatly improves learning efffciency. Therefore, we call our method Episodic Multi-agent reinforcement learning with Curiosity- driven exploration, called EMC.
We evaluate EMC in didactic examples, and broad set of StarCraft II micromanagement benchmark tasks [8. The didactic examples along with detailed visualization illustrate that our proposed intrinsic reward can guide agents' policies to novel or promising states, thus enabling effectively coordinated exploration. Empirical results on more complicated StarCraft II tasks show that EMC significantly outperforms other multi-agent sate-of-the-aad baselines.
2 Background
2.1 Dec-POMDP
cooperative multi-agent task can be modelled as a Dec-POMDP [3]. which is defined by tuple G =<<<>> I, A. P. R, f2, O, n. 7/ >, where I is the sets of agents, S is the global state space, A the finite action set, y E [0, 1) is the discount factor. We consider a partially observable setting in a Dec-POMDP, i.c., at each timestep, agent i E I only has access to the observation E S? drawn from the observation function (ss, i). Besides, each agent has an action-observation history E T= (52 x A)" x S2 and constructs its individual policy to jointly maximize team performance. With each agent i selecting an action a; E A, the joint action a = [a, E A = A leads to a shared reward
R(s, a) and the next state s' according to the transition distribution P(s'|s, a). The formal objective function to find a joint policy æ that maximizes a joint value function VT (s) = B 2'r = sp, T). or joint action-value function Q" (s, a) =r(s, a) + 7B [V* ( (s')).
2.2 Centralized Training With Decentralized Execution (CTDE)
CTDE is promising paradigm in deep cooperative multi-agent reinforcement learning B 23 24 where the local agents execute actions only based on local observation histories, while the policies can be trained in centralized manager which has access to global information. During the training process, the whole team cooperate to find the optimal joint action-value function Qior a) r(s, a) + [max,' (s a')l Due partial observability, we use Qtor a; 0) instead of Qior (s, a; 0). where T E T = T Then the Q-value neural network will be trained to minimize
the following expected TD-error:
[(0) Er (V
Quae a))]²
(1)
where D is the replay buffer and 0 denotes the parameters the target network, which is periodically updated by 0. And ((r';0-) is the one-step expected future return of the TD target. Local agents can only obtain local ocsionnobservatiio history and need inference based on individual Q-value functions (r(T, oz). Therefore, many works have made efforts in finding the factorization structures between joint Q-value functions Qtot and individual Q-functions Qi(Ti, a;) 5
3 Related Work
-uriosity-driven Exploration Curiosity-driven exploration has been well studied in single-agent reinforcement learning. Previous literature [25] 26 has provided good summary in this topic. Recently, curiosity-driven methods have made great progress in deep reinforcement learning. For example, some works use pseudo-state counts to get intrinsic rewards [9] 10 27] instead of count- based methods to get better scalability. Stadie et al. (28) use prediction errors in the feature space of an auto-encoder to measure the novelty of states and encourage exploration. On the other hand, Mohamed and Rezende (29 propose to use empowerment, measured by the information gain based on the entropy of actions, as intrinsic rewards for exploring novel states efficiently. Another information- [13] tries to maximize information gain about the agent's 's belief of the environment's
based method
3

---

dynamics as an exploration strategy. ICM [1] learns an inverse model which predicts the agent's s action given its current and next states and tries to predict the next state in the learned hidden space by current state and action. RND [2] uses curiosity as intrinsic rewards in a simpler but effective way, which uses fixed randomly initialized neural network as a representation network and directly predicts the embedding of the next state. Different from these methods, we are the first to propose an advanced curiosity-driven exploration method in MARL setting for diverse and coordinated exploration.
Multi-agent Exploration Although single-agent exploration is extensively studied and has achieved considerable success, few exploration methods were designed for cooperative MARL. Bargiacchiet al. (30) proposes an exploration method that can only be used in repeated single-stage problems. Jaques et al. [I defines intrinsic reward by "social influence" to encourage agents to choose actions that can influence other agents' actions. Iqbal and Sha [32] uses various simple exploration methods to learn simultaneously and then put the samples of every method in a shared buffer to achieve the coordinated exploration. Wang
14] use mutual information (MI) to capture the interdependence of the rewards and transitions between agents. MAVEN [I5] is the state-of-the-arr exploration method in MARL that uses a hierarchical policy to produce a shared latent variable and learns several state-action value functions for each agent. These works, although important, still face the challenge of achieving scalable and effective multi-agent exploration.
Our work is also related to episodic control reinforcement learning, which is usually adopted in single-agent settings for better sample efficiency. Previous works propose to use episodic memory in mear-deterministic environment (3 Model-free episodic control R4 uses a completely non-parametric table to keep the best Q-values of state-action pair
Episodic Control
aabular-based
memory and uses a k-nearest-neighbor fashion to find the sequence of actions that so far yielded the highest return from a given start state in the memory. Recently, several extensions have been proposed to integrate episodic control with parametric DQN. Gershman and Daw (37 uses episodic memory to retrieve samples and then average future returns to approximate the action values. EMDQN 21 uses fixed random matrix as representation function and uses the projection of states as keys to store the information of episodic memory into a non non-parametric model. Using the episodic-memenn based target as a regularization term to guide the training process, the performance of EMDQN is significantly improved compared with the original DQN. Despite the fruitful progress made in single-agent episodic reinforcement learning, few works study episodic control in multi-agent setting. To the best of our knowledge, we are the first to utilize the mechanism of episodic control in deep multi-agent reinforcement learning.
Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration
In this section, we introduce EMC, a novel episodic multi-agent exploration framework. EMC takes prediction errors of individual Q-value functions intrinsic rewards for guiding the diverse and coordinated exploration. After collecting informative experience, we leverage an episodic memory to memorize the highly rewarding sequences and use it as the reference of one-step TD target boost multi-agent Q-learning. First, we analyze the motivations for predicting individual Q-values, then we introduce the curiosity module for exploration. Finally, we describe how utilize episodic memory to boost training.
4.1 Curiosity- Exploration by Predicting Individual Q-values
As shown in Figure in the paradigm of CDTE, local agents make decisions based on individual Q-value functions, which take local observation histories as inputs, and are updated by the centralized module which has access to global information for training. The key insight is that, different from single-agent cases, individual Q-value functions in MARL are used for both decision-making and embedding historical observations. Furthermore, due to implicit credit assignment by global reward signal during centralized training, individual Q-value functions Q (T,,.) -) are influenced by environment as well as other agents' behaviors. More concretely, it has been proved by Wang et al. (20) that, when the joint Q-function Qtot is factorized into linear combination of individual Q-functions Q. i.e., Q(t+")(( (T, a) EN (((+1) (Ts; 0s), then Q ( (T,, o;) has the following closed-form solution:

---

Q 2 (T)
E (T0rL
(T (ppp))))))))
evaluation of the individuad action
(2)
E aa (ar-pp((A- ( (T', (T' twx(
counterfactual baseline
where y (T, = + maxa Q (T', (T') denotes the expected one-step TD target, and (P()))) denotes the conditional empirical probability of in the given dataset D. The notation TT denotes (Tí...., T ,, Ti, T++1 TT)) and --, denotes the elements of all agents except for agent i. A (T,) denotes the set of trajectory histories that may share the same latent-state trajectory as T₁. The residue term w [w:]i=1 is an arbitrary function satisfying VT r. Ei= wi((T))
Eq. 2 shows that by linear value factorization, the individual Q-value Qi a,) is not only decided by local observation histories but also influenced by other agents' action-observation histories. Thus predicting Q; can capture both the novelty states and the interaction between agents and lead agents to explore promising states. Motivated by this insight, in this paper, factorization module separate from the inference module to learn the individual value function Qi, and use the prediction errors of Qi intrinsic rewards guide exploration. In this paper, we define the prediction errors of individual Q-values as curiosity and propose our curiosity-drive exploration module as follows.
linear value
Value Factorization Framework
(n "e9"i--.Loo
[o/]==
Mixing
Environment
Contral ers
Qex (Q,, Oz. QN)
[q]]_=
Q
Gradients
Memory Loss
Trajectories
Loss 1-1
Predictors
Distance Fanction
bntnt (I,
State
Random Projection
Q
Gradients [0 (+,-))
(s))()))
MC
Targets @ueat
Mixing
Q
(r. r', s, ext)
Replay buffer
Curiosity Module
Episodic Memory (d)
(b)
(c)
Figure 2: An overview of EMC's framework
Figure 2p demonstrates the Curiosity Module, separated from the inference module (Figure 21. The curiosity module consists of four components: (i) The centralized training part with linear value factorization, which shares the same implementation rewards y eatt from the environment; (ii) the Target for prediction, i.e., the corresponding individual
VDN 44, but only trained with extrinsic
Q-values Quntt represented by recurrent Q-network; (iii) Predictor (T:), which is used for predicting Q and shares the same network architecture as Target QeIt, and (iv) Distance Function, which measures the distance between Quaa and Q:. c.g., Le distance. The predictors are trained by minimizing the Mean Squared Error (MSE) the distance in an end-to-end manner. For stable training, we use the soft-update target (38 of Q+2t to smooth the outputs of the targets. In general, (ii) trained with (i) and outputs individual Q-values, while (iii) is trained with (ii) and (iv), and aims to predict the soft-update target of individual Q-values. Motivated by the implicit credit assignment of linear value factorization (Eq. (2. the curiosity module predicts individual Q-values [ in
linear factorization, i.e., Qeat i= Q Then the curiosity- driven intrinsic reward is generated
N
by the following equation:
N
iin
( - (T..))
(3)
5

---

This intrinsic reward is used for the centralized training of the inference module, as shown in Figure 2
Linference (0) = E ET. [(U(T a) - Qoo a; 0))
(4)
where y(T, a) = rext + Brent + maxa' Qtot (T', a'; 0-), denoting one step TD target of the inference module, and / is the weight term of the intrinsic reward. We use a separate training model for inference (Figure 2a) to avoid the accumulation of projection errors of Q, during training.
The independence of inference module leads to another ad advantage, that EMC's architecture can be adopted into many sattee-aaattzziin-bbaee based multi-agent algorithms which utilize the CDTE paradigm, i.e., the general function f in Figure 2a can indicate specific (linear, monotonic and IGM) value factorization structures in VDN [., QMIX [3], and QPLEX (7], respectively. In this paper, we utilize these state-of to-tte--rttttttt algorithms for the inference module. With this curiosity- -riven bias plugged into ordinary MARL algorithms, EMC will achieve efficient, diverse and coordinated exploration.
4.2 Episodic Memory
Equipped with efficient exploration ability, another challenge is how to make the best use of good trajectories collected by exploration effectively. Recently, episodic control has been widely studied in single-agent reinforcement learning 21 22), which can replay the highly rewarding sequences, thus boosting training. Inspired by this framework, we generalize single-agent episodic control to propose a multi-agent episodic memory, which records the best memorized Monte-Carlo return in the episode, and provide a memory target H as reference to regularize the ordinary one-step inference TD target estimation in the inference module (Figure 2
Lmemory ( = EE. (H Qto((T a: 0))]
(5)
However, different from the single-agent episodic control, the action space of MARL exponentially grows as the number of agents increases, and partial observability also limits the information of individual value functions. Thus, maintain our episodic memory by storing the state-value function on the global state space and utilizing the global information during the centralized training process under the CTDE paradigm. Figure 2 shows the architecture of the Episodic Memory. We keep a memory table M to record the maximum remembered return of the current state, and use a fixed random matrix drawn from Gaussian distribution low-dimensional vectors d(s): : S + R², which are used as keys to look up corresponding global state value function H( ((())). When our exploration method collects a new trajectory, we update our memory table M as follows:
representation function to project states into
H H(((s))) = max((H((()))), R, a.))
(f(l(ss) - o <
(6)
Rs as)
otherwise
where ((+:) is o(s;)² 's nearest neighbor in the memory M, 8 a threshold, and R(st, az) represents the future return when agents taking joint action a, under global state 8; at the t-th timestep in a new episode. In our implementation, Φ(st) E M is indeed evaluated approximately based on the embedding distance. Specifically, when the key of the state o(st) is close enough to one key in the memory, we assume that ((s)) E M and find the best memorized Monte-Carke return correspondingly. Otherwise, we think ((s)) & M and record the state's return into the memory. Leveraging the episodic memory, we can directly obtain the maximum remembered return of the current state, and the one-step TD memory target H reference to regularize learning:
Ho((s), ar) = (r( a:) + (HH(((()))))))
Thus, the new objective function for the inference module is:
Lroral (0) Linforence (0) + XL mmmmoryy( (0)
EE.T. ( a) Qior (T, a1))) X (ō(si), a.) Qroi - (8)
where the weighting term to balance the effect of episodic memory's reference. Using the maximum return from the episodic memory to propagate rewards, we can compensate for the disadvantage of slow learning induced by the original one-step reward update and improve sample efficiency.

---

5 Experiments
In this section, we will conduct a large set of empirical experiments for answering the following questions: (1) Is exploration by predicting individual Q-value functions better than exploration by decentralized curiosity or global curiosity (Section 5.1 (2) Can our method perform efficient coordinated exploration in challenging multi-agent tasks (Section 5.2 5.3 (3) If so, what role does each key component play for the superior performance (Section 5.4 (4) Why do we choose to predict target Q for generating intrinsic rewards rather than other choices (Section 5.4b? We will propose several didactic examples and demonstrate the advantage of our method in coordinated exploration, and evaluate our method on the StarCraft II micromanagement (SMAC) benchmark [8] compared with existing state-of-the-ee multi-agent reinforcement learning (MARL) algorithms: QPLEX [7].
Weighted Reighted-QMIX [39, QTRAN 6 QMIX 5 VDN RODE and MAVEN 15]
5.1 Didactic Example
Figure shows an 11 x 12 grid world game that requires coordinated exploration. The blue agent and the red agent can choose one of the five actions: [up, down, left. right, stay/ at each time step. The wall shown in the picture isolates the two agents, and one agent cannot be observed by the other until it gets into the shaded area. The two agents will receive a positive global reward r 10 if and only if they arrive at the corresponding goal grid (referred to the character G in Figure 3) at the same time. If only one arrives, the incoordination will be punished by a negative reward
To evaluate the effectiveness of our curiosity-driver exploration, we implement our method into QPLEX, QMIX, and VDN (denoted as EMC- QPLEX. EMC-QMIX, and EMC-VDN, respectively) and test them in this
Wall
toy game compared with the state-of-the-ard MARL algorithms: VDN,
Moving Agent
IQL, QMIX, and QPLEX. Moreover, to demonstrate the motivation of predicting individual Q-functions, we add two more baselines: QPLEX with the prediction error of global state as intrinsic rewards (denoted as OPLEX-Global, and QPLEX with the prediction error of local joint histories as intrinsic rewards (denoted as QPLEX- use a fixed network to project the inputs into latent embedding. then predict the latent embedding to generate intrinsic reward, just like the Random Network Distillation (RND) [12.. We test different punishment degrees, i.e., different ps (which are deferred to Appendix C), and the results show QPLEX- Globall and QPLEX-Local are effective enough for exploration when p is relatively small. However, as increases, the task becomes more challenging since it requires sufficient and coordinated exploration. In Figure 4 median test win rate of all methods over random seeds when = 2, and only our methods can learn the optimal policy and win the game, while other methods failed.
Moving Agent
Both of them
Figure 3: Coordinated Toygame
we show the
To understand this result better, we have made several visualisations to demonstrate our advantage in coordinated exploration. Figure 4 shows the heatmaps of visitation and intrinsic reward by EMC-QPLEX, APLEX--lobal. and LoLllL During the early stage of training. all methods uniformly explore the whole area (Figure 41). As the exploration progresses, the global curiosity (QPLEX-Global) encourages agents to visit all configurations without bias, which is inefficient and fail to leverage the potential locality influence between agents (Figure 4b), resulting in extrinsic rewards beginning to dominate the behaviors (b) On the other hand, the visitation heatmap of QPLEX-Local shows the decentralized curiosity encourages agents to explore around the goal grid, but it cannot promise to encourage agents to coordinate and gain the reward due to the partial observability decentralized execution. In contrast, the heatmap of intrinsic reward for EMC-QPLEX shows that predicting individual Q-values will bias exploration into areas where individual Q-values are more dynamic due to the potential correlation between agents. Therefore, QPLEX-Local and OPLEX-Global both fail in this task (Figure 4)), while our methods are able to find the optimal policy. This didactic example shows the global curiosity or local curiosity may fail to handle complex tasks where coordinated exploration needs to be addressed. While since individual Q-values Q; are the embeddings of historical observations, and are dynamically updated by the backpropagation of the global reward signal gained through cooperation during centralized training. Thus Q, can implicitly reflect the influence from the environment and other agents, and predicting Qi can capture valuable and spare interactions among agents and bias exploration into new or promising states.
7

---

Phase
Phase 2
Phase Optimal Policy
Median Test Win Rate %
Uniform
Capture
Exploration
Interaction
100
EMC-QPLEX (Ours)
80
EMC QMIX (Ours)
(Ours)
60
QPLEX
QMIX
40
QPLEX-Global OPLEX-Local
20
0
0.0M
0.1M
0.2M
0.3M
0.4M
0.5M
OOL
25Ch
Timesteps
EMC-OPLEX
QPLEX-Local
OPLEX-Global
405. Visitation
Aittiii Intrinsic Reward
Hiilliime Visitation
sa raa Intrinsic Reward
2 Visitation
2 Intrinsic Reward
(a)
(b)
(c)
Figure 4: The heat map of gridworld game.
5.2 Predator Prey
Predator-Prey is a partially-observable multi-agent co- ordinated game with miscoordination penalties used by WQMIX [39]. As shown in Figure 55) since extensive exploration is needed to jump of the local optima, WQMIX the only baseline algorithm to find the optimal policy, due its shaped data distribution which be seen as a type of exploration. Other stateoof-the-ar multi- agent Q-learning algorithms, such as QPLEX and QMIX, fail to solve this task. For MAVEN, QPLEC-Global and QPLEX-Local, although equipped with improved explo- ration ability, they still failed to address coordination due to uniform exploration nature or partial observability. However. plugged with EMC, EMC-VDN, EMC-QMIX, and EMC-QPLEX can guarantee coordinated exploration effectively and achieve good performance.
40
Median Test Return
-40
-80
-120
160
QPLEXX-C444)
0.0M
0.2M
0.4M
0.6M
0.8M
1.0M
Timesteps
Figure 5: The performance of Predator Prey.
5.3 StarCrafth Micromanagement (SMAC) Benchmark
StarCraft II Micromanagement (SMAC) is popular
benchmark in MARL 4
5
40 39
We conduct ex-
EMC (muz))
QFLEX
periments
benchmark tasks StarCraft II, which
# Maps Best (out of (7)
contains 14 popular tasks proposed by SMAC 8 and three
OW-QMIX
RODE
more super hard cooperative tasks proposed by QPLEX [7].
In the micromanagement scenarios, each unit is controlled by an independent agent that must act based on its own local observation, and the enemy units are controlled by a built-in AI.
For evaluation, we compare EMC with the state-of siic--ofthhr the- art algorithms: RODE [40], QPLEX 77, MAVEN (15), and the two variants of QMIX 5) CW-QMIX and OW- QMIX (39]. All experimental results are illustrated with the median performance and 25-75% percentiles. Figure 6
0%
40%
60%
100%
Training Percentage
Figure 6: The number of scenarios in which the algorithm's median test win rate is the highest by as least 1/32.

---

EMC (Ours)
GPLEX
QTRAN
QMIX
VON
CW-QMMIX
OW-QMIX
Median Test Win Rate %
Median Test Win Rate %
Median Test Win Rate %
100
100
100
80
80
80
60
60
40
0.0M 0.4M 0.8M 1.2M 1.6M 2.0M Timesteps
0.0M 0.4M 0.8M 1.2M 1.6M
2.0M
0.0M 0.4M 0.8M 1.2M 1.6M 2.0M
Timesteps
Timesteps
(a) corridor
(b) 3s52_vs V3_362
(c) 6h_ vs_8z
Median Test Win Rate %
Median Test Win Rate %
Median Test Win Rate %
100
100
80
60
60
40
40
20
20
20
0
0.0M 0.4M 0.8M 1.2M 1.6M 2.0M Timesteps
0.0M 0.4M 0.8M 1.2M 1.6M 2.0M Timesteps
0.0M
0.4M 0.8M 1.2M 1.6M 2.0M Timesteps
(d) 1c3s8z vs_ 1c3992
(e) 5s10z
(l) 7s7z
Figure 7: Results of super hard maps in SMAC.
shows the overall performance of the tested algorithms in all these 17 maps. Due to the effective exploration with episodic memory which can efficiently use promising exploratory experience trajectories, EMC the best performer on up to 6 tasks, underperforms on just three tasks, and ties for the best performer on the rest tasks.
The advantages of our algorithm can be mainly illustrated by the results of the six hard maps which need sufficient exploration shown in Figure 7 The three maps in the first row are super hard, and solving them needs efficient, diverse and coordinated exploration. Thus, we can find that the EMC algorithm significantly outperforms other algorithms in corridor and 3s5z_vs vs 3s6z, and also achieves the best performance (equal to RODE) in 6h vs 8z. To the best of our knowledge, this will be the state-of-the-an results in corridor and 3s5z_ vs vs_3s6.. For the remaining three maps in the second row ( /c3s82_vs _c_3_33,,,,,,,, 5s/0z, and 7s72), where other baselines can also find winning strategies, due to the boost learning process via episodic memory along with efficient exploration, our algorithm EMC still performs the best the three maps, with fastest learning speed and the highest rates achieved.
5.4 Ablation Study
To understand the superior performance of EMC, we carry out ablation studies to test the contribution of its two main components: curiosity module and episodic memory. Following methods are included in the evaluation: (i) EMC without curiosity module (denoted by EMC-wo-C); (ii) EMC without episodic memory component (denoted by EMC-wo-M); (iii) QPLEX, which can be considered as EMC without the episodic memory component nor the curiosity module, provides a natural ablation baseline of EMC.
Figure 8 bbcc shows that in easy exploration maps, both EMC and EMC-wo-C achieve the sfate-of-the- art performance, which implies that in the easy tasks, sufficient exploration can be achieved simply by the popular e-greedy method. However, in super hard exploration maps (Figure g (a)), EMC-wo-C cannot solve this task but EMC has excellent performance. These empirical experiments show that the curiosity module plays a vital role in improving performance when sufficient and coordinated exploration is necessary. On the other hand, making the best use of good trajectories collected by exploration is also essential. As shown Figure EMC with episodic memory enjoys better sample efficiency than EMC-wo-Mr Mo in challenging (Figure Xh) and easy exploration tasks (Figure (Bb-c)). In general, the curiosity module and the episodic memory complement each other, and efficiently using promising exploratory experience trajectories leads the superior performance of EMC.
Like single-agent curiosity or RND 12 exploration methods, our approach looks simple yet effective. In addition, its design choices do not look straightforward before we know how to do it right.

---

Median Test Win Rate %
Median Test Win Rate %
100
Median Test Win Rate %
100
LMC (Chrs)
EMC
80
EMC-a0-C
EMC we
EMC-ww-M
EMC M
QPLEX
60
QPLEX
60
EMC-wo- C
20
20
QPLEX
0.0M 0.4M 0.8M 1.2M 1.6M
2.0M
0.0M 0.2M 0:1M 0.6M 0.8M 1.0M
0.0M 0.2M 0:4M 0.6M 0.8M 1.0M Timesteps
Timesteps
Timesteps
(a) corridor
(b) 2s3z
(c) 3s5z
Figure 8: Ablation study on the two major components.
Median Test Win Rutt %
Mean Test Win Rate %6
100
100
(Oum)
(Qumi
Meam Test Win Rate
(Ound)
NO
EMC-TD
NO
EMC-TD
80
EMC-TD
60
LMC-DDe
60
EMC-Drc EMC-Cun
60
EMC-Drc
EMC-Cun
EMC-Ces
40
OPLEX
QPLEX
40
QPLEX
20
20
20
0.0M
0.4M 0.8M 12M
1.6
2.0M
0.0M
0.4M 0.8M 1.2M 1.6M
2.0M
0.0M
0.4M
0.8M 1.2M
1.6M 2.0M
T Timesteps
Timesteps
Timestops
(a) corridor
(b) 3s5z_ _ 3s62
(c) 6h
Figure 9: Ablation study on design choice.
Therefore conduct additional ablation studies to demonstrate the effect of our elaborate formulation of curiosity bias. We introduce several baselines and compare them with EMC : (i) using the normalized TD-error of Qtotel as curiosity rewards, denoted EMC-TD; (ii) using the averaged error between the individual utilities and their targets as intrinsic rewards, denoted as EMC-Ind; (iii) using the TD error of centralized critic of the controllers which conditions all agents' histories and actions, denoted EMC-Cen; (iv) using the averaged prediction errors of Qurttidec which are trained in decentralized way. denoted as EMC-Dec. We aim to investigate the subtle implementation difference between EMC and EMC-TD as well EMC-Ind, and compare the exploration efficiency of our method with the global curiosity-driven exploration method (EMC-Cen) and local curiosity-drice driven exploration method (EMC-Dec) empirically.
We design a variant of the toygame mentioned in section which has an additional random noisy reward region. By visualizations, demonstrate that the agents of EMC-TD and End tend to get stuck in the noisy-reward region, thus resulting in sub-optimal policy, while our method show superior ability for avoiding such noise-spike problem. On the other hand, since EMC-Cen is based on global curiosity, which encourages agents to explore the whole state space without bias, it may fail in finding sparse but valuable interaction patterns in the exponentially growing space in complex tasks. When comparing with EMC and EMC-Dec, we find that the key difference is the counterfactual baseline (Eq. (2), which can theoretically reduce the variance of EMC 4T. Therefore, EMC can focus more on the individual specific contribution and achieve the significant improvements.
We test these baselines in SMAC and the results shown in Figure 9 and method significantly outperform other baselines. In general. by conducting these ablations, we demonstrate the robustness for noise spikes of design choice ((i) and (ii),, as well the efficiency and stability of our method compared with centralized or decentralized curiosity-drive -driven exploration method. More detailed discussions will be deferred Appendix E.
Conclusions and Future Work
This paper introduces EMC, a novel episodic multi-agent reinforcement learning algorithm with curiosity-drive exploration framework that allows for efficient coordinated exploration and boosted policy training by exploiting explored informative experiences. Based on the effective exploration ability, our method shows significant outperformance over state-of-the-are MARL baselines on challenging tasks in the StarCraft II micromanagement benchmark. The limitation of our work lies in the lack adaptive exploration methods to ensure robustness. Besides, the episodic memory may get problems in stochastic settings. For future work, may conduct further research in these directions.
10

---

Acknowledgments and Disclosure of Funding
We would like to thank the anonymous reviewers for their valuable comments and helpful suggestions. This work is supported in part by Science and Technology Innovation 2030 "New Generation Artificial Intelligence" Major Project (No. 2018AAA0100900), a grant from the Institute of Guo Qiang, Tsinghua University, and grant from Turing Al Institute of Nanjing.
References
[1] Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial informatics, 9())4277-338 2012.
[2] Maximilian Hüttenrauch, Adrian Šošić, and Gerhard Neumann. Guided deep reinforcement learning for swarm systems. arXiv preprint arXiv:1709.0600 2017.
[3] Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4))819 840, 2002.
[4] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. delle-decompooition networks for cooperative multi-agent learning based on team reward. In AAMAS, pages 2085-2087, 2018.
[5] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference Machine Learning, pages 4295-4304, 2018.
[6] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. arXiv preprint arXiv: 055408 2019.
[7] Jianhao Wang. Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent gelaarning. arXiv preprint arXiv: 01062, 2020.
[8] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon White- son. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 2186-2188, 2019.
[9] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in neural information processing systems, pages 1471-1479, 2016.
[10] Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International Conference on Machine Learning, pages 2721- 2730, 2017.
[11] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, 2017.
[12] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint 1110. 2018.
[13] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pages 109-1117, 2016.
[14] Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. Influence-based multi-agent exploration. In International Conference Learning Representations, 2019.
11

---

[15] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi- agent variational exploration. In Advances in Neural Information Processing Systems, pages 7613-7624, 2019.
[16] Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. Emi: Exploration with mutual information. arXiv preprint urXiv:1810.01 2018
[17] Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, al. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2019.
[18] Adrià Puigdomenech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvit- skyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pages 507-517. PMLR, 2020.
[19] Lihong Li, Thomas J Walsh, and Michael L Littman. Towards unified theory of state abstraction for mdps. ISAIM, 4:5, 2006.
[20] Jianhao Wang, Zhizhou Ren, Beining Han, Jianing Ye, and Chongjie Zhang. Towards un- derstanding linear value decomposition in cooperative multi-agent q-learning. arXiv preprint arXiv:2006.00587 2020.
[21] Zichuan Lin, Tianqi Zhao, Guangwen Yang, and Lintao Zhang. Episodic memory deep q- networks. IJCAI, 2018.
[22] Guangxiang Zhu, Zichuan Lin, Guangwen Yang, and Chongjie Zhang. Episodic reinforcement learning with associative memory. In International Conference on Learning Representations, 2019.
[23] Frans A Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
[24] Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008.
[25] Pierre-Yve Yves Oudeyer, Frdric Kaplan, and Verena Hafner. Intrinsic motivation systems for autonomous mental development. IEEE transactions on evolutionary computation, 11(2):265- 286, 2007.
[26] Pierre- Oudeyer and Frederic Kaplan. What intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics, 1:6, 2009.
[27] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pages 2753-2763 2017.
[28] Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predietive models. arXiv preprint arXiv: 2015.
[29] Shakir Mohamed and Danilo Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Proceedings of the 28th International Conference on Neural Information Processing Voltums-Volume 2, pages 2125-2133, 2015.
[30] Eugenio Bargiacchi, Timothy Verstraeten, Diederik Roijers, Ann Nowé, and Hado Hasselt. Learning to coordinate with coordination graphs in repeated single-stage multi-agent decision problems. In International conference on machine learning, pages 482-490. 2018.
[31] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pages 3040-3049, PMLR, 2019.
12

---

[32] Shariq Iqbal and Fei Sha. Coordinated exploration via intrinsic rewards for multi agent rein- forcement learning. arXiv preprint arXiv:1905. 12127, 2019.
[33] M Lengyel and Dayan. Hippocampal contributions to control: The third way. In Twenty-First Annual Conference on Neural Information Processing Systems (NIPS 2007), pages 889-896. Curran, 2008
[34] Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprint arXiv: 04460, 2016.
[35] Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In International Conference on Machine Learning, pages 2827-2222222222 2836. PMLR, 2017.
[36] Steven S Hansen, Pablo Sprechmann, Alexander Pritzel, André Barreto, and Charles Blundell. Fast deep reinforcement learning using online adjustments from the past. Proceedings of the 32nd International Conference Neural Information Processing Systems, pages 10590-10600, 2018.
[37] Samuel Gershman and Nathaniel D Daw. Reinforcement learning and episodic memory in humans and animals: integrative framework. Annual review of psychology, 68:101-128, 2017.
[38] Timothy P Lilliccaap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1xXiv1159.02997 2015.
[39] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted QMIX: ex- panding monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of the Annual Conference on Neural Information Processing Systems 2020 (NeurIPS 2020), 2020.
[40] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523, 2020.
[41] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Dop: Off- policy multi-agent decomposed policy gradients. In International Conference on Learning Representations, 2020.
[42] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei Efros. Large-scale study of curiosity-driver learning. arXiv preprint 180::04083.555 2018.
[43] Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint trXiv:1803.0093. 2018.
[44] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint Xrxiv: / 1.0555,, 2015.
[45] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty- AAAI conference on artificial intelligence, 2018
13


Reviews:
[{"review_id": "z3qDzalj8QU", "reviewer": "Reviewer_vJkd", "summary": "The paper presents a solution for improving exploration in multi-agent environments. Episodic Multi-agent reinforcement learning with Curiosity-driven exploration (EMC) adds a curiosity and episodic memory module on top of standard value-factorized Q-learning algorithms typically used in multi-agent control (e.g. QMix). First, an intrinsic reward is defined through the curiosity module. The curiosity module trains separate Q-functions from the policy to predict the Q-values only on external reward. In addition to these external-reward Q-functions, the curiosity module also has a collection of \"target\" Q-functions. The intrinsic curiosity reward is then derived as the scaled value of the L2 distance between target and external-reward Q-functions. The motivation behind this particular intrinsic reward is that each individual agent's external-reward Q-values contain information from every other agent due to the mixing function used to estimate the final combined Q-value, meaning that coordinated exploration is more likely to occur with this intrinsic reward than others based only on local state information. Additionally to the curiosity module, EMC uses an episodic memory which, similar to work on episodic control, stores the maximum obtained Monte-Carlo return of a particular state. Using a non-parametric look-up into the episodic memory provides the learning process the ability to regularize its target Q-function to an observed Monte-Carlo return.\n\nResults are demonstrated on a toy environment, Predator-Prey and SMAC, where EMC outperforms other baseline methods especially in environments where exploration is a significant factor towards success. An analysis on the toy environment additionally shows in more detail the improved coordinated exploration afforded by EMC.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper's proposed method, EMC, provides a straightforward and effective solution to the problem of curiosity-based exploration / intrinsic motivation for multi-agent reinforcement learning. The curiosity module resembles Random Network Distillation (RND) and other related methods, where the error between target and predictor network output represents the intrinsic reward provided to the agent. In EMC, the use of an external-reward Q-network as the target for the prediction is an interesting contribution by itself. It could seemingly also be applied to the setting of single-agent environments, where incorrect predictions of the Q-value by the predictor networks could signal the agent towards re-visiting states where its Q-values are changing more rapidly, which seems like a useful signal for learning. The paper convincingly motivates their formulation of the intrinsic reward both by (1) demonstrating that the per-agent Q-functions capture global information via the use of a mixing function during Q-learning and (2) extensive comparisons against a variety of baselines on both toy and standard benchmarks.\n\nThe other contribution of EMC, which is the episodic memory component, is maybe less well-motivated. Episodic control methods are potentially brittle to highly stochastic environments, since in those settings a stored maximum return is likely to be an outlier, i.e. caused by lucky environment dynamics and not necessarily due to an agent's own actions. Episodic control's sensitivity to outliers could be especially noticeable in multi-agent environments, which have multiple learning systems interacting in non-stationary ways. The ablations in Figure 8 reveal a maybe minor effect of the episodic memory component on learning speed, and an insignificant effect on final performance (although the figures are only on 3 of 17 SMAC tasks).\n\nIt would make the paper stronger if the authors can (1) expand more on the motivation of the memory component, (2) demonstrate that this memory is useful even with stochastic environments (or otherwise show that this isn't a serious problem more generally) and (3) have clearer empirical results demonstrating the performance improvement of the memory component (e.g. maybe ablations over the components, presented over all scenarios similar to Figure 6). Even with these unanswered questions about the episodic memory, I still rate the paper as accept due to the curiosity module representing an effective solution to multi-agent coordinated exploration.\n\nAlso, is there a reason that MAVEN, another exploration method for multi-agent RL mentioned in the paper, is missing from certain figures (Figures 4, 5, 6). \n\nIn terms of writing, the paper is clear and well-written. The method's explanation is easily understandable, with Figure 2 providing a direct overview. The connections to previous work are sufficiently highlighted.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper describes some limitations, but as mentioned in the main review, the episodic memory's sensitivity to outliers is one limitation that I foresee could have impact in non-stationary and highly stochastic environments.\n\nI don't believe there is large potential for negative societal impact, as the method is only a curiosity-based intrinsic reward for multi-agent reinforcement learning.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Y79grQuRb0_", "reviewer": "Reviewer_rGKf", "summary": "The paper proposes to use prediction errors of agent specific Q-functions as an intrinsic exploration bonus in multi-agent reinforcement learning (RL) similar to how prediction errors have been used in single agent RL. Another contribution is the application of episodic memory as an auxiliary loss in multi-agent RL. The proposed approach outperforms comparison methods in experiments.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "ORIGINALITY: \n\nThe proposed multi-agent RL approach is a novel combination of existing single agent techniques. To use curiosity and episodic memory in multi-agent RL the paper makes important design decisions and proposes a new framework.\n\nThe references require a thorough check. For example, references [18] and [19] refer to the same paper.\n\n\nQUALITY:\n\nThe Q-functions $Q_{tot}^{ext}$, $Q_i^{ext}$, etc. used inside the curiosity module are implemented as neural networks separate from $Q_i$ and $Q_{tot}$. This design choice could be discussed in more detail. What would be potential problems if using the same Q-functions instead?\n\nThe paper should explain why in Equations (4) and (5) the Q-function has as parameter $s$. Under partial observability $s$ is not observed? On Line 209 $Q_{tot}$ has as parameter the history.\n\nIn the episodic memory approach, selecting the highest reward observed so far seems to be a very heuristic and application specific approach. Is there analysis (in the single agent case?) on when it does not work? For example, in stochastic environments very unlikely events but with high reward can mislead the exploration process into sub-optimal solutions. The intrinsic motivation based reward seems a less \"dangerous\" feature of the proposed approach since Q-function prediction errors should decrease with more samples. Appendix B shows that the episodic memory module which is a greedy heuristic leads easily to sub-optimal solutions in some benchmarks. \n\nThere are two components in the proposed approach, episodic memory and intrinsic reward. Why is the episodic memory used with the global state and with the global Q function but intrinsic reward is used with individual Q-functions dependent on action-observation histories? Should not similar reasoning apply to both?\n\nWhat is the interplay of the episodic memory and intrinsic motivation? It was mentioned that intrinsic motivation is not sufficient on its own but the episodic memory is needed in addition. This connection could be explained in more detail.\n\nIn \"As the dynamics of an agent’s individual Q-value function captures the novelty of states and the influence from other agents\",\nwhat does \"dynamics of an agent’s individual Q-value function\" mean exactly?\n\nThe proposed approach has several hyper-parameters due to two features: intrinsic reward and episodic memory. Please, specify how hyper-parameters were chosen and how much computational effort was used to find hyper-parameters?\n\n\nCLARITY:\n\nThe paper is in many parts well written. Figure 2 is very useful.\n\nThe storyline in the intro is somewhat misleading. Centralized training with decentralized execution (CTDE) is not such a recent paradigm as advertised. Dec-POMDPs, where typical solutions involve computing a solution offline in a centralized manner and executing policies decentralized, have been around already for some time [Bernstein et al., 2002]. \n\nBernstein, D. S., Givan, R., Immerman, N., & Zilberstein, S. (2002). The complexity of decentralized control of Markov decision processes. Mathematics of operations research, 27(4), 819-840.\n\nOn Line 121: \"and attracted great attention.\". What does this mean exactly and why is it important?\n\nLine 194: What are \"orediction errors\"?\n\nLine 219: Please, rephrase \"how to use the best use of good trajectories collected by exploration\".\n\nLine 220: In \"Recently, episodic control has become very popular\", why is it importance that episodic control has become popular? Please, concrete statements: what are the benefits of episodic control and where do the benefits come from?\n\nLine 222: \"Inspires by this\" should be \"Inspired by this\"?\n\n\nSIGNIFICANCE:\n\nImproving exploration in multi-agent RL is important. The idea of using ideas from single agent RL exploration based on curiosity and extending them to multi-agent RL individual agents is interesting. The way episodic memory is used to improve utilization of experience should be better motivated and the weaknesses discussed properly / replaced with something less heuristic and greedy. The proposed framework is partly well motivated but needs further clarifications or/and improvements in order for other researchers to be able to take advantage of it.\n\nUPDATE:\n\nThe rebuttal addressed some of my concerns. I am still not completely satisfied with the theoretical motivation and will keep my score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "Ejan7Aj1jp3", "reviewer": "Reviewer_qZwR", "summary": "This paper studies curiosity-driven exploration for multi-agent deep reinforcement learning. Specifically, instead of deploying centralized curiosity exploration or decentralized curiosity exploration,  the paper learns a factorized centralized critic which is a monotonic linear combination of individual agents’ Q value. Then, the prediction errors of individual agent’s Q-values are used as intrinsic rewards that encourage exploration. In addition, an episodic memory is used to prioritize promising experience trajectories and speedup training. The proposed approach is compared with competitive baselines on StarCraft II multi-agent challenges (SAMC). ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well structured and easy to follow. The multi-agent exploration problem it aims to address is an important problem for our community. The experimental results look promising. However, the reviewer has some questions about the claims made in the papers. Please see the detailed comments below. \n\n\n1. The authors claim that the proposed approach achieves better coordinated exploration. However, it is unclear why and how the proposed curiosity module improves coordination. Is it because the intrinsic reward is assigned to a global state and thus it encourages agents to transition to the global state? If it is the case, why the proposed approach could achieve better coordination than global curiosity exploration. Moreover, the author mentions that the proposed curiosity module could `biases exploration into states where strong interdependence may lie between agents’. It is unclear to the reviewer how the proposed approach could achieve this. Could you elaborate?\n\n\n2. In MARL, the size of the global state space grows exponentially with the number of agents. Will the proposed curiosity module encourage agents to visit all states? If yes, could you comment on the scalability of the proposed approach?\n\n\n3. This paper considers multi-agent exploration in a dense-reward setting. On the other hand, previous multi-agent exploration work such as EITI /  EDTI [1] considers sparse-reward setting, i.e. agents only receive rewards when completing a task, which is much more challenging than dense-reward setting. Experiments, analysis, or comments on the performance of the proposed approach on sparse-reward environments would make the submission stronger. \n\n\n[1] Influence-Based Multi-Agent Exploration, Wang et. al. ICLR 2020\n\n====== Post Rebuttal ======  \n\nThanks for the reply.  The author response addressed my concern and the additional experimental results look good. I raise my score to 6.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the limitations of their work. Discussion on societal impact seems missing. \n", "review_text": "", "extra_scores": {"ethical_concerns": "The reviewer doesn't see any ethical issues with this paper.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "2-HTik9Mkcg", "reviewer": "Reviewer_XAps", "summary": "This work focuses on efficient exploration for MARL in the Centralized Training Decentralized Execution (CTDE) framework. It proposes to rely on two mechanisms to improve it: an intrinsic motivation based on the prediction error of the agents' Q-values and the use of episodic memory to store the highest return obtained from previously visited states.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "2. Strong points\n* The problem tackled by this work is important and well-motivated. \n* The proposed method performs well and can be applied to any MARL algorithm with Q-value factorization.\n* The method is tested against several baselines on several environments of varying difficulties. Visualizations are provided in a simple environment. \n* Ablation studies are presented. \n\n3. Possible improvements\n      1. The motivation behind this choice of intrinsic motivation could be made clearer. From what I understand it uses Random Network Distillation (RND) but with a changing target, the individual Q-values, that are learned by TD-learning. Therefore it encourages visiting less-visited states (at least in the beginning) but more importantly states for which the Q values have changed (change in the strategy, environment, etc.). It seems that this would not be specific to multi-agent learning or value factorization but could also be applied to single-agent RL. It would therefore be interesting to motivate why the MARL setting, in particular, is considered. Also, it would be interesting to assess how much the centralized training of the Q-values impacts the performance of the exploration. In addition to QPLEX-Global and QPLEX-Local an additional baseline could be to train the $Q^{ext}_i$ in a decentralized way and use the sum of their prediction error as intrinsic motivation. This should test how much the \"counterfactual baseline\" term of Eq (2) plays a role in the efficiency of the curiosity.\n    2. The implementation of the Episodic Memory with the memory table and the state projection could be made clearer. Also, could you explain why using the max return over previous trajectories does not break the learning? Indeed using the return of a previous outdated trajectory estimates the value under an outdated policy/team strategy. Could it be because it uses the max and that the optimal policy/team strategy should have the highest value for any state?\n    3.   How many additional parameters and updates are required to use the Curiosity module (training all the additional Q-network and predictors)?\n    4.   This work mentions \"coordinated\" exploration several times. Could you define what is meant, how to measure it, and how the proposed method improves it?\n\n\n4. Recommendation\nI think the paper is good but can be improved by addressing some points raised in 3. \n\n5. Additional Remarks \n    1. Could you comment on the use of an intrinsic reward based on the prediction error between Q-values trained in a centralized /factored way from TD and Q-values trained in a decentralized way (also from TD)\n    2. The notation seems to alternate between $Q_{tot}^{all}$ and $Q_{tot}$ which is confusing. Also, the $Q$ values sometimes depend on the state and sometimes on the trajectory history.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Quite rapidly, it could be developed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4,5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new exploration algorithm specifically for multi-agent RL with factorized and centralized value function (critic). The idea is to use individual agent's value prediction errors as intrinsic rewards based on the intuition that this captures the influence from other agents thanks to the factorized/centralized aspect. In addition, the paper proposes an episodic memory that exploits past good experiences to further improve sample-efficiency. 

Improving exploration in the context of multi-agent RL is also an important problem. All of the reviewers appreciated that the proposed idea is insightful and well-motivated, and the results are also strong. There is still a remaining concern regarding the effectiveness of the episodic memory especially in stochastic environments. The authors added a new result showing that the proposed method is still effective to some degree of stochasticity during the rebuttal period, and the reviewers are generally satisfied with this result. Therefore, I recommend accepting this paper and suggest the authors to include this for the camera-ready version.
Decision: Accept (Poster)