Episodic Multi-agent Reinforcement Learning with Curiosity-driven Exploration
Authors: Lulu Zheng
Jiarui Chen
Jianhao Wang
Jiamin He
Yujing Hu
Yingfeng Chen
Changjie Fan
Yang Gao
Chongjie Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: multi-agent reinforcement learning
multi-agent exploration
episodic control

Abstract: Efficient exploration in deep cooperative multi-agent reinforcement learning (MARL) still remains challenging in complex coordination problems. In this paper, we introduce a novel Episodic Multi-agent reinforcement learning with Curiosity-driven exploration, called EMC. We leverage an insight of popular factorized MARL algorithms that the ``induced" individual Q-values, i.e., the individual utility functions  used for local execution,  are the embeddings of local action-observation histories, and can capture the interaction between agents due to reward backpropagation during centralized training. Therefore, we use prediction errors of individual Q-values as intrinsic rewards for coordinated exploration and utilize episodic memory to exploit explored informative experience to boost policy training. As the dynamics of an agent's individual Q-value function captures the novelty of states and the influence from other agents, our intrinsic reward can induce coordinated exploration to new or promising states. We illustrate the advantages of our method by didactic examples, and demonstrate its significant outperformance over state-of-the-art MARL baselines on challenging tasks in the StarCraft II micromanagement benchmark.

Episodic Multi-agent Reinforcement Learning with Curiosity-driven Exploration
Lulu Zheng" Jiarui Chen 22 Jianhao Wang's Jiamin He*t, Yujing Hu², Yingfeng Chen',
Changjic Fan", Yang Gao", Chongjie Zhang'
Institute for Interdisciplinary Information Sciences, Tsinghua University, China Department of Computer Science and Technology, Nanjing University, China Fuui AI Lab, NetEase, China
Department of Computing Science, University of Alberta, Canada z1119@mails. tainghua. edu.
chenjiaruilsmmil.. wtw.::/mmmma......aea.... edu. cn
Sehis0maalls.tsinggiin edu.
jiamin120ualberta.ccm
(huyujing, deenyingfeng01, f Scorp
netease @coor.
com
gaoy@nju edu. en
Chongjie@tsinghuaaeduu
Abstract
Efficient exploration in deep cooperative multi-agent reinforcement learning (MARL) still remains challenging in complex coordination problems. In this paper, we introduce novel Episodic Multi-agent reinforcement learning with Curiosity-driven exploration, called EMC. We leverage an insight of popular factor- ized MARL algorithms that the "induced" individual Q-values, i.e., the individual utility functions used for local execution, are the embeddings of local action- observation histories, and can capture the interaction between agents due to reward backpropagation during centralized training. Therefore, we use prediction errors individual Q-values as intrinsic rewards for coordinated exploration and uti- lize episodic memory to exploit explored informative experience to boost policy training. As the dynamics an agent's individual Q-value function captures the novelty of states and the influence from other agents, intrinsic reward can induce coordinated exploration to new or promising states. We illustrate the advantages of our method by didactic examples, and demonstrate its significant outperformance over state-of-the-aa MARL baselines on challenging tasks in the StarCraft II micromanagement benchmark.
Introduction
Cooperative multi-agent reinforcement learning (MARL) has great promise to solve many real-world multi-agent problems, such as autonomous cars [ and robots [2). These complex applications post two major challenges for cooperative MARL: scalability, i.e., the joint-action space exponentially grows as the number of agents increases, and partial observability, which requires agents to make decentralized decisions based on their local cecton-observation histories due communication constraints. Luckily, a popular MARL paradigm, called centralized training with decentralized execution (CTDE) [3, is adopted to deal with these challenges. With this paradigm, agents' policies are trained with access to global information in a centralized way and executed only based on local
"Equal contribution. Work performed while visiting Tsinghua Univeristy.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

histories in a decentralized way. Based on the paradigm of CTDE, many deep MARL methods have been proposed, including VDN QMIX [5]. QTRAN 6 and QPLEX
core idea of these approaches is to use value factorization, which uses neural networks to represent the joint state-action value as function of individual utility functions, which can be referred to individial Q-values for terminological simplicity. For example, VDN learns a centralized but factorizable joint value function Qto represented as the summation of individual value function Qi- During execution, the decentralized policies can be easily derived for each agent by greedily selecting actions with respect its local value function Qi- By utilizing this factorization structure, an implicit multi-agent credit assignment is realized because Q, is represented as a latent embedding and is learned by neural network backpropagation from the total cemporal-differees global reward signal, rather than on a local reward signal specific to agent i. This value factorization technique enables value-based MARI, approaches, such as QMIX and QPLEX, to achieve state-of- the-art performance in challenging tasks such as the StarCraft unit micromanagement [8].
error on the single
Despite the current success, since only using simple e-greedy exploration strategy, these deep MARL approaches are found ineffective to solve complex coordination tasks that require coordinated and efficient exploration 77.. Exploration has been extensively studied in single-agent reinforcement learning and many advanced methods have been proposed, including pseudo-counts (9] [O curiosity [T] 12, and information gain [3]. However, these methods cannot adopted into MARL directly, due to the exponentially growing state space and partial observability. leaving agent exploration challenging. Recently, only a few works have tried address this problem. For instance, EDTI [4 uses influence-based methods to quantify the value of agents' interactions and coordinate exploration towards high-value interactions. This approach empirically shows promising results but, because of the need to explicitly estimate the influence among agents, it is not scalable when the number of agents increases. Another method, called MAVEN [15), introduces hierarchical control method with a shared latent variable encouraging committed, temporally extended exploration. However, since the latent variable still needs to explore in the space of joint behaviours [15. is not efficient in complex tasks with large state spaces.
In this paper, we propose novel multi-agent curiosity-drin -drivee exploration method. Curiosity is a type of intrinsic motivation for exploration, which usually uses prediction errors on different spaces (e.g., future observations [2]. actions [11] or learnable representation [16] as reward signal. Re- cently, curiosity-drive methods have achieved significant success in single-agent reinforcement learn- ing 122 [8) However, curiosity-dree -driven methods face a critical challenge MARL: in which space should we define curiosity? The straightforward method is to measure curiosity the global observa- tion [12 or joint histories in a centralized way. However, is inefficient to find structured interaction between agents, which seems too sparse compared with the exponentially growing state space when the number of agents increases. In contrast, if curiosity defined as the novelty of local observation histories during the decentralized execution, although scalable, it still fails to guide agents to coordi- nate due to partial observability. Therefore, we find a middle point of centralized curiosity and decen- tralized curiosity, i.c., utilizing the value factorization of the state-of- ooffthe-an art multi-agent Q-learning approaches and defining the prediction errors of individual Q-value functions as intrinsic rewards. The significance of this intrinsic reward is two- fold: 1) it provides a novelty measure of joint observation histories with scalability because in- dividual Q-values are latent embeddings (i.e., an effective state abstraction (19] of observation histories in factorized multi-agent Q-learning (c.g.. VDN or QPLEX); and 2) as shown in Figure [] it captures the influence from other agents due the implicit credit assignment from global reward signal during centralized train- ing 20). and biases exploration into promising states where strong interdependence may lie between agents. Therefore, with this novel intrinsic reward, our curiosity-driven method enables efficient, diverse, and coordinated exploration for deep multi-agent Q-learning with value factorization.
Global Curiosity
Global Information
Centralize Trai Training Qcocal
Mixing Gradients
Individual
Q-values
2
QN
Local
Loca actionn voservet in Notolee
Local action- abserce or Noorle
Local action- obsenstian Nitories
Curiosity
Figure 1: CTDE Framework
Besides efficient exploration, another challenge for deep MARL approaches is how to make the best use of experiences collected by the exploration strategy. Prioritized experience replay based on TD errors shows effectiieness in single-agent deep reinforcement learning. However, does
2

---

not carry this promise in factorized multi Q-learning, since the projection error induced by value factorization is also fused into the TD error and severally degrades the effectiveness of the TD error as measure of the usefulness of experiences. To efficiently use promising exploratory experience trajectories, we augment factorized multi-agent reinforcement learning with episodic memory [21 [22] This memory stores and regularly updates the best returns for explored states. We use the results in the episodic memory to regularize the TD loss, which allows fast latching onto past successful experience trajectories collected by curiosity exploration and greatly improves learning efffciency. Therefore, we call our method Episodic Multi-agent reinforcement learning with Curiosity- driven exploration, called EMC.
We evaluate EMC in didactic examples, and broad set of StarCraft II micromanagement benchmark tasks [8. The didactic examples along with detailed visualization illustrate that our proposed intrinsic reward can guide agents' policies to novel or promising states, thus enabling effectively coordinated exploration. Empirical results on more complicated StarCraft II tasks show that EMC significantly outperforms other multi-agent sate-of-the-aad baselines.
2 Background
2.1 Dec-POMDP
cooperative multi-agent task can be modelled as a Dec-POMDP [3]. which is defined by tuple G =<<<>> I, A. P. R, f2, O, n. 7/ >, where I is the sets of agents, S is the global state space, A the finite action set, y E [0, 1) is the discount factor. We consider a partially observable setting in a Dec-POMDP, i.c., at each timestep, agent i E I only has access to the observation E S? drawn from the observation function (ss, i). Besides, each agent has an action-observation history E T= (52 x A)" x S2 and constructs its individual policy to jointly maximize team performance. With each agent i selecting an action a; E A, the joint action a = [a, E A = A leads to a shared reward
R(s, a) and the next state s' according to the transition distribution P(s'|s, a). The formal objective function to find a joint policy æ that maximizes a joint value function VT (s) = B 2'r = sp, T). or joint action-value function Q" (s, a) =r(s, a) + 7B (V* ( (s')).
2.2 Centralized Training With Decentralized Execution (CTDE)
CTDE is promising paradigm in deep cooperative multi-agent reinforcement learning B 23 24 where the local agents execute actions only based on local observation histories, while the policies can be trained in centralized manager which has access to global information. During the training process, the whole team cooperate to find the optimal joint action-value function Qior a) r(s, a) + [max,' (s a')l Due partial observability, we use Qtor a; 0) instead of Qior (s, a; 0). where T E T = T Then the Q-value neural network will be trained to minimize
the following expected TD-error:
[(0) Er (V
Quae a,)]²
(1)
where D is the replay buffer and 0 denotes the parameters the target network, which is periodically updated by 0. And ((r';0-) is the one-step expected future return of the TD target. Local agents can only obtain local cottoonobservattin history and need inference based on individual Q-value functions Qi oz). Therefore, many works have made efforts in finding the factorization structures between joint Q-value functions Qtot and individual Q-functions Qi(Ti, a;) 5
3 Related Work
-uriosity-driven Exploration Curiosity-driven exploration has been well studied in single-agent reinforcement learning. Previous literature [25] 26 has provided good summary in this topic. Recently, curiosity-driven methods have made great progress in deep reinforcement learning. For example, some works use pseudo-state counts to get intrinsic rewards [9] 10 27] instead of count- based methods to get better scalability. Stadie et al. (28) use prediction errors in the feature space of an auto-encoder to measure the novelty of states and encourage exploration. On the other hand, Mohamed and Rezende (29 propose to use empowerment, measured by the information gain based on the entropy of actions, as intrinsic rewards for exploring novel states efficiently. Another information- [13] tries to maximize information gain about the agent's 's belief of the environment's
based method
3

---

dynamics as an exploration strategy. ICM [1] learns an inverse model which predicts the agent's s action given its current and next states and tries to predict the next state in the learned hidden space by current state and action. RND [2] uses curiosity as intrinsic rewards in a simpler but effective way, which uses fixed randomly initialized neural network as a representation network and directly predicts the embedding of the next state. Different from these methods, we are the first to propose an advanced curiosity-driven exploration method in MARL setting for diverse and coordinated exploration.
Multi-agent Exploration Although single-agent exploration is extensively studied and has achieved considerable success, few exploration methods were designed for cooperative MARL. Bargiacchiet al. (30) proposes an exploration method that can only be used in repeated single-stage problems. Jaques et al. [I defines intrinsic reward by "social influence" to encourage agents to choose actions that can influence other agents' actions. Iqbal and Sha [32] uses various simple exploration methods to learn simultaneously and then put the samples of every method in a shared buffer to achieve the coordinated exploration. Wang
14] use mutual information (MI) to capture the interdependence of the rewards and transitions between agents. MAVEN [I5] is the state-of-the-arr exploration method in MARL that uses a hierarchical policy to produce a shared latent variable and learns several state-action value functions for each agent. These works, although important, still face the challenge of achieving scalable and effective multi-agent exploration.
Our work is also related to episodic control reinforcement learning, which is usually adopted in single-agent settings for better sample efficiency. Previous works propose to use episodic memory in mear-deterministic environment (3 Model-free episodic control R4 uses a completely non-parametric table to keep the best Q-values of state-action pair
Episodic Control
aabular-based
memory and uses a k-nearest-neighbor fashion to find the sequence of actions that so far yielded the highest return from a given start state in the memory. Recently, several extensions have been proposed to integrate episodic control with parametric DQN. Gershman and Daw (37 uses episodic memory to retrieve samples and then average future returns to approximate the action values. EMDQN 21 uses fixed random matrix as representation function and uses the projection of states as keys to store the information of episodic memory into a non non-parametric model. Using the episodic-memenn based target as a regularization term to guide the training process, the performance of EMDQN is significantly improved compared with the original DQN. Despite the fruitful progress made in single-agent episodic reinforcement learning, few works study episodic control in multi-agent setting. To the best of our knowledge, we are the first to utilize the mechanism of episodic control in deep multi-agent reinforcement learning.
Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration
In this section, we introduce EMC, a novel episodic multi-agent exploration framework. EMC takes prediction errors of individual Q-value functions intrinsic rewards for guiding the diverse and coordinated exploration. After collecting informative experience, we leverage an episodic memory to memorize the highly rewarding sequences and use it as the reference of one-step TD target boost multi-agent Q-learning. First, we analyze the motivations for predicting individual Q-values, then we introduce the curiosity module for exploration. Finally, we describe how utilize episodic memory to boost training.
4.1 Curiosity- Exploration by Predicting Individual Q-values
As shown in Figure in the paradigm of CDTE, local agents make decisions based on individual Q-value functions, which take local observation histories as inputs, and are updated by the centralized module which has access to global information for training. The key insight is that, different from single-agent cases, individual Q-value functions in MARL are used for both decision-making and embedding historical observations. Furthermore, due to implicit credit assignment by global reward signal during centralized training, individual Q-value functions Q (T,,.) -) are influenced by environment as well as other agents' behaviors. More concretely, it has been proved by Wang et al. (20) that, when the joint Q-function Qtot is factorized into linear combination of individual Q-functions Q. i.e., Q(t+")(( (T, a) EN (((+1) (Ts; 0s), then Q ( (T,, o;) has the following closed-form solution:

---

Q 2 (T))a))
E (T0rL
(T (ppp))))))))
evaluation of the individuad action
(2)
E aa (ar-pp((A- (1 (T', (T' twx(
counterfactual baseline
where y (T, = + maxa Q (T', (T') denotes the expected one-step TD target, and (P()))) denotes the conditional empirical probability of in the given dataset D. The notation TT denotes (Tí...., T ,, Ti, T++1 TT)) and --, denotes the elements of all agents except for agent i. A (T,) denotes the set of trajectory histories that may share the same latent-state trajectory as T₁. The residue term w [w:]i=1 is an arbitrary function satisfying VT r. Ei= wT((T))
Eq. 2 shows that by linear value factorization, the individual Q-value Qi a,) is not only decided by local observation histories but also influenced by other agents' action-observation histories. Thus predicting Q; can capture both the novelty states and the interaction between agents and lead agents to explore promising states. Motivated by this insight, in this paper, factorization module separate from the inference module to learn the individual value function Qi, and use the prediction errors of Qi intrinsic rewards guide exploration. In this paper, we define the prediction errors of individual Q-values as curiosity and propose our curiosity-drive exploration module as follows.
linear value
Value Factorization Framework
(n "e9"i--.Lon
[o/]==
Mixing
Environment
Contral ers
Qex (Q,, Oz. QN)
[q]]_=
Q
Gradients
Memory Loss
Trajectories
Loss 1-1
Predictors
Distance Fanction
bntnt (I,
State
Random Projection
Q
Gradients [0 (+,-))
(s))()))
MC
Targets @ueat
Mixing
Q
(r. r', s, ext)
Replay buffer
Curiosity Module
Episodic Memory (d)
(b)
(c)
Figure 2: An overview of EMC's framework
Figure 2p demonstrates the Curiosity Module, separated from the inference module (Figure 21. The curiosity module consists of four components: (i) The centralized training part with linear value factorization, which shares the same implementation rewards y eatt from the environment; (ii) the Target for prediction, i.e., the corresponding individual
VDN 44, but only trained with extrinsic
Q-values Quntt represented by recurrent Q-network; (iii) Predictor (T:), which is used for predicting Q and shares the same network architecture as Target QeIt, and (iv) Distance Function, which measures the distance between Quaa and Q:. e.g., Le distance. The predictors are trained by minimizing the Mean Squared Error (MSE) the distance in an end-to-end manner. For stable training, we use the soft-update target (38 of Q+2t to smooth the outputs of the targets. In general, (ii) trained with (i) and outputs individual Q-values, while (iii) is trained with (ii) and (iv), and aims to predict the soft-update target of individual Q-values. Motivated by the implicit credit assignment of linear value factorization (Eq. (2. the curiosity module predicts individual Q-values [ in
linear factorization, i.e., Qeat i= Q Then the curiosity- driven intrinsic reward is generated
N
by the following equation:
N
iin
( - (T..))
(3)
5

---

This intrinsic reward is used for the centralized training of the inference module, as shown in Figure 2
Linference (0) = E ET. [(U(T a) - Qoo a; 0))
(4)
where y(T, a) = rext + Brent + maxa' Qtot (T', a'; 0-), denoting one step TD target of the inference module, and / is the weight term of the intrinsic reward. We use a separate training model for inference (Figure 2a) to avoid the accumulation of projection errors of Q, during training.
The independence of inference module leads to another ad advantage, that EMC's architecture can be adopted into many sattee-aaattzziin-bbaee based multi-agent algorithms which utilize the CDTE paradigm, i.e., the general function f in Figure 2a can indicate specific (linear, monotonic and IGM) value factorization structures in VDN [., QMIX [3], and QPLEX (7], respectively. In this paper, we utilize these state-of to-tte--rttttttt algorithms for the inference module. With this curiosity- -riven bias plugged into ordinary MARL algorithms, EMC will achieve efficient, diverse and coordinated exploration.
4.2 Episodic Memory
Equipped with efficient exploration ability, another challenge is how to make the best use of good trajectories collected by exploration effectively. Recently, episodic control has been widely studied in single-agent reinforcement learning 21 22), which can replay the highly rewarding sequences, thus boosting training. Inspired by this framework, we generalize single-agent episodic control to propose a multi-agent episodic memory, which records the best memorized Monte-Carlo return in the episode, and provide a memory target H as reference to regularize the ordinary one-step inference TD target estimation in the inference module (Figure 2
Lmemory ( = EE. (H Qto((T a: 0))]
(5)
However, different from the single-agent episodic control, the action space of MARL exponentially grows as the number of agents increases, and partial observability also limits the information of individual value functions. Thus, maintain our episodic memory by storing the state-value function on the global state space and utilizing the global information during the centralized training process under the CTDE paradigm. Figure 2 shows the architecture of the Episodic Memory. We keep a memory table M to record the maximum remembered return of the current state, and use a fixed random matrix drawn from Gaussian distribution low-dimensional vectors d(s): : S + R², which are used as keys to look up corresponding global state value function H( ((())). When our exploration method collects a new trajectory, we update our memory table M as follows:
representation function to project states into
H H(((s))) = max((H((()))), R, a.))
(f(l(ss) - o <
(6)
Rs as)
otherwise
where ((+:) is o(s;)² 's nearest neighbor in the memory M, 8 a threshold, and R(st, az) represents the future return when agents taking joint action a, under global state 8; at the t-th timestep in a new episode. In our implementation, Φ(st) E M is indeed evaluated approximately based on the embedding distance. Specifically, when the key of the state o(st) is close enough to one key in the memory, we assume that ((s)) E M and find the best memorized Monte-Carke return correspondingly. Otherwise, we think ((s)) & M and record the state's return into the memory. Leveraging the episodic memory, we can directly obtain the maximum remembered return of the current state, and the one-step TD memory target H reference to regularize learning:
Ho((s), ar) = (r( a:) + (H
Thus, the new objective function for the inference module is:
Lroral (0) Linforence (0) + XL mmmooryy( (0)
(8) where the weighting term to balance the effect of episodic memory's reference. Using the maximum return from the episodic memory to propagate rewards, we can compensate for the disadvantage of slow learning induced by the original one-step reward update and improve sample efficiency.
EE.T. ( a) Qior (T, a1))) X (ō(si), a.) Qroi -ro((T,

---

5 Experiments
In this section, we will conduct a large set of empirical experiments for answering the following questions: (1) Is exploration by predicting individual Q-value functions better than exploration by decentralized curiosity or global curiosity (Section 5.1 (2) Can our method perform efficient coordinated exploration in challenging multi-agent tasks (Section 5.2 5.3 (3) If so, what role does each key component play for the superior performance (Section 5.4 (4) Why do we choose to predict target Q for generating intrinsic rewards rather than other choices (Section 5.4b? We will propose several didactic examples and demonstrate the advantage of our method in coordinated exploration, and evaluate our method on the StarCraft II micromanagement (SMAC) benchmark [8] compared with existing state-of-the-ee multi-agent reinforcement learning (MARL) algorithms: QPLEX [7].
Weighted Reighted-QMIX [39, QTRAN 6 QMIX 5 VDN RODE and MAVEN 15]
5.1 Didactic Example
Figure shows an 11 x 12 grid world game that requires coordinated exploration. The blue agent and the red agent can choose one of the five actions: [up, down, left. right, stay/ at each time step. The wall shown in the picture isolates the two agents, and one agent cannot be observed by the other until it gets into the shaded area. The two agents will receive a positive global reward r 10 if and only if they arrive at the corresponding goal grid (referred to the character G in Figure 3) at the same time. If only one arrives, the incoordination will be punished by a negative reward
To evaluate the effectiveness of our curiosity-driver exploration, we implement our method into QPLEX, QMIX, and VDN (denoted as EMC- QPLEX. EMC-QMIX, and EMC-VDN, respectively) and test them in this
Wall
toy game compared with the state-of-the-art MARL algorithms: VDN,
Moving Agent
IQL, QMIX, and QPLEX. Moreover, to demonstrate the motivation of predicting individual Q-functions, we add two more baselines: QPLEX with the prediction error of global state as intrinsic rewards (denoted as OPLEX-Global, and QPLEX with the prediction error of local joint histories as intrinsic rewards (denoted as QPLEX- use a fixed network to project the inputs into latent embedding. then predict the latent embedding to generate intrinsic reward, just like the Random Network Distillation (RND) [12.. We test different punishment degrees, i.e., different ps (which are deferred to Appendix C), and the results show QPLEX- Globall and QPLEX-Local are effective enough for exploration when p is relatively small. However, as increases, the task becomes more challenging since it requires sufficient and coordinated exploration. In Figure 4 median test win rate of all methods over random seeds when = 2, and only our methods can learn the optimal policy and win the game, while other methods failed.
Moving Agent
Both of them
Figure 3: Coordinated Toygame
we show the
To understand this result better, we have made several visualisations to demonstrate our advantage in coordinated exploration. Figure 4 shows the heatmaps of visitation and intrinsic reward by EMC-QPLEX, APLEX--lobal. and LoLllL During the early stage of training. all methods uniformly explore the whole area (Figure 41). As the exploration progresses, the global curiosity (QPLEX-Global) encourages agents to visit all configurations without bias, which is inefficient and fail to leverage the potential locality influence between agents (Figure 4b), resulting in extrinsic rewards beginning to dominate the behaviors (b) On the other hand, the visitation heatmap of QPLEX-Local shows the decentralized curiosity encourages agents to explore around the goal grid, but it cannot promise to encourage agents to coordinate and gain the reward due to the partial observability decentralized execution. In contrast, the heatmap of intrinsic reward for EMC-QPLEX shows that predicting individual Q-values will bias exploration into areas where individual Q-values are more dynamic due to the potential correlation between agents. Therefore, QPLEX-Local and OPLEX-Global both fail in this task (Figure 4)), while our methods are able to find the optimal policy. This didactic example shows the global curiosity or local curiosity may fail to handle complex tasks where coordinated exploration needs to be addressed. While since individual Q-values Q; are the embeddings of historical observations, and are dynamically updated by the backpropagation of the global reward signal gained through cooperation during centralized training. Thus Q, can implicitly reflect the influence from the environment and other agents, and predicting Qi can capture valuable and spare interactions among agents and bias exploration into new or promising states.
7

---

Phase
Phase 2
Phase Optimal Policy
Median Test Win Rate %
Uniform
Capture
Exploration
Interaction
100
EMC-QPLEX (Ours)
80
EMC QMIX (Ours)
(Ours)
60
QPLEX
QMIX
40
QPLEX-Global OPLEX-Local
20
0
0.0M
0.1M
0.2M
0.3M
0.4M
0.5M
OOL
25Ch
Timesteps
EMC-OPLEX
QPLEX-Local
OPLEX-Global
405. Visitation
Aittiii Intrinsic Reward
Hiilliime Visitation
sa raa Intrinsic Reward
2 Visitation
2 Intrinsic Reward
(a)
(b)
(c)
Figure 4: The heat map of gridworld game.
5.2 Predator Prey
Predator-Prey is a partially-observable multi-agent co- ordinated game with miscoordination penalties used by WQMIX [39]. As shown in Figure 55) since extensive exploration is needed to jump of the local optima, WQMIX the only baseline algorithm to find the optimal policy, due its shaped data distribution which be seen as a type of exploration. Other stateoof-the-ar multi- agent Q-learning algorithms, such as QPLEX and QMIX, fail to solve this task. For MAVEN, QPLEC-Global and QPLEX-Local, although equipped with improved explo- ration ability, they still failed to address coordination due to uniform exploration nature or partial observability. However. plugged with EMC, EMC-VDN, EMC-QMIX, and EMC-QPLEX can guarantee coordinated exploration effectively and achieve good performance.
40
Median Test Return
-40
-80
-120
160
QPLEXX-C444)
0.0M
0.2M
0.4M
0.6M
0.8M
1.0M
Timesteps
Figure 5: The performance of Predator Prey.
5.3 StarCrafth Micromanagement (SMAC) Benchmark
StarCraft II Micromanagement (SMAC) is popular
benchmark in MARL 4
5
40 39
We conduct ex-
EMC (muz))
QFLEX
periments
benchmark tasks StarCraft II, which
# Maps Best (out of (7)
contains 14 popular tasks proposed by SMAC 8 and three
OW-QMIX
RODE
more super hard cooperative tasks proposed by QPLEX [7].
In the micromanagement scenarios, each unit is controlled by an independent agent that must act based on its own local observation, and the enemy units are controlled by a built-in AI.
For evaluation, we compare EMC with the state-of siic--ofthhr the- art algorithms: RODE [40], QPLEX 77, MAVEN (15), and the two variants of QMIX 5) CW-QMIX and OW- QMIX (39]. All experimental results are illustrated with the median performance and 25-75% percentiles. Figure 6
0%
40%
60%
100%
Training Percentage
Figure 6: The number of scenarios in which the algorithm's median test win rate is the highest by as least 1/32.

---

EMC (Ours)
GPLEX
QTRAN
QMIX
VON
CW-QMMIX
OW-QMIX
Median Test Win Rate %
Median Test Win Rate %
Median Test Win Rate %
100
100
100
80
80
80
60
60
40
0.0M 0.4M 0.8M 1.2M 1.6M 2.0M Timesteps
0.0M 0.4M 0.8M 1.2M 1.6M
2.0M
0.0M 0.4M 0.8M 1.2M 1.6M 2.0M
Timesteps
Timesteps
(a) corridor
(b) 3s52_vs V3_362
(c) 6h_ vs_8z
Median Test Win Rate %
Median Test Win Rate %
Median Test Win Rate %
100
100
80
60
60
40
40
20
20
20
0
0.0M 0.4M 0.8M 1.2M 1.6M 2.0M Timesteps
0.0M 0.4M 0.8M 1.2M 1.6M 2.0M Timesteps
0.0M
0.4M 0.8M 1.2M 1.6M 2.0M Timesteps
(d) 1c3s8z vs_ 1c3992
(e) 5s10z
(l) 7s7z
Figure 7: Results of super hard maps in SMAC.
shows the overall performance of the tested algorithms in all these 17 maps. Due to the effective exploration with episodic memory which can efficiently use promising exploratory experience trajectories, EMC the best performer on up to 6 tasks, underperforms on just three tasks, and ties for the best performer on the rest tasks.
The advantages of our algorithm can be mainly illustrated by the results of the six hard maps which need sufficient exploration shown in Figure 7 The three maps in the first row are super hard, and solving them needs efficient, diverse and coordinated exploration. Thus, we can find that the EMC algorithm significantly outperforms other algorithms in corridor and 3s5z_vs vs 3s6z, and also achieves the best performance (equal to RODE) in 6h vs 8z. To the best of our knowledge, this will be the state-of-the-an results in corridor and 3s5z_ vs vs_3s6.. For the remaining three maps in the second row ( /c3s82_vs _c_3_33,,,,,,,, 5s/0z, and 7s72), where other baselines can also find winning strategies, due to the boost learning process via episodic memory along with efficient exploration, our algorithm EMC still performs the best the three maps, with fastest learning speed and the highest rates achieved.
5.4 Ablation Study
To understand the superior performance of EMC, we carry out ablation studies to test the contribution of its two main components: curiosity module and episodic memory. Following methods are included in the evaluation: (i) EMC without curiosity module (denoted by EMC-wo-C); (ii) EMC without episodic memory component (denoted by EMC-wo-M); (iii) QPLEX, which can be considered as EMC without the episodic memory component nor the curiosity module, provides a natural ablation baseline of EMC.
Figure 8 bbcc shows that in easy exploration maps, both EMC and EMC-wo-C achieve the sfate-of-the- art performance, which implies that in the easy tasks, sufficient exploration can be achieved simply by the popular e-greedy method. However, in super hard exploration maps (Figure g (a)), EMC-wo-C cannot solve this task but EMC has excellent performance. These empirical experiments show that the curiosity module plays a vital role in improving performance when sufficient and coordinated exploration is necessary. On the other hand, making the best use of good trajectories collected by exploration is also essential. As shown Figure EMC with episodic memory enjoys better sample efficiency than EMC-wo-Mr Mo in challenging (Figure Xh) and easy exploration tasks (Figure (Bb-c)). In general, the curiosity module and the episodic memory complement each other, and efficiently using promising exploratory experience trajectories leads the superior performance of EMC.
Like single-agent curiosity or RND 12 exploration methods, our approach looks simple yet effective. In addition, its design choices do not look straightforward before we know how to do it right.

---

Median Test Win Rate %
Median Test Win Rate %
100
Median Test Win Rate %
100
LMC (Chrs)
EMC
80
EMC-a0-C
EMC we
EMC-ww-M
EMC M
QPLEX
60
QPLEX
60
EMC-wo- C
20
20
QPLEX
0.0M 0.4M 0.8M 1.2M 1.6M
2.0M
0.0M 0.2M 0:1M 0.6M 0.8M 1.0M
0.0M 0.2M 0:4M 0.6M 0.8M 1.0M Timesteps
Timesteps
Timesteps
(a) corridor
(b) 2s3z
(c) 3s5z
Figure 8: Ablation study on the two major components.
Median Test Win Rutt %
Mean Test Win Rate %6
100
100
(Oum)
(Qumi
Meam Test Win Rate
(Ound)
NO
EMC-TD
NO
EMC-TD
80
EMC-TD
60
LMC-DDe
60
EMC-Drc EMC-Cun
60
EMC-Drc
EMC-Cun
EMC-Ces
40
OPLEX
QPLEX
40
QPLEX
20
20
20
0.0M
0.4M 0.8M 12M
1.6
2.0M
0.0M
0.4M 0.8M 1.2M 1.6M
2.0M
0.0M
0.4M
0.8M 1.2M
1.6M 2.0M
T Timesteps
Timesteps
Timestops
(a) corridor
(b) 3s5z_ _ 3s62
(c) 6h
Figure 9: Ablation study on design choice.
Therefore conduct additional ablation studies to demonstrate the effect of our elaborate formulation of curiosity bias. We introduce several baselines and compare them with EMC : (i) using the normalized TD-error of Qtotel as curiosity rewards, denoted EMC-TD; (ii) using the averaged error between the individual utilities and their targets as intrinsic rewards, denoted as EMC-Ind; (iii) using the TD error of centralized critic of the controllers which conditions all agents' histories and actions, denoted EMC-Cen; (iv) using the averaged prediction errors of Qurttidec which are trained in decentralized way. denoted as EMC-Dec. We aim to investigate the subtle implementation difference between EMC and EMC-TD as well EMC-Ind, and compare the exploration efficiency of our method with the global curiosity-driven exploration method (EMC-Cen) and local curiosity-drice driven exploration method (EMC-Dec) empirically.
We design a variant of the toygame mentioned in section which has an additional random noisy reward region. By visualizations, demonstrate that the agents of EMC-TD and End tend to get stuck in the noisy-reward region, thus resulting in sub-optimal policy, while our method show superior ability for avoiding such noise-spike problem. On the other hand, since EMC-Cen is based on global curiosity, which encourages agents to explore the whole state space without bias, it may fail in finding sparse but valuable interaction patterns in the exponentially growing space in complex tasks. When comparing with EMC and EMC-Dec, we find that the key difference is the counterfactual baseline (Eq. (2), which can theoretically reduce the variance of EMC 4T. Therefore, EMC can focus more on the individual specific contribution and achieve the significant improvements.
We test these baselines in SMAC and the results shown in Figure 9 and method significantly outperform other baselines. In general. by conducting these ablations, we demonstrate the robustness for noise spikes of design choice ((i) and (ii),, as well the efficiency and stability of our method compared with centralized or decentralized curiosity-drive -driven exploration method. More detailed discussions will be deferred Appendix E.
Conclusions and Future Work
This paper introduces EMC, a novel episodic multi-agent reinforcement learning algorithm with curiosity-drive exploration framework that allows for efficient coordinated exploration and boosted policy training by exploiting explored informative experiences. Based on the effective exploration ability, our method shows significant outperformance over state-of-the-are MARL baselines on challenging tasks in the StarCraft II micromanagement benchmark. The limitation of our work lies in the lack adaptive exploration methods to ensure robustness. Besides, the episodic memory may get problems in stochastic settings. For future work, may conduct further research in these directions.
10

---

Acknowledgments and Disclosure of Funding
We would like to thank the anonymous reviewers for their valuable comments and helpful suggestions. This work is supported in part by Science and Technology Innovation 2030 "New Generation Artificial Intelligence" Major Project (No. 2018AAA0100900), a grant from the Institute of Guo Qiang, Tsinghua University, and grant from Turing Al Institute of Nanjing.
References
[1] Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial informatics, 9())4277-338 2012.
[2] Maximilian Hüttenrauch, Adrian Šošić, and Gerhard Neumann. Guided deep reinforcement learning for swarm systems. arXiv preprint arXiv:1709.0600 2017.
[3] Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4))819 840, 2002.
[4] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. delle-decompooition networks for cooperative multi-agent learning based on team reward. In AAMAS, pages 2085-2087, 2018.
[5] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference Machine Learning, pages 4295-4304, 2018.
[6] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. arXiv preprint arXiv: 055408 2019.
[7] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent gelaarning. arXiv preprint arXiv: 01062, 2020.
[8] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon White- son. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 2186-2188, 2019.
[9] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in neural information processing systems, pages 1471-1479, 2016.
[10] Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International Conference on Machine Learning, pages 2721- 2730, 2017.
[11] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, 2017.
[12] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint 1810. 2018.
[13] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pages 109-1117, 2016.
[14] Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. Influence-based multi-agent exploration. In International Conference Learning Representations, 2019.
11

---

[15] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi- agent variational exploration. In Advances in Neural Information Processing Systems, pages 7613-7624, 2019.
[16] Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. Emi: Exploration with mutual information. arXiv preprint urXiv:1810.01 2018
[17] Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, al. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2019.
[18] Adrià Puigdomenech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvit- skyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pages 507-517. PMLR, 2020.
[19] Lihong Li, Thomas J Walsh, and Michael L Littman. Towards unified theory of state abstraction for mdps. ISAIM, 4:5, 2006.
[20] Jianhao Wang, Zhizhou Ren, Beining Han, Jianing Ye, and Chongjie Zhang. Towards un- derstanding linear value decomposition in cooperative multi-agent q-learning. arXiv preprint arXiv:2006.00587 2020.
[21] Zichuan Lin, Tianqi Zhao, Guangwen Yang, and Lintao Zhang. Episodic memory deep q- networks. IJCAI, 2018.
[22] Guangxiang Zhu, Zichuan Lin, Guangwen Yang, and Chongjie Zhang. Episodic reinforcement learning with associative memory. In International Conference on Learning Representations, 2019.
[23] Frans A Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
[24] Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008.
[25] Pierre-Yve Yves Oudeyer, Frdric Kaplan, and Verena Hafner. Intrinsic motivation systems for autonomous mental development. IEEE transactions on evolutionary computation, 11(2):265- 286, 2007.
[26] Pierre- Oudeyer and Frederic Kaplan. What intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics, 1:6, 2009.
[27] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pages 2753-2763 2017.
[28] Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predietive models. arXiv preprint arXiv: 2015.
[29] Shakir Mohamed and Danilo Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Proceedings of the 28th International Conference on Neural Information Processing Vostums-Volume 2, pages 2125-2133, 2015.
[30] Eugenio Bargiacchi, Timothy Verstraeten, Diederik Roijers, Ann Nowé, and Hado Hasselt. Learning to coordinate with coordination graphs in repeated single-stage multi-agent decision problems. In International conference on machine learning, pages 482-490. 2018.
[31] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pages 3040-3049, PMLR, 2019.
12

---

[32] Shariq Iqbal and Fei Sha. Coordinated exploration via intrinsic rewards for multi agent rein- forcement learning. arXiv preprint arXiv:1905. 12127, 2019.
[33] M Lengyel and Dayan. Hippocampal contributions to control: The third way. In Twenty-First Annual Conference on Neural Information Processing Systems (NIPS 2007), pages 889-896. Curran, 2008
[34] Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprint arXiv: 04460, 2016.
[35] Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In International Conference on Machine Learning, pages 2827-2222222222 2836. PMLR, 2017.
[36] Steven S Hansen, Pablo Sprechmann, Alexander Pritzel, André Barreto, and Charles Blundell. Fast deep reinforcement learning using online adjustments from the past. Proceedings of the 32nd International Conference Neural Information Processing Systems, pages 10590-10600, 2018.
[37] Samuel Gershman and Nathaniel D Daw. Reinforcement learning and episodic memory in humans and animals: integrative framework. Annual review of psychology, 68:101-128, 2017.
[38] Timothy P Lilliccaap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1xXiv1159.02997 2015.
[39] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted QMIX: ex- panding monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of the Annual Conference on Neural Information Processing Systems 2020 (NeurIPS 2020), 2020.
[40] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523, 2020.
[41] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Dop: Off- policy multi-agent decomposed policy gradients. In International Conference on Learning Representations, 2020.
[42] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei Efros. Large-scale study of curiosity-driver learning. arXiv preprint 180::04083.555 2018.
[43] Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint trXiv:1803.0093. 2018.
[44] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint Xrxiv: /5/ 1.0555,, 2015.
[45] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty- AAAI conference on artificial intelligence, 2018
13


Reviews:
[{"review_id": "fotg8hl80K", "reviewer": "Reviewer_DyLw", "summary": "The paper proposes an exploration extension of value-decomposition algorithms in multi-agent reinforcement learning (MARL) based on two orthogonal components: (i) a intrinsic curiosity reward based on temporal difference errors of individual agents' utilities and (ii) an episodic memory module. The framework is evaluated for VDN, QMIX and QPLEX on a toy-task, a predator-prey task and 17 StarCraft micro-management tasks. The proposed methods and some ablations are compared with state-of-the-art baselines and show a significant advantage in hard exploration tasks, which seems to mostly be due to contribution (i).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an interesting problem and is generally well written. The proposed contribution (ii) seems to be mostly an application of existing techniques, but contribution (i) appears novel and sufficiently complex to warrant publication. However, the reviewer could not always follow the design decisions and would like to see some ablations to justify them. It seems to the reviewer that the two contributions address very different problems, which is not very well discussed. The existing experiments appear thorough enough, but the reviewer would like to see some additional comparisons in the toy examples, both for a fair comparison with literature and to clarify the different effects of the two contributions better.\n\nIn particular, the reviewer would like the authors to respond to the following criticisms:\n1. The curiosity module appears unnecessary complex. The reviewer could not see the point of regressing the VDN utilities, if the curiosity reward is just the average of the resulting errors. Why not use the TD-error of $Q^{ext}_{tot}$ as curiosity reward? The proposed method adds significant complexity, which needs to be justified by an empirical ablation (comparison to the above suggestion). \n2. The paper makes a big deal of separating local and global curiosity, but the distinction never seems to play a role in the method (except for QPLEX-{global,local}). Regressing $Q^{ext}_{tot}$ seems to be the authors attempt to use local curiosity, but global curiosity is never truly investigated. For example, you could make an ablation that uses the TD error of a centralized critic of the controllers $Q_i$, which conditions on all agents' histories and actions, as global curiosity for a proper comparison.\n3. The didactic example and the predator-prey task are not compared against MAVEN, which is curious, as this is the only explicit exploration method that is compared against. Right now EMC beats only methods that do not explicitly incentivize exploration, which is not a sufficient comparison on these domains.\n4. Both weighted QMIX and MAVEN are approaches to deal with non-monotonic value functions, and both the didactic example and predator-prey (with high enough punishment) exhibit such non-monotonic values. It is therefore necessary to test weighted QMIX (and ideally MAVEN) on the didactic example to draw the conclusion that \"other methods failed\" (l.276). Moreover, it begs the question which effect allows EMC (and probably CW/OW-QMIX) to succeed where others have failed: the exploration advantage or the non-monotonic values? The reviewer has the distinct impression that the two contributions of EMC affect these two components very differently and highly recommends to run the ablations EMC-wo-C and EMC-wo-M on both the didactic example and the predator-prey task to find out!\n\nThe reviewer may be swayed to increase the rating if the authors can either convincingly argue against these points or provide additional ablations that provide satisfactory answers. Otherwise the reviewer encourages the authors to improve the paper on these points before resubmission.\n\nREFERENCES\n- Lillicrap, et al. (2016): Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR). URL http://arxiv.org/abs/1509.02971.\n- Gupta et al. (2021): UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning. Proceedings of the International Conference on Machine Learning. URL https://arxiv.org/abs/2010.02974\n\nDETAILED COMMENTS\n- l.27: commata\n- l.31ff: do not call $Q_i$ value functions, as they do not obey any Bellman equation, but *utilities* (as in VDN)!\n- l.40: \"since\" -> \"since they\"\n- l.104: histories are in $(\\Omega \\times \\mathcal A) \\times \\Omega$\n- l.106: it must be a transition distribution, not a function\n- l.107ff: it makes more sense to introduce the policy and all values conditioning on $\\tau$ rather than $s$ (see [18])\n- eq.1: the use of $V$ is ambiguous (do you mean $V^\\pi$, $V^*$ or the value induced by the data distribution $D$?)\n- l.147: clarify which challenge are you referring to specifically\n- eq.2: the equation should be in the background section (as it is not yours). It is generally questionable whether this equation, which needs a lot of space and brain power to understand, is the best way to make your point. For example, Gupta et al. (2021) derive a much simpler analytic VDN solution in the appendix.\n- l.191f: the claim \"Thus predicting $Q_i$ can capture both the novelty of states and the interaction between agents and lead agents to explore promising states.\" seems to be the deduction \"$Q_i$ is influenced by other agents $\\to$ errors of $Q_i$ capture interactions between agents\", which you cannot make without further assumptions!\n- l.193: \"orediction\"\n- l.197 the four components are very confusing. Just say (ii) is a model that is trained with (i) and (iii) is a model that is trained with (iv).\n- l.204: don't refer to (eq.2) here, which is very confusing, better refer to (eq.1).\n- eq.4: rename $Q_{tot}^{all}$  to $Q_{tot}$, or even better rename $Q_{tot}$ to $Q_{tot}^{inf}$ consistently to differentiate it from $Q_{tot}^{ext}$.\n- l.211: Do you refer to soft-update of the target parameters $\\theta^-$? If so, cite Lillicrap et al. (2016) here. Otherwise reformulate to avoid confusion.\n- l.216: Don't say that you refer to QPLEX for the rest of the paper, as you apply EMC to VDN, QMIX and QPLEX later. \n- l.222: \"Inspires\" $\\to$ \"Inspired\"\n- l.236: clarify that you mean the return of the episode\n- sec.4.2: the memory approach depends massively on the similarity metric induced by the random hash function and is bound to produce \"wrong\" targets sometimes. Have you tried to decay $\\lambda$ to compensate for errors introduced by the projection?\n- l.245: \"experiments\" $\\to$ \"experimental\"\n- l.248: \"if\" $\\to$ \"If\"\n- l.259: goal grid should refer to character $G$ in Figure 3\n- fig.6+l.329f: this figure is extremely misleading! How significant are these comparisons really? If you define the metric mean1 - std1 > mean2 + std2, how does this change these numbers? Please stay away from comparisons like this, unless you count statistically significant events.\n- fig.7: MAVEN showed very differentiated exploration behavior when run longer. Maybe you should add a representative level (e.g. corridor) that runs for 10M steps to see how your approach fairs against the others in the long run?\n- l.353: The reviewer interpreted the ablation results differently: Fig.8a shows that in \"corridor\", the curiosity is the important quantity, whereas Fig.8b+c show a slight advantage introduced by the memory module. This indicates that the two contributions address different issues! In particular the sentence \"As shown Figure 8, EMC with episodic memory enjoys better sample efficiency than EMC-wo-M in challenging (Figure 8 (a)) and easy exploration tasks (Figure 8 (b-c)).\" seems questionable.\n- l.416+418: double reference\n\n**Post rebuttal comment**\nAfter reading the reviews and authors responses, and after a long discussion with the authors, the reviewer is convinced that the the discussed effect is significant and the paper is of sufficient interest to the community. However, the reviewer only recommends publication under the condition that (i) the promised changes are incorporated; (ii) the additional experiments and results are summarized in the main text; (iii) the full results the authors posted in their responses are added to the appendix; and (iv) the major points that emerged in the discussion should be integrated in the main text. The authors are welcome to emphasize their own point of view, but must at least mention other arguments and why they do not follow them.   ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no specific discussion of limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "TwqhkW6TReY", "reviewer": "Reviewer_hMh6", "summary": "This paper introduces a new curiosity-driven exploration framework for multi-agent reinforcement learning. The proposed framework introduces the following two modules to be added upon conventional centralized training with decentralized execution (CTDE) approaches: (1) a reward shaping exploration module which infers curiosity of agent based on the error of Q-value prediction; (2) a episodic memory module which saves informative experiences and form a regularization loss $L_{memory}$ to regularize the Q-value update. The proposed method is evaluated on a toy grid world game with 2 coordinated agents, a predator prey game, and 17 maps in  StarCraftII Micromanagement benchmark tasks.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nThis work combines two existing approaches, which is prediction-error based reward shaping and the usage of episodic memory.  Such combination is not new in deep RL literature as there is already prior work [1] that combines those two modules in single-agent deep RL settings. Therefore I feel the novelty of this work is very limited. \n\n\nQuality: \n\nThis method is technically sound. It is an application of prediction error-based reward shaping in multi-agent deep RL domains. The evaluation is very comprehensive. The proposed method achieves promising improvements over baseline approaches on three different multi-agent RL domains.  \n\n\nClarity:\n\nThe paper is easy to follow but both its organization and technical formulation are weak. Several figures are not put in appropriate places (i.e., Fig 2 and Fig 4) which makes it a bit hard to read. Related work section appears after Background section. Some terms in Eq (2) have not been properly defined, e.g., the meaning of x; why there is $\\tau'$ and $a'$ in term 1 of Eq (2), etc. \n\n\nSignificance:\n\nThough the empirical result of this work seems very promising, I think its significance is still somehow limited due to the limited novelty of the method. Also, the ablation study results could not verify that curiosity-driven exploration plays important roles in multi-agent deep RL.\n\n\nDetailed Comments:\n\nMy primary concern of this work is that it employs both curiosity-driven exploration and episodic memory but the connection between these two parts is very loose. The curiosity computation has nothing to do with the episodic memory and the episodic memory part is not something novel. Therefore this makes the novelty of this work very weak. Also, from multi-agent perspective, the curiosity computation seems also not that surprising. The reward bonus is computed upon each individual Q network and therefore is local, while its effect of capturing the global curiosity comes from using a summed target Q. A more novel perspective of inferring curiosity under a multi-agent RL setting is expected.\n\nMy second concern is that the ablation study result seems to show that curiosity computation is not a key to drive superior performance. At least in 2 out of the 3 ablation study domains, the baseline EMC without curiosity module performs really on-par with full EMC. In all the baselines, full EMC ends up with similar asymptotic convergent performance compared to some baselines. This makes me have a feeling that the effect of curiosity module is somehow weak and therefore the overall significance of this work becomes weakened.\n\nI am also curious how the baselines are implemented and where those baselines' performance can be easily replicated. In the original baseline papers, they use some disjoint evaluation domains and therefore it is hard to justify if the baselines are fully optimized. It is good if the authors provide implementation of the full compared methods for reproducibility check.\n\n[1] Episodic curiosity through reachability, ICLR2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe this work would not trigger any negative social impact as it is established upon synthetic environments.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hrs"}, "extra_text": {}}, {"review_id": "HyhzrcDJiF9", "reviewer": "Reviewer_DEY8", "summary": "This paper introduces a method for improving exploration in multi-agent Reinforcement Learning (RL) based on sharing local observations among agents as a measure for intrinsically motivated exploration. The proposed method can be easily integrated in many multi-agent RL algorithms, and show improvements in experimental evaluations upon baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method introduced in this paper is a simple, almost straightofrward, application of the intrinsically motivation paradigm in multi-agent Reinforcement Learning (RL). As described by the authors, their method is a simple addition to algorithms introduced in previous works. Overall, the theoretical and methodological contribution of this paper is quite limited.\nHowever, I think the authors did a good job providing reasonable motivations for their method, integrating it in a multi-agent RL scheme in an effective way, such that it can be seamlessly applied in different learning algorithms. The description of related works is satisfying enough. The description of the method in Section 4 is also clear, although the quality of the writing of subsection 4.2 is significantly lower than for the rest of the paper. Experimentally, the proposed method achieves remarkable results in meaningful problems, and it is compared to a satisfying amount of baselines. Overall, my opinion about the paper is that this is a simple contribution, yet well done, especially for its empirical analysis.\nI have few doubts that I’d like the author to address:\n\n- Can you clarify how your method scales w.r.t. the number of agents? I don’t have much experience with Starcraft problems, how many agents are trained in this task?\n- It seems that your method can incur in local optima, as also stated in the conclusion. How does this affect the success of your method in stochastic environments? It would be great if you can provide additional results on a stochastic variant of the grid world in the didactic example.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "FuQVxzVHoHK", "reviewer": "Reviewer_d4Cx", "summary": "The paper studies multi-agent RL, and introduces an exploration reward generated by individual agents based on local (i.e. decentralized) prediction errors by the individual Q-networks. There is also an extra loss term based on experience stored in a memory buffer. The experiments, including comparisons to established methods and ablation studies evaluating each contribution individually, show that both the intrinsic reward and the memory-based loss term lead to faster policy improvement.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The decentralized, individual agent based intrinsic reward is a nice contribution. The performance is compared to that of existing methods on established benchmarks, showing state of the art results. This paper presents a step forward for the MARL literature, and I believe it should be accepted.\n\nI do have some questions and concerns, which I would like to see addressed or at least discussed.\n\nThe intrinsic reward is based on prediction errors, like e.g. RND (Burda et al.) and ICM (Pathak et al.). The target here is different - using the individual Q-value prediction errors as intrinsic rewards has the upside that it is focused on reward-related regions of the observation-action space, which are relevant to the overall objective. However, this approach could easily become problematic when there is a need to explore long sequences without extrinsic reward, as both the collective and the individual Q-values would be zero there, leading to a perfect prediction and leaving the agents without an exploration incentive. A similar concern applies to situations where the policy becomes deterministic: then the prediction errors also vanish over time. This is a general weakness of prediction error based intrinsic rewards: there must be an interesting error to go after.\n\nRegarding the ‘key insight’ that individual Q-values embed local observation-action histories: if the individual Q-values are computed as the outputs of LSTMs, then the recurrent states of those LSTMs are likely to embed local observation-action histories, but this is not different from the single-agent case: there the recurrent states also embed action-observation histories. Can you explain what you see as the fundamental difference between single- and multi-agent settings here?\n\nThe description of the memory looks like it only operates on exact matches. Why did you make this choice? As the environments get more complex, the chances of encountering an exact match in memory decrease, which means the contribution of the memory-based loss term becomes less relevant (it is zero more often) as the environment is scaled up. Did you consider approximate matching, like in MFEC / NEC? Since you are using the memory in a way similar to the replay buffer (to draw training data from), have you considered merging it with the actual replay buffer? It seems that the matching and target substitution could be done there as well.\n\nA question about those baselines: the WQMIX curves you include look worse than the ones reported in the WQMIX paper (Rashid et al.), both on predator-prey and on some of the SMAC tasks. Can you comment on that?\n\nSome nits:\n- What is the difference between lower-case n and capital N in section 2.1? There are n agents, but in the definition of the joint action it seems there are N of them; similar for the definition of the joint observation-action history.\n- A round of language editing could improve the legibility of the paper. Articles are frequently missing, and in some places the phrasing is awkward (e.g. line 219 ‘how to use the best use of’).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations are briefly addressed at the end; I have asked a number of questions relating to potential limitations in the main review. The potential negative societal impact is not discussed, but for this specific work it does not seem very direct or different from the existing literature.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a well written paper, exploring how recent advances in RL can be adapted, extended and applied in MARL. The reviewers were satisfied with the authors responses (except one reviewer who did not respond). The experimental evaluation is good, the paper clearly written. During discussions, the reviewers expressed that the explanation of how all the constituent parts fit together should be expanded upon to assist the reader.
Decision: Accept (Poster)