Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies multi-agent RL, and introduces an exploration reward generated by individual agents based on local (i.e. decentralized) prediction errors by the individual Q-networks. There is also an extra loss term based on experience stored in a memory buffer. The experiments, including comparisons to established methods and ablation studies evaluating each contribution individually, show that both the intrinsic reward and the memory-based loss term lead to faster policy improvement.
main_review: The decentralized, individual agent based intrinsic reward is a nice contribution. The performance is compared to that of existing methods on established benchmarks, showing state of the art results. This paper presents a step forward for the MARL literature, and I believe it should be accepted.

I do have some questions and concerns, which I would like to see addressed or at least discussed.

The intrinsic reward is based on prediction errors, like e.g. RND (Burda et al.) and ICM (Pathak et al.). The target here is different - using the individual Q-value prediction errors as intrinsic rewards has the upside that it is focused on reward-related regions of the observation-action space, which are relevant to the overall objective. However, this approach could easily become problematic when there is a need to explore long sequences without extrinsic reward, as both the collective and the individual Q-values would be zero there, leading to a perfect prediction and leaving the agents without an exploration incentive. A similar concern applies to situations where the policy becomes deterministic: then the prediction errors also vanish over time. This is a general weakness of prediction error based intrinsic rewards: there must be an interesting error to go after.

Regarding the ‘key insight’ that individual Q-values embed local observation-action histories: if the individual Q-values are computed as the outputs of LSTMs, then the recurrent states of those LSTMs are likely to embed local observation-action histories, but this is not different from the single-agent case: there the recurrent states also embed action-observation histories. Can you explain what you see as the fundamental difference between single- and multi-agent settings here?

The description of the memory looks like it only operates on exact matches. Why did you make this choice? As the environments get more complex, the chances of encountering an exact match in memory decrease, which means the contribution of the memory-based loss term becomes less relevant (it is zero more often) as the environment is scaled up. Did you consider approximate matching, like in MFEC / NEC? Since you are using the memory in a way similar to the replay buffer (to draw training data from), have you considered merging it with the actual replay buffer? It seems that the matching and target substitution could be done there as well.

A question about those baselines: the WQMIX curves you include look worse than the ones reported in the WQMIX paper (Rashid et al.), both on predator-prey and on some of the SMAC tasks. Can you comment on that?

Some nits:
- What is the difference between lower-case n and capital N in section 2.1? There are n agents, but in the definition of the joint action it seems there are N of them; similar for the definition of the joint observation-action history.
- A round of language editing could improve the legibility of the paper. Articles are frequently missing, and in some places the phrasing is awkward (e.g. line 219 ‘how to use the best use of’).
limitations_and_societal_impact: Some limitations are briefly addressed at the end; I have asked a number of questions relating to potential limitations in the main review. The potential negative societal impact is not discussed, but for this specific work it does not seem very direct or different from the existing literature.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper introduces a new curiosity-driven exploration framework for multi-agent reinforcement learning. The proposed framework introduces the following two modules to be added upon conventional centralized training with decentralized execution (CTDE) approaches: (1) a reward shaping exploration module which infers curiosity of agent based on the error of Q-value prediction; (2) a episodic memory module which saves informative experiences and form a regularization loss $L_{memory}$ to regularize the Q-value update. The proposed method is evaluated on a toy grid world game with 2 coordinated agents, a predator prey game, and 17 maps in  StarCraftII Micromanagement benchmark tasks.
main_review: Originality:

This work combines two existing approaches, which is prediction-error based reward shaping and the usage of episodic memory.  Such combination is not new in deep RL literature as there is already prior work [1] that combines those two modules in single-agent deep RL settings. Therefore I feel the novelty of this work is very limited. 


Quality: 

This method is technically sound. It is an application of prediction error-based reward shaping in multi-agent deep RL domains. The evaluation is very comprehensive. The proposed method achieves promising improvements over baseline approaches on three different multi-agent RL domains.  


Clarity:

The paper is easy to follow but both its organization and technical formulation are weak. Several figures are not put in appropriate places (i.e., Fig 2 and Fig 4) which makes it a bit hard to read. Related work section appears after Background section. Some terms in Eq (2) have not been properly defined, e.g., the meaning of x; why there is $\tau'$ and $a'$ in term 1 of Eq (2), etc. 


Significance:

Though the empirical result of this work seems very promising, I think its significance is still somehow limited due to the limited novelty of the method. Also, the ablation study results could not verify that curiosity-driven exploration plays important roles in multi-agent deep RL.


Detailed Comments:

My primary concern of this work is that it employs both curiosity-driven exploration and episodic memory but the connection between these two parts is very loose. The curiosity computation has nothing to do with the episodic memory and the episodic memory part is not something novel. Therefore this makes the novelty of this work very weak. Also, from multi-agent perspective, the curiosity computation seems also not that surprising. The reward bonus is computed upon each individual Q network and therefore is local, while its effect of capturing the global curiosity comes from using a summed target Q. A more novel perspective of inferring curiosity under a multi-agent RL setting is expected.

My second concern is that the ablation study result seems to show that curiosity computation is not a key to drive superior performance. At least in 2 out of the 3 ablation study domains, the baseline EMC without curiosity module performs really on-par with full EMC. In all the baselines, full EMC ends up with similar asymptotic convergent performance compared to some baselines. This makes me have a feeling that the effect of curiosity module is somehow weak and therefore the overall significance of this work becomes weakened.

I am also curious how the baselines are implemented and where those baselines' performance can be easily replicated. In the original baseline papers, they use some disjoint evaluation domains and therefore it is hard to justify if the baselines are fully optimized. It is good if the authors provide implementation of the full compared methods for reproducibility check.

[1] Episodic curiosity through reachability, ICLR2019.
limitations_and_societal_impact: I believe this work would not trigger any negative social impact as it is established upon synthetic environments.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a method for improving exploration in multi-agent Reinforcement Learning (RL) based on sharing local observations among agents as a measure for intrinsically motivated exploration. The proposed method can be easily integrated in many multi-agent RL algorithms, and show improvements in experimental evaluations upon baselines.
main_review: The method introduced in this paper is a simple, almost straightofrward, application of the intrinsically motivation paradigm in multi-agent Reinforcement Learning (RL). As described by the authors, their method is a simple addition to algorithms introduced in previous works. Overall, the theoretical and methodological contribution of this paper is quite limited.
However, I think the authors did a good job providing reasonable motivations for their method, integrating it in a multi-agent RL scheme in an effective way, such that it can be seamlessly applied in different learning algorithms. The description of related works is satisfying enough. The description of the method in Section 4 is also clear, although the quality of the writing of subsection 4.2 is significantly lower than for the rest of the paper. Experimentally, the proposed method achieves remarkable results in meaningful problems, and it is compared to a satisfying amount of baselines. Overall, my opinion about the paper is that this is a simple contribution, yet well done, especially for its empirical analysis.
I have few doubts that I’d like the author to address:

- Can you clarify how your method scales w.r.t. the number of agents? I don’t have much experience with Starcraft problems, how many agents are trained in this task?
- It seems that your method can incur in local optima, as also stated in the conclusion. How does this affect the success of your method in stochastic environments? It would be great if you can provide additional results on a stochastic variant of the grid world in the didactic example.
limitations_and_societal_impact: Not applicable.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes an exploration extension of value-decomposition algorithms in multi-agent reinforcement learning (MARL) based on two orthogonal components: (i) a intrinsic curiosity reward based on temporal difference errors of individual agents' utilities and (ii) an episodic memory module. The framework is evaluated for VDN, QMIX and QPLEX on a toy-task, a predator-prey task and 17 StarCraft micro-management tasks. The proposed methods and some ablations are compared with state-of-the-art baselines and show a significant advantage in hard exploration tasks, which seems to mostly be due to contribution (i).
main_review: The paper addresses an interesting problem and is generally well written. The proposed contribution (ii) seems to be mostly an application of existing techniques, but contribution (i) appears novel and sufficiently complex to warrant publication. However, the reviewer could not always follow the design decisions and would like to see some ablations to justify them. It seems to the reviewer that the two contributions address very different problems, which is not very well discussed. The existing experiments appear thorough enough, but the reviewer would like to see some additional comparisons in the toy examples, both for a fair comparison with literature and to clarify the different effects of the two contributions better.

In particular, the reviewer would like the authors to respond to the following criticisms:
1. The curiosity module appears unnecessary complex. The reviewer could not see the point of regressing the VDN utilities, if the curiosity reward is just the average of the resulting errors. Why not use the TD-error of $Q^{ext}_{tot}$ as curiosity reward? The proposed method adds significant complexity, which needs to be justified by an empirical ablation (comparison to the above suggestion). 
2. The paper makes a big deal of separating local and global curiosity, but the distinction never seems to play a role in the method (except for QPLEX-{global,local}). Regressing $Q^{ext}_{tot}$ seems to be the authors attempt to use local curiosity, but global curiosity is never truly investigated. For example, you could make an ablation that uses the TD error of a centralized critic of the controllers $Q_i$, which conditions on all agents' histories and actions, as global curiosity for a proper comparison.
3. The didactic example and the predator-prey task are not compared against MAVEN, which is curious, as this is the only explicit exploration method that is compared against. Right now EMC beats only methods that do not explicitly incentivize exploration, which is not a sufficient comparison on these domains.
4. Both weighted QMIX and MAVEN are approaches to deal with non-monotonic value functions, and both the didactic example and predator-prey (with high enough punishment) exhibit such non-monotonic values. It is therefore necessary to test weighted QMIX (and ideally MAVEN) on the didactic example to draw the conclusion that "other methods failed" (l.276). Moreover, it begs the question which effect allows EMC (and probably CW/OW-QMIX) to succeed where others have failed: the exploration advantage or the non-monotonic values? The reviewer has the distinct impression that the two contributions of EMC affect these two components very differently and highly recommends to run the ablations EMC-wo-C and EMC-wo-M on both the didactic example and the predator-prey task to find out!

The reviewer may be swayed to increase the rating if the authors can either convincingly argue against these points or provide additional ablations that provide satisfactory answers. Otherwise the reviewer encourages the authors to improve the paper on these points before resubmission.

REFERENCES
- Lillicrap, et al. (2016): Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR). URL http://arxiv.org/abs/1509.02971.
- Gupta et al. (2021): UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning. Proceedings of the International Conference on Machine Learning. URL https://arxiv.org/abs/2010.02974

DETAILED COMMENTS
- l.27: commata
- l.31ff: do not call $Q_i$ value functions, as they do not obey any Bellman equation, but *utilities* (as in VDN)!
- l.40: "since" -> "since they"
- l.104: histories are in $(\Omega \times \mathcal A) \times \Omega$
- l.106: it must be a transition distribution, not a function
- l.107ff: it makes more sense to introduce the policy and all values conditioning on $\tau$ rather than $s$ (see [18])
- eq.1: the use of $V$ is ambiguous (do you mean $V^\pi$, $V^*$ or the value induced by the data distribution $D$?)
- l.147: clarify which challenge are you referring to specifically
- eq.2: the equation should be in the background section (as it is not yours). It is generally questionable whether this equation, which needs a lot of space and brain power to understand, is the best way to make your point. For example, Gupta et al. (2021) derive a much simpler analytic VDN solution in the appendix.
- l.191f: the claim "Thus predicting $Q_i$ can capture both the novelty of states and the interaction between agents and lead agents to explore promising states." seems to be the deduction "$Q_i$ is influenced by other agents $\to$ errors of $Q_i$ capture interactions between agents", which you cannot make without further assumptions!
- l.193: "orediction"
- l.197 the four components are very confusing. Just say (ii) is a model that is trained with (i) and (iii) is a model that is trained with (iv).
- l.204: don't refer to (eq.2) here, which is very confusing, better refer to (eq.1).
- eq.4: rename $Q_{tot}^{all}$  to $Q_{tot}$, or even better rename $Q_{tot}$ to $Q_{tot}^{inf}$ consistently to differentiate it from $Q_{tot}^{ext}$.
- l.211: Do you refer to soft-update of the target parameters $\theta^-$? If so, cite Lillicrap et al. (2016) here. Otherwise reformulate to avoid confusion.
- l.216: Don't say that you refer to QPLEX for the rest of the paper, as you apply EMC to VDN, QMIX and QPLEX later. 
- l.222: "Inspires" $\to$ "Inspired"
- l.236: clarify that you mean the return of the episode
- sec.4.2: the memory approach depends massively on the similarity metric induced by the random hash function and is bound to produce "wrong" targets sometimes. Have you tried to decay $\lambda$ to compensate for errors introduced by the projection?
- l.245: "experiments" $\to$ "experimental"
- l.248: "if" $\to$ "If"
- l.259: goal grid should refer to character $G$ in Figure 3
- fig.6+l.329f: this figure is extremely misleading! How significant are these comparisons really? If you define the metric mean1 - std1 > mean2 + std2, how does this change these numbers? Please stay away from comparisons like this, unless you count statistically significant events.
- fig.7: MAVEN showed very differentiated exploration behavior when run longer. Maybe you should add a representative level (e.g. corridor) that runs for 10M steps to see how your approach fairs against the others in the long run?
- l.353: The reviewer interpreted the ablation results differently: Fig.8a shows that in "corridor", the curiosity is the important quantity, whereas Fig.8b+c show a slight advantage introduced by the memory module. This indicates that the two contributions address different issues! In particular the sentence "As shown Figure 8, EMC with episodic memory enjoys better sample efficiency than EMC-wo-M in challenging (Figure 8 (a)) and easy exploration tasks (Figure 8 (b-c))." seems questionable.
- l.416+418: double reference

**Post rebuttal comment**
After reading the reviews and authors responses, and after a long discussion with the authors, the reviewer is convinced that the the discussed effect is significant and the paper is of sufficient interest to the community. However, the reviewer only recommends publication under the condition that (i) the promised changes are incorporated; (ii) the additional experiments and results are summarized in the main text; (iii) the full results the authors posted in their responses are added to the appendix; and (iv) the major points that emerged in the discussion should be integrated in the main text. The authors are welcome to emphasize their own point of view, but must at least mention other arguments and why they do not follow them.   
limitations_and_societal_impact: There is no specific discussion of limitations and potential negative societal impact.
review_title: null

Meta-review: This is a well written paper, exploring how recent advances in RL can be adapted, extended and applied in MARL. The reviewers were satisfied with the authors responses (except one reviewer who did not respond). The experimental evaluation is good, the paper clearly written. During discussions, the reviewers expressed that the explanation of how all the constituent parts fit together should be expanded upon to assist the reader.
Decision: Accept (Poster)