Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents an approach for modeling partially-observable multi-agent MDPs. The paper proposes two modifications on top of a general transformer for modeling the relationship between observation and action: first a memory module that maintains information from previous observations associated with all agents, and second a prediction layer that filters information for predicting actions by the arguments of that action (e.g., predicting to attack enemy i is conditioned on the observation information about enemy i). They evaluate on two MARL environments and show improvements over existing methodologies for modeling memory over time, including RNN-based methods.
strengths_and_weaknesses: Strengths:

This is an interesting problem and an interesting approach to the problem. The paper is relatively clear and experiments on both environments are mostly comprehensive and performance over compared methods is impressive.

--- 

Weaknesses:

The way that transformer-based working memory is introduced in 42-48 makes it seem as though the main contribution is applying an existing method to an existing problem. I'm not familiar enough with related work to make this judgment, however.

I was confused about what an observation includes: is o_ally the observation *of* an ally, or the ally's observation? If the latter, does this also hold for enemies, and if so, it seems as though an agent shouldn't have access to the enemy's observation. If the former, is there information sharing between agents (i.e., sharing their first-person observations, which may not overlap with another ally's)? How would this affect performance? If would help to have a formal definition of the task, including what exactly observations look like (are they images?). 
questions: * What would alternatives to the entity-bound action layer be? Using only memory of o_self to predict actions, or using some combination (e.g., concatenation) of all observations (which has a drawback of many parameters, some of which may be unnecessary)? I'm a bit confused what it means to ablate EBA in the experiments, and whether you experimented with all three settings (EBA, self-only, and all observations).
* Did you perform any interpretation of the memory? E.g., if memory/observation can be interpreted as an image.
limitations: The entity-bound action layer seems specific to the action spaces available in the two environments. What if the action space is much lower-level, such that there aren't semantic relationships between a particular action and a particular entity in the environment (e.g., instead of "attack enemy i", an agent may have to move to a strategic position and rotation with multiple navigation actions, then "attack" by firing some weapon in the direction it is pointed)? Would the entity-bound action layer still generalize to this?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper is about designing a new architecture, Agent Transformer Memory (ATM), for multiagent reinforcement learning where each agent is equipped with its own working memory and action space. Specifically, the authors propose to use a transformer-based architecture to process both the moving entities in the environment and the recent memory slots. The output of the transformer consists of an updated memory slot (to be pushed to the limited-capacity working memory) and the encoded representations for each entity. Then, each agent selects an action (from their respective action space) based on their most recent memory slot and the representations of their surrounding entities. Experiments were conducted on the well-established StarCraft Multi-Agent Challenge (SMAC) and Level-Based Foraging (LBF) environments. The authors test the proposed agent transformer memory with QMIX and QPLEX algorithms commonly used on SMAC, and with MAPPO and MAA2C for experiments on LBF. Empirically, it was shown that agents with memory and personalized action space perform better. The authors also investigate different memory updating schemes and show that using ATM works best.
strengths_and_weaknesses: **What I like about this paper**
- Each agent having their own point-of-view of the environment, own working memory, and own action space, while using the same model (same weights, different input $T_{in}$).
- The ablation study of the proposed architecture ATM. It does seem like having a working memory is important (even if only one slot).

**Potential weaknesses**
 - Lack of coordination between the agents. If I understood correctly, all agents predict their next action at the same time. That said, it seems to be a common approach in the MARL literature.
 - Even though this work takes place in a partially observable setting, the proposed technique requires to know in advance how many entities (and their attributes) there will be in the game. Since the proposed architecture requires the relative spatial positions between the self entity and all other entities (even if they are out of the field of view?), it is not clear to me what remains "partially observable".
 - It is not clear how the proposed model performs when playing against other learning agents.

**Originality, quality, clarity, and significance**

As pointed out in the related work, the concept of having a transformer-based working memory is not new. The paper would benefit from stating clearly how the proposed technique differs from each related work. The main thing I could see is the application of a transformer encoder in the multi-agent RL setting with agent-centric observations and working memory. It wasn't clear to me how the Entity-Bound Action Layer related to any previous work (if any). I found the paper technically sound and the ablation study is backing up the proposed agent transformer memory component.

I found the paper well-written and well-organized for the most part. In Figure 1, it is not clear what the gray squares represent, is it the spatial/temporal embeddings, or the entity/memory IDs? While it is unclear how the proposed model compares to the actual state-of-the-art on tested environments, the empirical results suggest the ATM architecture would be useful to others in the community.

Overall, I tend to recommend this paper for acceptance.

**Minor**
- p.2 (l.83): should $d_o$ be $d_e$, if not what is $d_o$?
- p.5 (l.164): "for four times" -> "four times"
- p.7 (l.243): "...agent need remember them." -> "..agent needs to ..."
- p.7 (l.244): "...sight field" -> "..sight range"
- p.7 (l.246): "...focusing fire" -> "..focused fire"
questions: - Based on Eq. 4 and 5, should $T_{in}$ and $T_{out}$ be indexed by $i$, i.e., $T_{in}^i$ and $T_{out}^i$?
- In Figure 5, why is Agent 5 heat map incomplete? Was it taken out of action around time step 20?
- From Figure 5, it looks like the memory is not very useful to attend to after the first few couples of steps. Why is that?
- Did the authors try making the trained agents with different architectures play against each other (instead of against the built-in AI)?
limitations: The authors discussed the limitations and social impacts in the Appendix. Notably, how the learning process requires some exploration that could lead to unsafe situations for both the agents and humans, and also how the Entity-Bound Action Layer requires expert knowledge to manually configure it.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new Multiagent Reinforcement Learning (MARL) method using Agent Transformer Memory (ATM). ATM consists of 4 parts; memory, self, allies and entities. Especially, for self, it duplicates the token to use it for Entity-Bound Action (EBA) layer which affects the performance clearly. For memory, ATM uses working memory concept, and other memory types are also evaluated as an ablation study. ATM-based MARL agent outperforms other methods for StarCraft Multi-Agent Challenge (SMAC) and its architecture (EBA and working memory) shows better performance than alternatives (e.g., relational memory or without EBA).

=================================

What happening if they apply appending style memory to their model is interesting and they didn't show that, but except that, they handled lots of concerns through author-reviewer discussion phase. I sustain my score.
strengths_and_weaknesses: Strengths:
- It proposes a new memory module, ATM which can infer the interaction between the agent, allies, entities and memory.
- It evaluates for SMAC, ATM-based agent outperforms GRU-based one which is usually used.
- It reports the ablation studies for EBA and memory types and the number of memory slots.

Weaknesses:
- As shown in [1], the agent with Transformer requires more computations, even though it can benefit for sample efficiency [1,2] or interactions between entities like this paper. For example, in Figure 2 (a) and (b), ATM-based models outperform GRU-based models in steps, but when comparing in wall-clock time, ATM-based model could be much slower than recurrent module-based models.
- The memory following the working memory mechanism is interesting, but it cannot attend the past explicitly. Other transformer-based agents [2,3] get the advantages from attending the past directly, while it cannot do that through ATM architecture.

[1] Parisotto, Emilio, and Ruslan Salakhutdinov. "Efficient transformers in reinforcement learning using actor-learner distillation." arXiv preprint arXiv:2104.01655 (2021).

[2] Parisotto, Emilio, et al. "Stabilizing transformers for reinforcement learning." International conference on machine learning. PMLR, 2020.

[3] Chen, Chang, et al. "Transdreamer: Reinforcement learning with transformer world models." arXiv preprint arXiv:2202.09481 (2022).
questions: - Could you share the plots in Figure 2 and 6 by comparing in wall-clock time? I think ATM should be slower than GRU, but want to know how much slower. If ATM is not slower than GRU, then it is interesting also.
- Did you try to apply appending style memory or TrXL style memory? Those one can be computationally expensive, but useful from the perspective that it can support to attend the past explicitly.
limitations: I think that one of the limitations of ATM is computationally more expensive than recurrent module. I don't want to say "so we don't use ATM". I want to discuss the comparison in the aspect of computation or time to get the action from the agents. Even though ATM is more expensive, sometimes we must pick ATM to solve some tasks requiring to infer huge interactions.
review_title: null

Meta-review: metareview: All reviewers agree that this paper makes a good contribution in developing a novel transformer-based memory structure for MARL. The developed approach is evaluated through comprehensive and solid experiments. The authors have also clearly addressed the questions/concerns raised by the reviewers.
confidence: Certain
Decision: Accept