Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes the use of sequence models (SMs), specifically Transformers, to implement the Multi-Agent Advantage Decomposition theorem. This is done by introducing the Multi-Agent Transformer (MAT), an encoder-decoder model that sequentially predicts the action of an agent based the actions taken by previous agents in a random permutation. MAT is evaluated on four different benchmarks: SMAC, Multi-Agent MuJoCo benchmark, Bi-DexHands and Google Research Football benchmark. MAT achieves strong performance on SMAC and MuJoCo compared to the provided baselines. Further experiments show evidence that MAT can be good at few-shot learning.
strengths_and_weaknesses: Strengths
- I found the application of Transformers’ causal masked attention to the Multi-Agent Advantage Decomposition theorem novel.
- The performed experiments were instructive and provide good insight into the MAT’s performance. I particularly appreciated it’s ability to model both homogeneous and heterogenous agent tasks.
- I thought the paper was well organized and well-written.

Weaknesses:
- Where is the related work section? This is an important part of any scientific paper as it positions this work in the general area, (i.e., MARL). Only covering two contemporary methods is not sufficient in my opinion.
- I found a MAPPO result mismatch with paper; for example, for MMM2, the original paper reports 90.6 win rate versus 81.8 in the current manuscript. Did the authors implement their own version? If so, why not use the published version of the results?
- I found the explanation of the few-shot experiments confusing and unclear (section 5.2). For instance what exactly was done in the last meta-column (MAT-from scratch)? Also to help with the flow, I would suggest moving the paragraph starting at line 265 to the beginning of section 5.2. 

questions: - This work seems relevant and should probably be addressed and possibly compared to:  
[1] Hu, Siyi, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. "Updet: Universal multi-agent reinforcement learning via policy decoupling with transformers." arXiv preprint arXiv:2101.08001 (2021).

- What would happen to its performance if we replaced the transformer by other types of recurrent models (GRU, LSTM)? 

limitations: Limitations were not addressed in the manuscript, as reported by the authors.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies transformer networks in an actor-critic algorithm with a multi-agent setting. The authors first picks a random order of agents, motivated by the advantage decomposition theorem. Given this ordering of agents and their observations and actions, they formulate multi-agent RL as a sequence modeling problem. A transformer encoder is first applied to a sequence of observations to generate observation encodings and value of each agent. Conditioned on observation encodings, a transformer decoder is applied to a sequence of actions to generate action distributions of every agent. A new ordering of agents is resampled for every iteration to prevent agents from changing their policy. Different from HAPPO, the authors use a different objective where parameters of all agents are updated simultaneously rather than sequentially, which can potentially be faster with gradient accumulation and batching. When evaluated on a set of cooperative multi-agent environments, the proposed model, MAT, outperforms strong baselines. It is also shown to perform well with scarce data.
strengths_and_weaknesses: The main strength is improved performance and shared experience from all agents in a flexible framework like transformers. Main weaknesses are lack of related work and some of the claims.

## Strengths
1. The paper is overall well written and easy to follow.
2. The authors use a transformer architecture in a multi-agent setting and creates a connection to the multi-agent advantage decomposition theorem.
3. Experimental results show improvements.

## Weaknesses
1. The main contribution of the paper is the transformer architecture but related work lacks any prior work on using sequential models in a multi-agent setting. It is not easy to position the paper without knowing if the encoder-decoder structure, transformers, or advantage decomposition has been studied in the past. Especially, UPDeT*, which also uses a transformer architecture and a similar encoder-decoder structure, should be compared.
2. You mention that "the transformer architecture allows sequential policies to be trained in parallel" but it is not clear what this means. While you are using a different objective with all actions of all agents are available, you still need to run the decoder on $\hat{a}^{i_{1:m-1}}_t$ to generate the action distribution for agent $i_m$. Main difference from the autoregressive variant seems to be batching of input and gradient accumulation but not clear if this is what you meant by parallelism.
3. It is not clear if all of transformer components are useful in this setting. For example, since you are shuffling agents at every iteration, position encoding might not be very relevant and can be discarded. I think an ablation study is needed to understand the effect of different components.


\* UPDET: UNIVERSAL MULTI-AGENT REINFORCEMENT LEARNING VIA POLICY DECOUPLING WITH TRANSFORMERS. Siyi Hu, Fengda Zhu, Xiaojun Chang, Xiaodan Liang.
questions: I have several questions regarding my concerns above.

1. Can you compare MAT to other sequential models, especially ones that use transformer architecture? If there are comparable approaches, can you add baseline experiments?
2. Can you clarify the advantage of a transformer decoder w.r.t. parallel training?
3. Can you add ablation studies to understand different components in the transformer? Also, another ablation on just using the decoder where observations and actions for all agents $i_{1:m-1}$ are provided as input to the decoder to generate value and action distribution for agent $i_m$? This would be useful to understand if the added complexity of encoder-decoder architecture, which is inspired by the decomposition theorem where observations of all agents are given, is needed to get improved results.
limitations: The authors addressed the limitations and included a source code with their submission.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces a sequence model problem formulation of the cooperative MARL setup. Multi-Agent Transformer (MAT) is proposed to extend HAPPO by an encoder-decoder architecture which maps sequences of agent observations to sequences of agent actions. MAT is evaluated in SMAC, MA-Mujoco, and Dexterous Hands Manipulation to demonstrate superior performance over MAPPO and HAPPO, generalization, and few-shot learning capabilities.
strengths_and_weaknesses: - **Originality**: The paper presents an interesting and novel direction in cooperative multi-agent learning using state-of-the-art sequence-to-sequence learning techniques.
- **Quality**: The paper is mostly technically sound. The explanations and experiments are presented well and the results adequately support the claims. However, the problem formulation in Section 2.1 actually represents a *Markov game* rather than a Dec-POMDP (according to [1] and the HAPPO paper). Please note that unlike the (unknown) state, the joint observation is not Markovian in Dec-POMDPs. Therefore, the value functions defined in Eq. 1 are only valid for Markov games but not for Dec-POMDPs [2].
- **Clarity**: The paper is very well-written, clearly structured, and easy to follow.
- **Significance**: The experimental results indicate a clear improvement over MAPPO and HAPPO with respect to performance, generalization, and few-shot learning capabilities. Due to the parallelizability of transformers, MAT potentially offers better computational efficiency than HAPPO although a report on wall clock time would have been nice.

Minor comments:
- Text and labels in Fig. 4 are very small and hard to read when printed. Labels in Fig. 5 are fine but the axes ticks are too small (I suggest to reduce the number of ticks and make the text larger).
- The numbers in brackets in Table 2 and 3 are unreadable when printed.

[1] Littmann, "Markov Games as a Framework for Multi-Agent Reinforcement Learning", ICML 1994

[2] Oliehoek et al., "Optimal and Approximate Q-value Functions for Decentralized POMDPs", JAIR 2008
questions: According to the SMAC documentation, the action spaces vary depending on the number of enemies. How does MAT deal with that in the experiments?
limitations: Limitations and potential negative societal impact have not been addressed in this paper.
review_title: null

Meta-review: metareview: The reviewers are largely in consensus that this is a well-executed application of a highly relevant technology (transformers) to an interesting and challenging problem, with pleasing results. I believe the community will be quite interested to see this development presented at NeurIPS.

Note: I agree with reviewer MEiB that important prior work also using transformers needs careful comparison, but the author's rebuttals gave confidence that will be done appropriately. Please do be very dilligent on this point in the final version.


confidence: Certain
Decision: Accept