Multi-Agent Reinforcement Learning is a Sequence Modeling Problem

Abstract: Large sequence models (SM) such as GPT series and BERT have displayed outstanding performance and generalization capabilities in natural language process, vision and recently reinforcement learning. A natural follow-up question is how to abstract multi-agent decision making also as an sequence modeling problem and benefit from the prosperous development of the SMs. In this paper, we introduce a novel architecture named Multi-Agent Transformer (MAT) that effectively casts cooperative multi-agent reinforcement learning (MARL) into SM problems wherein the objective is to map agents'  observation sequences to agents' optimal action sequences. Our goal is to build the bridge between MARL and SMs so that the modeling power of modern sequence models can be unleashed for MARL. Central to our MAT is an encoder-decoder architecture which leverages the multi-agent advantage decomposition theorem to transform the joint policy search problem into a sequential decision making process; this renders only linear time complexity for multi-agent problems and, most importantly, endows MAT with monotonic performance improvement guarantee. Unlike prior arts such as Decision Transformer fit only pre-collected offline data, MAT is trained by online trial and error from the environment in an on-policy fashion. To validate MAT, we conduct extensive experiments on StarCraftII, Multi-Agent MuJoCo, Dexterous Hands Manipulation, and Google Research Football benchmarks. Results demonstrate that MAT achieves superior performance and data efficiency compared to strong baselines including MAPPO and HAPPO. Furthermore, we demonstrate that MAT is an excellent few-short learner on unseen tasks regardless of changes in the number of agents.
See our project page at https://sites.google.com/view/multi-agent-transformer.

Reviews and author responses:

 We would like to say thank you for your deep engagement with our work, especially the direction you have suggested helps us make the paper more understandable. Thanks a lot.

Indeed, both UPDeT and MAT are trying to model the interrelationships between agents using Transformers. However, while UPDeT implements it by modeling the relationship between *matched observation-entities* within each agent's local observations and action spaces, MAT implements it by modeling the relationship between different agents' observations and actions directly and enjoys the advantages in the following three aspects:
1. MAT does not require observations to be *divisible* and thus is applicable for more scenarios. UPDeT needs to divide each observation into a set of *observation-entities* (each *observation-entity* contains information about an ally or enemy unit) before modeling, requiring each observation contains information of all units as well as is well-structured, i.e. *divisible*, and limiting its application. For example, considering scenarios where each observation does not contain information about all units, like Multi-agent MuJoCo whose observations involve only a current controlled joint and possibly some neighbor joints (sometimes they even contain only a single joint so that is hard to be divided), UPDeT might not be able to model the interrelationships between all agents without extra works to reorganize observations. By contrast, MAT models different agents' observations directly, and thus no assumptions are required about the content and structure of observations. 
2. MAT supports both discrete and continuous action spaces, while UPDeT supports only discrete action spaces that could be divided into a set of *action-groups*.
3. MAT provides an explicit guarantee for monotonic improvement of joint policies since it considers the relationship between actions of different agents with the auto-regressive decoder, while UPDeT still risks conflicting joint actions because it considers only the relationship between *agent-groups* of a single agent. 

 Thank you for the clarification. I would be increasing my score.

It would help if you can explain the difference between UPDeT and yours better. For example, UPDeT also uses transformers to model interrelationships between agents.

 I am satisfied with the rebuttal provided by the authors and the updates they made to the manuscript. I think the proposed approach has demonstrated novelty and its performance is strong compared to other recent methods (QMIX, HAPPO and MAPPO). 

I am raising my score by 1 point.

 ### We thank Reviewer rnUZ for his/her constructive comments that will surely turn our paper into a better shape.
> **Q1** The problem formulation is that of Markov games [1] rather than the stated Dec-POMDPs. The HAPPO paper uses the *Markov game* formulation as well, thus the deployed theoretical results are sound in this setting only. This should be corrected. 

**A1** The reviewr is right in pointing this out. We apologize for this error. We had changed *decentralized Partially Observable Markov Decision Processes (Dec-POMDPs)* to *Markov Games (MGs)* in the revision and keep the stated formulation.

> **Q2** MAT achieves computational-efficiency advantage w.r.t. HAPPO. It would be nice to see the exact performance comparison w.r.t. wall clock time.

**A2** Thanks so much for the suggestion, the wall clock time comparison between MAT, MAPPO and HAPPO w.r.t 10M environment steps is demonstrated under this **anonymized** [link](https://anonymous.4open.science/r/MAT_Rebuttal-0ABA/walltime.pdf). Also, we had added this comparison in our revised Appendix to better demonstrate the advantages.

> **Q3** Figures 4 & 5, as well Tables 2 & 3 are illegible due to size.

**A3** This is a valuable feedback. With the suggestion in mind, we will readjust the style of all figures and tables to make them more readable in the final version.

**Reference:**
[1] Littmann, "Markov Games as a Framework for Multi-Agent Reinforcement Learning", ICML 1994

 ### We thank Reviewer MEiB for his/her constructive comments that will surely turn our paper into a better shape.
> **Q1** The work lacks a *related works* section which makes it difficult to position the used techniques (advantage decomposition, transformers) in the MARL literature. The UPDeT algorithm, which also uses transformers, is not compared against.

**A1** We appreciate pointing out this flaw. The Policy Decoupling in UPDeT is relevant to our work. UPDeT focuses on observations of each single agent, handling various observation sizes by decoupling the observations into a set of observation-entities according to the physical meanings, matching them with different action-groups, and modeling the relationship between the matched observation-entities with self-attention mechanism for better representation learning [1]. In contrast, MAT focuses more on observations and actions from different agents, modeling the interrelationship between agents with Transformer architectures, promoting the monotonic improvement of joint policies without the requirement for divisible observations.

Besides, Section 1 was intended to provide the discussion about the related work and the motivation and blend them together, whereas Section 2 continued the literature further mathematically by covering the related work (such as the trust-region methods, e.g., MAPPO and HAPPO) that is closely related to the advantage decomposition theorem. That said, thanks for the reviewer’s comment, in the updated version, we have better positioned our work in the general area of MARL by:

1. adding the discussion about two more classic MARL methods, i.e. QMIX (value decomposition) and MADDPG (deterministic policy gradient), and a Transformer-based MARL method, i.e. UPDeT, in Sections 1 and 2.
2. Additionally, running more baseline experiments for MADDPG and UPDeT, where currently completed results are shown in this **anonymous** [link](https://anonymous.4open.science/r/MAT_Rebuttal-0ABA/baseline.pdf) (QMIX’s performance is covered already in Tab.1).


> **Q2** You mention that "the transformer architecture allows sequential policies to be trained in parallel" but it is not clear what this means. If you refer to parallelism in the agents' updates, then it comes as counterintuitive. After all, you need to run the decoder on $\hat{a}^{i_{1:m-1}}_t$ to generate the action distribution for agent $i_m$.

**A2** We apologize for the confusion. What we want to say here is that "In order to guarantee the monotonic improvement of joint policies, HAPPO has to update each policy one-by-one during training, by leveraging previous update results of $\pi^{i_1},...,\pi^{i_{m-1}}$ to improve $\pi^{i_m}$, which becomes critical at large size of agents. By contrast, the attention mechanism of Transformer architecture (especially the decoder) allows for batching the ground truth actions $a_t^{i_0},...,a_t^{i_{n-1}}$ in the buffer to predict $a_t^{i_1},...,a_t^{i_{n}}$ and update policies simultaneously, which significantly improves the training speed and makes it feasible for large size of agents". We had added this clarification to the last paragraph of Section 3 in our revision.


> **Q3** Ablation studies could be helpful to understand the effect of different components. For example, since agents are shuffled at every iteration, the position encoding might not be very relevant and can be discarded. Another example is to use the decoder where observations and actions for all agents $i_{1:m-1}$ are provided as input to the decoder to generate value and action distribution for agent $i_m$.

**A3** 
We are very grateful for the insightful suggestion and have conducted more ablation experiments w.r.t different encoding approaches and model architectures, as described at the end of revised appendices. 

Exactly as you mentioned, we found the position encoding in the vanilla Transformer [1] is not very relevant for MARL tasks in early experiments, and it has been already discarded in our implementation. Instead, we bind each observation with corresponding one-hot agent id, which also be applied to baseline methods for fairness. The ablation results under this **anonymous** [link](https://anonymous.4open.science/r/MAT_Rebuttal-0ABA/encoding.pdf) emphasized the importance of agent id encoding on heterogeneous settings.

With the suggestion in mind, we also compared the implementation with different model architectures, i.e. encoder-decoder, encoder only, decoder only, and other sequence models (GRU). Corresponding ablation results under this **anonymous** [link](https://anonymous.4open.science/r/MAT_Rebuttal-0ABA/structure.pdf) confirmed the advantage of Transformer and the necessity of encoder-decoder architectures. 

**Reference:**

[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, (2017)

 >**Q4** This work seems relevant and should probably be addressed and possibly compared to UPDeT [1].

**A4** We agree that the Policy Decoupling in UPDeT is relevant to our work. UPDeT focuses on observations of each single agent, handling various observation sizes by decoupling the observations into a set of observation-entities according to the physical meanings, matching them with different action-groups, and modeling the relationship between the matched observation-entities with self-attention mechanism for better representation learning [1]. In contrast, MAT focuses more on observations and actions from different agents, modeling the interrelationship between agents with Transformer architectures, promoting the monotonic improvement of joint policies without the requirement for divisible observations. We have introduced this work in Section 2 of the revision and will surely compare with it in the final version (currently completed results are show in this **anonymous** [link](https://anonymous.4open.science/r/MAT_Rebuttal-0ABA/baseline.pdf)).

> **Q5** What would happen to MAT's performance if we replaced the transformer by another language model (e.g., GRU, LSTM)?

**A5** Theoretically, the multi-agent sequential decision paradigm could be implemented with different type of sequence models including RNNs like GRU and LSTM. However, compared with Transformer, other RNN-based implementations pass the information between sequential tokens with recurrence and will suffer from critical information loss that impedes building long-distance dependencies [2]. In contrast, the attention mechanisms allow the Transformer to model inter-token dependencies regardless of their distance in the input or output sequences. 

Technically, we conducted an ablation experiment that replaces the encoder and decoder in Figure 2 with GRU, whose results verified the advantage of Transformer architecture (see the **anonymized** [link](https://anonymous.4open.science/r/MAT_Rebuttal-0ABA/structure.pdf) or the *ablation studies* section at the end of our revised appendices). 
We found this question very insightful and we thank the reviewer for it. 

**Reference:**

[1] Hu, Siyi, et al. "Updet: Universal multi-agent reinforcement learning via policy decoupling with transformers." arXiv preprint arXiv:2101.08001 (2021)

[2] Ashish Vaswani, et al. Attention is all you need. In Advances in neural information processing systems, (2017)

[3] Yu, Chao, et al. "The surprising effectiveness of ppo in cooperative, multi-agent games." arXiv preprint arXiv:2103.01955 (2021).

 ### We thank Reviewer YpeW for his/her constructive comments that will surely turn our paper into a better shape.
> **Q1** Why have the authors not include a *related works* section and only reviewed two trust-region mehtods? *Related works* serves an important role of positioning the paper in the MARL literature.

**A1** Section 1 was intended to provide the discussion about the related work and the motivation and blend them together, whereas Section 2 continued the literature further mathematically by covering the related work (such as the trust-region methods, e.g., MAPPO and HAPPO) that is closely related to the advantage decomposition theorem.  That said, thanks for the reviewer’s comment, in the updated version, we have better positioned our work in the general area of MARL by:

1. adding the discussion about two more classic MARL methods, i.e. QMIX (value decomposition) and MADDPG (deterministic policy gradient), and a Transformer-based MARL method, i.e. UPDeT, in Section 1 and 2.
2. Additionally, running more baseline experiments for MADDPG and UPDeT, where currently completed results are shown in this ***anonymous*** [link](https://anonymous.4open.science/r/MAT_Rebuttal-0ABA/baseline.pdf) (QMIX’s performance is covered already in Tab.1).


> **Q2** There seems to be a mismatch between the MAPPO performance from the original paper [3] and the one reported in this work. For example, on MMM2, MAPPO is known to achieve 90.6 win rate, while here authors report 81.8. Did the authors implement their own MAPPO version? If so, why?

**A2** We used the original MAPPO implementation from the MAPPO codebase. Also, we kept its hyperparameters unchanged. What seems to be the reason of the mismatch is that we ran MAPPO with seeds generated by us at random (MAPPO paper has not listed the details of their seeds). Therefore, random seeds are different between our paper and MAPPO paper. The striking difference comes from the fact that MAPPO performed particularly poorly with one of the seeds, which is reflected by the higher variance in Tab.1 (10.1 versus 2.8). For better demonstration, we here provide the plot after removing the outlier under the following ***anonymised*** [link](https://anonymous.4open.science/r/MAT_Rebuttal-0ABA/outlier.pdf), where MAPPO achieve 90.2 win rate. Moreover, MAT achieves 93.8 win rate on MMM2, which is still better than the known win rate of MAPPO.


> **Q3** Some parts of explanation of the few-shot experiments (Section 5.2) are confusing. For instance what exactly was done in the last meta-column (MAT-from scratch)? Also to help with the flow, I would suggest moving the paragraph starting at line 265 to the beginning of section 5.2.

**A3** We apologize for the confusion. The difference between MAT and "MAT-from scratch" in section 5.2 is that the MAT model is pre-trained on different tasks while the "MAT-from scratch" model is randomly initialized. Therefore, the "MAT-from scratch" is a control group experiment to demonstrate the effectiveness of pre-training process w.r.t MAT. Also, with reviewer's suggestion in mind, we have moved the paragraph mentioned to section 5.2 as well as clarified the "MAT-from scratch" in our revision since it does make our paper easier to follow.


 This work proposes the use of sequence models (SMs), specifically Transformers, to implement the Multi-Agent Advantage Decomposition theorem. This is done by introducing the Multi-Agent Transformer (MAT), an encoder-decoder model that sequentially predicts the action of an agent based the actions taken by previous agents in a random permutation. MAT is evaluated on four different benchmarks: SMAC, Multi-Agent MuJoCo benchmark, Bi-DexHands and Google Research Football benchmark. MAT achieves strong performance on SMAC and MuJoCo compared to the provided baselines. Further experiments show evidence that MAT can be good at few-shot learning. Strengths
- I found the application of Transformers’ causal masked attention to the Multi-Agent Advantage Decomposition theorem novel.
- The performed experiments were instructive and provide good insight into the MAT’s performance. I particularly appreciated it’s ability to model both homogeneous and heterogenous agent tasks.
- I thought the paper was well organized and well-written.

Weaknesses:
- Where is the related work section? This is an important part of any scientific paper as it positions this work in the general area, (i.e., MARL). Only covering two contemporary methods is not sufficient in my opinion.
- I found a MAPPO result mismatch with paper; for example, for MMM2, the original paper reports 90.6 win rate versus 81.8 in the current manuscript. Did the authors implement their own version? If so, why not use the published version of the results?
- I found the explanation of the few-shot experiments confusing and unclear (section 5.2). For instance what exactly was done in the last meta-column (MAT-from scratch)? Also to help with the flow, I would suggest moving the paragraph starting at line 265 to the beginning of section 5.2. 
 - This work seems relevant and should probably be addressed and possibly compared to:  
[1] Hu, Siyi, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. "Updet: Universal multi-agent reinforcement learning via policy decoupling with transformers." arXiv preprint arXiv:2101.08001 (2021).

- What would happen to its performance if we replaced the transformer by other types of recurrent models (GRU, LSTM)? 
 Limitations were not addressed in the manuscript, as reported by the authors.

 This paper studies transformer networks in an actor-critic algorithm with a multi-agent setting. The authors first picks a random order of agents, motivated by the advantage decomposition theorem. Given this ordering of agents and their observations and actions, they formulate multi-agent RL as a sequence modeling problem. A transformer encoder is first applied to a sequence of observations to generate observation encodings and value of each agent. Conditioned on observation encodings, a transformer decoder is applied to a sequence of actions to generate action distributions of every agent. A new ordering of agents is resampled for every iteration to prevent agents from changing their policy. Different from HAPPO, the authors use a different objective where parameters of all agents are updated simultaneously rather than sequentially, which can potentially be faster with gradient accumulation and batching. When evaluated on a set of cooperative multi-agent environments, the proposed model, MAT, outperforms strong baselines. It is also shown to perform well with scarce data. The main strength is improved performance and shared experience from all agents in a flexible framework like transformers. Main weaknesses are lack of related work and some of the claims.

## Strengths
1. The paper is overall well written and easy to follow.
2. The authors use a transformer architecture in a multi-agent setting and creates a connection to the multi-agent advantage decomposition theorem.
3. Experimental results show improvements.

## Weaknesses
1. The main contribution of the paper is the transformer architecture but related work lacks any prior work on using sequential models in a multi-agent setting. It is not easy to position the paper without knowing if the encoder-decoder structure, transformers, or advantage decomposition has been studied in the past. Especially, UPDeT*, which also uses a transformer architecture and a similar encoder-decoder structure, should be compared.
2. You mention that "the transformer architecture allows sequential policies to be trained in parallel" but it is not clear what this means. While you are using a different objective with all actions of all agents are available, you still need to run the decoder on $\hat{a}^{i_{1:m-1}}_t$ to generate the action distribution for agent $i_m$. Main difference from the autoregressive variant seems to be batching of input and gradient accumulation but not clear if this is what you meant by parallelism.
3. It is not clear if all of transformer components are useful in this setting. For example, since you are shuffling agents at every iteration, position encoding might not be very relevant and can be discarded. I think an ablation study is needed to understand the effect of different components.


\* UPDET: UNIVERSAL MULTI-AGENT REINFORCEMENT LEARNING VIA POLICY DECOUPLING WITH TRANSFORMERS. Siyi Hu, Fengda Zhu, Xiaojun Chang, Xiaodan Liang. I have several questions regarding my concerns above.

1. Can you compare MAT to other sequential models, especially ones that use transformer architecture? If there are comparable approaches, can you add baseline experiments?
2. Can you clarify the advantage of a transformer decoder w.r.t. parallel training?
3. Can you add ablation studies to understand different components in the transformer? Also, another ablation on just using the decoder where observations and actions for all agents $i_{1:m-1}$ are provided as input to the decoder to generate value and action distribution for agent $i_m$? This would be useful to understand if the added complexity of encoder-decoder architecture, which is inspired by the decomposition theorem where observations of all agents are given, is needed to get improved results. The authors addressed the limitations and included a source code with their submission.

 The paper introduces a sequence model problem formulation of the cooperative MARL setup. Multi-Agent Transformer (MAT) is proposed to extend HAPPO by an encoder-decoder architecture which maps sequences of agent observations to sequences of agent actions. MAT is evaluated in SMAC, MA-Mujoco, and Dexterous Hands Manipulation to demonstrate superior performance over MAPPO and HAPPO, generalization, and few-shot learning capabilities. - **Originality**: The paper presents an interesting and novel direction in cooperative multi-agent learning using state-of-the-art sequence-to-sequence learning techniques.
- **Quality**: The paper is mostly technically sound. The explanations and experiments are presented well and the results adequately support the claims. However, the problem formulation in Section 2.1 actually represents a *Markov game* rather than a Dec-POMDP (according to [1] and the HAPPO paper). Please note that unlike the (unknown) state, the joint observation is not Markovian in Dec-POMDPs. Therefore, the value functions defined in Eq. 1 are only valid for Markov games but not for Dec-POMDPs [2].
- **Clarity**: The paper is very well-written, clearly structured, and easy to follow.
- **Significance**: The experimental results indicate a clear improvement over MAPPO and HAPPO with respect to performance, generalization, and few-shot learning capabilities. Due to the parallelizability of transformers, MAT potentially offers better computational efficiency than HAPPO although a report on wall clock time would have been nice.

Minor comments:
- Text and labels in Fig. 4 are very small and hard to read when printed. Labels in Fig. 5 are fine but the axes ticks are too small (I suggest to reduce the number of ticks and make the text larger).
- The numbers in brackets in Table 2 and 3 are unreadable when printed.

[1] Littmann, "Markov Games as a Framework for Multi-Agent Reinforcement Learning", ICML 1994

[2] Oliehoek et al., "Optimal and Approximate Q-value Functions for Decentralized POMDPs", JAIR 2008 According to the SMAC documentation, the action spaces vary depending on the number of enemies. How does MAT deal with that in the experiments? Limitations and potential negative societal impact have not been addressed in this paper.

Meta-review: The reviewers are largely in consensus that this is a well-executed application of a highly relevant technology (transformers) to an interesting and challenging problem, with pleasing results. I believe the community will be quite interested to see this development presented at NeurIPS.

Note: I agree with reviewer MEiB that important prior work also using transformers needs careful comparison, but the author's rebuttals gave confidence that will be done appropriately. Please do be very dilligent on this point in the final version.



Decision: Accept