Multi-Agent Reinforcement Learning is a Sequence Modeling Problem
Authors: Muning Wen
Jakub Grudzien Kuba
Runji Lin
Weinan Zhang
Ying Wen
Jun Wang
Yaodong Yang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-Agent Reinforcement Learning
Sequence Modeling
Transformer

Abstract: Large sequence models (SM) such as GPT series and BERT have displayed outstanding performance and generalization capabilities in natural language process, vision and recently reinforcement learning. A natural follow-up question is how to abstract multi-agent decision making also as an sequence modeling problem and benefit from the prosperous development of the SMs. In this paper, we introduce a novel architecture named Multi-Agent Transformer (MAT) that effectively casts cooperative multi-agent reinforcement learning (MARL) into SM problems wherein the objective is to map agents'  observation sequences to agents' optimal action sequences. Our goal is to build the bridge between MARL and SMs so that the modeling power of modern sequence models can be unleashed for MARL. Central to our MAT is an encoder-decoder architecture which leverages the multi-agent advantage decomposition theorem to transform the joint policy search problem into a sequential decision making process; this renders only linear time complexity for multi-agent problems and, most importantly, endows MAT with monotonic performance improvement guarantee. Unlike prior arts such as Decision Transformer fit only pre-collected offline data, MAT is trained by online trial and error from the environment in an on-policy fashion. To validate MAT, we conduct extensive experiments on StarCraftII, Multi-Agent MuJoCo, Dexterous Hands Manipulation, and Google Research Football benchmarks. Results demonstrate that MAT achieves superior performance and data efficiency compared to strong baselines including MAPPO and HAPPO. Furthermore, we demonstrate that MAT is an excellent few-short learner on unseen tasks regardless of changes in the number of agents.
See our project page at https://sites.google.com/view/multi-agent-transformer.

Multi-Agent Reinforcement Learning is A Sequence Modeling Problem
Muning Wen Jakub Grudzien Kuba², RinjiLin',,
Weinan Zhang', Ying Wen' Jun Wang Yaodong Yang
Shanghai Jiao Tong University. 'Digital Brain Lab. Universityy of Oxford,
'Institute of Automation, Chinese Academy of Science, University College London, Beijing Institute for General AI, Institute for AI, Peking University
Abstract
Large sequence models (SM) such as GPT series and BERT have displayed out- standing performance and generalization capabilities in natural language process, vision and recently reinforcement learning. A natural follow-up question is how to abstract multi-agent decision making also an sequence modeling problem and benefit from the prosperous development of the SMs. In this paper, we introduce a novel architecture named Multi-Agent Transformer (MAT) that effectively casts co- operative multi-agent reinforcement learning (MARL) into SM problems wherein the objective is to map agents' observation sequences to agents' optimal action sequences. Our goal is to build the bridge between MARL and SMs so that the mod- eling power of modern sequence models can unleashed for MARL. Central to our MAT is an encoder-decodes architecture which leverages the multi-agent advantage decomposition theorem to transform the joint policy search problem into a sequen- tial decision making process; this renders only linear time complexity for multi- agent problems and, most importantly, endows MAT with monotonic performance improvement guarantee. Unlike prior arts such Decision Transformer fit only pre- collected offline data, MAT is trained by online trial and error from the environment in on- on-policy fashion. To validate MAT, conduct extensive experiments on StarCraftII, Multi-Agent MuJoCo, Dexterous Hands Manipulation, and Google Re- search Football benchmarks. Results demonstrate that MAT achieves superior per- formance and data efficiency compared to strong baselines including MAPPO and HAPPO. Furthermore, demonstrate that MAT is excellent few-short learner on unseen tasks regardless of changes in the number of agents. See our project page at hattps://sitee wew.iitssssogggee.eooeoooooooeee wwom/vie/nnee mul ti agent- transf transformer""
Introduction
Multi-agent reinforcement learning (MARL) [44, is challenging problem for its difficulty which arises not only from identifying each individual agent's policy improvement direction, but also from combining agents' policy updates jointly which should be beneficial for the whole team. Recently, such difficulty in multi-agent learning has been eased owing to the introduction of centralized training for decentralized execution (CTDE) [11. 45], which allows agents to access the global information and opponents' actions during the training phase. This framework enables successful developments of methods that directly inherit single-agent algorithms. For examples, COMA replaces the policy-gradiac (PG) estimate with multi-agent PG (MAPG) counterpart [11], MADDPG
(v1 Corresponding <yaodong saaoodoongyynggeee www.yauw..uuuuu.......m The source code could be accessed directly with this link https://github. com/PKU MARL/Multi Agent- Transf ormer.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

extends deterministic policy gradient into multi-agent settings with a centralized critic [20, 34], QMIX leverages deep Qnetworks for decentralized agents and introduces centralized mixing network for Q-value decomposition [29, 36, 26]. MAPPO endowing all agents with the same set of parameters and then training by trust-region methods [46]. PR2 [42] and GR2 [43] methods conduct recursive reasoning under the CTDE framework. These methods, however, cannot cover the whole complexity of multi-agent interactions; in fact, some of them are shown to fail in the simplest cooperative task [15]. To resolve this issue, the multi-agent advantage decomposition theorem was proposed [15, Theorem IJ which captures how different agents contribute to the return and provides an intuition behind the emergence of cooperation through sequential decision making process scheme. Based on it, HATRPO and HAPPO algorithms [15, 17, 16] were derived which, thanks to the decomposition theorem and sequential update scheme, established new tfate-of-the-art methods for MARI.. However, their limitation is that the agents' policies are unaware of the purpose to develop cooperation and still rely on carefully handcrafted maximization objective. Ideally, team of agents should be aware of the jointness of their training by design, thereby following a holistic and effective paradigm ideal solution that is yet be proposed.
In recent years, sequence models (SM) have made a substantial progress in natural language pro- cessing (NLP) [27]. For example, GPT series [3] and BERT models [9]. built on autoregressive SMs, have demonstrated remarkable performance on a wide range of downstream tasks and achieved strong performance on few-shot generalization tasks. Although SM are mostly used in language tasks due to its natural fitting with the sequential property of languages, the sequential approaches not confined to NLP only, but is instead widely applicable general foundation model [2]. For example, in computer vision (CV), one can split an image into sub-images and align them in a sequence if they were tokens in NLP tasks [9, 10, 12). Although the idea of solving CV tasks by SM is straightforward, it serves as the foundation to some of the best- best-performing CV algorithms [38, 41, 39]. Furthermore, sequential methods are starting to spawn powerful multi-modal visual language models such as Flamingo [1], DALL-E [28], and GATO [30] in the recent past.
Coming with effective and expressive network architectures such as Transformer [40], sequence modeling techniques have also attracted tremendous attention from the RL community, which results in series of successful offline RL developments based on the Transformer architecture [5, 14. 30, 23). These methods show great potentials in tackling some of the most fundamental RL training problems, such long-horizon credit assignment and reward sparsity [37, 24, 25]. For example, by training autoregressive models on pre-collected offline data in purely supervised way, Decision Transformer [5] bypasses the need for computing cumulative rewards through dynamic programming, but rather generates future actions conditioning on the desired returns, past states and actions. Despite their remarkable successes, none of these methods have been designed to model the most difficult (also unique to MARL) aspect of multi-agent systems-the agents' interactions. In fact, if we were to simply endow all agents with Transformer policy and train them independently, their joint performance still could not be guaranteed to improve [15, Proposition 1]. Therefore, while a myriad of powerful SMs are available, MARL- an area that would greatly benefit from SM- has not truly taken advantage of their performance benefit. The key research question to ask is then
How can we model MARL problems by sequence models? ?
In this paper, we take several steps to provide an affirmative answer to the above research question. Our goal is to enhance MARL studies with powerful sequential modeling techniques. To fulfill that, we start by proposing a novel MARL training paradigm which establishes the connection between cooperative MARL problems and sequence modeling problems. Central to the new paradigm are the multi-agent advantage decomposition theorem and sequential update scheme, which effectively transform multi-agent joint policy optimization into sequential policy search process. As nat- ural outcome of our findings, we introduce Multi-Agent Transformer (MAT), an encoder-decoder architecture that implements generic MARL solutions through SM. Unlike Decision Transformer [5], MAT is trained online based on trials and errors in an on- on-policy fashion; therefore, it does not require collecting demonstrations upfront. Importantly, the implementation of the multi-agent advantage decomposition theorem ensures MAT to enjoy monotonic performance improvement guarantee during training. MAT establishes a new state-of oftte--f-th-art baseline model for cooperative MARL tasks. We justify such claim by evaluating MAT on the benchmarks of StarCraftlI, Multi-Agen Agent MuJoCo, Dexterous Hands Manipulation, and Google Research Football; results show that MAT achieves superior performance over strong baselines, such as MAPPO [46], HAPPO [15], QMIX [29] and UPDeT [13]. Finally, we show that MAT possesses great potentials in task generalizations, which holds regardless of the agent number in new tasks.

---

2
Preliminaries
In this section, we first introduce the cooperative MARL problem formulation and the multi-agent advantage decomposition theorem, which serves as the cornerstone of our work. We then review existing MARL methods that relate to MAT, and finally familiarize the reader with the Transformer.
2.1 Problem Formulation
Cooperative MARL problems are often modeled by Markov games A, [19]. .N {I....,n} is the set of agents, O I O' is the product of local observation spaces of the agents, namely the joint observation space, A II- A' is the product of the agents' action spaces, namely the joint action space, R O x A 1- Rmax] is the joint reward function, P:O x A O - I. is the transition probability function, and 7 [0, is the discount factor. At time step i E N. an agent i E N observes an observation of Oi (2) (o (o' ..... o²) is "joint" observation) and takes an action ał according to its policy n². which is the hth component of the agents'
joint policy At each time step, all agents take actions simultaneously based their observation with no sequential dependencies. The transition kernel P and the joint policy induce the (improper) marginal observation distribution Px è E Pr((o, = o|T). At the end of each time step, the whole team receives a joint reward R(o,, a,) and observe Os+11 whose probability distribution is p( a.). Following this process infinitely long, the agents earn a discounted cumulative return of R
E R((o,, a.).
2.2
Multi-Agent Advantage Decomposition Theorem
The agents evaluate the value of actions and observations with Q~ a) and (o), defined as
Qx(o, a) e Panss -
R
Oo = ag = =a.
(1)
V (o) A Eno 01:00 ~ Pas [RT On = o
The jointness of the objective causes difficulties associated with the credit assignment problem- having received a shared reward, individual agents are unable to deduce their own contribution to the team's s success or failure [4]. Indeed, applying traditional RL methods which simply employ the above value functions leads to obstacles in training, such as the growing variance of multi-agent policy gradient (MAPG) estimates [17]. Hence, to tackle these, notions of local value functions [21] and counterfactual baselines [1 have been developed. In this paper, we work with the most general notions of this kind-the multi-agent observation-vale
functions [15]. That is, for arbitrary disjoint, J.). for m. h S n, we define the
ordered subsets of agents 1mmm = (i1..... 3m and j1:h =
multi-agent observation-vall function by
Qx a
O O, a km
a f1mm
which recovers the original state-action value function in Equation (1) when m = n, and the original iiservation-valus function when m 0 (i.e,, when the set 11:m is empty). Based on Equation (2), we can further measure the contribution of a chosen subset of agents to the joint return and deñin the multi-agent advantage function by
A (o, Siih alim) Quiaiiti (o, aiis a afi")) > Quih (0, araa).
(3)
The above quantity describes how much better/worse than average the joint action a will be if agents dism take the joint action a' aimmm once fi:h have taken adi.. Again, when = O, the advantage compares the value of am to the baseline value function of the whole team. This calue-funntional representation of agents' actions enables studying interactions between them, as well to decompose the joint value function signal, thus helping alleviate the severity of the credit assignment problem [29, 35, 22]. The insights of Equation (3) accomplished by means of the following theorem.
For notation convenience, we omit defining agents' observation functions that take the global state as the input and outputs local observation for each agent, but rather define agents' local observations directly.
3

---

Theorem 1 (Munti-Agent Advantage Decomposition [17]). Let 11:m be a permutation of agents. Then, for any joint observation holds with no further ussumption needed,
O and joint action a a E A, the following equation always
A (o, (o.n"
Σ m=1
A (o, a ammm)
Importantly, this theorem provides an intuition guiding the choice of incrementally improving actions. Suppose that agent 1 picks an action at with positive advantage. A (o, a²1) > 0. Then, imagine that for all j = ...... n. agent iz knows the joint action a of its predecessors. In this case, it can choose an action at for which the advantage A (o, a²"3-1,a") ) is positive. Altogether, the theorem assures that the joint action a f has positive ad advantage. Furthermore, notice that the joint action has been chosen in n steps, each which searched an individual agent's action space. Hence, the complexity of this search is additive, Di=1 (A4, in the sizes of the action spaces. If we were to perform the search directly in the joint action space, we would browse set of multiplicative size, [A| I i |A'| Later, we will build upon this insight to design a SM that optimizes joint policies efficiently, agent by agent, without the necessity considering the joint action space at once.
2.3 Existing Methods in MARL
We now briefly summarize two sfate-of-the-art MARL algorithms. Both of them build upon Proximal Policy Optimization (PPO) [33] RL method famous for its simplicity and its performance stability,
MАPPO [46] is the first, and the most direct, approach for applying PPO in MARL. It equips all agents with one shared set of parameters and use agents' aggregated trajectories for the shared policy's update; iteration k 1, it optimizes the policy parameter 0k+1 by maximizing the clip objective of
Toe o) No, (a)lo) ATE, a).
ma(aalo) To, (a'lo)
+c) ATH, (o. (o..a))
EouPTor To
min
where the clip operator clips the input value (if necessary) so that it stays within the interval [ 11-6, [+e]]. However, enforcing parameter sharing is equivalent to putting a constraint the joint policy space, which lead to an exponentially- worse sub- sub-optimal outcome [15]. This motivates a more principled development of heterogencous--gee agent trust-region methods, e.g.. HАPPО.
03, Vi.j E N on
HАРPO (15) is currently one of the SOTA algorithm that fully leverages Theorem (1) to implement multi-agent trust-region learning with monotonic improvement guarantee. During an update, the agents choose a permutation i1.m at random, and then following the order in the permutation, every Tm that maximizes the objective of
agent im picks π Te
min (r(*²m " (o, ftom (lmm))cli((((""")
A (o, 1i
ommPmaaia f1im wat Faom
where r(Tr"m) )= am (aim /o)/ (a aff" lo). Note that the expectation is taken over the newly-updated
previous agents' policies, i.e, Tnew -1m-", this reflects intuition that, under Theorem (1), the agent im reacts to its preceding agents 11:--1- However, one drawback of HAPPO is that agent's policies has to follow the sequential update scheme in the permutation, thus it cannot be run in parallel.
2.4 The Transformer Model
Transformer [40] was originally designed for machine translation tasks (c.g., input English, output French). maintains an encoder-decoder structure, where the encoder maps an input sequence of tokens to latent representations and then the decoder generates a sequence of desired outputs in an auto-regressive manner wherein at each step of inference, the Transformer takes all previously generated tokens as the input. One of the most essential component in Transformer is the scaled dot- dot-product attention, which captures the interrelationship of input sequences. The attention function
QK"" VV, where the Q K, V corresponds to the vector of queries, keys and values, which can be learned during training, and the dy represent the dimension of Q and K. Self-attentions refer to cases when Q K, V share the same set of parameters.
is written Attention Q, K. V) softmax
VTz

---

00
}
{
} } {
Figure 1: Conventional multi-agent learning paradigm (left) wherein all agents take actions simultane- ously vs. the multi-agent sequential decision paradigm (right) where agents take actions by following sequential order, each agent accounts for decisions from preceding agents red arrows suggest.
Inspired by the attention mechanism, UPDeT [13] handles various observation sizes by decoupling each agent's observations into sequence of observation entitnes, matching them with different action- groups, and modeling the relationship between the matched iuservation-entiie with Transformer- based function for better representation learning in MARI. problems. Apart from this, based on the sequential property described the Theorem (1) and the principle behind HAPPO [15], is intuitive implementation for multi-agent trust-region learning. By
to think about another Transformer-bant treating a team of agents as a sequence, the Transformer architecture allows us to model teams of agents with variable numbers and types, while avoiding drawbacks of MAPPO/HAPPO. We will describe in more details how a cooperative MARL problem can solved by a sequence model.
3 The Surprising Connection Between MARL and Sequence Models
To establish the connection between MARL and sequence models, Theorem (1) provides new angle of understanding the MARL problem from a SM perspective. If each agent knows its predecessors' actions with an arbitrary decision order, the sum of agents' local advantages A (o. a a1im-1,, a.m) will be exactly equal to the joint advantages A (o, a aft:) This orderly decision setting across agents simplifies the update of their joint policy, where maximizing each agent's own local advantage is equivalent to maximizing the joint advantage. As such, agents do not need to worry about interference from other agents anymore during the policy update; the local advantage functions have already captured the relationship between agents. This property revealed by Theorem (1) inspires us to propose multi aueti- sequential decision paradigm for MARL problems as show in Figure (1), where we assign agents with an arbitrary decision order (one permutation for each iteration); each agent can access its predecessors' behaviors, based which then takes the optimal decision. This sequential paradigm motivates us to leverage a sequential model, e.g., Transformer, to explicitly capture the sequential relationship between agents described in Theorem (1).
Underpinned by Theorem (1), sequence modeling reduces the complexity growth of MARL problems with the number of agents from multiplicative to additive, thus rendering linear complexity. With the help of the Transformer architecture, we can model policies of heterogeneous agents with an unified network but treat each agent discriminatively with different position, and thus ensuring high sample efficiency while avoiding the exponentially-worss outcome that MAPPO is facing. Besides, in order guarantee the monotonic improvement of joint policies, HAPPO has to update each policy one-by-one during training, by leveraging previous update results of T T'm to improve ,t", which becomes critical in computational efficiency large size of agents. By contrast, the attention mechanism of Transformer architectures allows for batching the ground truth actions at ...., a, fe-3 in buffer to predict af' ...., a;" and update policies simultaneously, which significantly improves the training speed and makes it feasible for large size of agents. Furthermore, in cases that the number and the type of agents are different, SM can incorporates them into an unified solution through its capability on modeling sequences with flexible sequence length, rather than treat different agent
5

---

v(a")
0
Emb
MLP
6
0
V(Oa)
Attention
V
Multi-Agent Observation Encoding
Nx
tm
fm-
t
Emb.
Masked Attention
K
Masked Attention
MLP
MLP
fmi
Auto-regressive Action Decoding
Nx
The encoder-decoder architecture MAT. At each time step, the encoder takes in sequence agents' observations and encodes them into a sequence of latent representations, which is then passed into the decoder. The decoder generate each agent's 's optimal action in a sequential and auto-regressive manner. The masked attention blocks ensures agents only access its preceding agents' actions during training. We list the full pscudocode of MAT in Appendix A and a video that shows dynamic data flow of MAT in https://sitens google. com/v htt//vee//multiin agent- transrrranrttn
Figure 2:
numbers as different tasks. To realize the above idea, we introduce practical architecture named Multi-Agent Transformer in the next section.
4 The Multi-Agent Transformer
To implement the sequence modeling paradigm for MARL, our solution is Multi-Agent Transformer (MAT). The idea of applying the Transformer architecture comes from the fact that the mapping between the input of agents' observation sequence (o' o²") and the output of agents' action are sequence modeling tasks similar to machine translations. As eluded by Theorem (1), the action atm depends on all previous agents' decisions a "1-m11 Hence, our MAT in Figure (2) consists of encoder, which learns representations of the joint observations, and decoder which outputs actions for each individual agent in an auto-regressive manner.
sequence (at , ..")
The encoder, whose parameters denote by takes sequence of observations (o'...... o'n) in arbitrary order and passes them through several computational blocks. Each such block consists of self-attention mechanism and multi-layer perceptron (MLP), well residual connections to prevent gradient vanishing and network degradation with the increase of depth. We denote the output encoding of the observations (o ..... 8² ), which encodes not only the information of agents (ii...., )n) but also the high-level interrelationships that represent agents' interactions. In order to learn expressive representations, the training phase, make the encoder approximate the value functions, whose objective is to minimize the empirical Bellman error by
Eiccuter (0)
R(o, - -VV3((
volo?-))
(4)
T'
where T is the target network's 's parameter, which is non-differentiable and updated every few epochs.
The decoder, whose parameters we denote by t, passes the embedded joint action afom 1,m {1....n) n (where ae is an arbitrary symbol indicating the start of decoding) to a sequence of decoding blocks. Crucially, every decoding block comes with a masked self-artention mechanism, where the masking makes sure that, for every ij. attention is computed only between the 7th and the 7th action heads wherein j that the sequential update scheme can be maintained. This is then followed

---

HalfCheetah
(b) CachOvvr2Undeeaam
DeorOpenloward
DeorClossOtward
Figure 3: Demonstrations of the Bi- Dends and the HalfCheetah environments.
NST HAPP
MATEN
MATIN HAWD
HARV
HARV
Figure 4: Performance comparisons on the Multi- Agent MuJoCo and the Bi-DexHands benchmarks.
by a second masked attention function, which computes the attention between the action heads and observation representations. Finally, the block finishes with an MLP and skipping connections. The output to the last decoder block sequence of representations of the joint actions, fa"0:t-1j.0 -'=1" This is fed to MLP that outputs the probability distribution of im 's action, namely, the policy To train the decoder, we minimize the following clipping PPO objective of
fm (afm fiom a 21:m-1). 1).
Decoder (0)
Tn Σ
Σ min (1= (0)A,, clip(r (0), (0),
(5)
Tg (at"
Sm
f1mm-1)
r/ (0)
Bas (a.
im
where Å, is an estimate of the joint advantage function. One can apply generalized advantage estimation (GAE) [32] with Vi = Σm= V(ōā.") Notably, the action generation process different between the inference and the training stage. In the inference stage, each action is generated auto-regressively, in the sense that a will be inserted back into the decoder again to generate am++1 (starting with a and ending with - While during the training stage, the output of all actions, a can be computed in parallel simply because a $1:--1 have already been collected and stored in the replay buffer.
as robust estimator for the joint value function.
The attention mechanism, which lies in the heart of MAT, encodes observations and actions with weight matrix calculated by multiplying the embedded queries, (g'1..... q²x ()) and keys, (q"", k5)). The embedded values (K41 (Kh...,;k" k""), "".. wfa) multiplied with the weight matrix to output representations. While the masked attention in the encoder uses a full weight matrix to extract the interrelationship between agents, i.e., Ô' the masked attentions in the decoder capture a with triangular matrices where (e(g",,**) 0 for r < j (see an visual illustration in Appendix A). With the properly masked attention mechanism, the decoder safely output the policy T finishes the implementation of Theorem (1).
where each of the weight wig"r,,'' k5)
&m+ Am=+1(a (a'm+t 6m, alt:",, which
The monotonic improvement guarantee. An MAT agent im optimizes trust-region objective that is conditioned on new decisions of agents 41-m-1 by means of conditioning its policy ratio on them (see Equation (5)). As such, it increases the joint return monotonically like if it followed the sequential update scheme of HAPPO [15, Theorem 2]. However, as oppose to that method, the MAT model does not require i. to wait until its predecessors make their updates, nor it uses their updated action distribution for importance sampling calculations. fact, as actions of all agents are outputs of MAT, their clipping objectives can be computed in parallel (during training), thus dominating
7

---

Table 1: Performance evaluations of win rate and standard deviation the SMAC benchmark, where UPDeT's official codebase supports several Marine-basede tasks only.
Task
Difficulty
MAT
MAT-Dec
MAPPO
НАРPО
QMIX
UPDeT
Steps
3m
Easy
100.0..)
100.0...
100.0.u.)
100.0u.:
96.913
100.0...
5e5
8m
Easy
100.0a..
97.5(25)
96.8
97.5GL
97.719
96.39.7
1e6
letado
Easy
100.0.4
100.0a.4
100.0.2)
97.5(18)
96.9(.
2c6
MMM
Easy
100.0.2
98. le.
95.6(45)
81.2
91.2
2c6
to 6krg
Hard
100.0..3)
95.9223
100.022
90.0448)
90.3:4m
5e6
34
Hard
100.0n17)
100.0.1)
100.0.2.5
91.91s3)
92.3/4.))
5e6
Hard
100.0..9
100.0:33)
72.5
90.0k..)
84.3:54)
3e6
Sm vs 6a
Hard
90.6:4.4
83. (4.)
88.2(62)
73.8(..)
75.8,37)
90.61a.
1e7
Hard
100.0
95.0cs6)
93.8
86.244.4
92.6:am
5e6
10m vs lim
Hard
100.0..4
100.022m
96.365
77.5.97)
95.8%6.
5c6
25m
Hard
100.0a.3)
86.9
100.0,27
70.1(R))
90,2
2.8
2e6
27m 30m
Hard+
100.0wa7)
95.3(22)
93.1Gam
5,6(2)
39.2
/
1e7
MMM2
Hard+
93.8.2.)
91.2(s3)
81.8aun
68.8/137
88.3.24)
/
1e7
6h va Br
Hard+
98.8(1))
93.8<a7>
88.4
0.3
9.76m
1c7
365 vs Johz
Hard+
96.5.13
85.3055
84.3
82.8
68.8212)
2c7
scademy_pess wwtpiiyypeessseeeee keeger
acedemy_ wit.1_ithheee keeper
ocademyy.cunteraltee
scoles
SCOPe
SCOOTES
MAT
MAT
Environment stops
Environment stops
Environment stops
Figure 5: Performance comparison the Google Research Foothall tasks with 2-4 agents from left to right respectively.
HAPPO on the time complexity. Lastly, assure that the limiting joint policy is such that none of the agents is incentivized to change its policy (Nash equilibrium). MAT requires permutating the sequential order of updates at every iteration, which is inline with the discovery HAPPO [15, Theorem 3].
5 Experiments and Results
MAT provides new solution paradigm for cooperative MARL problems. The key insights of MAT are the sequential update scheme, which is inspired by Theorem (1), as well as the encoder-decoder architecture, which provides highly-effin highly-efficient implementation for sequence modeling perspective. Importantly, MAT inherits the monotonic improvement guarantee, and agents' policies can be learned in parallel during training. We firmly believe MAT will become game changer for MARL studies.
To evaluate if MAT meets our expectations, we test MAT on the StarCraftII Multi-Agent Challenge (SMAC) benchmark [31] where MAPPO with parameter sharing [46] has shown superior performance, and the Multi-Agent MuJoCo benchmark [7] where HAPPO [15] shows the current state-of-the-arr performance. SMAC and MuJoCo environments
common benchmarks in the MARL field. On top of them, we also test MAT on the Bimanual Dexterous Hands Manipulation (Bi-Dex Dandds) [6] which provides a list of challenging bimanual manipulation tasks (see Figure (3)), and the Google Research Football [18] benchmark with a series of cooperation scenarios in football game.
We apply the same hyper-pane parammtrrssssrs of baseline algorithms from their original paper to ensure their best performance, and adopt the same hyper-parameter tuning process for our methods with details in Appendix B. To ensure fair comparisons to CTDE methods, we also introduce a CTDE-variant of MAT called MAT-Dec, which essentially adopts a fully decentralized actor for each individual agent (rather than using the decoder proposed in MAT) while keeping the encoder fixed. The critic's

---

back Toot
back shin
back thigh
0485
0000
MAT
MAT HAlides MARPO
MAT Matioee MARRO HAPP
1000
MAFFO
100
HAMPO
HAMO
Enviranment
Emviranment
Environment
forc, foot
foro shin
fore thigh
1001
1000
MAT
WAT
WAT
HAFFO
MATiDes
140
MAI Dhes
MAl DeC
Environment steps
Environment steps
Environment steps
Figure 6: Performance on the HalfCheetah task with different disabled joints shown in Figure (3a).
loss for MAT-Dec is L(Ø) = E R(o,, ar) + + 7 Em=1 V)(otT1) Em Vo(o(m)) and apply the local advantage estimation A,, aam)) guide the subsequent policy update.
5.1 Performance on Cooperative MARL Benchmarks
According to Table (1) and Figure (4), MAPPO significantly outperforms HAPPO in SMAC with higher sample efficiency. This verified the homogeneity of SMAC agents and the heterogeneity of multi-agent MuJoCo agents, that are also discovered by Kuba et al. [15]. Take the SMAC task 25m as an example, all the marines are equivalent and interchangeable so that agents can learn from their teammate's 's experience. Sharing parameters in this settings means leveraging 25 times more examples train each agents comparing with separated network of HAPPO, and thus enjoying higher learning efficiency. On the other hand, with the heterogeneous settings of multi-agent MuJoCo, training a "foot" agent with experience from "thigh" agent can surely harm its performance since they represent different functions on the Cheetah. However, MAT outperforms MAPPO and HAPPO in almost all tasks in Table (I) and Figure (4), indicating its modeling capability on both homogeneous and heterogeneous-ageen tasks. MAT also enjoys the superior performance over MAT-Dec, which emphasize the importance the decoder architecture in the MAT design. On the Bi-DexHands tasks, MAT outperforms MAPPO and HAPPO methods by large margin. We save the Google Football results to Figure (5), where the conclusion stays the same.
5.2 MAT as Excellent Few-short Learners
Since Transformer-based models often demonstrate strong generalization performance on few-short tasks [3, 9]. we believe MAT can possess strong generalization ability on unseen MARL tasks as well. To validate such an assumption,
design zero-shot and few-shot experiments SMAC and multi-agent MuJoCo tasks. For SMAC tasks, we pre-train agents on eight tasks involving five types of units (3m, 8m vs 9m, 10m vs 1 11m, 25m, 3s vs 3z, 2s3z, 3s5z, MMM) with 10M examples in total and then apply them on six separate and much harder tasks (5m vs 6m, 8m, 27m vs 30m, 2s vs Isc, Ic3s5z, MMM2) including seven types of units. This setting is designed generalization ability of MAT when training simple tasks but transferring to more diverse and complex downstream tasks. In terms of multi-agent MuJoCo, we reuse the models trained on the complete HalfCheetah robot as the pre-trained agent and then directly apply it to six new tasks, each with different leg being disfunctioned (see Figure (3a)) We investigate the generalization capability of pre-trained models on each downstream task with 0% (zero-shot), 1 1%, 5%, 10% few-short new examples, respectively. Note that common MARL baselines such as HAPPO assume fixed number of agents during training, thus it cannot directly handle the cases with changing number of agents.
evaluate the

---

Table Median evaluation win and the standard deviation the SMAC benchmark for pre-trained models with different number online examples.
Methous #examples
MAT
MAPPO
MAT-frum stratch
0%
5%
10%
1%
5%
10%
0%
1%
5%
10%
5m 6m
0.0pm
0.0pm
5.8
18.8
0.0cm
0.0mm
4.30
21,9
0.0pm
0.0ka
1.90
3.8
8m
100cm:
100ks
100ks;
100km
100kn
100
100ke)
10th..
0.0aa
10.6
92.5pm
100
27m 30m
0.0
6.3
13.8
71.2ss%
9,40
15x
26.2ms
26.8a%
0.0pm
0.0pm
0,0ea
0.3ese
2s vs 1sc
0.0cm
15.6
100km
100en
0.0cm
43.
100k.
10th.n
0.0aa
19.3axs
96.363
100ks:
Ic3s5z
3.
5,6.00
82.5
100
3,1on
4.34m:
73.8u50
97.5um
0.0am
7.514m
87,5gm
100
MMM2
0.0cw
0.0
33.8
62.5uz0
0.0cm
0.0cm
13.8m
36.2nm
0.0mmm
0.0rm
0.0cm
0.0cor
Table 3: Average evaluation score and standard deviation on Multi-Agent MuloCo for pre-trained models with different number of online examples.
Methods
MAT
MAPPO
MAT MAT-from scratch
#examples
1%
5%
10%
1%
10%
5%
10%
back foot
2100sn
2837cm
46912s
5646.m
2936.z0
3017:u9
322
3304use
-0.44am
-5.18
670
1635
back shin
4008..
4143
6077pm
7176.u
2406mm
2542ue4
2796
2955
-0.31a
-3.95
743
1252
back thigh
5361um
5641.s
7101mm
7460hmn
3043m
306th1e
3217mm
3353<n
-0.54am
-4.87m
930ccs
2067mam
fore foot
1313s12
1955
4856
6054
623
970
2025
2480zze
-0,37aa
-2.25mm
182
2877
fore shin
2435.xo
2617:o
3851:m
4373
1715
2457
3096
3310se
-0.15mm
-0.6ee
1461am
3003
fore thigh 5631cmm
6448km
7952
8347
3087
317
3340sm
35
40.29s
0.82a
1021mm
2600
We summarize the zero-shot and few-shot results of each algorithm in Table (2) and (3), where the bold number indicates the best performance. We also provide the performance of MAT if it was given the same amount of data but trained from scratch, the "MAT-from scratch", as the control group to demonstrate the effectiveness of pre-training process. As both tables suggest, bold numbers are mainly located in the area of MAT, which justify MAT's strong generalisation performance as a few-short learner. Surprisingly, we find that the few-shot MAT with only 10% data show even higher rewards than its counterpart that is purely trained on HalfCheetah with the same disabled joints (back foot, back shin and back thigh) and 100% full amount data, we believe it is because the pre-train process offers initial weights that are not only closer to optimum but also less likely to stuck bad local optima than random initialization.
6 Conclusion
In the past five years, large sequence models have achieved remarkable successes on solving visual language tasks. In this paper, we take the initial effort build the connection between multi-agent reinforcement learning (MARL) problems and generic sequence models (SM), with the ambition that MARL researchers can hereafter benefit from the prosperous development on the sequence modeling side. Specifically. we contribute by unifying a general solution to cooperative MARL problems into Transformer like encoder-decoder model. The proposed Multi-Agent Transformer (MAT) leverages the multi-agent advantage decomposition theorem, which essentially transforms the joint policy optimization process into a sequential decision making process that can be simply implemented by an auto-regressive model. We have demonstrated MAT's ss strong empirical performance on three challenging benchmarks against current state-of-the-a MARL solutions including MAPPO and HАPPO. Based on the established connection between MARL and SM, in the future, we plan to bring multi-agent learning tasks into large multi-modal SM, chasing for more generally intelligent models as the most recent success of GATO has already demonstrated [30].
Acknowledgment
The SJTU team is partially supported by "New Generation of AI 2030" Major Project (2018AAA0100900), Shanghai Municipal Science and Technology Major Project (202ISHZDZZX1122) Shanghai Sailing Program (2IYF1421900), and National Natural Sci- ence Foundation of China (62076161, 62106141).
10

---

References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. Stt.:2244.4199., 2022.
[2] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 2021.
[3] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. NeurlPS, 2020.
[4] Yu-Han Chang, Tracey Ho, and Leslie Kaelbling. All learning is local: Multi-agent learning in global reward games. Advances in neural information processing systems, 16, 2003.
[5] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint TrXiv:210600144. 2021.
[6] Yuanpei Chen, Yaodong Yang, Tianhao Wu, Shengjie Wang. Xidong Feng, Jiechuang Jiang, Stephen Marcus McAleer, Hao Dong, Zongqing Lu, and Song-Chun Zhu. Towards human-leve bimanual dexterous manipulation with reinforcement learning. arXiv preprint arXi::2206.0866 08686, 2022.
[7] Christian Schröder de Witt, Bei Peng, Pierre-Alexanrre Kamienny, Philip H. S. Torr, Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. CoRR, has/2003.06779, 2020.
[8] Xiaotie Deng, Yuhao Li, David Henry Mguni, Jun Wang, and Yaodong Yang. On the complexity of computing markov perfect equilibrium in general-sum stochastic games. arXiv preprint arXiv:2109.01795. 2021.
[9] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In NAACL-HLT (1), 2019.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[11] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi agget policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[12] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 2021.
[13] Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet: Universal multi-agent rein- forcement learning via policy decoupling with transformers. arXiv preprint arXiv:2101.08001 2021.
[14] Michael Janner, Qiyang I.,, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in Neural Information Processing Systems, 34, 2021.
[15] Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. ICLR, 2022.
[16] Jakub Grudzien Kuba, Xidong Feng, Shiyao Ding, Hao Dong, Jun Wang, and Yaodong Yang- Heterogeneous- mirror learning: continuum of solutions to cooperative marl, arXiv preprint arXiv: 01682, 2022.

---

[17] Jakub Grudzien Kuba, Muning Wen, Linghui Meng, Haifeng Zhang, David Mguni, Jun Wang. Yaodong Yang. et al. Settling the variance of multi-agent policy gradients. Advances Neural Information Processing Systems, 34: 1134-4887470 2021.
[18] Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. arXiv preprint arXiv: /80, 2019.
[19] Michael L Littman. Markov games as a framework for multi-agent reinforcement learning, In Machine learning proceedings 1994, pages 157-163. Elsevier, 1994.
[20] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed aaperative-competinee environments. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6382-6393, 2017.
[21] Xueguang Lyu, Yuchen Xiao, Brett Daley, and Christopher Amato. Contrasting centralized and decentralized critics in multi-agent reinforcement learning. arXiv preprint wwxxi:22122004400 2021.
[22] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. Advances in Neural Information Processing Systems, 32, 2019.
[23] Linghui Meng. Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang. Jun Wang, and Bo Xu. Offline pre-traine traitra ageti- decision transformer: One big sequence model conquers all starcraftii tasks. arXiv preprint arXiv: 1at.:2//2..2255 2021.
[24] David Mguni, Jianhong Wang, Taher Jafferjee, Nicolas Perez-Nieves, Wenbin Song, Yaodong Yang, Feifei Tong, Hui Chen, Jiangcheng Zhu, Yali Du, al. Learning to shape rewards using a game of switching controls. arXiv preprint arXiv: 103.09999, 2021.
[25] David Henry Mguni, Taher Jafferjee, Jianhong Wang, Nicolas Perez-Nieves, Oliver Slumbers, Feifei Tong, Yang Li, Jiangcheng Zhu, Yaodong Yang. and Jun Wang. Ligs: Learnable intrinsic- reward generation selection for multi-agent learning. In International Conference on Learning Representations, 2021.
[26] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, (11(77400:529-5333 2015.
[27] Prakash M Nadkarni, Lucila Ohno-Machado, and Wendy W Chapman. Natural language pro- cessing: an introduction. Journal of the American Medical Informatics Association, 18(5):544- 551, 2011.
[28) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[29] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference Machine Learning, pages 4295-4304.
PMLR, 2018
[30] Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. arXiv preprini arXiv:2205.06175. 2022.
[31] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nan- tas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. 2019.
12

---

[32] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 1506:0/550002233 2015.
[33] John Schulman, F. Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv,
2017.
[34] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference machine learning, pages 387-395. PMLR, 2014.
[35] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5887-5896. PMLR, 2019.
[36] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg. Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv: 1706. 05296, 2017.
[37] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[38] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Go- ing deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 32-42, 2021.
[39] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. arXiv preprint arXiv:2204.0166 2022.
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998 0008-600,0000 2017.
[41] Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu. Cross- former: A versatile vision transformer hinging on cross-scale attention. arXiv preprint arXiv:2188.00154 2021.
[42] Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. In International Conference on Learning Representa- tions, 2018.
[43] Ying Wen, Yaodong Yang, and Jun Wang. Modelling bounded rationality in multi-agent interactions by generalized recursive reasoning. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 414-421. International Joint Conferences Artificial Intelligence Organization, 7 2020. Main track.
[44] Yaodong Yang and Jun Wang. An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv preprint arXiv:20/ 0at::2///0000500 1,00583, 2020.
[45] Yaodong Yang, Ying Wen, Jun Wang, Liheng Chen, Kun Shao, David Mguni, and Weinan Zhang. Multi-agent determinantal q-learning. In International Conference on Machine Learning. pages 10757-10766. PMLR, 2020.
[46] Chao Yu, A. Velu, Eugene Vinitsky, Yu Wang, A. Bayen, and Wu. The surprising effectiveness of mappo in cooperative, multi-agent games. ArXiv, wtts//113.0111 2021.
13


Reviews:
[{"review_id": "MH0Q7Wfz7KX", "reviewer": "Reviewer_rnUZ", "summary": "The paper introduces a sequence model problem formulation of the cooperative MARL setup. Multi-Agent Transformer (MAT) is proposed to extend HAPPO by an encoder-decoder architecture which maps sequences of agent observations to sequences of agent actions. MAT is evaluated in SMAC, MA-Mujoco, and Dexterous Hands Manipulation to demonstrate superior performance over MAPPO and HAPPO, generalization, and few-shot learning capabilities.", "questions": "According to the SMAC documentation, the action spaces vary depending on the number of enemies. How does MAT deal with that in the experiments?", "limitations": "Limitations and potential negative societal impact have not been addressed in this paper.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- **Originality**: The paper presents an interesting and novel direction in cooperative multi-agent learning using state-of-the-art sequence-to-sequence learning techniques.\n- **Quality**: The paper is mostly technically sound. The explanations and experiments are presented well and the results adequately support the claims. However, the problem formulation in Section 2.1 actually represents a *Markov game* rather than a Dec-POMDP (according to [1] and the HAPPO paper). Please note that unlike the (unknown) state, the joint observation is not Markovian in Dec-POMDPs. Therefore, the value functions defined in Eq. 1 are only valid for Markov games but not for Dec-POMDPs [2].\n- **Clarity**: The paper is very well-written, clearly structured, and easy to follow.\n- **Significance**: The experimental results indicate a clear improvement over MAPPO and HAPPO with respect to performance, generalization, and few-shot learning capabilities. Due to the parallelizability of transformers, MAT potentially offers better computational efficiency than HAPPO although a report on wall clock time would have been nice.\n\nMinor comments:\n- Text and labels in Fig. 4 are very small and hard to read when printed. Labels in Fig. 5 are fine but the axes ticks are too small (I suggest to reduce the number of ticks and make the text larger).\n- The numbers in brackets in Table 2 and 3 are unreadable when printed.\n\n[1] Littmann, \"Markov Games as a Framework for Multi-Agent Reinforcement Learning\", ICML 1994\n\n[2] Oliehoek et al., \"Optimal and Approximate Q-value Functions for Decentralized POMDPs\", JAIR 2008", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GyoYTGyTgTH", "reviewer": "Reviewer_YpeW", "summary": "This work proposes the use of sequence models (SMs), specifically Transformers, to implement the Multi-Agent Advantage Decomposition theorem. This is done by introducing the Multi-Agent Transformer (MAT), an encoder-decoder model that sequentially predicts the action of an agent based the actions taken by previous agents in a random permutation. MAT is evaluated on four different benchmarks: SMAC, Multi-Agent MuJoCo benchmark, Bi-DexHands and Google Research Football benchmark. MAT achieves strong performance on SMAC and MuJoCo compared to the provided baselines. Further experiments show evidence that MAT can be good at few-shot learning.", "questions": "- This work seems relevant and should probably be addressed and possibly compared to:  \n[1] Hu, Siyi, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. \"Updet: Universal multi-agent reinforcement learning via policy decoupling with transformers.\" arXiv preprint arXiv:2101.08001 (2021).\n\n- What would happen to its performance if we replaced the transformer by other types of recurrent models (GRU, LSTM)? \n", "limitations": "Limitations were not addressed in the manuscript, as reported by the authors.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- I found the application of Transformers’ causal masked attention to the Multi-Agent Advantage Decomposition theorem novel.\n- The performed experiments were instructive and provide good insight into the MAT’s performance. I particularly appreciated it’s ability to model both homogeneous and heterogenous agent tasks.\n- I thought the paper was well organized and well-written.\n\nWeaknesses:\n- Where is the related work section? This is an important part of any scientific paper as it positions this work in the general area, (i.e., MARL). Only covering two contemporary methods is not sufficient in my opinion.\n- I found a MAPPO result mismatch with paper; for example, for MMM2, the original paper reports 90.6 win rate versus 81.8 in the current manuscript. Did the authors implement their own version? If so, why not use the published version of the results?\n- I found the explanation of the few-shot experiments confusing and unclear (section 5.2). For instance what exactly was done in the last meta-column (MAT-from scratch)? Also to help with the flow, I would suggest moving the paragraph starting at line 265 to the beginning of section 5.2. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4UaHMltxwiM", "reviewer": "Reviewer_MEiB", "summary": "This paper studies transformer networks in an actor-critic algorithm with a multi-agent setting. The authors first picks a random order of agents, motivated by the advantage decomposition theorem. Given this ordering of agents and their observations and actions, they formulate multi-agent RL as a sequence modeling problem. A transformer encoder is first applied to a sequence of observations to generate observation encodings and value of each agent. Conditioned on observation encodings, a transformer decoder is applied to a sequence of actions to generate action distributions of every agent. A new ordering of agents is resampled for every iteration to prevent agents from changing their policy. Different from HAPPO, the authors use a different objective where parameters of all agents are updated simultaneously rather than sequentially, which can potentially be faster with gradient accumulation and batching. When evaluated on a set of cooperative multi-agent environments, the proposed model, MAT, outperforms strong baselines. It is also shown to perform well with scarce data.", "questions": "I have several questions regarding my concerns above.\n\n1. Can you compare MAT to other sequential models, especially ones that use transformer architecture? If there are comparable approaches, can you add baseline experiments?\n2. Can you clarify the advantage of a transformer decoder w.r.t. parallel training?\n3. Can you add ablation studies to understand different components in the transformer? Also, another ablation on just using the decoder where observations and actions for all agents $i_{1:m-1}$ are provided as input to the decoder to generate value and action distribution for agent $i_m$? This would be useful to understand if the added complexity of encoder-decoder architecture, which is inspired by the decomposition theorem where observations of all agents are given, is needed to get improved results.", "limitations": "The authors addressed the limitations and included a source code with their submission.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength is improved performance and shared experience from all agents in a flexible framework like transformers. Main weaknesses are lack of related work and some of the claims.\n\n## Strengths\n1. The paper is overall well written and easy to follow.\n2. The authors use a transformer architecture in a multi-agent setting and creates a connection to the multi-agent advantage decomposition theorem.\n3. Experimental results show improvements.\n\n## Weaknesses\n1. The main contribution of the paper is the transformer architecture but related work lacks any prior work on using sequential models in a multi-agent setting. It is not easy to position the paper without knowing if the encoder-decoder structure, transformers, or advantage decomposition has been studied in the past. Especially, UPDeT*, which also uses a transformer architecture and a similar encoder-decoder structure, should be compared.\n2. You mention that \"the transformer architecture allows sequential policies to be trained in parallel\" but it is not clear what this means. While you are using a different objective with all actions of all agents are available, you still need to run the decoder on $\\hat{a}^{i_{1:m-1}}_t$ to generate the action distribution for agent $i_m$. Main difference from the autoregressive variant seems to be batching of input and gradient accumulation but not clear if this is what you meant by parallelism.\n3. It is not clear if all of transformer components are useful in this setting. For example, since you are shuffling agents at every iteration, position encoding might not be very relevant and can be discarded. I think an ablation study is needed to understand the effect of different components.\n\n\n\\* UPDET: UNIVERSAL MULTI-AGENT REINFORCEMENT LEARNING VIA POLICY DECOUPLING WITH TRANSFORMERS. Siyi Hu, Fengda Zhu, Xiaojun Chang, Xiaodan Liang.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept