Offline Pre-trained Multi-Agent Decision Transformer
Authors: Linghui Meng
Muning Wen
Yaodong Yang
chenyang le
Xi yun Li
Haifeng Zhang
Ying Wen
Weinan Zhang
Jun Wang
Bo XU
Venue: ICLR 2022 Submitted (2022)
Keywords: Multi-Agent Reinforcement Learning
Offline Reinforcement Learning
Machine Learning

Abstract: Offline reinforcement learning leverages static datasets to learn optimal policies with no necessity to access the environment. This is desirable for multi-agent systems due to the expensiveness of agents' online interactions and the demand for sample numbers. Yet,  in multi-agent reinforcement learning (MARL), the paradigm of offline pre-training with online fine-tuning has never been reported, nor datasets or benchmarks for offline MARL research are available. In this paper, we intend to investigate whether offline training is able to learn policy representations that elevate performance on downstream MARL tasks. We introduce the first offline dataset based on StarCraftII with diverse quality levels and propose a multi-agent decision transformer (MADT) for effective offline learning. MADT integrates the powerful temporal representation learning ability of Transformer into both offline and online multi-agent learning, which promotes generalisation across agents and scenarios. The proposed method demonstrates superior performance than the state-of-the-art algorithms in offline MARL. Furthermore, when applied to online tasks, the pre-trained MADT largely improves sample efficiency, even in zero-shot task transfer. To our best knowledge, this is the first work to demonstrate the effectiveness of pre-trained models in terms of sample efficiency and generalisability enhancement in MARL.

Under review as a conference paper at ICLR 2022
OFFLINE PRE- MULTI-AGENT DECISION TRANSFORMER
Anonymous authors Paper under double-blind review
ABSTRACT
Offline reinforcement learning leverages static datasets to learn optimal policies with no necessity to access the environment. This desirable for multi-agent systems due to the expensiveness of agents' online interactions and the demand for sample numbers. Yct, in multi-agent reinforcement learning (MARL), the paradigm of offline pre-training with online fine-tuning has never been reported, nor datasets or benchmarks for offline MARL research are available. In this paper, we intend to investigate whether offline training is able learn policy represen- tations that elevate performance on downstream MARL tasks. We introduce the first offline dataset based on StarCraftIl with diverse quality levels and propose a multi-agent decision transformer (MADT) for effective offline learning. MADT integrates the powerful temporal representation learning ability Transformer into both offline and online multi-agent learning, which promotes generalisation across agents and scenarios. The proposed method demonstrates superior performance than the wate-off-hhe-ar algorithms in offlin MARL. Furthermore, when applied to online tasks, the pre-trained MADT largely improves sample efficiency, even in zero-shot task transfer. To our best knowledge, this is the first work to demon- strate the effectiveness pre-trained models in terms of sample efficiency and generalisability enhancement in MARL.
INRTODUCTION
Multi-Agent reinforcement learning (MARL) plays essential role for solving complex decision- making tasks by learning from the interaction data between machine or autonomous agents and simulated physical environments. It typically applies self-driving (Shalev-Shwartz al., 2016), traffic control (Bazzan, 2009), and recommendation system (Xian et al., 2019), and, to specifically mention, surpasses human beings on some video games (Bellemare al., 2013; Wang et al., 2020). However, the experience-based policy learning scheme requires the algorithms with high sample efficiency because of the limited computing resources and high cost resulting from the data collec- tion (Haarnoja et al., 2018; Munos et al., 2016; Espeholt et al., 2019; 2018). Furthermore, even in domains where the online environment is feasible, we might still prefer to utilize previously collected data instead, for example, if the domain's 's complex requires large datasets for effecttve generalization. In addition, a policy trained on one scenario usually cannot perform well on another though under the same task. Therefore, universal policy is critical for saving the training time of general reinforcement learning. Offline RL (Levine et al., 2020) attracts more researchers motivated by improving online reinforcement learning with offline datasets (Fu al., 2020). Those methods like CQL (Kumar et al., 2020), BEAR (Kumar et al., 2019), and BCQ (Fujimoto et al., 2019) conventionally leverage the current off- off-ppoicy RL algorithms. e.g., DQN (Mnih al., 2015), with provable constraints (Rashidinejad et al., 2021). They mainly focus on solving the overestimation of Q values learned in offline datasets and the distribution shift between offline and online. However, those constraints limit the performance with insufficiently offline dataset exploitation (Kumar et al., 2019). Recently, the transformer shows its advantage of representing the sequential data for the classification or prediction tasks. Most related to our work, Chen et al. (2021) replace the constrained Q networks in conventional offline RL algorithms with causal transformer and demonstrate the superiority comparing with baselines such as Behavior Cloning (BC) and state-of-the-an offline algorithms in single-agent offline RL. Nevertheless, the overestimation problems emerge in the offline multi-agent RI. when directly applies the sequential modeling method. We propose leveraging

---

Under review as a conference paper at ICLR 2022
store data
SOTA Policy/
Offline dataset
Demonstration
Offline
data
Dalaset
Ioad data
Trainer
train
MADT
rollout
Online
collect
Buffer
Online Env
Figure 1: Overview of the pipeline for pre-training the general policy and fine-tuning it online.
the representation capability of the transformer and then load the pre-trained model part of the policy the fine-tuning phase.
This paper proposes Multi-Agent Decision Transformers (MADT) for pre-training the general policy on offline datasets, capable of generalizing onto other seen/unseen environments. Firstly, we collect the offline datasets from the well-known, challenging MARL environment, StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019), interacting with the state-of-the-ae algorithm, multi- agent PPO (Yu et al., 2021). Then we construct transformer variants for multiple agents with parameter sharing or acing as a part. We give the paradigm of pre-training model with the transformer and fine-tuning with MARL. We conclude the main contribution follows:
We propose series of transformer variants for offline MARL via leveraging the sequential modeling of the attention mechanism.
We derive framework with offline pretraining and online fine-tuning to improve sample efficiency and generality across different scenarios.
We are the first to apply the pre-trained model across different multi-agent scenarios to validate the transferring ability. Experimental results show fast adaptation and superior performance.
We build a dataset with different skill-level covering all SMAC scenarios for benchmarking progress in MARL pre-training.
2 PRELIMINARY
Multi-Agent Reinforcement Learning. For the Markov Game, which is multi-agent extension from the Markov Decision Process (MDP), there is a tuple representing the essential elements S, A. R, P. n y 2, where denotes the state space of n agents S1 x S2 S, + S. A, the action space of each agent i, P S, x A; + PD((S) denotes the transition function emiting the distribution over the state space and function of each agent and takes action following their policies (als) E II, S + PD(A) from the policy space 1I, where II; denotes the policy space of agent i, a A., and s ~ S,. Each agent aims to maximize its long-term reward Σ, T'T,, where rf E Rt denotes the reward of agent i in time and y denotes the discount factor.
the joint action space, R x A, - R is the reward
Attention-based Model. Attention- dieettionbbased model has shown its stable and strong representation capability. The scale produccion attention uses the self-attention mechanism demonstrated in (Vaswani al., 2017). Let Q TR', Xd, be the quries, K Rdx be the keys, and V E * be the values, where f. are the element numbers of different inputs and d, are the corresponding element dimensions. Normally, 4 = t, and dy = dz. The outputs of self-attention are computed as:
QKT
Attention
Cantiion(,,,n Q K, V) softmax
V
Vd
where the scalar 1/ /d is used to prevent the softmax function into regions that has very small gradients. Then introduce the multi-head attention process follows:
MultiHead Q, K, V) = Concat (onada(head),..... head,,) W
(2)
head, Attention Q Qw? KW. VW!)
(3)
2

---

Under review as a conference paper at ICLR 2022
The position-wis wwise feed-forward network is another core module of the transformer. It consists of two linear transformations with a ReLU activation in between. The dimensionality of inputs and outputs is mondel. and that of the feef forward layer is dzz-- Specially,
(FN max(O, xW1 b1) b2
where W1 E Rdmoddexd4y and W2 E R2 Xdmodet are the weights, and bj Rd/ and ba E Rdmoodt are the biases. Across different positions are the same linear transformations. Note that the position encoding for leveraging the order of the sequence follows:
PE(pos, 24) sin pos/ 2000024/dmooo
(5)
PE(pos, 2i + 1) cos pos/ 10000 2:/ /mddeel
METHODOLOGY
In this section, we demonstrate how the transformer applied to our offline pre-training MARI. framework. Firstly, we introduce offline MARL method, in which the transformer maps be- tween the local observations and actions of each agent in the offline dataset via parameter sharing sequentially. Then we leverage the hidden representation as the input of the multi-agent decision transformer (MADT) to minimize the cross-entrops loss. Furthermore, we introduce how to integrate the online MARL with MADT in constructing policy. To accelerate the online learning, we load the pre-trained model as a part of MARL algorithms and learn the policy based on experience in the latest buffer stored from the online environment. To train a universal MARL policy quickly adapting to other tasks, we bridge the gap between different scenarios from observations, actions, and available actions, respectively. Figure 1 overviews our method from the perspective of offline pre-training with supervised learning and online fine-tuning with MARL algorithms.
whole framework to train universal MARL
3.1
MUINTI-GGENT DECISION TRANSFORMER
Algorithm shows the training process of Multi-Agent Decision Transformer (MADT), in which we autoregressively encode the trajectories from the offline datasets in offline pre-trained MARL and train the transformerbbaee network with supervised learning. We carefully reformulate the trajectories as the inputs of the causal transformer different from those in the Decision Transformer (Chen et al., 2021). Denote that we deprecate the reward-to- and actions that are encoded with states together in the single-agent DT. We will interpret the reason for this the next section. Similar to the seq2seq models, MADT is based on the autoregressive architecture with the reformulated sequential inputs across timescale. The left part Figure shows the architecture. The causal transformer encodes the agent i's trajectory sequence T at the time step to a hidden representation h (h1, h2, ..... ./)) with a dynamic mask. Given h2, the output at the time step t only based on the previous data then consumes the previously emitted actions additional inputs when predicting new action.
Algorithm 1: MADT-Offine: Multi-Agent Decision Transformer
Input: Offline dataset D {T; (8f, of, af, of, di, P of denotes the available action Initialize 0 for the Causal Transformer:
Initialize a as the learning rate, as the context length, and n as the maximize agent number
for 7 = {r,,..., ......, Ta in D do
Chunk the trajectory into T, = {ri, sf at heic as the ground truth samples, where C is the context length, and mask the trajectory when df is true
for T. = = of ... T.'} in T; do
Mask illegal actions via P(ay Tr, azy;; 0') 0 0f of is True
Predict the action ày = arg max, P(alTe, d'r;
Update / with 0 = arg max C Eの= P(a;) log P(ä (T, 0')
end end return 0
Trajectories Reformulation as Input. We model the lowest granularity at each time step as a modeling unit x, from the static offline dataset for the concise representation. MARL has many
3

---

Under review as a conference paper at ICLR 2022
Critic Loss e.g.. Huber
Add & Norn
Actor Loss e.g., PPO
aval 1-1
aval 1+1
aval
Feod Fooword
Causal Transformer
Actor Network
Critic Network
Offline Datasets
X agent
Padded Agent Observation Embedding
Local Observation
Global
Padded Agent Padded Action Embedding
Agent's Action
All Agents' Actions
Figure The detailed model structure for offline and online MADT.
elements such as (gl state, is reasonable for sequential modeling methods to model them in Markov Decision Process (MDP). Therefore, we formulate the trajectory as follows:
observation) different from the single agent.
(#11....
.T) where = (st, o a
where si denotes the global shared state, of denotes the individual observation for agent at time step and 0j denotes the action. We take 2e as token similar to the input of natural language processing.
Output Sequence Constructing, To bridge the gap between training with the whole context trajec- tory and testing with only previous data, we mask the context data to autoregressively output in the time step & with previous data in (1 (l.... 1). Therefore, MADT predicts the sequential actions at each time step using the decoder as follows:
y = ar arg max ma T, 0i,..., at-1)
where 0 denotes the parameter of MADT and denotes the trajectory including the global state local observation o before the time step I, Pa is the distribution over the legal action space under the available action
Core Module Description. MADT differs from the transformers in conventional sequence modeling tasks that take inputs with position encoding and decode the encoded hidden representation autore- gressively. We use the masking mechanism with lower triangular matrix to compute the attention:
QKT
Attention (Q, K, V) softmax
+ M)V
VTTE
where M the mask matrix which ensures the input the time step & only correlate with the input from .1....,t 1). We employ the Cross-entropy (CE) as the total sequential prediction loss and utilize the available action ensure agents taking those illegal actions with probability zero. The CE loss can be represented as follows:
-
Lce(0) - C
Σ P(a.) log P(arlr,,@cs;0), where C is the context length 1=1
(8)
where 0, is the ground truth action, includes {814, )1:). à denotes the output of MADT. The cross-entropy loss shown above aims to minimize the distribution distance between the prediction and the ground truth.
3.2 MULTI-AGENT DECISION TRANSFORMER WITH PPO
The method above can fit the data distribution well resulting from the sequential modeling capacity of the transformer. But it fails to work well when pre-training the offline datasets and improving continually by interacting with the online environment. The reason originates from the mismatch between the objective in MADT and ignore the value of each action. When the pre-trained model
4

---

Under review as a conference paper at ICLR 2022
is loaded to interact with the online environment, the buffer will only collect actions conforming to the distributions of the offline datasets rather than those corresponding to high reward at this state. That means the pre-trained policy is regarded as good once the selected action is identical to that in the dataset, even though it leads to low reward. Therefore, we need to design another paradigm, MADT-PPO, to integrate RI. and supervised learning for fine-tuning continually in Algorithm 2. Figure 2 shows the pre-training and fine-tuning framework. A direct method is to share the pre-trained model across each agent and implement the REINFORCE algorithm (Williams, 1992). However, only actors result in higher variance, and the employment of critic to assess state values is necessary. Therefore, in online MARL, we leverage an extension of PPO, the tfte-of-the--ar algorithm on tasks of StarCraft, MPE, and even return-based game Hanabi (Mordatch & Abbeel, 2018).
Algorithm 2: MADT-Online: Multi-Agent Decision Transformer with PPO
Input: Offline dataloader D, Pretrained MADT Policy with parameter 0 Initialize 0 and & are the parameters of an actor To To(a: (a:loz) and critic (s()) respectively, which could be inherited directly from pre-trained models. Initialize as the agent number, 7 as the discount factor, and e as clip ratio
for 7 = {T [ ................................ To..... ]n. in D do
Sample T; {s", of, at heI.T as the ground truth, where T' is the context length Compute the advantage function A(s, a.) = E, " a.) (s(s)) To (:: (;) Tome (0: a4)
Computing the important weight Update 0g for é E 1.... n via:
0, arg max E Ber arott ond [clip( 1 +, c)
Compute the MSE loss La = 11E,7 [=) Update the critic network via @ arg min Ls
end return 0, ó
3.3 UNIVERSAL MODEL ACROSS SCENARIOS
To train universal policy for each of the scenarios in the SMAC which might be vary in agent number, feature space, action space and reward ranges, we consider the modification list below.
Parameters Sharing Across Agents. When offline examples collected from multiple tasks or the test phase owns the different agent numbers from the offline datasets, the difference of agent number across tasks is intractable problem for deciding the number of actors. Thus, we consider sharing the parameters across all actors with one model as well as attaching one-hot agent IDs into observations, for compatibility with a variable number of agents.
Feature Encoding. When the policy needs to generalize to new scenarios that arises from different feature shapes, We propose encoding all features into universal space by padding zero at the end and mapping to a low-dimensional space with fully connected networks.
Action Masking. Another issue is the different action space across scenarios. For example, less enemies in scenario means less potential attack options as well as available actions. Therefore, an extra vector utilized to mute the unavailable actions so that their probabilities are always zero during both learning and evaluating process.
Reward Scaling. Different scenarios might be vary reward ranges and lead to unbalanced models during multi-task offline learning. To balance the influence of examples from different scenarios, we scale their rewards into the same range to ensure the output models have comparable performance between different tasks.
EXPERIMENTS
We show three experimental settings: offline MARL, online MARL by loading the pre-trained model, and few-shot or zero-shot offline learning. For the offline MARL, we expect to verify the performance of our method in pre-traine mre-trainin the policy and directly testing the corresponding maps. For the fine-tuning in the online environment, we aim to demonstrate the capacity of the pre- pre-trained policy on the original or new scenarios. Experimental results in offline MARL shows MffT-offline
5

---

Under review as a conference paper at ICLR 2022
230 125
5.
0.8
2.4
Tiestes
23.4
27.
14
6.4
0.5
3.2
Trunts
Arreee Bean
Leary Ma
(a) 2632. (eass)
(b) 3:52 (hard)
(c) 3652 vs. 3562 (super hard)
(d) corridor (super hard)
Figure 3: Performance of offline MADT comparing with baselines on four easy or (super-)hard SMAC maps. The dotted lines represent the mean values in the training set. Columns (a-d) are average returns from (poor, medium, good) datasets from top to the bottom.
in 3.1 outperforms the mate-offtheear methods. Furthermore, MADT-online in 3.2 can improve the sample efficiency across multiple scenarios. Besides, the universal MADT trained from multi-task data with MADT-online generalize well in each scenario in few-shot even zero-shot setting.
4.1 OFFLINE DATASETS
The offline datasets are collected from the running policy, MAPPO (Yu et al., 2021), the well- known SMAC task (Samvelyan et al., 2019). Each dataset contains a large number of trajectories: datasets consider the
= (st, ot, de, r2, done,, vt),-11 Different from D4RL (Fu et al., 2020), property of DecPOMDP, which owns local observations and available actions for each agent. In Appendix, list the statistical properties of the offline datasets in Table and Table 3.
4.2 OFFLINE MULTI-AGENT REINFORCEMEN LEARNING
In this experiment, aim to validate the effectiveness of MADT offline version in Section 3. as a framework for offline MARL on the static offline datasets. We train a policy on the offline datasets with various qualities and then apply it to online environment, StarCraft (Samvelyan et al., 2019). There are also baselines under this setting, such as Behavior Cloning (BC) as kind of imitation learning method showing stable performance on single-agent offline RL. Besides, we employ the conventional effective single-agent offline RL algorithms,, BCQ (Fujimoto et al., 2019), CQL (Kumar et al., 2020), and ICQ (Yang et al., 2021), then add the mixing network for multi-agent setting, denoting "XX-MA". To verify the quality of our collected datasets, we choose data from different level and train the baselines as well as our MADT. Figure shows the overall performance various quality datasets. The baseline methods enhance their performance stably, indicating the quality of our offline datasets. The results also show that our MADT outperforms the offline MARL baselines and converges faster across easy, hard, and super hard maps (2s3z, 3s5z, 3s52 vS_ _v__3662 corridor).
4.3
OFFLINE PRE-TRA TRAINING AND ONLINE FINE-TUNING
The experiment designed in this subsection intends to answer the question: the pre-training pro- cess is necessary for online MARL? Firstly compare the online version of MADT in Section 3.2 with and without loading the pre-trained model. If training MADT only by online experience, we can view transformer-hased MAPPO replacing the actor and eritic backbone networks with the

---

Under review as a conference paper at ICLR 2022
17.5
15.0
13.0
Auenge Return
Auenge Retum
12.3
30.0
30.0
MAA moze!
arotiannn
oveare
5.0
Timasiaaa
Timasuapa
many baneling (casy)
(b) 5m 6m (hard)
(c) 3s5z
356z (super hard)
Figure 4: The average returns with and without the pre-trained model..
transformer. Furthermore, we validate that our framework MADT with the pre-trained model can improve sample efficiency on most easy, hard, and super hard maps.
Necessity Pretrained Model. We train our MADT based on the datasets collected from a map and fine-tune it on the same map online with the MAPPO algorithm. For comparison fairness, we use the transformer as both actor and critic networks with and without the pre-trained model. Primarily, we choose three maps from Easy, Hard, and Super Hard maps to validate the effectiveness of the pre- pre-trained model in Figure 4. Experimental results show that the pre- pre-trained model converges faster than the algorithm trained from scratch, especially in the challenging maps.
Improving Sample Efficiency. For validating the improvement of the sample efficiency via loading our pre-trained MADT and fine-tuning it with MAPPO,
compare the overall framework with the state-of-the-aa art algorithm, MAPPO (Yu et al., 2021), ), without the pre-training phase. We measure the number of interactions with the environment in Table I, and our pre-trained model needs much less than the traditional MAPPO to access the same win rate.
Maps
the Wis
Maçu
Acess Win
20%
40%
60%
50%
100%
20%
40%
60%
80%
1005
2t la (Bway)
Re4/
1e5t
1.3c5)
L.5e5..
1.665/
3o vs. Se (Hard)
Bedt-
8.5c5i.
8.7c51 Se4
9c505e4
206/1.55
(Easy)
3.2631
8.364/
3.265/
7.2
365t
4c5/el
Sc50te5
1.5e6555
vs Ise (Bay))
led:
2.5el/
Sedr
Ro44464
3651 265
5t 15, 9ta (Haud)
Bedt
6e5t
oc2..26
(Easy)
2.5e5/
3e5t
6.2651l4
7.35513.55
9e5/2965
(Hart)
1.5e60e5
2.5er8ee5
Se662e6
35 vs. (Basy)
3651
4e5%
Se5
6.2c51
1.5//1.865
365e (Haud)
Be5ti Sed
1.3e6/te5
1.5e54e55
1.9e00e6
2.5e6026
baneling
3.
165444
3.2657644
Sc51864
Te66
10m 11m (Hend)
3.5cf
4e512,8e4
1.Tel/
466/2.555
Mes (Bary)
Sedk
5 M6dee5r1.6 fe5
MMMM2 (Super Hart)
166/
1.te6
2.5e6/
4etr
00/30
MMM (llry)
5.2641
3etr-
4.5c56
1.BeWNe
3dte (Seper Hand)
2.5e6
3eWNed
Sed/lel
bate vs. bane (Bary)
3.2
2636
3.3e5
465
comióor (Saper hand)
1.5e//-
265-
2.No0
Table 1: The sample numbers needed to access the win rate 20%, 40%, 60%, 80%, (MAPPO/pre-trainee MADT). "-" represents that no more samples is needed to reach the target win rate. "xo" represents that policies cannot reach the target win rate.
and and 100% with
4.4 GENERALIZTION WITH MULTI-TAL TASK PRE-TRAN TRAINING
Experiments in this section are to explore the transferability of the universal MADT mentioned in Section 3.3, which is pre-trained with mixed data from multiple tasks. According to whether the downstream tasks have been seen not, the few-shot experiments are designed to validate the adaptibilty on seen tasks, while the zero-shot experiments are designed for the held-out maps.
Few-shot Learning. The results in Figure 5a shows that method can utilize multi-task datasets to train a universal policy and generalize to all tasks well. Pre-trained MADT can achieve higher return than the model trained from scratch when we limit the interactions with environment.
Zero-shot Learning. Figure 5b shows that our universal MADT can surprisingly improve perfor- mance on downstream task even if has not been seen before. (3 stalkers vs. 4 zealots).
7

---

Under review as a conference paper at ICLR 2022
20.0
From streich
Fine tuned from unteersal MADT
17.5
Universal MADT (ours)
20
Avorage Roturn
15.0
Train from scratch
Averge Retrm
12.5
10.0
1.0
5.0
2,5
2m
25,05
20,00.3
8.000
0.25
0.50
0.75
1.00 Timesteps
1.25
1.50
1.75
2,00
(b)
Figure 5: Few-shot and Zero-shot validation results. (a) shows the average returns of the universal MADT pre-trained from all five tasks data and the policy trained from scratch, individually. limit the environment interaction to 2.5M steps. (b) shows the average returns of held-out map [3s_v__ 22),, where the universal MADT is trained from data on (2m 12, _s_ 1sc, 3m, _s_ss vs _2).
14
14
12.5
Avarage naturn
Awragg Rotum
Average Rettum
10
30
MADT with MAPO
755
NAOT with MAUTO Trecurtn
5.
Stat, only
stide
0.0
0.6
(b)
(c)
Figure 6: Ablation results on a hard map, 5m_ vs_ 6m, for validating the necessity of (a) MAPPO MADT-
Online, (b) input formulation, (c) online version of MADT.
4.5 ABLATION STUDY
The experiments in this subsection are designed to answer the following research questions: RQ1: Why we choose MAPPO for online phase? RQ2: Which kind of input shall we use to make the pre-trained model beneficial for the online MARL? RQ3: Why cannot the offline version of MADT be improved in the online fine-tune period after pre-training?
Suitable Online Algorithm. Although the selection of MARL algorithm for online phase should be flexible according to specific task, we design experiments to answer RQ1 here. As discussed in Section 3, we can train Decision Transformer for each agent and online fine-tur tune with MARL algorithm. An intuitive method is to load the pre-trained transformer and take it the policy network for fineituning with the policy gradient method, e.g. REINFORCE (Williams, 1992). However, for the reason of high variance mentioned in Section 3.2, choose MAPPO the online algorithm and compare their performance in improving sample efficiency during the online period in Figure 6a.
Dropping Reward-to-go in MADT. To answer RQ2, compare different inputs embedded into the transformer, including the combination of state, reward-to-go. and action. We find the reward-to- go harmful to online tne-tuning performance, shown in Figure 6b. We suppose the distribution of reward-to-ge is mismatch between offline data and online samples. That is. rewards of online samples are usually lower than offline data due stachastic exploration the beginning of the online phase. It deteriorates the fine-tuning capability of the pre-trained model, and based on Figure 6b, we only choose states as our inputs for pre-training and fine-tuning.
Integrating Online MARL with MADT. To answer RQ3, we directly apply the offline version of MADT pre-training and fine-tune online. However, Figure 66 shows cannot be improved during the online phase. We analyze the results from the absence of motivation for chasing higher rewards, and conclude that offline MADT is in fact supervised learning and tends to fit its collected experience even with unsatisfactory reward.
8

---

Under review as a conference paper at ICLR 2022
5 RELATED WORK
Offline Deep Reinforcement Learning. Conventionally, offline RL trains a policy based on the static and previously collected data without any environmental interaction (Levine al., 2020). This policy is then leveraged to interact with the online environment to obtain promising results. A straightforward method for the current reinforcement learning is to use the off- off-policy algorithm and regard the offline datasets as a replay buffer to learn policy with good performance. However, experience existing in offline datasets and interaction with online environment have different distributions which cause the overestimation in the off-policy (value-based) method (Kumar et al., 2019). Substantial works presented in offline RI. aim resolving the distribution shift between the static offline datasets and the online environment interaction (Kumar et al., 2020; Fujimoto et al., 2019; Kumar et al., 2019). Yang et al. (2021); Jiang& & Lu (2021) constrain off-policy algorithms in offline MARL. Related to our work for the improvement of sample efficiency, Nair et al. (2020) load the pretrained model to generate weight for actor-critie online RL algorithms. Recently, the Decision Transformer outperforms many state-of-the-art offline RL algorithms via regarding the training process as a sequential modeling phase and test on the online environment (Chen et al., 2021; Janner et al., 2021). ). In contrast, we show transformer-basee method in the multi-agent field, attempting to transfer across many scenarios without extra constraints.
Multi-Agent Reinforcement Learning. As natural extension from single-agent RL, MARL attracts much attention to solve more complex problems under Markov Games (Zhang et al., 2021). Related algorithms conventionally enforce multiple agents to interact with the environment online and collect the experience to train the joint policy from scratch (Rashid et al., 2018; Sunehag et al., 2017; Son et al., 2019; Mahajan et al., 2019; Yu al., 2021). The framework of traditional multi- agent learning employs the centralized training and decentralized execution framrwork due to local observation and increasing action space. We build offline datasets by collecting elements from the DecPOMDP and train the general policy with parameter sharing. We utilize transformer to learn policy with previously collected data from the golden policy in advance and fine-tune it online.
Transformer. Transformer (Vaswani et al., 2017) has achieved a great breakthrough to model relations between the input and output sequence with variable length, for the sequence-to-sen to-seueece problems (Sutskever et al., 2014), especially on machine translation (Wang et al., 2019) and speech recognition (Dong et al., 2018) Recent works even reorganize the vision problems as the sequential modeling process and construct the SOTA model with pretraining, named ViT (Han et al., 2020; Dosovitskiy et al., 2020; Liu et al., 2021). Due to the Markovian property of trajectories in offline datasets, we can utilize Transformer as that in language modeling. Therefore, Transformer can bridge the gap between supervised learning in the offline setting and reinforcement learning in the online interaction because of the representation capability. We claim that the components under Markov Games are sequential then utilize the transformer for each agent to fit transferable MARL policy, Furhtermore, we fine-tune the learned policy under via trial-and-error.
CONCLUSIONS
In this work, we propose MADT, an offline pre-trained model for MARL, which integrates the transformer to improve sample efficiency and generalizability. We build an offline MARI. dataset based on SMAC. Our method outperforms the state-of-the-an methods in offline MARL, including BC, BCQ, CQL, and ICQ. Our whole framework can also drastically improve the sample efficiency of online MARL via loading pre-trained model. Furthermore, we apply our method to train a universal policy over the multi-task datasets and fine-tur fnn-tune it under the few-shot and zero-shot settings. Results demonstrate that this universal policy adapts fast to new tasks and promises performance on different level maps.
To our best knowledge, we are the first to pre-train the multi-agent model with offline datasets and fine-tune it under the sho-shot and zero-shot settings. Future research will pre-train a more general policy, which can transfer across more diverse tasks.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Ana LC Bazzan, Opportunities for multiagent systems and multiagent reinforcement learning in traffic control. Autonomous Agents and Multi-Agent Systems, 18(3)3342. 2009.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint
2021.
Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformer:: no-recurrence sequence-to-sequenee model for speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5884-58888 5888. IEEE, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning. pp. 1407-1416. PMLR, 2018.
Lasse Espeholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski. Seed rl: Scalable and efficient deep-rl with accelerated central inference. In International Conference on Learning Representations, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020. 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, 2052-2062. PMLR, 2019.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05900. 2018.
Kai Han, Yunhe Wang. Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. survey on visual transformer. arXiv preprint arXiv:2012. 12556, 2020.
Michael Janner, Qiyang Li, and Sergey Levine. Reinforeement learning as one big sequence modeling problem. arXiv preprint arXiv:2106. 02039, 2021.
Jiechuan Jiang and Zongqing Lu. Offline decentralized multi-agent reinforcement learning, arXiv preprint 1=Xiv:2108.018.. 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32:11784-11777 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative g-learning for offline reinforcement learning. arXiv preprint arXiv:2006. 20t.:20000007779, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021.
10

---

Under review as a conference paper at ICLR 2022
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. Advances in Neural Information Processing Systems, 32:7613-7624, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 (17754)) 2015.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations, In Thirty-second AAAI conference on artificial intelligence, 2018.
Remi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off off-policy reinforcement learning. Advances Neural Information Processing Systems, 29:1054-1066 1062, 2016.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09333 2020.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 4295 4304. PMLR, 2018.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein- forcement learning and imitation learning: A tale of pessimism. arXiv preprint arXiv: 12021, 2021.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung. Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: 0tti::9022: 1902.04033 2019.
Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv: 2016.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Farl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 5887-5896. PMLR, 2019.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, et al. salooseeconoositioon networks for cooperative multi-agent learning. arXiv preprint arXiv: / 000. 2017.
Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 12, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.000
2017.
Qiang Wang, Bei Li. Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. arXiv preprint arXiv: 997600/777, 2019.
Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.0152. 2020.
Ronald J Williams. Simple statistical gradient-followin algorithms for connectionist reinforcement learning. Machine learning, 8(3):229 256, 1992.
Yikun Xian, Zuohui Fu, S Muthukrishnan. Gerard De Melo, and Yongfeng Zhang. Reinforcement knowledge graph reasoning for explainable recommendation. In Proceedings of the 42nd Inter- national ACM SIGIR Conference Research and Development in Information Retrieval, pp. 285-294, 2019.
Yigin Yang, Xiaoteng Ma. Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. arXiv preprint 10X1v:2100.03400, 2021.

---

Under review as a conference paper at ICLR 2022
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of mappo in cooperative, multi agent games. arXiv preprint arXiw:2103.0195,, 2021.
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pp. 321-384, 2021.
12

---

Under review as a conference paper at ICLR 2022
A PROPERTIES OF DATASETS
We list the properties of our offline datasets Tables and 3.
Reward Distribution
Maps
Difficulty
Data Quality
# Samples
(mean (#std))
3m-poor
62528
6.29 (+2.17)
3m
Easy
3m-medium
3m-good
1775159
19.99 (+0.18)
8m-poor
157133
6.43 (+2.411
8m
Easy
8m-medium
297439
11.95 (+0.94)
8m-good
2781145
20.00 (=0.16)
2s3z-poor
147314
7.69 (+1.77)
2s3z
Easy
2s3z-medium
441474
12.85 (=1.37)
2s3z-good
4177846
19.93 (=0.67)
2s_vs_lsc-poon
12887
6.62 (+22.74)
2s_ vs Ise
Easy
2s vs_ _s__sccmedium
33232
11.70 (=0.73)
2s vs_lsc-good
1972972
20.23 (=0.02)
3s vs_4z-poor
216499
7.58 (#1.45)
38 vs_______ 4z
Easy
vs 3s_vs__2_mmss 4z-medium
335580
12.13 (=1.38)
38 vs. 4z-good
3080634
20.19 (=0.40)
MMM-poor
326516
7.64 (#2.05)
MMM
Easy
MMM-medium
648115
12.23 (#1.37)
MMM-good
2423605
20.08 (=1.67)
so ma_may__aa Sirany_baneling-ae
1542
9.08 (+0.66)
so manm baneling
Easy
so many mediuu
59659
13.31 (=1.14)
so many baneling-gow good
1376861
19,46 (=1.29)
3s 3z-poor
52807
8.10 (#1.37)
3s_v8_3z
Easy
38 vs 32-medium
80948
11.87 (=1.19)
38_v8_37e vx__oooo
149906
20.02 (+0.09)
2m vs. 1z-poor
25333
5.20 (#1.66)
2m 1z
Easy
2m vs Iz-medium
300
11.00 (=0.01)
2m vs _s__vss__oo_o__ee Iz-good
120127
20.00 (+0.00)
bane vs bssbbane-por
63
1.59 (#3.56)
bane bane
Easy
hane_ vs. bene-medium
3507
14.00 (+0093)
hane_vs vs www.-e-_aaneoodddd
458795
19.97 (+0.36)
Ic3s5z-poor
52988
8.10 (.66.)))
1c3s5z
Easy
Ic3s57-medium
180357
12.68 (=: 1 1.42)
1c3s5z-good
2400033
19.88 (=0.69)
Table 2: The properties for our offline dataset collected from the experience of multi-agent PPO on the easy maps of SMAC.
13

---

Under review as a conference paper at ICLR 2022
Reward Distribution
Maps
Difficulty
Data Quality
Samples
(mean (#
5m vs_6m-poor
1324213
8.53 (#1.18)
5m_vs 3m_vs_6m
Hard
5m vs_ 6mm-mmmiimmm
657520
11.03 (#0.58)
5m vs_ "s_6m-good
503746
20 (#0.01)
10m vs 1ssllmmppppp 1mmpoor
140522
7.64 (+2.39)
10m_ son_vs__uu vs_ vim
Hard
10m mmm-mmiumm
916845
12.72 (#1.25)
10m_vs_ll we____lllmoddddd
895609
20 (+:0.0)
2c ss_642g-poor
10830
8.91 (#11.0)
_c_vs_ ys_64zg
Hard
2c_vs vs 64zg-medium
97702
13.05 (111.37)
_c_vs. _a__v__eee 2e__s_647gg-g00 647g-good
2631121
19.95 (+11244
_m_v
vs vs_9m-poor
184285
8.18 (#2.14)
8m_vv__0
vs 9s_
Hard
8m vs 9m-medium
743198
12.19 (#1.14)
8m vs >s_9--god
911652
20 (=0.01)
3s_vs vs_5z-ppor
423780
6.85 (+2.00)
3s 3s_vs__z 5z
Hard
vs _s_vs_ vs_5z-medium
686570
12.12 (+1.39)
3s vs__s-gooood
2604082
20.89 (#1.38)
3s57-poor
365389
8.32 (+1.44)
3s5z
Hard
3s5z-medium
2047601
12.61 (#1.32)
3s5z-good
1448424
18.45 (12203)
155z_vs_3s62-po0
594089
7.92 (#1.77)
3s5z_ 25_s__s_3s62
Super Hard
3s5z_vs_3s66e 333_2m_-mmmmm
2214201
12.56 (#1137)
3s5z_vs_3s66 _s___62-o3oooodd
1542571
18.35 (L2.04)
27m vs__mmmmo0orrrrr
102003
7.18 (+2.08)
27m _s_ vs_30m
Super Hard
27m_vs 1rm__ss30m-mmdium
456971
13.19 (+1.25)
27m mim_vs_30m-goo
412941
17.33 (+1.97)
MMM2-poor
1017332
7.87 (#1.74)
MMM2
Super Hard
MMM2-medium
1117508
11.79 (+1.28)
MMM2-good
541873
18.64 (#1.47)
corridor-poor
362553
4.91 (#1.71)
corridor
Super Hard
corridor-medium
439505
13.00 (+1.32)
corridor-good
3163243
19.88 (+0.99)
Table The properties for our offline dataset collected from the experience of multi-agent PPO on the hard and super hard maps of SMAC.
14

---

Under review as a conference paper at ICLR 2022
B DETAILS OF HYPER-PARAMETERS
Details of haper-parameters used for MADT experiments are listed from Table to 8.
Hyper-parameter
Value
Hyper-parameter
Value
Hyper-parameter
Value
offline_train______ train_eritic
True
_im_tieettep
400
eval_epochy
32
n layer
2
n head
n_embd
32
online_ buffee___ffi_ii
64
model_type
state_only
hini_batch_size
128
Table 4: Common hyper-parameters for all MADT experiments
Maps
affline_episode. num
offline_Ir _r
2s3z
1000
le-4
3s5z
1000
le-4
3s5z 3s6z
1000
5e-4
corridor
1000
5e-4
Table 5: Hyper-parameters for MADT experiments in Figure 3
Maps
offline episode num
offline Ir
online
online _pt_pppppppeeeeeeeeppeeeeppppepp epochs
_c_vs s__v___444 rs_6
1000
5c-4
5e-4
10
10m_ws vs_ 1sm
1000
5e-4
5e-4
10
8m vs 9m
1000
le-4
5e-4
10
3s_vs_5z
1000
le-4
5e-4
3s5z
1000
le-4
5e-4
10
3m
1000
le-4
5c-4
15
2s _s_lsc
1000
le-4
5e-4
15
MMM
1000
le-4
le-4
so_many bancling
1000
le-4
le-4
8m
1000
le-4
lc-4
38 vs 3z
1000
le-4
le-4
3s vs 4z
1000
le-4
le-4
bane vs. bane
1000
le-4
lc-4
2m vs lz
1000
le-4
le-4
2c vs 2__4
1000
le-4
le-4
5m_ys_6n
1000
le-4
le-4
10
corridor
1000
le-4
le-4
10
3s5z, vs_ _
1000
lc-4
lc-4
10
Table 6: Hyper-parameters for MADT experiments in Figure 4, 6 and Table
15

---

Under review as a conference paper at ICLR 2022
Hyper-parameter
value
offline map_misp lists
[3s vz_4z, 2m vs_ 1z, 3m, _s_vs_
vs_ Isc, 3s vs_32]
dilline_episode_mi
[200, 200, 200, 200, 200]
offline_Ir
5e-4
online_Ir
le-4
online_ppo_epochs
5
Table 7:
for MADT experiments in Figure 5a
Hyper-parameter
value
lists
[2m_vs, vs_ lz, 3m, _s_vs___________________________ Isc, 33]
offline_episode_nmm
[250, 250, 250, 250]
offline Ir
5e-4
online_Ir
le-4
ooline__poo_ppccs
5
Table 8: Hyper-parametern for MADT experiments in Figure 5b
16


Reviews:
[{"review_id": "vlHIZnpXEMs", "reviewer": "Reviewer_3RgR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n\n- The offline trainig and online tuning setting for multi-agent tasks is new and very interesting. \n- Authors show the promise of using a state-of-the-art architecture from the CV or NLP field to RL. \n- The paper is also well-written and easy to follow, with a clear introduction of the backgrounds, methods, and experiments.\n- Authors conduct extensive experiments in SMAC, and compare against recent state-of-the-art methods including BCQ, CQL, and ICQ. MADT outperforms baselines significantly.\n\nCons: \n- I'm concerned about the appropriateness of MADT-Online based on MAPPO. Although it is in the online fine-tuning stage, from Algorithm 2, MAPPO still learns from the offline dataset D. However, since MAPPO is an on-policy method, I'm concerned about whether it is correct for directly applying an on-policy method without any correction technique?\n- Compared to baselines including BCQ, CQL, and ICQ, using the transformer architecture can incur more computation costs. \n", "summary_of_the_paper": "The paper studies the offline training and online fine-tuning setting for MARL, and show that it is promising to learn policy representatrions to improve the performance on downstream MARL tasks. The paper proposes to integrate the transformer architecture to improve efficiency and generalization ability. Authors conduct extensive experiments to validate its effectiveness.", "summary_of_the_review": "The setting is very appealing, and the paper is clearly written, but I'm concerned about the MADT-Online method (see above).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "RN9aUTl-AT9", "reviewer": "Reviewer_yY9Q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Use Case: To me the biggest strength of this paper is the use case, it would be very practical for many multi-agent settings to consider efficient deployment in a new environment after pre-training on offline data. I also think that the data set provided is a significant contribution to the community to foster more research in this area. That said, the value of this data is somewhat diminished by being from a trained neural network model rather than humans. It possibly could be interesting to at least add behaviors from different models and different architectures to provide more variety to the behavior policy of each agent. Indeed, optimization in the context of changing behavior policies for each agent is one of the main challenges of multi-agent RL due to the non-stationarity it introduces for each agent, which this paper does not really address or discuss.  \n\nMADT Architecture: it is a good idea to apply transformers as an emerging architecture choice for RL problems and it does seem to improve generalization capabilities for the agents. However, it is unclear how these innovations tie to the prior literature and concepts in MARL and offline RL in particular. For example see these surveys \"Offline Reinforcement Learning: Tutorial, Review,and Perspectives on Open Problems\" Levine et al., 2020 and \"A Survey and Critique of Multiagent Deep Reinforcement Learning\" Hernandez-Leal et al., 2019. In particular I would have expected to see more discussion of topics like off-policy correction, importance sampling, and over-estimation bias. The imitation learning style objective implemented in this work is not justified in light of this theory of offline RL and clearly is very restrictive in terms of needing a consistently high quality behavior policy to get beneficial results during pre-training. Moreover, these off-policy correction and distributional miss-match problems become even more severe in multi-agent environments as each agent's policy is critical to the environment evolution and as such policies cannot be simply considered in isolation. The proposed approach was not clearly related to the multi-agent literature at all and popular approaches were not compared to even on a theoretical/intuitive level. \n\nExperiments: Another selling point of the paper is that the proposed method does well in comparison to popular approaches for single agent offline RL. However, there is very little theoretical understanding of where these performance gains come from. It seems conceivable that the transformer model alone is what is improving performance and that the way offline training and fine-tuning is done add little to performance. Indeed, transformer models have recently been demonstrated to have advantages for RL in general, so it is unclear if there is any value in the context of this multi-agent use case in particular. It would be very nice to see other proposals for offline training using this transformer architecture as well and validate comparisons to approaches from the multi-agent literature as opposed to only considering single agent approaches applied to multi-agent problems. \n\nMinor: there were a number of typos and grammatical issues throughout the paper.\n", "summary_of_the_paper": "This is the first paper to pre-train a multi-agent model with offline datasets and online fine-tune it to a particular task. The authors propose MADT an offline pre-training architecture for multi-agent RL integrating transformers with imitation learning on an a collected data set of offline data. Towards this end, the authors contribute a data set of offline multi-agent interactions collected from running multi-agent PPO on 5 tasks from SMAC. On this data set, the proposed MADT method is shown to outperform popular single agent offline RL algorithms from the literature and excel at fine-tuning a pre-trained offline model. \n", "summary_of_the_review": "I really liked a lot about this paper, including the cool and quite practical use case that they consider for the first time. Another strong point is that they provide a data set to jump start research in this area, although its overall value to the community is probably not that high as it is from a trained model and not humans. It is also interesting that the authors show how transformer models can provide value in this setting. The trouble is just that it is unclear how the contributions of this paper are related to the literature on MARL and offline RL, making it necessary to provide more analysis to understand whether these gains are connected to the particular setting considered here or rather hold in general. It is great that this approach can improve on popular algorithms for offline single agent RL, but no algorithms from the multi-agent literature were compared to and there is very little theoretical understanding about the value of particular design choices within the algorithm.  As a result, I lean towards rejection at this time believing that this paper can make a much larger impact to the community following another significant round of revisions.   ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L6wDHYA9-Kx", "reviewer": "Reviewer_mKDU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Novelty:\nThis paper is not novel as it seems to be a straightforward combination of transformer and MAPPO. \n\nPresentation:\nThe charity of the paper needs to be improved especially in experiments. As the main point of the paper is the pre-trained transformer, it is not necessary to spend much space to discuss the experimental results of MADT in the offline RL setting. It would better to discuss the challenge of the setting of offline pre-training with online fine-tuning.\n\nSignificance:\nIt is obvious that using transformer improves the ability of representation, and the results are not surprising. Research papers such as BREMEN and MUSBO from the deployment constrained RL setting are not discussed.", "summary_of_the_paper": "This paper proposes a multi-agent decision transformer to improve the learning performance in the setting of offline pre-training with online fine-tuning. Experimental results show that it outperforms offline RL algorithms, BCQ, CQL and ICQ in multiple Starcraft II games.", "summary_of_the_review": "This paper studies a new problem in MARL. However, its main algorithm is not novel and the presentation is not clear. Experimental evaluations are not sufficient, the comparison of MADT and BREMEN is needed. As BREMEN also studies the problem of online tuning RL with offline training. Thus I recommend the rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2TxL-2xd2u-", "reviewer": "Reviewer_nCt4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well constructed and the main idea is clear. The experimental evaluation is adequate. I really appreciate the efforts in building the offline SMAC dataset.\n\nThe contribution and novelty of this paper, however, are limited. Pre-training the RL model from demonstration is a common technique especially in complex tasks. It is trivial to transfer this technique into the multi-agent field, and MADT does not improve this paradigm for the characteristics in multi-agent environments. The authors stress several times that MADT is the first to pre-train the multi-agent model with offline datasets. I suggest toning down the claim.\n\nTransformer has strong representation ability, however, do you think the action distribution is so complex that MLP model cannot deal with it? In BCQ and CQL, MLP model is able to fit the behavior action distribution. \n\nI notices that PPO is trained from the merged offline and online dataset during the online interaction from Figure 1. If so, the training of PPO is incorrect. PPO is an on-policy algorithm, which requires that the distribution of the experiences for training could be far away from the distribution under the policy. However, there is a large distribution shift between the offline dataset and the online dataset. \n\nFor experiments, since MADT fits the action distribution by supervised learning, what is the difference between BC and MADT? I feel confused about the meaning of the X-axis (Timesteps) in Figure 3. I understand that xx-MA means mixing the individual Q values to the joint Q value by a QMIX-style network. However, this part should be elaborated.\n\nI suggest removing section 3.3 to the Appendix since it describes the details about SMAC rather than the proposed method.\n", "summary_of_the_paper": "The authors propose MADT which adopts the attention-based model to pre-train the agent policies from the offline dataset and fine-tune the policies by online interaction. The authors also provide an offline SMAC dataset with different skill-level.", "summary_of_the_review": "The contribution is limited and the main idea has been observed in existing work. The current version cannot reach the borderline of ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have raised relevant concerns that preclude acceptance and the authors have not provided a response. At this time, all reviewers concur that this paper should be rejected and I agree.
Decision: Reject