Year: 2021

Abstract:
Offline reinforcement learning leverages static datasets to learn optimal policies with no necessity to access the environment. This is desirable for multi-agent systems due to the expensiveness of agents’ online interactions and the demand for sample numbers. Yet, in multi-agent reinforcement learning (MARL), the paradigm of offline pre-training with online fine-tuning has never been reported, nor datasets or benchmarks for offline MARL research are available. In this paper, we intend to investigate whether offline training is able to learn policy representations that elevate performance on downstream MARL tasks. We introduce the first offline dataset based on StarCraftII with diverse quality levels and propose a multi-agent decision transformer (MADT) for effective offline learning. MADT integrates the powerful temporal representation learning ability of Transformer into both offline and online multi-agent learning, which promotes generalisation across agents and scenarios. The proposed method demonstrates superior performance than the state-of-the-art algorithms in offline MARL. Furthermore, when applied to online tasks, the pre-trained MADT largely improves sample efficiency, even in zero-shot task transfer. To our best knowledge, this is the first work to demonstrate the effectiveness of pre-trained models in terms of sample efficiency and generalisability enhancement in MARL.

heading: 1 INRTODUCTION
text: Multi-Agent reinforcement learning (MARL) plays an essential role for solving complex decisionmaking tasks by learning from the interaction data between machine or autonomous agents and simulated physical environments. It typically applies to self-driving (Shalev-Shwartz et al., 2016), traffic control (Bazzan, 2009), and recommendation system (Xian et al., 2019), and, to specifically mention, surpasses human beings on some video games (Bellemare et al., 2013; Wang et al., 2020). However, the experience-based policy learning scheme requires the algorithms with high sample efficiency because of the limited computing resources and high cost resulting from the data collection (Haarnoja et al., 2018; Munos et al., 2016; Espeholt et al., 2019; 2018). Furthermore, even in domains where the online environment is feasible, we might still prefer to utilize previously collected data instead, for example, if the domain’s complex requires large datasets for effective generalization. In addition, a policy trained on one scenario usually cannot perform well on another even though under the same task. Therefore, a universal policy is critical for saving the training time of general reinforcement learning. Offline RL (Levine et al., 2020) attracts more researchers motivated by improving online reinforcement learning with offline datasets (Fu et al., 2020). Those methods like CQL (Kumar et al., 2020), BEAR (Kumar et al., 2019), and BCQ (Fujimoto et al., 2019) conventionally leverage the current off-policy RL algorithms, e.g., DQN (Mnih et al., 2015), with provable constraints (Rashidinejad et al., 2021). They mainly focus on solving the overestimation of Q values learned in offline datasets and the distribution shift between offline and online. However, those constraints limit the performance with insufficiently offline dataset exploitation (Kumar et al., 2019). Recently, the transformer shows its advantage of representing the sequential data for the classification or prediction tasks. Most related to our work, Chen et al. (2021) replace the constrained Q networks in conventional offline RL algorithms with causal transformer and demonstrate the superiority comparing with baselines such as Behavior Cloning (BC) and state-of-the-art offline algorithms in single-agent offline RL. Nevertheless, the overestimation problems emerge in the offline multi-agent RL when one directly applies the sequential modeling method. We propose leveraging
the representation capability of the transformer and then load the pre-trained model as part of the policy in the fine-tuning phase.
This paper proposes Multi-Agent Decision Transformers (MADT) for pre-training the general policy on offline datasets, capable of generalizing onto other seen/unseen environments. Firstly, we collect the offline datasets from the well-known, challenging MARL environment, StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019), interacting with the state-of-the-art algorithm, multiagent PPO (Yu et al., 2021). Then we construct transformer variants for multiple agents with parameter sharing or acing as a part. We give the paradigm of pre-training a model with the transformer and fine-tuning it with MARL. We conclude the main contribution as follows:
• We propose a series of transformer variants for offline MARL via leveraging the sequential modeling of the attention mechanism.
• We derive a framework with offline pretraining and online fine-tuning to improve sample efficiency and generality across different scenarios.
• We are the first to apply the pre-trained model across different multi-agent scenarios to validate the transferring ability. Experimental results show fast adaptation and superior performance.
• We build a dataset with different skill-level covering all SMAC scenarios for benchmarking progress in MARL pre-training.

heading: 2 PRELIMINARY
text: Multi-Agent Reinforcement Learning. For the Markov Game, which is a multi-agent extension from the Markov Decision Process (MDP), there is a tuple representing the essential elements < S,A, R, P, n, γ >, where S denotes the state space of n agents : S1 × S2 . . . Sn → S. Ai is the action space of each agent i, P : Si × Ai → PD(Si) denotes the transition function emiting the distribution over the state space and A is the joint action space, Ri : S × Ai → R is the reward function of each agent and takes action following their policies π(a|s) ∈ Πi : S→ PD(A) from the policy space Πi, where Πi denotes the policy space of agent i, a ∼ Ai, and s ∼ Si. Each agent aims to maximize its long-term reward ∑ t γ
trt, where rti ∈ Ri denotes the reward of agent i in time t and γ denotes the discount factor.
Attention-based Model. Attention-based model has shown its stable and strong representation capability. The scale dot-production attention uses the self-attention mechanism demonstrated in (Vaswani et al., 2017). Let Q ∈ Rtq×dq be the quries, K ∈ Rtk×dk be the keys, and V ∈ Rtv×dv be the values, where t∗ are the element numbers of different inputs and d∗ are the corresponding element dimensions. Normally, tk = tv and dq = dk. The outputs of self-attention are computed as:
Attention(Q,K,V) = softmax( QKT√ dk )V (1)
where the scalar 1/ √ dk is used to prevent the softmax function into regions that has very small gradients. Then we introduce the multi-head attention process as follows:
MultiHead(Q,K,V) = Concat(head1, . . . , headh)WO (2)
headi = Attention(QW Q i ,KW K i ,VW V i ) (3)
The position-wise feed-forward network is another core module of the transformer. It consists of two linear transformations with a ReLU activation in between. The dimensionality of inputs and outputs is dmodel, and that of the feef forward layer is dff . Specially,
FFN(x) = max(0, xW1 + b1)W2 + b2 (4)
where W1 ∈ Rdmodel×dff and W2 ∈ Rdff×dmodelare the weights, and b1 ∈ Rdff and b2 ∈ Rdmodel are the biases. Across different positions are the same linear transformations. Note that the position encoding for leveraging the order of the sequence as follows:
PE(pos, 2i) = sin pos/100002i/dmodel
PE(pos, 2i+ 1) = cos pos/100002i/dmodel (5)

heading: 3 METHODOLOGY
text: In this section, we demonstrate how the transformer is applied to our offline pre-training MARL framework. Firstly, we introduce an offline MARL method, in which the transformer maps between the local observations and actions of each agent in the offline dataset via parameter sharing sequentially. Then we leverage the hidden representation as the input of the multi-agent decision transformer (MADT) to minimize the cross-entropy loss. Furthermore, we introduce how to integrate the online MARL with MADT in constructing our whole framework to train a universal MARL policy. To accelerate the online learning, we load the pre-trained model as a part of MARL algorithms and learn the policy based on experience in the latest buffer stored from the online environment. To train a universal MARL policy quickly adapting to other tasks, we bridge the gap between different scenarios from observations, actions, and available actions, respectively. Figure 1 overviews our method from the perspective of offline pre-training with supervised learning and online fine-tuning with MARL algorithms.

heading: 3.1 MULTI-AGENT DECISION TRANSFORMER
text: Algorithm 1 shows the training process of Multi-Agent Decision Transformer (MADT), in which we autoregressively encode the trajectories from the offline datasets in offline pre-trained MARL and train the transformer-based network with supervised learning. We carefully reformulate the trajectories as the inputs of the causal transformer different from those in the Decision Transformer (Chen et al., 2021). Denote that we deprecate the reward-to-go and actions that are encoded with states together in the single-agent DT. We will interpret the reason for this in the next section. Similar to the seq2seq models, MADT is based on the autoregressive architecture with the reformulated sequential inputs across timescale. The left part of Figure 2 shows the architecture. The causal transformer encodes the agent i’s trajectory sequence τ ti at the time step t to a hidden representation h t i = (h1, h2, . . . , hl) with a dynamic mask. Given ht, the output at the time step t only based on the previous data then consumes the previously emitted actions as additional inputs when predicting a new action.
Algorithm 1: MADT-Offline: Multi-Agent Decision Transformer Input: Offline dataset D : {τi : 〈sti, oti, ati, vti , dti, rti〉Tt=1}ni=1, vti denotes the available action Initialize θ for the Causal Transformer; Initialize α as the learning rate, C as the context length, and n as the maximize agent number for τ = {τ1, . . . , τi, . . . , τn} in D do
Chunk the trajectory into τi = {rti , sti, ati}t∈1:C as the ground truth samples, where C is the context length, and mask the trajectory when dti is true for τ ti = {τ1i . . . τ ti . . . τTi } in τi do
Mask illegal actions via P (aj |τi, â<j ; θ ′ ) = 0 if vti is True Predict the action ât = arg maxa P (a|τi, â<t; θ ′ )
Update θ with θ = arg max θ′ 1 C
∑C j=1 P (aj) logP (âj |τi, â<j ; θ ′ )
end end return θ
Trajectories Reformulation as Input. We model the lowest granularity at each time step as a modeling unit xt from the static offline dataset for the concise representation. MARL has many
elements such as 〈global_state, local_observation〉, different from the single agent. It is reasonable for sequential modeling methods to model them in Markov Decision Process (MDP). Therefore, we formulate the trajectory as follows:
τ i = (x1, . . . xt . . . , xT ) where xt = (st, oit, a i t)
where sit denotes the global shared state, o i t denotes the individual observation for agent i at time step
t, and ait denotes the action. We take xt as a token similar to the input of natural language processing.
Output Sequence Constructing. To bridge the gap between training with the whole context trajectory and testing with only previous data, we mask the context data to autoregressively output in the time step t with previous data in 〈1 . . . t− 1〉. Therefore, MADT predicts the sequential actions at each time step using the decoder as follows:
y = at = arg max a
pθ(a|τ, a1, . . . , at−1) (6)
where θ denotes the parameter of MADT and τ denotes the trajectory including the global state s, local observation o before the time step t, pθ is the distribution over the legal action space under the available action v.
Core Module Description. MADT differs from the transformers in conventional sequence modeling tasks that take inputs with position encoding and decode the encoded hidden representation autoregressively. We use the masking mechanism with a lower triangular matrix to compute the attention:
Attention(Q,K,V) = softmax( QKT√ dk +M)V (7)
where M is the mask matrix which ensures the input at the time step t can only correlate with the input from 〈1, . . . , t− 1〉. We employ the Cross-entropy (CE) as the total sequential prediction loss and utilize the available action v to ensure agents taking those illegal actions with probability zero. The CE loss can be represented as follows:
LCE(θ) = 1
C C∑ t=1 P (at) logP (ât|τt, â<t; θ), where C is the context length (8)
where at is the ground truth action, τt includes {s1:t, o1:t}. â denotes the output of MADT. The cross-entropy loss shown above aims to minimize the distribution distance between the prediction and the ground truth.

heading: 3.2 MULTI-AGENT DECISION TRANSFORMER WITH PPO
text: The method above can fit the data distribution well resulting from the sequential modeling capacity of the transformer. But it fails to work well when pre-training on the offline datasets and improving continually by interacting with the online environment. The reason originates from the mismatch between the objective in MADT and ignore the value of each action. When the pre-trained model
is loaded to interact with the online environment, the buffer will only collect actions conforming to the distributions of the offline datasets rather than those corresponding to high reward at this state. That means the pre-trained policy is regarded as good once the selected action is identical to that in the dataset, even though it leads to a low reward. Therefore, we need to design another paradigm, MADT-PPO, to integrate RL and supervised learning for fine-tuning continually in Algorithm 2. Figure 2 shows the pre-training and fine-tuning framework. A direct method is to share the pre-trained model across each agent and implement the REINFORCE algorithm (Williams, 1992). However, only actors result in higher variance, and the employment of a critic to assess state values is necessary. Therefore, in online MARL, we leverage an extension of PPO, the state-of-the-art algorithm on tasks of StarCraft, MPE, and even return-based game Hanabi (Mordatch & Abbeel, 2018).
Algorithm 2: MADT-Online: Multi-Agent Decision Transformer with PPO Input: Offline dataloader D, Pretrained MADT Policy with parameter θ Initialize θ and φ are the parameters of an actor πθ(ai|oi) and critic Vφ(s) respectively, which could be inherited directly from pre-trained models. Initialize n as the agent number, γ as the discount factor, and as clip ratio for τ = {τ1, . . . , τi, . . . , τn} in D do
Sample τi = {st, oti, ati}t∈1:T as the ground truth, where T is the context length Compute the advantage function A(s, ai) = ∑ t γ
tr(s, ai)− Vφ(s) Computing the important weight w = πθ(oi, ai)/πθold(oi, ai) Update θi for i ∈ 1 . . . n via:
θi = arg maxEs∼ρθold ,a∼πθold [clip(w, 1− , 1 + )A(s, ai)] Compute the MSE loss Lφ = 12 [ ∑ t γ
tr − Vφ(s)]2 Update the critic network via φ = arg min
φ Lφ
end return θ, φ

heading: 3.3 UNIVERSAL MODEL ACROSS SCENARIOS
text: To train a universal policy for each of the scenarios in the SMAC which might be vary in agent number, feature space, action space and reward ranges, we consider the modification list below.
Parameters Sharing Across Agents. When offline examples are collected from multiple tasks or the test phase owns the different agent numbers from the offline datasets, the difference of agent number across tasks is an intractable problem for deciding the number of actors. Thus, we consider sharing the parameters across all actors with one model as well as attaching one-hot agent IDs into observations, for compatibility with a variable number of agents.
Feature Encoding. When the policy needs to generalize to new scenarios that arises from different feature shapes, We propose encoding all features into a universal space by padding zero at the end and mapping to a low-dimensional space with fully connected networks.
Action Masking. Another issue is the different action space across scenarios. For example, less enemies in a scenario means less potential attack options as well as available actions. Therefore, an extra vector is utilized to mute the unavailable actions so that their probabilities are always zero during both learning and evaluating process.
Reward Scaling. Different scenarios might be vary in reward ranges and lead to unbalanced models during multi-task offline learning. To balance the influence of examples from different scenarios, we scale their rewards into the same range to ensure the output models have comparable performance between different tasks.

heading: 4 EXPERIMENTS
text: We show three experimental settings: offline MARL, online MARL by loading the pre-trained model, and few-shot or zero-shot offline learning. For the offline MARL, we expect to verify the performance of our method in pre-training the policy and directly testing on the corresponding maps. For the fine-tuning in the online environment, we aim to demonstrate the capacity of the pre-trained policy on the original or new scenarios. Experimental results in offline MARL shows our MADT-offline
in 3.1 outperforms the state-of-the-art methods. Furthermore, MADT-online in 3.2 can improve the sample efficiency across multiple scenarios. Besides, the universal MADT trained from multi-task data with MADT-online generalize well in each scenario in few-shot even zero-shot setting.

heading: 4.1 OFFLINE DATASETS
text: The offline datasets are collected from the running policy, MAPPO (Yu et al., 2021), on the wellknown SMAC task (Samvelyan et al., 2019). Each dataset contains a large number of trajectories: τ := (st, ot, at, rt, donet, vt) T t=1. Different from D4RL (Fu et al., 2020), our datasets consider the property of DecPOMDP, which owns local observations and available actions for each agent. In Appendix, we list the statistical properties of the offline datasets in Table 2 and Table 3.

heading: 4.2 OFFLINE MULTI-AGENT REINFORCEMENT LEARNING
text: In this experiment, we aim to validate the effectiveness of MADT offline version in Section 3.1 as a framework for offline MARL on the static offline datasets. We train a policy on the offline datasets with various qualities and then apply it to an online environment, StarCraft (Samvelyan et al., 2019). There are also baselines under this setting, such as Behavior Cloning (BC) as a kind of imitation learning method showing stable performance on single-agent offline RL. Besides, we employ the conventional effective single-agent offline RL algorithms, BCQ (Fujimoto et al., 2019), CQL (Kumar et al., 2020), and ICQ (Yang et al., 2021), then add the mixing network for multi-agent setting, denoting as “xx-MA”. To verify the quality of our collected datasets, we choose data from different level and train the baselines as well as our MADT. Figure 3 shows the overall performance on various quality datasets. The baseline methods enhance their performance stably, indicating the quality of our offline datasets. The results also show that our MADT outperforms the offline MARL baselines and converges faster across easy, hard, and super hard maps (2s3z, 3s5z, 3s5z_vs_3s6z, corridor).

heading: 4.3 OFFLINE PRE-TRAINING AND ONLINE FINE-TUNING
text: The experiment designed in this subsection intends to answer the question: Is the pre-training process is necessary for online MARL? Firstly we compare the online version of MADT in Section 3.2 with and without loading the pre-trained model. If training MADT only by online experience, we can view it as a transformer-based MAPPO replacing the actor and critic backbone networks with the
transformer. Furthermore, we validate that our framework MADT with the pre-trained model can improve sample efficiency on most easy, hard, and super hard maps.
Necessity of Pretrained Model. We train our MADT based on the datasets collected from a map and fine-tune it on the same map online with the MAPPO algorithm. For comparison fairness, we use the transformer as both actor and critic networks with and without the pre-trained model. Primarily, we choose three maps from Easy, Hard, and Super Hard maps to validate the effectiveness of the pre-trained model in Figure 4. Experimental results show that the pre-trained model converges faster than the algorithm trained from scratch, especially in the challenging maps. Improving Sample Efficiency. For validating the improvement of the sample efficiency via loading our pre-trained MADT and fine-tuning it with MAPPO, we compare the overall framework with the state-of-the-art algorithm, MAPPO (Yu et al., 2021), without the pre-training phase. We measure the number of interactions with the environment in Table 1, and our pre-trained model needs much less than the traditional MAPPO to access the same win rate.
Maps # Samples to Acess the Win Rate Maps # Samples to Acess the Win Rate
20% 40% 60% 80% 100% 20% 40% 60% 80% 100%
2m vs. 1z (Easy) 8e4/- 1e5/- 1.3e5/- 1.5e5/- 1.6e5/- 3s vs. 5z (Hard) 8e5/- 8.5e5/- 8.7e5/1.5e4 9e5/5e4 2e6/1.5e5
3m (Easy) 3.2e3/- 8.3e4/- 3.2e5/- 4e5/- 7.2e5/- 2c vs. 64 zg (Hard) 2e5/- 3e5/- 4e5/8e4 5e5/1e5 1.8e6/5e5
2s vs 1sc (Easy) 1e4/- 2.5e4/- 3e4/- 8e4/4e4 3e5/1.2e5 8m vs. 9m (Hard) 3e5/- 6e5/- 1.4e6/2e4 2e6/8e4 ∞/2.2e6
3s vs. 3z (Easy) 2.5e5/- 3e5/- 6.2e5/1e4 7.3e5/1.5e5 8e5/2.9e5 5m vs. 6m (Hard) 1.5e6/2e5 2.5e6/8e5 5e6/2e6 ∞/∞ ∞/∞
3s vs. 4z (Easy) 3e5/- 4e5/- 5e5/- 6.2e5/- 1.5e6/1.8e5 3s5z (Hard) 8e5/6.3e4 1.3e6/1e5 1.5e6/4e5 1.9e6/1e6 2.5e6/2e6
so many baneling (Easy) 3.2e4/8e3 1e5/4e4 3.2e5/7e4 5e5/8e4 1e6/6.4e5 10m vs. 11m (Hard) 2e5/- 3.5e5/- 4e5/2.8e4 1.7e6/1.2e5 4e6/2.5e5

heading: 4.4 GENERALIZTION WITH MULTI-TASK PRE-TRAINING
text: Experiments in this section are to explore the transferability of the universal MADT mentioned in Section 3.3, which is pre-trained with mixed data from multiple tasks. According to whether the downstream tasks have been seen or not, the few-shot experiments are designed to validate the adaptibilty on seen tasks, while the zero-shot experiments are designed for the held-out maps.
Few-shot Learning. The results in Figure 5a shows that our method can utilize multi-task datasets to train a universal policy and generalize to all tasks well. Pre-trained MADT can achieve higher return than the model trained from scratch when we limit the interactions with environment.
Zero-shot Learning. Figure 5b shows that our universal MADT can surprisingly improve performance on downstream task even if it has not been seen before. (3 stalkers vs. 4 zealots).

heading: 4.5 ABLATION STUDY
text: The experiments in this subsection are designed to answer the following research questions: RQ1: Why we choose MAPPO for online phase? RQ2: Which kind of input shall we use to make the pre-trained model beneficial for the online MARL? RQ3: Why cannot the offline version of MADT be improved in the online fine-tune period after pre-training?
Suitable Online Algorithm. Although the selection of MARL algorithm for online phase should be flexible according to specific task, we design experiments to answer RQ1 here. As discussed in Section 3, we can train Decision Transformer for each agent and online fine-tune it with a MARL algorithm. An intuitive method is to load the pre-trained transformer and take it as the policy network for fine-tuning with the policy gradient method, e.g. REINFORCE (Williams, 1992). However, for the reason of high variance mentioned in Section 3.2, we choose MAPPO as the online algorithm and compare their performance in improving sample efficiency during the online period in Figure 6a.
Dropping Reward-to-go in MADT. To answer RQ2, we compare different inputs embedded into the transformer, including the combination of state, reward-to-go, and action. We find the reward-togo harmful to online fine-tuning performance, as shown in Figure 6b. We suppose the distribution of reward-to-go is mismatch between offline data and online samples. That is, rewards of online samples are usually lower than offline data due to stachastic exploration at the beginning of the online phase. It deteriorates the fine-tuning capability of the pre-trained model, and based on Figure 6b, we only choose states as our inputs for pre-training and fine-tuning.
Integrating Online MARL with MADT. To answer RQ3, we directly apply the offline version of MADT for pre-training and fine-tune it online. However, Figure 6c shows that it cannot be improved during the online phase. We analyze the results from the absence of motivation for chasing higher rewards, and conclude that offline MADT is in fact supervised learning and tends to fit its collected experience even with unsatisfactory reward.

heading: 5 RELATED WORK
text: Offline Deep Reinforcement Learning. Conventionally, offline RL trains a policy based on the static and previously collected data without any environmental interaction (Levine et al., 2020). This policy is then leveraged to interact with the online environment to obtain promising results. A straightforward method for the current reinforcement learning is to use the off-policy algorithm and regard the offline datasets as a replay buffer to learn a policy with good performance. However, experience existing in offline datasets and interaction with online environment have different distributions which cause the overestimation in the off-policy (value-based) method (Kumar et al., 2019). Substantial works presented in offline RL aim at resolving the distribution shift between the static offline datasets and the online environment interaction (Kumar et al., 2020; Fujimoto et al., 2019; Kumar et al., 2019). Yang et al. (2021); Jiang & Lu (2021) constrain off-policy algorithms in offline MARL. Related to our work for the improvement of sample efficiency, Nair et al. (2020) load the pretrained model to generate a weight for actor-critic online RL algorithms. Recently, the Decision Transformer outperforms many state-of-the-art offline RL algorithms via regarding the training process as a sequential modeling phase and test on the online environment (Chen et al., 2021; Janner et al., 2021). In contrast, we show a transformer-based method in the multi-agent field, attempting to transfer across many scenarios without extra constraints.
Multi-Agent Reinforcement Learning. As a natural extension from single-agent RL, MARL attracts much attention to solve more complex problems under Markov Games (Zhang et al., 2021). Related algorithms conventionally enforce multiple agents to interact with the environment online and collect the experience to train the joint policy from scratch (Rashid et al., 2018; Sunehag et al., 2017; Son et al., 2019; Mahajan et al., 2019; Yu et al., 2021). The framework of traditional multiagent learning employs the centralized training and decentralized execution framrwork due to local observation and increasing action space. We build offline datasets by collecting elements from the DecPOMDP and train the general policy with parameter sharing. We utilize transformer to learn a policy with previously collected data from the golden policy in advance and fine-tune it online.
Transformer. Transformer (Vaswani et al., 2017) has achieved a great breakthrough to model relations between the input and output sequence with variable length, for the sequence-to-sequence problems (Sutskever et al., 2014), especially on machine translation (Wang et al., 2019) and speech recognition (Dong et al., 2018). Recent works even reorganize the vision problems as the sequential modeling process and construct the SOTA model with pretraining, named ViT (Han et al., 2020; Dosovitskiy et al., 2020; Liu et al., 2021). Due to the Markovian property of trajectories in offline datasets, we can utilize Transformer as that in language modeling. Therefore, Transformer can bridge the gap between supervised learning in the offline setting and reinforcement learning in the online interaction because of the representation capability. We claim that the components under Markov Games are sequential then utilize the transformer for each agent to fit a transferable MARL policy. Furhtermore, we fine-tune the learned policy under via trial-and-error.

heading: 6 CONCLUSIONS
text: In this work, we propose MADT, an offline pre-trained model for MARL, which integrates the transformer to improve sample efficiency and generalizability. We build an offline MARL dataset based on SMAC. Our method outperforms the state-of-the-art methods in offline MARL, including BC, BCQ, CQL, and ICQ. Our whole framework can also drastically improve the sample efficiency of online MARL via loading pre-trained model. Furthermore, we apply our method to train a universal policy over the multi-task datasets and fine-tune it under the few-shot and zero-shot settings. Results demonstrate that this universal policy adapts fast to new tasks and promises performance on different level maps.
To our best knowledge, we are the first to pre-train the multi-agent model with offline datasets and fine-tune it under the few-shot and zero-shot settings. Future research will pre-train a more general policy, which can transfer across more diverse tasks.

heading: A PROPERTIES OF DATASETS
text: We list the properties of our offline datasets in Tables 2 and 3.

heading: B DETAILS OF HYPER-PARAMETERS
text: Details of hyper-parameters used for MADT experiments are listed from Table 4 to 8.

Review summaries:
The paper studies the offline training and online fine-tuning setting for MARL, and show that it is promising to learn policy representatrions to improve the performance on downstream MARL tasks. The paper proposes to integrate the transformer architecture to improve efficiency and generalization ability. Authors conduct extensive experiments to validate its effectiveness.

This is the first paper to pre-train a multi-agent model with offline datasets and online fine-tune it to a particular task. The authors propose MADT an offline pre-training architecture for multi-agent RL integrating transformers with imitation learning on an a collected data set of offline data. Towards this end, the authors contribute a data set of offline multi-agent interactions collected from running multi-agent PPO on 5 tasks from SMAC. On this data set, the proposed MADT method is shown to outperform popular single agent offline RL algorithms from the literature and excel at fine-tuning a pre-trained offline model.