Offline Multi-Agent Reinforcement Learning with Knowledge Distillation
Authors: Wei-Cheng Tseng
Tsun-Hsuan Wang
Yen-Chen Lin
Phillip Isola
Venue: NeurIPS 2022 Accept (2022)
Keywords: offline multi-agent reinforcement learning
multi-agent
offline reinforcement learning

Abstract: We introduce an offline multi-agent reinforcement learning ( offline MARL) framework that utilizes previously collected data without additional online data collection. Our method reformulates offline MARL as a sequence modeling problem and thus builds on top of the simplicity and scalability of the Transformer architecture. In the fashion of centralized training and decentralized execution, we propose to first train a teacher policy as if the MARL dataset is generated by a single agent. After the teacher policy has identified and recombined the "good" behavior in the dataset, we create separate student policies and distill not only the teacher policy's features but also its structural relations among different agents' features to student policies. Despite its simplicity, the proposed method outperforms state-of-the-art model-free offline MARL baselines while being more robust to demonstration's quality on several environments.

Offline Multi-Agen Reinforcement Learning with Knowledge Distillation
Wei-Cheng Tseng', Tsun-Hsuan Wang", Lin Yen-Chen', PainCCh Isola?
University of Toronto, "MIT CSAIL
weicheng. tseng@mail utoronto.ca. www.:yyeecheel.hhll..pp...
Abstract
We introduce an offline multi-agent reinforcement learning (offline MARL) frame- work that utilizes previously collected data without additional online data collection. Our method reformulates offline MARL a sequence modeling problem and thus builds on top of the simplicity and scalability of the Transformer architecture. the fashion of centralized training and decentralized execution, we propose to first train a teacher policy who has the privilege to access every agent's 's observations, actions, and rewards. After the teacher policy has identified and recombined the "good" behavior the dataset, we create separate student policies and distill not only the teacher policy's 's features but also its structural relations among different agents' features to student policies. We show that our framework significantly improves performances
range of tasks and outperforms state-of-the-art offline MARL baselines. Furthermore, we demonstrate that the proposed method has a better convergence rate, more sample efficient, and is more robust to various demonstration qualities compared with baselines.
Introduction
The online learning paradigm assumed by existing multi-agent reinforcement learning (MARL) algorithms is one of the biggest obstacles to their widespread adoption. To apply MARL, one has to repeatedly perform the following two steps: (a) collect experiences by deploying multiple agents, typically with their latest learned policies, and (b) use the collected experiences to improve the policies. In many scenarios, frequently performing the first step impractical because deploying multi-agents to the environment can be expensive and dangerous (e.g., self-driving cars). Therefore, devising offline MARL algorithms that can simply learn from previously collected datasets without interaction with the environment is an important step toward solving real-world problems.
Recently, several works have extended offline RL algorithms under the single-agent setting to offline MARL 45). These works focus on addressing the distribution shift issue that causes the values of unseen state-action pairs be erroneously estimated. However, these methods are based on temporal difference (TD) learning and thus require bootstrapping for credit assignment, problem that is especially challenging under the multi-agent setting as the interactions between the agents and the environment can be highly complex.
To mitigate these issues, explore the possibility transform offline MARL into sequence modeling problem. This paradigm shift, first introduced by Decision Transformer [3] under the single-agent setting, allows to bypass bootstrapping and perform credit assignment directly via self-attention. A concurrent work, MADT (26). proposes to adapt Decision Transformer to the multi-agent setting by (a) sharing model parameters across agents and (b) attaching one-hot agent IDs to observations. Different from MADT, we propose framework based on policy distillation that achieves better performance (as demonstrated in the experiments).
Our method first trains teacher policy (instantiated as decision transformer) to model the entire offline MARL dataset sequentially by accessing each agent's observation, action, and reward. The
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(x) Goal
(b) (Ramdorn Walks)
Figure 1: Illustrative Example. (a) Two agents are tasked to explore as many blocks possible. Explored blocks are colored (e.g., blue). (b) The training dataset consists of agents' dom walk trajectories and per-step re- wards. (c) Agents trained with behav- ior cloning result in suboptimal perfor- mance. (d) A centralized Decision Trans- former (Teacher DT) achieves superior performance but assumes privileged infor- mation. (c) Our framework structurally distills Teacher DT's policy into student policies for decentralized execution.
Behavior Cloning
Teacher DT
Student
privilege to access every agent's information helps the teacher policy understand the underlying interaction across agents and predict actions that encourage cooperative behavior. However, the teacher policy cannot be deployed test time in the fashion of decentralized execution. To address this issue, initialize separate student policy for each agent and distill the knowledge of the teacher policy into the student policy. In addition to allowing decentralized execution, this step is helpful because (a) the teacher policy identify and recombine "good" behavior in suboptimal dataset [3]. (b) the centralized teacher allows credit assignment across agents via self-attention, and (c) distillation from privileged model (teacher) provides richer and stable learning signal the student policy (2). Furthermore, we propose a novel distillation objective that transfers structural relations of student policies' features rather than actual values of individual features. In our empirical results, we show that this objective is complementary to the classical policy distillation and helps us outperform state-of-the-art baselines analysis on convergence rate, sample efficiency, and robustness to different demonstration qualities. Qualitative results are presented in our project website
range benchmarks. Additionally, provide further
In summary, our contributions are as follows:
A framework that reformulates offline MARL sequential modeling and policy distillation for using self-attention to perform multi-agent credit assignment.
A novel multi-agent policy distillation objective that focuses preserving structural rela- tionships among policies.
State-of-the-arr results on broad range of tasks in terms of performance, convergence rate, sample efficiency and robustness to demonstration quality compared with baselines.
Related Works
MARL. Applying RL to multi-agent settings has been an active research domain for years. One line of work adapts value-based RL algorithms to perform credit assignment among agents given the joint
reward signal 40 9 32
38 13]. COPA (2 and REFIL (15) uses dynamic team composition to further improve the training efficiency for tasks that require cooperative behaviors.
Another line of work extends the actor-critic framework to the multi-agent setting. For instance, MADDPG (55 uses DDPG (1 to jointly update decentralized policies and centralized critie. For better scalability, several works [16][7 further introduce the attention mechanism into the critic to prevent noises from irrelevant information.
unique challenge proposed various communication mechanisms tackle this issue. By exchanging information between agents, they can better understand the tasks and other agents. Some works g 28 24 20
aalti- environments 14 30 is local observation. Previous works have
incorporate the attention mechanism improve communication efficiency, while others 37 design a gating mechanism to decide when to communicate.
Offline RL. Different from the typical RL setting, offline RL trains the policy with static, collected dataset and does not need any online interaction with the environment. Then, this policy is leveraged
httpe:://veichhonttsnng github..co http::/:w....p..a.o.ooooo..oo/na

---

(a) Independent Decision Transformer (IDT)
(b]Muiti-Agent Decision Transformer (MADT)
a++1
Decision Transformer
Decision Transformer
Ritt
oft)
R
of
X
Xn
Shared Weight
Decision Transsformer
Non Shhardd Weight
Decision Transformer
Ri
R
Offine Datasot
Dataset
(c) Ours
Centralized Decision Transforrmr
Policy Distillation
Decision Transformer
Xn
R
R
0 8
Offline Dataset
Figure 2: Overview. We compare (a) independent decision transformer (IDT), (b) multi-agent decision transformer (26) (MADT), and (c) approach. (a) IDT trains independent decision transformer for each agent separately. (b) MADT extends IDT by sharing parameters across multiple agents and concatenate agent's one-shot IDs the observations. (c) Our approach first train teacher policy, instantiated by centralized decision transformer, and then distill both its features and structural relations among features to IDT.
to interact with the online environment to ohtain promising results or assist exploration 43 However, trajectories existing in offline datasets and interaction with the online environment have different distributions. One of the strategies to mitigate this issue is applying policy constraints [27 48) Another direction to alleviate this difficulty to regularize the value estimation in reinforcement learning [8] or take uncertainty into consideration T.
Recently, the Decision Transformer 3 outperforms many state-of-the-art offline RL algorithms by regarding the training process as a sequential modeling phase and testing on the online envi- ronment. This approach can bypass the drawback of TD-learning and overcome sparse reward difficulty. MADT [26) further extends Decision Transformer to a multi-agent domain by making agents controlled by shared weight transformer-based policy.
Knowledge Distillation. Knowledge distillation (KD) transfers knowledge from one deep learning model (the teacher) to another (the student). The objective originally proposed by [12] minimizes the KL divergence between the teacher and student outputs. This formulation makes intuitive sense when the output is distribution, such as a probability mass function over classes. Recent works focus on how to leverage the relationships between instances in the dataset further provide meaningful representation learning 42 41 RKD (29] extracts the relation between instances from the feature space by defining distance- wise relation and angle-wise relation.
Knowledge distillation is also adapted to RL scenario. 6 proposes to distill multiple task specifcci policies into a single policy which is more parameter-efficient. DPD [19] utilizes two policies that interact with the same environment with different initialization to explore different perspectives the environment and extract knowledge from each other to enhance their learning. M&M 77] combines curriculum learning as well as distillation to allow agents to perform well in an environment with large action space.
The typical goal of knowledge distillation is model compression by making the student model smaller. Our method is different because our goal is to enable decentralized execution in the multi-agent setting.
Method
In this section, we introduce our framework based on sequence modeling and policy distillation. We first describe baseline, Independent Decision Transformer (IDT), that naively adapts Decision Transformer to the multi-agent setting in Section 3.1 Then, present novel structural relation

---

Algorithm Our Offline MARL
Input: offline dataset D (Ti ai 11)=1) la Initialize: 8 as the parameters of Teacher Φ
the parameters of π 1mm
I/ training centralized decision transformer
for T in D do
d argmax Teentralized Im (22 0)
8 a V Lgentalized
end for
training decision transformers for agents Freeze the weight 0 for in D do
à!""h = argmax Treacher (a 1:x 1al==aaaa aa 0)
for i= do
af argmax Ti (i'ce ot)
0 o' V øi Lacioon + aelipet + (Lkr))
end for end for
distillation for multi-agent policy distillation in Section 3.2 Algorithm
The overall algorithm is presented in
3.1 Independent Decision Transformer
An intuitive method to transform offline MARL into sequence modeling problem is to treat each agent's 's trajectory as inde- pendent sequence. For each agent i, sep- arate decision transformer I is trained to predict the next action d'y1 given the past trajectories (o(I-B):) (t=):::: where B stands for the maximum num- ber of past steps consider. Since these agents are trained independently without access to other agents' information, call this method Independent Decision Transformer (IDT). Its learning objective can be formulated that aims to predict the next action, L'aation | d;+1 ll We note that although each agent can leverage self-attention to perform credit assignment along their own trajectories, this algorithm cannot perform credit assignment across agents. This limitation prohibits agents to achieve cooperative behavior that attains maximal returns.
Conventional Policy Distilation
Polcy Distillation
of
OC
O
Figure 3: Policy distillation objectives we consider in our work. (a) Convential policy distillation 34 transfers individual outputs from a teacher model (green dots) to student model (blue dots) point-wise, while (b) the pro- posed Relational Policy Distillation transfers structural relations of multi-agents' features.
B):t1 B)t)
sequence modelling problem
3.2 Teacher-Student Policy Distillation
To address IDT's cross-agents credit assignment issue, we propose to first train a centralized teacher policy Weacher that takes in all agents' observations, actions, and rewards (o! B):e: B) (t- m)a) combined via concatenation to predict all agents' actions I.n at the next step. We represent the teacher policy Teacher with a decision transformer and train it to minimize the prediction error of all agents' actions. Since the teacher policy has access to all agents' information, it can perform credit assignment across agents with self-attention to better foster cooperative behavior. This is similar to how centralized critic helps actor critic- based methods in online MARL [16] 25 However, the centralized teacher policy cannot be deployed distributedly. To fix this, we propose distill the teacher policy's features to separate student policies. We note that even though student policies operate independently during test time, they are very different from policies learned by IDT because (a) the supervision comes from a teacher policy that can perform credit assignment across agents, and (b) distillation provides more stable learning signal to the student policy (2] One intuitive policy distillation objective is to minimize the KL divergence between the actions predicted by the teacher

---

policy at and the student's policy :
liK Dxz(a"aa))
The other common distillation objective is to minimize the euclidean distance between the teacher's features fr and the student's ' features J 351
T Tijeature
11 J 1
(2)
However, both of these widely- used distillation strategies do not consider the structural relation between multi-agent students.
Structural Relation Distillation. To help the student policies learn as much as possible from the teacher policy, we introduce a new distillation objective tailored to the multi-agent setting. Our main insight is that in addition to transferring the teacher policy's 's actual feature values, we wish to preserve the structural relation among multi-agents' features. For example, two agents belong to the same type of units and have similar observations in SMAC [35], their outputs should be close to each other. To this end, we define the relation between agents as the angle between feature vectors, and we intend to make the relation between student policies Timm mimic the relation between agents controlled by
Twacher In other words, we would like to minimize
Liee Σ H((cos (( (f' c' (
where H denotes the Huber loss. During the policy distillation procedure, the weight of the centralized policy Twacher is frozen. The comparision IDT, MADT 26 and our approach is summarized in
Fig. 2
which use an MLP projection head to provide flexibility for contrastive representation learning. we propose use pair of mapping networks (M, N)) to help policy distillation and prevent from loss of information induced by relational distillation loss. By leveraging mapping networks that remove information irrelevant to structural relation yet potentially useful in downstream policy learning, more information can be formed and maintained in the feature f. Specifically. M and N are simple MLPs that transform the features before relational policy distillation. Note that the weight of mapping networks non-shared (M + N) in our setting since the teacher and student policies are inherently asymmetric in the amount of information allowed for reasoning (the teacher has access to privileged information from all agents). Therefore, the relational policy distillation becomes
Mapping Networks. Inspired by recent works 4
Liee
EH(cos H(coos'(MM ((M())
Σ JZi
Cos (NNTN)
However, learnable and non-fixed M effectively forms a moving target in the objective, causing unstable learning. To stabilize the distillation process, we adopt momentum-lik, update for mapping networks. We make the update frequency of the teacher's 's mapping network M lower than the student's mapping network N. In other words, we update M every e updates of N. We empirically found that setting to e.
improves the convergence speed, but the converged performance is insensitive
In summary, the overall learning objective for agent is
Liotal L'uction + aLiet + BL
(4)
where a and | are hyperparameters that determine the importance of the proposed policy distillation.
4 Experiments
We execute a series of experiments to evaluate whether the proposed method is effective at solving offline MARL problems. Specifically, our experiments seek to answer the following questions: first, does our method perform favorably against wide range of existing approaches based on sequence modeling, imitation learning, and offline reinforcement learning (Section 4.1) Specifically, we compare method with model-free offline MARL methods based on TD-learning and MADT,
5

---

Fill-In
Equal
Space
Grid-World
Highway
BC
12.43 0.21
-9.35 0.64
1.49 1 0.16
13.38 + 1.14
IDT
3
4.42
-7.99 0.42
1.52 + 0.27
18.71 = 1.53
MADT 26
6.51
-6.91 =0.92
1.57 + 0.34
18.78 = 1.27
MA-CQL
18
9.41
-6.99 -0.38
1.44 0.30
17.69 = 1.32
MA-ICQ
45
9.72
-7.12 0.29
1.62 1 0.29
18.01 = 1.27
MA-BCQ 1O
8.21
T.06 0.59
1.49 0.44
17.92 = 1.48
MA-GAIL
39
3.11
-843 0.42
1.51 + 0.32
16.48 = 1.80
MA-AIRL
1.07
88.33 0.63
1.52 + 0.37
16.76 = 1.95
Ours
3.41
2243 0.72
2.09
+ 0.22
23.35 1 0.91
SMAC 3S
2s3z
3s5z
8m9m
3s5z vs 3s6z
BC
14.77 + 1.01
11.32 + 0.79
11.45 + 1.14
10.86 + 0.99
DTT 3
17.63 + 1.80
15.99 + 1.11
15.93 1:03::00 0.86
16.33 + 1.93
MADT 26
18.09 + 1.26
16.18 1.05
17.11 1.83
16.91 + 2.10
MA-CQL
18
17.04 1.38
15.02
14.92 --92+110 1.87
15.32 + 2.42
MA-ICQ 45
17.42 + 1.52
15.36 2.01
14.72+1.2. 1.22
14.99
2.21
MA-BCQ 10
17.08
1.12
15.09 1.84
14.32 1.02
15.78 1 1.64
MA-GAIL 39
15.01 + 1.12
13.99 0.84
13.99
+ 0.69
14.98 + 2.04
MA-ATRI
47
15.11 + 1.12
14.02 + 0.84
14.01
0.79
14.95 + 2.18
Ours
18.12
1.31
16.98 + 1.19
18.33 0.99
18.78 + 2.01
Table 1:
Quantitative Results. We show the average and standard deviation of return pre-agent, and all the experiments are merged with 10 random seeds. As for detailed information about the offline dataset, please refer to appendix D
Grid-World
Highway
Dataset Quality
gnod
normal
good
normal
BC
1.49 1. 0.16
1.29
1: 0.05
1.01
0.12
13.38
1.14
10.23 1 0.91
8.71
1 0.72
1.32 + 0.27
1.45
0.12
1.43
0.09
18.71
1.53
18.01
1.39
17.58
1.01
MADT
26
1.57 1 0.34
0.10
1.44
0.11
18.78
1:
1.27
18.03
1 1.02
17.84
1 0.84
MA-CQL
18
1.44 0.30
0.21
1.31
0.11
17.69
16.88
0.93
0.67
MA-ICQ
451
1.62
0.29
1.37
1.32=
1.32
0.09
18.01
1.27
16.54
0.74
0.74
16.03
0.83
BCO
10
1.49
1 0.44
1.39
0.19
1.37
0.05
17.92
1.48
16.89
1 0.78
15.73
1 0.58
MA-GAIL
1.19
MA-AIRI
47
1.52
0.37
1.21 0.27
1.11 1.11 0.25
16.76
1.95
15.82
1.93
10.22
1.87
Ours
2.09
+ 0.22
1.89
0.112
1.81 0.19
23.35
0.91
20.01
1.01
17.13
1.12
Table 2: Offline learning results offline trajectories with different quality. In general, our approach outperform other baselines when the quality the offline dataset is not perfect.
concurrent work that also solves MARL via sequence modeling. Next, conduct careful ablation studies to evaluate the contribution of each component within our framework (Section 4.2) To test the sample efficiency of various approaches, we further show the performance of various methods when different numbers of demonstrations are provided (Section 4.4). Finally, we analyze the convergence rate (Section 4.5) and discuss the scalability (Section 5) of our method.
Basclines. compare our approach with three group baselines including sequence modeling (IDT, MADT), offline RL (MA-CQL, MA-ICQ, MA-BCQ) and imitation learning/inverse RL (BC, MA-GAII, MA-AIRL), and we briefly introduce them as follow.
BC: behavior cloning. ITT: each agent is represented by decision transformer (3] independently. MADT 26 a weight-sharing decision transformer used to represent each agent's policy. MA- CQL [8 conservative Q-learning (CQL), which aims to address the extrapolation error by learning a conservative Q-function such that the expected value of policy under this Q-function its true value. MA-ICQ 455: implicit constraint Q-learning (ICQ), which improves upon MA-CQL by only trusting the state-action pairs given in the dataset for value estimation under the multi-agent setting. MA-BCQ [TO]: : batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset the given data. MA-GAIL [39]: an inverse reinforcement learning for multi-agent scenario. MA-AIRL [47]: framework for multi-agent inverse reinforcement learning, which is effective and scalable for Markov games with nigh-dimensional state-action space and unknown dynamics. For detail experimental setting about our approach and baselines, please refer to appendix A.

---

Environments. We test our approach on three multi- agent environments. Fill-In: grid-world environ- ment that agents are required to pass all the blocks map. Equal Space: a particle system environment that agents need to keep the same distance between each other. SMAC [28]: a collaborative multi-agent reinforcement learning based on Blizzard's StarCraft RTS game, and we choose four different scenarios, 2s3z, 3s5z, 8m9m, and 3s5z vs 3s6z. Grid-World: a simple grid-worke environment that allow agents to conduct discrete actions move and collect the corresponding objects. Highway: a multi-agent envi- ronment that simulate highway traffic scenario. Rach controllable agent drives a car and the goal is to reach high speed while not colliding into neighboring hicles. For detail experimental setting, please refer to appendix B.
(a)
(c)
(e)
Figure Environments we used in our ex-
Offline Dataset The offline MARL dataset for n
periments: (a) Fill-In (b) Equal Space (c)
agents is represented (o! ol r!!")?=-- For Grid- fordd and Highway, we train a centralized policy with PPO [36] and treat it
set of trajectories
Grid- World (d) SAMC Highway. See the
detailed description of each task in Section 4
as the expert to generate demonstrated trajectories. The offline datasets for SMAC 335 are collected
by running a policy trained with MAPPO 46.. appendix C.
We also provide more information about datasets in
Fill-In
Equal Space
Grid-World
Highway
Ours (conventional distillation)
-4.11 0.22
-5.43 0.51
1.62 = 0.29
18.33 = 1.27
Ours (M = N)
-3.11 0.22
-2.61
1.42 = 0.19
14.92 = 1.87
Ours Momentum Update
-3.92 + 0.31
+
1.79 = 0.19
18.62 + 1.98
Ours Mapping Network
-3.32 0.42
-287
1.88 = 0.11
18.21 - 1.99
Ours KL. Distillation
3388 1 0.33
-2.99
0.27
1.99 - 0.51
20.11 1.88
Ours KI. Distillation
Mapping Network
-3.30 =
-2.71
0.42
1.59
0.44
18.40 1.09
Ours Full method
-3.41 + 0.12
-2.43 + 0.72
2.09
0.22
23.35 0.91
SMAC 35
2s3z
3s5z
8m9m
3s52 vs 3s6z
Ours (conventional distillation)
16.11 1.87
15.78
1.78
15.99 1.11
17.93
1.32
Ours (M
12.78
1.08
15.4022.1
12.27 + 1.11
Ours Momentum Update
15.69 + 1.91
15.20
+ 0.81
16.39 = 2.70
17.51
1.49
Ours Mapping Network
15.23 1.81
15.01 0.31
16.87=0.99
17.01 + 0.32
Ours KL Distillation
16.32 1 0.91
16.21 1 0.55
17.01 1.26
17.80 1 1.08
Ours KL Distillation
15.01 + 1.23
14.98 +: 1.06
16.42
1.22
17.71 +: 1.88
MMapping Network
Ours Full method
18.12 + 1.31
16.98 + 1.19
18.33 + 0.99
18.78 + 2.01
Table 3: Ablation study. We show the average and standard deviation of return pre-agent, and all the experiments are merged with 10 random seeds.
4.1 Quantitative Results
We present the quantitative results in Table I For all the experiments, we report the mean and standard deviation of rewards based on 10 runs using different random seeds. The results show that our approach outperforms other baselines in all three environments. In general, we find that offline RL approaches reach higher performance compared with imitation learning and inverse RL methods. It is because offline RL approach can use reward signals to judge the quality of the policy and try to infer optimal strategy. Sequence modeling methods slightly outperform offline RL ones since Sequence modeling methods allow us to bypass bootstrapping and perform credit assignments directly via self-attention. Besides, comparing ours IDT, we can find that the proposed structural knowledge distillation does provide performance benefits.

---

fill-in
equal space
grid world
hig tway
0.0
20
reuure
-7.5
10
-10.0
20
40
60
30
100
60
50
100
20
50
100
20
40
50
100
traininng tteratioon (1a-3)
training Berraion (1id3)
tratting Iberation (1w3)
training (teration (1e3))
Figure 6: Learning curves of online finetuning. The experimental results are merged with 10 random seeds, and the shaded area represent the standard deviation.
We also find that our approach performs better than inverse MARL methods. Although inverse MARL discards the necessity of reward signal, it assumes the quality of demonstration trajectories is perfect. Under this assumption, is possible infer reward signals in an adversarial manner. However, since it's hard to guarantee the quality of multi-agent trajectories due to its dependency other agents' behaviors, the drawback of inverse MARL becomes obvious.
In Table 2] we present the quantitative results with different qualities of offline demonstrations and find that our method outperforms all baselines except for one setting. The results suggest that our method is more robust to various demon- stration qualities across tasks. We hy- pothesize our method performs on par with baselines Highway-poor because learning strong centralized decision transformer from poor demonstrations is challenging. This turns affects the per- formances of student policies.
Normal Distilation (KL divergence)
Relational Distillation (Ours)
Figure 5: Qualitative results on equal space environment. By observing the behavior of the agents, our approach encourage agents to behave more efficiently.
4.2 Ablation Study
To validate the effectiveness of each component in approach. To be more specific, compare our approach with following variant. Ours Momentum Update: the variant that both mapping network (M and N) are updated with the same frequency. Ours Mapping Network: the variant that relation knowledge distillation is directly applied to feature space of decision transformer. Ours - KL divergence: our approach without the KL divergence between teacher and student distillation. Ours (conventional distillation): our approach but use KL divergence for knowledge distillation from the teacher policy. Ours (M .N: the variant of our approach that the mapping networks for teacher policy and student policies share the same weights.
The results show that all the components in our approach are required. Specifically, Ours Mapping Network achieves poor performance on 7 out of 8 tasks. It shows that Mapping Networks that transform the students' and teacher's features into the same space for distillation are essential. Additionally, according to the standard deviation of the performance, we find that approaches that do use momentum update are less reliable. KL divergence also provide performance benefit, but not significant compared with mapping networks and momentum update.
We also find that ours (conventional distillation) performs the worst, which highlight that the proposed our knowledge distillation is more effective than transitional knowledge distillation. It is because structural relation among multi-agent can potentially represent how agent interact with each other, which is the critical signal for multi-agent task. Fig. shows the behavior of the policies trained by our approach and ours with conventional distillation, respectively. To make agents keep the same distance from each other, policy obtained by our method tends to make agent move the the closest corner and form an equilateral triangle structure. On other hand, policy obtained by offline learning and conventional distillation encourage specific agents to go specific corners, which is sub-optimal solution. The performance Ours (M = N) drops lot, which represents that making the weights of the two mapping networks non con-sd is reasonable choice.

---

Fil-In
Equal Space
Highway
Grid-Worid
20
return pergeent
retum RigHtte
perageent
2.00
-5
1.50
-20
-12
1.00
-14
-16-
100
200
300
400
500
200
400
500
800
1000
200
800 number of trajectories
400
500
1000
1000
2000
3000
4000
number of trajecto 1o5
number trajectorios
number of trajectories
SC
MADT
MA-EQ
MA-CQI
MA-BCO
Ours lconventional distiletion)
Durs
Figure 7: Data efficiency in offline learning setting on highway and grid-world environment. The experimental results merged with 10 random seeds, and the shaded area represents the standard deviation. general, our approach can have better data efficiency.
4.3 Finetuning
To verify whether the offline pre-trained framework can be further improved, we finetune the pre- trained framework with MAPPO [46]. In Fig we show the comparison of learning curves between finetuming based on the pre-trained policy and policy trained from scratch. The experimental results show that the pre-trained policy can get improvement with an online multi-agent reinforcement learning approach.
4.4 Data Efficiency
We also show the performance with a different number of trajectories Fig. 7 The results show that our relational knowledge distillation achieves better performance than conventional knowledge distillation when the number of offline trajectories is relatively low. Besides, we also find that offline RL approaches perform well with limited offline trajectories compared with our approach. We hypothesize that the cause is that the transformer-baser approach requires large amount of data. We stress that a common offline MARL setting allows abundant yet non-interactive demonstration and thus our approach still meaningful and desirable. Finally, we find that naive imitation learning and behavior cloning perform the worst in all the cases, and we hypothesize that imitation learning suffers from covariate shift, which can be more severe when the training data is few and unable to cover a sufficiently large domain.
4.5
Convergence Rate
We observe better comparable convergence with the centralized decision transformer for all environments compared to MADT, c.g., in SMAC 3s5z vs 3s6z with poor quality data, the centralized transformer is ~8% better 20k iterations (not fully converged). For knowledge distillation, requires roughly similar total training iterations for the student policies to converge. Besides, adding centralized decision transformer doesn't introduce large overhead in training time. In general, adding one more centralized DT only requires less than an additional 10% training time in our experiment, the training time as well as learning curves for each experiment are presented in Appendix
Limitation and Broader Impact
The scalability of multi-agent frameworks is important. Our approach depends centralized decision transformer, which predicts all actions of each agent based on the returns and actions of all the agents. Therefore, the centralized decision transformer may fall short in scalability. However, experiments find that we can still reach comparable performance compared with MADT when the number of agents increases. We leave scalability in multi-agent policy distillation for future work. Additionally, we note that offline MARI. algorithms may lead to negative applications such as flying fleet of drones for surveillance.
Conclusion
In this work, we propose an offline MARL algorithms that first trains a teacher policy as if the dataset generated by single agent. Once the teacher policy is trained, we use its predictions to supervise the student using novel distillation objective that considers agents' relations.

---

References
[1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective offline reinforcement learning, 2020.
[2] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In CoRL, 2019.
[3] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurlPS, 2021.
[4] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
[5] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, 2020.
[6] Wojciech M. Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszez, and Max Jaderberg. Distilling policy distillation. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, 2019.
[7] Wojeiech Marian Czarnecki, Siddhant M. Jayakumar, Max Jaderberg, Leonard Hasenclever,
Yee Whye Teh, Simon Osindero, Nicolas Heess, and Razvan Pascanu. Mix match agent curricula for reinforcement learning, 2018.
[8] Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. TarMAC: Targeted multi-agent communication. In ICMI, 2019.
[9] Jakob Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learinng awareness. In AAMAS, 2018
[10] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In ICML, 2019.
[11] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie. and Ross Girshick. Momentum contrast for unsupervised visual representation learning. CVPR, 2020.
[12] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
[13] Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. (UPD/et: Universal multi-agent (rl) via policy decoupling with transformers. In ICLR, 2021.
[14] Wenlong Huang, Igor Mordatch, and Deepak Pathak. One policy control them all: Shared modular policies for agent-agnostic control. In ICMI, 2020.
[15] Shariq Iqbal, Christian A Schroeder de Witt, Bei Peng. Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In ICML, 2021.
[16] Shariq Iqbal and Fei Sha. Actor-attentiin-crittc for multi-agent reinforcement learning. In ICML, 2019.
[17] Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent operation. In NeurlPS, page 7265-7275, Red Hook, NY, USA, 2018. Curran Associates Inc.
[18] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative gelearning for offline reinforcement learning. 2020.
[19] Kwei-Herng Lai, Daochen Zha, Yuening Li, and Xia Hu. Dual policy distillation, 2020.
[20] Sheng Li, Jayesh K. Gupta, Peter Morales, Ross Allen, and Mykel J. Kochenderfer. Deep implicit coordination graphs for multi-agent reinforcement learning. In AAMAS, 2021.
10

---

[21] Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez. Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
[22] Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. In ICMI,, 1994.
[23] Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, and Animashree Anandkumar, Coach-player multi-agent reinforcement learning for dynamic team composition, 2021.
[24] Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. Multi-agent game abstraction via graph attention neural network. In AAAI, 2020.
[25] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch, Multi-agent actor-critic for mixed cooperative-competttiee environments. Neural Information Processing Systems (NIPS) 2017.
[26] Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang. Ying Wen, Haifeng Zhang, Jun Wang, and Bo Xu. Offline pre-trained multi-agent decision transformer: One big sequence model tackles all SMAC tasks. 2021.
[27] Ofir Nachum and Bo Dai. Reinforcement learning via fenchel-rockafellar duality, 2020.
[28] Yaru Niu, Rohan Paleja, and Matthew Gombolay. Multi-agent graph-attention communication and teaming, In AAMAS, 2021.
[29] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, 2019.
[30] Deepak Pathak, Chris Lu, Trevor Darrell, Phillip Isola, and Alexei A. Efros. Learning to control self- assembling morphologies: A study of generalization via modularity. In NeurlPS, 2019.
[31] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[32] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In ICML, 2018.
[33] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: urXiv:1412.65500 2014.
[34] Andrei Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint
2015.
[35] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung. Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043,, 2019.
[36] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
[37] Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Individualized controlled continuous communication model for multiagent cooperative and competitive tasks. In ICLR, 2019.
[38] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In ICML, 2019.
[39] Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adver- sarial imitation learning, 2018.

---

[40] Peter Sunehag, Guy Lever, Audrunas Gruslys. Wojciech Marian Czarnecki, Vinicius Zambaldi. Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, and Thore Graepel. Valus-decompositioon networks for cooperative multi-agent learning based on team reward. In AAMAS, 2018.
[41] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation, 2020.
[42] Frederick Tung and Greg Mori. Ammilarity-preservvie knowledge distillation, 2019.
[43]
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma. Jiantao Jiao, Sergey Levine, and Karol Hausman. Jump-start reinforcement learning. https:/arriii.rr//////200...33.2 /22p://bbss220442222.,.. 2022.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[45] Yiqin Yang. Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. In NeurlPS, 2021.
[46] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games, 2021.
[47] Lantao Yu, Jiaming Song, and Stefano Ermon. Multi-agent adversarial inverse reinforcement learning, 2019.
[48] Chi Zhang, Sanmukh Kuppannagari, and Prasanna Viktor. Brac+: Improved behavior regular- ized actor critic for offline reinforcement learning. In Proceedings The 13th Asian Conference on Machine Learning, 2021.
12


Reviews:
[{"review_id": "sxMPDKarhBK", "reviewer": "Reviewer_dysR", "summary": "The paper designs an offline MARL framework based on decision transformer and policy distillation. And proposes a novel distillation loss to distillate the structural relations among different agents from a centralized teacher policy. The experiment results illustrated the improvement brought by policy distillation.", "questions": "See weeknesses before.", "limitations": "The authors mentioned the scalability problem. Also, the tested environments most have relatively short horizons. I wonder whether this framework can still remains a relatively fast training speed (additional 10% training time) if the horizons are long since attention has O(L^2) complexity. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The idea of combine knowledge/policy distillation and decision transformer for multi-agent scenario is interesting. \n2. The proposed design of relational policy distillation loss is reasonable in multi-agent cases.\n3. The authors conduct thorough experiments on multiple environments. The proposed framework achieves state-of-the-art results and the ablation shows the performance raise brought by relational policy distillation loss is important.\n\nWeaknesses:\n1. Although the proposed framework achieves significant performance raise on the four environments designed by the authors. The improvements on SMAC benchmarks are not significant compared to MADT. In the ablation studies, the conventional distillation method performs worse than IDT or MADT on multiple tasks. Is this because the teacher policy itself does not take a big lead on these tasks?\n2. The authors did not mention the settings for initial target return, which would significantly influence the performance of DT in single-agent settings. The setting of target return in multi-agent cases should be more complicated as it involves credit assignment. \n3. The authors did not analysis the influence of number of parameters, which might be important as DT is more complicated than networks used in previous offline MARL methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nrUjow3tAX", "reviewer": "Reviewer_2W5g", "summary": "In this paper, the authors propose a distillation-based offline MARL method for improving the quality of pre-trained multi-agent policy online. The authors construct the teach and student policy with access to the global and local information, repsectively. Experimental results on the intuitive and public environments show improvement over many strong baseline methods.", "questions": "1. The idea of this paper is similar to a recent work named Jump-Start Reinforcement Learning, which learn to use the guide policy to inform the whole online policy. Can you show the difference between the proposed approach of this method?\n\n2. Can you show more detailed distillation process used in this work? And how to distilled a reasonable student policy?", "limitations": "The scalability may limit the application of this work, which the complexity will increase with the scale of multiple agent enlarges as described above.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The authors propose the utilization of global and local information with teach and student policy distillation.\n1. The authors show the reasonable knowledge distillation on the teach policy pre-trained with global information. Then the student policy can be learned by distilled the parameter individually with relational constraints to improve the cooperation. The idea can be validated from the centralized training and decentralized execution (CTDE) framework.\n2. The clear writing and sufficient experiments show the effectiveness of this approach compared with several strong baselines including sequential modelling on MARL, MADT. \n\nWeakenesses: The learning complexity may increase with the scale of multi-agent enlarges.\n1. I think this work has the obvious limitation in the computation complexity including learning the teacher policy from offline dataset, and constraining the relational distillation.\n2. In line 296, the authors conclude the supereme in the poor offline data compared with MADT. I want to know the theoretical analysis of this results in detail.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4hZoA-e9QgS", "reviewer": "Reviewer_Vg5F", "summary": "The authors aim to tackle the multi-agent offline RL domain by leveraging the power of transformers for credit assignment using sequence modeling, hence removing the reliance on TD-learning-based bootstrapping methods. The proposed method first trains a “teacher” decision transformer on the full dataset to learn cross-agent interactions. Then the teacher distills relevant knowledge into the “student agents” via “structural relation loss.” Thus the proposed method utilizes the attention mechanism for learning interactions among the agents better than the current offline RL methods.", "questions": "1. In the Algorithm, shouldn't the student-agent gradient update equation (φi = φi − α ∇φi (Li action + αLi rel + βLi KL)) come inside the inner for-loop?\n2. I am not sure how the **Fill-In environment** works. If agent 1 has passed a particular block, will that block be then inaccessible for agent 2, and vice-versa?\n3. **If the answer to the above is yes, then skip this point.** But if it is not, it is unclear why the agents here even require cross-agent credit assignment during training?\n4. Re: Table 11 - Why does the return distribution for the Highway dataset have the opposite order, i.e., poor > normal > good? It seems like a typo to me.\n5. Referring to Line 563 - why are obstacles and apples needed in the Fill-In environment? Is this a typo?\n6. In the **equal space environment**, what are the initial positions of the three agents?\n7. In Table 2, how were the trajectories categorized into “good,” “normal,” and “poor”? Could the authors shed more light on how different qualities of datasets were collected?\n8. MADT has been cited with citation number 18 in many places (including the tables), but citation 18 refers to MA-BCQ. The correct citation number for MADT is 27. The authors should thoroughly cross-check the citations in the main text and the supplementary.", "limitations": "- Authors haven't discussed potential negative societal impacts of their work. \n- Authors have addressed the scalability of their method as a limitation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1. Leverages transformers for learning beneficial cross-agent interactions and distills them into student policies via well-motivated structural-relation loss constraint\n2. Achieves strong empirical performance in various multi-agent environments\n3. Authors conducted extensive ablation studies to show the importance of every component in the proposed method, as well as discussed.\n\n# Weaknesses\n\n1. Figure 6 compares only the proposed method w/ and w/o pretraining. Authors should include a comparison with MADT-online too.\n2. My main concern is that the method does not seem scalable. In environments like SMAC (where number of agents > 2), simpler transformer-based method MADT performs comparable to the proposed method. \n3. Minor: Lots of typos and grammatical errors in the text", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept