MOSDT: Self-Distillation-Based Decision Transformer for Multi-Agent Offline Safe Reinforcement Learning
Authors: Yuchen Xia
Yunjian Xu
Venue: NeurIPS 2025 poster ()
Keywords: multi-agent offline safe reinforcement learning
decision transformer
self-distillation
Primary area: reinforcement_learning
TL;DR: For multi-agent offline safe reinforcement learning (MOSRL), we propose the first algorithm MOSDT, and the first dataset and benchmark MOSDB.

Abstract: We introduce MOSDT, the first algorithm designed for multi-agent offline safe reinforcement learning (MOSRL), alongside MOSDB, the first dataset and benchmark for this domain. Different from most existing knowledge distillation-based multi-agent RL methods, we propose policy self-distillation (PSD) with a new global information reconstruction scheme by fusing the observation features of all agents, streamlining training and improving parameter efficiency. We adopt full parameter sharing across agents, significantly slashing parameter count and boosting returns up to 38.4-fold by stabilizing training. We propose a new plug-and-play cost binary embedding (CBE) module, which binarizes cumulative costs as safety signals and embeds the signals into return features for efficient information aggregation. On the strong MOSDB benchmark, MOSDT achieves state-of-the-art (SOTA) returns in 14 out of 18 tasks (across all base environments including MuJoCo, Safety Gym, and Isaac Gym) while ensuring complete safety, with only 65% of the execution parameter count of a SOTA single-agent offline safe RL method CDT. Code, dataset, and results are available at this website: https://github.com/Lucian1115/MOSDT.git

MOSDT: Self-Distillation-Basee Decision Transformer for Multi-Agent Offline Safe Reinforcement Learning
Yuchen Xia' Yunjian Xul
'The Chinese University of Hong Kong ycxia@link cuhk. edu. hk, yjxuêmae cuhk. edu, hk
Abstract
We introduce MOSDT, the first algorithm designed for multi-agent offline safe rein- forcement learning (MOSRL), alongside MOSDB, the first dataset and benchmark for this domain. Different from most existing knowledge distillation-based multi- agent RL methods, we propose policy self-distillation (PSD) with new global information reconstruction scheme by fusing the observation features of all agents, streamlining training and improving parameter efficiency. We adopt full parameter sharing across agents, significantly slashing parameter count and boosting returns up to 38.d-fold by stabilizing training. We propose a new plug-am lnng--nd-yn and-play cost binary embedding (CBE) module, which binarizes cumulative costs as safety signals and embeds the signals into return features for efficient information aggregation. On the strong MOSDB benchmark, MOSDT achieves state-of-tte of-heoofthhe-rt (SOTA) returns in out of 18 tasks (across all base environments including MuJoCo, Safety Gym, and Isaac Gym) while ensuring complete safety, with only 65% of the execution param- eter count of SOTA single-agent offline safe RL method CDT. Code, dataset, and 15/MOSDT.giit results are available at this website: https://gttub..cm//i
Introduction
Offline reinforcement learning (Offline RL) leverages static datasets to derive policies (1. Its applications span diverse domains, such large language models (LLMs))
MOSDB dataset
2], robotics 33, and power systems [4]. Offline RL has spawned specialized subfields, including offline multi-
Policy supervised learning
CBE
1
agent RL (offline MARL) 5 6 for systems with inter- acting agents, and offline safe RL. 7 for safety-critical problems. However, multi-agent offline safe RL (MOSRL) remains largely unstudied. MOSRI. seeks to learn safe policies for multiple agents in an interaction-fii fering significant potential for distributed safety-critical applications, like autonomous vehicle coordination [), power grid scheduling [10), and robot collaboration [II]
Reward-safety signal fused features
Data-wisc considerations
Forward propagation
way, of-
Full parameter shared student networks
Foowwrrd propagation
PSD2
Teacher network
While existing offline RL methods provide solid foun-
dation, they may not fully address the complexities of multi-agent cooperation, offline learning, and safety assur- ance within such contexts.
Model-wise considerations
(Coot binary embodding "Policy self aelf-disiilattoon
MOSDT model
Figure 1: Training MOSDT on MOSDB.
Addressing MOSRL requires tackling the key challenges from both offline MARL and offline safe RL. For offline MARL, the centralized training with decentralized execution (CTDE) framework has emerged as
*correppondin author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

prevalent communication-fee solution. Within this paradigm, knowledge distillation (KD) [ based methods (like MADTKD) [5] show strong performance by training a teacher network and then distilling policy from the teacher to students. Self-distillation (SD) 13 is a streamlined KD variant with teacher-student integration and the synchronization of supervised learning and distillation. SD well-suited for multi-agent applications due to its efficiency. Despite the success of SD in computer vision [13 [4 [5], its effectiveness in MARL remains untested. Besides, MARL methods generally suffer from training instability resulting from dynamic policy improvements among different agents 16 17.
In offline safe RL, decision transformer (DT) 18 based methods, such as CDT M achieve notable performance by leveraging the causal transformer network [19]. These approaches need targets for returns and cumulative costs to launch the execution. Existing methods typically set cost targets as specific values (like predetermined thresholds (7), converging actual cumulative costs to the predetermined values [20 21]. Unfortunately, such convergence may lead to suboptimal and/or potentially unsafe actions along trajectories. In addition, as causal transformers can only attend to preceding tokens, they would overlook the cumulative cost or the return, specifically whichever occurs later in the input sequence.
This work proposes MOSDT (multi-agent offline safe decision transformer), the first algorithm tailored to MOSRL. Fig. | shows the process of training MOSDT on our proposed MOSDB dataset. To streamline the training process and the network structure, we propose policy self-distillation (PSD) to build an efficient CTDE framework instead of adopting the two-stage conventional KD (used in MADTKD ()) For MARI,, PSI) distinguishes from SD [3 with new global information reconstruction scheme by summing up the observation features of all agents. PSI) integrates stu- dent networks within the teacher network and is performed synchronously with policy supervised learning-a pioneering design reducing training parameter count by 35% and training time by 24%. MOSDT marks the first demonstration that SD is effective in MARL, hinting at broader applicability for existing methods.
To ensure training stability, we adopt full parameter sharing across all agents. while other offline MARL methods with sharing designs [22 23 share a subset of parameters. Compared to the case with no/partial parameter sharing, full parameter sharing significantly boosts returns for tasks with three or more agents (in MuJoCo environment (24): 10.1 I to 38.4-fold for a three-agent task, 2.2 to 2.3-fold for a four-agent task, and 1.6 to 5.6-fold for a six-agent task. Full parameter sharing significantly slashes training parameter count by 47% and execution parameter count by 58%, making MOSDT much more lightweight and scalable.
To improve existing cost processing methods, we propose cost binary embedding (CBE), a plug-and- play module consisting of cost binarization and safety signal embedding. CBE fuzzifies cost targets to prevent actual cumulative costs from converging to predetermined values in execution. Towards this goal, we binarize cumulative costs as safety signals during training. We embedded the safety signals into returns, explicitly passing reward-cost correlations to causal transformers for sharper information aggregation. While ensuring safety, CBE improves returns on 14 out of 18 tasks (across
all base environments, including MuJoCo [24. Safety Gym (25 and Isaac Gym 26) in MOSDT.
Finally, we build the MOSDB dataset by collecting training data from 2 online algorithms [27] on all safe MARL tasks in Safety Gymnasium (28), following the online-collection paradigm in DSRL 29 (a widely used single-agent safe RL dataset). Given the absence of direct baselines for MOSRL, we follow E0O to compare MOSDT with centralized single-agent offline safe RL methods on MOSDB. Extensive experiments demonstrate the state-of sate-of-theeart (SOTA) performance of MOSDT. Ablation experiments validate the contributions of PSD, full parameter sharing, and CBE to return enhancement and risk control.
Our contributions are: (1) MOSDT, the first MOSRL algorithm, achieving high parameter efficiency and a streamlined training process by PSD. (2) A full parameter sharing design that stabilizes training and makes MOSDT lightweight and scalable. (3) CBE, A plug-and-pla module, offering an intuitive alternative to specific cost targets and improving information aggregation. (4) MOSDB, the first MOSRL dataset and benchmark, on which MOSDT achieves SOTA performance in 14 out of 18 tasks with only 65% of the execution parameter count of our base model CDT 7
2

---

2 Related work
Offline RL DT 18] proposes a powerful paradigm that formulates offline RL as a sequence modeling problem, inspiring subsequent research. For example, MGDT [31] achieves rapid adaptation to new tasks through fine-tuning. EDT [32] supports trajectory stitching by adjusting history lengths. HDT [33] leverages subgoal states to make decisions. However, the performance of DT in MOSRL remains unstudied.
Offiiee MARL For multi-agent systems, the CTDE framework eliminates information dependen- cies between training and execution, bypassing the communication requirements posed by partial observability. Recent works [
34 35 adopt KD to implement the CTDE framework. MADTKD [5] aligns the output between partially observable student DT's and globally observable teacher DT. PTDE 34 tailors the global information to each agent and distills into partial observations. LDPD 35 uses L.LSs as teachers to train compact students. The two-stage training of KD may lead to scalability issues in MARL [5] SD [3] offers a streamlined alternative to KD, although its suitability for MARL remains unknown. Additionally, while existing offline MARI methods at most partially share parameters across agents [22 23), full parameter sharing has not been attempted.
Offline safe RL DT--aaed methods again show impressive performance. For example, CDT 7 sends augmented data into a causal transformer in the order of returns, cumulative costs, and observations predict actions. Saformer (2 leverages cost-related tokens and posterior safety checks to craft constraint-complian policies. SDT [20) employs signal temporal logic to define time-sensitive safety rules for agents. Despite these strides, predetermined cost targets would make actual cumulative costs converge to predetermined targets. Moreover, due to the masked attention mechanism, causal transformers inherently overlook either the cumulative cost or the return whichever appears later in the sequence.
Most offline safe RL algorithms are trained the DSRL dataset (2] is gathered from environ-
ments like Safety Gymnasium 28 However, there is a lack of datasets tailored for MOSRL.
3 Methods
Following CDT 7. we formulate MOSRL. as a probability distribution learning problem, in which agent (i 1,..., N and N is the number of agents) uses offline data to learn safe policy T : x A' + [], where S is the state space and A' is the action space.
For MOSRL, we propose the first algorithm, MOSDT. Following CDT (7). we adopt DT as the base network. Inspired by MADTKD [5], we also build the CTDE architecture through distillation. Specifically, for agent at time in centralized training, teacher network Tř regresses its action at from a historical trajectory T = (R1, C1, 01, al...., Re, Ct, ot), containing the global information of. R. = Σm r- is the system reward return (r- is the system reward at time and M is the maximum episode length). C. = ΣM Cr the system cumulative cost (c, is the system cost at time 7). is the global observation, and a, is the joint action ground truth. In decentralized execution, àt predicted by a student network T''s from partial information T = (R), Cj, of. af s..... Ri C,, of). For the tasks where environments offer only system rewards and costs (cf. Table in Appendix we set R = R₁ and C = C..
preceding
3.1 CTDE via policy self-distillation (PSD)
Given the intrinsic applicability of SD [3 for MARL, propose to utilize it to build the CTDE architecture. Student outputs, teacher outputs, and ground truths are aligned by three synchronous processes during training: decentralized student supervised learning (DSSL), centralized teacher supervised learning (CTSL), and policy self-distillation (PSD). Fig. 2 illlustrates the network structure of MOSDT.
Decentralized student supervised learning (DSSL) Each agent adopts an individual DSSL module based on CDT 7 to directly align the policy to ground truths. We propose to adopt full parameter sharing across all DSSLs (as detailed in Section 3.2) For agent at time t, the details of DSSL

---

PSD
MOSDT (at time t)
CTSL
Hs
F
0,c
Ps
DSSL
H
P
Pf
a
FRB.
Fo
Fatt
Fo.t
Student causal transformer
Teacher causal transformer
JRD.
fott
fac
CBE
F.t: global feature
R
C
B
a
Multiple agents
encoder
II: action head
PSD
Ft
PSD
Full parameter sharing
P: feature projector
DSSLI
DSSL
policy supervised learning
The network structure of the proposed MOSDT algorithm. The purple areas indicate our proposed innovative modules. The green areas represent data, while the blue areas represent networks. We enlarge one of the DSSL modules and display it in detail in the upper left cornr. We only illustrate the situation at time
Figure 2:
are shown in the "DSSL"" box in Fig. Each input is encoded into a feature representation by distinct encoder. In this encoding process, our proposed CBE module binarizes Ct as safety signal Bi and obtains reward-safety signal fused feature J' (as detailed Section 3.3 The encoded features fi, (* E {RB, o, a}) are refined by a student causal transformer. We use F" to denote the refined features. àt is generated by student action head H'S from the refined observation feature Fi Following MADTKD 5 we insert student feature projector P before H3s to perform feature distillation.
Supervised learning for student policies (the blue dashed circles at the top-left of Fig. 2 minimizes the following loss function:
M Σ 1=1
Loss's M log T'S (at Ti X/s
(T'))
where M is the training sequence length, and H (T's) weighted by learnable X/s is widely used entropy regularization term [36.
Centralized teacher supervised learning (CTSL) CTSL directly aligns teacher outputs with ground truths. At time f, the details of CTSL are shown in the "CTSL," box in Fig. 2 Inspired by success of SD in computer vision [33 14 15), integrate student networks into the teacher network decision making. To this end, we propose a new global information reconstruction scheme for SD in MARL, by summing up F across all agents to gain a global feature:
N
F.J.
F
as indicated by the purple straight arrows Fig. Similar to the DSSL module, a teacher causal transformer and teacher action head H7 (with teacher feature projector Pt.) are used to map the global feature F., into the action output áj. The teacher causal transformer is shared across agents, while Pi and H possess distinct parameters for each individual i.
keep Fí the computational graph of Eq. instead detaching to get better results (cf. Toble|TO in Appendix B.5). Through Eq. ( students serve as components of the teacher for teacher-student policy integration, and teacher able access global information Tr, achieving a centralized teacher training.

---

Supervised learning for the teacher policy (the blue dashed circles in the "CTSL, box in Fig. minimizes the following loss function:
LosST
Σ
log पं (at Tr) H (TT),
(3)
M
where N/. is the learnable weight for entropy regularization.
In execution, we only use student policies a's to predict actions a from partial information T/ decentralizing the execution.
Policy self-distillation (PSD) PSD aligns student outputs with teacher outputs, achieving novel in- tegration of SD and MARL. We bridge the performance gap between partially observable students and the teacher with global information access by aligning both their actions and feature representations. We minimize the following loss function:
Loss w WADKE (T's m/.) + wF Fo,s Forrll2:
shown by the purple curved arrows with "PSD" in Fig. 2 DEL (* *) is the KL divergence. Fo,s is the whole observation feature generated by the student feature projector Ps. and Forr is the whole observation feature generated by the teacher feature projector Pf. ||*|| is the Euclidean norm. WA and Wp are weights that indicate the importance of action and feature distillation, respectively (both are set to 0.5 with reference to MADTKD (5)). In Eq. (4). the teacher policy oh and the teacher projected feature F oor are detached from the computational graph so that PSD only updates student policies.
Following SD 13 we carry out PSD synchronously with DSSL and CTSL to streamline the training process. Specifically, we minimize the following total loss function:
N
Loss
Loss's + Loss + Loss
(5)
All loss functions are averaged across samples in mini-batch.
Network details Network details are consistent with those in CDT 7 if available. The data augmentation in CDT 77] is also adopted in MOSDT with the same settings. We use linear layers as the encoders. Timestamp embeddings are added to the encoded features to incorporate temporal information. Feature projectors
linear layers followed by GELU activation functions [37]. Each action head consists of 2 linear layers: one for computing mean values of action distributions and another for standard deviations (the latter is not used in execution). More network details of MOSDT are summarized Appendix A The algorithms the training and execution processes of MOSDT presented in Appendix A.3
3.2 Full parameter sharing among agents
To tackle the training instability problem caused by dynamic policy improvements among different agents and to improve model scalability in offline MARL, we propose to adopt full parameter sharing across all agents, for the first time. Specifically, each student shares identical parameters, as expressed in the following equation:
x =.....=
implying the equivalence of each network component across students throughout the training process. In practice, we only create one instance of the Student class for all agents.
In the entire MOSDT network, only the teacher action heads and the teacher feature projectors distinct for different agents.
3.3 Cost binary embedding (CBE)
Cost binary embedding (CBE) is straightforward and plug-and-play method (the purple dashed "CBE" box Fig. 2 that provides an intuitive alternative to specific cost targets and enhances information aggregation. CBE consists of cost binarization and safety signal embedding.
5

---

Cost binarization In DT-based offline safe RL methods 7 20], at the beginning of execution 1). R and C₁ need be assigned as targets. Most existing algorithms tend to generate trajectories with cumulative costs approaching the predetermined targets, leading to suboptimal and/or potentially unsafe actions along trajectories.
To tackle this issue, CBE applies fuzzification to cost targets, preventing the cumulative costs along actual trajectories from approaching predetermined specific values during execution. Towards this goal, in training we map Cj as a safety signal:
Bi - 50 if C S. otherwise,
where is the cost threshold (often fixed). "0" represents "safe", and "1" represents "unsafe". After training MOSDT with Eq. (8) we input B; = 0 to the model throughout execution, guiding the model yield safe trajectories. (Rt is set to a task-related large value as CDT 77).
Safety signal embedding CDT 7 separately processes R2 and C; by a causal transformer, which prevents the model from capturing their correlations during the processing of Rt until Cý starts being processed, due to the masked attention mechanism. To allow MOSDT to always access the prior information about the reward-cost correlation, we embed the safety signal feature fè, into the return feature fi, by concatenating them together. By slight abuse of notation, we use fhB, to denote the fused feature:
fNB.1 concatenate (fh,t JB..)
(8)
fR. and fB. are of 64 dimensions, half the size of fort and fa,t- Their concatenation results in 128-dimensional S ensuring the dimensional consistency throughout MOSDT.
3.4 MOSDB dataset
We introduce MOSDB, the first dataset and benchmark for MOSRL. To obtain diverse trajectory data, we train 2 online safe MARI methods, MACPO and MAPPO-L.agrangia, [27], across all 18 safe MARL tasks Safety Gymnasium 28 (except for tasks "Goal 0" series with constantly zero cost). For each task, training was conducted with 3 cost thresholds: 25 (the default), 15, and 5 (two lower values selected to ensure dense trajectory distributions in sub-threshold regions). All other training hyperparameters retained their default settings. The collected trajectory data is filtered by the density filter in the DSRL dataset (29] to maintain suitable densities over return-cumulative cost planes (cf. Fig. 5 in Appendix C MOSDB dataset comprises approximately x 107 data tuples,
occupying 18.7 GB on Linux.
The MOSDB benchmark contains 3 task sets:
MOS Velocity. Robots are required to move as quickly as possible while adhering to velocity constraints. Multiple agents need to control distinct body segments cooperatively.
MOS Goal. Each agent is required to reach its color-designated target while avoiding collisions and hazardous terrain.
MOS Isaac Gym. focuses on collaborative robotic tasks, such as coordinated ball- handovers between dual manipulators, with enforeed safety constraints joint movements.
4
Experiments
We use PyTorch to program on Linux. Experiments are conducted at an NVIDIA@ GeForce RTXTM 4090 D GPU (24 GB VRAM) with an AMD Ryzen 9 7950X CPU.
4.1 MOSDB benchmark and the performance of MOSDT
We use the MOSDB dataset to train MOSDT and all single-agent offline safe RL methods provided by the DSRL dataset [29]:
BBC 7 Behavior cloning.

---

Table MOSDB benchmark and the performance of MOSDT. Results are in the "return (cumulative cost)" format. The cost threshold e is 25 (consistent with the original online tasks), and the maximum Blue: Safe policies with the highest rewards.
return of each task is shown in Table 13 in Appendix C
Red: Unsafe policies. Due to space constraints, the sample standard deviation across multiple runs of each experiment shown in Table
in Appendix B.6
Task
BC
BCO-Lag B
BEAR-Lag
CDT
COpsDDIE [I
CPQ B
MOSDT (ours)
MOS Velocity
2x Swimmer
5.08 (7.27)
(11.33)
(9.33)
9:25 (12.70)
0.97 (12.13)
8.20 (13.30)
11.64 (20.33)
2xcHallCheetah
2043.94 (22 83)
2162.67 (62 A3)
(57,
2067.33 (40.03)
2029.00 00 (10.03)
406 106.90(8.99)
2052.64 (22.27)
Walkerdd
1512.75 (0.00)
1578.59 (10.17)
1515.60 (3.53)
1526.43 (3.67)
1540.41 (0.00)
315.22 (15.997
1584.87 (3.83)
2x1Ant
2361.63 (0.00)
2172.33
2217.89 (0.90)
2116.81
2125.14 (1.67)
-1488.45 (0.50)
2054.88
Hopper
31.63-00.30
40.20 831
82.24 (8.33)
27.15(0.000)
69:02 (7.40)
121.5% (3.0)
1122.23
4x2Ant
831.82 (0.00)
779.85 (0.00)
792.78 (3.00)
923.72 (0.00)
816.86
-466.22 (0.83)
2083.85 (3. (3.53)
Hallhhaaa
447.13 (0.00)
339.63 (0.03)
334,62 (0.00)
397 80 (0.00)
321.46 46 (0.00)
-201 40 (0.87)
1853.64 (.97)
91HHumanoid
575.73 (20.70)
(19.27)
(18.57)
554.4 (15.27)
(18.53)
MOS Goal
Multi-Ami
23:41 (17.00)
15.78 (23.85)
21,31 (9.50)
28.44 (10.6)
33.75 (14.00)
061 (0.00)
38 (14.50)
Muhti-An22
2.59 (8.00)
(21.02)
2:71 (16.17)
4.93 (17.500 50)
3.55 (17.33)
0.51 (0.00)
2.96 (7.50)
Multi-Point
6.47
-1.43 (25.48)
4.28 (7.67)
9.25 (14.83)
2.11 (8.17)
2.84 (7.67)
9.65 (12.67)
Multi-Poim22
-0.06 (4.33)
-8.59 (36.53)
(14.00)
(19.53)
-1,23
0,95 (9:67)
-1.88 (21.00)
MOS Isnae Gym
CloteDmeerMMA
(1.00)
-5.15 (0.80)
4.49 (3.57)
-3.45 (0.00)
-5.28 (0.00)
-5.43 (0.00)
-3,45 (0.00)
PicLAndPlacceM
(3.73)
4.92 (0.27)
..26 (0.00)
(0.000
(2.67)
(2.47)
-2.67 (0.00)
CatchiFingerMAA
(2.60)
(0.00)
(0.00)
CarcJoiniMA
0.21 (0:00)
0.19 (0.00)
0.20 (0.00)
0.26 (0.80)
0.21 (0.00)
0.15 (5.977
0.31 (0.00)
OverFingerMA
0.43 (0.00)
0.46 (0.0F)
0.44 (0.00)
0.52 (0.00)
0.44 (0.00)
0.39 (8.00)
0.52 (0.00)
OverlointMA
0.45 (0.00)
0.47 (0.00)
0.46 (0.00)
0,46
0.44 (0.00)
0,42 (8.00)
0.47 (1.03)
Summary
(safe)
(unsafic)
(unsafe)
4SSOAA (unsafe)
(safe)
0 (safe)
SOTA (safe)
BC-Safe
7
Behavior cloning using only safe trajectories (results are presented in Table
in Appendix B.T.
BCQ-Lag
g
A BCQ
38 based method incorporating cost thresholds with PID-
Lagrangian
39
BEAR-Lag
8 : A BEAR
40 bbased method that deals with safety constraints by PID-
Lagrangian
339
CDT 77) A DT [8 -based method that sends augmented data into causal transformer in
the order of returns, cumulative costs, and observations to predict actions.
COptiDICE [41]: OptiDICE
[42 -based method using the Lagrangian approach to main-
tain safety.
CPQ [3: A constrained Q-updating method that incorporates penalties for unseen or unsafe actions.
Baseline algorithms are trained within the CTCE framework, accessing global information and thus yielding better performance. All methods are trained for 10yh steps, with evaluations conducted at 40 checkpoints during training. Each evaluation is averaged over 10 interaction episodes with Safety repeated across 3 fixed random seeds. All above hyperparameters are consistent with those in the DSRL dataset [29]. More training and evaluation settings are presented in Appendix A.2 We report the highest returns achieved under the cost threshold as final performance. For unsafe policies, we report the returns with the lowest cumulative costs. Table summarizes the MOSDB benchmark and the performance of MOSDT.
Gymnasium (28) Experiments
While the strong baseline methods leverage global information, MOSDT utilizes only partial in- formation and achieves the highest returns on 14 out of 18 tasks (across all base environments including MuJoCo [24), Safety Gym (25). and Isaac Gym (26) while guarantceing safety on all tasks, demonstrating its outstanding capability in balancing return maximization and risk control. MOSDT delivers very strong performance tasks with more than three agents, "3x "3xpeer""r", "4x2Ant", and "6x HallChhhhhttt"" indicating remarkable capability to address multi-agent challenges. MOSDT attains SOTA returns across all challenging "MOS Isaac Gym" tasks with high- nigh-dimensional action spaces (cf. Table 13 in Appendix C). demonstrating its superiority in making complex decisions.
We evaluate the parameter efficiency of MOSDT against base model CDT 7 by averaging their total/execution parameter counts across all tasks. The parameter counts of CDT 7 MOSDT, and MOSDT variants are presented in Fig- 3 Due to the lightweight student network design and the full
parameter sharing, MOSDT requires only 65% of the execution parameter count of CDT [7] while
7

---

Table 2: Ablation study results. Results are in the "return (cumulative cost)" format. The cost threshold is 25 (consistent with the original online tasks). The maximum return of each task is shown in Table 13 in Appendix C "Il": Policies with lower returns than full MOSDT or unsafe policies. Red: Unsafe policies. "Partial PS": Without parameter sharing for student action heads. "No PS No parameter sharing. "w/o SF Without safety signal embedding. Due to space constraints, the
sample standard deviation across multiple runs of each experiment is shown in Table 12 in Appendix
B.6
Task
MOSDT
PSD
Partial PS
CBE
MOS Vellcity
2x Swimmer
11.44(20.30
11.58 (9.40)
10.97 (15.33)
(4.22
10:40 (16.80)
11:08 (23.00)
2x3lHalchheetah
2052.64 (22.27)
2081.00 (39.83)
2050. (23.00)
2013.23 (22.13)
2024.48 (22.03)
( 2009.30 (20.63)
Wallaleer2d
1584.87 (3.83)
11776.03 (4.23)
1585,52 (2.07)
1597.48 (2.03)
1577.00 (2.93)
0 1568..7 (6.77)
2x4Ant
2054.88 (0.87)
2033.22 (2.97)
2049 (1.73)
2059.04 (7.90)
2008.38 (6.
(2.93)
3x 3olpereerr
1122.23 (4.00)
1239 49 (10.67)
110.93 (1.33)
29.26 (0.00)
1115.51 (13.60)
1112.48 (6.03)
4v2Ant
2083.85 (3.53)
2189.01 (5.57)
962.59 (0.00)
914.51 (0.00)
2128..66 (3.43)
2070.38 (8.00)
6allHaalhheetah
1853.64 (21.97)
1823.33 (27.27)
J 1169.26 (0.10)
528.80 (0.00)
1835,89 (22.933
0 1823.27 (22.53)
918Humanoid
444.71 (22.80)
477.70
474.87 (21.73)
482.17 (21.60)
453.52 (21.67)
456.21 (21.87)
MOS Goal
Multi-Antl
38.38 (14.50)
37.79 (20.33)
I) 36.87 (10.50)
33,91 (8.83
40.13 (19.33)
30,93 (16.50) 50)
Multi-An12
(11.83)
2.29
2.28
Multi-Point]
9.65 (12.67)
13.09 (14.67)
10:42 (14, 83)
14.13 (18.7))
10.87 (12.33)
I 6,90 (12.50)
-1.08 (21.00)
4222.17 (11.50)
0.33 (7.83)
1,18 (16.67)
2.84 (17.00)
-2.20 (8.67)
MOS Isane Gym
CroseDra
-3.45 (0.00)
1|73.74 (0.00)
-3446 00.46.0.000 (0.00)
-3.32 (0.00)
-3.49 (0.00)
-3.21 (0.00)
PickAndPlaceMA
(0.00)
(0.00)
-2.24 (0.00)
-2.00 (0.00)
.2.34 (2.67)
22277 (0.00)
CachhinnerMA
0.25 (6.33)
[| 0.21 (4.40)
[| 0:23 (5.33)
(3.3
0,22 (2.87)
| 0.21 (7.40)
CarchJoinMMA
0.31 (0:00)
0.23 (0.00)
0.27
0.266 (0.0000.00)
0.23 00)
OverFingerMA
0.52 (0.00)
0.45 (0.93)
0.48 (0.00)
0.44 (0.00)
0.45 (0.00)
(3.53)
OverJointMA
(1.03)
J) 0.44 (0.30)
0.46 (0.00)
0.45 (0.00)
0.45 (0.00)
(5,17)
Summary
(sali)
12 (j (unsafe)
12 () (safe)
11 1 (safe)
12 (safe)
14 U (safe)
achieving better performance. Therefore, MOSDT is well-suited for memory-constrained application scenarios, such as micro-robots.
MOSDT consumes less execution time than CDT 7 owing the lightweight design of the student network. MOSDT performs an inference for one agent in 0.80 milliseconds on average, compared to 0.86 milliseconds for CDT [7]. Detailed efficiency analysis is presented in Appendix B.2
4.2 Ablation study
We conduct massive ablation experiments to evaluate the effectiveness of each component in MOSDT. The training and evaluation settings are consistent with those in Section 4.1 Table@]summane the ablation study results.
1.6
Training parameters
1.49
Number of parameters (M)
1.4
Execution parameters
1.2
1.17
1:13
1.0
0.8
a.76
0.76
0,76
The effect of PSD To analyze the effectiveness of PSD, we remove it by minimizing only Eq. ( dur- ing training, converting MOSDT into decentralized training with decentralized execution (DTDE) archi- tecture. The "w/o PSD" column of Table 2 shows the critical role of PSD. Its absence degrades returns on 12 out of 18 tasks and renders the model unsafe. Given that safety is a fundamental requirement in MOSRL, PSD is vital for MOSDT.
0.6
0.47
0.47
0.47
0.4
0.2
0.0
CDT
MOSDT Using KD No w/o CBB Method
Figure 3: Number of parameters. "Using
KD": Using conventional KD instead of PSD. "No PS": No parameter sharing. MOSDT only requires 65% of the execution param- eter count of CDT. PSD reduces the train- ing/execution parameter counts by 47%/58%
We further verify the effectiveness of the action distil- lation and the feature distillation (the two terms of Eq.
44 finding that removing either impairs performance B.33)
(cf. Table 6 in Appendix B.3).
To analyze the efficiency improvements provided by PSD, we train MOSDT by the conventional KD used
MADTKD [5) for comparison. In this setting, policy distillation is performed after the completion of centralized teacher training. The comparison between the "MOSDT" column and "Using KD
8

---

(a)
Reward return
(b)
Reward retur with no PS
(c)
Cumulative cost
600
27.5
Threshold
Reward return
Reward return
25.0
Cumalative cast
20
400
22.5
10
200
MOSDT
20.0
MOSDT
No PS
No PS
0-
0-
17.5
75 100
50
75
100
100
Training step (k)
Training step (k)
Training step (k)
Figure 4: Visualization of the training on the "3x "3xpHr"per" task. "No PS": No parameter sharing. The purple curve in Subgraph (a) is enlarged and displayed in detail in Subgraph (b) for better viewing. Shaded areas represent sample standard deviations across multiple runs. We adopt the smoothing in TensorBeard to better view time-series data trends.
column in Fig. reveals that PSD reduces the number of training parameters by 35% compared to using KD, due to the proposed teacher-studee policy integration. In addition, PSD decreases the average training time by 24% (cf. Table Appendix B.2 through the synchronization of policy supervised learning and policy distillation.
The effect of full parameter sharing To verify the effects of full parameter sharing, we train MOSDT with parameter sharing. The Comparison between the "MOSDT" column and "No PS column of Table
that full parameter sharing enhances returns in 11 out of 18 tasks without compromising safety. We observe significant improvements on tasks involving least three agents in MuJoCo environment [44:: : a 38.4-fold increase on the "3x Hoppprr" task with three agents, a 2.3-fold increase on the "4x2Ant" task with four agents, and a 3.5-fold increase on the "6x HalfCheetah' task with six agents.
To verify the role of full parameter sharing in stabilizing training, we visualize the training process on the "3x Hopper" task with full and no parameter sharing in Fig. 4 Fig. (a) shows a constantly rising return curve with full parameter sharing, contrasting with persistently low return curve without it. Focusing on the case without full parameter sharing in Fig. (b), the return curve exhibits fluctuation with sharp decline in the early training stage, indicating instability. Additionally, the absolute variation coefficient across multiple runs is 0.27 and 0.35 (averaged over all tasks) with full and no parameter sharing, further showing the stabilizing effect of full parameter sharing.
Fig. (c) compares the cumulative cost curves with full and no parameter sharing. With full parameter sharing, MOSDT engages in more aggressive exploration during training, achieving effective return-cost tradeoff.
To benchmark against partial parameter sharing, we keep the sharing of all parameters except the student action heads H-- The result is shown in the "Partial PS" column of Table indicating that performance degradation occurs on 12 out of 18 tasks. Training MOSDT without parameter sharing for any one of the network components causes performance loss on 12 to 14 (out of 18) tasks (cf. Table 8 in Appendix B.4).
In Fig. 3 the comparison between the "MOSDT" and "No PS" columns shows that full parameter sharing substantially reduces the number of training parameters by 47% and the number of execution parameters by 58%, contributing the scalability MOSDT. Full parameter sharing introduces no additional time consumption in both training and execution (cf. Table Appendix B.2
The effect of cost CBE To demonstrate the efficacy of CBE, we set cost targets as the given threshold and separately send return features and cumulative cost features into the causal transformer as CDT [7]. The comparison between the "MOSDT" and "w/o CBE" columns in Table reveals that CBE increases returns on 14 out of tasks, covering all base environments including MuJoCo 24 Safety Gym 25), and Isaac Gym (26). In addition, CBE introduces no additional parameter requirement or computational burden (cf. Appendix B.2). As a plug-and-play method, CBE can be
seamlessly integrated into existing safe RL algorithms.

---

The comparison between the "MOSDT" and "w/o SE" columns in Table shows that the safety signal embedding boosts returns on 12 out of 18 tasks. It enables MOSDT to always access the prior information of correlations between returns and cumulative costs, avoiding MOSDT overlooking either of them (whichever arises later the input sequence).
According to the training curves (cf. the link in the abstract), CBE has three main advantages on training dynamics:
Stabilizing training. On most tasks (e.g., "2x3HalfCheetah, "2x3Walker2d"" and "3x "xopeer"""", CBE results in smoother learning curves, compared with MOSDT with- out CBE. CBE mitigates volatility and reduces sharp declines. The cost binarization in CBE eliminates the noise in the cost information, thereby stabilizing the training.
Better training initialization. In most tasks (e.g., "2x4Ant", "4x2Ant", and "Multi-PointI") CBE yields higher returns with safe cumulative costs at the beginning of training. CBE directly transmits the prior information about the reward-cost correlation MOSDT by embedding safety signals into returns, enabling agents to learn more efficiently. The cost binarization in CBE eliminates the noise of cost data, improving data efficiency, thereby offering a better training initialization.
Better safety control. In most tasks (e.g., "2ximmimm"r", "2x4Ant", and "OverFingerMA"", CBE yields cumulative cost learning curves with faster and more stable convergence. For 11 out of 18 tasks, CBE reduces the cumulative costs (cf. the ablation study results in Table 2). CBE provides better safety control by improving the processing of cost information.
Summary PSD is indispensable for safety, while full parameter sharing and CBE deliver the most substantial and the broadest return enhancements, respectively. These three components together enable MOSDT to address the challenges of multi-agent cooperation, offline learning, and safety assurance, achieving SOTA overall performance on MOSDB. Moreover, they reduce parameter counts and time consumption instead of introducing additional overhead.
4.3
Analysis of unsafe cases
There are three major causes for the violation of safety constraints:
High risk in the early stage of training. As shown in the training curves (cf. the link in the abstract), most failure cases show excessive cumulative costs at the initial stages of training. For example, on the "2x3HalfCheetah" task (the cost limit is 25), when 10% of the training (10k steps) is completed, the complete MOSDT only yields a cumulative cost of 13.1, while the MOSDT without PSD yields cumulative cost of 18.8, and CDT 7 yields a cumulative cost of 62, resulting the violation of safety constraints of the latter two methods (cf. Tables] I 2nd
Instability of the cumulative costs in training. In most failure cases, the training curves of cumulative costs show severe fluctuations (e.g.. the training curves of BCQ-Lag [8] on the "Mulii-Pointe task). This instability indicates that the model often enters unsafe states and cannot maintain safety.
Lack of communication among agents. Maintaining global safety requires cooperation among agents. Lack of communication among agents could cause them violate safety constraints, For example, in the "2x3HalfCheetah" task, rollout visualizations of the MOSDT without PSD (i.e., lacking centralized training) revealed that the agent's hind leg persistently accelerated, while its front leg remained stationary to decelerate. This uncoordinated action led to over-speeding and eventual task failure (cf. Table 2).
Conclusion
This work lays the foundation for MOSRL, providing its first algorithm MOSDT with the first dataset and benchmark MOSDB. Leveraging a novel architecture integrating PSD, full parameter sharing, and CBE, MOSDT achieves a superior tradeoff among returns, safety, and scalability on MOSDB, compared to SOTA single-agent offline safe RL methods (in the CTCE framework).
10

---

References
[1] R. F. Prudencio, M. R. Maximo, and E. L. Colombini, "A survey on offline reinforcement learning: Taxonomy, review, and open problems," IEEE Transactions Neural Networks and Learning Systems, 2023.
[2] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, "Direct preference optimization: Your language model is secretly reward model," Advances in Neural Information Processing Systems, vol. 36, 53 728-53 741, 2023.
[3] S. Sinha, A. Mandlekar, and A. Garg. "S4rl: Surprisingly simple self-supervision for offline reinforcement learning in robotics," " in Conference on Robot Learning. PMLR, 2022, pp- 907-917.
[4] G. Zhang, C. Zhang, W. Wang, H. Cao, Z. Chen, and Y. Niu, "Offline reinforcement learning control for electricity and heat coordination in a supercritical chp unit," Energy, vol. 266, p. 126485, 2023.
[5] W.-C. Tseng, T.-H. J. Wang, Y.-C. and P. Isola, "Offline multi-agent reinforcement learning with knowledge distillation," Advances in Neural Information Processing Systems, vol. 35, pp. 226-231, 2022.
[6] Y. Yang, X. Ma. Ci. Z. Zheng, Q. Zhang, G. Huang, J. Yang, and Q. Zhao, "Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning," Advances in Neural Information Processing Systems, vol. 34, pp. 10 299-10312, 2021.
[7] Z. Liu, Z. Guo, Y. Yao, Z. Cen, W. Yu, T. Zhang, and D. Zhao, "Constrained decision transformer for offline safe reinforcement learning," in International Conference on Machine Learning. PMLR, 2023, pp. 21 61 1-21 630.
[8] H. Xu, X. Zhan, and X. Zhu, "Constraints penalized q-learning for safe offline reinforcement learning," in Proceedings of the AAAI Conference Artificial Intelligence, vol. 36, no. 8, 2022, pp. $753-8760.
[9] Y. Chen, C. Ji, Y. Cai, T. Yan, and B. Su, "Deep reinforcement learning in autonomous car path planning and control: A survey," arXiv preprint arXiv:2404.00340, 2024.
[10] X. Chen, G. Qu, Y. Tang, S. Low, and N. Li, "Reinforcement learning for selective key applications in power systems: Recent advances and future challenges," IEEE Transactions on Smart Grid, vol. 13, no. 4, pp. 2935 2958, 2022.
[11] B. Singh, R. Kumar, and V. P. Singh, "Reinforcement learning in robotic applications: a comprehensive survey," Artificial Intelligence Review, vol. 55, no. 2, pp. 945-990, 2022.
[12] G. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in neural network," arXiv preprint Xiviv:550330000 0,,31, 2015.
[13] L. Zhang, J. Song, A. Gao, J. Chen, C. Bao, and K. Ma, "Be your own teacher: Improve the performance of convolutional neural networks via self distillation." in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 3713-3722.
[14] L. Zhang, C. Bao, and K. Ma, "Self-distillation Towards efficient and compact neural networks." IFEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 8, pp. 4488, 2021.
[15] Z. Allen-Zhu and Y. Li, "Towards understanding ensemble, knowledge distillation and self- distillation in deep learning," arXiv preprint arXiv:2012.09881 2020.
[16] z. Zhu, M. Liu, L. Mao, B. Kang, M. Xu, Y. Yu, S. Ermon, and W. Zhang, "Madiff: Offline multi- agent learning with diffusion models," Advances in Neural Information Processing Systems, vol. 37, pp. 4177-4206, 2024.
[17] P. Barde, J. Foerster, D. Nowrouzezahrai, and A. Zhang, "A model-based solution to the offline multi-agent reinforcement learning coordination problem." arXiv preprint 2xXv:23055111999 2023.
[18] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and Mordatch, "Decision transformer: Reinforcement learning via sequence modeling," Advances in neural information processing systems, vol. 34, pp. 15 085-15 202,
11

---

[19] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention all you need," Advances in neural information processing systems, vol. 30, 2017.
[20] 7. Guo, W. Zhou, and W. Li, "Temporal logic specification-condiiioned decision transformer for offline safe reinforcement learning," " arXiv preprint wxXi:2242.172117 2024.
[21] Q. Zhang, L.. Zhang. H. Xu, I.. Shen, B. Wang. Y. Chang, X. Wang, B. Yuan, and D. Tao, "Saffomer: A conditional sequence modeling approach to offline safe reinforcement learning." arXiv preprint arXiv:2301. 12203, 2023.
[22] Z. Zhu, M. Liu, L.. Mao, Kang, M. Xu, Y. Yu, S. Ermon, and W. Zhang. "Madiff: Offline multi- agent learning with diffusion models," Advances in Neural Information Processing Systems, vol. 37, pp. 4177-4206, 2024.
[23] Q. Tian, K. Kuang, F. Liu, and B. Wang, "Learning from good trajectories in offline multi-agent reinforcement learning," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 10, 2023. pp. 11 672-11 680.
[24] E. Todorov, T. Erez, and Y. Tassa, "Mujoco: A physics engine for model-based control," in 2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2012, pp. 5033.
[25] A. Ray, J. Achiam, and D. Amodei, "Benchmarking safe exploration in deep reinforcement learning," arXiv preprint 1rXiv:1910.01708 vol. 7, no. 1, p. 2, 2019.
[26] V. Makoviychuk. L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa et al., "Isaac gym: High performance gpu-based physics simulation for robot learning," arXiv preprint 2 10470, 2021.
[27] S. Gu, J. G. Kuba, Y. Chen, Y. Du, L. Yang, A. Knoll, and Y. Yang. "Safe multi-agent reinforcement learning for multi-robot control," Artificial Intelligence, vol. 319, p. 103905, 2023.
[28] J. Ji, B. Zhang, J. Zhou, X. Pan, W. Huang. R. Sun, Y. Geng, Y. Zhong, J. Dai, and Y. Yang, "SSfftty gymnasium: A unified safe reinforcement learning benchmark," Advances in Neural Information Processing Systems, vol. 36, pp. 18 964-18 9933, 2023.
[29] Z. Liu, Z. Guo, H. Lin, Y. Yao, J. Zhu, Z. Cen, H. Hu, W. Yu, T. Zhang, J. Tan al., "Datasets and benchmarks for offline safe reinforcement learning," arXiv preprint 1rXiv:2306.09300 2023.
[30] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg. M. Lanctot, N. Sonnerat, J. Z. Leibo. K. Tuyls et al., "Yalue-decomposition networks for cooperative multi-agent learning," arXiv preprint arXiv:1706.000 072066 2017.
[31] K.-H. Lee, O. Nachum, M. S. Yang. L. Lee, D. Freeman, S. Guadarrama, I. Fischer, W. Xu, decision transformers." Advances in Neural
E. Jang, H. Michalewski et al., "Multi- Information Processing Systems, vol. 35, pp. 27 921-27 936, 2022.
[32] Y.-H. Wu, X. Wang, and M. Hamaya, "Elastic decision transformer," Advances in neural information processing systems, vol. 36, pp. 18 582-1888 550, 2023.
[33] A. Correia and L. A. Alexandre, "Hierarchical decision transformer." in 2023 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS). IEEE, 2023, pp. 1661-1666.
[34] Y. Chen, H. Mao, J. Mao, S. Wu, T. Zhang, Zhang, W. Yang, and H. Chang, "Ptde: Personal- ized training with distilled execution for multi-agent reinforcement learning," arXiv preprint arXiv:2210.cm 00.8872, 2022.
[35] J. Liu, C. Xu, P. Hang, J. Sun, M. Ding, W. Zhan, and M. Tomizuka, "Language-die driven policy distillation for cooperative driving in multi- agent reinforcement learning,"" IEEE Robotics and Automation Letters, 2025.
[36] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor," in International conference on machine learning. Pmir, 2018, pp. 1861-1870.
[37] D. Hendrycks and K. Gimpel, "Gaussian error linear units (gelus))" arXiv preprint arXiv:1606.08411,,2 2016.
12

---

[38] S. Fujimoto, D. Meger, and D. Precup, "Off-policy deep reinforcement learning without explo- ration," " in International conference on machine learning. PMLR, 2019, pp. 2052-2062.
[39] A. Stooke, Achiam, and P. Abbeel, "Responsive safety in reinforcement learning by pid lagrangian methods," in International Conference on Machine Learning. PMLR, 2020, pp. 9133-9143.
[40] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine, "Stabilizing off-policy q-learning via bootstrapping error reduction," " Advances in neural information processing systems, vol. 32, 2019.
[41] J. Lee, C. Paduraru, J. Mankowitz, N. Heess, D. Precup, K.-E. Kim, and A. Guez, "Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation," " arXiv preprint arXiv:2204.08957, 2022.
[42] J. Lee, W. Jeon, B. Lee, J. Pineau, and K.-E. Kim, "Optidice: Offline policy optimization via stationary distribution correction estimation," in International Conference on Machine Learning. PMLR, 2021, pp. 6120-6130.
[43] J. L.. Ba, R. Kiros, and G. E. Hinton, "Layer normalization," arXiv preprint 107:v:1607.06450, 2016.
[44] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: simple way to prevent neural networks from overfitting, " The journal of machine learning research, vol. 15, no. I, pp. 1929-1958, 2014.
[45] D. P. Kingma, "Adam: A method for stochastic optimization." " arXiv preprint arXiv:1412.6980, 2014.
[46] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
[47] S. Gu. L. Yang. Y. Du, G. Chen, F. Walter, J. Wang, and A. Knoll, "A review of safe reinforce- ment learning: Methods, theories and applications," JEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly outline our motivations, approaches, and results, aligning closely with the paper's actual contributions without overstatement. The points in the abstract and introduction are supported by the detailed descriptions and experimental results in the following part of the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include a "Limitations and future work" section in the appendix to discuss the key limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper focuses on empirical contributions in multi-agent offline safe reinforcement learning rather than theoretical results. No formal theorems, proofs, or assumptions are presented.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides all the information needed to reproduce the main experi- mental results by directly stating or citing in the main text and the appendix, rendering complete reproduction. The open access to our code and data also ensures reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The paper explicitly states that "Code, dataset, and results are available at this website" in the abstract, indicating providing open access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https wttp:/ii/ggiiee//eeeeeeeeeeeeeee httpc//gguiieeeeeeSuuuutiiiii www.oodeuumiiisoonn....
/ //nips.
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The paper details all the training and evaluation details necessary understand the results by directly stating or citing in the main text and the appendix. The open access to the code and data also ensures complete details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We detail the sample standard deviations of all results from tables in the appendix. In addition, Fig. 4 notes that "Shaded areas represent sample standard deviations across multiple runs", showing statistical variability.
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative етгог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides the information of GPU and CPU, and the time consump- tion of training and execution in the main text and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that NeurlPS Code of Ethics httppp///.eourr.sccccco///////tt
didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the httpeeerrrrrpeepppeereeaeeeaeaae
Answer: [Yes]
The paper focuses on technical advancements multi-agent offline safe reinforcement learning without involving sensitive data, human subjects, methods that raise ethical concerns, or other components that may violate the NeurIPS Code of Ethics.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
17

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper lists several potential positive applications in the introduction and also explicitly states that "MOSRL may be used for negative applications such drone surveillance" in the "Limitations and future work" section in the appendix, indicating both potential positive societal impacts and negative societal impacts of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The paper explicitly states that "Upon closer inspection, no data at high risk of such misuse is observed in MOSDB" in the "Limitations and future work" section in the appendix, providing a safeguard for the misuse of the proposed MOSDB dataset, such as the drone surveillance.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
18

---

Answer: [Yes]
Justification: We build our model and dataset based on several existing code repositories on GitHub. We properly cite their papers and strictly respect their licenses, most of which are Apache License 2.0.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersaippettteee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- e-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide detailed guidance documents for using our code and dataset the website of our open-access.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper involves no crowdsourcing or human subjects, focusing solely on algorithmic and dataset development in multi-agent offline safe reinforcement learning.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
19

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No human subjects are involved, so IRB approvals or equivalent are not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration not required.
Answer: [NA]
Justification: LLMs are not parts of our core methodology. We use LLMs only for writing, editing, or formatting purposes.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy pettps://neurjje cwtpuriisscc/Cooffeee / /LLM) for what should or should be described.
20

---

Appendix
A
More implementation details
A.1 Network details of MOSDT
Each linear layer is a single-layer fully connected network. The linear layers for encoding returns and safety signals are 64-dimensional, and other linear layers are 128 dimensional. MOSDT contains 3 causal transformer layers with 128 dimensions and 8 attention heads. Layer Normalization (LN) 43 and Dropout 44 (drop ratio 0.1) are added before the student causal transformer. We adopt INs subsequent to both student and teacher causal transformers.
The features before the student causal transformer
arranged in ordered sequence JRBB M Jo,M- fa, where M the training sequence length. The refined feature sequence is divided into 3 subsequences (f., f...))))).))) (* E (RB,o,
A.2 More training and evaluation settings
For baseline methods, we adopt the same training and evaluation hyperparameters as DSRL 22]] except for the batch size, which is set to 64 in all baseline methods (so as the proposed MOSDT) to reduce training time and yield better performance. For MOSDT, we adopt the same hyperparameters as CDT (one of the baseline methods). The learning rate is 1 x 10 We use the Adam optimizer with 31 0.9 and B2 0.999. The value of Weight Decay
x 10-4, We clip gradients to maximum value of 0.25. We take the first 500 training steps warm-up [46]
45
Following HDT 33 evaluation, let CDT 7 and MOSDT perform each execution twice, with the return target of the maximum value in training data and half of the maximum value (the better result recorded as the final performance). Unlike HDT, we do not large return targets they
degrade performance.
A.3 MOSDT algorithm
Algorithm I and Algorithm 2 summarize the training and execution processes MOSDT, respec- tively.
Algorithm MOSDT training algorithm
Require: Offline dataset D 1: for each training step do
2: Sample trajectory mini-bated B D. B contains Ty (R)
Ris, CM d3s))
and ais. where M the training sequence length
3: for each agent do
4: Map Cj to Bi by Eq.
5:
Generate student features F and action estimations aj by student network T/.
6:
Compute the student supervised learning loss Loss's by Eq.
7:
end for
8:
Generate global feature F.,, by Eq.
9:
Generate action estimations af by teacher networks / Compute the teacher supervised learning loss Loss,, by Eq. 3
10:
11:
Compute the PSD loss Loss), by Eq. (4)
12:
Update ग. (full parameter shared across all agents) and पे by minimizing the total loss in
Eq. (5)
13: end for
B Further experimental results
The training and evaluation settings are consistent with those in Section 4.1
21

---

end for Algorithm 2 MOSDT execution algorithm
Require: Trained agent policies T environment Env)
1: Get an initial observation: of + Env.reset()
2: Initialize an input sequence 'i' <- (R) = R Bi - 0. where R' is predetermined large
return targets (task-related).
1, (Mt is the maximum episode length) do 4: for each agent do
3: for 1= 1,
5: Predict action àj from T; by the trained policy TT'
6: end for
7:
Execute the action: 0'41,7i, ci t- Env.step (a),.... ( a,)
8:
Compute input sequences for the next step:
9:
TH (R/+1 R r/, B/ 0,04+1)
10: if Envteerminated(> = True then
11: Break
12: end
13:
B.1 The performance of BC-Safe
Table I presents the performance of the methods trained on the entire MOSDB dataset. The perfor- mance of BC-Safe (trained by only safe trajectories) is shown in Table It achieves SOTA return on no task.
Table 3: The performance of BC-Safe on the MOSDB dataset. Results are in the "return sample standard deviation (cumulative cost + sample standard deviation)" format. The cost threshold is 25, and the maximum return each task is shown Table 13 in Appendix
Task
BC-Safe 7
MOS Velocity
Swimmer
+ 1.05 (2.30 + 1.97)
2x3HallCheetah
2041.64 = 13.71 (19.60 2.6)
2x3Walker2d
1562.08 + 42.31 +
2x4Ant
2406.86 = (0.00
3x 3x1Hopper
28.16 5.96
4x2Ant
803.14 54.20
6x HaaahChheet
407.18 1: 61.72 61.72
Hummmadd
563.67 40.13 (20.37
MOS Goal
Multi-Ant1
17.18 = 4.58 41..
Multi-Ant2
Multi-Poine
5.53 2.68 (10.33 9.61)
Multi-Point2
0.83 0.75 (12.33 11.79)
MOS Isaac Gym
CloseDrawerMA
-5.23 = (1.00 + 1.73)
PidPaacae
-5.72 0.57 (3.73 4.03)
CatchFingerMA
0.19 - 0.03 (0.00 = 0.00)
CatchJointMA
0.21 + 0.01 (0.00 = 0.00)
OverFingerMA
0.43 = 0.01 (0.00 = 0.00)
OverJointMA
0.45 0.02 (0.00 + 0.00)
Summary
SOTA (safe)
B.2 Detailed efficiency analysis
Table summarizes the parameter counts of CDT M7. MOSDT, and MOSDT variants. Table 5 summarizes their time consumption for training and execution. Compared with CDT 77, MOSDT requires only 65% of the execution parameters and consumes less execution time while achieving better performance. MOSDT needs about twice the training time of CDT. PSD, full parameter sharing, and CBE reduce parameter counts and time consumption rather than introducing additional overhead.
22

---

Table The details of parameter counts. Results are in millions (M). "TP": Total parameter count.
"EP": Execution parameter count. "Using KD": Using conventional KD instead of PSD. "No PS" No parameter sharing.
CDT
MOSDT
Using KD
No PS
w/o CBE
Task
TP
EP
TP
EP
TP
EP
TP
EP
TP
EP
MOS Velocity
2x1Swimmer
0.7315
0.7287
0.7771
0.3468
0,9102
0.3468
1.1238
0.6935
0.7774
0.3470
2x3HaliCheetah
0.7377
0.7326
0.7801
0.3487
0.9159
0.3487
1.1287
0.6974
0.7803
0.3489
Wx3Waller2d
0.7377
0.7326
0.7801
0.3487
0.9159
0,3487
1.1287
0.6974
0.7803
0,3489
2x4Antt
0,7436
0.7359
0.7822
0,3503
0,9209
0,3503
1.1326
0.7007
0.7825
0,3506
3x Hopper
0.7376
0.7319
0.7944
0,3473
0.9304
0,3473
1.4889
1.0418
0.7946
0.3475
4x2Ant
0.7606
0.7443
0.8147
0,5481
1.3584
0,5481
2.4590
2.1924
0,8150
0.5484
6x Halfhheatt
0.7634
0.7454
0.8458
0,5467
1,3911
0,5467
3.5793
3,2801
0.8461
0,5469
918Humanoid
0.9265
0.8287
0.8314
0.5952
1.4571
0.5952
1.4266
1.1904
0.8317
0.5955
MOS Goal
Multi-Antl
0.8398
0.7853
0.8090
0.5733
1.3916
0,5733
1,3823
1.1467
0.8093
0.5736
Multi-Ant2
0.8398
0.7853
0.8090
0.5733
1.3916
0,5733
1.3823
1.1467
0.8093
0.5736
Multi-Pointl
0.8063
0.7663
0.7964
0,5638
1.3631
0,5638
1.3603
1.1277
0.7967
0,5641
Multi-Point2
0.8063
0.7663
0.7964
0.5638
1,3631
0.5638
1,3603
1.1277
0.7967
0.564
MOS Isaac Gym
CloseDrawerMA 0.6264
0.6146
0.6623
0.4261
1.0749
0.4261
1.0885
0.8523
0.6626
0.4264
Pic
0.6290
0.6158
0.6630
0.4268
1.0769
0.4268
1.0897
0.8535
0.6632
0.4270
CatchFingerMA
0.7334
0.6756
0.6997
0.4547
1.1630
0.4547
1.1544
0,9094
0.6999
0,4550
CatchJointMA
0.7334
0.6756
0.6997
0.4547
1.1630
0.4547
1.1544
0,9094
0.6999
0,4550
OverFingerMA
0.7226
0.6679
0.6927
0.4509
1.1514
0,4509
1.1436
0,9017
0.6930
0,4511
OverJointMA
0.7226
0.6679
0.6927
0,4509
1.1514
0.4509
1.1436
0.9017
0.6930
0,4511
Average
0.7554
0.7223
0.7626
0.4650
1.1717
0.4650
1.4293
1.1317
0.7629
0.4653
B.3 Detailed ablation study for PSD
Table shows the results detailed ablation experiments for PSD (the sample standard deviation of each result is presented in Table
The "w/o ASD" column and the "w/o FSD" column in Table show that removing the action/feature distillation (the two terms of Eq. reduces returns on 13/12 (out of 18) tasks.
The "Using SRSD" column in Table shows that the structural relation distillation in MADTKD not well compatible with the MOSDB dataset or the MOSDT architecture.
B.4 Detailed ablation study for partial parameter sharing
Table & shows the results of partial parameter sharing (the sample standard deviations are presented
in Table 99. Partial parameter sharing leads to performance losses on 12 to 14 (out of 18) tasks.
B.5 Ablation study for network designs
Table 10 presents the results of detaching the agent features Fi, from the computational graph Eq. showing performance degradation on 10 out of 18 tasks.
B.6
Standard deviation results
Table 11 shows the sample standard deviation of each value in Table Table 12 shows the sample standard deviation of each value in T
C More details of the MOSDB dataset
Table shows the details of the MOSDB dataset. Fig. presents the data distribution over the reward-cost plane of each task in MOSDB.
23

---

Table 5: The details of time consumption. Results are in milliseconds (ms). "TT": The time consumption of one training step. "ET": The time consumption of an inference for one agent. "Using KD": Using conventional KD instead of PSD. "No PS": No parameter sharing. The results in "Using KD-ET" are identical to those in "MOSDT-ET", because using KD has effect during execution.
CDT
MOSDT
Using KD
No PS
w/o CBE
Task
TP
EP
TP
EP
TP
EP
TP
EP
TP
EP
MOS Velocity
2x Swimmer
6.65
0.76
11.19
0.54
15.27
0.54
11.68
0.56
10.92
0.55
2aafhhhhheetth
6.06
0.75
11.30
0.54
14.15
0.54
11.23
0.55
10.97
0.55
2x3 Walker2d
6.21
0.92
10.97
0.58
14.35
0.58
11.77
0.56
11.02
0.54
2x4Ant
6.28
0.76
11.13
0.56
14.42
0.56
11.61
0.58
11.14
0.56
3x Hopper
6.19
0.92
13.84
0.59
16.84
0.59
14.88
0.66
14.71
0.57
4x2Ant
6.24
0.77
18.75
0.71
22.13
0.71
20.51
0.73
19.42
0.68
6x HalfCheetah
6.32
0.76
26.44
0.71
29.63
0.71
30.13
0.74
26.33
0.70
918Humanoid
7.21
0.88
12.84
0.85
17.42
0.85
12.45
0.78
13.00
0.76
MOS Goal
Multi-Ant
6.85
0.79
11.76
0.74
16.70
0.74
12.17
0.74
12.22
0.75
Multi-Ant2
6.59
0.78
12.30
0.72
16.59
0.72
12.08
0.75
12.35
0.76
Multi-Pointl
6.65
0.78
11.95
0.73
16.40
0.73
12.22
0.76
11.97
0.73
Multi-Point2
6.48
0.79
11.95
0.75
16.32
0.75
12.65
0.78
12.07
0.75
MOS Isaac Gym
DraweDAA
6.63
1.02
11.61
1.08
16.46
1.08
12.05
1.10
11.53
1,41
6.62
1.03
11.57
1.07
16.35
1.07
11.82
1.09
11.65
1.07
CatchFingerMA
6,68
0.94
12.11
1.06
16.72
1.06
12.53
1.11
12.01
1.05
CatchJointMA
6.81
0.93
12.81
1.06
17.05
1.06
12.79
1.12
12.49
1.06
OverFingerMA
6,80
0.93
12.21
1.08
16.62
1.08
12.72
1.08
11.95
1.05
OverJointMA
6.82
0.93
12.53
1.08
16.88
1.08
13.33
1.14
11.95
1.08
Average
6.56
0.86
13.18
0.80
17.24
0.80
13.81
0.82
13.21
0.81
D Limitations and future work
MOSRL may be used for negative applications such as drone surveillance. Upon closer inspection, no data at high risk of such misuse is observed in MOSDB. In addition, the cost binarization in CBE may constrain the adaptability of models to dynamic cost thresholds. This drawback may not be an issue in many applications with fixed cost thresholds [10] 47). Future efforts will focus expanding the MOSDB dataset.
24

---

Table The results of detailed ablation experiments for PSD. Results are the "return (cumulative cost)" format. The cost threshold e is 25, The maximum return of each task is shown in Table T3 in Appendix " "I Policies with lower returns than full MOSDT or unsafe policies. Red: Unsafe policies. "w/o ASD": Without action self-distillation. "w/o FSD": Without feature self-distillation.
"Using SRSD": Adding a structural relation distillation [5] term in Eq. (4) Due to space constraints,
the sample standard deviation across multiple runs of each experiment shown in Table 7
Task
MOSDT
w/o ASD
w/o FSD
Using SRSD
MOS Velocity
2x Swimmerr
11.64 (20.33)
1. 9.48 (10.20)
11.43 (10.67)
11.02 (12.53)
2x3HaafChheetah
2052.64 (22.27)
2074.78 (24.17)
2084.67 (35.43)
I 2050.06 (24.00)
Wallall
1584.87 (3.83)
I/ 1583.14 (3.07)
1529.36 (0.10)
& 1562.50 (4.57)
Ant
2054.88 (0.87)
1 1971.57 (1.97)
2045.89 (5.60)
1975.55 (0.90)
3x Hopper
1122.23 (4.00)
1236.94 (7.10)
1199.86 (11.43)
y 1067.13 (8.53)
4x2Ant
2083.85 85 (3.53)
1, 2080.61 (2.93)
2191.79 (3.50)
2093.81 81 (0.57)
GalHhhhhee
1853.64 (21.9)
1. 1835,71 (23.97)
1808.95 (25.73)
1862.54 (22.80)
918Huanooid
444.71 (22.80)
477.64 (21.73)
510.74 (21.47)
460.61 (22.60)
MOS Goal
Multi-Antl
38.38 (14.50)
J 30.11 (11.33)
44.95 (14.33)
| 35,30 (14.17)
Multi-Ant2
2,96 (7.50)
| 2.85 (15.33)
v. 3.82 (13,50)
| 1.55 (8.33)
Multi-Pointl
9.65 (12.67)
| 5.61 (23.00)
11.33 (20.83)
11.88 (21 .50)
Multi-Point2
-1.08 (21.00)
| -1.79 (15.00)
-4.35 (8.33)
0.85 (12.67)
MOS Isaac Gym
CI
3.45 (0.00)
I 33,45 (0.00)
.3.73 (0.00)
3.44 (0.67)
PickAndPlaceMA
-2.67 (0.00)
-2544 (0.00)
2.07 (0.00)
L 2-877 (0.00)
CatchFingerMA
0.25 (6.33)
I 0.211 (4.47)
0.1.7 (5.57)
0,27 (3.50)
CatchJoinLMA
0.31 (0.00)
J 0.31 (0.00)
0.23 (0.00)
L 0.23 (0.00)
OverFingerMA
0.52 (0.00)
J. 0.46 (0.00)
0.433 (1.97)
J 0.44 (0.00)
OverloimLMA
0.47 (1.03)
0.48 (0.00)
L 0.44 (1.60)
& 0.46 (0.00)
Summary
(safc)
13 (safe)
12 I) (safe)
11 (safc)
Table 7: The results of detailed ablation experiments for PSD (sample standard deviation). Results are in the "return sample standard deviation (cumulative cost sample standard deviation)" format. "w/o ASD": Without action distillation. "w/o FSD": Without feature distillation. "Using SRSD": Adding structural relation distillation
5
term in Eq. (4)
Task
MOSDT
w/o ASD
w/o FSD
Using SRSD
MOS Velocity
2x1 Swimmer
1.41 (2.61)
1.43 (3.37)
1.05 (4.92)
2.43 (5.07)
2x 2x3halfCheetah
23.79 (1.88)
4.53 (0.57)
37.07 (6.09)
10.40 (0.89)
2x3 Walker2d
23.95 (5.43)
52.60 (2.80)
28.64 (0.17)
52.20 (7.39)
An4Ant
73.62 (1.33)
10.77 (2.18)
32.35 (2.56)
48.81 (0.90)
3x Hopper
131.42 (4.65)
40.69 (8.02)
4.59 (6.48)
167.69 (9.86)
4x2Ant
64.59 (3.39)
41.88 (0.76)
21.65 (1.21)
48.40 (0.15)
6x HalfChectth
42.95 (1.15)
4.24 (1.21)
26.32 (1.26)
19.34 (1.54)
Hummundd
8.88 (0.10)
31.93 (0.35)
11.80 (0.71)
11.29 (0.92)
MOS Goal
Multi-Antl
0.84 (12.56)
11.11 (8.96)
3.40 (10.13)
8.19 (11.00)
Multi-Ant2
1.56 (8.35)
0.85 (5.11)
1.39 (6.54)
0.57 (14.43)
Multi-Pointl
2.52 (1.53)
0.71 (1.50)
2.94 (3.51)
4.00 (2.18)
Multi-Point2
2.89 (4.09)
2.40 (6.06)
3.95 (4.54)
3.23 (8.39)
MOS Isaac Gym
ClwweDra
0.38 (0.00)
0.38 (0.00)
0.03 (0.00)
0.58 (1.15)
PickAndPlaceMA
0.52 (0.00)
(0.00)
0.27 (0.00)
1.07 (0.00)
CatehFingerMA
0.03 (0.15)
0..460..0)
0.01 (0.85)
0.11 (0.75)
CatchJointMA
0.09 (0.00)
0.1000.000
0.03 (0.00)
0.01 (0.00)
OverFingerMA
0.10 (0.00)
0.04 (0.00)
0.01 (2.89)
0.01 (0.00)
OverlointMA
0.03 (1.05)
0.05 (0.00)
0.03 (0.72)
0.04 (0.00)
25

---

Table 8: The results of partial parameter sharing. Results are in the "return (cumulative cost)" format. The cost threshold is 25. The maximum return of each task is shown in Table i3 in Appendix C "Il": Policies with lower returns than full MOSDT or unsafe policies. "woo PSH": Without parameter sharing for student action heads. "w/o PSFP": Without parameter sharing for student feature projector. "w/o PST": Without parameter sharing for student causal transformers. "w/o PSE": Without parameter sharing for encoders. Due to space constraints, the sample standard deviation across multiple runs of
each experiment shown in Table
Partial parameter sharing
Task
MOSDT
w/o PSH
w/o PSFP
PST
w/o PSE
MOS Velocity
2x 2x1Swimmer
11.64
J 10.97 (15.33)
9.85 (18.10) 10)
U 10.85 (15.67)
8.37 (3.83)
hhalhhhhhetaah
2152.64 (22.27)
J 2050.14 14 (23.00)
2050.25 (23. 13)
2049.17 (22.27)
2041.74 (23.87)
2x3Waleer2d
1584.87 (3.83)
1585.52 (2.07)
1558.54 (2.93)
1578.55 (1.43)
1557.33 (2.80)
2x4Ant
2054.88 (0.87)
|) 2049.88 (1.73)
1988.90 (0.00)
1983.47 (2.30)
2015.30 (0.33)
3x. Hopper
1122.23 (4.00)
I 110.93 (1,33)
47.56 (2.53)
33.54 (0.00)
34.09 (0.03)
4x2Ant
2083.85 (3.53)
962.59 (0.00)
934.06 (0.00)
910.53 (0.00)
934.31 (0.00)
6x HallChheeth
1853,64 (21 197)
J 1169 26 (0.10)
407.84 (0.00)
I| 330.70 (1100)
436.95 (0.00)
918Humanoid
444.71 (22.80)
474.87 (21.73)
489 29 (22.50)
508. 10 (20.77)
459.40 (2197)
MOS Goal
Multi-Antl
38.38 (14.50)
J. 36.87 (10.50)
36.22 (20.00)
I| 34.36 (17.50)
33,96 (14.00)
Multi-Ant2
2.96 (7,50)
4.60 (11.67)
3.20 (7.33)
4,29 (19.67)
$ 2,60 (17.67)
Multi-Point]
9.65 (12.67)
10.42 (14.83)
9.70 9.70 (14.67)
IJ 8.18 (20.17)
12.41 (19.83)
Multi-Poin12
-1,08 (21.00)
0.33 (7.83)
-0822 (14.17)
3.83 (18.17)
-3667 (11.00)
MOS Isaac Gym
ClaseDrawerMA
-3.45 (0.00)
|| -3.46 (0.00)
-3.68 (0.20)
(0.67)
3.228
A
-2.67 (0.00)
2.24
-2885 (0.00)
|| -2.99 (0.00)
(1.83)
CatchFingerMA
0.25 (6.33)
J 0.23 (5.33)
0.20 (3,10)
[| 0.20 (7.03)
$ 0.22 (7.23)
CatchJointMA
0.31 (0.00)
J 0.27 (0.00)
0,32 (0.00)
0,36 (0.00)
$ 0.25 (0.00)
OverFingerMA
0.52 (0.00)
(0.00)
0.49 (0.00)
(0.00)
0.49 (0.00)
OverJointMA
(1.03)
J. 0.46 (0.00)
0.46 (0.00)
4 0.46 (0.00)
4 0,46 (0.00)
Summary
(safe)
12 J. (sufe)
13 (safe)
13 J. (sufe)
14 (safe)
The results of partial parameter sharing (sample standard deviation). Results are in the "return sample standard deviation (cumulative cost sample standard deviation)" format. "w/o PSH": Without parameter sharing for student action heads. "w/o PSFP": Without parameter sharing for student feature projector. "w/o PST": Without parameter sharing for student causal transformers.
Table 9:
"w/o PSE": Without parameter sharing for encoders.
Partial parameter sharing
Task
MOSDT
w/o PSH
w/o PSFP
w/o PST
w/o PSE
MOS Velocity
2x1Swimmer
1.41
2.51 (3.19)
2.41 (4.12)
3.46 (3.61)
1.47 (2.40)
2x3HalfCheetah
23.79 (1.88)
4.77 (0.89)
19.05 (1.66)
0.84 (0.90)
14.62 (0.21)
2x3 Walker2d
23.95 (5.43)
59.87 (1.37)
66.45 (2.11)
12.86 (1.45)
2.44 (4.85)
2x4Ant
73.62 (1.33)
147.35 (1.70)
14.82 (0.00)
51.64 (2.89)
58.57 (0.35)
3x1Hopper
131.42 (4.65)
125.36 (2.31)
10.97 (4.39)
6.79 (0.00)
5.46 (0.06)
4x2Ant
64.59 (3.39)
9.76 (0.00)
16.76 (0.00)
4.02 (0.00)
22,29 (0.00)
Hallhhhhha
42.95 (1.15)
115.20 (0.17)
67.88 (0.00)
24.06 (0.00)
67.84 (0.00)
918Humanoid
8.88 (0.10)
10.98 (0.75)
29.37 (0.36)
60.67 (1.64)
24.41 (0.35)
MOS Goal
Multi-Ant1
0.84 (12.56)
9.15 (9.73)
4.86 (3.61)
4.25 (3.28))
3.81 (2)12)
Multi-Ant2
1.56 (8.35)
2.54 (7.52)
1.11 (6.83)
2.29 (6.21)
1.83 (5,39)
Multi-Pointl
2.52 (1.53)
3.23 (8.22)
6.89 (6.11)
2.33 (3.18)
2.69 (2.02)
Multi-Point2
2.89 (4.09)
2.20 (7.75)
1.86 (10.75)
4.30 (4.48)
5.26 (10.76)
MOS Isaac Gym
0.38 (0.00)
0.32 (0.00)
0.44 (0.35)
0.43 (1.15)
0.33 (1.15)
AickAndPlaceMA
0.52 (0.00)
0.41 (0.00)
0.75 (0.00)
0.25 (0.00)
0.47 (3.18)
CatchFingerMA
0.03 (0.15)
0.08 (2.31)
0.04 (2.04)
0.02 (0.95)
0.06 (0.70)
CatchJointMA
0.09 (0.00)
0.06 (0.00)
0.09 (0.00)
(0.00)
0.09 (0.00)
OverFingerMA
(0.00)
0.07 (0.00)
0.03
0.02
(0.00)
OverJointMA
0.03
0.03 (0.00)
0.0300.00)
0.00
0.03 (0.00)
26

---

Table 10: The results of detaching agent features. Results are in the "return + sample standard deviation (cumulative cost + sample standard deviation)" format. The cost threshold c is 25. The maximum return of each task is shown in Table 13 in Appendix C "4" Policies with lower returns than full MOSDT or unsafe policies. "Detaching": Detaching agent features F from the computational graph of Eq. (2).
Task
MOSDT
Detaching
MOS Velocity
2x 1ximmere
11.64 21.1
2054.66 = 2112
2xHheeChheetah
2052.64 + 22.27 (23.79 1.88)
2x3Walker2d
1584.87 3.83 (2435
I 1557..64 13.47 (2.37
2x4Ant
2054.88 1 1.33)
1. 2006.04 46.42 (0.63
3x 3x1Hopper
1122.23 = 4.00 (165.4224.655)
= (13.13
4x2Ant
3.53
2114.28 31.37 (6.13 8.11)
6x HalfCheetah
21.97 (42.95 (42.95
1.15)
1869.79 + 29.71 (23.30 22.31
988Humanoid
444.71 + 22.80 (8.88 + 0.10)
460.98 + 2.91 (21.50 + 0.95)
MOS Goal
Multi-Antl
38.38
12.56)
| 28.44 12.25 (15.50 5,50)
Multi-Ant2
2.96 1: 7.50 (1.56
| 2.01 = 0.90 (16.67
Multi-Pointl
9.65 12.67 (2.52
9.16 = (17.50
Multi-Point2
-1.08
4.09)
-0.02 + 1.16 10.75)
MOS Isaac Gym
CloseDrawerMA
-3.45 0.00 0.00)
-3.25 0.32 (0.67 1.15)
PickAndPlaceMA
-2.67 = 0.00 0.00)
2243 0.57 (2.67
CatchFingerMA
0,25 + 6.33 (0.03 0.15)
4 = 0.01 (4.03 2.61)
CatchJointMA
0.31 +
0.30 = 0.08 (0.00 0.00)
OverFingeRMA
0.52 0.00 1
0.448 = 0.04 (0.000.0.)))
OverJointMA
0.47 + 1.03 (0.03 + 1.05)
$ 0.047 = 0.04 (1.60 2.77)
Summary
(safe)
10 4 (safe)
Table 11: MOSDB benchmark and the performance of MOSDT (sample standard deviation). Results are in the "return sample standard deviation (cumulative cost sample standard deviation)" format.
Task
RC
BCQ-Tag
S
REAR-I ag X
CTVT
COptiICCE (4
CPQ
MOSDT (curs)
MOS Velocity
2x1 Swimmer
1.03 (2:57)
(4.17)
3.64 (8.04)
1.65 (6.35)
0.77 (2.26)
10.47 (11 (48)
(2.61)
2xHHalCheerah
6.54 (0.97)
38.19 (12.88)
67.62 (26.85)
19.86
(1.45)
607. 92. (9.27)
23.79 88)
2x3Walerr2d
43.72 (0.00)
87.21 (11.53)
(6.46(6.03) (6.03)
(3.188)
12.95 (0.00)
201.40 (6.30)
(5.43))
2x4Ant
124.08 (0.00)
8.48 (11.72)
(1.550
25.58
48.48 (1.14)
1156.58 (0.87)
(1.33)
3xpeepper
4.63 (0.52)
23.33 (3.09)
(3.23)
(0.000
48,45 (10.25)
98.51 (1.85)
4x2Ant
(0.00)
58.78 (0.00)
(0.00)
(0.00)
6.10 (0.00)
(0.84)
64,59 (3.39)
6x IHallhhhhetta
35.43 (0.00)
87.69 (0.06)
49.45 (0.00)
20.85 (0.00)
38.27 (0.00)
122.73 (1.50)
42.95 (1.15)
918Humanoid
19.84 (1.30)
28.77
35.03 (0.47)
(1.17)
38.79 (7.79)
MOS Goal
Multi-AntI
7.24
(0.96)
5.80 (12.76)
10.04 (9.57)
(6.73)
0,34 (0.00)
(12.56)
Multi-nr2
0.98 (7.09)
0.16 (1.28)
1,13 (12.27)
2.35 (4.09)
1.24 (7.29)
0.74 (0.00)
1,56 (8.35)
Point1
(4.75)
(5.20)
(12.41)
1.22 (6.83)
(13.28)
(1.53)
Multi-Poini2
0.7% (7.51)
4,51 (18.30)
3,02 (6.24)
2,23 (5.13)
0.86 (4.65)
0.78 (9.50)
2,89 (4.09)
Gym
CloseDrwerMA
(1.73)
0.26 (0.87)
1.53 (4.07)
0.43 (0.00)
0.15 (0.00)
0.35 (0.00)
0.38 (0.00)
PicLAndPlaceMA
1.75
CatchFingerMA
0.03 (0.00)
0.01 (1.28)
0.02 (0.00)
0.02 (0.10)
0.05 (0.00)
0.05 (1. 50)
0:03 (0.
CatchJointMA
0.01
0.04 (0.00)
0.03 (0.00)
0.01 (0.70)
0.06 (0
0.08
0.09
OverTingerrA
0.01 (0.00)
0.02 (0.12)
0.02 (0.00)
0.05 (0.00)
0.03 (0.00)
0.00 (0.00)
0.10 (0.00)
OverJeintMA
0.02 (0.00)
0.01 (0.00)
0,03 (0.00)
0,01 (1.55)
0.02 (0.00)
0.01 (0.00)
0.03 (1.05)
27

---

Table 12: Ablation study results (sample standard deviation). Results are in the "return sample standard deviation (cumulative cost sample standard deviation)" format. The cost threshold is
25. The maximum return of each task is shown Table [3 [3 in Appendix C "Partial PS Without
parameter sharing for student action heads. "No PS": No parameter sharing. "w/o SE": Without safety signal embedding.
Task
MOSDT
w/o PSD
Partial PS
No PS
w/o SE
w/ CBE
MOS Velocity
2x1Swimmer
1.41 (2.61)
1.49 (4.16)
2.51 (3.19)
1.89 (2.66)
1.57 (7.72)
2.39 (1.4))
2x3HaliCheetah
23.79 (1.88)
32,45
4.77 (0.89)
21.86 (2.70)
3.41 (3.86)
47.33 (5.58)
2x3 Walkerd
23,95 (5.43)
73.25
59.87 (1.37)
89.35 (1.88)
54.69 (5.08)
104.87 (4.76)
2x4Ant
(1.33)
20.82
147.35 (1.70)
88.48 (10.75)
34.64 (8.67)
17.59 (2.46)
3x Hopper
131.42 (4.65)
16.26 (10.93)
125.36 (2.31)
2.80 (0.00)
171.57
59.61 (7.27)
4x2Ant
64.59 (3.39)
22.42 (1.42)
9.76 (0.00)
L83 (0.00)
48.07 (2.56)
58.66 (7 48)
HalfCheetah
42.95 (1.15)
51.38 (1.80)
115.20 (0.17)
82.59 (0.00)
13.82 (2.42)
4.23 (0.74)
918Humanoid
8.88 (0.10)
18.53 (1.01)
10.98 (0.75)
51.44 (0.36)
24.66 (0.55)
9.89 (0.15)
MOS Goal
Multi-Antl
0.84 (12.56)
(2.25)
9.15 (9.73)
11.09 (11.93)
8.38 (4.01)
9.81 (4.92)
Multi-Ant2
1.56 (8.35)
1.61 (10.32)
2.54 (7.52)
0.49 (5.03)
0.50 (0.90)
0.33 (4.54)
Multi-Pointl
2.52 (1.53)
(5.06)
3.23 (8.22)
(4.86)
1.46 (8.69)
0,51 (11.82)
Multi-Point2
2.89 (4.09)
(10.33)
2.20 (7.75)
(3.69)
0.71 (9.96)
1.44 (7.77)
MOS Isaac Gym
CloseDrawerMA
0.38 (0.00)
0.06 (0.00)
0.32 (0.00)
0.48 (0.00)
0.20 (0.00)
0.14 (0.00)
PickAndPlaceMA
0.52 (0.00)
(0.00)
0.41 (0.00)
(0.00)
0.69 (4.62)
0.44 (0.00)
CatchlFingerMA
0.03 (0.15)
(1.82)
0.08 (2.31)
0,02 (3.94)
0.03 (3.07)
0.02 (0.53)
CatchJointMA
0.09 (0.00)
(0.00)
(0.00)
(0.00)
0.06
0.02 (0.00)
OverFingerMA
(0.00)
0.01 (0.92)
0.07 (0.00)
0.01 (0.00)
0.02 (0.00)
0.15 (3.50)
OverJointMA
0.03 (1.05)
0.02 (0.44)
0.03 (0.00)
0.0200.000
0.000)
0.09 (3.91)
Table 13: The details the MOSDB dataset. "Full name": Full name Safety Gymnasium 28
"Observation dimension": The sum of the observation dimensions of all agents. "Action dimension": The sum of action dimensions of all agents. "Credit assignment".
means the environment yields
respective returns and costs for each agent. " "X"" means the environment yields system returns and costs. "Maximum return": The maximum return in the MOSDB dataset.
Number of Observation dimension
Action dimension
Credit
Maximum
Task
Full name
assignment
MOS Velocity
2x1 1w1mmmmer
Safety2x1 SalltyyxiSwimmmeeeleeeooo Velecit--v
2
20
12.99
2x 3aFahhhhhetahh
38
X
1672.97
Balaalerr2d
Sufeey2::WWakkr/22eeeeeixxx
38
X
2031.20
2x4Ant
Sufoty2x4AAm/Vloociyy--0
2
58
X
1910.1
3x Hopper
Safetyāx HeeeeeixllHappeeellaiiiyyyyy
X
1326.51
4x2Ant
Sufeey4x2AntVeloccyy-xd
124
X
1882. 12
6x1laalCheetah
138
X
1626.55
918Humanodd
756
17
X
1438.35
MOS Goal
Multi-Ant
SafoyyAutMutiiioal11000
420
16
36.76
Multi-Ant2
SaferyAAmMulliiGoll22200
420
16
8.47
Multi-PointI
308
17.99
Multi-Point2
308
14.49
MOS Isane Gym
CloseDrawerMA
FroighfFankaClloeeDoverr
90
12
10.30
PickAndPlaceMA
FreighfFannkPPcckAudPloe
100
12
-0.83
CatchFingerMA
SaddoHHaadCaddDDeeuudddcee.m Safe_ finger
446
52
7.19
CatchJoinMM
Andd Safe, jaint
446
52
6.21
OverFingerMA
Handow Safe finger
422
40
6.12
OverJoinIMA
Shadow Shankw.landdOerr HandOver
422
40
6.58
28

---

Safety2x3HallCheetall Velocity-v0
Trajectory retur
20
Trajectory return
1000
0-
20
0
20
40
20
40
Trajectory cumulative cost
Trajectory cumulative cost
Safety22sWaleeeeeeeelooiyyyyy
Safety2x44AttVelocity-vo
2000
2000
Trajectory retur
Trajectory return
1000
1000
20
40
20
Trajectory cumulative cost
Trajectory cumulative cost
Safety3x Safetyix
Safety-422AmtVolocttt-oo
Trajectory return
Trajectory retum
1000
1000
500
0
20
40
0
20
40
Trajectory cumulative cost
Trajectory cumulative cost
Safety6x HalaaaaCChhhaaaaaoiiio-----
Safety9 VelummmniddVllcciyyy...
1500
Trajectory return
Trajectory return
1000
1000
500
0
0
20
40
0
20
40
Trajectory cumulative cost
Trajectory cumulative cost
Figure 5: The data distribution over the reward-cost plane of the proposed MOSDB dataset (part 1/3).
Each point represents a eeturn-cumulative cost tuple trajectory. The cost limit set to 25.
29

---

AafetyAntMuliiiiallll
Saféty-AntMulltGGooll2--o0
40
Trajectory retum
Trajectory return
10
20
5
0
0
20
40
20
40
Trajectory cumulative cost
Trajectory cumulative cost
SaferyPointMMutiGGoall--
PateyyPointMuttiGGoal2---
20
Trajectory return
Trajectory return
10
10-
20
40
20
40
Trajectory cumulative cost
Trajectory cumulative cost
FreighErannaCClosDDaaeer
FreightFannkPPikAAnPPece
Trajectory return
Trajectory retum
-10
-20
-20
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
Trajectory cumulative cost
Trajectory cumulative cost
lance 2Underarm Safe finger
ShadowilandCCtchhoeer
2Underarm Safe joint
Trajectory return
Trajectory retum
5.0
5
2.5
0.0
4
2
Trajectory cumulative cost
Trajectory cumulative cost
Figure 5: The data distribution over the reward-cost plane of the proposed MOSDB dataset (part 2/3). Each point represents a return-cumulative cost tuple a trajectory. The cost limit set to 25.
30

---

ShadowHandOvver Safe finger
ShadowHanddOver Safe joint
Trajectory return
Trajectory return
4
4
2
2
4
2
4
Trajectory cumulative
Trajectory cumulative cost
Figure 5: The data distribution the reward-cost plane of the proposed MOSDB dataset (part 3/3).
Each point represents a return-cumulative cost tuple trajectory. The cost limit set to 25.
31


Reviews:
[{"review_id": "cjgAAJ6vIB", "reviewer": "Reviewer_Twb8", "summary": "This paper investigates multi-agent offline safe reinforcement learning. Built on top of CDT and MADTKD, this paper proposes MOSDT, which performs (1) decentralized student supervised learning; (2) centralized teacher supervised learning; (3) policy self-distillation. Also, this paper utilizes several implementation tricks: (1) full parameter sharing and (2) \nCost binary embedding. Extensive experiments are conducted on a self-collected dataset MOSDB to demonstrate the effectiveness of the proposed algorithm.", "questions": "See weakness", "limitations": "n/a", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "### Strengths\n1. Comprehensive ablation studies are conducted to verify each component of the proposed algorithm.\n2. The techniques are simple and easy to follow.\n\n### Weaknesses\n\n1. The setting seems too incremental for considering offline, multi-agent, safety RL. Please motivate this setting with several important application scenarios. It would be better to include a benchmark that tackles one more practical scenario to make this setting well-justified.\n\n2. The proposed algorithm is also incremental over two existing algorithms, one from offline safety RL algorithm, CDT, and one from offline multi-agent RL algorithm, MADTKD. There is not any clear motivation in the methodology to justify why we should design MOSRL algorithms as in MOSDT.\n\n3. The quality of the built benchmark MOSDB seems low. As shown in Table 1 and Table 11, MOS Goal and MOS Isaac Gym cannot provide informative comparison between MOSRL algorithms, as the performance gap can be filled even with several random seeds. Same phenomenon can be overserved in Table 2 and Table 12. This makes the reported performance based on SOTA on how many settings meaningless. Also, it is weird to report mean values and standard deviations in two separate tables. Please consider merge them in one table.\n\n4. The motivation of the paper is clear. It states that multi-agent offline safe Reinforcement Learning is lacking in the existing literature, but it does not specify why this particular technique is necessary. What specific problem is this paper to address? Why are existing solutions inadequate for this purpose?\n\n5. What is the exact novelty compared to reference [5]? \n\n6.  MOSDB in Section 3.4: What is the nature of the tasks within the datasets? How are the trajectories collected (what is the meaning of the trajectories in the dataset, anyway)? The authors are expected to elaborate further on this newly created dataset, including the motivation, tasks, evaluation methods, baselines, etc.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer Twb8 for recognizing the completeness and usability of our work. Below, we respond to the weaknesses.\n\n**(W1) Motivation for multi-agent offline safe RL (MOSRL)**\n\nWe further emphasize that MOSRL is not merely an incremental extension but a critical advancement addressing challenges in many important application scenarios.\n\n(1) Autonomous vehicle coordination [1]. Driving policies for autonomous vehicles should be safe and cooperative to avoid any collisions with each other. Online exploration is impractical due to safety risk.\n\n(2) Power grid scheduling [2]. A power grid involves multiple agents (e.g., generators). Safety constraints, such as preventing overload, are crucial. Online trial-and-error is infeasible due to potential catastrophic consequences.\n\n(3) Robot collaboration [3]. In industrial settings such as manufacturing and logistics, multiple robots need to coordinate under strict safety constraints (e.g., collision avoidance during collaborative machining and transportation). Online learning may cause unacceptable asset losses.\n\nWe mentioned scenarios (1) and (2) in Section 1 (lines 29-30). We will emphasize more on the motivation of MOSRL in the revised version. The complex requirements in these scenarios reveal the limitations of existing RL methods. The developed MOSRL framework, algorithm, and benchmarks have the potential to meet the requirement of learning safe, cooperative policies from offline data.\n\nIndeed, the MOSDB dataset has incorporated 18 practical tasks, including bionic robot racing, robot navigation and obstacle avoidance, and throwing/catching objects by bionic robotic hands. These tasks have significant implications in various real-world engineering domains, including autonomous vehicles and industrial automation.\n\nWe recognize the value in expanding the scope for greater practical significance. As stated in Appendix D (lines 522-523), future efforts will focus on expanding the MOSDB dataset, by introducing a series of power system operation tasks with real-world data. \n\n**(W2) Design motivation of MOSDT**\n\nInstead of a simple combination of CDT and MADTKD, MOSDT is a new algorithm tailored for the MOSRL problem, with several key innovations that significantly enhance its performance in multi-agent environments with safety constraints.\n\nWe propose two innovative designs to address key challenges in multi-agent environments, including policy self-distillation (PSD) and full parameter sharing. Inspired by the spirit of MADTKD that introduces policy distillation to offline MARL, we propose PSD to extend CDT to multi-agent settings. Different from MADTKD, PSD embeds student networks into the teacher and performs distillation synchronously with supervised learning, reducing training parameter count by 35% and training time by 24%. As shown in the following table (with the same setting as Table 2), compared with the PSD in our proposed MOSDT, policy distillation (in MADTKD) causes return losses on 11 out of 18 tasks and leads to unsafe policy on 1 task (Task No. 2). \n\nTable A\n\n|  |  |  |  |  |  |  |  |  |  |\n|---|---|---|---|---|---|---|---|---|---|\n| Task No. | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9  |\n| Return (cost) | ↓10.56 (13.90) | 2096.33 (39.07, unsafe) | ↓1546.32 (3.30) | ↓2035.09 (4.43) | 1256.79 (10.23) | 2197.64 (5.47) | ↓1808.23 (24.63) | 499.15 (20.47) | ↓31.60 (14.83)  |\n| Task No. | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18  |\n| Return (cost) | 3.35 (14.00) | 11.33 (8.33) | 3.32 (16.83) | ↓-3.74 (0.00) | ↓-2.69 (0.00) | ↓0.20 (2.97) | ↓0.24 (0.00) | ↓0.43 (0.80) | ↓0.43 (1.10)  |\n\nThe full parameter sharing among agents reduces training parameter count by 47% and execution parameter count by 58% (cf. Fig. 3).\n\nWe propose a plug-and-play module, cost binary embedding (CBE), to address key challenges in safety-critical settings. For MOSDT, CBE enhances the cost information processing capabilities of the CDTs by binarizing cumulative costs as safety signals and embedding the signals into returns. CBE improves returns on 14 out of 18 tasks (cf. Table 2).\n\n**(W3) Quality of the MOSDB benchmark**\n\nOur benchmark MOSDB can provide an informative comparison on most of the tasks in a widely adopted environment, *Safety Gymnasium* [4]. Indeed, MOSDT achieves SOTA returns on 14 out of 18 tasks (cf. Table 1). On 9 out of these 14 tasks, MOSDT achieves a lead of more than 5% compared to the suboptimal method. On some tasks, MOSDT achieves a return several times higher than the suboptimal method (9.2 times on “3x1Hopper”, 4.2 times on “6x1HalfCheetah”, and 2.3 times on “4x2Ant”). On 13 out of these 14 tasks, MOSDT achieves a lead of more than 10% compared to the average performance of the baseline models.\n\nA few tasks in the MOSDB benchmark demonstrate small performance gaps among methods—we note that this is not unusual in other well-known benchmarks. For instance, in Safety Gymnasium [4], the best method achieves a lead of more than 5% compared to the suboptimal method on only 2 out of 6 Safety Velocity tasks. We agree with Reviewer Twb8 that for MOSDB, the statistical significance of performance comparison requires more attention.\n\nThe offline data in the MOSDB dataset is collected strictly following the procedure of a widely used single-agent offline safe dataset, DSRL [5].  Following existing well-known RL benchmarks, such as Safety Gymnasium [4], DSRL [5], and D4RL [6], we evaluate the quality of algorithms based on achieving SOTA average returns on how many tasks.\n\nIn the original submission, we separated mean values and standard deviations due to space constraints. We will merge them into one table in the revised version.\n\n**(W4) Necessity of MOSRL**\n\nAs mentioned in Section 1 (lines 29-30), MOSRL is promising in various application scenarios, such as autonomous vehicle coordination and robot collaboration (cf. our reply to W1 for more details), where agents need to learn safe, cooperative policies using offline data.\n\nAs stated in Section 1 (lines 21-26), existing offline RL methods focus solely on either multi-agent or safety settings, leaving a gap in algorithms specifically designed for MOSRL. Existing offline MARL methods (e.g., MADT, MADiff, and PTDE) lack built-in safety mechanisms, rendering them ineffective for safety-critical applications. Current offline safe RL methods (e.g., CDT, SaFomer, and SDT) are tailored for single-agent settings, lacking the necessary coordination mechanisms for multi-agent systems. \n\nTo lay the foundation for MOSRL, we provide its first algorithm (MOSDT) and first dataset (MOSDB). We train MOSDT and other baseline methods to set up the MOSDB benchmark and verify the superiority of MOSDT in addressing MOSRL challenges.\n\n**(W5) Novelty Compared to MADTKD**\n\nCompared to MADTKD, the proposed MOSDT incorporates three key innovations.\n\n(1) Policy self-distillation (PSD). As a two-stage method, it first trains an independent teacher network and then distills the policy to students. We introduce policy self-distillation (PSD) to MARL. PSD synchronously performs policy distillation alongside supervised policy learning, integrating student networks into the teacher network to streamline training and enhance parameter efficiency.\n\n(2) Full parameter sharing. Existing offline MARL methods (including MADTKD) assign distinct network parameters to different agents. We propose full parameter sharing across all agents to ensure training stability and make MOSDT more lightweight.\n\n(3) Safety property with CBE. As an offline MARL algorithm, MADTKD was not designed to address safety constraints. To enhance the safety performance of MOSDT, we propose a new plug-and-play module, CBE, which i) binarizes cumulative costs as safety signals, and ii) embeds the signals into returns.\n\n**(W6) Details of the MOSDB Dataset**\n\nThe offline data in the MOSDB dataset is collected from Safety Gymnasium [4]. Please refer to [4] for detailed introduction on the nature of the tasks. We briefly describe these tasks below and will supplement this information in the revised version.\n\n(1) MOS Velocity. Robots are required to move as quickly as possible while adhering to velocity constraints. Multiple agents need to control distinct body segments cooperatively.\n\n(2) MOS Goal. Each agent is required to reach its color-designated target while avoiding collisions and hazardous terrain.\n\n(3) MOS Isaac Gym. It focuses on collaborative robotic tasks, such as coordinated ball-handovers between dual manipulators, with enforced safety constraints on joint movements.\n\nRegarding trajectory collection, as stated in Section 1 (line 77), we train two multi-agent safe RL algorithms on Safety Gymnasium [4], logging agent-environment interactions as offline data. As a commonly used concept in offline RL, “trajectory” is a fixed sequence of historical state-action-reward (cost) transitions.\n\nAs mentioned in Section 2 (line 114), the lack of MOSRL datasets is our key motivation for building the MOSDB dataset. The evaluation methods are described in Section 4.1 (lines 241-246). The baseline methods trained on the MOSDB dataset are introduced in Section 4.1 (lines 232-239).\n\nWe will integrate some of the above details of MOSDB into Section 3.4 in the revised version.\n\n**References**\n\n[1] Chen Y, et al. Deep reinforcement learning in autonomous car path planning and control: A survey. arXiv (2024).\n\n[2] Chen X, et al. Reinforcement learning for selective key applications in power systems: Recent advances and future challenges. IEEE Transactions on Smart Grid (2022).\n\n[3] Singh B, et al. Reinforcement learning in robotic applications: a comprehensive survey. Artificial Intelligence Review (2022).\n\n[4] Ji J, et al. Safety gymnasium: A unified safe reinforcement learning benchmark. NeurIPS (2023).\n\n[5] Liu Z, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv (2023).\n\n[6] Fu J, et al. D4rl: Datasets for deep data-driven reinforcement learning. arXiv (2020)."}, {"review_id": "wdculWfAVU", "reviewer": "Reviewer_vSoS", "summary": "This paper introduces MOSDT, a decision transformer architecture for multi-agent offline safe reinforcement learning. The method consists of several points of this approach, including (1) policy self-distillation integrated with global information reconstruction, (2) full parameter sharing across agents to stabilize training and reduce model complexity, and (3) cost binary embedding that replaces specific cost targets with binary safety signals for more flexible safety handling. The authors also release MOSDB, the first benchmark dataset for MOSRL, and show that MOSDT achieves SOTA performance on 14 out of 18 tasks, ensuring complete safety while requiring only 65% of the execution parameter count compared to baselines.", "questions": "Can the authors provide a deeper analysis of the failure cases observed in the ablation study (e.g., the four tasks where safety constraints were violated)? What factors contributed to these violations?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This paper addresses the understudied and challenging domain of multi-agent offline safe reinforcement learning, where issues of scalability and safety remain largely unresolved.\n- On the new MOSDB benchmark, MOSDT achieves state-of-the-art (SOTA) performance on 14 out of 18 tasks, spanning diverse environments such as MuJoCo, Safety Gym, and Isaac Gym, while strictly adhering to safety constraints.\n- Full parameter sharing reduces the execution parameter count by 58% and improves inference time compared to CDT.\n\nWeaknesses:\n- Limited real-world baseline comparison: The evaluation primarily adapts single-agent baselines to the multi-agent setting. Strong multi-agent baselines—such as recent diffusion-based methods (e.g., MADiff) or model-based safe MARL algorithms (e.g., constraint-aware MBRL)—are not included. These could offer valuable comparisons, especially under tight safety budgets or dynamic constraints.\n- Cost binarization limitation: The proposed CBE encodes safety using hard binary thresholds, which may limit adaptability in settings with soft, dynamic, or probabilistic constraints.\n- Limited safety analysis: Although all reported tasks satisfy safety thresholds, the failure modes (e.g., the four unsafe cases in the ablation study) are not thoroughly analyzed. Deeper investigation into these cases could inform future improvements in robustness.", "was_revised": true, "final_justification": "Thank the authors for detailed replies. It addresses my concerns. After reviewing the replies of the authors to me and other reviewers, I would like to keep my score.", "rebuttal": "We appreciate Reviewer vSoS for recognizing the significance and quality of our work. Below, we respond to the weakness and the question.\n\n**(Weakness 1) Limited baseline comparison**\n\nWe appreciate the Reviewer vSoS’s suggestion to broaden our evaluation by incorporating advanced offline MARL methods. We train MADiff [1] (a diffusion-based offline MARL method) and MOMA-PPO [2] (a model-based offline MARL method) on the proposed MOSDB dataset with their default settings for the tasks with the highest state dimensions. To make these two methods applicable to safety constraints, we input cumulative costs into them along with returns. The results are shown in the tables below (with the same setting as Table 1).\n\nTable A: Results of training MADiff [1] on the MOSDB dataset.\n\n|  |  |  |  |  |  |  |  |  |  |\n|---|---|---|---|---|---|---|---|---|---|\n| Task No. | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9  |\n| Return (cost)  | ↓8.29 (18.40) | 2131.33 (55.23, unsafe) | ↓1560.67 (0.63) | 2197.03 (2.47) | ↓81.67 (8.57) | ↓880.59 (0.00) | ↓357.98 (0.00) | 613.12 (19.93) | ↓20.44 (18.17)  |\n| Task No. | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18  |\n| Return (cost) | ↓2.20 (19.28) | ↓3.50 (7.00) | ↓-3.35 (21.47) | ↓-5.19 (1.00) | ↓-3.95 (0.00) | ↓0.24 (0.00) | ↓0.20 (0.00 | ↓0.44 (0.00) | ↓0.45 (0.07)  |\n\nTable B: Results of training MOMA-PPO [2] on the MOSDB dataset.\n\n|  |  |  |  |  |  |  |  |  |  |\n|---|---|---|---|---|---|---|---|---|---|\n| Task No. | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9  |\n| Return (cost) | ↓8.84 (19.23) | ↓665.63 (10.67) | ↓1557.61 (1.63) | 2296.26 (5.17) | ↓31.33 (0.00) | ↓734.56 (0.00) | ↓335.55 (0.00) | 552.37 (19.50) | ↓17.01 (19.97)  |\n| Task No. | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18  |\n| Return (cost) | ↓2.22 (16.50) | ↓0.32 (24.78) | 0.59 (15.17) | ↓-5.33 (1.00) | ↓-4.26 (0.00) | ↓0.24 (1.33) | ↓0.21 (0.00) | ↓0.45 (0.03)  | 0.47 (0.47)  |\n\nCompared with our proposed MOSDT, MADiff [1] yields lower returns on 15 out of 18 tasks and leads to unsafe policy on 1 task (Task No. 2). MOMA-PPO [2] yields lower returns on 14 out of 18 tasks compared with MOSDT, while maintaining safety. We will supplement these new results in the revised version and upload their training curves.\n\nAlthough advanced offline MARL methods like MADiff [1] and MOMA-PPO [2] perform well in standard offline MARL settings, they lack safety-oriented designs, limiting their applicability to safety-critical MOSRL tasks.\n\n**(Weakness 2) Limitations of CBE**\n\nWe acknowledge that the cost binarization in CBE may limit its adaptability in settings with soft, dynamic, or probabilistic constraints, as stated in Appendix D (lines 520-522). However, in most cases, the cost limits are fixed values, such as the temperature limits of cables in power systems [3] and the distance limit between vehicles and human in autonomous driving [4]. Many widely used safe RL benchmarks, such as Gymnasium [5] and DSRL [6], also adopt fixed values as their cost limits. In most cases where the cost limits are fixed, CBE offers a better alternative to specific cumulative costs, given its powerful effect (improving returns on 14 out of 18 tasks in Table 2), plug-and-play feature, and intuitive design.\n\nIn the future work, we will exploration several directions to improve our approach to accommodate soft, dynamic, or probabilistic constraints, enhancing its adaptability to more complex and realistic settings.\n\n(1) Soft safety constraints: Replacing hard binary signals with a continuous cost representation. This would allow the model to reason about the severity of safety violations rather than treating them as all-or-nothing events, improving performance in environments with graded safety settings.\n\n(2) Dynamic safety constraints: Introducing mechanisms to adjust thresholds based on contextual factors or trajectory history. This could enhance flexibility in settings where safety requirements evolve.\n\n(3) Probabilistic safety constraints: Integrating uncertainty-aware models to handle probabilistic safety constraints, which could improve performance in applications where safety requirements are not deterministic.\n\n**(Weakness 3 & Question 1) Analysis of failure cases**\n\nWe appreciate Reviewer vSoS’s suggestion on analyzing the failure cases without safety constraints violated. There are three major causes for the violation of safety constraints.\n\n(1) High risk in the early stage of training. As shown in the uploaded training curves (cf. the link in the abstract), most failure cases show excessive cumulative costs at the initial stages of training. For example, on the “2x3HalfCheetah” task (the cost limit is 25), when 10% of the training (10k steps) is completed, the complete MOSDT only yields a cumulative cost of 13.1, while the MOSDT without PSD yields a cumulative cost of 18.8, and CDT [7] yields a cumulative cost of 62, resulting the violation of safety constraints of the latter two methods (cf. Tables 1 and 2).\n\n(2) Instability of the cumulative costs in training. In most failure cases, the training curves of cumulative costs show severe fluctuations (e.g., the training curves of BCQ-Lag [8] on the “Multi-Point2” task). This instability indicates that the model often enters unsafe states and cannot maintain safety.\n\n(3) Lack of communication among agents. Maintaining global safety requires cooperation among agents. Lack of communication among agents could cause them to violate safety constraints. For example, in the \"2x3HalfCheetah\" task, rollout visualizations of the MOSDT without PSD (i.e., lacking centralized training) revealed that the agent's hind leg persistently accelerated, while its front leg remained stationary to decelerate. This uncoordinated action led to over-speeding and eventual task failure (cf. Table 2).\n\nWe will supplement the above analysis in the revised version.\n\n**References**\n\n[1] Zhu Z, Liu M, Mao L, et al. Madiff: Offline multi-agent learning with diffusion models[J]. Advances in Neural Information Processing Systems, 2024, 37: 4177-4206.\n\n[2] Barde P, Foerster J, Nowrouzezahrai D, et al. A model-based solution to the offline multi-agent reinforcement learning coordination problem[J]. arXiv preprint arXiv:2305.17198, 2023.\n\n[3] Chen X, Qu G, Tang Y, et al. Reinforcement learning for selective key applications in power systems: Recent advances and future challenges[J]. IEEE Transactions on Smart Grid, 2022, 13(4): 2935-2958.\n\n[4] Chen Y, Ji C, Cai Y, et al. Deep reinforcement learning in autonomous car path planning and control: A survey[J]. arXiv preprint arXiv:2404.00340, 2024.\n\n[5] Ji J, Zhang B, Zhou J, et al. Safety gymnasium: A unified safe reinforcement learning benchmark[J]. Advances in Neural Information Processing Systems, 2023, 36: 18964-18993.\n\n[6] Liu Z, Guo Z, Lin H, et al. Datasets and benchmarks for offline safe reinforcement learning[J]. arXiv preprint arXiv:2306.09303, 2023.\n\n[7] Liu Z, Guo Z, Yao Y, et al. Constrained decision transformer for offline safe reinforcement learning[C]//International conference on machine learning. PMLR, 2023: 21611-21630.\n\n[8] Xu H, Zhan X, Zhu X. Constraints penalized q-learning for safe offline reinforcement learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(8): 8753-8760."}, {"review_id": "iq7XDiztFt", "reviewer": "Reviewer_mXFE", "summary": "This paper studies the setting of multi-agent offline safety reinforcement learning. They proposed the first algorithm named MOSDT in this setting and showed its effectiveness through experiments. Besides, they constructed the first dataset and benchmark for this setting. They build their algorithm based on constrained decision transformer (safety offline RL) and centralized training with decentralized execution framework (multi-agent offline RL). Their innovations are from three perspectives. First, they proposed a cost binary embedding module to convert the safety signal to binary signal which solves the issue of unsafe intermediate actions. Second, they introduced self-distillation technique to multi-agent offline RL which simplifies the training pipeline, decreases training parameters and saves the training time. Last, they propose a full-parameter sharing design across students to stabilize the training.", "questions": "1. Can the authors explain more on the full parameter sharing design? Based on my understanding, it is to initialize the network of all student agents with the same parameters before the training pipeline starts.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. They proposed the first algorithm for the setting of multi-agent offline safety reinforcement learning which initiates the study in this setting.\n2. The proposed techniques including cost binary embedding and self-distillation look reasonable to me for this problem. \n3. They did comprehensive ablation studies to show the effect of different components in their algorithm network.\n\nWeaknesses:  \nNot much. The self-distillation technique is not tailored to safety RL which means it can be applied to multi-agent offline RL without safety constraints. Would be interesting to see how it works outside the safety setting compared to works such as MADTKD. The experiments are all based in the safety setting.", "was_revised": true, "final_justification": "I am recommending an accept since their proposed method is reasonable under the studied setting and the experimental results are satisfactory. But I agree the novelty is limited somehow. Basically, their proposed method is a combination of existing modules (CDT and CTDE) plus some small new tricks (full-parameter sharing and CBE).", "rebuttal": "We appreciate Reviewer mXFE for recognizing the significance, reasonability, and completeness of our work. Below, we respond to the weakness and the question.\n\n**(Weakness 1) Effectiveness of policy self-distillation (PSD) in offline MARL**\n\nWe appreciate the Reviewer mXFE for the suggestion to analyze PSD without safety settings. To verify the effectiveness of PSD in offline MARL without safety constraints, we removed the safety constraints in the environments of the MOSDB benchmark and let agents run 1,000 steps during evaluation. In this setting, we train MOSDT with the proposed PSD and with the conventional knowledge distillation (used in MADTKD [1]), respectively. The results are shown in the tables below (with the same setting as Table 2).\n\nTable A: Results of training MOSDT with PSD on the MOSDB benchmark (without safety constraints).\n\n|  |  |  |  |  |  |  |  |  |  |\n|---|---|---|---|---|---|---|---|---|---|\n| Task No. | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9  |\n| Return | 13.70 | 2050.09 | 1783.18  | 2171.59  | 1184.62  | 2053.42  | 1885.41 | 1126.75  | 37.90  |\n| Task No. | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18  |\n| Return  | 5.01 | 11.38 | 5.06 | 1.14 | -2.87 | 2.59 | 2.24 | 3.11  | 2.92  |\n\nTable B: Results of training MOSDT with the conventional knowledge distillation on the MOSDB benchmark (without safety constraints).\n\n|  |  |  |  |  |  |  |  |  |  |\n|---|---|---|---|---|---|---|---|---|---|\n| Task No. | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9  |\n| Return  | ↓12.53 | ↓2027.44 | 1802.80 | ↓2150.33 | ↓1169.34 | 2118.23 | ↓1850.15 | 1208.25 | ↓37.07  |\n| Task No. | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18  |\n| Return  | ↓4.45 | 11.51 | 9.86 | ↓-0.02 | -1.97 | ↓2.00 | ↓1.93 | ↓1.28 | ↓2.81  |\n\nAccording to these two tables, PSD improves returns on 12 out of 18 tasks (without safety constraints) compared to the conventional knowledge distillation. PSD improves most offline MARL tasks (with or without safety settings).\n\nThe primary focus of our paper is to tackle the novel and underexplored problem of multi-agent offline safe RL (MOSRL). Offline MARL methods, such as MADTKD [1], cannot address safety constraints, making them unsuitable for direct comparison within our safety-focused experimental framework. Our experiments thus prioritize the safety setting to demonstrate the risk control capabilities of MOSDT. We will conduct more in-depth research into the effectiveness of PSD in offline MARL without safety settings in subsequent work.\n\n**(Question 1) Clarification on the full parameter sharing design**\n\nWe thank Reviewer mXFE’s interest in our full parameter sharing design. We note that full parameter sharing is not merely initializing all student networks with the same parameters before training. We share all parameters across all student networks throughout the training process, as described in Eq. (6). This design keeps the training stable during the entire process and significantly reduces model parameters. In practice, we only create one instance of the Student class for all agents. In the entire MOSDT network, only the teacher action heads and the teacher feature projectors are distinct for different agents. According to the ablation study, compared with no/partial parameter sharing, full parameter sharing yields the best performance (cf. Table 2 in Section 4.2 and Table 8 in Appendix B.4) and slashes parameter count to the greatest extent.\n\nIn the revised version, we will enhance the description of full parameter sharing. We appreciate Reviewer mXFE for highlighting this point, which allows us to improve the clarity of the paper.\n\n**References**\n\n[1] Tseng W C, Wang T H J, Lin Y C, et al. Offline multi-agent reinforcement learning with knowledge distillation[J]. Advances in Neural Information Processing Systems, 2022, 35: 226-237."}, {"review_id": "Ur5S3MZHqE", "reviewer": "Reviewer_eJKo", "summary": "The paper proposes MOSDT, an algorithm for multi-agent offline safe reinforcement learning (MOSRL), and introduces MOSDB, a corresponding benchmark and dataset. The method incorporates policy self-distillation with a global information reconstruction mechanism that fuses observation features across agents. It employs full parameter sharing and introduces a cost binary embedding (CBE) module to encode safety constraints. Empirical results on MOSDB, which includes MuJoCo, Safety Gym, and Isaac Gym environments, show that MOSDT achieves higher returns on 14 out of 18 tasks, while maintaining safety and reducing execution-time parameter count compared to a single-agent baseline.", "questions": "Please refer to the weakness.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n- The authors extend multi-agent offline reinforcement learning and safe reinforcement learning to the underexplored setting of multi-agent safe offline reinforcement learning.\n- They provide a benchmark and dataset tailored to this new problem formulation.\n- The proposed approach achieves state-of-the-art performance under safety-constrained evaluation scenarios. A comprehensive ablation study offers insight into the rationale behind key design choices.\n\n# Weaknesses\n- There is room for improvement in the paper’s writing. For example, figure captions currently provide limited information and could be more descriptive to aid understanding. In Line 147, the symbol L appears to denote the loss, whereas Equation (1) uses the term Loss. This inconsistency in notation could be clarified.\n- While full parameter sharing across agents helps stabilize training, it may limit the applicability of the proposed method in scenarios where agents have heterogeneous action spaces.\n- Although the proposed cost binary embedding (CBE) module appears to improve information aggregation and support specific cost targets, its impact on training stability is unclear. It would be helpful if the authors could provide learning curves or additional ablation results to better assess the influence of CBE on training dynamics.", "was_revised": true, "final_justification": "Given authors' response, I change the scores accordingly.", "rebuttal": "We appreciate Reviewer eJKo for recognizing the significance, quality, and completeness of our work. Below, we respond to the weaknesses.\n\n**(Weakness 1) Writing improvements**\n\nWe appreciate Reviewer eJKo’s feedback on the writing quality. In the revised version, we will expand the captions to include more information and explanation.\n\n(1) Figure 1: Training MOSDT on the MOSDB dataset. The arrows represent the sequence of operations and data flow among the dataset, the student networks, and the teacher network under the proposed policy self-distillation (PSD) framework.\n\n(2) Figure 2: The network structure of the proposed MOSDT algorithm. The purple areas indicate our proposed innovative modules. The green areas represent data, while the blue areas represent networks. We enlarge one of the DSSL modules and display it in detail in the upper left corner. We only illustrate the situation at time t.\n\n(3) Figure 3: Number of parameters. “Using KD”: Using conventional KD instead of PSD. “No PS”: No parameter sharing. MOSDT only requires 65% of the execution parameter count of CDT. PSD reduces the number of training parameters by 47% and the number of execution parameters by 58%.\n\n(4) Figure 4: Visualization of the training on the “3x1Hopper” task. “No PS”: No parameter sharing. The purple curve in Subgraph (a) is enlarged and displayed in detail in Subgraph (b) for better viewing. Shaded areas represent sample standard deviations across multiple runs.\n\n(5) Figure 5: The data distribution over the reward-cost plane of the proposed MOSDB dataset. Each point represents a return-cumulative cost tuple of a trajectory. The cost limit is set to 25.\n\nRegarding the notation clarity, we confirm that the “L” in line 147 denotes the training sequence length instead of loss. We acknowledge that “L” could indeed be misunderstood as loss, so we will replace it with another notation, such as “M”, in the revised version. We will further check the clarity of notations to eliminate any possible misunderstandings.\n\n**(Weakness 2) Limitations of the full parameter sharing design**\n\nFor agents with heterogeneous action spaces, full parameter sharing can effectively stabilize training and improve performance at the scale of the MOSDB dataset (up to 756 dimensions of observation spaces and 52 dimensions of action spaces). Full parameter sharing improves returns on 11 out of 18 tasks compared with no parameter sharing (cf. Table 2) and 12 to 14 out of 18 tasks compared with partial parameter sharing (cf. Table 8 in Appendix B.4).\n\nFor all tasks in the MOSDB dataset, agents have heterogeneous action spaces.\n\n(1) MOS Velocity. Agents need to control distinct robot segments to move as quickly as possible while adhering to velocity constraints. The agents are heterogeneous because they play different roles in movements (e.g., the front and rear legs of “HalfCheetah” need to have different behavior patterns to move forward).\n\n(2) MOS Goal. Multiple agents are required to navigate to targets matching their designated colors, while evading collisions and hazardous terrain. The agents are heterogeneous because they have different objectives (navigating to targets in different colors) and do not share rewards and costs (cf. Table 13 in Appendix C), which makes them not interchangeable.\n\n(3) MOS Isaac Gym. It focuses on collaborative tasks between robotic agents, such as dual manipulators performing coordinated object manipulation like ball-handovers. Agents are heterogeneous because they play different roles in collaborations (e.g., dual manipulators are responsible for throwing and catching the ball, respectively). \n\nTo verify the effectiveness of full parameter sharing in more challenging scenarios with agents having heterogeneous action spaces, we will introduce a series of higher-dimensional power grid scheduling tasks to future versions of the MOSDB dataset.\n\n**(Weakness 3) Impact of CBE on training dynamics**\n\nWe appreciate Reviewer eJKo for the suggestion to analyze the effect of CBE on training dynamics. According to the uploaded learning curves (cf. the link in the abstract), CBE helps improve training stability on almost all tasks by eliminating the noise in cost information. For training dynamics, CBE provides better training initializations and safety controls on most tasks.\n\nThe following statements need to be read in conjunction with the uploaded learning curves (cf. the link in our abstract). We observe three main advantages of CBE on training dynamics. \n\n(1) Stabilizing training. On almost all tasks (e.g., “2x3HalfCheetah”, “2x3Walker2d”, and “3x1Hopper”), CBE results in smoother learning curves, compared with MOSDT without CBE. CBE mitigates volatility and reduces sharp declines. The cost binarization in CBE eliminates the noise in the cost information, thereby stabilizing the training.\n\n(2) Better training initialization. In most tasks (e.g., “2x4Ant”, “4x2Ant”, and “Multi-Point1”), CBE yields higher returns with safe cumulative costs at the beginning of training. CBE directly transmits the prior information about the reward-cost correlation to MOSDT by embedding safety signals into returns, enabling agents to learn more efficiently. The cost binarization in CBE eliminates the noise of cost data, improving data efficiency, thereby offering a better training initialization.\n\n(3) Better safety control. In most tasks (e.g., “2x1Swimmer”, “2x4Ant”, and “OverFingerMA”), CBE yields cumulative cost learning curves with faster and more stable convergence. For 11 out of 18 tasks, CBE reduces the cumulative costs (cf. the ablation study results in Table 2). CBE provides better safety control by improving the processing of cost information.\n\nWe will add the above analysis and related learning curves to the revised version."}]

Author rebuttal:

Meta-review: This paper addresses a novel problem formulation that combines aspects of MARL and OSRL in a non-trivial way. While the solution draws on ideas that are already present in the community, the way they are brought together is new, and the approach demonstrates clear empirical benefits. The formulation itself is interesting and contributes to expanding the scope of problems considered in this area.

The reviewers raised several concerns, most notably regarding the degree of novelty given that the method integrates existing modules (e.g., CDT, CTDE) along with smaller modifications (e.g., parameter sharing, CBE). Some reviewers questioned whether this rises to the level of a NeurIPS contribution. There were also a number of questions around the problem setting and empirical results. 

The authors provided thoughtful and detailed responses to these points, which help clarify the contribution. In particular, the rebuttal makes a stronger case for the non-triviality of the problem formulation and the benefits of the proposed design choices. 

Overall, this is a promising paper that presents an interesting formulation, demonstrates empirical gains, and helps expand an active research area. With revisions to better highlight the problem’s novelty and the rationale behind the design choices, the paper will be stronger.
Decision: Accept (poster)