Title: MOSDT: Self-Distillation-Based Decision Transformer for Multi-Agent Offline Safe Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: multi-agent offline safe reinforcement learning

decision transformer

self-distillation
Abstract: We introduce MOSDT, the first algorithm designed for multi-agent offline safe reinforcement learning (MOSRL), alongside MOSDB, the first dataset and benchmark for this domain. Different from most existing knowledge distillation-based multi-agent RL methods, we propose policy self-distillation (PSD) with a new global information reconstruction scheme by fusing the observation features of all agents, streamlining training and improving parameter efficiency. We adopt full parameter sharing across agents, significantly slashing parameter count and boosting returns up to 38.4-fold by stabilizing training. We propose a new plug-and-play cost binary embedding (CBE) module, which binarizes cumulative costs as safety signals and embeds the signals into return features for efficient information aggregation. On the strong MOSDB benchmark, MOSDT achieves state-of-the-art (SOTA) returns in 14 out of 18 tasks (across all base environments including MuJoCo, Safety Gym, and Isaac Gym) while ensuring complete safety, with only 65% of the execution parameter count of a SOTA single-agent offline safe RL method CDT. Code, dataset, and results are available at this website: https://github.com/Lucian1115/MOSDT.git


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes MOSDT, an algorithm for multi-agent offline safe reinforcement learning (MOSRL), and introduces MOSDB, a corresponding benchmark and dataset. The method incorporates policy self-distillation with a global information reconstruction mechanism that fuses observation features across agents. It employs full parameter sharing and introduces a cost binary embedding (CBE) module to encode safety constraints. Empirical results on MOSDB, which includes MuJoCo, Safety Gym, and Isaac Gym environments, show that MOSDT achieves higher returns on 14 out of 18 tasks, while maintaining safety and reducing execution-time parameter count compared to a single-agent baseline.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The authors extend multi-agent offline reinforcement learning and safe reinforcement learning to the underexplored setting of multi-agent safe offline reinforcement learning.\n- They provide a benchmark and dataset tailored to this new problem formulation.\n- The proposed approach achieves state-of-the-art performance under safety-constrained evaluation scenarios. A comprehensive ablation study offers insight into the rationale behind key design choices.\n\n# Weaknesses\n- There is room for improvement in the paper’s writing. For example, figure captions currently provide limited information and could be more descriptive to aid understanding. In Line 147, the symbol L appears to denote the loss, whereas Equation (1) uses the term Loss. This inconsistency in notation could be clarified.\n- While full parameter sharing across agents helps stabilize training, it may limit the applicability of the proposed method in scenarios where agents have heterogeneous action spaces.\n- Although the proposed cost binary embedding (CBE) module appears to improve information aggregation and support specific cost targets, its impact on training stability is unclear. It would be helpful if the authors could provide learning curves or additional ablation results to better assess the influence of CBE on training dynamics.\n\nQUESTIONS:\nPlease refer to the weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the setting of multi-agent offline safety reinforcement learning. They proposed the first algorithm named MOSDT in this setting and showed its effectiveness through experiments. Besides, they constructed the first dataset and benchmark for this setting. They build their algorithm based on constrained decision transformer (safety offline RL) and centralized training with decentralized execution framework (multi-agent offline RL). Their innovations are from three perspectives. First, they proposed a cost binary embedding module to convert the safety signal to binary signal which solves the issue of unsafe intermediate actions. Second, they introduced self-distillation technique to multi-agent offline RL which simplifies the training pipeline, decreases training parameters and saves the training time. Last, they propose a full-parameter sharing design across students to stabilize the training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. They proposed the first algorithm for the setting of multi-agent offline safety reinforcement learning which initiates the study in this setting.\n2. The proposed techniques including cost binary embedding and self-distillation look reasonable to me for this problem. \n3. They did comprehensive ablation studies to show the effect of different components in their algorithm network.\n\nWeaknesses:  \nNot much. The self-distillation technique is not tailored to safety RL which means it can be applied to multi-agent offline RL without safety constraints. Would be interesting to see how it works outside the safety setting compared to works such as MADTKD. The experiments are all based in the safety setting.\n\nQUESTIONS:\n1. Can the authors explain more on the full parameter sharing design? Based on my understanding, it is to initialize the network of all student agents with the same parameters before the training pipeline starts.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MOSDT, a decision transformer architecture for multi-agent offline safe reinforcement learning. The method consists of several points of this approach, including (1) policy self-distillation integrated with global information reconstruction, (2) full parameter sharing across agents to stabilize training and reduce model complexity, and (3) cost binary embedding that replaces specific cost targets with binary safety signals for more flexible safety handling. The authors also release MOSDB, the first benchmark dataset for MOSRL, and show that MOSDT achieves SOTA performance on 14 out of 18 tasks, ensuring complete safety while requiring only 65% of the execution parameter count compared to baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This paper addresses the understudied and challenging domain of multi-agent offline safe reinforcement learning, where issues of scalability and safety remain largely unresolved.\n- On the new MOSDB benchmark, MOSDT achieves state-of-the-art (SOTA) performance on 14 out of 18 tasks, spanning diverse environments such as MuJoCo, Safety Gym, and Isaac Gym, while strictly adhering to safety constraints.\n- Full parameter sharing reduces the execution parameter count by 58% and improves inference time compared to CDT.\n\nWeaknesses:\n- Limited real-world baseline comparison: The evaluation primarily adapts single-agent baselines to the multi-agent setting. Strong multi-agent baselines—such as recent diffusion-based methods (e.g., MADiff) or model-based safe MARL algorithms (e.g., constraint-aware MBRL)—are not included. These could offer valuable comparisons, especially under tight safety budgets or dynamic constraints.\n- Cost binarization limitation: The proposed CBE encodes safety using hard binary thresholds, which may limit adaptability in settings with soft, dynamic, or probabilistic constraints.\n- Limited safety analysis: Although all reported tasks satisfy safety thresholds, the failure modes (e.g., the four unsafe cases in the ablation study) are not thoroughly analyzed. Deeper investigation into these cases could inform future improvements in robustness.\n\nQUESTIONS:\nCan the authors provide a deeper analysis of the failure cases observed in the ablation study (e.g., the four tasks where safety constraints were violated)? What factors contributed to these violations?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates multi-agent offline safe reinforcement learning. Built on top of CDT and MADTKD, this paper proposes MOSDT, which performs (1) decentralized student supervised learning; (2) centralized teacher supervised learning; (3) policy self-distillation. Also, this paper utilizes several implementation tricks: (1) full parameter sharing and (2) \nCost binary embedding. Extensive experiments are conducted on a self-collected dataset MOSDB to demonstrate the effectiveness of the proposed algorithm.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. Comprehensive ablation studies are conducted to verify each component of the proposed algorithm.\n2. The techniques are simple and easy to follow.\n\n### Weaknesses\n\n1. The setting seems too incremental for considering offline, multi-agent, safety RL. Please motivate this setting with several important application scenarios. It would be better to include a benchmark that tackles one more practical scenario to make this setting well-justified.\n\n2. The proposed algorithm is also incremental over two existing algorithms, one from offline safety RL algorithm, CDT, and one from offline multi-agent RL algorithm, MADTKD. There is not any clear motivation in the methodology to justify why we should design MOSRL algorithms as in MOSDT.\n\n3. The quality of the built benchmark MOSDB seems low. As shown in Table 1 and Table 11, MOS Goal and MOS Isaac Gym cannot provide informative comparison between MOSRL algorithms, as the performance gap can be filled even with several random seeds. Same phenomenon can be overserved in Table 2 and Table 12. This makes the reported performance based on SOTA on how many settings meaningless. Also, it is weird to report mean values and standard deviations in two separate tables. Please consider merge them in one table.\n\n4. The motivation of the paper is clear. It states that multi-agent offline safe Reinforcement Learning is lacking in the existing literature, but it does not specify why this particular technique is necessary. What specific problem is this paper to address? Why are existing solutions inadequate for this purpose?\n\n5. What is the exact novelty compared to reference [5]? \n\n6.  MOSDB in Section 3.4: What is the nature of the tasks within the datasets? How are the trajectories collected (what is the meaning of the trajectories in the dataset, anyway)? The authors are expected to elaborate further on this newly created dataset, including the motivation, tasks, evaluation methods, baselines, etc.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-05"}