Title: Boundary-to-Region Supervision for Offline Safe Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Offline safe reinforcement learning
Abstract: Offline safe reinforcement learning aims to learn policies that satisfy predefined safety constraints from static datasets. Existing sequence-model-based methods condition action generation on symmetric input tokens for return-to-go and cost-to-go, neglecting their intrinsic asymmetry: RTG serves as a flexible performance target, while CTG should represent a rigid safety boundary. This symmetric conditioning leads to unreliable constraint satisfaction, especially when encountering out-of-distribution cost trajectories. To address this, we propose Boundary-to-Region (B2R), a framework that enables asymmetric conditioning through cost signal realignment . B2R redefines CTG as a boundary constraint under a fixed safety budget, unifying the cost distribution of all feasible trajectories while preserving reward structures. Combined with rotary positional embeddings , it enhances exploration within the safe region. Experimental results show that B2R satisfies safety constraints in 35 out of 38 safety-critical tasks while achieving superior reward performance over baseline methods. This work highlights the limitations of symmetric token conditioning and establishes a new theoretical and practical approach for applying sequence models to safe RL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a critical limitation in existing sequence-model-based approaches for offline safe reinforcement learning (RL), namely the symmetric conditioning of reward-to-go (RTG) and cost-to-go (CTG) in decision transformer-style models. The authors propose Boundary-to-Region (B2R), a novel framework that redefines CTG as a strict boundary constraint, enabling asymmetric conditioning. The B2R framework includes trajectory filtering, CTG realignment, and the use of Rotary Positional Embeddings (RoPE) to enable safe and effective policy learning. Theoretical analysis under simplifying assumptions is provided, and experiments across 38 safety-critical tasks compare B2R with baselines including BC-All, BC-Safe, CDT, BCQ-Lag, CPQ, COptiDICE, and TraC.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper identifies a concrete and well-motivated issue in the use of symmetric token conditioning in safe RL via supervised learning, which is both conceptually sound and practically impactful.\n\n- The B2R framework innovatively addresses the reward–cost asymmetry by aligning all safe trajectories to the deployment-time constraint budget through CTG realignment, coupled with trajectory filtering. The design is simple yet powerful.\n\n- The method is extensively validated on the DSRL benchmark, showing state-of-the-art results in many environments, outperforming CDT, BC-Safe, and others, while preserving constraint satisfaction.\n\n- Safety guarantees are derived under clear assumptions, providing both expected and probabilistic bounds on cost violation.\n\nWeaknesses\n- The method’s reliance on trajectory filtering limits its applicability when safe data is sparse or when behavior policies collect highly unsafe trajectories. This is acknowledged by the authors.\n\n- While the authors provide a valuable comparison of multiple CTG realignment strategies, the clear empirical superiority of the simple \"Shift\" method may suggest that additional exploration of more complex alternatives offers diminishing returns in this context.\n\n- Although discussed briefly, potential negative use cases (e.g., overly conservative policies in safety-critical control) or mitigation strategies are not detailed.\n\nQUESTIONS:\nQuestions\n1. The authors mention that B2R’s performance may degrade in settings with limited safe data and suggest data augmentation or generative modeling as possible solutions. Do the authors have any preliminary insights or intuitions on which strategies might be most promising in this context? Any specific challenges they foresee in integrating such approaches with the B2R framework?\n\n2. B2R assumes a fixed cost budget across all episodes for CTG realignment. Have the authors considered how B2R might extend to non-stationary cost regimes, for instance, where cost thresholds vary due to environment drift or contextual constraints? Do they anticipate significant challenges in adapting the CTG realignment process in such settings?\n\n3. While B2R ensures constraint satisfaction via hard filtering of unsafe trajectories, might this strategy limit the diversity of the training signal by excluding high-reward but marginally unsafe behaviors? Could the authors comment on the potential of soft filtering mechanisms to trade off safety and exploration more flexibly?\n\nMinor clarity issue: \nThe abstract introduces acronyms such as RTG, CTG without any definition or expansion. While these terms are common within the RL community, abstracts should remain accessible to broader audiences.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a fundamental limitation in existing sequence-model-based offline safe reinforcement learning (RL) methods, such as those inspired by Decision Transformers (DT). These methods typically treat return-to-go (RTG) and cost-to-go (CTG) signals equivalently. The authors argue that this approach neglects the intrinsic difference between reward and cost: RTG is a flexible performance target to maximise, while CTG should represent a rigid safety budget or boundary to strictly adhere to. This oversight leads to unreliable constraint satisfaction, especially when policies encounter cost trajectories outside the training data distribution. To overcome this, the paper proposes Boundary-to-Region (B2R), a novel framework that enables asymmetric conditioning through cost signal realignment\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Problem and issues articulated well\n2. Interesting and intuitive methodology\n3. Detailed experimental results and strong empirical performance.\n\nWeaknesses:\n\n1. There is a reliance on high-quality safe trajectories. \n2. While authors have provided comparison against transformer based approaches, intuitive issues with more existing approaches (e.g., TraC) have not been provided. \n3. The assumptions employed in theoretical results are unclear.\n\nQUESTIONS:\n1. In the problems where the results provided are not safe, can the authors elaborate on why the solution does not work well? Is it because of tight constraints or complex cost functions or something else? It would help to provide a more detailed analysis. \n\n2. Transformer based approaches for RL are known to struggle in environments where there is transitional uncertainty.  Can authors clarify how they have addressed this issue?\n\n3. The ablation study on CTG realignment strategies is insightful, with \"Shift\" performing the best overall. However, the paper also mentions that \"environment-specific design choices can further enhance performance\". Could the authors provide more detailed guidance or propose a meta-strategy for adaptively selecting or tuning the CTG realignment method (e.g., Shift, Avg, Rand, Scale) based on dataset characteristics, environment properties, or observed real-time performance, rather than a fixed choice?\n\n4. The assumption of \"prediction-error bound\" seems to be hard to visualize. Can authors clarify where this assumption will be violated?\n\n5. How would B2R deal with situations where there are not many high quality safe trajectories available?\n\n6. How does B2R perform better than the TraC method intuitively?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces B2R (Boundary-to-Region), a framework for offline safe reinforcement learning that improves constraint satisfaction in sequence-model-based policies. The key innovation is treating cost-to-go (CTG) as a rigid safety boundary rather than a flexible optimization target, in contrast to return-to-go (RTG). The method involves:\n- Trajectory filtering to remove unsafe samples.\n- CTG realignment to align all safe trajectories with the deployment-time cost constraint.\n- Use of rotary positional embeddings to enhance temporal modeling. \n\nEmpirical results on 38 tasks show that B2R outperforms baselines, and the paper provides theoretical analysis to justify its safety guarantees.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The asymmetry between RTG and CTG is a relevant.\n- Trajectory filtering and CTG realignment are simple yet effective interventions.\n- B2R achieves strong empirical results, outperforming or matching baselines in both reward and constraint satisfaction on 35 out of 38 tasks.\n- The method is evaluated through ablations on core components including CTG realignment, RoPE, and filtering.\n\nWeaknesses:\n- B2R depends on access to a sufficient number of safe trajectories and lacks stitching capabilities.\n- The CTG realignment strategy (“Shift”) is heuristic and not learned or adaptive.\n- The method does not generalize to multiple constraints without retraining, yet remains as computationally heavy as CDT.\n\nMinor:\n- Line 238: The paper states that “these approaches discard potentially useful trajectories once they exceed the limit,” but B2R also performs filtering based on the same cost threshold. Could the authors clarify how B2R’s filtering is fundamentally different or more selective?\n- Appendix C.3: It seems that the cost thresholds for SafetyGymnasium and BulletSafetyGym may have been inadvertently swapped.\n\nQUESTIONS:\n- Please check weaknesses.\n- How does B2R perform in datasets with sparse or skewed distributions of safe trajectories, such as under tight cost limits (e.g., FISOR-style setups)?\n- Could the CTG realignment strategy be made adaptive or learned, potentially conditioned on environment dynamics or trajectory characteristics?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Boundary-to-Region (B2R) Supervision, a framework for offline safe RL using transformer-based sequence models. The key insight is that reward-to-go and cost-to-go should not be treated symmetrically as in CDT, especially when cost-to-go (CTG) corresponds to rigid safety constraints. B2R first filters unsafe trajectories from the offline dataset, then realigns the cost-to-go of all remaining (safe) trajectories to match the safety threshold, enriching the training data for the sequence model. Empirical results show that it achieves better reward while satisfying the cost constraint in most tasks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n* Comprehensive empirical study: 38 tasks from the DSRL benchmark are evaluated, with strong results on both safety and reward metrics. Ablations of key components are included.  \n\n* Theoretical analysis: Provide probabilistic guarantees on safety and reward performance of B2R.\n\n* Simple, practical method: Trajectory filtering and cost realignment are easy to implement and does not introduce more hyperparameters. The approach can be integrated into standard Decision Transformer-style methods.\n\n### Weaknesses\n\n1. **Obvious baseline not included**: The policy oscillation around the boundary in Fig2 is to be expected if the model is trained only on near-boundary trajectories. A natural baseline --- CDT trained with a buffer (i.e., a margin below the actual constraint) --- should be included (CDT with a margin), to directly test if realignment is strictly necessary or just a margin effect.\n\n2. **Unclear use of region-conditional supervision**: The role of region-conditional cost supervision (R_t) is not clearly explained. After cost-to-go shifting, is the model still conditioned on a range, or only on the boundary? Is region conditioning used at training or inference, or both?  \n\n3. **RoPE rationale and ablation missing**: Rotary position embeddings are used but not discussed in detail. The authors should explain why RoPE helps in this context, and provide comparisons with standard positional embeddings as in Decision Transformer / RvS.  \n\n4. **Assumption of sufficient safe data after trajectory filtering**: B2R filters out unsafe trajectories entirely. In domains with limited safe data, this may significantly reduce coverage or reward diversity.\n\n5. **Incremental novelty**: The incremental tweaks --- CTG realignment, trajectory filtering, RoPE --- seems a little bit arbitrary and heuristic. The authors might want to highlight the novel contribution of this paper.\n\n6. **Related work lacks differentiation**: The Related Work section lists prior approaches but does not explicitly position or differentiate B2R from them.  A concise positioning paragraph would help readers and reviewers assess novelty.\n\n7. **Typo in main text**: Line 197 refers to Equation 4.3, which I think should be Equation 9.\n\nQUESTIONS:\n1. Did the author assess “buffered threshold” baseline (CDT trained with a stricter threshold)?\n\n2. How does B2R handle cases where safe trajectories are rare? What is the minimum “safe coverage” required?\n\n3. How does RoPE compare to standard positional embeddings in CDT or Decision Transformer? Any reason to prefer RoPE?\n\n4. Can the authors clarify exactly how region-conditional supervision is used during training and inference? Is it purely boundary-conditioned after cost-to-go shifting?\n\nLIMITATIONS:\nThe paper pointed out the limitation of the method in environment with limited safe data in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}