Boundary-to-Region Supervision for Offline Safe Reinforcement Learning
Authors: Huikang Su
Dengyun Peng
Zifeng Zhuang
YuHan Liu
Qiguang Chen
Donglin Wang
Qinghe Liu
Venue: NeurIPS 2025 poster ()
Keywords: Offline safe reinforcement learning
Primary area: reinforcement_learning

Abstract: Offline safe reinforcement learning aims to learn policies that satisfy predefined safety constraints from static datasets. Existing sequence-model-based methods condition action generation on symmetric input tokens for return-to-go and cost-to-go, neglecting their intrinsic asymmetry: RTG serves as a flexible performance target, while CTG should represent a rigid safety boundary. This symmetric conditioning leads to unreliable constraint satisfaction, especially when encountering out-of-distribution cost trajectories. To address this, we propose Boundary-to-Region (B2R), a framework that enables asymmetric conditioning through cost signal realignment . B2R redefines CTG as a boundary constraint under a fixed safety budget, unifying the cost distribution of all feasible trajectories while preserving reward structures. Combined with rotary positional embeddings , it enhances exploration within the safe region. Experimental results show that B2R satisfies safety constraints in 35 out of 38 safety-critical tasks while achieving superior reward performance over baseline methods. This work highlights the limitations of symmetric token conditioning and establishes a new theoretical and practical approach for applying sequence models to safe RL.

Boundary-to-Region Supervision for Offline Safe Reinforcement Learning
Huikang S Dengyun Peng**
Zifeng Zhuang
Yuhan Liu
Qiguang Chen Donglin Wang*
Qinghe Liu
Harbin Institute of Technology. Weihai, Westlake University, Hangzhou
Harbin Institute of Technology. Harbin
suhuikang 120 23@gmai1. com, dypeng@ir www.hhtth.... edu. cn qingheliuGhitwh edu.cn
Abstract
Offline safe reinforcement learning aims to learn policies that satisfy predefined safery constraints from static datasets. Existing sequence-model--ne methods condition action generation on symmetric input tokens for return-to-go and cost- to-go, neglecting their intrinsic asymmetry: return-to-go (RTG) serves as a flex- ible performance target, while cost-to-go (CTG) should represent rigid safety boundary. This symmetric conditioning leads to unreliable constraint satisfaction, especially when encountering autoo--distribution cost trajectories. To address this, we propose Boundary-to- conditioning through cost signal realignment. B2R redefines CTG boundary constraint under a fixed safety budget, unifying the cost distribution of all feasible trajectories while preserving reward structures. Combined with rotary positional embeddings. it enhances exploration within the safe region. Experimental results show that B2R satisfies safety constraints in 35 out of 38 safety-critical tasks while achieving superior reward performance over baseline methods. This work highlights the limitations of symmetric token conditioning and establishes a new theoretical and practical approach for applying sequence models to safe RL. Our code is available https: //git///tttt /gggttu........ com/ /HuikangSuBBBR.
(B2R), a framework that enables asymmetric
Introduction
Offline reinforcement learning (RL) enables policy learning from static datasets without risky online interactions [28, 7], a critical capability for safety-sensitive applications such as autonomous driving [21, 23, 37], robotics [4, 5], and industrial control systems [36]. While conventional offline RL focuses on maximizing rewards under distributional shift [17, 38, 24], real-world deployments often demand adherence to safety constraints [9, 3]. This necessitates offline safe RL, which seeks policies that maximize cumulative rewards while ensuring expected costs remain below predefined thresholds.
Recent advances in Reinforcement Learning via Supervised Learning (RvS), exemplified by the Decision Transformer (DT) [2], have shown promise by autoregressively generating actions con- ditioned on historical states, actions, and return-to-go (RTG) signals. However, extending DT to safe RL reveals fundamental limitation: existing methods naively apply symmetric conditioning mechanisms to both RTG and cost-to-go (CTG), overlooking their inherent asymmetry. Specifically, RTG serves flexible performance target to pursue, while CTG represents rigid safety budget to enforce a distinction that existing approaches fail to capture.
"quual contribution
Corresponning Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

B2R
DT
D'rrjectory @: cost path
Trajectory Filtering 2
CCTG Realigment
Trajectory Dataset cost the puth
filtered by threshold
Shift CTG to threshold
Generated Policy
Policy Training
Generated Polley
Safe Region
Constraint Boundary
Causal Transforner
Cost Expectation = 1.5
Cost Expectation
Goal: The robot need to arrive the end
Constraint Threshold: cost must not exceed 2.
Figure 1: Overview of the B2R framework compared DT methods. DT approaches rely boundary-aligned trajectories whose costs happen to match the constraint threshold, making difficult to supervise diverse safe behaviors and often resulting in unsafe, high-cost actions. To address this, the B2R pipeline introduces Trajectory Filtering to remove unsafe samples and CTG Realignment align all remaining trajectories with the deployment-time cost threshold. This transforms sparse boundary supervision into consistent training over a broader Safe Region, reducing expected cost (1.5). contrast, DT methods lack such filtering and alignment, frequently generating actions beyond the constraint, leading to higher expected cost (2.0), as shown in the right subfigure.
This oversight leads to unreliable constraint satisfaction [1], particularly when policies encounter cost trajectories outside the training distribution. Methods like Constrained Decision Transformer (CDT) [26] treat RTG and CTG as equivalent input tokens, conflating the orthogonal objectives of reward maximization and safety assurance. To address this, we propose Boundary- to-Regon (B2R), a framework that introduces asymmetric conditioning through CTG realignment. Our core innovation lies in unifying feasible trajectories under a fixed safety budget by redistributing cost signals while preserving reward structure. This redefines CTG boundary constraint rather than variable target, decoupling safety guarantees from reward optimization. Combined with trajectory filtering and rotary positional embeddings [31], B2R enables comprehensive exploration of the safe action space while maintaining strict cost adherence. Figure illustrates how B2R broadens supervision beyond narrow constraint-alignedd trajectories, enabling stable learning over the entire safe region.
Experiments on 38 safety-critical tasks demonstrate B2R's effectiveness: it satisfies safety constraints in 35 environments while achieving competitive rewards [25]. Our findings underscore the necessity of abandoning symmetric token conditioning when applying sequence models to safe RL. The contributions of this work are threefold:
1. Problem Identification: We identify and formalize fundamental symmetry fallacy in existing RvS methods [2, 26), where the flexible nature rewards and the rigid nature of costs are improperly treated symmetric signals.
Methodology and Validation: We propose region-level supervision, a new paradigm for offline safe RL. We then introduce B2R, logically coherent framework designed to instantiate this paradigm, demonstrating how components like trajectory filtering and CTG realignment form a mutually reinforcing system, not collection of ad-hoc tweaks.
Theoretical Foundation: We provide initial theoretical analysis of B2R's safety compliance [1] under simplified assumptions and empirically validate its superiority in balancing reward and safety across 38 diverse tasks.
Related Work
2.1
Offline Reinforcement Learning and RvS
Offline RI. learns policies from static datasets [20], with methods addressing distributional shift by constraining policies near the behavior policy, such BCQ [8], CQL [17], BPPO[39], and BEAR

---

[16]. While these methods leverage value-based objectives [33] to mitigate policy deviations, they often face challenges such as the "deadly triad" [32].
Reinforcement Learning via Supervised I Learning [6] reframes reinforcement learning as supervised learning problem [30, 27], where policies are conditioned signals such goals rewards. Decision Transformer [2] uses transformer to autoregressively generate actions conditioned on states and returns-to-go. Reinformer [40] improves upon DT by incorporating return maximizing objective into sequence models, guiding action selection without relying on pre-specified RTG. ConDT [15] introduces an enhanced contrastive loss to structure retum-dependdnt representations. leading to significant performance gains.
2.2 Offiine Safe Reinforcement Learning
Offline Safe RL combines safe RI. [9, 12] and offline RL by enforcing safety constraints in static datasets to ensure reliable policy learning. Early methods, such as Lagrangian optimization [34] and distribution correction [19], incorporate cost constraints during training but less flexible with diverse datasets. A key focus Offline Safe RL is enhancing dataset quality to address safety challenges. OASIS [35] uses a conditional diffusion model reshape datasets for improved safety compliance, while partitioning-basee approaches [10] classify trajectories into desirable and undesirable subsets to avoid unsafe behaviors. Constrained Decision Transformer [26] highlights the importance of trajectory data by leveraging sequential modeling to enforce safety constraints.
FAWAC [13] uses a learned feasibility critic down-we wwigh unsafe actions during policy optimization in offline RL. LSPC [14] constrains policy learning in a latent trajectory space inferred from both reward and safety signals.
While prior methods have made significant strides, B2R is distinctly positioned by addressing the fundamental challenge of sparse and symmetric supervision in sequence-basee offline safe RI. Unlike approaches such as CDT [26]. which are limited to sparse boundary supervision by conditioning on specific cost-to-go values, B2R introduces region-wide supervision. By realigning the costs of all safe trajectories to a unified boundary, it learns from a much denser and more diverse set of behaviors. Furthermore, B2R differs from filtering- based methods like TraC [10], which classify and often discard large portions of the dataset. Instead of merely selecting safe trajectories, B2R's s core contribution to transform them, retaining their behavioral diversity to create more robust policy. Thus, B2R's 's novelty lies in its principled approach to reshaping the supervision signal itself to resolve the underlying symmetry fallacy.
3 Preliminaries
3.1 Offline Safe Reinforcement Learning
We formalize the safe reinforcement learning problem using the Constrained Markov Decision Process (CMDP) framework [1]. A CMDP M defined as a tuple (S, P.r, c, Ho). where S and A denote the state and action spaces, S x A x S [0, is the transition probability function, r: S x A x S - IR the reward function, e: A x S [0, Cmax] is the cost function bounded by known constant Cmax 2 0, and Mo is the initial state distribution over S.
{(st, de, Te, ce) as R(T) En r(st, at, $t+1) and C(7) = "Eto c(st; ae, 8t+1), respectively. where H is the horizon length. A policy S x A + [0, 1] is considered feasible if its expected cumulative cost does not exceed the threshold & E [0, +oo), i.c., [[C(T)] S K.
We define the cumulative reward and cost of a trajectory
The goal of offline safe reinforcement learning is to learn feasible policy æ that maximizes the expected return while satisfying a cost constraint:
π arg max B, [[(T]] subject to (C()) S K.
(I)
In the offline setting, the agent has no access to environment interaction and must instead learn from static dataset D = [ collected by unknown or suboptimal behavior policies. This introduces significant challenges such as distributional shift and limited coverage, making it difficult to simultaneously achieve high return and strict safety compliance.
3

---

3.2 Reinforcement Learning via Supervised Learning
To address distributional shift in offline reinforcement learning, Reinforcement Learning via Su- pervised Learning [6] reformulates policy learning as conditional sequence modeling, Instead of estimating value functions or applying dynamic programming, RvS learns conditional distribu- tions π (ax (z)), where is user-specified signal like return, goal, trajectory-level attributes.
representative approach in the RvS paradigm is the Decision Transformer [2], which treats offline reinforcement learning as conditional sequence modeling. In safety-aware variants, each trajectory is represented as sequence of tokens (R,, C₁ St. at), where return-to-ge oo Rs = Σ "% and cost-to-go go č = E
C denote future cumulative reward and cost. The model is trained predict actions autoregressively from prior context via behavior cloning:
Lac (0) = ED [ log To (a." (a(")
- C S K:t _
6(m)
In this formulation, (()) ~ D denotes a trajectory from the offline dataset, the parameter K defines the length of the context window, and 0 denotes the parameters of policy model. The model predicts the next action a, given fixed-length context window of return, cost, state, and action tokens from steps K to t. During deployment, the model generates actions autoregressively given the initial condition tokens (Ro, Co) and the observed state sequence.
4 Method
To address the symmetry fallacy identified the introduction, this section details the Boundary-to- Region framework. B2R is not collection of ad-hoc components but a coherent system designed to implement our proposed region-wide supervision paradigm. This system consists of three interlocking parts: (1) trajectory filtering to define the safe region, (2) CTG realignment to create dense and uniform supervision signal, and (3) RoPE to preserve the temporal dynamics of this signal. The following subsections will elaborate on the motivation and implementation of each component within this unified system.
4.1 Limitations of Symmetric Token Conditioning
In CMDP formulations, the optimization objective (Eq. 1) reveals fundamental asymmetry: reward signals rank feasible policies, while cost signals define whether policy is feasible all. This asymmetry is not merely modeling artifact-it is essential to the semantics of safe decision making,
However, modern DT-style sequence models do not reflect this asymmetry. Instead, they encode both re- ward and cost as RTG and CTG tokens, treating them symmetrically to condition action prediction. At each timestep, these tokens are computed from the trajec- tory's future cumulative reward and cost, and are used conditioning signals during training, as defined by the loss in Eq. 2. At deployment time, the model is queried with user-specified initial values Ro and Co representing the desired return and the target cost bud- get. This failure to reflect the asymmetry between reward and cost, gives rise to two key challenges in safe offline policy deployment.
Meal V
Non V
D2R.
Time 930
Figure 2: Velocity profiles in simplified MetaDrive scenario. Training on boundary- aligned trajectories results in unstable behav- ior and frequent violations (Non-ideal_ _V) while B2R achieves smooth, constraint- compliant control.
The first is the difficulty of token selection itself. In DT-style models, both Rg and Co are treated sym- metrically as conditioning signals, but they serve very different purposes: Ro sets a performance target, while Co enforces a safety limit. Choosing these values requires balancing ambition and feasibility. A large return target RS encourages high-return behavior, but overly optimistic values may push the policy into out-of distrrbiittii (OOD) regions not supported by the offline data. At the same time, determining compatible cost token Co that sup- ports the desired return while satisfying safety constraints is particularly difficult, as the appropriate

---

trade-off is often unknown and hard to infer from data. Notably, trajectories with cost close to the safety threshold are sparse, and high-retum behaviors may occur lower cost levels.
The second challenge remains even when the initial CTG fixed to a known deployment constraint, typically Co = K. In this case, the model is expected to imitate high-reward behavior under constraint threshold, but the dataset may contain few trajectories whose cumulative cost is close K, limiting the quality of supervision available at training time. Moreover, while reward prediction errors degrade performance, cost modeling errors can cause constraint violations. This risk is especially high near the constraint threshold, where small errors can lead to unsafe, hard-to- tot--errerrllzee behavior.
Figure 2 illustrates this issue more clearly in simplified MetaDrive environment. The agent is rewarded for higher speeds but penalized when velocity exceeds 10. This structure creates a distinction within the safe region itself: behaviors the boundary (v=10) are less robust and relatively riskier than those with a safety margin (v<10). The Non-ideal_ curve, trained only on boundary-aligned (v=10) trajectories, learns a brittle policy that constantly overcorrects, failing respect this nuance. In contrast, B2R, by learning from diverse behaviors deep within the safe region, develops a robust control policy that successfully maintains this crucial safety margin.
4.2 From Boundary Supervisionto Region- Wide Supervision
To address these challenges, we introduce a consistent training strategy that aligns all trajectories with an initial CTG token Co τ, while broadening supervision beyond the narrow set around Noundary-alignedd examples. B2R accomplishes this by CTG realignment in the dataset to expose the full spectrum of safe behaviors. We now formalize the supervision distinction central to this design:
Definition 1 (Constraint Boundary- vs. Safe Region-Conditional Supervision). At time step I, the learner receives cost-to-go token C2 E (0, ks). Given suffix trajectory Tr (St, , sH), define its remaining cost Ci(T) Σ 1
k=t c(sk, ax))
We define two conditioning strategies over supervision labels: coaudaar-cooditiona supervision uses trajectories within a narrow band (C,, c), while region-conditions supervision includes all trajectories in the safe region (C,), without modifying the model or training objective.
B,(CC,,)) c) - (TT Ca(T) e ICs - C2 ddd R.(C.) - - - [C(() <a]
(3)
where is small tolerance around the constraint threshold.
B2R achieves this through trajectory filtering and cost realignment. Filtering removes un- safe trajectories, ensuring feasibility, while re- alignment modifies costs to match the con- straint token without changing state-action se- quences. Additionally, we use rotary position embeddings (RoPE) to encode temporal struc- ture, maintaining consistent conditioning across the safe region.
Algorithm Houndary-to-Reiinne Framework
Require: Offline dataset D, cost threshold k,
model parameters 0
Ensure: Policy aa(a; Roo,, Cu.e, Su:1, au:-11
1: 1. Trajectory Filtering:
2: Filter safe trajectories:
3: D = (T D C(T) S n}
4: 2. CTG Realignment:
5: for each 7 E D do
Trajectory Filtering. To prevent unsafe tra- jectories from negatively impacting the policy, the B2R framework employs trajectory filtering as preprocessing step. We define safe trajec- tory trajectory Tsale satisfying C(() S K, and define the corresponding safe dataset Daate as:
6: Shift CTG: C = C2 + (n C(r))
7: end for
8: 3. Tokenization with RoPE:
9: For each timestep I, construct context:
10: = -Re-K: C/ St-Ka, at-K
Deate = fr E 'D C(T) S x).
(4)
11: Encode positions with RoPE
12: 4. Policy Training:
This filtering step ensures that all training tra- jectories are compliant with the deployment constraint K, enabling subsequent token con- ditioning to be consistent with test-time usage.
13: Train Transformer to minimize:
14: L(Ø) = Er~DAe |- log To o.)]
15: return π&8
5

---

CTG Realignment. As shown in Figure 3, conventional methods provide supervision only from trajectories whose cost lies near the constraint threshold (orange dots), leading to sparse and unstable learning signals. These methods discard large portions of feasible data simply because their cost does not exactly match the conditioning token.
In contrast, B2R leverages all safe trajectories by adjusting their costs to match the constraint, en- abling consistent supervision under a fixed token while drawing on diverse behaviors originally scat- tered across the safe region (orange box). This process decouples the conditioning input from the supervision signal: Ö = k. the model is consis- tently conditioned on a single boundary token, but learns to associate it with the diverse behaviors sourced from the entire safe region.
Return
Return
Return
Iaret R
target R
larget R
Ceroaski
Bratek
(4)
Constraint Boundary
Safe Region
coff-condition trajectorics
Offcondition trajectories
Figure 3: Supervision strategy comparison. Conventional methods (a. b) rely on sparse, Houndary-alignee trajectories. In contrast, B2R (c) realigns all safe trajectories to the constraint threshold (dashed line), transforming sparse boundary data into dense, region- wide supervi- sion. Orange dots denote compliant trajectories; dashed arrows show the realignment.
We adopt realignment strategy, called CTG-shift to ensure consistency between each trajectory and the fixed initial CTG:
C = C. + (n (n- C(T)).
(5)
Here the shifted CTG sequence { is constructed by adding a constant offset λ CTT to each step, such that Čj K, and the temporal profile of the original CTG is preserved. This adjustment ensures that the entire sequence remains strictly decreasing, and that each supervision token semantically aligned with the deployment-timg constraint. We compare four CTG realignment strategies: Shift (uniform offset), Avg (even redistribution), Rand (random redistribution), and Scale (multiplicative normalization). Detailed discussion are provided in Appendix B.1.
Model and Inference. The policy model in B2R adopts transformer-based autoregressive structure, augmented with RoPE, whose relative positional en- coding is better suited for capturing the step-by-stey cost dynamics introduced by our CTG realignment strategy. As shown in Figure 4, these tokens are pro- cessed by a Transformer encoder, which outputs the mean 14 of a Gaussian action distribution (a, |.) N N(H4, of). The model is trained via behavior cloning on CTG-aligned trajectories using fixed initial CTG
Linear
B2R Transformer
Enh- RaFE
Figure 4: Architecture of the Transformer model in the B2R framework. The model takes tokenized inputs consisting of states, actions, RTG, and CTG, and augments them with RoPE for improved temporal modeling.
C% K, and is queried autoregressively inference with consistent input, ensuring alignment with the deployment-ting safety budget. The model minimizes negative log-likeling likelihoood of action:
(BC = -ppiwaaa -log Ahe (a." (n)
(d(a) ((m) K Co Kir af
((m))
(6)
4.3 Theoretical Analysis
While B2R is conceptually simple, it fundamentally reshapes how RvS agents generalize under constraints. To explore how B2R balances reward maximization with cost constraint satisfaction, we analyze its theoretical foundations, focusing on the relationship between the learned policy's return and its adherence to the cost constraint. We first focus on safety adherence, analyzing how B2R ensures cost constraint satisfaction while maintaining safety margin. This analysis is built upon Assumptions and 2, leading to the results presented in Theorem 1.
Assumption (Safe-Aligned Data) After CTG realignment every training trajectory obeys
CH = Ć/ Cτ Cc. 0 S c S Cmaxy
Essentially, this ensures the agent never observes "unsafe" cost patterns during training-it learns from data where budget consumption evolves as if the agent were truly constrained. Additionally, we

---

assume that the dataset used during training remains static, ensuring consistent training data over the course of the learning process.
Assumption 2 (Prediction-Froon Bound). At deployment, the agent's ' implict cost predictions é (conditioned on trajectory history 7 must be sufficiently accurate:
H-1
Elee. a.l] o, O<oH < <,,
Cask- 2
(8)
1=
Here a bounds the per-step prediction error, while ensures the planned cost stays safely below k (accounting for potential error accumulation). The requirement oH prevents enror compounding from overwhelming the safety margin. In practice, this assumption can be violated if the agent encounters highly out-of-distribuiion states (leading to large single-step error that breaches the bound) or its planned trajectory is overly optímistic, leaving an insufficient safety margin a to account for cumulative errors. This assumption is based the premise that the model has sufficient expressive capacity to accurately predict the cost and can learn to replicate the policy.
Under Assumptions 1-2, B2R provides probabilistic and expected-cost safety guarantees:
Theorem 1(Safety Guarantees of B2R). Let T be generated with initial CTG Co K. Then: Pr[[ (7) S 2 - exp
(4- (1-at1)) 2HC
[
(T) S K o
Here, denotes an upper bound on the per-step cost prediction error during deployment. The proof Appendix A.I. Equation provides a dual guarantee of biggh-rooabbility and expected safety at the trajectory level for B2R: the probabilistic guarantee shows that larger safety margins d make constraint violations exponentially rare. The cost expection bound ensures the average trajectory stays 5 H below the budget,
under worst-case error.
We now analyze B2R's performance terms of maximum cumulative reward (Equation 1). Unlike safety, which minimizes cost, our focus here is whether enforcing safety degrades reward. Under the following assumption, B2R can preserve or even improve reward:
Assumption Apptimal-Coveragel The dataset contains a safe trajectory T" such that
C(T') and R(T*) max
R(T)
(10)
reDa
This assumption ensures that the return-optimal trajectory is within the feasible region, so B2R's filtering and realignment do not exclude from training.
Theorem 2 (Performance Superiority of B2R over Boundary Supervision). Under Assumptions 1 and 3. B2R guarantees:
RB2R (R) 2 R
Rboundary (1s)
(11)
That is, B2R achieves reward no worse than boundary-conditional supervision while satisfying
C(r) < k. The proof is provided in Appendix A.2.
Experiments
We evaluate the B2R framework on two fronts: (1) its ability to balance reward maximization and safety compliance under cost constraints, and (2) the individual impact of its key components, particularly cost realignment. We also examine B2R's effectiveness in selecting feasible actions compared to baseline methods.
Environments. Experiments are conducted on the DSRL. benchmark [25], which includes 38 sequential decision-makinn tasks of varying difficulty. This suite provides diverse and realistic testbed for safety-critical offline RL. Full environment details are in Appendix C.2.
Metrics. We report both reward and cost, aiming to maximize the former while keeping the latter below the constraint threshold. Each method is evaluated under three cost limits and three random seeds per task. Normalization details and metric definitions provided in Appendix C.3.
Baselines. We compare our approach with several state-of-the-art algorithms, including BC-AII, BC-Safe, CDT [26). BCQ-Lag (8), CPQ [34]. COPtiDICE [19], and TraC [10]. Some experimental results are obtained from the DSRL implementation, while others are sourced from the official codebase. See Appendix C.4 for details.
7

---

Table 1: Normalized reward and cost results. t indicates that higher values are better, while + indicates that lower values are preferable. Metrics are averaged over 3 cost thresholds, 20 evaluation episodes, and 3 random seeds. Bold marks safe agents with normalized costs below 1, while Blue highlights safe agents achieving the highest reward.
BCQ--a5
CPQ
OOPIDICE
10700
rewand?
cloa]
sewandd
roward"
PrintRation
0.1
1.05
0,06
0.52
0.55
1.68
0..24
1.73
0.99
3.2
0.13
1.35
0.17
0.91
0.11
0.96
PristBatin2
0.27
2.02
0.16
1.1
0.46
1.57
0.4
2.65
0.55
-.3
0.15
1.51
0.16
0.91
0.16
0.91
0.79
3.58
0.41
0.16
01.59
0.00
0.54
2.35
0.43
0.75
0.55
5,51
0.50
0,07
0.54
0.31
innnCindee
0.96
0.17
0.48
0.99
01.6
1.05
0.66
2.5
0.24
3.58
0.55
K.61
0.61
0.86
0.61
0.80
Pontaall
0.65
1.95
0.43
I.S4
0160
1:12
0.71
0.96
0.57
0.38
0.49
1.66
0.44
0.36
0.55
0.71
PoatGeal2
0.51
1.97
0.29
I.75
0.59
1.34
0.67
3.15
0.4
1.31
0.38
1.92
0.31
0.59
0.34
0.68
FomiPrsh]
0.19
11.1
0.13
8.43
0.24
0.88
0.35
0.86
0.2
0.83
0.13
0.83
0.15
0.42
0.23
0.67
FomiPrah2
0.15
0.91
0.11
0.8
0.21
0.65
0.25
0.99
0.11
L.04
0.00
1.18
0.15
0.5
0.16
0.76
CaBaltall
0.03
1.38
0.07
0.21
1.6
0.44
0.66
20.00
1.68
0.59
0J04
0.65
CaBulti22
-0113
1.24
.0.01
0.33
1.58
0.06
2.13
0.37
12.51
40007
1.99
0.62
-001
0.62
2.84
CaeClc
CarCeall
1.21
IL4T
0.76
1.42
0.72
CurGoall
0.23
1,05
0.48
1.25
0.5
1.44
0.52
0.22
0.66
CarPashi
0.22
0.36
0,33
0.31
0,4
0.25
0.43
4003
0.95
0.25
0.18
0.35
0.56
CarPush?
0.14
0.9
0.25
0.19
1.3
0.15
1.58
0.24
4.25
0.00
0,08
0.54
6.11
0.79
SwimmeNNslocin
0.49
4.72
0.51
1.07
0.66
0.96
0.45
6.58
0.13
1.66
0.53
7.58
0.55
3.21
0.49
0.46
Velciit
0.65
6.29
0.36
0.67
0.63
0.61
0.75
502
0.14
2.11
0.15
1.51
0.57
0.98
0,63
0.53
Horra Velucity
0.97
13.1
0.58
RSS4
1.0
6.01
1OS
15:31
0.29
0.74
0.85
6.8
0.96
2.5
0.00
Waller//Sttocitt
0.79
3.88
0.79
IL14
0.78
0.06
0.79
0.17
0.14
0.21
0,12
0.74
I164
0.06
0.79
0.81
AutVddacity
0.98
3.72
0.98
0.29
0.98
0.39
100
4.15
-1.01
0.8
1.0
3.25
0.97
0.15
0.99
0.22
Sym Average
0.46
3,33
0.54
1.06
0.5
3.29
0.27
2.79
0.37
2.65
0.40
0.92
0.42
0.73
BallRa
0.76
1.27
0.31
DrandRan
9.50
0.72
8.91
0.76
0.03
0.02
BalliCincee
0.74
0.37
1,00
0.69
2.36
0.64
0.76
0.59
0.07
0.59
CarChrcle
0.55
3.14
0.84
0.75
0.95
0.6
1.89
0.71
0.53
0.76
0.71
0.88
DroneCincle
0.72
3.63
0.56
0.57
0.63
0.98
0.5
3.07
-0.22
1.28
0.26
1.02
0,67
0.57
0.34
AncCiieee
0.58
4.9
0.4
0.96
0.54
1.78
0.55
2.87
0.0
0.0
0.17
5.04
0.47
0.98
0.45
1.36
BylletGym Average
0.64
3.36
0.52
0,82
0.68
1.04
0.74
3.13
0.33
1.12
0.25
2,55
0.61
0.52
0.62
0.50
sasysparse
0.17
1.54
0.11
0.21
0.17
0.23
0.75
5.03
40.66
0.67
0.96
5.34
0.35
0.05
0.77
0,70
eosymeat
0.63
2.82
0.04
0.29
0.45
0.54
0.7
3.64
-U.07
0.87
0.56
3.97
0.39
0.05
0.77
0.99
casylouse
0.27
0.32
0.62
0.26
0.47
-0.16
0.06
0.76
0.69
melismsuuse
0:37
1.1
0.44
1.16
-0.15
0.87
0.71
0.06
0.93
0.58
mediummens
0:75
0.75
1.53
-0.55
0.76
0.06
0.85
0.63
0.41
0.5
0.06
0.48
0.47
2.56
0.06
0.50
0.58
harodenue
0.98
0,21
-004
0.68
0.58
MetaDrive Averngs
0.42
2.06
0.18
0.58
0.42
0.8
0.54
2.05
-0.66
0.06
0.58
2.73
0.35
0,06
0.72
0.62
5.1 Results on DSRL Benchmark
Table shows that B2R effectively balances reward and cost, satisfying safety constraints in 35 out of 38 tasks and achieving the highest rewards in 20. It also ranks first in average score across Sifety-Gymmasium, Bullet Safety-Gym, and MetaDrive. B2R's ' robustness is notable, though it struggles in few exceptionally challenging environments like CarCircle1/2 and AntCircle, where most baselines also fail. These failures are attributable to the datasets: high-reward trajectories are coupled with high-cost actions, leaving the offline data with insufficient examples of the precise, long-horizon control required to navigate the narrow safe action space near unforgiving boundaries.
Compared to Transformer- based methods like CDT, B2R achieves lower cost in most environments while maintaining competitive reward. CDT's sensitivity to tight safety constraints stems from its supervision. This fundamental limitation cannot be reliably solved by simply training with a stricter, buffered cost threshold an approach that our experiments show leads an unstable safety-performance trade-off. In contrast, B2R's s cost realignment enables region-wide supervision, drawing from a denser and more diverse set of safe trajectories to ensure robust and consistent constraint adherence. A detailed analysis of the buffered baseline comparison is available in Appendix B.3.
reliance on sparse
B2R also demonstrates superior performance over methods like BC-Safe and TraC. While all approaches begin by filtering for safe trajectories, the fundamental difference lies in how they utilize the resulting safe data. BC-Safe Iearns from a sparse signal, while TraC relies on coarse, binary classification of trajectories into "desirable" and "undesirable" sets. In contrast, B2R's 's core innovation is to realign all safe trajectories, which transforms the learning problem into fine-grained temporal regression task. This creates a dense supervision signal that allows the Decision Transformer framework to better learn the causal dynamies between actions and future costs, enabling more effective adaptation in diverse safety-critical settings.
Figure 5 presents B2R's 's average reward and cost at each constraint level (LI-L3), grouped by benchmark suite. In MetaDrive, we observe a non-monotonic cost trend: cost increases from LI to
8

---

L2 before dropping at L3. This is likely because LI filtering excludes many high-cost trajecto- ries, biasing the dataset toward safer behavior the lowest constraint. Overall, B2R main- tains stable or improving rewards and shows decreasing cost under tighter constraints in most cases, demonstrating its robustness across vary- ing safety levels.
Reward Trend
Cost Trend
Normalized Vahe
0.8
0.4
BS
SG
MD
F1
12
Furthermore, to stress-test B2R's 's robustness un- der particularly stringent cost limits, we con- ducted direct comparison against FISOR using its original, stricter evaluation protocol. The re- sults, detailed in Appendix B.6, show that B2R achieves a superior safety-pes perfty-performance balance even in these data-scarce scenarios.
Constraint Level
Constraint Lovel
Figure 5: Average B2R performance on Bullet- SafetyGym (BS). SafetyGymnasium (SG), and MetaDrive (MD) across three constraint levels (LI- L3). Tighter constraints generally decrease cost while rewards remain stable or improve. The non- monotonic trend in MD is likely artifact of LI filtering bias (see Appendix C.3 for thresholds).
5.2 Ablation of B2R Components
We ablate three core components of B2R CTG realignment, RoPE, and trajectory filtering each removed independently while keeping the rest of the system fixed (Figure 6). We evaluate their impact reward and safety across tasks from three environment groups covering diverse dynamics and constraint structures.
AmtRun R
BallRun
CarCizele
AmRun
BallRun
CarCirele
B2R
WVO Realign
WrO RoPE
W/O
CarGoa12
AntVelocity R
HopperVelocity R
CarCioa22 C
AntVelocity
HopperVelocity
B2R
W/ Realign
RoPE
casymean R
mediumepprese
R
hardsparse
easymean
C
mediumsparese C
hardspamee
C
B2R
WVO Realign
RoPE
WPO Filter
0.0
0.5
0.0
Figure 6: Ablation study of B2R across tasks from the DSRL. benchmark. We assess the effect of three components: cost-to-go realignment (W/O Realign), trajectory filtering (W/O Filter), and our choice of positional embeddings. For the latter, we compare our default RoPE against standard absolute positional embeddings, which is labeled as "W/O RoPE' in the plots. The full B2R model achieves favorable reward and cost performance compared to ablated variants, highlighting the contribution of each module.
CTG Realignment. Removing CTG realignment results in higher cumulative cost across nearly all environments, with frequent violations of the safety threshold (e.g., BallRun, CarCircle, AntVelocity). Reward remains identical to the full model, aligning with theoretical expectation: CTG realignment alters only the cost signal while leaving the reward structure untouched.
Positional Embeddings. The results show that replacing RoPE with standard absolute positional embeddings (APE, labeled as 'W/O RoPE' in Figure 6) leads to significant performance drops in tasks such CarCircle, HopperVelocity, and easymean, with both higher costs and lower rewards.

---

This empirically confirms that RoPE's ability to capture relative temporal dependencies is more effective for modeling the mnneggriied, step-by-step cost dynamics inherent our framework, a task for which APE is less suited.
Trajectory Filtering. Removing trajectory filtering causes marked increase in cost and more unstable performance across seeds, particularly in tasks such as CarCircle, HopperVelocityy and hardsparse. While reward may slightly improve in some settings due to exposure higher-reward but unsafe data, the model frequently violates constraints. This trade-off highlights filtering's 's role in enforcing feasible supervision and stabilizing constraint satisfaction.
5.3 Cost-Aware Action Selection under Aligned Supervision
To examine how B2R improves constraint sat- isfaction, we compare its action distribution with a baseline in the easysparse environment using t-SNE (Figure 7). Actions are marked unsafe if their one-step cost exceeds the per- step budget. The baseline produces notable number of unsafe actions, scattered across dis- tinct state-action clusters. This suggests that aoundery-aligned supervision provides limited coverage, causing the model to associate cost only with narrow set of behaviors and gener- alize poorly to high-risk scenarios. In contrast, B2R reduces unsafe actions and yields more con- centrated, constraint-complin action clusters remain largely within the safe region, indicating better generalization to the intended constraint threshold.
With 122
WAD
Actions
a Uneafe
States
States
Figure 7: t-SNE visualization of actions from the baseline (right) and B2R (left) in the easysparse environment (threshold = 10). Actions are labeled as safe (blue) their one-step cost is below the threshold divided by episode length, and unsafe (red) otherwise. B2R selects fewer unsafe actions than the baseline, reflecting its cost-aligned train- ing under a unified constraint.
responses.
Conclusion and Future Work
In this work, we proposed Boundary-to-Reet reinforcement learning. By addressing the reward-cost asymmetry. B2R realigns the cost-to-go tokens of all trajectories reflect the desired safety threshold and filters out infeasible behaviors, ensuring the model learns safe, cost-compliant policies. Extensive experiments on benchmark tasks validate B2R's effectiveness, achieving leading performance in 20 environments while maintaining safety compliance in 35 out of 38 cases. These results demonstrate B2R's s ability to optimize rewards under strict constraints while ensuring safety.
simple yet effective framework for offline safe
A key consideration for B2R is its reliance on the availability of high-quality safe trajectories, a common challenge in offline safe RL. In environments where safe trajectories are exceedingly rare, B2R's 's performance may degrade due to the reduced size of the training set. To investigate this, we conducted few-shot experiments (detailed in Appendix B.4), which show that B2R exhibits graceful degradation profile and remains data-efficient due to its region- wide supervision. Nevertheless, for extreme cases of data scarcity, future work could explore integrating B2R with data augmentation or generative modeling techniques to synthesize diverse, safe trajectories.
While the non-adaptive CTG realignment strategy used in this work effectively validates our core paradigm, a promising future direction is to explore learned realignment strategies. An adaptive mechanism could, for example, allocate the safety margin non-uniformly to better handle high-risk states, offering more fine-gratne ine-grained control. We view this as a valuable extension of the B2R framework.
Additionally, while this work focuses on a single safety threshold for clarity and practical relevance, our framework is extensible to multi-target scenarios. We present a nooo-of--coneept Appendix B.5, demonstrating that single B2R agent can be trained to adhere to various constraint levels without retraining while maintaining comparable performance.
10

---

Acknowledgements
I wish to thank the anonymous reviewers for their constructive comments, which have greatly improved the quality of this paper. Also, many thanks YM. An, my life partner. She gently eased all the anxiety and pressure born from this work. Her presence what gave meaning the entire endeavor.
References
[1] Eitan Altman. Constrained markov decision processes with total cost criteria: Occupation measures and primal 1p. Mathematical Methods of Operations Research, 43:45 1996. URL https://api. semanticscholar Tarr//orpussDD. 1orrpssDD:67023455 16702345.
[2] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee. Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 11550044559777 2021.
[3] Qiguang Chen, Dengyun Peng, Jinhao Liu, HuiKang Su, Jiannan Guan, Libo Qin, and Wanxiang Che. Aware first, think less: Dynamic boundary aelf-awareness drives extreme reasoning efficiency in large language models. arXiv preprint arXiv:2508. 11822, 2025.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241123666, 2023.
[5] Wenhao Ding, Laixi Shi. Yuejie Chi, and Ding Zhao. Seeing is not believing: Robust reinforcement learning against spurious correlation. Advances in Neural Information Processing Systems, 36, 2024.
[6] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rys: What is essential for offline rl via supervised learning? arXiv preprint urXiv:2112.10073 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint 20Xiv:2004.07219, 2020.
[8] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without explo- ration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[9] Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480. 2015.
[10] Ze Gong, Akshat Kumar, and Pradeep Varakantham. Offline safe reinforcement learning using trajectory classification. arXiv preprint XXiv:24/2./5429, 2024.
[11] Sven Gronauer. Bullet-safety-gyne gym: framework for constrained reinforcement learning. 2022.
[12] Hao Jiang, Tien Mai, Pradeep Varakantham, and Minh Huy Hoang. Solving richly constrained rein- forcement learning through state augmentation and reward penalties. arXiv preprint arXiv:2301.11592. 2023.
[13] Prajwal Koirala, Zhanhong Jiang, Soumik Sarkar, and Cody Fleming. Fawac: Feasibility informed advantage weighted regression for persistent safety in offline reinforcement learning, 2024. URL https: //arxiv. .rrg abr/2112.24...... 088800
[14] Prajwal Koirala, Zhanhong Jiang, Soumik Sarkar, and Cody Fleming. Latent safety-constrained policy approach for safe offline reinforcement learning, 2024. URL https::/ //ar:v/aavvve//aa org/ /ttp//2222./////
[15] Sachin G Koman, Esmacil Seraj, and Matthew Gombolay, Contrastive decision transformers. In Conference on Robot Learning, pages 2159-2169. PMLR, 2023.
[16] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. via bootstrapping error reduction. Advances neural information processing systems, 32, 2019.
Stabilizing off-polic g-learning
[17] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 13:1119-1191, 2020.
[18] Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pages 6120 6130. PMLR. 2021.
11

---

[19] Jongmin Lee, Cosmin Paduraru. Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arXiv preprint NKKY:2204.089577 2022.
[20] Sergey Levine. Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0/66 2020.
[21] Jinning Li, Xinyi Liu, Banghua Zhu, Jiantao Jiao, Masayoshi Tomizuka, Chen Tang, and Wei Zhan. Guided online distillation: Promoting safe reinforcement learning by offline demonstration. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 7447-7454, 2024. doi: 10.1 111100//AA5114.... 202. 10611000221111111 1123.
[22] Quanyi Li, Zhenghao Peng, Lan Feng. Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 15(3):3461-3477 2022.
[23] Haohong Lin, Wenhao Ding. Zuxin Liu, Yaru Niu, Jiacheng Zhu, Yuming Niu, and Ding Zhao. Safety- aware causal representation for trustworthy offiine reinforcement learning in autonomous driving. IEEE Robotics and Automation Letters, 2024.
[24] Jinxin Liu, Ziqi Zhang, Zhenyu Wei, Zifeng Zhuang. Yachen Kang, Sibo Gai, and Donglin Wang. Beyond ood state actions: Supported cross-domain offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13945-13953, 2024.
[25] Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint xiv::23066008 09303, 2023.
[26] Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Con- strained decision transformer for offline safe reinforcement learning. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, pages 2161 1-21630. PMLR, 2023.
[27] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113-1132, PMLR, 2020.
[28] Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy. review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[29] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1900.00 1910.01788. 7(1):2 2019.
[30] Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojoiech Jaškowski, and Jürgen Schmidhuber. Training agents using upside-down reinforeement learning. arXiv preprint arXiv: 1912.02877, 2019.
[31] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
[32] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[33] Christopher JCH Watkins and Peter Dayan. Q-learning, Machine learning, 8:279-292, 1992.
[34] Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learnig for safe offline rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8753-8760, 2022.
[35] Yihang Yao, Zhepeng Cen, Wenhao Ding, Haohong Lin, Shiqi Liu, Tingnan Zhang, Wenhao Yu, and Ding Zhao. Oasis: Conditional distribution shaping for offline safe reinforcement learning. arXiv preprint 24t.:24007.16533. 2024.
[36] Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 4680-4688, 2022.
[37] Zhili Zhang, Songyang Han, Jiangwei Wang, and Fei Miao. tempoll--mpprlaaaaa aware safe multi ageti rein- forcement learning of connected autonomous vehicles in challenging scenarios. In 2023 IEEE International Conference on Robotics and Automation (ICRA). pages 5574-5580 IEEE, 2023.
12

---

[38] Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan. and Jingjing Liu. Safe offline reinforcement learning with feasibility-guided diffusion model. arXiv preprint urXiv:2401.10700 2024.
[39] Ziteng Zhuang, Kun Lei. Jinxin Liu, Donglin Wang, and Yilang Guo. Behavior proximal policy optimiza- tion. arXiv preprint arXiv:200.11331. 2023.
[40] Zifeng Zhuang. Dengyun Peng, Jinxin Liu, Ziqi Zhang, and Donglin Wang. Reinformer: Max-retum sequence modeling for offline arXiv preprint arxiv:2405.08777 2024.
13

---

Appendix
Table of Contents
Proofs
14
A.1
Proof of Theorem
14
A.2
Proof of Theorem 2
15
B
Supplementary experiments
16
B.1
CTG Realignment Strategies
16
B.2
Comparison with Recent Offline Safe RL Baselines
18
C Implementation details
21
C.I Inference Algorithm
21
C.2 Training Environment
21
C.3 Evaluation Metries
22
C.4 Baseline Methods
22
C.5 Hyperparameter Settings
22
C.6 Computing resources.
23
A Proofs
A.1 Proof of Theorem
We begin by establishing notation and core relationships. Let e; := Ct - è denote the instantancous
cost estimation error, and define the cumulative estimation error D. = Σi 0 &i. The filtration Fr captures all historical information up to time specifically F, == a(so, do, Co.... St, az Ci).
Step Cost Accounting via Telescoping Sums The CTG mechanism maintains the budget estimate C₂ through the recursion:
Cu+ = - -c, Co =
(12)
Telescoping this relationship over H steps reveals:
n-1
Σ c = Co - -C Cn
(13)
1=0
Consequently, the total realized cost C(T) = k Cn. with budget violation C(r) > equivalent to
ČH < O.
Step 2: Martingale Construction for Error Propagation We analyze the cumulative error D, through the martingale difference sequence:
M, == D, - F.D.F, Mo 0
(14)
The martingale property E[M[F[,1]] M,-1 follows immediately from the tower property of conditional expectation.
Step 3: Bounding Martingale Increments The difference sequence satisfies:
M M, < le + E[ee]|[Fl S 20max =: Ö
(15)
where the first inequality uses the triangle inequality and the second follows from our bounded cost assumption leel Cmax combined with Jensen's 's inequality.
14

---

Step 4: Concentration via Azuma-Hoeffding Inequality Applying the Azuma-Hoeffünn inequality the martingale {M, yields:
Pr [M 2 7 - exp
(16)
2H Č2
Setting = - -H (positive by assumption oH < E) gives the probability bound:
8= exp (- (5 (-18-o111
(-2 maax
(17)
Step 5: Error Control Under Good Event Define the favorable event & : {MH < oHl On this event:
Dn Mn B[(Du [11-1]
(18)
H-1 (E
oH) Σ Ele,/Fn
(19)
S
(20)
where 20 follows from our behavioral cloning assumption H[|e:|]F--1] o.
Step 6: Conservative Budget Planning By the training data constraint Cdata (T) S and
d-accurate cost estimation:
H-1
Σ
a 2
(21)
Step 7: Probabilistic Safety Guarantee Combining results event E:
C(T)
Σ à DH
(22)
k 2
+ K
(23)
Thus the probability of budget violation satisfies:
(8 (--o1))²
Pr(C(r) > x) S (
(-200 2HC max
(24)
Step 8: Expected Cost Analysis Finally, the unbiasedness of cost estimates E[ee] = 0 implies:
H-1
B[C(T)] = Σ Ela] +
E
(25)
t=
S の - (e HH)
(26)
completing the proof.
A.2 Proof of Theorem 2
Step Supervision Set Inclusion (Based on Definition 1) By Definition boundary-conditionall supervision uses trajectories in:
B. [ G, +a}
(27)
while region-conditional supervision (B2R) uses:
R₂(C,) {Te. C((T) < cc
(28)
15

---

After filtering, for the entire trajectory (f = 0), this implies:
hounddry {T C(T) E [ k]} S Tregion [T k}
(29)
Implication:
B2R retains all trajectories available to boundary supervision.
Step 2: Maximization Scope Let
boundury = arg max R(T)
(30)
TE Thunndry
By Assumption there exists r* Tregion with:
C(T') < and R((T)) max R(()) rEDat
(31)
Two cases arise:
If E T then
Ra (k) = R((T)) Rebundary (1s)
(32)
2. If E Tboundary then
Raae (K) = R((T) Febaundary (xs)
(33)
Step 3: Strict Improvement via Safe Region Case leverages Assumption 3: the existence of Thoundary ensures:
T* E Tregion
BBR (k)
R(T*)
pboundary (k)
(34)
This aligns with B2R's ability exploit strictly safer, higher-reward trajectories excluded by boundary supervision.
Step 4: Guaranteed Lower Bound Even if T* E Toounduyy set inclusion ensures:
RB2R (R))
phoundary (16)
(35)
B2R cannot perform worse includes all boundary-
trajectories.
Remark. To simplify the analysis, we do not consider trajectory stitching. As result, B2R conservatively learns only from full trajectories that satisfy the safety threshold C(T) S K, ensuring that supervision remains within the feasible region throughout training.
Supplementary experiments
B.1 CTG Realignment Strategies
While B2R aligns CTG tokens to a fixed deployment- budget, the choice realignment strategy remains flexible. Our main results a simple Shift method that applies a uniform offset each trajectory's 's CTG sequence. However, alternative designs-based on assumptions such as uniformity, proportionality, stochasticity-may influence reward optimization and constraint satisfaction differently.
To investigate this, we evaluate four representative strategies, each reflecting distinct inductive bias in how cost should be redistributed. All methods enforce C/ K, but vary in how they shape the remaining cost sequence. Below, we summarize their design principles:
Shift (default). The Shift strategy adds a constant offset A -C(() all cost-to-go tokens along the trajectory:
C- C.
(36)
This preserves the original temporal profile of cost decay, maintaining consistency with the behavior's natural cost progression. It assumes that preserving cost shape helpful for learning, and only alignment at the start token is necessary.
16

---

Avg. The Avg strategy evenly distributes the total offset A = C(() across all steps of the trajectory, modifying per-step costs as:
A 4 c H²
for all t .,..., H
(37)
where is the trajectory length. The updated cost-to-go sequence is then recomputed as:
C = Σ 4
(38)
This approach enforces uniform per-step adjustment, flattening cost variation across time. It assumes that smoothing cost signals may stabilize learning, especially in environments with noisy sparse cost feedback.
Rand. Rand randomly reallocates the excess budget across eligible timesteps. In discrete environ- ments (e.g.. SafetyGym), this involves flipping randomly chosen C₁ = 0 steps to c = until A is exhausted. In continuous environments (e.g.. MetaDrive), sample ca k/ H and update:
c/
A+
H -c)
a.)
(39)
The CTG then recomputed as:
C Σed c
(40)
k=d
This method introduces stochasticity, testing the model's robustness to non-smooth cost supervision.
Scale. The Scale strategy rescales the entire CTG sequence by multiplicative factor k/C(T):
C = a Ć..
(41)
This preserves the relative shape of the cost curve while adjusting its magnitude. assumes that cost proportionality, rather than absolute values, is the key for generalization under fixed constraint.
Each strategy ensures Cj but their internal structure introduces distinct biases that may affect policy behavior. The following sections compare their empirical effects.
AntRun R
BallCirele -R
CarCirele R
AntRun C
BallCirele C
CarCirele C
Shift
Aug
Rand
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.0
0.5
1.0
CarGoal2
AntVelocity R
VolpotVelocity R
CarGoal2 -
AntVelocity C
HogperVVloeity
Shift
Avg
Rand
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
mediumsppreee
hardsparse
easymenn
C
mediumspancse C
hardsparse- C
Shift
Avg
Rand
0.0
0.5
1.00
0.0
0.5
1.0
0.0
0.5
0.0
0.5
0.00
0.25
0.50
Figure 8: Reward and cost comparison of four CTG realignment strategies across nine representative tasks. Shift achieves the best overall trade-off.
17

---

Experimental Setup. We evaluate four CTG realignment strategies described above under the B2R, keeping all other components fixed. Experiments are conducted on representative tasks from the DSRL benchmark, selected to cover diverse dynamics and constraint regimes. For each strategy, we report normalized reward, normalized cost, and constraint violation rate, averaged over three random seeds and all constraint levels. This setup allows us to isolate the impact of CTG structure on policy behavior.
Results and Analysis. Figure 8 report the reward and cost performance of the four realignment strategies across nine representative tasks.
Overall, the Shift strategy achieves the most balanced trade-off: consistently maintains low cumulative cost while achieving near-optimal reward. This supports the hypothesis that preserving the original cost structure, while globally aligning to the target budget, is sufficient for effective constraint-aware training.
The Scale strategy occasionally yields higher reward (e.g., AntVelocity), but at the expense of increased cost and more frequent constraint violations, particularly in sparse-cost environments such as HopperVelocity: This suggests that proportional rescaling can overstretch cost dynamics and lead to over-optimism in high-return regions.
Avg performs reasonably but tends underperform tasks with strongly structured or sparse cost signals (e.g., CarCircle, easymean), likely due to over-smoothing the CTG sequence and erasing important temporal variations.
Rand performs surprisingly well across many tasks, maintaining competitive reward and low cost. This suggests that although the reallocation is stochastic, it still respects the underlying cost structure of the environment. As a result, it preserves budget feasibility while introducing minimal structural distortion, making it more robust than Avg in sparse high-variance environments.
While the four CTG realignment strategies differ in their structural assumptions and empirical performance, all of them outperform the bascline model without realignment. This confirms that the B2R framework itself is robust to variations in cost token design, and that the realignment mechanism-reguun
of its specific form-i essential for effective constraint- aware training. These results validate B2R as general paradigm for aligning training-tine supervision with deployment- time constraints. Moreover, the observed differences across strategies suggest that environment- specific design choices can further enhance performance, without undermining the overall value of the framework.
B.2 Comparison with Recent Offline Safe RL Baselines
To assess the broader applicability of B2R, we compare it against several recent offline safe RL. methods, including FAWAC [38], OASIS [35], LSPC [14], and FISOR [38]. These baselines leverage generative modeling or latent constraint inference to improve safety in offline settings.
Table 2: Extended comparison between B2R and recent safe offline RL methods: FISOR, OASIS,
LSPC, and FAWAC. Metrics report normalized reward (f) and cost (L), averaged over 3 constraint levels, 20 episodes, and 3 seeds. Bold indicates safe agents (cost < 1); Blue highlights safe agents achieving the highest reward. B2R consistently achieves strong reward while maintaining low cost across diverse tasks.
FISOR
OASIS
LSPC
FAWAC
(2R(s
Task
reward
cost.
reward
cost.
reward
cost
reward
cost
reward
cost.
BallRun
0.17
0.04
0.28
0.79
0.14
0.00
0.24
0.31
0.23
CarRun
0.85
0.15
0.02
0.97
0.97
0.96
0.08
DroneRun
0.44
2.52
0.13
0.79
0.57
0.00
0.57
0.12
0.56
0.03
BallCirele
0.28
0.00
0.70
0.45
0.47
0.01
0.61
0.89
0.67
0.59
CarCircle
0.24
0.15
0.76
0.89
0.72
0.42
0.71
0.68
DroneCircle
0.49
0.02
0.25
0.58
0.60
0.57
0.79
0.57
0.34
Average
0.41
0.48
0.55
0.53
0.57
0.13
0.56
0.46
0.63
0.33
As shown Table 2, B2R achieves the highest average reward while maintaining competitive or lower cost across diverse tasks. While some methods (c.g., OASIS, LSPC) perform well in specific
18

---

environments, they often suffer from higher cost or limited generalization across constraint levels. In contrast, B2R delivers consistently strong performance without requiring task-specific tuning or architectural modifications. This highlights its robustness and effectiveness general-purpose framework, Method details are provided Section 2.2.
B.3 Comparison with Buffered CDT Baseline
To verify that the superiority of B2R is not merely margin effect, we conducted a comparative analysis against a natural baseline: the CDT trained with a stricter buffered cost threshold. This experiment directly evaluates whether simply adjusting the target CTG is sufficient for achieving robust safety, if B2R's cost realignment mechanism offers more fundamental advantage. We benchmarked B2R against CDT variants trained with target CTGs set to 70%, 80%, 90%, and 100% of the true constraint threshold K.
The results in table 3 reveal the limitations of simple margin-based baseline. Such an approach often creates unfavorable trade-off between safety and performance and can be unreliable in complex tasks where no simple buffer value guarantees safety. We identify the root cause the supervision paradigm: a buffered baseline is still constrained by sparse boundary supervision, learning only from trajectories near specific cost target. In contrast, B2R 's region supervision leverages the entire safe dataset create denser learning signal. This confirms that B2R's 's advantage is not margin effect but stems from a more robust and data-efficient learning paradigm.
Table 3: Comparison of B2R against CDT with buffered cost thresholds. The CDT variants (CDT-X%) trained with a target CTG set to X% of the true constraint threshold k. Metries report normalized R (f) and C ()), averaged over 3 constraint levels, 20 episodes, and 3 seeds. Bold indicates safe agents (cost < 1); Blue highlights safe agents achieving the highest reward.
CDT-70 %
CDT-80% %
CDT-90% %
CDT-100%
B2R
Task
R
C1
Rt
C1
R
C1
R1
C4
R1
CJ
AntVelocyty
0.95
0.40
0.99
0.47
0.99
0.53
0.99
0.55
0.99
0.42
CarCircle
0.71
0.61
0.71
0.70
0.72
0.76
0.72
0.90
0.71
0.68
mediummean
0.86
1.85
0.87
1.97
0.82
1.86
0.81
1.92
0.88
0.63
harddense
0.19
0.40
0.20
0.40
0.24
0.55
0.25
0.65
0.48
0.58
B.4 Performance under Safe Data Scarcity
To quantify the robustness of B2R to the sparsity of safe data, we conducted a data ablation study, We retrained B2R on subsets of the filtered safe dataset, sampled at 5%, 20%, 50%, and 100% the originally available safe trajectories. The results on four representative tasks are shown in Table 4.
Table 4: B2R performance under data scarcity, with cost threshold k = 10. Metrics report normalized (1) and (L), averaged over 20 episodes, and 3 seeds. Bold indicates safe agents (cost < 1); Blue highlights safe agents achieving the highest reward. The policy's performance degrades gracefully, showing resilience even with only 20% of the safe data in many tasks.
5%
20%
50%
100%
Task
Rt
C1
R1
C1
R1
CL
R1
C|
BallCirele
0.59
2.50
0.61
0.88
0.64
0.80
0.67
0.59
DroneCircle
0.37
2.37
0.44
0.78
0.53
0.26
0.57
0.34
easysparse
0.60
0.27
0.66
0.63
0.59
0.00
0.63
0.26
mediummean
0.35 0.33
0.43
2.57
0.42
0.09
0.75
0.06
The results demonstrate B2R's robustness. Even with only 20% of the data, B2R maintains safe policies in BallCircle and DroneCircle. This resilience stems from our region-wide supervision, which creates denser and more robust learning signal than boundary-focused methods, making our approach less sensitive to data scarcity.
19

---

B.5 Extension to Multiple Constraint Targets
The main body of our work focuses on a single, fixed safety threshold, as this setting mirrors many practical safety-critical applications where reliability under a specific constraint paramount. However, the B2R framework is flexible and can be extended to handle multiple cost targets within single model, avoiding the need for retraining for each new constraint.
The methodology for this multi-target extension is as follows:
1. Multi-Target Data Preparation: We define a set of target cost thresholds K (Ki, J. For each hi K, we filter the original offline dataset obtain safe subset k
2. Per-Target CTG Realignment: For every dataset k we apply our CTG-realigmment procedure using its specific threshold k, This yields a set of distinct realigned datasets.
3. Unified Conditional Training:
We merge all realigned datasets into a single mixed set and train one conditional policy. At inference time, the desired cost threshold k; is provided as an initial condition to the model.
We conducted an exploratory experiment to validate this approach. Table compares the performance of a single B2R agent trained a mixed dataset for thresholds {10, 20, 40} against specialized agents trained for each single threshold.
The results show that single multi-targe B2R agent can achieve comparable performance to specialized agents across various thresholds. This confirms the promise of this extension, though deeper investigation into the performance trade-offs remains topic for future work.
Table Performance comparison between a single multi-target B2R agent and specialized single- target agents. The multi-target agent is trained once on mixed dataset and evaluated at different target thresholds (k)) Metrics report normalized R (t) and C (L), averaged over 20 episodes, and seeds. Bold indicates safe agents (cost < 1); Blue highlights safe agents achieving the highest reward. The comparable performance demonstrates the feasibility of extending B2R to handle multiple constraints without retraining.
Multi- Multe-Target
Single- Target
Task
K
Rt
CJ
R
C|
10
0.70
0.48
0.63
0.26
easysparse
20
0.81
1.12
0.80
1.08
40
0.86
0.70
0.89
0.81
10
0.68
0.90
0.65
0.81
BallCirele
20
0.71
0.76
0.70
0.79
40
0.73
0.39
0.65
0.16
10
1.00
0.73
0.98
0.45
AntVelocity
20
1.00
0.53
0.99
0.46
40
0.99
0.32
1.00
0.34
B.6 Comparison with FISOR under Stringent Cost Limits
To specifically evaluate B2R's performance in scenarios with sparse safe data resulting from stringent cost constraints (a "FISOR-sty style setup"), we conducted head-to-head comparison with FISOR. Following the protocol from the original FISOR paper, we used tight cost thresholds of = 10 for Syfety-Gymmasium tasks and k 5 for others, which are stricter than main experiments.
The results Table show that B2R not only remains safe under these stringent conditions but also consistently achieves significantly higher rewards. This suggests that while FISOR's 's hard-constraint approach can be overly conservative, B2R's region-wide supervision allows learn a more effective policy that better balances safety and performance, even when safe data is sparse.
20

---

Table Comparison B2R and FISOR under stringent, low cost-limits. Bold indicates safe agents
(cost < 1): Blue highlights safe agents achieving the highest reward. B2R consistently achieves
higher rewards while maintaining robust safety, demonstrating a better perfyypperfrrmacce balance than the more conservative FISOR approach in these settings.
B2R
FISOR
Task
R1
C1
R1
C₁
BallCircle
0.60
0.98
0.34
0.00
DroneCircle
0.53
0.58
0.48
0.00
easysparse
0.60
0.00
0.38
0.53
medmean
0.74
0.08
0.39
0.08
harddense
0.42
0.14
0.30
0.34
Ant Velociiy
0.97
0.99
0.89
0.00
C Implementation Details
C.1 Inference Algorithm
The following Algorithm provides the pseudocode for the specific implementation of the B2R framework during the inference phase. In this phase, the trained policy is used to make decisions based on the input environment states, while ensuring that safery constraints are respected throughout the process. The goal of this inference phase is generate sequence of actions by generating from the trained policy, leveraging both reward and cost information accumulated from past decisions. The process follows a step-by-step approach:
Algorithm 2 Inference with Boundary-to- Framework
Require: Trained policy ne, constraint threshold K, target return Rg, initial state 8o Ensure: Generated trajectory
{(*,, de, Fr, a))
Initialize: RS f- Ro, Co +- k, 80 + env. env.reset(). (), aco +- 0
2: for / = 0 to H do
3: Construct tokenized context:
0: = [Re-Kas Kits Cr- 8t- Qt-
4: Predict action: a, | oz)
5: Execute in environment: st+1, ct done +- env.step(a)
6: Update cost and return: Ret1 R. Te: Ctp1 t C2 Ct
7: if done then
8: break
9: end if
end for
11:
return = {(5t, ot, rt, ct) tang
C.2 Training Environment
Our experiments utilize the DSRL benchmark [25], which consists of 38 diverse sequential decision- making tasks covering robotic control, navigation, and autonomous driving. These tasks are specifi- cally chosen to challenge agents in a variety of realistic, safety-critica environments. They are drawn from several well-established benchmarks that aim to push the limits of reinforcement learning in both safety and performance. These benchmarks include:
Gymnasymmmasium [29]: A suite of Mujoco-based environments designed for safe reinforcement learning, featuring diverse tasks (Goal, Button, Push, Circle) with adjustable difficulty levels and various safety constraints.
21

---

BulletSafetyGym [11]: Built on the PyBullet physies engine, this benchmark extends SafetyGymna- sium with additional agents (Ball, Car, Drone. Ant) and shorter episode horizons.
MetaDrive [22]: A self-driving simulator based on the Panda3D game engine, providing complex road conditions and dynamic traffic interactions to evaluate safe RL in realistic driving scenarios.
C.3 Evaluation Metries
We evaluate the performance of B2R using the metrics from the DRSL to ensure fair comparisons across different tasks. For tasks within MetaDrive and BulletSafetyGym, we set the cost thresholds to 10, 20, and 40, ensuring balanced trade-off between performance and constraint adherence. In contrast, for environments in and 80, to account for the different cost dynamics and safety requirements in these settings.
the cost thresholds are higher, specifically 20, 40,
Normalized Reward
The normalized reward measures policy performance and is computed as:
R minn(MM
Roomnalized
x 100,
(42)
Tmax (M) (mi
where R is the evaluated reward return, and max (M) and (M) are the empirical maximum and minimum rewards for task M.
Normalized Cost
The normalized cost is defined as:
Cn +
Cnormalized
(43)
K +
where C is the evaluated cost return, K is the target threshold, and e is a small positive number ensuring numerical stability.
C.4 Baseline Methods
To ensure a comprehensive evaluation, we compare our approach against several state-of-the-ee art offline safe reinforcement learning algorithms. These baselines encompass range of methodologies, including behavior cloning, sequential modeling, distribution correction, and Lagrangian based ap- proaches. Some results are directly obtained from the DSRL benchmark, while others are reproduced using their official implementations.
BC-AIl / BC-Safe: Behavior Cloning (BC) serves fundamental baseline by training policies to mimic expert demonstrations. BC-All utilizes the entire dataset, whereas BC-Safe exclusively uses safe trajectories to ensure policy compliance with constraints.
CDT (Conditional Decision Transformer) [26]: A Decision Transformer based approach that incorporates safety constraints by conditioning cost-related tokens, allowing the model to learn safe policies an autoregressive manner.
BCQ-Lag [8]: A Lagrangian- extension of BCQ that penalizes unsafe actions while optimizing for reward. The Lagrangian multiplier dynamically adjusts to enforce safety constraints.
CPQ (Constraint-Penalizee Q-learning) [34]: Treats out-of- distriibbiiiino actions as unsafe and penalizes them by modifying the Q-value function, preventing policy optimization on unsafe state-action pairs.
COptiDICE [19]: A distribution correction estimation (DICE)-based offline safe RL method that extends OptiDICE [18], explicitly enforcing cost constraints while optimizing policy performance.
TraC Semnjectory-Consrraie RL) [10]: A trajectory-based approach that incorporates cost-awareness into safe RL by guiding the policy learning process with explicit constraints.
C.5 Hyperparameter Settings
All models are trained for 20 epochs, each consisting of 5000 gradient steps, totaling 100,000 training steps. To evaluate robustness, we use three random seeds for all experiments. Environment- specific cost thresholds are listed in Table 7, along with other key hyperparameters.
22

---

Table Hyperparameter settings for B2R experiments.
Category
Hyperparameter
Value
Optimizer Type
Lamb
Learning Rate
0.0001
Optimizer
Batch Size
2048
Gradient Clipping
0.25
Steps per Epoch
5000
Total Epochs
20
Training Strategy
Early Stopping
Reward stagnation
Hidden Dimension
128
Attention Heads
Transformer Model
Transformer Layers
3
Dropout
0.1
Context Length (MetaDrive)
3
Environment-Speccftc Settings
Context Length (Others)
10
C.6 Computing resources.
The experiments are conducted on a Linux-based server equipped with an Intel Core 19- 19-149 22-Core Processor, one NVIDIA GeForce RTX 4070 GPU, and 64 GB of RAM. The implementation is based on PyTorrch (v1.13.1) with CUDA 12.4. Training is performed for 100K steps, with gradient updates running on a single GPU. The total training time per task averages approximately 2 hours, which is comparable to CDT, another braneee method. Evaluation is conducted separately on CPU/GPU to ensure efficiency.
23

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As shown in Abstract and Introduction, we present main claims and outline the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

We present simplified theoretical framework in Section 4.3, while the complete proofs are provided in Appendix A.I.
Justification:
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide a detailed description of our algorithm in Algorithm 1, and elaborate on the experimental settings in Table 7.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
the contribution primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
If the contribution is either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
model (e.g., g., large language model), then there should
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
25

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have uploaded the complete code and documentation for the main experi- ments in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
wwwsssssslllllioooe onPolicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / wttpp.ccppppplipppppp wwtpccccppcccceececcccccccccttlt cooollc/guidess
seabbmmmsssooPPe ssiiinolicyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide a detailed description of our algorithm in Algorithm 1, and elaborate on the experimental settings in Table 7.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We tested and showed the mean and std results with three different random seeds across almost all experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed specifications of our computational resource usage in Appendix C.6.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tprrip///uuiss.... hptpulic//thissuiddeeeeopopeppee
Answer: [Yes]
Justification:
We convinced that comply with NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This research carries minimal risk of being exploited for detrimental societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
27

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This research carries minimal risk of being exploited for detrimental societal impacts.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly cited all relevant source materials and obtained necessary permissions for any third-party resources used in this study.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
28

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode com conddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We have included comprehensive documentation with the submitted code repository.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This study does involve any human subject research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This study does not involve any human subject research.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
This study is entirely unrelated to prompt engineering.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy fttta://neuripa. cc/ wwt.c//oeeeeeneee/222222222 for what should or should not be described.
30


Reviews:
[{"review_id": "UfyERsxZZX", "reviewer": "Reviewer_yMKb", "summary": "This paper proposes Boundary-to-Region (B2R) Supervision, a framework for offline safe RL using transformer-based sequence models. The key insight is that reward-to-go and cost-to-go should not be treated symmetrically as in CDT, especially when cost-to-go (CTG) corresponds to rigid safety constraints. B2R first filters unsafe trajectories from the offline dataset, then realigns the cost-to-go of all remaining (safe) trajectories to match the safety threshold, enriching the training data for the sequence model. Empirical results show that it achieves better reward while satisfying the cost constraint in most tasks.", "questions": "1. Did the author assess “buffered threshold” baseline (CDT trained with a stricter threshold)?\n\n2. How does B2R handle cases where safe trajectories are rare? What is the minimum “safe coverage” required?\n\n3. How does RoPE compare to standard positional embeddings in CDT or Decision Transformer? Any reason to prefer RoPE?\n\n4. Can the authors clarify exactly how region-conditional supervision is used during training and inference? Is it purely boundary-conditioned after cost-to-go shifting?", "limitations": "The paper pointed out the limitation of the method in environment with limited safe data in the conclusion.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n* Comprehensive empirical study: 38 tasks from the DSRL benchmark are evaluated, with strong results on both safety and reward metrics. Ablations of key components are included.  \n\n* Theoretical analysis: Provide probabilistic guarantees on safety and reward performance of B2R.\n\n* Simple, practical method: Trajectory filtering and cost realignment are easy to implement and does not introduce more hyperparameters. The approach can be integrated into standard Decision Transformer-style methods.\n\n### Weaknesses\n\n1. **Obvious baseline not included**: The policy oscillation around the boundary in Fig2 is to be expected if the model is trained only on near-boundary trajectories. A natural baseline --- CDT trained with a buffer (i.e., a margin below the actual constraint) --- should be included (CDT with a margin), to directly test if realignment is strictly necessary or just a margin effect.\n\n2. **Unclear use of region-conditional supervision**: The role of region-conditional cost supervision (R_t) is not clearly explained. After cost-to-go shifting, is the model still conditioned on a range, or only on the boundary? Is region conditioning used at training or inference, or both?  \n\n3. **RoPE rationale and ablation missing**: Rotary position embeddings are used but not discussed in detail. The authors should explain why RoPE helps in this context, and provide comparisons with standard positional embeddings as in Decision Transformer / RvS.  \n\n4. **Assumption of sufficient safe data after trajectory filtering**: B2R filters out unsafe trajectories entirely. In domains with limited safe data, this may significantly reduce coverage or reward diversity.\n\n5. **Incremental novelty**: The incremental tweaks --- CTG realignment, trajectory filtering, RoPE --- seems a little bit arbitrary and heuristic. The authors might want to highlight the novel contribution of this paper.\n\n6. **Related work lacks differentiation**: The Related Work section lists prior approaches but does not explicitly position or differentiate B2R from them.  A concise positioning paragraph would help readers and reviewers assess novelty.\n\n7. **Typo in main text**: Line 197 refers to Equation 4.3, which I think should be Equation 9.", "was_revised": true, "final_justification": "All major concerns are addressed and authors have provided additional result to strengthen their claims. Final review rating raised to 5.", "rebuttal": "We sincerely thank reviewer yMKb for your valuable comments and recognition of our work. As per your suggestions, we have responded to each point below and revised the manuscript accordingly.\n>**W1:Obvious baseline not included.** The policy oscillation around the boundary in Fig2 is to be expected if the model is trained only on near-boundary trajectories. A natural baseline --- CDT trained with a buffer (i.e., a margin below the actual constraint) --- should be included (CDT with a margin), to directly test if realignment is strictly necessary or just a margin effect.\n\n>**Q1:** Did the author assess “buffered threshold” baseline (CDT trained with a stricter threshold)?\n\nWe thank the reviewer for the invaluable suggestion to benchmark against **CDT with a buffered threshold**. We have conducted the proposed experiment comparing B2R against CDT trained with buffered thresholds (i.e., target CTGs of 70%, 80%, 90%, and 100% of the constraint κ). The results not only highlight B2R's superiority but also expose the fundamental limitations of the buffered baseline approach.\n\nThe experimental results are as follows:\n\n|Task|Met|70%|80%|90%|100%|B2R|\n|----|---|--|--|--|---|---|\n|AntVelocity|R|0.95|0.99|0.99|0.99|**0.99**|\n||C|0.40|0.47|0.53|0.55|**0.42**|\n|CarCircle|R|0.71|0.71|**0.72**|0.72|0.71|\n||C|0.61|0.70|**0.76**|0.90|0.68|\n|mediummean|R|0.86|0.87|0.82|0.81|**0.88**|\n||C|1.85|1.97|1.86|1.92|**0.63**|\n|harddense|R|0.19|0.20|0.24|0.25|**0.48**|\n||C|0.40|0.40|0.55|0.65|**0.58**|\n\nOur key findings are as follows:\n\n- **CDT faces a direct safety-performance conflict**: As the buffered threshold tightens (from 70% to 100% of the constraint value κ), cost decreases but reward is sacrificed, revealing a typical yet unfavorable trade-off.\n\n- **Margin adjustment is unstable and unreliable**: In the medmean task, CDT's margin-cost relationship becomes unstable, and no margin setting satisfies safety constraints, while B2R remains safe (0.63) with higher reward. This shows that for complex tasks, simply tuning the target cost-to-go is a fragile and unreliable solution.\n\nThe root cause lies in the learning signal: The \"buffered CDT\" approach is limited by sparse \"boundary supervision\"—it only mimics trajectories whose cumulative costs are close to a specific target CTG during inference. In contrast, B2R enables \"region supervision\" through cost realignment, leveraging all safe trajectories. This yields denser and more diverse training signals, allowing B2R to learn more robust policies and break the inherent limitations of baseline methods.\n\nThese results confirm that B2R's core contribution lies in its unique supervision paradigm. We will include this important comparison in the appendix.\n\n> **W2: Unclear use of region-conditional supervision.** The role of region-conditional cost supervision (R_t) is not clearly explained. After cost-to-go shifting, is the model still conditioned on a range, or only on the boundary? Is region conditioning used at training or inference, or both?\n\n>**Q4:** Can the authors clarify exactly how region-conditional supervision is used during training and inference? Is it purely boundary-conditioned after cost-to-go shifting?\n\nWe thank the reviewer for this important question, which allows us to clarify the central mechanism of our work.\n\nRegion-conditional supervision refers to the training strategy in which the model is supervised by every trajectory whose cost-to-go satisfies the safety constraint, i.e., $C_0 \\le \\kappa$. This collection of trajectories forms the “safe region” $\\mathcal{R}_t$. Unlike conventional approaches that rely only on samples within a narrow margin around the boundary, $C_0\\in[\\kappa-\\varepsilon,\\kappa+\\varepsilon]$, **region-conditional supervision exploits a broader and more diverse set of feasible behaviors, thereby improving both training stability and generalization.**\n\nTo achieve this, we preprocess the data with CTG Realignment: after discarding unsafe trajectories whose cost exceeds the threshold $\\kappa$, we apply a constant offset to the CTG sequence of every remaining safe trajectory so that all initial CTGs in the “safe region” are aligned to the constraint boundary $\\kappa$. This ensures that **while the CTG input is boundary-conditioned to a fixed $\\kappa$, the supervision signal is drawn from the entire safe region.**\n\n**This design unifies training and inference.** At inference, we provide the same initial condition $\\kappa$. Because the model has learned to associate this boundary condition with the wide spectrum of safe-region behaviors, **it generates stable and reliable actions that satisfy the constraint, rather than merely imitating a few boundary trajectories.**\n\n> **W3:RoPE rationale and ablation missing.** Rotary position embeddings are used but not discussed in detail. The authors should explain why RoPE helps in this context, and provide comparisons with standard positional embeddings as in Decision Transformer / RvS.\n\n>**Q3:** How does RoPE compare to standard positional embeddings in CDT or Decision Transformer? Any reason to prefer RoPE?\n\nThank you for pointing this out; we apologize for the lack of clarity in the original manuscript, which led to an important misunderstanding: The “W/O RoPE’’ ablation Section 5.2 and Figure 6 is the requested comparison against the standard absolute positional encoding (APE), and it empirically shows RoPE's superiority. The results clearly show that the RoPE-equipped B2R variant significantly outperforms the APE variant in cost control, empirically demonstrating the superiority of RoPE within our framework.\n\nThe choice of RoPE is theoretically grounded: B2R performs **additive CTG realignment**, which demands that the model capture the fine-grained, step-by-step evolution of costs. RoPE’s relative positional encoding **offers a strong inductive bias for learning these local, step-level dynamics**, enabling the model to better understand the relative causal relationships between actions and cost changes. By contrast, APE provides only absolute positional information and is less suited to modeling such relative dynamics.\n\nWe will update the label and add a more detailed theoretical explanation to the paper.\n\n>**W4:Assumption of sufficient safe data after trajectory filtering.** B2R filters out unsafe trajectories entirely. In domains with limited safe data, this may significantly reduce coverage or reward diversity.\n\n>**Q2:** How does B2R handle cases where safe trajectories are rare? What is the minimum “safe coverage” required?\n\nWe thank the reviewer for this critical question regarding performance with sparse safe data. This is a crucial consideration for any practical offline safe RL method, and we agree it represents a fundamental challenge for the field. \n\nTo address this, we conducted a new few-shot experiment. Due to character limits, we respectfully refer to our detailed response to Reviewer LsnC (W1) for the full results. The results show that while performance is naturally affected by less data, B2R exhibits remarkable stability and a graceful degradation profile. Our region-wide supervision is designed to be highly data-efficient, as it extracts a learning signal from every single available safe trajectory. \n\n**Regarding a \"minimum safe coverage,\" this experiment shows it is highly task-dependent.** For extreme cases, we believe future work could further enhance B2R's performance by integrating data augmentation or generative modeling techniques to create more diverse safe samples.\n\n>**W5:The incremental tweaks.** --- CTG realignment, trajectory filtering, RoPE --- seems a little bit arbitrary and heuristic. The authors might want to highlight the novel contribution of this paper.\n\nWe agree that when viewed in isolation, trajectory filtering, CTG realignment, and RoPE may appear as incremental tweaks. However, we wish to emphasize that their combination is not an ad-hoc assembly but a deliberately engineered, interlocking whole designed to instantiate a fundamentally new paradigm.\n\nThe true novelty of our work lies in identifying the symmetry fallacy inherent in prior methods—the failure to recognize the asymmetry between the flexibility of rewards and the rigidity of costs, which hinders effective learning of the safety boundary.\n\nTo resolve this foundational issue, we introduce a region-level supervision paradigm that conditions the policy on the cost-to-go distribution of the entire safe region rather than on sparse boundary samples. Implementing this paradigm demands a coherent pipeline:\n\n1. **Trajectory filtering** is the prerequisite that **defines the safe region**. Without this step, the concept of learning from the whole region is moot.  \n2. **CTG realignment** is the **core mechanism** that converts heterogeneous safe trajectories into a **uniform, dense supervision signal** anchored at the single safety boundary $\\kappa$.  \n3. **RoPE** is the **synergistic component** that preserves the temporal fidelity of this refined signal. As detailed in our responses to W3 & Q3, RoPE provides the inductive bias required to model the incremental cost dynamics introduced by realignment.\n\nTogether, these three components form a **logically coherent and mutually reinforcing system** purpose-built from the ground up to realize our central idea. Empirical results robustly validate the efficacy of this system in achieving the proposed paradigm.\n\n>**W6:Related work lacks differentiation.** The Related Work section lists prior approaches but does not explicitly position or differentiate B2R from them. A concise positioning paragraph would help readers and reviewers assess novelty.\n\nWe agree B2R's positioning needed strengthening. Per your suggestion, we will add a positioning paragraph in revised paper to explicitly differentiate B2R from key methods like CDT and TraC to highlight its novelty.\n\n>**W7:Typo in Line 197.**\n\nWe fixed this typo in our revised version and carefully checked the manuscript."}, {"review_id": "YRQ22wRHqR", "reviewer": "Reviewer_LsnC", "summary": "This paper introduces B2R (Boundary-to-Region), a framework for offline safe reinforcement learning that improves constraint satisfaction in sequence-model-based policies. The key innovation is treating cost-to-go (CTG) as a rigid safety boundary rather than a flexible optimization target, in contrast to return-to-go (RTG). The method involves:\n- Trajectory filtering to remove unsafe samples.\n- CTG realignment to align all safe trajectories with the deployment-time cost constraint.\n- Use of rotary positional embeddings to enhance temporal modeling. \n\nEmpirical results on 38 tasks show that B2R outperforms baselines, and the paper provides theoretical analysis to justify its safety guarantees.", "questions": "- Please check weaknesses.\n- How does B2R perform in datasets with sparse or skewed distributions of safe trajectories, such as under tight cost limits (e.g., FISOR-style setups)?\n- Could the CTG realignment strategy be made adaptive or learned, potentially conditioned on environment dynamics or trajectory characteristics?", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The asymmetry between RTG and CTG is a relevant.\n- Trajectory filtering and CTG realignment are simple yet effective interventions.\n- B2R achieves strong empirical results, outperforming or matching baselines in both reward and constraint satisfaction on 35 out of 38 tasks.\n- The method is evaluated through ablations on core components including CTG realignment, RoPE, and filtering.\n\nWeaknesses:\n- B2R depends on access to a sufficient number of safe trajectories and lacks stitching capabilities.\n- The CTG realignment strategy (“Shift”) is heuristic and not learned or adaptive.\n- The method does not generalize to multiple constraints without retraining, yet remains as computationally heavy as CDT.\n\nMinor:\n- Line 238: The paper states that “these approaches discard potentially useful trajectories once they exceed the limit,” but B2R also performs filtering based on the same cost threshold. Could the authors clarify how B2R’s filtering is fundamentally different or more selective?\n- Appendix C.3: It seems that the cost thresholds for SafetyGymnasium and BulletSafetyGym may have been inadvertently swapped.", "was_revised": true, "final_justification": "The authors have addressed the concerns I raised, and the additional empirical results are valuable. While the method is not highly novel, the empirical performance is strong and the findings are worth publishing. I am therefore maintaining my positive rating.", "rebuttal": "We sincerely thank you for your constructive feedback and recognizing the value of our work. Below, we clarify our method, discuss future extensions, and address your comments.\n\n>**W1:** B2R depends on access to a sufficient number of safe trajectories and lacks stitching capabilities.\n\nWe thank the reviewer for this insightful comment, which addresses two important characteristics of our method. We respond to both points below.\n- **On the dependency on sufficient safe trajectories:** \n\nWe agree with the reviewer that B2R's performance, like almost all offline safe RL methods, is predicated on the availability of safe trajectories. To quantitatively characterize B2R's robustness to data sparsity, we conducted a data ablation study. We retrained B2R on subsets of the filtered safe dataset, sampled at 5%, 20%, 50%, and 100%. The results on four representative tasks are shown below (with a cost threshold of $\\kappa$=10).\n\n|Task|Metric|5%|20%|50%|100%|\n|---|---|---|---|---|---|\n|**BallCircle**|**R**|0.59|0.61|0.64|**0.67**|\n||**C**|2.50|0.88|0.80|**0.59**|\n|**DroneCircle**|**R**|0.37|0.44|0.53|**0.57**|\n||**C**|2.37|0.78|0.26|**0.34**|\n|**easysparse**|**R**|0.60|**0.66**|0.59|0.63|\n||**C**|0.27|**0.63**|0.00|0.26|\n|**mediummean**|**R**|0.35|0.43|0.42|**0.75**|\n||**C**|0.33|2.57|0.09|**0.06**|\n\nThese results demonstrate B2R's robustness. Our key findings from this data ablation study are:\n\n**Graceful Degradation and Safety under Moderate Scarcity:** Performance degrades predictably, not catastrophically. With just 20% of the data, B2R maintains safe policies in BallCircle (Cost: 0.88) and DroneCircle (Cost: 0.78), both well under the cost threshold.\n\n**Remarkable Resilience at Extreme Scarcity:** At the stark 5% data limit, while B2R violates the safety threshold in tasks with the fewest safe trajectories like BallCircle (Cost: 2.50 from 4 trajs) and DroneCircle (Cost: 2.37 from 13 trajs), its overall resilience remains clear. Under the same scarcity, it still achieves a high reward of 0.60 in easysparse (from 18 trajs) with a low cost of 0.27, and remains both effective and safe in mediummean (Cost: 0.33 from 21 trajs).\n\nThis resilience stems directly from our region-wide supervision. Unlike boundary-supervision methods whose sparse signal can collapse when data is reduced, B2R’s signal is drawn from all safe trajectories. This creates an inherently denser and more robust learning signal, making our approach less sensitive to data scarcity.\n\n- **On the lack of stitching capabilities:**\n\nYou are correct that our method, like other models based on the Decision Transformer framework, does not use an explicit \"stitching\" mechanism. Our work addresses a different, but equally critical, problem: the sparse supervision signal in offline safe RL.\n\nB2R's core innovation is to create a dense, unified learning signal from all safe trajectories. Not only does this significantly boost performance on its own, but we also believe it provides an implicit generalization benefit. By unifying the cost budget and using RoPE to better model dynamics, B2R creates a more consistent learning signal that may empower the model to better combine behaviors from different trajectories, achieving some of the goals of stitching through a different mechanism.\n\n>**W2:** The CTG realignment strategy (“Shift”) is heuristic and not learned or adaptive.\n\n>**Q2:** Could the CTG realignment strategy be made adaptive or learned, potentially conditioned on environment dynamics or trajectory characteristics?\n\nWe thank the reviewer for this insightful question. Our use of a simple, non-adaptive \"Shift\" strategy was a deliberate choice for this initial work. Our primary goal was to cleanly isolate and validate the core concept of \"region-wide supervision\" without introducing confounding variables from a more complex, learnable module.\n\nWe agree that an adaptive or learned realignment strategy is a promising direction for future research. Such a mechanism, potentially conditioned on trajectory characteristics, could offer more fine-grained control over the cost-reward trade-off. For instance, a learned module could allocate the \"safety margin\" more intelligently by preserving it for high-risk regions of the state space, potentially unlocking further performance gains. This represents a valuable extension that can be built upon the framework we have established. We will incorporate this discussion into the Future Work section of our revised manuscript.\n\n> **W3:** The method does not generalize to multiple constraints without retraining, yet remains as computationally heavy as CDT.\n\nOur current focus on a **fixed-threshold** setting was deliberate, as it mirrors many safety-critical systems (e.g., autonomous driving speed limits, industrial control set-points) where **reliability under a single, strict constraint is paramount**. Our experiments show that for this key scenario, B2R’s specialized approach is demonstrably more effective and reliable than generalist models.\n\nHowever, the underlying principle of B2R is flexible and can be extended to handle **multiple cost targets** within a single model, thus avoiding the need for retraining for each new constraint. To demonstrate this, we propose a straightforward extension to our framework which allows a single agent to learn policies for various cost thresholds simultaneously. The methodology is as follows:\n\n**1.Multi-Target Data Preparation:** We abandon a single cost threshold $\\kappa$ and instead define a set $\\{\\kappa_1, \\kappa_2, \\dots, \\kappa_N\\}$.  For each $\\kappa_i$, we filter the original offline dataset to obtain $\\mathcal{D}_{\\text{safe},i}=${$\\tau \\in \\mathcal{D} \\mid C(\\tau)\\le\\kappa_i$}.\n\n**2.Per-Target CTG Realignment:** \nFor every $\\mathcal{D}_{\\text{safe},i}$ we apply our CTG-realignment procedure using its specific $\\kappa_i$. \nThis yields $N$ distinct realigned datasets {$\\mathcal{D}'_1, \\mathcal{D}'_2, \\dots, \\mathcal{D}'_N$}.\n\n**3.Unified Conditional Training:** We merge all realigned datasets into a single mixed set $\\mathcal{D}_{\\text{unified}} = \\mathcal{D}'_1 \\cup \\mathcal{D}'_2 \\cup \\dots \\cup \\mathcal{D}'_N.$\n\nThe experimental results of this exploratory method are shown in the table below, demonstrating the feasibility and promise of this multi-target learning:\n\n| Task | $\\kappa$ | multi(R) | multi(C) | single(R) | single(C) |\n|---|---|---|---|---|---|\n| **easysparse** | 10 | 0.70 | 0.48 | 0.63 | 0.26 |\n|  | 20 | 0.81 | 1.12 | 0.80 | 1.08 |\n|| 40 | 0.86 | 0.70 | 0.89 | 0.81 |\n| **BallCircle** | 10 | 0.68 | 0.90 | 0.65 | 0.81 |\n|  | 20 | 0.71 | 0.76 | 0.70 | 0.79 |\n| | 40 | 0.73 | 0.39 | 0.65 | 0.16 |\n| **AntVelocity** | 10 | 1.00 | 0.73 | 0.98 | 0.45 |\n| | 20 | 1.00 | 0.53 | 0.99 | 0.46 |\n|  | 40 | 0.99 | 0.32 | 1.00 | 0.34 |\n\nFor example, in easysparse, BallCircle, and AntVelocity tasks across different κ values, a single 'multi-target' B2R agent achieves comparable R and C performance compared to 'single-target' agents, though multi-target performance often shows slightly higher R and C values. We will discuss the feasibility of this approach in the appendix of the revised paper.\n\n> **W4:** Line 238: The paper states that “these approaches discard potentially useful trajectories once they exceed the limit,” but B2R also performs filtering based on the same cost threshold. Could the authors clarify how B2R’s filtering is fundamentally different or more selective?\n\nWe sincerely thank the reviewer for highlighting this confusing sentence. The reviewer is absolutely correct—our original phrasing was imprecise and did not accurately represent our method's mechanism, our intention was to compare how different methods utilize the information within the safe dataset. We apologize for this and will replace the sentence entirely in the revised manuscript.\n\nThe fundamental difference lies in the quality and utilization of the supervision signal that each method constructs from the safe trajectories: BC-Safe's signal is sparse and scattered, while TraC's is a coarse, binary classification signal based on partitioning data into \"desirable\" and \"undesirable\" sets. In contrast, B2R’s key innovation is to realign all safe trajectories to create a single, dense, and fine-grained supervision signal for the desired constraint.\n\n> **W5:** Appendix C.3: It seems that the cost thresholds for SafetyGymnasium and BulletSafetyGym may have been inadvertently swapped.\n\nWe sincerely thank you for your meticulous catch. The values were swapped due to a typo and have been corrected in the revised manuscript. We appreciate your help in improving our paper's accuracy.\n\n> **Q1:** How does B2R perform in datasets with sparse or skewed distributions of safe trajectories, such as under tight cost limits (e.g., FISOR-style setups)?\n\nWe understood the reviewer's question as a stress test of B2R's performance under \"FISOR-style setups\"—challenging scenarios with stringent, low cost-limits that result in a sparse distribution of available safe data.\n\nTo address this directly, we conducted a head-to-head comparison against FISOR using its own paper's strict threshold settings ($κ$ =10 for Safety-Gymnasium, $\\kappa$ =5 for others). The results are as follows:\n\n|Task|**B2R(R)**|**B2R(C)**|**FISOR(R)**|**FISOR(C)**|\n|---|---|---|---|---|\n|**BallCircle**|**0.60**|**0.98**|0.34|0.00|\n|**DroneCircle**|**0.53**|**0.58**|0.48|0.00|\n|**easysparse**|**0.60**|**0.00**|0.38|0.53|\n|**medmean**|**0.74**|**0.08**|0.39|0.08|\n|**harddense**|**0.42**|**0.14**|0.30|0.34|\n|**AntVelocity**|**0.97**|**0.99**|0.89|0.00|\n\nThe conclusion from this experiment is clear: B2R not only remains robustly safe under these stringent conditions but also consistently achieves significantly higher rewards. While FISOR's hard-constraint approach can be overly conservative, B2R learns a more effective policy that better balances safety and performance."}, {"review_id": "Wz0nIS7eJf", "reviewer": "Reviewer_EuNY", "summary": "This paper addresses a fundamental limitation in existing sequence-model-based offline safe reinforcement learning (RL) methods, such as those inspired by Decision Transformers (DT). These methods typically treat return-to-go (RTG) and cost-to-go (CTG) signals equivalently. The authors argue that this approach neglects the intrinsic difference between reward and cost: RTG is a flexible performance target to maximise, while CTG should represent a rigid safety budget or boundary to strictly adhere to. This oversight leads to unreliable constraint satisfaction, especially when policies encounter cost trajectories outside the training data distribution. To overcome this, the paper proposes Boundary-to-Region (B2R), a novel framework that enables asymmetric conditioning through cost signal realignment", "questions": "1. In the problems where the results provided are not safe, can the authors elaborate on why the solution does not work well? Is it because of tight constraints or complex cost functions or something else? It would help to provide a more detailed analysis. \n\n2. Transformer based approaches for RL are known to struggle in environments where there is transitional uncertainty.  Can authors clarify how they have addressed this issue?\n\n3. The ablation study on CTG realignment strategies is insightful, with \"Shift\" performing the best overall. However, the paper also mentions that \"environment-specific design choices can further enhance performance\". Could the authors provide more detailed guidance or propose a meta-strategy for adaptively selecting or tuning the CTG realignment method (e.g., Shift, Avg, Rand, Scale) based on dataset characteristics, environment properties, or observed real-time performance, rather than a fixed choice?\n\n4. The assumption of \"prediction-error bound\" seems to be hard to visualize. Can authors clarify where this assumption will be violated?\n\n5. How would B2R deal with situations where there are not many high quality safe trajectories available?\n\n6. How does B2R perform better than the TraC method intuitively?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Problem and issues articulated well\n2. Interesting and intuitive methodology\n3. Detailed experimental results and strong empirical performance.\n\nWeaknesses:\n\n1. There is a reliance on high-quality safe trajectories. \n2. While authors have provided comparison against transformer based approaches, intuitive issues with more existing approaches (e.g., TraC) have not been provided. \n3. The assumptions employed in theoretical results are unclear.", "was_revised": true, "final_justification": "Authors have answered all my questions very well and I do not have any concerns.", "rebuttal": "We sincerely thank you for your constructive feedback and for recognizing the value of our work. In our response below, we address each of your comments, offering clarifications on our method and discussing potential future extensions.\n\n> **W1:** There is a reliance on high-quality safe trajectories.\n\n> **Q5:** How would B2R deal with situations where there are not many high quality safe trajectories available?\n\nWe thank the reviewer for highlighting this important consideration regarding data dependency, a common challenge in offline safe RL. We address this point below by presenting new empirical evidence and discussing potential extensions for data-scarce scenarios.\n\nB2R is designed with data efficiency in mind, and our experiments demonstrate its robustness even under limited safe data. To directly evaluate this, we conducted a data ablation study by retraining B2R on subsets of the filtered safe dataset (sampled at 5%, 20%, 50%, and 100%). For a detailed visualization of these results across four representative tasks (with cost threshold κ=10), please refer to our response to Reviewer LsnC (W1), where these results are presented in full (due to space constraints here).\n\nKey insights:\n- With 20% data, B2R maintains safety (e.g., costs below 1.0 in BallCircle and DroneCircle) while achieving competitive rewards.\n- Even at 5% data, B2R shows resilience, delivering high rewards with reasonable costs in easysparse and mediummean.\n\nThis robustness arises from B2R's region-wide supervision, which extracts dense signals from all available safe trajectories, unlike sparse boundary supervision methods that falter with reduced data.\n\nFor scenarios with extremely scarce high-quality safe trajectories, we recommend integrating generative models synergistically with B2R. For instance,  one could train a conditional diffusion model on the available safe data to generate novel safe trajectories, densifying the safe region.\n\n> **W2:** While authors have provided comparison against transformer based approaches, intuitive issues with more existing approaches (e.g., TraC) have not been provided.\n\n> **Q6:** How does B2R perform better than the TraC method intuitively?\n\nB2R's advantage over TraC stems from its fundamentally different supervision mechanism.\n\n- **Supervision Signal:** TraC uses a coarse, binary classification signal, labeling entire trajectories as \"desirable\" or \"undesirable\". B2R employs a fine-grained, temporal regression signal. Through Cost-to-Go (CTG) Realignment, it provides a precise, step-by-step regression target for managing the safety budget at every point in time.\n\n- **Underlying Framework:** TraC's Behavior Cloning framework learns to imitate a general class of \"good\" behaviors. B2R's Decision Transformer framework, enhanced by our realignment, learns the precise, causal dynamics between actions and future costs.\n\n- **Data Utilization:** B2R's \"region-wide supervision\" creates a dense learning signal from all available safe trajectories for the target constraint κ. TraC's partitioning may discard useful behaviors in its \"undesirable\" set, leading to a sparser signal.\n\nIn short, B2R's superior performance stems from its fine-grained, temporal supervision, allowing it to learn the dynamics of safe behavior more effectively than TraC's coarse, classification-based approach. We will clarify this distinction in the revised manuscript.\n\n> **W3:** The assumptions employed in theoretical results are unclear.\n\n> **Q4:** The assumption of \"prediction-error bound\" seems to be hard to visualize. Can authors clarify where this assumption will be violated?\n\nWe will now clarify specific scenarios where our assumption would be violated.\n\n**1. Violation of Single-Step Prediction Accuracy**  $(\\mathbb{E}[|c_t - \\hat{c}_t|] \\le \\sigma)$\n\nThis assumption is violated when the agent encounters a significantly out-of-distribution (OOD) state not represented in its training data.\n\n**Scenario**: An autonomous car, trained exclusively on data from sunny weather, is deployed in a winter storm and encounters black ice for the first time. The model, unable to comprehend the extreme danger, might predict a low cost similar to a wet road. The true cost would be substantially higher, causing a large prediction error $(|c_t - \\hat{c}_t| \\gg \\sigma)$ that violates the assumption.\n\n**2. Violation of Conservative Long-Term Estimation**  $(\\sum \\hat{c}_t \\le \\kappa - \\delta/2)$\n\nThis assumption is violated if the model is overly optimistic and generates a sequence of actions without a sufficient safety margin.\n\n**Scenario**: For a task with a total cost budget of $\\kappa = 100$, the model generates a sequence of actions with a predicted total cost of $\\sum \\hat{c}_t = 99.9$. This generated sequence is invalid because it leaves virtually no safety buffer ($\\delta \\approx 0$). Any minor, unpredicted cost overrun during execution would cause the true accumulated cost to breach the hard constraint.\n\n> **Q1:** In the problems where the results provided are not safe, can the authors elaborate on why the solution does not work well? Is it because of tight constraints or complex cost functions or something else? It would help to provide a more detailed analysis.\n\nB2R's success in 35 out of 38 tasks confirms its efficacy. However, its failures in CarCircle 1&2 and AntCircle stem from extreme properties of their offline datasets and environments. As our main experiments show, most other baseline methods also struggle significantly or fail to achieve safety in these particular tasks.\n\nThese environments present deceptive data landscapes and feature complex, sensitive dynamics with steep cost penalties. High-reward trajectories in these environments are often high-cost, leading to extremely sparse or absent high-reward/low-cost safe examples. In such scenarios, the effective safe action space can become extremely narrow near unsafe regions, implying that the environment itself offers very limited safe solutions during critical moments. Achieving safety in these cases necessitates precise, long-horizon control. The lack of diverse, high-quality safe data demonstrating robust control near these unforgiving safety boundaries limits B2R's ability to learn zero-violation performance.\n\n> **Q2:** Transformer based approaches for RL are known to struggle in environments where there is transitional uncertainty. Can authors clarify how they have addressed this issue?\n\nThank you for this insightful question on transitional uncertainty in Transformer-based RL. Upon reviewing of the SafeRL benchmarks (e.g., DSRL, Safety-Gymnasium, and BulletSafetyGym), we confirm that the state transitions in these environments are predominantly deterministic. These tasks are abstractions of real-world physical systems, where the next state is fully determined by physical laws given the current state and action—without inherent stochasticity in the transition function. Stochasticity, if present, typically arises from observation noise or external disturbances rather than probabilistic transitions.\n\nHowever, if transitional uncertainty were present (e.g., in game-like environments with random elements, such as randomized obstacle placement akin to a 2048-style puzzle where tile spawns are probabilistic), B2R incorporates features to mitigate Transformer limitations:\n\n- **Rotary Positional Embeddings (RoPE):** B2R uses RoPE to improve temporal modeling, which helps Transformers better capture sequential dependencies in uncertain transitions. RoPE's rotational invariance allows the model to generalize across varying sequence lengths and patterns, reducing sensitivity to transitional noise.\n\n- **Region-Wide Supervision:** B2R's realignment of cost-to-go signals creates a denser, more diverse training distribution, enabling the Transformer to learn robust representations that generalize under uncertainty. This contrasts with sparse boundary supervision, which can amplify errors in stochastic settings.\n\nWe will clarify this in the revision and note experiments on stochastic variants as an important direction for future work.\n\n>**Q3:** The ablation study on CTG realignment strategies is insightful, with \"Shift\" performing the best overall. However, the paper also mentions that \"environment-specific design choices can further enhance performance\". Could the authors provide more detailed guidance or propose a meta-strategy for adaptively selecting or tuning the CTG realignment method (e.g., Shift, Avg, Rand, Scale) based on dataset characteristics, environment properties, or observed real-time performance, rather than a fixed choice?\n\nThis is an excellent and forward-looking question. We are grateful to the reviewer for engaging so deeply with the ablation study in Appendix B.1.   \n\nOur choice of the simple, non-parametric \"Shift\" strategy in the main paper was a deliberate one, primarily motivated by scientific clarity. As the first work to propose and validate the \"region-wide supervision\" paradigm, our goal was to cleanly isolate and demonstrate the effectiveness of this core concept, ensuring that observed performance gains were directly attributable to our central idea without confounding effects from a more complex, learnable module. \n\nWe fully agree that an adaptive or learned realignment strategy is a promising direction for future research. Such a mechanism, potentially conditioned on trajectory characteristics or environment dynamics, could offer more fine-grained control over the cost-reward trade-off. For instance, a learned module could allocate the \"safety margin\" more intelligently by preserving it for high-risk regions of the state space, potentially unlocking further performance gains.  This represents a valuable extension that can be built upon the framework we have established. We will incorporate this promising direction—developing an adaptive realignment strategy—into the Future Work section of our revised manuscript."}, {"review_id": "H5UDdRbtJx", "reviewer": "Reviewer_AgLh", "summary": "This paper addresses a critical limitation in existing sequence-model-based approaches for offline safe reinforcement learning (RL), namely the symmetric conditioning of reward-to-go (RTG) and cost-to-go (CTG) in decision transformer-style models. The authors propose Boundary-to-Region (B2R), a novel framework that redefines CTG as a strict boundary constraint, enabling asymmetric conditioning. The B2R framework includes trajectory filtering, CTG realignment, and the use of Rotary Positional Embeddings (RoPE) to enable safe and effective policy learning. Theoretical analysis under simplifying assumptions is provided, and experiments across 38 safety-critical tasks compare B2R with baselines including BC-All, BC-Safe, CDT, BCQ-Lag, CPQ, COptiDICE, and TraC.", "questions": "Questions\n1. The authors mention that B2R’s performance may degrade in settings with limited safe data and suggest data augmentation or generative modeling as possible solutions. Do the authors have any preliminary insights or intuitions on which strategies might be most promising in this context? Any specific challenges they foresee in integrating such approaches with the B2R framework?\n\n2. B2R assumes a fixed cost budget across all episodes for CTG realignment. Have the authors considered how B2R might extend to non-stationary cost regimes, for instance, where cost thresholds vary due to environment drift or contextual constraints? Do they anticipate significant challenges in adapting the CTG realignment process in such settings?\n\n3. While B2R ensures constraint satisfaction via hard filtering of unsafe trajectories, might this strategy limit the diversity of the training signal by excluding high-reward but marginally unsafe behaviors? Could the authors comment on the potential of soft filtering mechanisms to trade off safety and exploration more flexibly?\n\nMinor clarity issue: \nThe abstract introduces acronyms such as RTG, CTG without any definition or expansion. While these terms are common within the RL community, abstracts should remain accessible to broader audiences.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- The paper identifies a concrete and well-motivated issue in the use of symmetric token conditioning in safe RL via supervised learning, which is both conceptually sound and practically impactful.\n\n- The B2R framework innovatively addresses the reward–cost asymmetry by aligning all safe trajectories to the deployment-time constraint budget through CTG realignment, coupled with trajectory filtering. The design is simple yet powerful.\n\n- The method is extensively validated on the DSRL benchmark, showing state-of-the-art results in many environments, outperforming CDT, BC-Safe, and others, while preserving constraint satisfaction.\n\n- Safety guarantees are derived under clear assumptions, providing both expected and probabilistic bounds on cost violation.\n\nWeaknesses\n- The method’s reliance on trajectory filtering limits its applicability when safe data is sparse or when behavior policies collect highly unsafe trajectories. This is acknowledged by the authors.\n\n- While the authors provide a valuable comparison of multiple CTG realignment strategies, the clear empirical superiority of the simple \"Shift\" method may suggest that additional exploration of more complex alternatives offers diminishing returns in this context.\n\n- Although discussed briefly, potential negative use cases (e.g., overly conservative policies in safety-critical control) or mitigation strategies are not detailed.", "was_revised": true, "final_justification": "The rebuttal addressed all of my earlier points. The additional ablation results confirmed B2R’s robustness under reduced safe data, and the preliminary multi-target extension showed feasibility for non-stationary cost regimes. I don’t see any remaining issues, so I am keeping my score at 5 (Accept).", "rebuttal": "We are deeply grateful for your exceptionally positive and insightful review. The forward-looking questions are particularly valuable, and we are glad to engage with these ideas, which we believe will help chart the future of this research direction.\n\n> **W1:** The method’s reliance on trajectory filtering limits its applicability when safe data is sparse or when behavior policies collect highly unsafe trajectories. This is acknowledged by the authors.\n\n> **Q1:** The authors mention that B2R’s performance may degrade in settings with limited safe data and suggest data augmentation or generative modeling as possible solutions. Do the authors have any preliminary insights or intuitions on which strategies might be most promising in this context? Any specific challenges they foresee in integrating such approaches with the B2R framework?\n\nThank you for this forward-looking question. Our primary insight comes from a data ablation study we conducted, which we believe lays the groundwork for exploring the strategies you mentioned.\n\nTo quantitatively evaluate B2R's robustness and gain these initial insights, we conducted a comprehensive data ablation study. We retrained B2R on subsets of the filtered safe dataset, sampled at 5%, 20%, 50%, and 100%. The results on four representative tasks (with a cost threshold of κ=10) are shown in our response to Reviewer LsnC (W1). We kindly direct your attention to that section for the detailed results, as space constraints here prevent a redundant presentation. \n\nOur key findings from this data ablation study are:\n\n- With 20% data, B2R maintains safety (e.g., costs below 1.0 in BallCircle and DroneCircle) while achieving competitive rewards.\n- Even at 5% data, B2R shows resilience, delivering high rewards with reasonable costs in easysparse and mediummean.\n\nOur intuition is that this resilience stems directly from B2R's core mechanism: region-wide supervision, which creates an inherently denser and more robust learning signal compared to boundary-focused methods.\n\nFor extreme scarcity scenarios, we believe the most promising approach would be to integrate generative models in a way that is highly synergistic with the B2R framework. Instead of generic data augmentation, one could train a conditional generative model on the filtered safe dataset. This model would learn the distribution of safe state-action sequences and generate novel yet plausible safe trajectories, enriching the training set beyond the diversity present in the original offline data. This approach presents both opportunities and challenges:\n\n- **Opportunity:** The core strength of B2R is its ability to leverage the entire safe region. By using a generative model to densify this region with new, valid samples, we could further enhance policy robustness and performance, especially in extremely sparse-data regimes.\n- **Challenge:** The primary challenge lies in ensuring the fidelity of the generated data. The generated trajectories must be both dynamically feasible and genuinely safe. A generative model that produces unrealistic or subtly unsafe behaviors could poison the training process. Verifying the safety and feasibility of generated data is an important research problem.\n\n> **W2:** While the authors provide a valuable comparison of multiple CTG realignment strategies, the clear empirical superiority of the simple \"Shift\" method may suggest that additional exploration of more complex alternatives offers diminishing returns in this context.\n\nWe appreciate the reviewer's insightful observation on our CTG realignment ablation and the empirical strength of the \"Shift\" method. We agree that complex alternatives might offer diminishing returns in many contexts, underscoring the efficiency of simple designs.\n\nOur paper's primary aim is to establish B2R's core effectiveness, demonstrating how region-wide supervision addresses key offline safe RL limitations. Prioritizing the simple yet powerful \"Shift\" strategy, which preserves original cost decay profiles, allowed us to clearly isolate the benefits of this novel supervision. This simplicity also boosts robustness and generalizability. Our comprehensive ablation study confirmed that cost realignment itself, regardless of specific technique, robustly provides significant benefits over baselines.\n\nLooking ahead, we believe there remains ample room for innovation in this direction. For instance, future work could explore more elegant, data-adaptive realignment mechanisms—such as learned modules that dynamically adjust based on trajectory characteristics or environmental dynamics. This could involve investigating how to learn optimal \"cost shapes\" that go beyond simple shifts, potentially addressing complex scenarios with highly sparse or noisy cost signals, or cases where the optimal cost-reward trade-off is more nuanced.\n\n> **W3:** Although discussed briefly, potential negative use cases (e.g., overly conservative policies in safety-critical control) or mitigation strategies are not detailed.\n\n> **Q3:** While B2R ensures constraint satisfaction via hard filtering of unsafe trajectories, might this strategy limit the diversity of the training signal by excluding high-reward but marginally unsafe behaviors? Could the authors comment on the potential of soft filtering mechanisms to trade off safety and exploration more flexibly?\n\nWe thank the reviewer for these insightful question, which astutely identify the key trade-off in our hard filtering design. Our use of hard filtering was a principled choice for this foundational work. It provided an unambiguous \"safe region\" that was instrumental for our theoretical analysis and allowed for a clean evaluation of our core concept.\n\nWe agree that soft filtering is a powerful mitigation strategy. For instance, one might retain all trajectories but down-weight the loss contribution of those whose cost marginally exceeds κ in proportion to their degree of violation. This approach would:\n- Permit learning from high-reward yet slightly unsafe behaviors,  \n- Facilitate exploration of the Pareto frontier between reward and safety,  \n- Better suit applications where safety constraints act as strong preferences rather than absolute rules.  \n\nThis approach, of course, requires careful calibration of the weighting scheme to define the \"softness\" of the boundary—an important research question in itself.\n\n> **Q2:** B2R assumes a fixed cost budget across all episodes for CTG realignment. Have the authors considered how B2R might extend to non-stationary cost regimes, for instance, where cost thresholds vary due to environment drift or contextual constraints? Do they anticipate significant challenges in adapting the CTG realignment process in such settings?\n\nThe reviewer's question highlights a crucial point: our B2R framework is primarily designed for scenarios with a fixed cost budget. While this addresses a common need in safety-critical applications, generalizing to dynamic cost regimes, where thresholds vary due to environment drift or contextual constraints, represents a significant challenge for our current approach.\n\nHowever, recognizing this importance, we are exploring a novel extension. The core idea is to enable a single B2R agent to adapt to multiple, varying cost targets, avoiding retraining for each new constraint. Our proposed scheme for extending B2R to handle multiple cost targets involves a three-step process:\n\n**1.Multi-Target Data Preparation:** Instead of using a single fixed cost threshold $\\kappa$, we define a set of target thresholds $\\{\\kappa_1, \\kappa_2, \\dots, \\kappa_N\\}$. For each $\\kappa_i$, we filter the original offline dataset to create a corresponding safe subset: $\\mathcal{D}_{\\text{safe},i} = \\{\\tau \\in \\mathcal{D} \\mid C(\\tau) \\le \\kappa_i\\}$.\n\n**2.Per-Target CTG Realignment:** For every $\\mathcal{D}_{\\text{safe},i}$, we apply B2R's existing CTG realignment procedure, but crucially, using its specific target $\\kappa_i$. This process yields $N$ distinct realigned datasets: $\\{\\mathcal{D}'_1, \\mathcal{D}'_2, \\dots, \\mathcal{D}'_N\\}$.\n\n**3.Unified Conditional Training:** Finally, these $N$ realigned datasets are merged into a single, unified training set: $\\mathcal{D}_{\\text{unified}} = \\mathcal{D}'_1 \\cup \\mathcal{D}'_2 \\cup \\dots \\cup \\mathcal{D}'_N$. The B2R model is then trained on this combined dataset, with the specific target $\\kappa_i$ implicitly influencing the conditioned action generation during training and inference.\n\nPreliminary validation experiments confirm the feasibility and promise of this multi-target learning. Results below show a single \"multi-target\" B2R agent achieves comparable reward and cost performance across different κ values against \"single-target\" agents.\n\n| Task | $\\kappa$ | multi(R) | multi(C) | single(R) | single(C) |\n|---|---|---|---|---|---|\n| **easysparse** | 10 | 0.70 | 0.48 | 0.63 | 0.26 |\n|| 20 | 0.81 | 1.12 | 0.80 | 1.08 |\n| | 40 | 0.86 | 0.70 | 0.89 | 0.81 |\n| **BallCircle** | 10 | 0.68 | 0.90 | 0.65 | 0.81 |\n| | 20 | 0.71 | 0.76 | 0.70 | 0.79 |\n| | 40 | 0.73 | 0.39 | 0.65 | 0.16 |\n| **AntVelocity** | 10 | 1.00 | 0.73 | 0.98 | 0.45 |\n| | 20 | 1.00 | 0.53 | 0.99 | 0.46 |\n| | 40 | 0.99 | 0.32 | 1.00 | 0.34 |\n\nThese exploratory results demonstrate the viability of extending B2R to non-stationary cost regimes without a complete redesign. Despite encouraging preliminary results, extending B2R to non-stationary cost regimes critically challenges its generalization robustness under data distribution shifts and its fine-grained control over dynamic cost signals, **demanding the model grasp the dynamic semantics of cost constraints beyond simple numerical adaptation.**"}]

Author rebuttal:

Meta-review: This paper considers the problem of offline safe reinforcement learning (OSRL) where the goal is to learn a reward maximizing policy with respect to a given cost constraint from a given offline dataset. Specifically, it identifies a fundamental limitation of sequence modeling based OSRL methods (e.g., decision transformer): they treat return-to-go (RTG) and cost-to-go (CTG) equivalently. The paper argues why this is not appropriate because RTG is a flexible objective to maximize whereas CTG corresponds to a strict safety budget constraint -- leads to unreliable safety, especially for OOD cost trajectories. The paper proposes a intuitive approach referred to as Boundary-to-Region (B2R) to address this limitation by enabling asymmetric conditioning of RTG and CTG via trajectory filtering, CTG realignment, and rotary positional embeddings. Under simplifying assumptions, B2R is analyzed theoretically. Experiments on 38 benchmark tasks demonstrate the efficacy of B2R over prior baseline methods.

All the reviewers' appreciated the paper's contributions but also raised several good questions. The author response addressed these questions adequately resulting in increase of scores in some cases. Overall, this is a good paper on an important topic. I recommend accepting the paper and strongly encourage the authors' to incorporate all the discussion into the final paper and also release the code.
Decision: Accept (poster)