Adaptable Safe Policy Learning from Multi-task Data with Constraint Prioritized Decision Transformer
Authors: Ruiqi Xue
Ziqian Zhang
Lihe Li
Cong Guan
Lei Yuan
Yang Yu
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Safe Reinforcement Learning
Offline Reinforcement Learning
Multi-task Reinforcement Learning
Primary area: reinforcement_learning
TL;DR: This paper introduces CoPDT, a method of using one unified and adaptable DT model for multi-task (multi-budget or multi-constraint) offline safe RL.

Abstract: Learning safe reinforcement learning (RL) policies from offline multi-task datasets without direct environmental interaction is crucial for efficient and reliable deployment of RL agents. Benefiting from their scalability and strong in-context learning capabilities, recent approaches attempt to utilize Decision Transformer (DT) architectures for offline safe RL, demonstrating promising adaptability across varying safety budgets. 
However, these methods primarily focus on single-constraint scenarios and struggle with diverse constraint configurations across multiple tasks. 
Additionally, their reliance on heuristically defined Return-To-Go (RTG) inputs limits flexibility and reduces learning efficiency, particularly in complex multi-task environments. To address these limitations, we propose CoPDT, a novel DT-based framework designed to enhance adaptability to diverse constraints and varying safety budgets. Specifically, CoPDT introduces a constraint prioritized prompt encoder, which leverages sparse binary cost signals to accurately identify constraints, and a constraint prioritized Return-To-Go (CPRTG) token mechanism, which dynamically generates RTGs based on identified constraints and corresponding safety budgets. Extensive experiments on the OSRL benchmark demonstrate that CoPDT achieves superior efficiency and significantly enhanced safety compliance across diverse multi-task scenarios, surpassing state-of-the-art DT-based methods by satisfying safety constraints in more than twice as many tasks.

A daptabbe Safe Policy Learning from Multi-task Data with Constraint Prioritized Decision Transformer
Ruiqi Xue Ziqian Zhang" Lihe Li Cong Guan Lei 123, Yang Yu 12,3.
National Key Laboratory for Novel Software Technology, Nanjing University School of Artificial Intelligence, Nanjing University Polixir Technologies
Nanjing, China
(xuerq, zhangzq, 1ilh, guanc, yuan1)0lamde nju. edu. cn yuyenju. edu. en
Abstract
Learning safe reinforcement learning (RL.) policies from offline multi-task datasets without direct environmental interaction is erucial for efficient and reliable deploy- ment of RL agents. Benefiting from their scalability and strong in-context learning capabilities, recent approaches attempt to utilize Decision Transformer (DT) archi- tectures for offline safe RL, demonstrating promising adaptability across varying safety budgets. However, these methods primarily focus on single-constraint sce- narios and struggle with diverse constraint configurations across multiple tasks. Additionally, their reliance on heuristically defined Return- To-Go (RTG) inputs limits flexibility and reduces learning efficiency, particularly in complex multi-task scenarios. To address these limitations, we propose CoPDT, a novel DT-based framework designed enhance adaptability to diverse constraints (i.e., cost fune- tions) and varying budgets. Specifically, CoPDT introduces a constraint prioritized prompt encoder, which leverages sparse binary cost signals to accurately identify constraints, and constraint prioritized ReturnToo-Go (CPRTG) token mechanism, which dynamically generates RTGs based identified constraints and correspond- ing safety budgets. Extensive experiments on the OSRL benchmark demonstrate that CoPDT achieves superior efficiency and significantly enhanced safety com- pliance across diverse multi-task scenarios, surpassing state-of ofti---f-hh--arr DT-based methods by satisfying safety constraints more than twice many tasks.
Introduction
Safe reinforcement learning (RL., focusing deriving policies that explicitly satisfy predefined safety constraints, has garnered significant attention due to its applicability in critical domains such as autonomous driving (Zhang al., (02I). robotic control (Brunke al. 2002), and aligning large language models with human values (Dai et al. 2024). However, conventional online RL methods rely heavily on inherently risky trial-and-error interactions. limiting their practical deployment in safety-critical environments (Xu et al. 2022b). To address this limitation, offline safe RL, aiming to learn safe policies purely from previously collected datasets without direct environmental interaction, has emerged as prominent research direction (Liu et al. 2024b Chemingui et al.) (025).
Typical offline safe RL approaches predominantly rely on static datasets collected under predefined safety constraints, employing methods such as Lagrangian constraint optimization (Le et al. 2019) and Hamitton-Jccoii (HJ) reachability analysis (Zheng al.) (2024) derive policies with robust safety guarantees. Despite their effectiveness, these methods exhibit limited adaptability in realistic
Cooresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

multi-task scenarios involving multiple safety constraints (i.e., multiple cost functions) or dynamically changing safety budgets. For instance. in autonomous driving tasks, vehicles must respect diverse constraints, including lane-dependent speed limits (multt-constraiin) and varying fuel consumption budgets along a driving trajectory (multi-budget) (Kiran et al., 202T). Addressing these multi-task settings via independently trained individual policies is computationally prohibitive and inefficient, as it neglects the intrinsic structural similarities across tasks. This exacerbates learning complexity, particularly in practical offline safe RL contexts, where available data is inherently scarce (Dulac-
Arnold et al. 202T; Gu et al. 2022).
Leveraging nowledge-sharingg mechanisms, multi-tas task learning approaches (Zhang & Yang 2021 I
Gronauer & Diepold (022) have notably enhanced sample efficiency across interrelated tasks. Transtormer architectures Vaswaai et al. [201]] [slam et al. (2024), celebrated for their scalability and in-context learning ability, have demonstrated remarkable versatility in fields such as natural language
processing (Kalyan et al. (021), computer vision (Khan et et. al 2022), and decision-making (Yang et al.,
2023). Recent developments have also expanded Transformers into offline safe RL, incorporating
explicit constraint representations such as Cost-To-Go (CTG) tokens (Zhang al.] 2023 Liu et al.
2023) or logic tokens (Guo et al., (024)) These tokens analogously extend the Retur-To-Go
(RTG) formalism introduced in DT (Chen et al., 2021 Li et al. 2023), facilitating autoregressive policy training under explicitly modeled safety constraints. Despite significant progress, these approaches currently face limitations in adaptive safety within complex multi-task environments: they inadequately distinguish between diverse constraint specifications, resulting in reduced adaptability when cost functions differ across tasks. Additionally, inherent tensions between maximizing rewards and maintaining safety force these methods to rely extensively on predefined RTG objectives derived from prior knowledge, thus restricting dynamic adjustment capabilities in response to evolving safety budgets during policy execution, and ultimately constraining their flexibility.
To overcome these limitations, we introduce Constraint Prioritized Decision Transformer (CoPDT), a novel DT-based framework designed for offline safe RL that exhibits enhanced adaptability to multiple safety constraints and dynamically changing budgets in multi-task settings. Specifically, we propose a constraint prioritized prompt encoder, which explicitly partitions trajectory data into safe and unsafe segments guided by observed cost signals, individually encoding each segment to effectively capture constraint-speciffe characteristics. This approach enables CoPDT to clearly differ- entiate among diverse constraints, facilitating adaptive decision-making across varying conditions. Moreover, we introduce a constraint prioritized Return-To-Go (CPRTG) token mechanism, dynam- ically generating RTG targets based on the constraint-speciffc encodings and current CTG tokens. Unlike traditional methods reliant upon static RTG inputs, the proposed CPRTG mechanism achieves enhanced flexibility by adaptively accommodating changing safety budgets, thereby producing more robust and effective policies. Extensive experiments on the OSRL benchmark (Liu et al., 2024b) demonstrate that CoPDT significantly surpasses existing DT-based approaches in multi-task safety adaptability, successfully satisfying constraints in over twice as many tasks, and exhibiting superior efficacy in safe transfer learning scenarios.
2 Preliminaries
2.1 Multi-task offline safe RL
Safe RL is typically modeled as Constrained Markov Decision Process (CMDP), defined as tuple (S, A, r, c, b). Here, S and A denote the state and action spaces, respectively; r S x A + [- Rmmxx Rmaa] and S x A- - {0, 1} represent the reward and binary cost functions. P: x A x S [0. 1] specifies the transition dynamics, ? E (0. 1) is the discount factor, and b is the safety threshold. A policy 5 + A(A) maps states to action distributions. Under policy ग, the expected discounted reward return and cost return are defined as I(TT) = ET-P, 7 at)] and C(T)- Ey,p, EE= ?'((s, a.), where т = (so. ag. $1, a1....)
P. denotes a trajectory induced by and the environment dynamics P. Thus, the objective of solving CMDP is to find a policy that maximizes reward return while ensuring the cost return remains below the safety threshold:
max R(T), s.l. C(T) b.
In multi-task offline safe RI., a policy " must effectively adapt across multiple safety budgets or constraints. Each task is formulated as distinct CMDP, characterized by variations in elements

---

such as states S, actions A, transition dynamics P. reward functions r, cost functions c, and safety thresholds (budgets) b. Specifically,, multi-budget scenarios involve tasks that differ primarily in their safety threshold b, where, for example, a lower budget (b- = 10) enforces more conservative behavior compared higher budget (b- = 100), which allows more aggressive policies. In contrast, multi- constraint scenarios entail variations in the cost functions themselves; for instance, one task may employ a cost function C₁ = Lcondition (v > 1.0), whereas another task adopts 02 = Lcondition (v > 2.0), where llcondtiion (-) is the indicator function and v denotes state variable such velocity. Additionally, other CMDP components (S, A, P.r) may also vary across tasks. During training, the policy is exposed to collection of tasks {ī) and their offline datasets {D while at deployment, must generalize effectively to a given task T. If {Ti)}= the policy receives a single expert trajectory for task identification in multi-constraint settings. Otherwise, for efficient transfer to an unseen task, it is provided with L expert trajectories from task T.
2.2 Decision Transformer for offline safe RT
Decision Transformer (DT) is one of the most prominent methods that apply sequence modeling to decision-making. It uses Transformer framework, modeling RL's reward maximization problem as a sequence prediction task. When applied to offline safe RI., DT models the trajectory as the following to support training and generation with Transformers:
T T= T((Ōi, RI, $1, 41, C2, R2, 82, a2,..., Ör, Ri, ST, sT,),,
(2)
where Rg = Ei T3 is the Return-To-Go (RTG) token at time step L, and G. = Eizr 4 is the Cost-To-Go (CTG) token. Let T- = (Či-k, Ri-K. St-K, Qt-K. Cr-1, C4-1,Ri, Rt-1,,
DT's policy can be expressed (ā, KT, Co2 R2, st), inferring the current action based on the previous K-step trajectory, the current RTG, CTG and state. The policy is trained by minimizing the difference between the inferred actions å, and actions a, in offline datasets. During deployment, the DT's policy Tor requires initial RTG token R.. CTG token E₁ (which is exactly the safety threshold b), and state 81 to generate actions, with the RTG and CTG updated using Ret1 Rr rt and Ci+1 C. Cz.
3 Method
This section gives the detailed CoPDT, a novel algorithm for adaptable multi-task offline safe RL (Figure D. Section 3.1 presents CoPDT's procedure for prompt encoding, Section 3.2 illustrates the process of CPRTG token generation, while Section 3.3 introduces CoPDT's overall algorithm.
3.1 Constraint prioritized prompt encoder learning
To enable CoPDT to effectively adapt to tasks with varying constraints, the policy must accurately identify these constraints. Therefore, we introduce a prompt encoder that extracts relevant information from task trajectories to generate task representations, helping distinguish between constraints. Environment-speeefie Encoders First, considering the presence of tasks with different state action spaces, using single unified neural network for all tasks becomes challenging due to the inconsistency in input dimensions. Therefore, we categorize different tasks into distinct environments, where tasks within the same environment share identical (S, A, P). Then, for each environment, apply enviromment-speccin Specifically, for the i-th environment E,, introduce two encoders, e,, for states and ea, for actions, along with decoders dy,; and da,5- To ensure that the action encodings retain sufficient information from the original actions, Ca,i and du,s trained using the reconstruction loss:
encoders to align the dimensionality of states and actions across tasks.
min Ea,~D, (Ca..((a..(aa))) - a1) dani
(3)
where D: represents the combined offline dataset for all tasks within environment & and ag is the sampled action. As for ex,; and da,,, we introduce additional inverse dynamics model gi, and train them by simultaneously minimizing the reconstruction error and the inverse dynamics error to incorporate both state information and dynamics transition information into state encodings:
min E E (dd,, - =,)² + (g.(e.. e.,()))) - (a.))),
(4)
3

---

@
Few-shot Demonstrations
causal transformer
Reconstruction Learning
s'
r
Safe Patch
Decoder
a
Ps
prompt
Pu
NLL Loss
(a) Constrained Prioritized Prompt Encoder
deployment
training
(b) Constrained Prioritized Return-To-Go
Figure I: Structure of CoPDT.
where Sc, ae, si are the sampled state-action transitions.
In previous DT-based methods for handling same-environment tasks with varying rewards, the typical method Prompt-DT (Xu et al.) 2022c) uses limited K -tstep trajectory segments as prompts for task representation. However, due to the sparse nature of costs, short trajectory segments may not contain sufficient information for constraint recognition. Therefore, new methods are needed to more effectively extract cost-related information from the trajectories.
Constraint Prioritized Prompt Encoder address the challenge posed by the sparse nature c, propose the constraint prioritized prompt encoder pe- Specifically, Pe (ps, Pu) consists of two sub-prompt encoders. Given a reference trajectory T (s1, 01 F1, c....... ST, ar, TT, eT) for task T, where rz, Ct the reward and cost of time step the prompt encoding is computed follows:
T
= -=P(TT))
Σ 1=1
(Tconditioon (c: = O)ps(st, a2, rı) Icoudition (c. = 1)),(s(, oz, r)),
(5)
T
where Icoudition is the indicator function. Since task differences may arise from variations in state spaces (environments), reward functions, and cost functions, it is crucial for Pe capture information from all three factors ensure accurate task differentiation. To accomplish this, we introduce three additional decoder networks Ss fr. and Se, and train them by minimizing prediction errors:
min PeJarrdfe
ET E Brraaaa (ff(St, at, Pe(r")) - x(2²
(6)
+(fr(se, ar (e())) r2) (fe(s), at, Pe(p)))())))))))))))))))))))))) 2
where Dy is the dataset for task T, containing trajectories with both reward and cost information, 5* refers to the trajectory that includes si, Tt, Cz.
The constraint prioritized prompt encoder exploits the binary nature of cost signals to decouple cost information from the prompt encoder's input and uses to select the appropriate sub-encoder network. This design enables efficient utilization of cost information by capturing task distinctions through variations the input distributions of states, actions, and rewards across different encoders. The resulting prompt encoding serves as part of the input of the DT policy for task identification.
3.2 Constraint prioritized RTG token generation
Leveraging constraint-level representations from the prompt encoder and budget-level signals from the CTG, a unified DT policy can potentially support adaptable multi-task offline safe learning. However, prior methods often overlook a key issue in multi-budget settings: the conflict between reward and cost objectives, which are represented by the RTG token ît, and the CTG token C, in DT, respectively. Handling varying budgets requires adapting to diverse CTG inputs, but selecting a suitable RTG under changing CTGs remains challenging. When RTG values are misaligned, the agent tends to prioritize rewards due to its limited ability to resolve the conflict between RTG and

---

CTG, risking safety violations. Therefore, our goal is to automatically generate a conflict-free RTG based on the identified constraint representation and the current CTG.
Modeling RTG Conditioned on CTG To achieve the generation of conflict-free RTGs in CoPDT for different budgets, straightforward approach is model RTG as conditioned on CTG, i.e., learning the model p( [e|CC CC; from the offline data. Since the relationship between RTG and CTG is primarily derived from offline trajectories, where multiple RTGs might correspond to the same CTG, we further constrain the generation process by incorporating state information each time step, and model p as non-deterministcc normal distribution N approximated by a neural network ds. Formally, given the offline data D {((t, at Tt, Tt, Ct, Re, Ct, ) we have:
as (-((,,, st) = (Cp 8t), Eg(C,, st)),
where us and Eg are the mean and standard deviation networks, respectively, and C₁ and represent the CTG and state at step L. To maximize the probability of generating Rt conditioned on the given Ct and St, the model as is optimized by the following negative log- lekeikllooodd objective:
min 4s
oc, R. log 4o (Ro Ct, 8r]]-
(8)
CTG-based B-quantile Sampling for Safe And Expert Inference However, such modeling only prioritizes CTG without considering the need for expert-level inferences after ensuring safety. Therefore, in addition to maximizing p( Ru Cr, we also aim to maximize p p(Re,Gr,, Cr. 8,) by introducing a variable G, that indicates the trajectory is expert after time step I. Similar to MGDT (Lee al. 2022), we apply Bayes' theorem obtain the following:
D(R, Cr st) x p(Ree Cr 84 st)p(G.) &t, 8t),
where p BRe, St)
st) represents the probability that the future trajectory expert given the current RTG, CTG, and state. Intuitively, when fixing Cr, a higher probability is attributed to p(Ge Re, Cr 8t) R. possesses a larger value. Therefore, this term could be maximized by sampling large R. However, when RG becomes excessively large, the conditional probability p( R& Cr, 8x) tends to decrease. To optimize the joint likelihood p( (R Rr,, Ct, st), it is therefore essential to select an appropriate R, that strikes suitable balance. To this end, strategy- Specifically, we first sample X values from qs (-((( st), and then choose the B-quantile among them as the final RTG token. To find a balanced fł, we propose the CTG-based 3 decay:
adopt quantile-baut based selection
3 = min + (Bsun Bend) Ć. C
Bena),
(10)
where Õ₁ is the initially given safety threshold, 3 and Bend are two hyperparameters. When CTG is innge-inii more room for potential future safety violations - larger B3 for more aggressive decision decision-making! is acceptable. Conversely, when CTG is small, the policy should be more conservative, resulting in a smaller B,.
Overall Generation Process In conclusion, at time step i, we sample X candidate values from C-(--C,, 8t), and chose the 34-quantile value as the CPRTG token, denoted as R. This token provides a simple but efficient method for adjusting policy conservatism while attaining high-rewarding behaviors during deployment. If the policy does not meet safety requirements, lowering Astn or Bend can increase conservatism without altering model parameters. Similarly, adjustments can be made to improve reward return when the policy is too conservative. In practice, we typically fix Baun as 0.99 and adjust Bend only. Additionally, the past trajectory T T= is incorporated into qø to improve inference accuracy and maintain greater consistency with training. Additional explanations and theoretical analyses of CPRTG from the perspective of offline RL. are provided in Appendix A
3.3 Overall algorithm
With the design above, we can apply CoPDT to multi-task scenarios to learn safe policies. Below, we briefly outline CoPDT's training and deployment. Detailed pseudo-codes are provided in Appendix B and the approach for task identification in unknown environments is provided in Appendix C
5

---

During training, CoPDT first learns the environment-specn speciff encoders by Equation (3) and Equa- tion (4) to align the dimensionalities of state, action spaces. Then, the constrained priorfizze prompt encoder Pe is trained via Equation ( to produce prompt encoding = that captures constraint-relevant information. Given the prompt encoding z as an additional input, the CPRTG generator go is then similarly optimized by Equation (8) to generate suitable RTGs under various constraints and budgets. With all components prepared, we finally learn the DT policy. Let the policy network be denoted as To, which consists of two output heads: T for actions and Te, for costs. The additional cost head is designed to help the policy capture cost-related patterns, thereby improving task identification. Given the expert trajectory T", the learned Pe, environment ID i (for choosing environment cerromment--pecific encoders), and sampled trajectory T-Kit; Re, 8t from task T's offline dataset, the input can be repre- sented as or (T-Ket; Cr, Rr, St, (T"), i). The cost output head Te. is modeled deterministically, while the action output head is modeled as a normal distribution:
T8
(0)) Eg,a(o:))
(11)
where 148,a and Eg.o are the mean and standard deviation networks for the action output head, respectively. We optimize the policy by minimizing the negative log-likelihood loss and negative entropy loss of the actions, as well as the difference between the predicted costs and true costs:
min E TT T000
[E
i K. Rr. |- log na,a T0 (or
(12)
A H To ( Ao (o(TTo,(O.)) - c0)
where H is the Shannon entropy regularizer (Haarnoja et al. 2018), Ås and de are two hyperparame- ters that control the weighting of the entropy regularization and the cost loss.
During deployment, the initial task safety threshold Ć₁ is provided, and in each time step, the CPRTG RG is computed to replace the original RTG Rg At this point, the policy's 's input is or (T-Kit Co, Re, St, Pele,e(TT po(T") ))), where
Deployment
--k:t = (Cr-R, Re-k, 81- at a........... CL-1, Rr-1, 81-1
(13)
4 Experiments
In this section, we present our experimental analysis conducted on 26 tasks from the OSRL (Liu et al. (024b) dataset to answer the following questions: (1) Can CoPDT outperform other baselines across various multi-task settings (Section 4 (2) How the design of CoPDT contributes to its performance (Section /. (3) Can multi-task learning bring about benefits, and whether each
component of CoPDT contribute effectively to the performance (Section 4.4) ? For page limits,
additional experimental results will be provided in Appendix F
4.1 Baselines and tasks
To evaluate the performance of CoPDT, we conduct experiments across several baselines in various multi-task settings. We first compare CoPDT to CPQ (Xu et al. 2022a). a widely used single- task offline safe RL method based on conservative estimation, and to CDT (Lu et al.) 2023), the Additionally, we consider FISOR (Zheng et al., (024) and LSPC (Koirala et al. (025), two recent baselines that integrate hard constraint modeling and demonstrates SOTA safety performance in single-task settings. To assess performance in more general multi-constraint scenarios, we extend CDT in two ways: (1) by training it on multiple constraints, resulting in MTCDT, and (2) by incorporating expert trajectory segments as prompts (Xu et al. 2022c), referred to as Prompt-CDT. These baselines are evaluated under three settings: Single-constrain Multi-budget, Multi-constraint Single-budger and Multi-constraint Multi-budged where each method aims to handle four distinct budget levels [10, 20, 40, 80) per constraint in multi- bullet settings. For training, single-task baselines like CPQ are trained separately for each budget level, while FISOR, which does not support budget conditioning, is trained four times independently and evaluated under the same protocol to ensure fair comparison. In conti-coonttaint settings, unified policy is trained for each method handle all 26 OSRL tasks simultaneously.
SOTA DT--bsed method designed for multi-budget Decision-making
The tasks selected from the OSRL dataset used in our experiments consist of 16 navigation tasks and 10 velocity tasks. Specifically, the navigation tasks involve two types of robots (Point and Car)

---

Table I: Overall normalized rewards and costs. Each value is averaged over 20 evaluation episodes, random seeds, and the given budgets in each setting. Bold: Safe agents. Gray: Unsafe agents. Blue: Safe agent with the highest reward in each setting.
Smplecmmtm Math-halge
Klatt conarare Maitshadgst
Phrmp-CIT
Prorp- CDD
41
PrinBrnol
D:7
5.25
0.54
5.16
1.30
0.16
1.50
0:65
0.06
0.45
4.06
0.57
6.00
8/12
0.44
0.69
4.17
0.55
4.50
0.14
0.55
0.53
0.04
0.45
4.32
0.11
1.1
0.17
1.70
0.14
1.41
0.45
4.65
0.43
4.32
0.09
1.43
0.38
3.51
0.40
4.22
0,08
0.98
PonCidel
0.94
0.58
0.55
6.64
0.59
0.43
0.51
0.52
0.53
0.88
0.52
0.47
0.55
0.ST
0.48
0.67
0.25
6:27
0.36
11.56
1150
1.20
0.71
1.66
11.27
1151
1.28
0.63
1.66
0.24
0.39
3.45
0.54
2.84
0.29
1.25
0.25
IUSS
0.31
1.22
0.46
2.50
0.55
5.34
0.19
0.00
0445
2.01
0.54
2.54
0.26
0.66
Panntal
1.50
0.227
1.42
0.89
0.13
B.97
0.19
0.888
11.26
1.18
0.24
1.49
11235
1.11
0.34
1:25
0.12
1.42
0.38
1.55
Curuuool
15:40
0.20
11.54
5.42
0.25
7.05
0.35
4.61
0.29
6.76
CuBanod
0:29
19.32
0.14
4.70
.0.03
0.40
-0.15
1.52
.002
1.33
0.24
5.58
0.25
6.01
.001
1.28
0.22
5.19
0.25
5.46
-0/12
0.94
CuCadel
4004
4.50
0.55
4.08
D:69
5.35
0.45
3.50
0.51
3.34
0.50
3.11
0.40
3.40
0.42
2.51
0.55
3.47
0.51
3.39
0.50
2.50
CurCccle?
1.31
0.63
6:26
0.56
7.44
0.55
6:11
0.56
6.37
0.57
5.61
1.67
CaCooll
2.20
0.64
1.30
0.54
0.55
CarCodl
0.57
4.72
0.42
2.59
6:07
0.27
0.12
0.59
0.19
0.51
0.52
1.50
0.40
3.05
0.12
1.20
0.30
1.53
0.38
2.70
0.13
0.91
Cattabl
D03
1.00
0.29
1196
8:25
0.43
0.22
1.56
0:20
00007
0.25
1.16
0.25
1.35
0.14
0.47
1.25
0.544
0.25
1U95
0.18
0.488
Cattabd
0.18
7.50
0.15
2.50
0.09
BU9
0.73
1.115
2.16
0.14
2.5
0.57
2.31
0.07
2:31
0.0%
0.71
1.40
0.65
-0/04
0.44
0.59
0.38
0.96
0L56
0.76
0.2
01.40
0.06
0.74
D:04
2.00
0.854
11.92
0.30
0.23
1.97
8:19
1.50
15.08
2.11
O66
6.55
13:33
5.01
0.57
4.23
0:15
1:49
0.32
1.50
0.86
0.97
0:31
1.17
1.31
0.70
6.85
0.15
1:15
0.63
5.50
0.27
1.00
2.24
0.36
0.55
15:50
1.05
35:91
14.71
2.20
0.598
1.02
0.95
01350
0.54
1.16
0.95
1.22
Waler2/ieiociig vo
0:84
0.46
0.29
1.91
0.13
1.11
0.40
4.39
0.32
2.00
1.27
25.63
0.95
14.59
0.34
4.58
1.25
24.53
0.81
14.30
0.35
4.44
W1
8:83
0.36
0.79
0.09
0.53
0.80
0.78
0.65
0:75
01.42
0.78
0.28
0.77
0.39
0:71
0.99
0.79
0.36
0.76
0.33
9.66
0.73
AutVtbecyyD
0.00
0.50
0.90
0.54
0.55
1.41
0.96
2.95
0.85
0.96
2.56
AstVdlocayV
Average
3.54
0.36
2.19
0.42
0.40
1.11
0337
4.48
0.39
4.81
1.00
0359
4.30
0.55
6.38
0.38
145
across four distinct scenarios: Button, Circle, Goal, and Push, with two tasks per scenario. On the other hand, the velocity tasks cover five different robots-Ant, HalfCheetah, Hopper, Swimmer, and Walker2d witt two tasks for each robot. Additional details can be found in Appendix D
4.2 Competitive results
In this section, we present the overall performance of CoPDT and all baselines across the three evaluation settings, and the results are summarized in Table I First, in the Single-constraint Multi- budget setting, CPQ exhibits poor performance, primarily due to its limited capacity to handle multi-budget datasets, which often include unsafe trajectories. In contrast, FISOR demonstrates robust safety performance, highlighting the effectiveness of its hard constraint modeling. However, both CPQ and FISOR fail to support adaptation to multiple budgets via a unified policy. On the other hand, the Transformer-bar based method CDT, which leverages the DT architecture, successfully extends the policy to handle multiple budgets simultaneously, achieving competitive reward performance. Nevertheless, its reliance on manually specified RTGs hinders its adaptability, leading to suboptimal safety outcomes due to conflicts between RTGs and CTGs. In comparison, our method, CoPDT, demonstrates better safety performance than LSPC and comparable results to FISOR, while being able to seamlessly adapt to multiple budgets using a unified policy, effectively resolving conflicts between RTGs and CTGs.
Next, in the multi-constrain settings, MTCDT experiences further degradation in safety performance, primarily due to its inability to distinguish between different constraints. In contrast, Prompt- CDT, which incorporates additional expert trajectory segments for constraint identification, achieves marginally better results, making safe decisions in a few more tasks compared to MTCDT. However, the limited availability of cost signals in the expert trajectory segments restricts their informativeness. leading to insufficient constraint- ountraint-awwarensss awareness and ultimately suboptimal performance. In contrast, our method CoPDT delivers consistent and significant improvements in safety performance across both single-budget and multi-budget settings, demonstrating the effectiveness of the constraint prioritized prompt encoder in accurately identifying and adapting to a variety of task task-specific constraints. Detailed standard deviations are provided in Appendix F.15
7

---

Normalloed Rewourd Aff Defferen Target
wosh Diftrenn Tarpet
Pointtutiont
3.504
X0
0.532
2.40
33.05
$4.79
17.3
5006
Torget rowward
Torget fteward
Tugel Cood
(a)
(b)
(c)
(d)
Figure 2: (a) Visualization of task PointButton. (b) The normalized rewards of CDT with different initial RTGs and CTGs. (c) The real costs of CDT with different initial RTGs and CTGs. (d) The safety performance of CoPDT and CDT across various budgets.
4.3
Visualization experiment
To reveal how our method CoPDT works, we here design visualization experiments to show the inherent conflict between reward and cost, and assess whether the proposed CPRTG mechanism can generate conflict-free RTGs that enable safe and adaptive decision-making under varying budget constraints. Specifically, we first investigate the conflict between CTGs and RTGs objectives in DT by analyzing CDT's behavior under four different target rewards and four budgets in PointButton] (Figure (a)). As illustrated in Figure 2 (b) and (c), we observe that both the reward and cost of the DT policy vary significantly with changes in the target reward. In contrast, the cost remains relatively insensitive to changes in the target cost. This asymmetric behavior indicates a conflict between reward and cost objectives, where the DT policy tends to prioritize reward, often at the expense safety. These findings indicate the necessity of generating suitable RTGs to achieve safe decision-making under different budget constraints. To further quantitatively reveal how the prioritization of CTGs influences safety, we compare the performance of CoPDT and CDT under ten different budgets. As shown in Figure 2 (d), CoPDT consistently satisfies safety constraints across all budgets, with cumulative cost decreasing in accordance with stricter budgets. In contrast, CDT exhibits severe safety violations, especially under stricter budget constraints, illustrating that CoPDT's CPRTG can effectively generate reasonable RTGs based on the CTG, ensuring the higher priority of CTG and enabling safe decision- -eaisinn-making under multi multi-budget scenarios. More results can be seen in Appendix F2 Appendix F66
We design additional experi- ments to further assess our constraint prioritized prompt encoder's ability in constraint identification. First, visu- alize the projection of the gen- erated prompt encodings, shown in Figure 3 (a), the en- codings corresponding to dif- ferent tasks are well-separated, indicating strong task discrim- inability. Next, to verify that the constraint prioritized prompt en- coder can effectively utilize cost signals for task identification, we conduct a visualization experiment in the PointCircle environment. In this experiment, we first randomly generat 5,000 (s, a, samples without including the cost c. Each of these samples is then encoded using both p. and Pu. followed by dimensionality reduction. The visualization results are shown in Figure (b). As can be observed, the encoded representations from p. and Pa exhibit clear differences, indicating that the cost label e significantly influences the encoding process. This demonstrates that the constraint prioritized prompt encoder effectively decouples the learned representations from the behavior policy. Finally, we conduct a prompt quality analysis to assess the encoder's robustness under varying prompt con- ditions, comparing our encoder with a simple MLP encoder across four prompt qualities: Expert, Mediuml, Medium2, and Random. Mediuml and Medium2 correspond to prompts consisting of unseen trajectories with high costs and low rewards. As shown in Figure (c), encoder maintains
CaPOT Waggaa
(4)
Figure 3: (a) Visualization of prompt encodings. (b) Comparison of safe patch encodings and unsafe patch encodings. (c) Prompt quality analysis on costs.
8

---

Gerendiaiion reauts
1318
140
Porea 201
(a)
(b)
(d)
Figure 4: (a) Zero-shot generalization results. (b) Transfer rewards in ChopperVelocing V2. (c) Transfer costs in HopperVelocityVV. (d) Main ablation results.
more stable performance across these degraded prompt conditions, demonstrating its robustness even under noisy or suboptimal inputs. Further details can be found in Appendix F.6land Appendix F.13
4.4 Benefits of multi-task learning and ablations
Since CoPDT trained in a multi-task setting. we here design experiments to validate its potential for promoting knowledge sharing. Specifically, we first evaluate its shot generalization to unseen constraints (unseen velocity limits) across three robot types. As shown Figure f (a), we can find that CoPDT outperforms Prompt-CDT in terms of average safety generalization, suggesting that its learned task representations capture richer, and our design could cover more transferable information for policy learning. However, noticeable performance gap remains when compared to policies trained directly on task-specific offline data, indicating the need for few-shot transfer learning. To this end, we introduce 10 expert trajectories under fixed budget of 10 as additional transfer data and compare two fine-tuning methods- full fine-tuning (FFT) and LoRA (Hu et al. 2022) agginstt learning from scratch. As shown in Figure a (b) and (c), regardless of the fine-tuning method, multi-task pretraining significantly enhances the policy's 's performance across both reward and cost metries, highlighting the effectiveness of CoPDT 's multi-task learning in enabling efficient adaptation to novel, yet related tasks. Additional transfer results are presented in Appendix [77
Finally, we conduct ablation studies on all selected OSRL tasks in the Multi-constraint Multi-budget setting to evaluate the contributions of key components in CoPDT. The following variants are considered: (1) W/o PE omits the constraint prioritized prompt encoder. (2) Simp PE replaces the
encoder with a simple MLP without separating safe and unsafe patches. (3) W/o CP removes the CPRTG module. (4) W/o CD disables CTG-based 3 decay. (5) Small Model utilizes DT backbone with fewer parameters. As shown in Figure (d), the extremely poor safety performance of W/o PE highlights the necessity of effective constraint identification, while the higher safety violations in Simp PE further underscore the effectiveness of our constraint prioritized prompt encoder design. The performance degradation observed in W/o CP and W/o CD emphasizes the critical roles of the CPRTG module and the CTG-based / decay mechanism in generating reliable RTGs that effectively balance reward and safety. Finally, the drop in safety performance seen in the Small Model reinforces the importance of model capacity, highlighting the necessity of using Transformers to ensure scalability. Additional results can be found in Appendix E:10 Appendix F.1
Related work
Safe RL.
Safe RL ensures policy deployment under safety constraints in addition to reward maximization, which is often formulated as constrained optimization problems (Garcıa & Fernández
2015 Wachi et al. 2024), and typically solved using Lagrangian multiplier methods (Wachi et al.
2024). These methods learn a cost value function and a parameterized multiplier, adjusting the multiplier based on cumulative cost to enforce safety (Chow et al., 2018, Stooke et al.) (0020) Tessler et al. 2019). However, they will involve unsafe real-world interactions during training, making them impractical. To address this, recent work focuses on offline safe RL, learning safe policies from pre-collected data to avoid unsafe exploration. These methods evaluate safety conservatively by treating dustt--iittiotttioo (OOD) samples as unsafe (Le et al. 2019: Xu al.) 2022a] /Zeng et al.)
2024 Yao et al. 2024). thereby reducing extrapolation errors (Fujimoto et al. 2019).

---

Transformers in RL. Transformers have shown impressive performance in complex sequential tasks like large language models (Zhao al. 2023). inspiring their use in offline RL. DT (Chen et al.) 202T), for example, employs GPT--lke Transformer (Achiam et al. 2023) with historical sequences and RTG tokens to predict optimal actions, breaking traditional RI. paradigms and circumventing extrapolation enrors directly. Extensions such as MGDT (Lee et al.) 2022) and Prompt-DT (Xu et al., 2022ch adapt DT to multi-task settings using visual inputs or expert trajectory prompts. For safe RL, CDT (Liu et al., 2023) introduces CTG tokens, while SDT (Guo et al. 2024 uses signal temporal logic tokens to encode richer safety information. However, these methods overlook the differing priorities between RTG and safety tokens, leaving the trade-off between reward and safety constraints unresolved, which motivates our work. More related work is discussed Appendix G
6 Final remarks
In this work, we propose CoPDT, novel algorithm for learning adaptable, multi-task offline safe policies within the DT architecture. CoPDT first introduces a constraint prioritized prompt encoder to utilize sparse binary cost information for accurate task identification under different constraints. Then, the CPRTG mechanism is designed to generate reasonable RTGs across varying budgets, effectively resolving conflicts between CTGs and RTGs. Extensive experiments across diverse multi-task settings demonstrate the superior adaptability and safety performance of CoPDT. Looking forward, scaling policy capacity and training across broader task distributions and larger datasets present a promising path toward enhancing knowledge transfer. Furthermore, achieving zero-cost exploration in safe RL using large language models (Wang et al., (2024) and deploying safe RL in embodied robots (Liu
et al. 2024a Feng al. 2025) also highly promising future research directions.
Acknowledgments
This work is supported by the AI & AI for Science Project of Nanjing University, the National Science Foundation of China 022495099,,25061535 U24A20324), the Natural Science Foundation of liangsu (BK20241 199, BK20243039). We thank the anonymous reviewers for their support and helpful discussions on improving the paper.
References
Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained policy optimization. In International Conference Machine Learning, pp. 22-31, 2017.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint Stti::2303.087744: 2023,
Alshickh, M., Bloem, R., Ehlers, R., Könighofer, B., Niekum, S., and Topeu, U. Safe reinforcement learning via shielding. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Brunke, L.., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S., Panerati, J., and Schoellig, A. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, 2022.
Robotics, and Autonomous Systems, 5(1 5(1)):11-444)
Chemingui, Y., Deshwal, A., Wei, H., Fern, A., and Doppa, J. Constraint-adappive policy switching for offline safe reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 15722-15730, 2025.
Chen, L., LU, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. Decision transformer: reinforcement learning via sequence modeling. In Proceedings of the 35th International Conference on Neural Information Processing Systems, pp. 15084-15097, 2021.
Cheng, R., Orosz, G., Murray, R. M., and Burdick, J. W. End-to-end safe reinforcement learning through barrier functions safety-critical continuous control tasks. Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 3387-3395, 2019.
10

---

Chow, Y., Ghavamzadeh. M., Janson, L.,, and Pavone, M. fisk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167):1-511 2018.
Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang. Y., and Yang. Y. Safe rlhf: Safe reinforcement learning from human feedback. In International Conference on Learning Representations, 2024.
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., and Hester, T. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 10(9))2419-2444 2021.
Eysenbach, B., Chaudhari, S., Asawa, S., Levine, S., and Salakhutdinov, R. Off-dynamics rein- forcement learning: Training for transfer with domain elassifiers. In International Conference on Learning Representations, 2021.
Feng, Z., Xue, R., Yuan, L., Yu, Y., Ding, N., Liu, M., Gao, B., Sun, J., and Wang. G. Multi-agent embodied ai: Advances and future directions. arXiv preprint urXiv:2505.05 2025.
Fujimoto, S., Meger, D., and Precup. D. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062 2019.
Ganai, M., Gong, Z., Yu, C., Herbert, S., and Gao, S. Iterative reachability estimation for safe reinforcement learning. Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 19764-69797 2023.
Garcia, J. and Fernández, F. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Gronauer, S. and Diepold, K. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review, 55(2):895 943, 2022.
Gu, S., Yang, L., Du, Y., Chen, G., Walter, F., Wang. J., and Knoll, A. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205. 10330, 2022.
Gu, S., Kuba, J. G., Chen. Y., Du, Y., Yang, L., Knoll, A., and Yang. Y. Safe multi-agent reinforcement learning for multi-robot control. Artificial Intelligence, 319:10 103905, 2023.
Guan, C., Xue, R., Zhang, Z., Li, Li, Y.C.,, Yuan, L., and Yu, Y. Cost-aware offline safe meta reinforcement learning with robust nn-distribution online task adaptation. In Proceedings of the 23rd International Conference Autonomous Agents and Multiagent Systems, pp. 743-751, 2024.
Guo, Z.,, Zhou, W., and Li, W. Temporal logic inneffccationccondttionned decision transformer for offline safe reinforcement learning. International Conference on Machine Learning, 2024.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pp. 1861-1870, 2018.
Hu, J., Wallis, P., Allen-Zhu Z., Li, Y., Wang, S., Wang, L., Chen, W., al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Huang, W., J., Xia, C., Zhang, B., and Yang, Safedreamer: Safe reinforcement learning with world models. In International Conference Learning Representations, 2024.
Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., and Pedrycz, W. A comprehensive survey on applications of transformers for deep learning tasks. Expert Systems with Applications, 241: 2024.
Janner, M., Fu, J., Zhang, M., and Levine, S. When to trust your model: model-based policy opti- mization. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 12519-12530, 2019.
Ji, J., Zhang. B., Zhou, J., Pan, X., Huang, W. Sun, R., Geng, Y., Zhong, Y., Dai, J., and Yang, Y. hameay-gymmasium: a unified safe reinforcemei learning benchmark. In Proceedings of the 37th International Conference Neural Information Processing Systems, pp. 18964-18993, 2023.
11

---

Kalyan, K. S., Rajasekharan, A., and Sangeetha, S. Ammus: A survey of transformer based pretrained models in natural language processing. arXiv preprint arXiv:2108.05542, 2021.
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., and Shah, M. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):1 2022.
Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. Morel: model-based offline rein- forcement learning. Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 21 21810-21823, 2020.
Kiran, R., Sobh, I., Talpaert, V., Mannion, P., Al Sallab, A., Yogamani, S., and Pérez, Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 2366):999--49288 2021.
Koirala, P. Jiang, 7., Sarkar, S., and Fleming, C. Latent safety-constrained policy approach for safe offline reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025.
Kostrikov, I., Nair, A., and Levine, Offline reinforcement learning with implicit q-learning. In International Conference Learning Representations, 2022.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative q-learning for offline reinforcement learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 1179-1191, 2020.
Le, H., Voloshin, C., and Yue, Y. Batch policy learning under constraints. In International Conference on Machine Learning, pp. 3703-3712, 2019.
Lee, K.-H., Nachum, O., Yang, S., Lee, L.,, Freeman, C. D., Guadarrama, S., Fischer, L,, Xu, W., Jang, E., Michalewski, H., et al. Multi-game decision transformers. In Advances Neural Information Processing Systems, 2022.
Li, L., Yang, R., and Luo, D. Focal: Efficient fully-offline meta-reinforcemmnt learning via dis- tance metric learning and behavior regularization. In International Conference Learning Representations, 2021.
Li, W., Luo, H., Lin, Z., Zhang, C., Lu, Z., and Ye, D. A survey transformers in reinforcement learning, arXiv preprint Stt::23001.3044. 2023.
Liu, Y., Chen, W., Bai, Y., Luo, Song, X., Jiang, K., Li, Z., Zhao, G., Lin, J., Li, G., al. Aligning cyber space with physical world: A comprehensive survey on embodied ai. arXiv preprint arXiv:2407.000 068866 2024a.
Liu, Z., Cen, Z., Isenbaev, V., Liu, W., Wu, S., Li, B., and Zhao, D. Constrained variational policy optimization for safe reinforcement learning. In International Conference Machine Learning, pp. 13644-13668, 2022.
Liu, Z., Guo, Z., Yao. Y., Cen, Z., Yu, W., Zhang, T., and Zhao, D. Constrained decision transformer for offline safe reinforcement learning In International Conference on Machine Learning, pp. 2161 1-16300, 2023.
Liu, Z., Guo, Z., Lin, H., Yao, Y., Zhu, J., Cen, Z., Hu, H., Yu, W., Zhang, T., Tan, J., and Zhao, D. Datasets and benchmarks for offline safe reinforcement learning. Journal of Data-centric Machine Learning Research, 2024b.
Luo, F.M.,, Xu, T., Cao, X., and Yu. Y. consis--o dynamics models are strongly general- izable for offline reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.
Nagabandi, A., Clavera, I., Liu, S., Fearing, R. S., Abbeel, P., Levine, S., and Finn, C. Learning to adapt dynamic, real- wol-world environments through meta-reinforcemeet learning. International Conference Learning Representations, 2019.
12

---

Prudencio, R. F., Maximo, M. R., and Colombini, E. L. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Rakelly, K., Zhou, A., Finn, C., Levine, S., and Quillen, D. Efficient oft-policy meta-reinforcement learning via probabilistic context variables. In Proceedings of the International Conference on Machine Learning, pp. 5331-5340, 2019.
Stooke, A., Achiam, J., and Abbeel, P. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp. 2020. 2020.
Tessler, C., Mankowitz, D. J., and Mannor, S. Reward constrained policy optimization. In Interna- lional Conference on Learning Representations, 2019.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-50263 2012.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, L.,, and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Wachi, A., Shen, X., and Sui, Y. A survey of constraint formulations in safe reinforcement learning. arXiv preprint arXiv:2402.02022 2024.
Wang, L., Ma, C., Feng, X., Zhang, Z.., Yang, H., Zhang, J., Chen, Z.,, Tang, J., Chen, X., Lin, Y., et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024.
Wu, Z., Tang, B., Lin, Q., Yu, C., Mao, S., Xie, Q., Wang, X., and Wang, D. Off-policy primal-dual safe reinforcement learning. In International Conference on Learning Representations, 2024.
Xiao, W., Lyu, Y., and Dolan, J. Model-based dynamic shielding for safe and efficient multi-agent reinforcement learning. In Proceedings of the 2023 International Conference Autonomous Agents and Multiagent Systems, pp. 1587-1596, 2023.
Xu, H., Zhan, X., and Zhu, X. Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8753-8760, 2022a.
Xu, M., Liu, Z., Huang, P., Ding, W. Cen, Z., Li, B., and Zhao, D. Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability. arXiv preprint arXiv:2209.008 08025, 2022b.
Xu, M., Shen, Y., Zhang, S., Lu, Y., Zhao, D., Tenenbaum, J., and Gan, C. Prompting decision transformer for few-shot policy generalization. In International Conference on Machine Learning, pp. 24631-24645 2022c.
Yang. S., Nachum, O., Du, Y., Wei, J., Abbecl, P., and Schuurmans, D. Foundation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04129, 2023,
Yao, Y., Liu, Z. Cen, Z., Zhu, J., Yu, W., Zhang, T., and Zhao, D. Constraint-conditoons policy optimization for versatile safe reinforcement learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 12555-12568. 2023.
Yao, Y., Cen, Z., Ding, W., Lin, H., Liu, S., Zhang, T., Yu, W., and Zhao, D. Oasis: Conditional distribution shaping for offline safe reinforcement learning. arXiv preprint arXwv.2407116533. 2024.
Yu, D., Ma, H., Li, S., and Chen, J. Reachability constrained reinforcement learning. In International conference on machine learning, pp. 25636-25655, 2022.
Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J., Levine, S., Finn, C.,, and Ma, T. Mopo: model- based offline policy optimization. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 14122, 2020.
13

---

Yuan, H. and Lu, Z. Robust task representations for offline meta- retffoeiiforemmmnt learning via contrastive learning. International Conference on Machine Learning, pp. 25747-25759 2022.
Zhai, J., Zhang, S., Chen, J., and He, Q. Autoencoder and its various variants. In 2018 IEEE 2018.
international conference on systems, man, and cybernetics, pp.
Zhang, L., Zhang, R., Wu, T., Weng, R., Han, M., and Zhao, Y. Safe reinforcement learning with stability guarantee for motion planning of autonomous vehicles. IEEE transactions on neural networks and learning systems, 32(12):5435- 55454-444444, 2021.
Zhang. Q. Zhang, L., Shen, L., Xu, H., Wang. B., Yuan, B.. Chang, Y., and Wang, X. Saformer: A conditional sequence modeling approach offline safe reinforcement learning. In International Conference on Learning Representations 2023 Workshop on Scene Representations for Autonomous Driving, 2023.
Zhang. X., Qiu, W., Li. Y-C.,, Yuan, L., Jia, C., Zhang, Z., and Yu, Debiased offline representation learning for fast online adaptation in non- non-stationary dynamics. In International Conference on Machine Learning, 2024.
Zhang, and Yang, Q. A survey on multi-task learning. IEEE transactions on knowledge and data engineering, 34( 12):55866668- 5609, 2021.
Zhao, W. X., Zhou, K., Li. J., Tang, T., Wang. X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303. 2.33::2333.18233, 2023.
Zheng, Y., J., Yu, D., Yang, Y., Li, S. E., Zhan, X., and Liu, J. Safe offline reinforcement learning with feasibility-guidee diffusion model. In The Twelfth International Conference on Learning Representations, 2024.
Zhu, Z., Lin, K., Jain, A. K., and Zhou, Transfer learning deep reinforcement learning: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We provide discussion on limitations in Appendix H
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
14

---

The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-texl system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
The theoretical result and proofs are provided in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Detailed algorithms and hyperparameters provided in Appendix and Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
15

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The data and code are provided in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ htlliii/ggiess//e (CodeSSbmissiooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips.cc cc/ http://lll/ggg..eee/eeeeeeueeeee http:///otteuubmsssooooooooo.ooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
16

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Detailed algorithms and hyperparameters provided in Appendix and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The standard deviation of the main experiment is provided in Appendix F.15
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma eтror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The time complexity analysis is provided in Appendix F
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
17

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide discussion societal impacts in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
18

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators original owners of assets used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pataasess.paperswtt paperswi thcode. cow/coo////ooooo /atasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
19

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy /ettps://neurr cc/ //onfffrencees22255555 for what should or should be described.
20

---

A Additional interpretations of CPRTG from the perspective of offline RL
A fundamental challenge in offline RL lies in mitigating extrapolation errors that arise when the policy encounters out-of-distribution (OOD) regions (Prudencio et al.] 2023). While DT partially address this issue by constraining policy learning to the support the offline dataset through supervised learning, they remain vulnerable to OOD generalization when RTG values are treated as also part of the input state. In particular, RTG values that deviate from those observed in the offline data can still induce distributional shift, leading to unsafe or suboptimal behavior. The CPRTG mechanism addresses this by generating RTG targets that remain within the empirical support of the offline dataset, thereby reducing extrapolation risk and enhancing safety. From this perspective, we can make some theoretical analysis about the advantage of using CPRTG.
Lemma A.I. (Jamner el al. 2019) Suppose we have two distributions p1(æ, y) = (z) (x)))())a) and v) = P2
We can bound the total variation distance (TVD) of the joint as
Div(P1 (I, (I, (p (2 u)) S (rv(p(())lppd) + Ezwpi [Dr((p (vl))
(14)
Lemma A.2. (Janner et al. 2019) Suppose the expected TVD between two dynam- ics distributions is bounded as max; Bawpi (Dr(pP (8')s, a) (P2(s' (s, a))] S C.. and maxs Drv(m (12((s)))) S em, where pi (8) is the state distribution of Ty under dynamics p1(8 (811,, a). Then the returns are bounded as:
In - 72 S 2Bmas + mm)
2
+
(15)
(1 (1-7)²
1-9
where тą is the expected reward return under Ty and ps, Ty is the shared discount factor and Rmax is the maximum possible reward.
The two Lemmas mentioned above, although both derived from MBPO (Janner al., 2019), do not require any additional assumptions for the policy training process (such as being model-based or online training). They only measure the performance of the policy during deployment. Therefore, we can shift our perspective and treat the CTG and RTG distributions from offline data as the true distributions given by the environment. In this context, the policy during deployment can be viewed as being rolled out in learned model of that environment. This environment model learns state transitions accurately, but the CTG and RTG transitions are not perfectly accurate. According to Lemma A.2, we can conclude that as the CTG and RTG transitions become more aligned with their true distributions, the return obtained by the policy in the learned model will more closely resemble the return in the real environment, which means, in our case, the return during policy deployment will be more similar to the return observed in the offline data. (Under the following thcorem and proof, we ignore the past trajectory -- for simplification. After incorporating T-Kit, it can similarly be regarded as part of the state. As a result, the proof remains unchanged- modification simply introduces T- Ka an additional condition in all probability terms.)
Theorem A.3. Suppose the transition distribution of CTG given the next state dur- ing deployment is Pi (Cr+1166', s, Re, Cr,a), a), and that induced from the offline dataset is p2(Cr-1l'', 8, Re, Cr, The transition distribution of RTG given the next state and next CTG during deployment is p1( P1(Retal (RealCCe41,s s',, Re, Ci, a), and that induced from the offline dataset is P2(4411 Citi,s', s', RG, a), Let the TVD between the CTG transition distribution D Div(pi (Cr+ils', 8, Re, Ce, a))lpz(Cr-11 ss, Re, a)) be IV(C,t), and the TVD between the Re4alCe41, s', ,,, Re, Ci, a)) be
RTG transition distribution Drv(pI (Re+i ( C44,, Re, TV( t). If
max E spep (VCC,, SEc.
[V(C,
(16)
max E
TV(R, ()] 1 SE ER.
(17)
then we have
2Rmax + cc + tEn
2Rmar
m
1/2
R
71-
1-7
(18)
27(Er t + +ER
2
c
< 1/2
+
+
(19)
(1 7)-
1-7'
21

---

where pi (s) is the state distribution of the learned DT policy in timestep t, pi(-ls, Rd, Ct a) is the dynamics transition distribution of the target task, 77 7h are the expected reward return and cost return for the learned DT policy during deployment, and np 12 is the expected reward return and cost return for the behavior policy under the state, CTG, RTG transition induced from the dataset.
Proof. We view RTG RG and CTG O₁ from a different perspective, rather than the condition, but part of the state. Then, we take the state, RTG and CTG transition distribution induced from the offline dataset p2(s'. Re-1; Ci+1|s, Re, Cr a) as the ground truth transition distribution, but the state, RTG and CTG transition distribution during deployment as the environment model transition.
First, applying Bayes rule we have
pi(s', Reti, Cit1l0, Ru, Cr,a) = ppi(s'ss, Re, Cr p)p s', Rv, Co (ppp((Be C s', s, Rs, C₁ a),
(20)
i 1, and (s Rr, a) (s ss Re, Cr a) due to the same state transition distribution. Therefore, apply Lemma A. we can obtain
Drv(pi(s', Reti, Citils, Rr. a))a))22(( R Citals, Re, Cr,a))
Div(pi(s' RG, Cr ail (2 (5'1s Rr, Cc,
$ S Bl C4 8, Rí, Cr a)llp (Re Citils, 8, Ru, Co a)) Rr, Co a) (Ci Rs, Cr,
B R, C, D
8 Re, Co a) |p2 [2e Rr, Cr a)]]]
(21)
Since
max L Bnp (VCC,) EC,
(22)
max E (ww. (T((R,)) S eR.
(23)
and thus
max Bawp((s) (Drv(pns REIL, Cit1ls, Re, Cr a)llpz(s', Reti, Cetils, Re, Cr. a))
max E max
(V(C,
(24)
TV(R,
<Ec +
Therefore, treat pils', RetI, Crtil0, Re, Cr a) as the state transition pi(5'18,,) in Lemma A.2 we further obtain
1n/2 -1/ 2Rmax + + EER)
2Rmaxe
+
(25)
7)2
1-7
27(ex + 60 + +(0)
20
Inf 1 5
+
(26)
71
1-7
Finally, we have
2Reax 7(5 + + +ER)
2Rmax
1/1 1/2
R R
(27)
(1-
7
27(er +e0 + +ER)
2
C <
1/2
C
+
7)-
+
$1 1-
(28)
Theorem A.3
provides upper bound on the performance gap between the DT policy during deployment and the offline data-driven behavior policy. This gap is primarily influenced by three factors: Eg. 6c, and Eg. En is mainly determined by the degree of optimization of the policy loss
22

---

function, which is difficult to alter. As for ec, we rely on the generalization ability of the Transformer for the CTG to adapt to different safety thresholds, and thus, we do not wish to modify the initial settings or update method of the CTG. Therefore, a natural approach to improving the lower bound of policy performance is to reduce the value of en- In this context, the CPRTG generator CoPDT can be viewed as a neural network approximation of p2 (ft++ (C ( 8, Re, Ce a), which helps to lower é during deployment.
Future research could also explore addressing OOD CTG values, for instance, by mapping large initial CTGs (those exceeding the maximum in the offline dataset) to the dataset's maximum, thus ensuring safety while further mitigating OOD-related extrapolation errors.
B Practical algorithms
In this part, we will offer the detailed algorithms of CoPDT in multi-constraint scenarios. As described in the main paper, the workflow of CoPDT primarily includes four parts: do training, Pe training, policy training, and policy deployment. For ds and Pe (Dz, Pa), we both use simple multi-layer perceptron (MLP) networks, and additional prompt embeddings. environment IDs, and previous K-step trajectories will be used as inputs for qo:
min ET (T) E Ka... C. log 90 (Rec St.Pe("") O]]. 4o
(29)
where T- is concatenated into single vector, and is used to improve the accuracy of distribution fitting and maintain consisteney between the inputs of qs and πī.. For computational efficiency, we adopt a lightweight MLP architecture here. However, a Transformer-bate based model, similar to the one used in the DT policy, can also be employed to potentially enhance performance. Exploring such a design remains an interesting direction for future research.
The use of ps and p., in Pa is similar to traditional context-based meta RL (Rakelly al., 2019 let al. [0221; Yuan & Lu, 2022). Assume that the safe patch, classified using cost information, is represented as [**,, az, Vx into a single vector Ti.- Then, T2 is passed through the MLP neural network ps to obtain output vector 2g. Consequently. we obtain T, output vectors for the safe patch. Similarly, we can obtain Tr output vectors for the unsafe patch. By averaging these T, + Tw output vectors, we obtain the final prompt embedding z. During the training of Pe, the prompt embedding is further input into MLP networks fs. fr. and fe to attempt to predict the corresponding s'. r. and e values based on the given (s, a) information (f., fr. and fo are decoupled from the DT policy). The prediction is then used to compute regression loss, which allows gradients to be backpropagated into pa and Pu. The relationship between pa. and Ss. fr. Se. is essentially that of the encoder and decoder in traditional autoencoder (Zhai et al. 2018). They are trained jointly before DT training, but only the frozen encoder (not updated with DT) is required for the DT policy training and deployment phase.
T } For each sample (st, at, r²) within this patch, we first concatenate it
Detailed pseudo-codes for ds training, Pe training, and policy training are provided in Algorithm while the pscudo-codes for policy deployment are provided in Algorithm
C Distinguish constraints in unknown environments
In this section, will introduce the task (constraint) identification method when the environment ID is unknown. Based on the definition in Section 2.1 an environment is determined by its state space, action space, and dynamics transition. Therefore, on this information. First, we filter out potential candidate environments from the previously seen environments based on the state space and action space dimensions of the unknown environment. encoders, environment-speciin decoders, and the
need infer the true environment ID based
Then, we sequentially use the aevvrooment-speccinnn inverse dynamics model from the candidate environments to test the given trajectory in the unknown environment. Specifically, given the trajectory (se, dr; s/);" and the set {Ex,is Ca,;; da da gi t=1 of all N' candidate environments, the objective is as follows:
T
min T
((d. sx)² (ai(e,)()))) (.)())) -aa))) (30)
23

---

Algorithm 1 CoPDT Training
task set {T, f=1 (T)})= trajectory length K, hyperparameters Ano Ae- Initialize: constraint prioritized prompt encoder Pe, decoders fr. fer state action encoders and decoders for each environment {ea,f, ca da d gs | CPRTG generator qo- DT policy my.
Input:
environment set {Ei offline dataset for each task (DT, } DT
for step in aavironment-specifie training steps do for each environment & do
Merge each task dataset in this environment to get the environment dataset D; Sample batch {(t, at, s
Update ca,; and da,; with Equation (3)
Update Ca,i da,i and 9i with Equation end for end for
for step in prompt encoder training steps do for each task T with its environment ID do
Sample batch {(T" St, d2, Te, ct] from D7. Encode states sampled with Cx2i and actions sampled with Ca.i- Update Pe: Ss. fr, fo with Equation (6).
end for end for
for step in policy training steps do for each task T with its environment ID i do
Sample a batch {(-", Teee; Co Rg 8t, a2 ce) from Dr-
Update अe with Equation (12
Update with Equation 22
end for end for Return (ce,s, Ca.i L.
Pes 9p. To.
Algorithm 2 CoPDT Deployment
Input: initial CTG Ö₁ environment ID i, constraint prioritized prompt encoder Pe, state action
encoders C,.s, Co.is CPRTG generator ge, DT policy To. expert trajectory r", DT trajectory length K, hyperparameters X, Sstatt Pend. Initialize: input sequence т = [.
Encode states and actions in 7" with and Ca.i- Compute the prompt encoding according to Equation (5) for t=I.....TT .T do
Observe current state 5t. Compute 3 according to Equation
10
Sample X values in distribution qs ( K :). C. $1, 2) and select the Bu-quantile of it R.-
Sample action a from (o.a(-17[-K Gt Ri, 8t; 2, Step action 04 in the task environment to get rt, Ot-
Compute Cr+1 = C₂ Ct. Append (C,, R4, St, at) to r.
end for
24

---

(a) Navigation Tasks
(b) Velocity Tasks
Figure 5: Tasks used in this paper. (a) Navigation Tasks based on Point and Car robots. (b) Velocity
Tasks based on Ant, HalfCheetah, Hopper, Walker2d, and Swimmer robots.
Table 2: Detailed velocity thresholds for new designed tasks.
Tasks
Velocity Threshold
AntVelocity
2.52
MallCCheetahVelociiy V2
3.05
HopperVclocityV2
0,56
SwimmerVelocity-V V
0.18
Walker2ddeelcctyy22
2.00
where the first term is the action reconstruction loss, which aims to ensure consistency the action consistency in the state space; the second term the state reconstruction loss, which aims consistency in the dynamics space; and the third term the inverse dynamics error, which ensures transition. Once the environment ID is determined, revert the previous setup, where the trajectory is passed through the environment-specife state encoder, environment-speciff action encoder, and the constraint prioritized prompt encoder obtain the prompt encoding, which serves as the basis for task identification.
D Detailed description of the tasks and baselines
D.1 Tasks and Datasets
All pretraining tasks used this paper are derived from the Safety-Gymmasiuun' Navigation Tasks and Velocity Tasks. In the Navigation Tasks, there are two different types of robots: Point and Car, which we need to control to navigate through the environment and earn rewards by reaching target points, pressing the correct buttons, moving in designated directions. Different tasks also have varying costs, such as avoiding collisions with specific targets, preventing incorrect button presses, and staying within designated boundaries.
The Velocity Tasks are built on traditional MuJoCo (Todorov et al.) 2012) simulations, requiring robots such as Ant, HalfCheetah, Swimmer, and Walker2d to move, where higher speeds result in higher rewards. However, each robot has specific safety velocity thresholds for different tasks, and exceeding these thresholds leads to unsafe states. For detailed descriptions of each task, refer to the original
paper (Ji et al. 2023). Besides the mentioned tasks, we designed five new Velocity tasks for task transfer, which differ from previous ones only in their velocity thresholds, as detailed in Table 2
The offline datasets used for each task are sourced from OSRL (Liu et al. 2024b). Specifically, the datasets for the Velocity and Velocity tasks were additionally collected using OSRL's original data collection methods.
D.2 Baselines
We provide more detailed introduction to the basclines of the experiment in this section.
25

---

CPQ is MMDP-based offline safe RL algorithm built upon the classic offline RL method CQL (Kumar et al., 2020). It incorporates the conservative regularization operator from CQL into the cost critic, treating out-of-ddisrributio samples as unsafe. Unlike traditional methods that use Lagrangian multipliers for policy updates, CPQ directly truncates the reward critic to 0 for unsafe state-action pains, preventing unsafe policy execution. This algorithm has become one the most common baselines in offline safe algorithms and is considered -aate-of-the-all in CMDP-based offline safe algorithms.
CDT is the previous SOTA algorithm under sequence modeling for offline safe RL. After incorporating CTG into DT, CDT also seeks to address the conflict between safety constraints and reward maximization. To tackle this issue, CDT proposes a data augmentation approach, where reward returns for certain safe but low-reward trajectories in the offline dataset are re-labeled with higher values. However, the effectiveness of this data augmentation method is still limited by the amount augmented data and does not fundamentally resolve the underlying issue. In experiments, all CDT-based methods utilize manually specified initial RTG values for different constraints and budgets, as provided by the original OSRL implementation (authored by the same team behind CDT)
FISOR is the SOTA offline safe RL algorithm based hard constraint modeling. Similar to other algorithms under hard constraints Yu et al. (2022); Ganai et al. (2023). it first divides the state space into feasible and infeasible regions. It then uses the Kostriky et al.) (2022) algorithm to learn feasible value functions for offline feasible region identification. Next, FISOR sets distinct learning objectives for the feasible and infeasible regions. In the feasible region, it aims to maximize the reward while ensuring feasibility, while the infeasible region, it focuses minimizing constraint violations. Finally, FISOR employs diffusion model to represent and learn the policy. In the experiments, although the policy learning of FISOR independent of the safety thresholds, operating under fully hard-constrained setting, all cost results are still normalized using four safety thresholds: [10, 20, 40, 80).
LSPC is a recently proposed hard-constrained offline safe RL algorithm. Similar to FISOR, it employs IQL to learn feasible value functions. Once the feasible value functions are obtained, it minimizes the feasible value as a weighting to train the decoder of a CVAE. The trained CVAE decoder can then be regarded as policy that minimizes safety violations, referred to as LSPC-S. Furthermore, an additional policy (2]ss) is trained to generate the latent encoding imput for the CVAE decoder. where is optimized maximize the expected reward. Finally, hyperparameter is introduced to control the optimization strength of æ, balancing safety satisfaction and reward maximization. This variant referred as LSPC-O. In experiments, we adopt LSPC-O, which provides a balanced trade-off between safety and reward, as the baseline, and evaluate using the same protocol as FISOR.
MTCDT a straightforward multi-task extension CDT. It addresses the varying state and action dimensions in cross coosseenvionment tasks by utilizing distinct input and output heads for each environment. Additionally, MTCDT attempts to identify different tasks based solely on the sequential inputs of DT for multi-task decision-making.
Prompt-CDT is also multi-task extension of CDT. Building on MTCDT, Prompt-CIT utilizes additional expert trajectory segments as prompts assist task identification.
E Hyperparameters
The training and deployment of CoPDT both involve the selection of hyperparameters. To ensure reproducibility, this section outlines the specific hyperparameters used in our experiments. CoPDT is implemented based on CDT within the OSRL framework, and the default parameters are retained for any hyperparameters not explicitly mentioned in Table 3
26

---

Table 3: Hyperparameter choices of CoPDT.
Hyperparameter
Value
state encode dim action encode dim
32 2
all network hidden layers
[128, 128, 128
environment-speciine encoders
update steps batch size
100000
2048
learning rate
0.0001
prompt encode dim all encoder hidden layers
16 [256, 256, 256]
all decoder hidden layers
[128, 128]
prompt encoder Pe
update steps batch size
100000
2048
learning rate
0.0001
environmennt-peccffee state input head output dim environment-specifee state input head hidden layers
64
[128, 128, 128
generator hidden layers
[1024, 256, 1 128]
RTG generator q
update steps batch size
100000
2048
learning rate
0.0001
DT embedding dim
512
DT num layers
3
DT num heads
8
DT sequence len
20
environment -eroonimett-peccffi state input head output dim cetroomment-specifn state input head hidden layers environment speccffc action input head output dim environment-speccffc action input head hidden layers environment--pecciicc action output head input dim environment-specife action output head hidden layers update steps batch size learning
64
[128, 128, 128)
32
[128, 128, 128]
32
policy learning and deployment
[128, 128, 128]
200000
1024
0.0001
A/
0.1
X
0.02
Saut
0.99
Aend
0.8
X
1000
Table 4: Time complexity comparison.
CoPDT (Single-constraint)
CDT
CoPDT (Multi-constraint)
MTCDT
Prompt-CDT
Prompt Encoder Training
1.330
Policy Training
15.737
CPRTG Generator Training
0.250
1,404
Depleyment
0.012 sistep
0.008 s/step
0.017 s/step
0.008 s/step
0.014 s/step
F More experimental results
F.1
Time complexity analysis
Time complexity during policy training and deployment is a critical issue real-world applications. Therefore, in this section, provide the analysis of CoPDT's time complexity. Due the use of neural networks, it is not feasible to provide quantitative analysis. Instead, we first compare CoPDT qualitatively with other baseline algorithms and present the specific results through actual data.
27

---

CoPDT
CDT
Real-Targer
PolniButton1
PointCircle1
PointGoall
PointPush1
Real Cost
00
Rewarr
06
0.4
02
00
10
10
Target Cost
Target Cost
Target Cost
Target Cost
Figure 6: Performance of CoPDT and CDT under various safety thresholds. Target costs and real costs are normalized by 10.
First, during the training process in the Single-constrane setting (no need for constraint identification), the policy training for CoPDT and CDT is identical, with the only difference being in the CPRTG generator's training. Since we use a traditional MLP neural network in the CPRTG, its computational complexity is much lower than that of the large Transformer networks used in policy training, resulting in minimal additional overhead for CoPDT during training. In the Multi-constraint setting, CoPDT, compared to MTCDT and Prompt-CDT, involves not only the CPRTG generator but also the training of the prompt encoder. This training includes both environment-specii and constraint prioritized prompt encoder training. Similar CPRTG training, the prompt encoder training only involves MLP networks, so it does not introduce significant additional overhead. The In the Single-constraint scenario, we use the training of fixed
encoder training
specific data is shown in Table 4 task as the result, and in the Multi-constraio scenario, we report the average training time per task (constraint). During deployment, both Single-constraint and Multi-constraint scenarios use the same task for testing. All experiments were conducted on a single NVIDIA GeForce RTX 4090, and fairness was ensured even the presence of CPU resource contention. The results the table align with our earlier analysis, showing that the primary overhead during training comes from policy training. with the additional MLP network training overhead being minimal. In contrast, Prompt-CD' incurs higher training overhead due to the use of sequence-based prompts.
In the policy deployment process, the additional time complexity for CoPDT mainly arises from the use of the CPRTG generator. By observing the last row of Table 4 we can see that in the Single-constraint setting, the use of the CPRTG generator does introduce some extra overhead. In the Multi-constraiie setting, the additional overhead increases, as the use of prompts adds computational complexity both during policy inference and in the CPRTG generator, but the additional overhead is still within an acceptable range (smaller than 0.01 second) for real-world deployment. Further analysis of the additional time complexity introduced by the CPRTG generator is provided in Appendix F.10
F.2 More results on multi-budget decision making
In this section, we provide more results on the adaptation capability of CoPDT and CDT to different safety thresholds, shown in F It shown that CoPDT is able effectively adapt under any safety threshold, ensuring the safety performance of the policy. At the same time, both its cost and reward exhibit clear increasing trend as the safety threshold rises. Although CDT shows some advantages in terms of reward, it clearly lags behind in terms of safety. In three of the environments, fails demonstrate adaptability different safety thresholds, resulting poor safety performance when the safety threshold is low. These experimental results validate the strong adaptation capability of CoPDT across different safety thresholds.
F33 Deployment phase conservatism adjustment with different Send
28

---

Bend serves critical hyperparameter that controls the conservatism of CPRTG generation. To evaluate its impact on policy performance, we conduct sensi- evident
Performance with different Bend
1.00
1.50
cost
tivity analysis, as illustrated in F that as Pnd increases, both the reward and cost of the policy also gradually rise, indicating that CoPDT can effectively modulate the level of conservatism in ceiision-making through tuning Send, demonstrating its flexibility and adaptability across different safety- reward trade-offs.
0.22%
1.44
Reward
0.994
0.36
0.34 1.20
1.38
1.32
Based on the previous analysis, the conservatism of CoPDT can be flexibly adjusted by modifying Send. To further explore this adaptability, we intro- Figure 7: Parameter sensitivity study Bend- duce new experimental setting within the Single- Constraint Multi-Budget setting, referred to as Oracle.
Bend
In this setting, algorithms are allowed to fine-tune hyperparameters for each task with different con- straints and even across varying budgets within the same constraint to achieve optimal performance. The following algorithms are evaluated under this setting: (1) BC-Safe: For each constraint and budget, trajectories from the offline dataset that satisfy the corresponding budget filtered, and behavior cloning is performed based on the filtered safe trajectories. (2) FISOR:
Separate reverse- expectile values are used for tasks with different constraints. (3) CoPDT: Different Pend values applied across tasks with varying constraints.
The experimental results, summarized in Table 5 highlight the strengths and weaknesses of the three algorithms. BC-Safe demonstrates superior average performance but satisfies the safety constraints in the fewest number of tasks. FISOR, on the other hand, ensures compliance with safety constraints across all tasks but shows limited ability to optimize rewards under safety conditions. CoPDT strikes a balance: while it fails to meet safety constraints in small number of tasks, it exhibits strong reward optimization capabilities under safety constraints, offering greater flexibility. Additionally, a unique advantage of CoPDT over the other two algorithms lies in its ability to adjust conservatism during the deployment phase. Unlike BC-Safe and FISOR, CoPDT allows for adjustments to Send without requiring policy retraining, enabling efficient and flexible tuning.
E4 Analysis stitching ability
In this section, we compare the stitching ability of CoPDT and CDT. Specifically, we construct challenging test setting by removing all trajectories with cumulative cost < 10 from the offline datasets of eight Point tasks, and evaluate the policies under a safety budget of 10. As shown in Table CDT fails to learn safe policies in this setting, exhibiting significant safety violations due the lack of safe trajectories in the offline data. contrast, CoPDT' successfully learns safety-satisfiing policies in more than half of the tasks. These results demonstrate that the CPRTG module in CoPDT enables effective trajectory stitching beyond simple behavior cloning, facilitating safe decision even in the absence of directly safe trajectories.
F.5 Visualization of CPRTGs
In this section, we visualize how R₁ changes in response to variations in C₁ and 8t, as shown in Figure 8 The results indicate that R, increases significantly with higher values of Ć₁ and also rises notably as 3e increases. This confirms the rationale behind modeling RTG under CTG conditions and applying decay to 8,, allowing the policy to gradually adjust its conservatism based on potential future safety violations.
F.6 Visualization of prompt encodings
In this section, we additionally explored the encoding process and properties of the constraint priori- tized prompt encoder. First, we visualized the state distributions after encoding each environment's state separately using the environment-specifee state encoder (Figure 9(a)) The results reveal clear
separations between different environments, though tasks within the same environment remain in- distinguishable. Next, visualized the results after applying the prompt encoder (Figure 9(b). At
29

---

Table 5: Overall performance in the Oracle setting.
Oracle FISOR
Task
BC-Safe
CoPDT
rr
c/
r
c.p
r
c|
PointButton
0.04
0.74
-0.01
0.28
0.09
0.91
PointButton2
0.15
1.75
0.05
0.43
0.08
0.92
PointCirclel
0.38
0.16
0.05
0.06
0.54
0.62
PointCircle2
0.45
0.99
0.20
0.00
0.61
0.98
PointGoall
0.38
0.53
0.03
0.01
0.51
0.87
PointGoal2
0.29
1.13
0.05
0.08
0.29
0.91
PointPushl
0.13
0.67
0.31
0.89
0.19
0.88
PointPush2
0.13
1.05
0.09
0.29
0.13
0.63
CarButtonl
0.07
0.87
-0.02
0.78
0.07
0.74
CarButton2
-0.03
1.25
0.02
0.40
-0.02
0.89
CarCirclel
0.29
1.66
0.21
0.24
0.49
2.96
CarCircle2
0.51
5.17
0.40
0.42
0.28
0.98
CarGoal1
0.28
0.39
0.43
0.72
0.39
0.75
CarGoal2
0.14
0.57
0.07
0.27
0.19
0.81
CarPush1
0.15
0.45
0.25
0.43
0.28
0.96
CarPush2
0.05
0.63
0.13
0.59
0.09
0.88
Swimmer SeimmerVelocity VO
0.52
0.08
-0.04
0.31
0.62
0.98
Swimmer Velocity V1
0.50
0.63
-0.04
0.14
0.44
0.87
VepoeiteelcityVV VO
0.50
0.25
0.30
0.23
0.18
0.52
HopperVelocire V1
0.42
0.65
0.16
0.86
0.18
0.86
HalfCheetah Velocity VO
0.92
1.11
0.89
0.00
0.67
0.38
MalCCheetahVelociity V1
0.89
0.75
0.89
0.00
0.84
1.00
Walker2dVelocity VO
0.24
1.45
0.05
0.12
0.32
2.90
Walker2dVelocity
0.79
0.01
0.53
0.80
0.78
0.12
AntVelocityVVi
0.86
0.61
0.77
0.00
0.90
0.84
AntVelocityVI
0.96
0.38
0.89
0.00
0.97
1.58
Average
0.39
0.92
0.25
0.32
0.39
0.99
Table 6: Stitching ability analysis results.
CDT
CoPDT
Task
rt
cl
rf
cl
PointButton
0.56
12.66
0.01
0.88
PointButton2
0.50
10.28
0.04
1.49
PointCircle1
0.53
0.91
0.51
0.61
PointCirele2
0.59
1.79
0.59
1.52
PointGoall
0.74
4.65
0.25
0.98
PointGoal2
0.69
13.53
0.22
1.37
PointPushl
0.27
4.16
0.13
0.90
PointPush2
0.16
5.25
0.09
0.87
Average
0.51
6.65
0.23
1.08
this stage, tasks within the same environment also successfully differentiated, confirming the effectiveness of our design and loss function selection.
Next, we aim to further explore the properties of the constraint prioritized prompt encoder, specifically whether it focuses more on differences rather than on the state-action distribution itself. To test this, we specifically selected tasks where the cost function significantly affects the state-action distribution (e.g., PointCirclel and PointCircle2).
acte-action distribution driven by varying cost information,
Using the policy of PointCirclel, we collected data in both tasks to ensure consistency in state-action distribution for the prompts. The results, as shown Figure and Figure [d]] reveal that the
constraint prioritized prompt encoder effectively distinguishes tasks based cost information, while
30

---

Pointumon1
Anticloocty
CTO
(a) PointButton
(b) AntVO
Figure 8: Generated RG under different C. and Be.
(a)
(b)
(c)
(d)
Figure 9: Different visualization results. (a) Visualization of state encodings after using the environment-speciff state encoders. (b) Visualization of prompt encodings of the constraint priori- tized prompt encoder using expert trajectories as prompts. (c) Visualization of prompt encodings of the constraint prioritized prompt encoder using trajectories collected
behavior policy for PointCircle1 and PointCircle2. (d) Visualization of prompt encodings of the simple MIP encoder using trajectories collected by same behavior policy for PointCirclel and PointCircle2.
the traditional MIP encoder suffers from task confusion due to the similar state-action distribution. This highlights the robust cost information extraction ability of the constraint prioritized prompt encoder.
F.7 More policy transfer results
In this section, first provide transfer results more tasks, as shown in Figure 10 In four out the five tasks, although the CoPDT pretrained model can't guarantee zero-shot safety, fine-tuning with FFT or LoRA shows clear advantages both safety performance and reward compared training from scratch. In one environment, the pretrained model ensures zero-shot safety, but fine-tuning results in a performance drop, likely due insufficient trajectory coverage. Comparing FFT and LoRA, both perform similarly, possibly due to the relatively small model size, with LoRA offering lower fine-tuning overhead. Based on the above results, it can be found that the pretraining of CoPDT can effectively improve the policy's adaptation ability in new tasks, enhancing applicability.
Then, to demonstrate that our prompt encoder design can include more effective information than directly inputting sequence prompts during task transfer, we compare the results with Prompt-CDT under two fine-tuning methods: FFT and LoRA. Additionally, we introduce new environment, AntCircle, which has a lower similarity to the pretraining task. In AntCircle, the state space, action space, and dynamics transition are consistent with AntVelocity, but both the reward function and cost function undergo significant changes. The reward function is modified to represent the speed at which the Ant robot moves along a circle, while the cost function now penalizes the robot's x-coordinate rather than its speed. The results are shown in Figure I1] First, by observing the results in similar tasks, we see that Prompt-CDT's multitask pretraining also provides certain level improvement in task transfer in environments other than HalfCheetah. However, compared to CoPDT, Prompt-CDT still exhibits inferior task transfer performance. In HalfCheetah, leads to a significantly higher violation of safety constraints. In Hopper, FFT shows noticeable drop in performance, while LoRA
31

---

from scratch
FFT
LoRA
bound
AntV2
HallChectahi22
HopperV/2
SwimmerV2
Walker2iV2
1.00
1.00
0.59
0.00
Reward
0.00
0.93
0.00
0.00
0.00
0.40
0.40
0.90
0.20
2.6
100
2.6
6.0
10.0
0.5
2.6
6.0
7.6
10.0
0.0
9.5
7.6
13.0
50
25
5.0
100
1.8
10.0
12.5
12.0
Cast
2/3
10.0
20
5.5
5.0
0.2
0.0
2.5
100
5.0
10.0
0.2
13.0
100
Step (1e4)
Step (le44
Step (1e4)
Step (1e4)
Step (1e4)
Figure 10: The transfer results of CoPDT in 5 new tasks.
from, scratch
CoPDT_LoRA LooR
Prompt CDT CDT_LoRR LoRA
CaPDT_FF
Prompt-CDT_ FFT
cost bound
AntV2
HalKCheetahV2
HopperV2
AntCirdeev
1.00
1.50
Reward
0.40
0.80
0.80
0.59
0.69
0.20
0.00
200
6.0
20.0
Coot
150
100
5.0
0.5
0.5
Step (1e4)
Step (1e4)
Step (1e4)
Step (1ee4
Figure 11: Transfer comparison with Prompt-CDT and transfer results in dissimilar task.
causes instability in safety. This clearly indicates that CoPDT's use of the prompt encoder provides more effective information for knowledge transfer than directly using sequence prompts.
Next, by observing the results in dissimilar tasks, we find that even in scenarios with low task similarity, CoPDT's pretraining still provides some performance improvement compared to learning from scratch. However, due to the limited amount of transferable knowledge, this improvement is less significant than similar tasks. In contrast, Prompt-CDTTs pretraining results noticeable decline in safety performance. This indicates that the sequential prompts used in Prompt-CDT do not always bring additional information gain and may sometimes interfere with the extraction of effective information. Furthermore, these results suggest that in low-similarity scenarios, few-shot adaptation may not always yield stable results, and using larger datasets for training might be better alternative. Additionally, increasing the diversity of tasks during pretraining is an effective way to enhance the policy's 's transferability.
Finally, we further evaluated transfer performance on two unseen robots, Ball and Drone, using 10 expert trajectories with a safety budget of 10. Due to the differences in state and action spaces, we first trained new environment-specifee encoders for these four tasks, followed by fine-tuning the rest of the CoPDT architecture. As shown Figure although the parameter-efficient LoRA fine-tuning method performs suboptimally, FFT consistently outperforms training from scratch across all tasks
32

---

from scratch
FFT
LORA
cost bound
BallRun
BallCiicle
DroneRun
DroneCircle
1.50
0.36
0.80
2.0
Reward
1.00
0.00
0.5
0.00
2.0
0.40
0.90
0.90
0.25
0.0
2:00
10
10
10
7
10
32.0
Coet
4.0
4.
30
Step (1e4)
Step (1e4)
Step (1e4)
Step (1e4)
Figure 12: The transfer results of CoPDT to unseen robots.
in terms safe decision-making makinn.. These results further validate that CoPDT's multi-task learning framework enables effective knowledge transfer to unseen tasks under limited data.
F8 Discussion on zero-shot constraint generalization
This section gives the detailed generalization results in Figure 4 (a) of the main paper, shown in T The results reveal that while CoPDT demonstrates better zero-shot performance compared to Prompt-CDT, it still exhibits substantial safety violations, making it unsuitable for zero-shot deployment. We attribute this primarily to the insufficient number similar tasks encountered during multi-task training, which fails to support the emergence of robust zero-shot generalization. This finding suggests that in typical decision-mate making scenarios with limited tasks and data, task transfer remains a more practical and reliable approach than relying on policy generalization for unseen tasks.
Table 7: Zero-shot generalization to tasks with unseen constraints.
CoPDTT
Prompt-
Task
reward
cost
reward
cost
AntV2
0.99
2.18
1.10
5.16
HalfCheetahV2
1.02
0.01
1.15
7.74
H2pperV2
0.52
9.96
0.70
16.14
To further support our hypothesis that pretraining broader range of similar tasks enhance the zero-shot generalization capability of the learned policy, we conduct an additional experiment in which policies are trained under larger number of velocity thresholds (from 2 to 6) in the AntVelocity and HopperVelocity tasks. Specifically, in the 2-threshold setting, the thresholds are [2.57, 2.62] for Ant and [0.37, 0.74] for Hopper. In the 6-threshold setting, we use [2.42, 2.47, 2.55, 2.57, 2.62, 2.67] for Ant and [0.37, 0.42, 0.52, 0.62, 0.67, 0.74] for Hopper. For evaluation, beyond the original zero-shot generalization tasks (AntV2 with threshold 2.52 and HopperV2 with threshold 0.56), we further design two additional zero-shot generalization tasks: AntV7 (threshold 2.4) and V7 (threshold 0.35). Notably, the V2 tasks correspond interpolation generalization settings, while the V7 tasks represent extrapolation settings. All results are obtained under a budget of 10, and evaluated using random seeds. The experimental results are shown in Table g It can be observed that when CoPDT is pre-trained on larger number of similar tasks similar to the setup in meta-RL- the model exhibits significant improvements in safety performance on both interpolation (V2) and extrapolation (V7) generalization tasks. The improvement is particularly notable in the Hopper environment, which may be attributed to the large threshold gap between the original two Hopper tasks, making pre-training on only two tasks insufficient. These results support the claim that our constraint prioritized prompt encoder not only captures task-specific memory but also possesses a certain degree of generalization capability to unseen constraints.
33

---

Table 8: Zero-shot generalization results with more similar training tasks.
CoPDT (2 similar tasks)
CoPDT (6 similar tasks)
Task
r
cJ
rf
AntV2
0.99
2.18
0.98
1,56
AmtV7
0.99
3.28
0.96
2.34
HopperV2
0,52
9.96
0,48
0.04
HopperV7
0.24
21.68
0.43
4.57
Average
0.69
9.28
0.71
2.13
Table 9: Comparison results of coupled and decoupled prompt encoder training.
Multi- conti-conntraint Multi Multi-budget
Task
CoPDT
coupled
CoPDT decoupled
rf
c/
cl
P
0.03
0.39
0.04
0.55
PointButton2
0.12
1.48
0.08
0.98
PointCircle
0.50
0.54
0.55
1.09
PointCirele2
0.51
0.58
0.57
1.75
PointGoal1
0.24
0.35
0.24
0.30
PointGoal2
0.25
1.04
0.26
0.66
PointPush1
0.14
0.57
0.12
0.69
PointPush2
0.09
0.54
0.11
0.83
CarButton
0.05
0.87
0.04
0.89
2arButton2
-0.02
0.80
-0.02
0.94
CarCirelel
0.50
2.92
0.50
2.89
CarCircle2
0.31
1.76
0.34
1.67
CarGoall
0.20
0.39
0.22
0.32
CarGoal2
0.15
0.98
0.13
0.91
CarPushl
0.17
0.31
0.18
0.48
CarPush2
0.06
0.85
0.06
0.62
Swimmer Veloeeereeeeyyyyvvv VO
0.68
0.82
0.69
0.84
Swimmer Velocitty V1
0.59
0.59
0.61
0.74
HopperVelocity VO
0.63
3.05
0.57
4.28
Hloppereelocity
0.21
1.02
0.27
1.09
HallCheetah Velocity VO
0.66
0.18
0.70
0.36
HellCheetta
0.69
1.73
0.75
1.22
Walker2dVelocity VO
0.39
5,25
0.35
4,44
Walker2dVelocittyV
0.73
1.15
0.66
0.73
AntVelocity
0.96
4.60
0.95
4.89
AntVelocityVI
0.86
5.10
0.92
3,42
Average
0.37
1.46
0.38
1.45
F.9 Discussion on coupled or decoupled prompt encoder training
In context-based multi-task RL, whether the prompt encoder should be trained jointly (coupled) with the policy remains open research question. Coupled training allows for richer parameter updates and potentially more accurate adaptation, while decoupled training better preserves the desirable properties obtained during pretraining. To investigate this, we conduct experiments comparing the coupled and decoupled training of the prompt encoder within CoPDT. As shown in Table coupling the prompt encoder with policy optimization does not yield significant performance gains. This suggests that, in our setting, the simpler decoupled training scheme is already sufficient. Nevertheless, the broader question of whether and when prompt encoders should coupled with policy training warrants further investigation.
34

---

Comple che Amidyail CUROET
(a)
(b)
(c)
Figure 13: (a) Performance ablation on X. (b) Time complexity analysis of X. (c) Performance
ablation on whether using the inverse dynamics model g in 4 tasks.
E.10 Ablation on CPRTG sample number X
In this section, first conducted ablation study different choices of the CPRTG sample number X to investigate the impact of this hyperparameter on policy performance. The results are shown in Figure [3(a) It can be observed that as X increases from 100 2000, there is almost significant change in the policy's reward, but a noticeable reduction in the policy's cost. This result confirms that the CPRTG sample number increases, the sampled values are closer the desired quantile points, leading to better performance and a certain improvement in safety.
Additionally, performed further analysis of the time complexity of the CPRTG generation process in CoPDT, with the results presented in Figure 13(b) From the figure, we can draw two conclusions. First, when CPRTG used, the time consumption does indeed increase compared to not using CPRTG, indicating that the CPRTG generation process introduces additional computational overhead. Second, when increases from 100 2000, the time overhead remains almost unchanged. suggesting that the additional cost brought by CPRTG mainly comes from the inference of the CPRTG generator's neural network, rather than the sampling of quantile points. Therefore, in practice, can increase X as much as possible to achieve better policy performance without introducing significant additional computational costs.
F.1 Discussion and ablation on inverse dynamics model g
we introduced an additional inverse dynamics model g to compute the inverse dynamics error for training the environment--peciff state encoders. The primary motivation for using this model to address tasks with identical state and action spaces but different dynamics transitions. While such tasks have not appeared in main experiments, they still quite common (Nagabandi ctt al.) 2019 [Eysenbach et al. [2021] Zhang et al.) 2024). In these cases, the inverse dynamics error based the inverse dynamics model can effectively produce different state representations during the environment-specific state encoder learning phase, thereby reducing the learning difficulty for the constraint prioritized prompt encoder. Moreover, described in Appendix C when the environment ID is unknown, the inverse dynamics error based on the inverse dynamics model becomes the core method for distinguishing these tasks, making it essential component. We also conducted an additional ablation study to ensure that the use of the inverse dynamics model does not negatively impact the policy performance, with results shown in Figure 13(c) which aligns with our expectations.
In Section 3.1
F.12 Ablation on LoRA rank
In LoRA, performance is mainly influenced by the LoRA rank r and the LoRA a (Hu et al. 2022). Following standard practice, we set to twice the value of r and conducted experiments on task transfer with various values. The results, shown improves as r increases, except in the HalfCheetah task, where an anomaly occurred, consistent with previous findings. These results suggest that when the model has relatively few parameters, increasing the number of tune-tuned parameters positively impacts performance.
Tigure14 indicate that performance generally
35

---

rank 4
rank
rank 16
rank 32
bound
AntV2
HailCheetahvz
Hoppervz
SwimmerV2
Walker2dv2
1.00
1.00
1.10
Rewerd
2.05
0.33
0.33
0.66
1.30
0.55
0.00
0.2
1.01
0.66
0.00
1.00
0.23
00
25
10.0
5.0
100
23
13
100
0.0
5.
25
230
10.0
1.5
1.0
4.0
0.0
8.0
20
1.0
6.
2.0
4.
2.0
0.0
0.0
00
2.5
75
100
0.0
160
2.5
7.5
10.0
5.0
25
120
Step (1o4)
Step (1o4)
Step (1e4)
Step (164)
Step (1m4)
Figure 14: Ablation on different LoRA ranks.
Table 10: Detailed results of the prompt quality analysis done in the Multi-constrain Multi- -udget setting. The red results indicate results that are significantly affected by changes in prompt quality.
CarDI
C.FDT Stggle PC
Paangi-CUr
Thak
e7
I7
21
47
27
4]
K]
47
17
FL
8/8
0.9
0.84
0.99
0.00
9.85
0.10
1.21
0.00
0.68
0/12
9.59
0U90
9.66
1140
4.22
D41
4.23
0L95
3.29
0.46
500
0:35
03.51
0.51
8.59
0.59
0.33
0.53
0.53
0.35
3.00
0.59
0:54
0.34
0.34
0,52
0:57
13.50
0.50
0.51
272
1155
0.52
0:52
1.45
2400
0358
D661
3.37
2.62
hirrkoll
8.34
16.35
0.34
8.24
0.37
0.36
0.25
8:36
4.32
0.36
0.58
D.67
1.47
1.58
0.88
hiiDOell2
6.26
6:25
0.55
8:25
0.55
hishall
8.12
0660
04.44
0.74
0.16
0.15
0.43
0.62
0.55
0.11
0.11
0.12
0.51
1L.3
0.45
0.24
1.25
9.26
1.42
0.25
1.3
0.27
121
Caftancal
1U99
1U84
0.68
0.64
9.64
9.66
0.67
0.84
1U72
0.51
1U81
0.70
1374
6.34
533
0.76
585
0.78
600
CrBaaee22
4003
-001
0.91
0:25
5.10
538
CaClack
0.50
01:93
2.54
0.50
2.54
0.45
2.54
01.51
3.27
0.31
3,10
0.51
3.16
2.44
0.31
3.39
0.55
3.23
0.55
3.20
0.39
3.38
CaClrild
11:35
0.23
0.50
4.92
11.22
0.23
0:51
1:33
3.39
0.57
5.01
0.26
6.88
0.57
330
0.59
Catboll
8.22
11.34
0.24
4.76
0.22
0.21
0.20
0.21
0.36
0.36
0.56
0:56
1.58
3423
Carkolt
6.15
0.91
1.33
0.17
1.00
0116
1.01
0.54
1.94
0.17
8.16
4.56
6.90
0.31
0.36
2.48
0.39
2.75
0.34
2.50
Calaal
CaPaad
8966
1662
1083
0.46
0.55
0.55
0.59
0.85
0.61
0.66
1U75
0.11
253
1423
2.77
0.17
2.27
0.17
2.13
330
0.37
2.45
SwireekitrstVV
861
0.74
08.56
0.38
0.57
0.30
6.57
8.6
0.99
0.74
0.57
0.70
8.57
4.5
1359
0.50
0.66
BG2
0.66
8:49
0.62
002
Waveyy
0:70
1.00
09:37
1:35
13.30
101
0.31
1.00
11755
1:43
0.33
1.34
0:79
1.66
1.66
1.58
5.50
D/2
6.11
0.77
6:83
0.71
0.57
EalClenniVikeeig00 VD
6:00
0.88
0.59
5:3
0.70
0.52
10.29
25:97
25.89
1.58
26.06
20:54
072
1.22
9.62
0.55
6A6
0.26
0.55
0.20
0.56
0.63
9.55
0.55
0.41
8.55
0.42
0.46
0.56
8:38
15:35
0.35
D.47
9.07
11:03
0.38
0.09
5:40
15:00
13:41
15:33
1.23
8/6
1664
1:11
0.31
1.00
0.72
0.58
0.68
8.71
0.69
8.76
8.7A
0.31
6.13
Ax
805
4.00
1096
5.10
0.56
5.00
096
4.5
0.39
8.45
0.58
5.55
0.06
3.50
0.97
8.19
0.96
2.55
2.51
0096
2.43
0.54
1,65
Ax
Avtagg
8:38
1,5
0.37
1.41
0.35
0.55
2.37
0.39
1.69
0.38
1.88
0.00
2.33
0.39
3.19
0.58
4.38
0:58
4.55
0.50
4.8
0.59
531
F.13 Detailed prompt quality analysis results
In this section, we evaluate the impact of prompt quality on the performance of CoPDT, CoPDT Simple PE, and Prompt-CDT. Specifically, CoPDT Simple PE replaces the constraint prioritized prompt encoder in CoPDT with a standard MLP encoder. We consider four categories of prompt quality: Expert iin-iitribution expert trajectories seen during prompt encoder training), Mediuml (unseen trajectories with severe safety violations), Medium2 (unseen trajectories with significantly lower reward performance), and Random (trajectories collected from a random policy). As shown in Table 10 while all methods are affected by prompt quality to some extent, CoPDT demonstrates substantially greater robustness. This highlights the superior generalization capability of the constraint prioritized prompt encoder in handling unseen prompts.
36

---

Table 11: Detailed results of the ablation studies done in the Multi-constraint Multi-budget setting.
Ours (MT)
W/o CP
Det CP
W/o CD
Wo PE
Simp PE
Small
DT
Task
rt
cl
rt
c]
r1
r.
r7
c]
c]
rT
c]
PointButton1
0.04
0.55
0.49
3.94
0.03
0.62
0.09
1.14
0.08
0.94
0.02
0.60
0.07
0.73
PointBatton2
0.08
0.98
0.32
3.31
0.03
0.95
0.11
1.41
0.07
1.01
0.10
1.21
0.11
1.19
PointCinclel
0.55
1.09
0.57
0.93
0.52
1.05
0.55
1.05
0.48
0.70
0.53
0.86
0.53
0.71
PointCirele2
0.57
1.75
0.60
1.92
0.55
1.85
0.57
1.85
0.57
2.59
0.55
1.51
0.53
1.46
PointGoall
0.24
0.30
0.62
1.44
0.20
0.35
0.33
0.53
0.22
0.32
0.26
0.36
0.28
0.48
PointGeal2
0.26
0.66
0.49
2.33
0.23
0.63
0.30
0.89
0.22
0.79
0.22
0.77
0.27
0.99
PoiniPushi
0.12
0.69
0.25
1.40
0.10
0.57
0.52
0.12
0.56
0.17
0.69
PointPush2
0.11
0.83
0.15
1,25
0.10
0.78
0.10
0.94
0.11
0.77
0.11
0.80
0.14
1.20
CarButton
0.23
4.70
0.02
0.65
0.67
0.02
CurButton2
-0.02
0.94
0.17
4.77
-0.02
0.77
-0,05
1.56
0.01
1.07
-0.01
1.05
-0.04
1.14
CarCircle1
0.50
2.89
0.52
4.10
0.49
2.87
0.50
3.17
0:42
2.44
0.51
3.27
0.56
4.53
CarCircle2
0.34
1.67
0.55
4.61
0.31
1.40
0.35
2.05
0.46
4,06
0.32
1,37
0.38
2.41
CarGoall
0.22
0.32
0.50
1.32
0.19
0.29
0.24
0.37
0.19
0.28
0.21
0.34
0.26
0.45
CarGoal2
0.13
0.91
0.32
1.83
0.15
0.94
0.20
1.01
0.15
0.76
0.14
0.94
0.19
1.06
CarPushl
0.18
0.48
0.24
0.73
0.16
0.43
0.20
0.45
0.17
0.33
0.18
0.35
0.19
0.38
CarPush2
0.06
0.62
0.16
0.04
0.46
0.07
0.54
0.61
0.06
0.81
SwimmerVelocity VO
0.69
0.84
0.72
0.72
0.53
0.64
0.72
0.82
0.67
5.79
0.70
1.93
0.67
3.12
VwimmerrVelociyy1
0.74
0.66
0.43
1.38
0.74
0.56
HopperVelocityVo
0.57
4.28
0.83
2.31
0.41
3.77
0.60
4.13
0.55
2.69
0.55
3.25
0.37
1.85
HopperVelocityV1
0.27
1.09
0.64
3.53
0.23
0.96
0.37
1.60
0.44
0.52
0.28
1.43
0.16
2.08
0.70
0.36
0.95
0.78
0.67
0.31
0.76
0.40
0.75
13:77
0.7
0.38
0.78
0.17
HallCheettahNelocctyy
0.75
1.22
0.95
0.27
0.73
1.06
0.80
1.13
0.73
0.65
0.88
0.49
0.67
0.69
Valker24VelocityY00
0.35
4.44
0.28
2.47
0.36
4.51
0.36
4.49
1.44
27.45
0.40
5.57
0.36
4.64
Walker2WVelocityVV
0.66
0.73
0.74
0.09
0.66
0.72
0.67
0.72
0.69
0.71
0.72
0.59
0.58
2.28
VotVolocityV
0.95
4.89
0.94
1.65
0.96
5.01
0.96
7.02
8.45
0.99
10.13
V1tVelocityV1
0.92
3.42
0.98
0.71
0.91
3.80
0.95
3.13
0.92
5.24
0.96
3.44
0.96
3.29
Average
0.38
1.45
0.53
0.35
0.41
3.16
1.60
0.38
1.84
E114 Detailed main ablation results
In this section, we also provide the detailed results of the ablation studies, as shown in Table 11 Here. we provide an additional baseline Det CP, which models qe deterministically. The results show that Det CP achieves improved safety performance, but this comes at the cost lower reward performance. Moreover, due to its deterministic nature, Det CP is unable to adjust conservatism through Bend. lacking flexibility.
F.15 Standard deviation of the main results
In this section, we present the standard deviation of CoPDT and various baselines three different random seeds, as shown in Table 12, The results demonstrate that CoPDT consistently achieves significantly lower standard deviation compared other baselines, indicating superior stability.
G More details about related work
Safe RL Safe RL is a kind of machine learning approach aimed at learning policies that maximize cumulative rewards while adhering to additional predefined safety constraints (Gu et al. (022). Safe
RL algoritmss are broadly divided into two categories: safe exploration and safe optimization (Gareıa
& Fernández 2013). Safe exploration algorithms do not focus on directly optimizing the policy. Instead, they aim to modify the policy's behavior through additional mechanisms to prevent violations of safety constraints. A typical example of these algorithms is shielding based methods (Alshiekh) al. 2018, Cheng et al. 2019 Xiao et al., 2023), which construct or learn logical structures known as "shields" or "barriers" that ensure the actions taken in a given state comply with the safety constraints. However, the decoupling from policy learning of safe exploration methods results in lower learning efficiency, leading to growing focus safe optimization algorithms. Safe
37

---

Table 12: Detailed standard deviation results of the main experiment, All results are computed using three different random seeds. For experiments involving multiple budgets, the reported standard deviation is averaged across all budgets.
Single constrine Milli budger
Mult coustruar Sisgle-rudge
Muti constraint Muugs-rudgg
Task
CPQ
CDT
FISOR
CdFOT
MICDT
Prongt CDT
CoPDT
MFTCDT
ProngeCDD
CoPDT
r_Gal
c_3sl
r.ald
r_46
c_vo
s.aid
c..c
r.46i
c.44
c.aid
c_.6d
T.46
c_avl
c_.ol
PainBuncel
0.30
0.48
0.05
1.32
0.01
0.07
0.01
0.03
0.00
1.31
0.03
0.04
0.27
PeletBunce2
0.08
1.25
0.04
0.52
0.63
1.10
0.04
0.44
0.07
0.20
0.01
0.80
0.00
0.54
0.07
0.78
0.01
0.96
0.05
0.33
PeiaCaclel
1003
0.08
0.01
0.1
0.00
0.01
0.02
0.13
0.20
PeinCreke
0.24
5.19
0.01
0.17
0.04
4.91
0.01
0.12
0.01
0.47
0.01
0.21
0.03
0.11
0.01
0.20
0.02
0.51
0.03
0.59
PrinGaall
0.50
Pointisat2
0.14
1440
003
0.64
0.06
1154
0101
0.05
0.04
0.49
0.04
0,41
0103
0.04
0.52
010
0.57
0.03
0.20
PoinPuhh
0.77
0.25
0.05
0.01
6.22
0.00
0.02
0.02
0.12
0.41
0.03
0.12
PoinPPuh2
0.08
0.27
0.03
0.32
0.04
0.87
0.01
0.59
0.03
0.66
0.05
0.40
0.01
0.50
0.04
0.62
0.05
0.44
0.01
0.30
Ceffrianl
2.13
0.03
0.46
0.05
0.12
0.00
0.03
0.07
0.02
0.29
0.06
1:15
0.05
0.96
0.01
0.34
Carfumon2
0:10
1.94
0.05
0.55
0100
0.05
0.00
0.02
0.04
0.15
0.33
Carlinlll
0.09
2.35
0.04
0.65
0.03
4,30
0.05
1.09
0.01
0.22
0.06
0.57
0.00
0.33
0.01
0.29
0.122
0.78
0.02
0.19
CaCCnde2
0.99
0.01
CaCacal
0.04
0.39
0.00
0.26
0.63
0.57
0.04
0.04
0.04
0.30
0.06
0.45
0.05
0.00
0.08
0.52
0.05
0.52
0.05
0.00
Carinal
0.57
0.06
0.01
0.04
0.0%
0.25
CarPubl
0.16
4.97
0.12
0.21
0.02
11445
0.02
0.07
0.01
0.56
0.03
6.17
0.022
0.54
0.02
0.31
0112
0.34
0.01
0.35
CaPudd
2,06
0.03
0.82
0.05
0.71
0.01
6.02
0.04
0.01
0.36
0.38
0.67
0.04
0.50
0.03
0.40
SwinmeeKkeriy MD
0.07
0.72
0.01
0.33
0.01
0.17
0.00
0.18
0.02
2.29
0.00
0.06
0.01
0.04
0.02
3.07
0.00
0.06
0.01
0.06
Swinme K1
1.02
0.07
0.79
0.01
0.20
0.09
0.12
0.02
0.30
0.01
0.19
0.01
0.20
0.01
0.16
0.01
0.56
0.0%
0.21
Hopper Vo
1.59
0.04
0.23
0.06
0.09
0.20
0.13
2.34
Boppe VI
0.14
2,05
0.04
1,39
0.08
0.65
0.03
0.15
0.01
1.53
0.06
5.32
0.01
0.26
0.04
1.46
0.07
3.17
0.09
0.29
HallChesskkciseeyy0
1.55
0.01
Halk Vonshccecyy
6:21
1.45
0:00
0.15
0.00
0.00
0:07
0.00
0.01
0.68
0.02
0.35
0.00
0.27
0.04
0.75
0.02
0.56
0.08
0.44
0:46
0.59
0112
6.36
0.42
0.00
0.37
9:37
0.38
0.08
0.47
Walee 2
0.02
0.25
0.01
0.06
0.12
0.59
0.02
0.07
0:01
0.20
0.08
0.29
0.07
0.61
0.01
0.23
0.06
0.13
0.05
0.39
AntVelecit VD
0.00
0.09
0.01
0.00
0.01
0.00
0.00
0.02
0.00
0.07
0.01
0.43
0.02
1.59
AntVclociy V1
0.00
0.00
0.01
0.13
0.01
0.00
0.01
0.59
0.00
0.12
0.00
0.06
0.04
1.25
0.00
0.11
0.01
0.12
0.05
0.39
Average
0.09
1.41
0.03
0.41
0.04
1.16
0.02
0.27
0.05
1.60
0.04
1,02
0.04
0.50
0.05
1.75
0.04
1.17
0.04
0.42
optimization algorithms typically model the problem as CMDP, with Lagrangian multiplier-based algorithms being the mainstream solution, as discussed in the main paper. Other than Lagrangian multiplier-based algorithms, trust region methods are among the most prevalent approaches in safe optimization. They attempt to keep policies within a safe trust region during updates via low-order Taylor expansions (Achiam et al., 2017) variational inference (Liu et al., 2022). Due to their robust learning process, trust region methods are also further applied to multi-agent scenarios (Gu et al. 2023). However, their on-policy nature results in lower data efficiency. In response, recent works have increasingly focused on off-policy safe optimization. CAL (Wu et al. (024) imprrves the optimization of Lagrange multipliers using the augmented Lagrangian method and enhances the conservatism of the cost function learned off-policy via the use of upper confidence bound. Meanwhile, SafeDreamer (Huang et al.) (224) increases data efficiency by learning an environment model and using model rollouts for data augmentation. Recently, more attention has been directed toward conditiend RL. CCPO Yao et al. (203) effectively adapts to different safety thresholds in an online algorithm by incorporating the safety threshold into the input of CVPO. On the other hand, SDT (Guo et al. 2024) attempts to integrate safety prior knowledge expressed through temporal logic into the input of DT, further enhancing the policy's 's safety performance while adapting to various temporal logic safety constraints. This provides fresh perspective for the practical application of safety RL. With the growing body of research on safe RL, these algorithms have found increasing applications various fields. Notable examples include ensuring the safety of vehicles in autonomous driving (Kiran et al. 2021) and safeguarding robots in industrial settings (Brunke et al. (022)..
Offline RL trains policies using pre-collected datasets, avoiding real-world trial and error, which is critical for deploying RL in practical settings. Its primary challenge is addressing the extrapolation errors (Prudencio et al. 2023). Methods like BCQ (Fujimoto et al.) 2019) and CQL (Kumar et al., 2020) tackle this by constraining actions to those seen in the offline data or by penalizing unseen actions. Others, such as MOReL (Kidambi et al., 2020) and MOPO (Yu et al.) 2020), learn the environment models from the offline data and utilize these models with uncertainty estimates to avoid OOD regions with low model accuracy. However, these methods only focus on reducing extrapolation errors, without addressing the challenge of generalizing in OOD areas. To tackle this limitation, MOREC (Luo et al. 2024) employs adversarial learning in the model learning process, improving model generalization abilities.
Offline RL
38

---

Meta RL Meta RL is similar to multi-task RL, with both involving multi-task training. However, Meta RL does not receive additional expert trajectories as prompts during testing. Instead, must collect data in the unknown environment to generate prompts. Additionally, it focuses on training across large-scale similar tasks for generalization to new ones (Zhu et al., 2023). PEARL. (Rakelly) et al. 2019) uses a probabilistic encoder to facilitate task identification and employs Thompson sampling for data collection in new environments. Other works, like FOCAL (Li et al. (021), and CORRO (uaan & Lu 2022), focus on designing contrastive loss functions for the encoder, improving task encoding robustness. COSTA (Guan et al. 2024) first considers safety in meta RL, designing a cost-based contrastive loss and safety-aware data collection framework, improving policy safety in both task identification and deployment.
H Discussion on limitations
CoPDT does indeed have some limitations. Firstly, CoPDT does not exhibit strong generalization capabilities, which is actually anticipated. One reason is that in our multi-task training, we did not select large set of similar tasks (such as numerous variations of velocity) but rather chose a variety of tasks with significant differences. We believe this scenario is more common in practice. In contrast, past meta-RL methods, which emphasize generalization, typically demonstrate limited generalization ability only when trained on large number (dozens or even hundreds) of similar tasks. Another reason that generalizing to unseen safety constraints has limited practical value. Safety constraints are stringent, and in real-world applications, there is often preference for retraining or fine-tuning model to obtain safer policy rather than relying on an unprovable generalized policy. Secondly, CoPDT may introduce larger errors in RTG modeling for tasks with larger and more diverse reward distributions (e.g., velocity tasks), leading to drop in performance compared to tasks with smaller and more concentrated reward distributions (e.g.. point, car tasks). Therefore, further research into how to model RTG effectively for tasks with wide-ranging reward distributions is needed. Finally, CoPDT uses different input-output heads for tasks in different environments. As result, when fine-tuning in unseen environments, it requires the re-initialization of additional input-output heads, which may reduce the efficiency of knowledge transfer. Thus, how facilitate knowledge sharing and transfer across tasks in different environments (with different state action spaces, and transition functions) remains important avenue for future research.
I Broader societal impacts
The goal of the work presented in this paper is to advance the development of sequence modeling- based methods in offline safe RL. These methods, built on the Transformer, are intended more effectively scale to multi-task scenarios and beyond, laying a foundation for future research on scaling policies in RL and safe RL. In the future, our approach holds promise for broader application across variety of decision- does not pose any foreseeable negative societal impacts.
scenarios involving budget constraints. Mcanwhile, we believe our work
39


Reviews:
[{"review_id": "a34ijLP6wd", "reviewer": "Reviewer_9wTp", "summary": "The paper introduces CoPDT (Constraint Prioritized Decision Transformer), a novel framework for offline safe reinforcement learning (RL) in multi-task settings where multiple safety constraints and budgets vary across tasks. COPDT introduced two key components: a constraint prioritized prompt encoder that distinguishes safe and unsafe trajectory segments to identify task-specific constraints, and a constraint prioritized returns-to-go generator that dynamically produces conflict-free RTG values conditioned on safety budgets and contextual signals. These allow CoPDT to generate adaptive, safe policies using only offline data, without requiring environment interaction. Extensive experiments demonstrate that CoPDT outperforms state-of-the-art baselines in both safety compliance and generalization, particularly excelling in multi-budget and multi-constraint scenarios.", "questions": "Questions echo the weakness,\n1. Choosing $\\beta_t$ relies on the structure that $q_\\phi(R_t|C_t)$ is Gaussian, which is a strong assumption. What if the underlying distribution is not Gaussian?\n2. Any examples or theoritical intuitions of defining the causal structure $p(R_t|C_t)$ in line 163?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n1. The writing is clear.\n2. The motivation and background of safe multi-task RL are well explained.\n3. The problems described in the paper are adequate and handled properly.\n4. The separation of safe and unsafe encoders looks novel to me.\n5. The causal structure of CTG and RTG, and introducing the variable $G_t$ indicating the optimality, looks novel to me, which might create novel viewpoints in balancing the returns and safety in the community of safe RL.\n6. The experimental results are extensive and convincing.\n\n## Weakness\n1. The beta-decay design seems artificial, although it makes sense.\n2. The notation system does not highlight the multi-task nature of the problem formulation, which should be the core contribution of the paper. The authors should revise the notations.\n3. The causal structure, despite its novelty, lacks some intuitive explanations and examples. It will be good to include some examples.", "was_revised": true, "final_justification": "I have read the comments from other reviewers and AC. I will keep my score.", "rebuttal": "Thank you for your kind and thoughtful reviews. We appreciate your feedback and hope that our response addresses your concerns.\n\n### Q1 Choosing $\\beta_t$ when the underlying distribution is not Gaussian?\n\nIn our current implementation, we model $q_\\phi$ as a Gaussian distribution, as it is generally sufficient to handle most practical scenarios. However, we acknowledge that this assumption may not hold in more complex scenarios. In such cases, we believe that **leveraging the strong multi-modal distribution modeling capabilities of generative models—such as diffusion models or flow matching—provides a more suitable alternative** for modeling $q_\\phi$. Under this setup, the choice of $\\beta_t$ can remain consistent with the Gaussian case. Specifically, to sample $\\hat{R}_t$, we first draw $X$ samples from a standard normal distribution. These samples are then transformed using a diffusion model or flow matching to generate $X$ candidate values. Finally, we select the $\\beta_t$-quantile among these candidates as the final $\\hat{R}_t$.\n\n### Q2 Intuitive explanations about modeling RTG on CTG?\n\n**Intuitively:** To illustrate the motivation behind our design, consider a simple example. In our framework, the CTG represents the goal of safety satisfaction, while the RTG corresponds to reward maximization. The DT policy acts as a **goal-conditioned policy**, where both CTG and RTG jointly specify the intended behavior.\n\nSuppose an agent operates along a 1D axis and starts at the origin. The CTG defines a safety region—e.g., $X \\in [-10, 10]$—within which the agent must remain. Meanwhile, the reward function incentivizes movement in the positive X direction, so the RTG effectively specifies a target position. If the RTG is set to 20, the agent will attempt to reach $X = 20$, which lies outside the safe region, resulting in a conflict between the safety and reward objectives. In this case, the agent may violate safety constraints in pursuit of the reward.\n\nTo avoid such conflicts, our **CPRTG mechanism** retrieves feasible (CTG, RTG) pairs from the offline dataset. For instance, for a given CTG, the dataset might contain trajecotries with RTG values of 5, 8, 9, and 9.8. **Since these trajectories already satisfy the safety constraints, their associated RTGs are guaranteed to be compatible with the CTG**. Among these candidates, we select the one with the **highest return** (e.g., 9.8), thereby maximizing reward **while ensuring safety**.\n\n**Formally:** More detailly, modeling RTG using $q_\\phi$ conditioned on CTG can be interpreted from two complementary perspectives: **(1) Resolving the conflict between RTG and CTG in multi-budget safe RL; (2) Reducing extrapolation error from an offline RL perspective.**\n\nFor the first point, we have visualized the conflict between RTG (representing reward maximization) and CTG (representing adherence to safety constraints) in **Fig. 2(b) and 2(c) of Sec. 4.3**. The plots show that, for a DT model, when CTG is held fixed and RTG is increased, both the reward and cost performance of the resulting policy increase accordingly. In contrast, when RTG is fixed and CTG is varied, the performance remains largely unchanged. This asymmetry indicates that the **DT model tends to prioritize RTG over CTG, thereby potentially neglecting safety requirements**. To address this issue in safe RL—where satisfying safety constraints is often a primary concern—we propose using $q_\\phi$ to automatically generate RTG values conditioned on a given CTG. These RTG values are learned from offline data and are generally **more conservative than externally specified RTG targets**, which helps mitigate the conflict between reward and safety objectives.\n\nFrom the offline RL perspective, DT learns policies through behavior cloning via supervised learning. Hence, its performance heavily relies on how well the **input distribution aligns with that of the offline dataset**. If CTG is regarded as an externally specified safety target and **RTG is treated as part of the input state**, minimizing the extrapolation error requires aligning the policy input distribution with that of the offline data. To achieve this, we model RTG using $q_\\phi$ learned from the offline dataset. This interpretation is further elaborated in **App. A**.\n\n### Q3 The notation system does not highlight the multi-task nature\n\nThank you for the constructive suggestion. We will revise our notation system accordingly in the next version. Specifically, we will explicitly incorporate the task embedding into the formulations in Sec. 3.2 and 3.3 to better highlight the multi-task nature of our approach."}, {"review_id": "VLuNVNvO0y", "reviewer": "Reviewer_2Z9j", "summary": "This paper proposes CoPDT (Constraint Prioritized Decision Transformer), a method for offline safe reinforcement learning in multi-task settings with varying constraints and budgets. CoPDT introduces two key components:\n- A constraint prioritized prompt encoder that distinguishes between tasks.\n- A CPRTG mechanism that dynamically generates Return-To-Go values conditioned on safety budgets, resolving conflicts between reward and safety objectives.\n\nCoPDT outperforms several baselines under different settings of the OSRL benchmark.", "questions": "- Do you train CoPDT on a mix of PointButton1 and PointButton2 datasets, or across all Point robot datasets? Or does each environment have multiple cost functions? If it’s the latter, where do these multiple cost signals come from? Simply modifying the constraint threshold (e.g., a velocity limit that triggers a binary cost) may not justify calling it a separate constraint. Multiple costs should correspond to qualitatively different constraints. For a true multitask setup, training on a mix of datasets with different task semantics (e.g., PointButton and PointCircle) would be more compelling. This needs clarification.\n- How are safe vs unsafe trajectory segments defined? Is the model trying to drive costs to zero, or just keep them under the given budget?\n- If the expert trajectory used for prompt identification has zero cost, how does it convey information about the constraint? Or do expert demonstrations consume the full budget? \n- How is the constraint type identified from the trajectory itself? If the constraint is not directly observable (like a rule about staying in a lane), how does the model infer its presence from data?\n- Is Equation 9 used to train the CPRTG model q_phi? How exactly is this done, and how is Gt (the expert indicator) defined in practice? What counts as an expert trajectory?\n- Comparing to FISOR in the multi-budget setting is problematic, as FISOR is not budget-aware and its performance is mostly fixed unless re-tuned.", "limitations": "Yes, the authors adequately address limitations and societal impact considerations. They explicitly acknowledge:\n- The limitations of relying on binary cost signals.\n- The difficulty of scaling to broader task distributions.\n- The challenge of extrapolation errors in offline RL, mitigated in part by the CPRTG design.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The work addresses an important challenge in safe RL: learning from offline data and generalizing to multiple constraints and budgets.\n- The CPRTG mechanism is a novel and well-motivated contribution that replaces manual RTG conditioning with a learned, CTG-conditioned alternative.\n- The paper presents strong empirical results across a variety of evaluation setups.\n- It includes extensive ablation studies and visualizations.\n\nWeaknesses:\n- The definition of multitask or multi-constraint settings is unclear. If the tasks only differ by simple modifications like velocity thresholds, it’s questionable whether they represent genuinely distinct or meaningful benchmarks. A more diverse or semantically varied tasks/constraints are required.\n- The architecture includes many components: multiple encoders, decoders, and learned modules. This increases complexity and raises concerns about over-engineering.\n- The comparison to CPQ is limited in value for the Single-constraint Multi-budget setting; BC-Safe (used in the oracle ablation) or more recent methods (below) show that CPQ is not a strong baseline. \n  - OASIS [arXiv:2407.14653]\n  - Latent Safety-Constrained Policies [arXiv:2412.08794]\n  - FAWAC [arXiv:2412.08880]\n  - Trajectory Classification for Safe RL [arXiv:2412.15429]\n  - Constraint-Adaptive Policy Switching [arXiv:2412.18946], referenced in the introduction, notably supports multi-budget generalization.", "was_revised": true, "final_justification": "The authors have addressed my concerns, which has given me a better understanding of the paper’s merits. As a result, I have increased my score.", "rebuttal": "Thank you for your careful reviews and constructive suggestions. We appreciate your feedback and hope that our response addresses your concerns.\n\n### Q1 Clarification of multi-constraint settings\n\nThank you for your valuable comments. We would like to clarify that **our multi-constraint setting does not refer to handling multiple constraints within a single environment (e.g., PointButton1)**. Instead, **as described in Section 4.1, we train a single DT policy using a unified dataset that spans 26 diverse tasks**, including **PointButton1 through AntVelocity-v1**. This universal policy is then evaluated across all these environments and further tested in few-shot transfer scenarios to previously unseen tasks.\n\nDue to the heterogeneity in state and action spaces among the 26 tasks—for example, the state dimension of PointButton is 76, while that of CarCircle is 40—we employ task-specific encoders and decoders. These components help standardize the input and output dimensions for the shared Transformer core, allowing it to effectively operate across diverse environments.\n\nWe sincerely apologize for the earlier lack of clarity in our explanation, and we truly appreciate your feedback. We will revise the manuscript to provide a more precise and detailed description of the multi-task setting to avoid any future confusion.\n\n### Q2 Definition of safe and unsafe\n\nWe thank the reviewer for the opportunity to clarify our problem formulation. Our work is situated within the **soft constraint formulation of safe RL**, a prominent and widely-adopted paradigm in the field [1, 2]. To be more precise, in this formulation, a trajectory is considered 'safe' if its total accumulated cost does not exceed a predefined budget. Therefore, the core objective of our model is to learn a policy that generates trajectories compliant with these specified cost constraints.\n\n### Q3 Zero-cost expert trajectory\n\nCoPDT, is designed to be robust and **does not necessitate unsafe examples in the expert data to function**. In the specific scenario where an expert trajectory has zero cost, our method gracefully handles it as follows:\n\n- The entire trajectory is effectively considered a single, continuous 'safe patch'.\n- This 'safe patch' is then processed exclusively by the safe encoder $p_s$ to generate its representation.\n- Since no cost violations exist, the unsafe encoder $p_u$ is naturally bypassed and does not contribute to the final embedding.\n\nCrucially, our formulation in **Eqn. (5)** is designed to unify these cases, allowing the final representation to be computed seamlessly. We will clarify this point in the revised version to avoid possible misunderstandings.\n\n### Q4 When constraint is not directly observable\n\nTypically, we assume that cost labels are directly available in offline datasets. However, in many real-world scenarios, we often encounter situations where only rule-based descriptions of costs are provided, without explicit cost labels or with only ambiguous or noisy cost labels. \n\nUnder this setting, we believe a promising approach is to **apply the commonly used language-guided reward generation methods [3,4] to cost modeling, and integrate them with CoPDT**. Specifically, cost-related rule descriptions, along with observation and other relevant information, can be input into a LLM. The LLM can then generate a cost function that aligns with the provided rule. After cost function generation, its correctness can be verified and refined through human inspection or a small number of labeled samples. Then, this cost function can subsequently be used to annotate the offline dataset without cost labels. We have conducted preliminary experiments along this line, as discussed in our response to **Q2 of Reviewer YyH8**.\n\n### Q5 Meaning of Eqn. (9)?\n\nThank you for pointing this out. We would like to clarify that **Eqn. (9) is not used to train $q_\\phi$**—rather, it is trained using Eqn. (8). **Eqn. (9) defines the target we aim to sample from $q_\\phi$**. In this formulation, $G_t$ serves as an expert indicator. Under our setup, among trajectories that satisfy the CTG budget constraint, those **with higher returns are considered more \"expert”**. Therefore, $p(\\hat{R}_t \\mid G_t, \\hat{C}_t, s_t)$ represents the desired sampling distribution over $\\hat{R}_t$ conditioned on $\\hat{C}_t$ and $s_t$, such that the sampled $\\hat{R}_t$ is more likely to correspond to trajectories aligned with the expert indicator $G_t$. Eqn. (9) then applies Bayes’ rule to decompose this distribution into two components: The first term reflects the likelihood under $q\\_\\phi$, i.e., the probability is higher when the sample is closer to the mean of $q\\_\\phi$; The second term, $p(G_t \\mid \\hat{C}_t, \\hat{R}_t)$, denotes the likelihood that a trajectory with the given CTG and RTG is considered an expert. When CTG is fixed, trajectories with higher RTG are more likely to satisfy $G_t$ (importantly, $G_t$ is not used to directly label trajectories—it is an interpretable latent variable). The takeaway from Eqn. (9) is that, during sampling from $q\\_\\phi$, the objective is **not only to sample high-probability RTGs but also to prefer relatively larger RTGs that are more likely to satisfy the expert indicator**. We will improve the clarity in the next version.\n\n### Q6 Comparison with FISOR?\n\nWhile FISOR is not inherently budget-aware, we include it as a strong baseline due to its SOTA performance in safety. FISOR is designed for hard constraint settings and does not accept a budget as input, we **conducted 12 separate runs for each task and grouped them into four categories, using $[10, 20, 40, 80]$ to normalize the cost accordingly**. From the safety perspective, this setup gives FISOR an advantage during evaluation, as the normalization scale of cost is relaxed for it. Despite this, CoPDT achieves comparable safety performance to FISOR, demonstrating that CoPDT maintains strong safety properties. We acknowledge that this comparison may not be perfectly aligned in terms of problem assumptions, but we believe it still provides useful insights into safety performance across settings.\n\n## Q7 New baselines\n\nThank you very much for your valuable suggestion. We have additionally provided experimental results comparing CoPDT with **OASIS [5] and LSPC [6]** in the **Single-constraint Multi-budget setting, and the conclusions remain consistent with our previous findings**. Due to time constraints, these experiments were conducted on the first 16 tasks. Since the official OASIS code does not include hyperparameter settings for the SafetyGymnasium tasks we tested, we followed a similar practice to that used in their BulletGym experiments and adopted the hand-set (RTG, CTG) pairs provided in the OSRL codebase as generation-condition hyperparameters for OASIS. \n\nThe results are shown in the table below. OASIS, which leverages diffusion models for data generation, achieved promising results in the BulletGym environment. However, due to time constraints, we were unable to perform extensive hyperparameter tuning for OASIS in the SafetyGymnasium environments, which may have limited its performance and failed to fully showcase its potential. On the other hand, LSPC—a hard-constraint baseline similar to FISOR—demonstrates strong safety performance and achieves constraint satisfaction in most environments. Nevertheless, **CoPDT still outperforms LSPC in terms of overall safety, underscoring the effectiveness of the CPRTG mechanism in balancing reward and safety under the multi-budget setting**, in line with our prior conclusions. We will include the corresponding experimental results in the next version.\n\n| Task | OASIS | LSPC | CoPDT |\n| --- | --- | --- | --- |\n|  | r↑  c↓ | r↑  c↓ | r↑  c↓ |\n| PointButton1 | **0.00, 0.97** | 0.16, 1.90 | **0.05, 0.66** |\n| PointButton2 | -0.16, 1.12 | 0.17, 1.70 | 0.14, 1.41 |\n| PointCircle1 | 0.59, 5.11 | 0.55, 6.64 | **0.50, 0.63** |\n| PointCircle2 | 0.69, 7.47 | 0.62, 5.31 | **0.61, 0.98** |\n| PointGoal1 | **0.41, 0.97** | **0.25, 0.27** | **0.36, 0.56** |\n| PointGoal2 | 0.37, 1.71 | **0.25, 0.85** | 0.31, 1.02 |\n| PointPush1 | **0.00, 0.67** | **0.13, 0.97** | **0.19, 0.88** |\n| PointPush2 | **-0.72, 0.52** | **0.11, 0.89** | 0.19, 1.47 |\n| CarButton1 | 0.00, 9.29 | -0.07, 1.37 | **0.07, 0.74** |\n| CarButton2 | -0.62, 1.30 | -0.15, 1.52 | -0.02, 1.33 |\n| CarCircle1 | 0.53, 5.26 | 0.45, 3.50 | 0.51, 3.34 |\n| CarCircle2 | 0.61, 17.71 | **0.26, 0.00** | **0.28, 0.98** |\n| CarGoal1 | **-0.57, 0.18** | **0.24, 0.51** | **0.33, 0.47** |\n| CarGoal2 | 0.01, 3.50 | **0.12, 0.59** | **0.19, 0.81** |\n| CarPush1 | -0.02, 1.41 | **0.22, 0.86** | **0.20, 0.67** |\n| CarPush2 | -0.18, 1.21 | **0.09, 0.93** | **0.07, 0.73** |\n| Average | 0.06, 3.65 | 0.21, 1.74 | 0.24, 1.04 |\n\n[1] Garcıa J, Fernández F. A comprehensive survey on safe reinforcement learning[J]. Journal of Machine Learning Research, 2015, 16(1): 1437-1480.\n\n[2] Gu S, Yang L, Du Y, et al. A review of safe reinforcement learning: Methods, theories and applications[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.\n\n[3] Yu W, Gileadi N, Fu C, et al. Language to rewards for robotic skill synthesis[J]. arXiv preprint arXiv:2306.08647, 2023.\n\n[4] Xie T, Zhao S, Wu C H, et al. Text2reward: Reward shaping with language models for reinforcement learning[J]. arXiv preprint arXiv:2309.11489, 2023.\n\n[5] Yao Y, Cen Z, Ding W, et al. Oasis: Conditional distribution shaping for offline safe reinforcement learning[J]. Advances in Neural Information Processing Systems, 2024, 37: 78451-78478.\n\n[6] Koirala P, Jiang Z, Sarkar S, et al. Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning[C]//The Thirteenth International Conference on Learning Representations, 2025."}, {"review_id": "4h9mGtGxFH", "reviewer": "Reviewer_YyH8", "summary": "This work proposes a novel offline safe reinforcement learning (RL) framework built upon the Decision Transformer (DT) architecture, designated as Constraint-Prioritized Decision Transformer (CoPDT). The proposed framework is designed to tackle the limited adaptability of existing DT-based approaches in multi-task scenarios characterized by diverse constraints and dynamic safety budgets. CoPDT encompasses two pivotal innovations: firstly, a constraint-prioritized prompt encoder that utilizes sparse binary cost signals to precisely identify task-specific constraints; secondly, a Constraint-Prioritized Return-To-Go (CPRTG) mechanism, which dynamically generates return-to-go targets based on Cost-To-Go (CTG) and safety budgets, thus reconciling the conflict between reward maximization and adherence to safety requirements. Comprehensive experiments conducted on 26 tasks from the OSRL benchmark validate that CoPDT retains competitive reward performance while demonstrating its efficacy in multi-constraint and multi-budget settings.", "questions": "1.The constraint prioritized prompt encoder relies on binary cost signals (c=0/1), yet real-world data often contains noisy or ambiguous cost labels. How does CoPDT perform under noisy cost signals? A robustness analysis would enhance confidence in its practical application, and improved performance in noisy environments would elevate the quality score.\n\n2.CPRTG utilizes CTG-based β decay to balance conservatism, with β_start and β_end being fixed. How sensitive is CoPDT to these hyperparameters across different tasks? A sensitivity analysis would clarify their impact on the risk-reward trade-off, and stable performance across varying values would strengthen the reliability of the method.\n\n3.This paper focuses on the offline setting, but many practical scenarios require online adaptation. Can CoPDT be extended to online fine-tuning (e.g., with limited environmental interactions)? A preliminary evaluation of online adaptability would enhance its significance, as successful results would broaden its applicability.\n\n4.The capability of zero-shot generalization to unseen constraints is limited (Fig. 4a). Could the integration of meta-learning (e.g., learning constraint embeddings across more diverse tasks) improve this? Evidence that meta-learning enhances zero-shot performance would address a key limitation and improve the overall score.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths：\n\n1. The method proposed in this paper is strongly motivated, with both the constraint-prioritized fast encoder and the CPRTG mechanism embodying clear design principles. \n\n2. In terms of experiments, it covers a variety of tasks, conducts rigorous comparisons with baseline methods (CPQ, CDT, FISOR), and verifies the contributions of core components (fast encoder, CPRTG) through ablation experiments.\n\n3. This paper skillfully combines existing methods to address the challenges of offline reinforcement learning in multi-task and multi-constraint scenarios. The modular design of CoPDT and sufficient empirical evidence provide some insights for relevant researchers.\n\nWeaknesses：\n\n1. Both the immediate encoding and inverse dynamics model presented in this paper are existing algorithms, and the core method of this paper is a combination of such existing methods, which significantly diminishes the innovativeness of this work.\n\n2. The constraint-prioritized immediate encoding strategy relies on accurate cost estimation, and deviations in cost estimation appear to significantly impact model performance.\n\n3.A clearer exposition of how β decay in CTG balances safety and reward gains would facilitate readers' understanding of the method proposed in this paper.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful reviews. We appreciate your feedback and hope that our response addresses your concerns.\n\n### Q1 Novelty?\n\nCoPDT tackles the **unique challenges of combining offline safe RL with multi-task sequence modeling**, introducing the following key innovations:\n\n- **Unified modeling for multi-task offline safe RL:** CoPDT is the first to explicitly learn a unified model adaptable to tasks with varying constraints and budgets. This improves offline data efficiency and few-shot transfer to new constraints in limited-data scenarios.\n- **Resolving reward-safety conflicts in multi-budget settings:** In such settings, optimizing for both high reward and strict safety often leads to conflicts that limit safety performance. To address this, CoPDT introduces the CPRTG token, conditioning RTG generation on CTG to prioritize safety while encouraging maximal feasible rewards. Experiments demonstrate that CoPDT achieves significantly better safety performance than CDT and other baselines under multi-budget conditions, validating the effectiveness of this approach. To our knowledge, no prior work has addressed this problem from the perspective of prioritizing CTG during RTG generation, making this a novel solution strategy.\n- **Constraint prioritized task representation for multi-constraint adaptation:** Inspired by hyper-networks, we propose a constraint prioritized prompt encoder explicitly integrates binary cost signals into encoder structure design for more effective constraint representation. As shown in **Fig. 3(b) of Sec. 4.3** and the experiments in **App. F.6**, this encoder outperforms standard MLP-based encoders, especially when prompt quality is low—demonstrating stronger robustness in constraint representation.\n\nIn summary, CoPDT introduces **key innovations in CPRTG modeling and structured prompt encoding**, offering a **targeted solution to multi-task offline safe RL** rather than merely combining existing techniques.\n\n### Q2 Noisy or ambiguous cost signals\n\nIn line with most prior work in offline safe RL, we focus on settings with clear, binary cost signals. However, we acknowledge that dealing with noisy or ambiguous cost signals is an important and practical challenge, and is valuable for future works.\n\nTo address this, we assume access to a **language-based descriptions** **for the cost-related rules.** Under this setting, we believe a promising approach is to **apply the language-guided reward generation methods [1,2] to cost modeling, and integrate them with CoPDT**. Specifically, cost-related rule descriptions, along with observation and other relevant information, are input into a LLM, which generates a cost function aligned with the described rules. This function can then be verified and refined via human inspection or a few labeled samples, and used to annotate datasets with noisy or ambiguous cost labels.\n\nWe conducted preliminary experiments in this direction, discarding the original (potentially noisy) cost labels and using GPT-4o-mini to generate a new, slightly more conservative cost function ( to mitigate the impact of ambiguity or imprecision in the rule-based descriptions) based on the language description. As shown in the table below, CoPDT trained on LLM-labeled data maintains strong safety performance—**even surpassing results obtained using the original ground-truth labels in terms of safety**. These findings validate the effectiveness of our proposed approach in scenarios with noisy or even absent cost labels.\n\n| Task | LLM-labled CoPDT |\n| --- | --- |\n|  | r↑  c↓ |\n| PB1 | **-0.01, 0.26** |\n| PB2 | 0.08, 1.02 |\n| PC1 | **0.33, 0.00** |\n| PC2 | **0.49, 0.45** |\n| PG1 | **0.20, 0.36** |\n| PG2 | **0.21, 0.64** |\n| PP1 | **0.11, 0.39** |\n| PP2 | **0.09, 0.56** |\n| CB1 | **0.00, 0.35** |\n| CB2 | **-0.03, 0.63** |\n| CC1 | 0.44, 2.60 |\n| CC2 | 0.40, 2.26 |\n| CG1 | **0.19, 0.32** |\n| CG2 | **0.11, 0.80** |\n| CP1 | **0.15, 0.38** |\n| CP2 | **0.07, 0.91** |\n| Average | 0.18, 0.75 |\n\nIn the absence of language-based cost rule descriptions, we advocate exploring **robust algorithmic designs** and shifting to an **online training setup**. In offline settings with potentially erroneous cost labels, it is difficult to reliably distinguish trustworthy from corrupted data. In contrast, under an online setting, one can **leverage existing safe and robust reinforcement learning frameworks by introducing an adversary that perturbs the cost labels of online samples with a certain probability** [3]. This enables learning policies that are both robust to label noise and maintain strong safety—representing a promising direction for future research.\n\n### Q3 Sensitivity analysis on $\\beta-$decay\n\nDifferent values of $\\beta_{\\text{end}}$ do affect reward and cost, but **the impact on cost remains within 0.1**, indicating minimal influence on policy stability. Instead, it offers CoPDT a useful mechanism for **fine-tuning performance during deployment**.\n\nWe agree that that studying the impact of $\\beta$-decay on the risk-reward trade-off is important. To this end, we conduct a sensitivity analysis of $\\beta_{\\text{end}}$ in **App. F.3 (Fig. 7)**, under the multi-constraint multi-budget setting across all 26 tasks.\n\nAs shown in the figure, there is a clear trend: **as $\\beta_{\\text{end}}$ increases from 0.5 to 0.9, both the average reward performance and cost performance of the policy show an increase trend, but the increase of cost remains within 0.1**. This indicates that when $\\beta_{\\text{end}}$ is smaller, the policy emphasizes safety satisfaction more strongly; whereas when $\\beta_{\\text{end}}$ is larger, the policy places more emphasis on reward maximization.\n\nSince $\\beta_{\\text{end}}$ is only applied during **evaluation**, this observation inspired us to explore an **Oracle setting**, where $\\beta_{\\text{end}}$ is slightly tuned for each task based on actual test-time performance. The results under this setting are shown in **Tab. 5 of App. F.3**. We observe that such post-hoc tuning during evaluation can further improve the overall safety performance of the policy.\n\n### Q4 Extending CoPDT to online setting\n\nWhile we focus on the offline safe RL setting in this work, we agree that extending CoPDT to online scenarios is a highly promising direction for future research. In **Sec. 4.4 (Fig. 4)** and **App. F.7 (Fig. 10–12)**, we have demonstrated that CoPDT can perform efficient adaptation when provided with a small number (e.g., 10) of expert trajectories, indicating its potential for low-data transfer.\n\nHowever, in fully online settings, directly applying supervised learning-based approaches such as Online Decision Transformer (ODT) [4] may lead to poor learning efficiency, primarily due to the difficulty of exploring trajectories that satisfy safety constraints. This poses unique challenges in safe exploration. To address this, new algorithmic designs will be required—for example, determining how to choose the initial CTG and RTG values during exploration, how to sample them dynamically, and how to update the policy under safety-aware objectives.\n\n**We consider this an important and challenging direction, and leave it as a key avenue for future work.**\n\n### Q5 Zero-shot generalization when learned with more similar tasks\n\nThanks for the valuable suggestion, **pre-training on a larger set of similar tasks can indeed effectively enhance CoPDT’s zero-shot generalization capabilities**.\n\nWe conducted **an additional experiment where we trained policies under a larger number of velocity thresholds** (from 2 to 6) in the AntVelocity and HopperVelocity environments. Specifically, in the 2-threshold setting, the thresholds were [2.57,2.62] for Ant and [0.37,0.74] for Hopper. In the 6-threshold setting, we used [2.42,2.47,2.55,2.57,2.62,2.67] for Ant and [0.37,0.42,0.52,0.62,0.67,0.74] for Hopper.\n\nFor evaluation, beyond the original zero-shot generalization tasks (**AntV2** with threshold 2.52 and **HopperV2** with threshold 0.56), we further designed two additional zero-shot generalization tasks: **AntV7** (threshold 2.4) and **HopperV7** (threshold 0.35). Notably, the V2 tasks correspond to **interpolation** generalization settings, while the V7 tasks represent **extrapolation** settings. All results were obtained under a budget of 10, and evaluated using 3 random seeds.\n\nThe experimental results are shown in the table below. It can be observed that when CoPDT is pre-trained on a larger number of similar tasks—similar to the setup in meta-RL—the model exhibits **significant improvements in safety performance** on both interpolation (V2) and extrapolation (V7) generalization tasks. The improvement is particularly notable in the Hopper environment, which may be attributed to the large threshold gap between the original two Hopper tasks, making pre-training on only two tasks insufficient. These results support the claim that our **constraint-prioritized prompt encoder not only captures task-specific memory but also possesses a certain degree of generalization capability to unseen constraints**.\n\n|  | CoPDT (2 similar tasks) | CoPDT (6 similar tasks) |\n| --- | --- | --- |\n|  | r↑  c↓ | r↑  c↓ |\n| AntV2 | 0.99, 2.18 | 0.98, 1.56 |\n| AntV7 | 0.99, 3.28 | 0.96, 2.34 |\n| HopperV2 | 0.52, 9.96 | 0.48, 0.04 |\n| HopperV7 | 0.24, 21.68 | 0.43, 4.57 |\n| Average | 0.69, 9.28 | 0.71, 2.13 |\n\n[1] Yu W, Gileadi N, Fu C, et al. Language to rewards for robotic skill synthesis[J]. arXiv preprint arXiv:2306.08647, 2023.\n\n[2] Xie T, Zhao S, Wu C H, et al. Text2reward: Reward shaping with language models for reinforcement learning[J]. arXiv preprint arXiv:2309.11489, 2023.\n\n[3] Li Z, Hu C, Wang Y, et al. Safe reinforcement learning with dual robustness[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.\n\n[4] Zheng Q, Zhang A, Grover A. Online decision transformer[C]//International Conference on Machine Learning. PMLR, 2022: 27042-27059."}, {"review_id": "sCZfwLtcm3", "reviewer": "Reviewer_gaMd", "summary": "This paper tackles the challenge in safe reinforcement learning: how to learn policies from offline data that can adapt to different safety requirements across multiple tasks. The authors focus on scenarios where an agent must handle various safety constraints (like different speed limits for different robots) and adapt to changing safety budgets (how much violation is tolerable) without direct environment interaction. The key insight is that existing decision transformer approaches struggle when faced with multiple constraints because they rely on manually specified reward targets (Return-To-Go) that often conflict with safety requirements (Cost-To-Go). When these conflict, the agent tends to prioritize rewards over safety, leading to constraint violations.", "questions": "- The results show constraint violations in zero-shot settings (e.g., HopperV2 with 9.96±3.06 cost vs 1.0 budget). This suggests the prompt encoder is memorizing specific constraint patterns rather than learning generalizable safety concepts. Have you investigated alternative architectures that could learn more compositional constraint representations? For example, using contrastive learning objectives that explicitly maximize distances between different constraint types while maintaining invariance to trajectory-specific features?\n\n- Could you please provide analysis on what the prompt encoder actually learns? Specifically, visualization of prompt embedding gradients with respect to different trajectory features might reveal whether it's capturing constraint semantics or trajectory statistics.\n\n- Would it be possible to integrate CoPDT with existing safety verification frameworks? For example, could the learned constraint representations be used to define control barrier functions?\n\n- Is it possible to derive PAC-style bounds on constraint violation probability given finite offline data?", "limitations": "-  The fundamental limitation to binary costs is not discussed, despite many real-world safety metrics being naturally continuous\n\n- Should explicitly state that CoPDT provides no formal safety certificates or probabilistic bounds, making it unsuitable for truly safety-critical applications without additional safeguards\n\nConstructive Suggestions:\n\n- Would it be possible to add a \"When NOT to use CoPDT\" section explicitly stating: systems requiring formal guarantees, continuous cost functions, strict real-time constraints, or high-stakes deployment without extensive testing", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Technical Strengths:\n\n- The constraint prioritized prompt encoder leverages the binary nature of costs to create separate neural pathways for safe and unsafe trajectory segments, amplifying the sparse cost signal that would otherwise be dominated by dense state-action information.\n- Rather than relying on fixed Return-To-Go values, the CPRTG mechanism learns a probabilistic model of appropriate reward targets conditioned on current safety budgets, solving the conflict between reward maximization and constraint satisfaction in decision transformers.\n-The prompt encoder maintains discriminative power even when evaluated on out-of-distribution trajectories (Medium1/Medium2 prompts), suggesting learned representations capture fundamental constraint properties rather than superficial trajectory statistics.\n\nTechnical Weaknesses:\n\n- The approach fails to develop robust constraint representations that transfer to unseen cost functions - the learned prompt embeddings are essentially memorizing specific constraint patterns rather than learning generalizable safety concepts.\n\n- The CPRTG generator struggles with high-variance reward distributions, as evidenced by velocity tasks showing systematically worse safety performance than navigation tasks with concentrated rewards.\n\n- The architecture cannot incrementally learn new constraints without retraining from scratch, as the prompt encoder's fixed dimensionality and the environment-specific components prevent online adaptation.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful reviews. We appreciate your feedback and hope that our response addresses your concerns.\n\n### Q1 More analysis on zero-shot generalization\n\nCurrently, for each task type, CoPDT is pre-trained using only 2 similar tasks. We acknowledge that this limited diversity may restrict the model’s ability to achieve stronger zero-shot generalization performance. **However, this limitation can be mitigated by pre-training on a broader set of similar tasks**.\n\nTherefore, we conducted **an additional experiment where we trained policies under a larger number of velocity thresholds** ($2 \\to 6$) in the AntVelocity and HopperVelocity environments. The goal is to examine whether exposing CoPDT to more similar tasks during training can improve its zero-shot generalization ability. Please refer to our response to **Q5 of Reviewer YyH8** for detailed experimental settings and results.\n\nIt can be observed that when CoPDT is pre-trained on a larger number of similar tasks—similar to the setup in meta-RL—the model exhibits **significant improvements in safety performance** on both interpolation (V2) and extrapolation (V7) generalization tasks. The improvement is particularly notable in the Hopper environment, which may be attributed to the large threshold gap between the original two Hopper tasks, making pre-training on only two tasks insufficient. These results support the claim that our **constraint-prioritized prompt encoder not only captures task-specific memory but also possesses a certain degree of generalization capability to unseen constraints**.\n\n### Q2 CPRTG generator with high-variance reward distributions\n\nWe acknowledge that this reflects a current limitation of the CPRTG generator under the **Gaussian modeling assumption**. However, we believe that **adopting more powerful generative models with strong multi-modal distribution modeling capabilities**—such as diffusion models or flow matching—offers a promising solution. Enhancing the CPRTG generator in this way is an important direction for future research.\n\n### Q3 Incrementally learn new constraints\n\nWe would like to clarify that **CoPDT is inherently capable of incrementally adapting to new constraints and environments**. \n\nIn the multi-constraint setting, both the prompt encoder and the DT policy in CoPDT adopt a three-layer architecture: the bottom and top layers are environment-specific encoders and decoders, designed to handle different state and action space dimensions across environments, as stated in **Sec. 3.1**. **The middle layer, which forms the core of both the prompt encoder and DT policy, is shared across all tasks and environments to enable effective knowledge transfer.**\n\nAs a result, when encountering new tasks with previously unseen state and action spaces, we can train new encoders and decoders for those tasks while **reusing the shared middle layers** of the prompt encoder and DT policy. This design naturally supports incremental adaptation without retraining the entire model from scratch.\n\nWe have evaluated this capability in an experimental setup described in **App. F.7 (Fig. 12)**, where we perform few-shot transfer to unseen robots whose state and action spaces were not present during training. The results show that CoPDT with full fine-tuning achieves better safety performance compared to learning from scratch, even in these entirely new environments—demonstrating the effectiveness of multi-task training and knowledge sharing.\n\n### Q4 What encoder learns?\n\nActually, our prompt encoder is **capable of effectively extracting and representing information from the cost signal**.\n\nSince the cost signal in our prompt encoder is **not used to compute gradients**, but instead serves as a **switch to select between sub-networks**, we are unable to provide traditional gradient-based interpretation methods. However, we propose an alternative and more intuitive visualization approach: **directly comparing the outputs of the same samples (s,a,r,s’) processed through $p_s$ and $p_u$**.\n\nTo do this, we collected 5,000 transition samples in PointCircle1 and passed them through both $p_s$ and $p_u$, obtaining two sets of 5,000 embedding vectors. While we are unable to include t-SNE visualizations of these vectors due to rebuttal limitations, we compute the **cosine similarity** between the mean vectors of the two sets, which yields a value of **0.415**. This relatively low similarity indicates that the encodings produced by $p_s$ and $p_u$ differ substantially, confirming that **the selection mechanism based on cost information has a significant effect on the encoding**.\n\nThese findings provide empirical support for the effectiveness of our constraint prioritized prompt encoder, showing that the cost signal meaningfully influences the learned task representation, even without direct gradient supervision.\n\n### Q5 Integrate CoPDT with existing safety verification frameworks\n\nIndeed, the constraint representations learned in CoPDT can be **naturally integrated with existing safety verification frameworks**, such as control barrier functions (CBFs).\n\nIn the case of neural-network-based CBFs, the constraint representation can be directly treated as part of the state and used as an additional input to the neural network, enabling task-adaptive safety filtering.\n\nFor non-parametric CBFs, one can construct a set of prototype task embeddings by generating constraint representations from a small number of trajectories for each training task. During testing, a trajectory from the target task can be used to compute its constraint representation, which is then matched to the most similar prototype. The corresponding CBF of the matched prototype task can then be applied to enforce safety in the target task.\n\n### Q6 PAC-style bounds on constraint violation probability？\n\nSince CoPDT is trained using the behavior cloning objective from DT, **it does not naturally lend itself to PAC-style theoretical guarantees** as commonly found in traditional TD-learning frameworks.\n\nTo complement this limitation, we provide empirical evidence in App. A by reporting **the performance gap between the learned policy and the behavior policy in the offline dataset**. These results highlight the importance of using the CPRTG generator to better align the RTG with the offline data distribution, thereby mitigating distributional shift and improving policy performance.\n\n### Q7 Adding a \"When NOT to use CoPDT” section\n\nThank you very much for your valuable suggestion. We will incorporate this section in the next version of the paper, explicitly stating:\n\n> \"Systems that require formal safety guarantees, continuous cost functions, strict real-time constraints, or high-stakes deployment without extensive testing are not suitable for deploying CoPDT.\"\n> \n\nThis clarification will help better delineate the scope and applicability of our method."}]

Author rebuttal:

Meta-review: This paper proposes encoding cost constraints to inform value prediction, which is a novel and well-motivated idea. The approach is clearly presented, and the experimental results show meaningful benefits across the evaluated settings. The reviewers were generally positively inclined toward the paper, recognizing both the conceptual contribution and the empirical evidence.

Some concerns were raised during the review process, including questions about the scope of the evaluation, the generality of the approach, and whether the novelty lies primarily in combining existing ideas. The authors provided clear and thoughtful responses to these points, which helped to address the main issues raised. Incorporating these clarifications into the final version will further strengthen the paper and make the contribution more evident.
Decision: Accept (poster)