A Provable Approach for End-to-End Safe Reinforcement Learning
Authors: Akifumi Wachi
Kohei Miyaguchi
Takumi Tanabe
Rei Sato
Youhei Akimoto
Venue: NeurIPS 2025 poster ()
Keywords: safe reinforcement learning
gaussian process
Primary area: reinforcement_learning
TL;DR: We propose a provable safe RL approach with end-to-end safety guarantee.

Abstract: A longstanding goal in safe reinforcement learning (RL) is a method to ensure the safety of a policy throughout the entire process, from learning to operation. However, existing safe RL paradigms inherently struggle to achieve this objective. We propose a method, called Provably Lifetime Safe RL (PLS), that integrates offline safe RL with safe policy deployment to address this challenge. Our proposed method learns a policy offline using return-conditioned supervised learning and then deploys the resulting policy while cautiously optimizing a limited set of parameters, known as target returns, using Gaussian processes (GPs). Theoretically, we justify the use of GPs by analyzing the mathematical relationship between target and actual returns. We then prove that PLS finds near-optimal target returns while guaranteeing safety with high probability. Empirically, we demonstrate that PLS outperforms baselines both in safety and reward performance, thereby achieving the longstanding goal to obtain high rewards while ensuring the safety of a policy throughout the lifetime from learning to operation.

A Provable Approach for End-to-End Safe Reinforcement Learning
Akifumi Wachi"
Kohei Miyaguchi'
Takumi Tanabe"
Rei Sato
Youhei Akimoto
*LY Corporation University of Tsukuba *RIKEN AIP
(akifumi.. wachi, kmiyaguc, takumi. tanabe, sato. retoo retooree)@@ycorp....o co. jo. akimoto@cs. tsukuba.ac.
Abstract
A longstanding goal in safe reinforcement learning (RL. is method to ensure the safety of a policy throughout the entire process, from learning to operation. However, existing safe RL paradigms inherently struggle to achieve this objective. We propose a method, called Provably Lifetime Safe RL (PLS), that integrates offline safe RL with safe policy deployment address this challenge. Our proposed method learns policy offline using returnconnditooned supervised learning and then deploys the resulting policy while cautiously optimizing a limited set parameters, known as target returns, using Gaussian processes (GPs). Theoretically, mathematical relationship between target and actual returns. We then prove that PLS finds near-optimal target returns while guaranteeing safety with high probability. Empirically, we demonstrate that PLS outperforms baselines both in safety and reward performance, thereby achieving the longstanding goal to obtain high rewards while ensuring the safety policy throughout the lifetime from learning to operation.
we justify the use of GPs by analyzing
Introduction
Reinforcement learning (RL) has exhibited remarkable capabilities in wide range of real problems, including robotics (32), data center cooling (34] finance [23), and healthcare [60] RL has attracted
significant attention through its successful deployment in language models [21 38 or diffusion
models 77. RL becomes core component of advanced Al systems that affect our daily lives, ensuring the safety of these systems has emerged as critical concern. Hence, while harnessing the immense potential of RL, we must simultaneously address and mitigate safety concerns [4].
Safe RL 18 20 is a fundamental and powerful paradigm for incorporating explicit safety considera- tions into RL. Given its wide range of promising real-world applications, safe RL naturally spans broad scope and involves several critical considerations in its formulation. For example, design choices must be made regarding the desired level of safety (e.g., safety guarantees are required in expectation or with high probability), the phase in which safety constraints are enforced (c.g..
post-convergence or even during training), and other related aspects 27 (5)
longstanding goal in safe RL to develop methodology with a safety guarantee throughout the entire process, from learning operation. However, existing safe RL paradigms inherently struggle to achieve this goal. In online safe RL, where an agent learns its policy while interacting with the environment, ensuring safety especially challenging during the initial phases of policy learning. While safe exploration (5T), sim-to-real safe RL 24 or end-to-end safe RL have been actively
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Offline <- Deployment Online
Target returns
Return- conditioned policy
Return- conditioned policy
Offline dataset
Actual returns
Return- conditioned
Fixed target returns after convergence
policy
Gaussian processes
Offline policy learning Absolute safety guarantee
Safe optimization of target returns
Long-term operation
High probability near-optimality and safety quarantees
High probability safe quarantee
Figure 1: A conceptual illustration of PLS. After learning a return-connoee conuin--onddtooned policy using offline safe RL, PLS optimizes target returns through safe online policy evaluation via Gaussian processes. key advantage PLS that safety is guaranteed at least with high probability the entire process.
studied, they typically rely strong assumptions, such (partially) known state transitions. Also, in offline safe RL, where a policy is learned from a pre-collected dataset, it remains difficult to deploy a safe policy in real environment due to distribution mismatch issues between the offline data and the actual environment, even though training can proceed without incurring any immediate safety risks.
Our contributions. We propose Provably Lifetime Safe RL (PLS), an algorithm designed to address the longstanding goal in safe RL. PLS integrates offfine policy learning with online policy evaluation and adaptation with high probability safety guarantee, as illustrated in Figure [ gins by training a policy using an offline safe RL algorithm based on return- couurn-oondiioned supervised learning (RCSL.. Given this resulting return-onnditoone policy, PLS then seeks optimize set of target returns by maximizing the reward return subject to a safety constraint during actual environ- mental interaction. Through rigorous analysis, demonstrate that leveraging Gaussian processes (GPs) for this optimization is theoretically sound, which enables PLS to optimize target returns in a Bayesian optimization framework. We further prove that, with high probability, the resulting target returns are near-optimal while guaranteeing safety. Finally, empirical results demonstrate that I) PLS outperforms baselines in both safety and task performance, and 2) PLS learns policy that achieves high rewards while ensuring safety throughout the entire process from learning to operation.
Specifically, PLS be-
2 Related Work
Safe RL [18 is promising approach to bridge the gap between RL and critical decision-making problems related to safety. constrained Markov decision process (CMDP, (3) a popular model for formulating safe RL problem. In this problem, an agent must maximize the expected cumulative reward while guaranteeing that the expected cumulative safety cost less than a fixed threshold.
Online safe RL. Although safe RI. in CMDP settings has been substantially investigated, most the existing literature has considered "online" settings, where the agent learns while interacting with the environment (55). Prominent algorithms fall into this category, as represented by constrained policy
optimization (CPO. M]) Lagrangian- -aaeed actor-critic 8, and primal-dual policy optimization [39 58). In online safe RL, satisfaction of safety constraints is not usually guaranteed during learning. and many unsafe actions may be executed before converging. To mitigate this issue, researchers have investigated safe exploration [5] 51 53). formal methods [7]. or end-to-end safe RL [11] 25]. These techniques, however, typically rely on strong assumptions (e.g.. known state transitions), and excessively conservative policies tend to result in unsatisfactory performance or inapplicability to complex systems. Therefore, simultaneously achieving both reward performance and guaranteed safety within the online safe RL paradigm is inherently difficult.

---

Offline safe RL. Offline reinforcement learning (RL) [3 40 dataset of previously collected experiences. Since the agent does not interact with the environment during training, no potentially unsafe actions are executed during learning. Extending this setup to incorporate explicit safety requirements has led to the area of offline safe RI. [30
trains an agent exclusively on fixed
56 In this context, the objective is to maximize expected cumulative reward while satisfying pre-specified safety constraints, all from a static dataset. Because the policy is never deployed during training, offline safe RL is especially appealing for safety-critical domains. Le et al. (30) pioneered this direction with an algorithm that optimizes return under safety constraints using only offline data. Liu et al. (37] proposed constrained decision transformer (CDT) that solves safe RL problems by sequence modeling by extending decision transformer [10] architectures from unconstrained to constrained RL settings. Despite such progress, offline safe RL still suffers from central difficulty: learned policies often become either unsafe or overly conservative, largely due to the intrinsic challenges of off-policy evaluation (OPE) in stateful environments [15
Versatile safe RI. Our PLS is also related to versatile safe RL, where an agent needs to incorporate set of thresholds rather than single predefined value. For example, in online safe RL settings, Yao et al. [59] proposes a framework called constraint-conditionee policy optimization (CCPO) that consists of versatile value estimation for approximating value functions under unseen threshold conditions and conditioned variational inference for encoding arbitrary constraint thresholds during policy optimization. Also, Lin et al. (35] proposes an algorithm to address offline safe RL problems with real-time budget constraints. Finally, Guo et al. [22] proposes an algorithm called constraint- conditioned actor-critic (CCAC) that models the relations between state- actiin distributions and safety constraints and then handles nut-of-distribution data and adapts to varying constraint thresholds.
3 Problem Statement
We consider sequential decision decision-making problem in a finite-horizon constrained Markov decision process (CMDP, 3) defined as tuple
M = S1,
(I)
where S is a state space, A is an action space, and P 8 A + A(S) is the state transition probability, where (XX) denotes the probability simplex over the set X. For ease of notation, we define a transition kernel Pr. : S x A + A(R2 x XS) associated with (P.r.g). Additionally, H Z+ is the fixed finite length of each episode, 81 E S is the initial state, r S x A + [0, 1] is the normalized reward function bounded in [o, 1]. While we assume that the initial state fixed to 81, our key ideas can be easily extended to the case of initial state distribution A(S). A key difference from a standard (unconstrained) MDP lies in the (bounded) safety cost function g : S x - (0. 1]. For succinct notation, we use s; and a, to denote the state and action at time t, and then define & := (st, ar, Tt, 9t) for all E [H], where T; = = a;) and g: = g(s;, at).
Episodes are defined as sequences of states, actions, rewards, and safety costs E == {Er)" e (S x A x RR) where St+1 ~ P(- st, a;) for all t E [H]] The t-th context o; of an episode refers to the partial history I₁ (51, E2. ... E(-1, for 1 S t < H + 1, where we let s 1 be
dummy state. Let X₁ == (S x A IR, ( S be the set of all t-th contexts and X = U X, be the sets of all contexts at time steps S 1 H.
We consider context-dependent policy tion, subsequently identifying a joint probability distribution paT on E such that a, T(x;) and (Te: St, 8141) Pr(se, ar) for all t [M] Given a trajectory T (Ei, ,,, returns are given Given
X - A(A) to map context action distribu-
by R(r)= = Σn" r(st, 0t) for reward and G(7) = -" g(a, ar) for safety cost, respectively. We now define the following two metrics that are respectively called reward and safety cost returns, where the expectation is taken over trajectories T induced by a policy π and the transition kernel Pr:
[ and Jg (T) = E, Pr [ā(7].
Jr (T) = E,
(2)
Dataset. We assume access to offline dataset D = ( where Z, is positive integer. Let B X + A(A) denote a behavior policy. The dataset D comprises n independent episodes
In this paper, we focus on context-dependent policies, broader class than the state-dependen policies that dominate most prior RL work.
3

---

generated by 5; that is, D ~~ distribution ((z)) is conditionally independent of past rewards frx [ri and safety costs {9h 14-1 given past states and actions Vt \ {rr, ( )-1
(P We also assume that, for any Iz E X, the behavior action 1t-1
Goal. We solve a versatile safe RL problem in the CMDP, where the safety threshold b is chosen within a set of candidate thresholds B := [0, H]. Specifically, our goal is to optimize single policy ı that maximizes (x) while ensuring that 1g (TT) is less than a threshold b B:
max (T) subject to
Jg(nT) - b, E
In contrast to the standard safe RL problems, we additionally address two fundamental and important challenges. First, our goal is to leam, deploy, and operate policy for solving (3) while guaranteeing safety throughout the entire safe RL process from learning to operation, at least with high probability. Second, we aim to train single policy that can adapt to diverse safety thresholds b E B.
4 Preliminaries
4.1 Return-Conditioned Supervised Learning
Return-conditioned supervised learning (RCSL) is a methodology to learn the return-conditional distribution of actions in each state and then define a policy by sampling from the action distribution with high returns. RCSL was first proposed in online RL settings [29] 43 46 and then extended
to offline RL settings [10 14]. In offline RL settings, RCSL aims at estimating the return-conditioned
I = R = R); that is, the action distribution conditioned the return R R [0, H and the context 24
behavior (RCB) policy (a x) = P (a.
X. According to the Bayes' rule, the RCB policy BR: : X + ((A) is written as the importance-weighted behavior policy
dB x) = f(R x fa)f((RR 2) d33(a) x),
(4)
where f(R .)) = aR p ( < R xt = x) and f(R x. = d Po P = x, 0t
Po
a)
respectively denote the conditional probability density functions of the behavior return
4.2 Decision Transformer
Decision transformer (DT, [10] is representative instance of the RCSL. In DT, trajectories modeled as sequences of states, actions, and returns (i.e., reward-to-go). DT policies are typically learned using the GPT architecture [I] with a causal self-attention mask; thus, action sequences are generated in an autoregressive manner. The pre- pre-training of DT can be seen as a regularized maximum likelihood estimate (MLE) of the neural network parameters
i argmin Pee
In po (at
F (0) + (())
(5)
nH
where P := (a x, Roee parametric model of conditional probability densities, and (()) 2 0 is penalty term representing inductive biases in parameter optimization. The output of DT is then given by Tō where NØ.8 denotes the policy associated with ( R).
4.3 Constrained Decision Transformer
Constrained decision transformer (CDT, B7) is a promising paradigm that extends the DT to constrained reinforcement learning by conditioning the policy on both reward and safety-cost returns. Specifically, CDT parameterizes policy to take states, actions, reward returns, and safety cost returns input tokens, and then generates the same length of predicted actions as output. Although practical implementations often truncate the input to a fixed context length, we simplify the analysis by assuming that the entire history æ; is provided to the model.
"Strictly speaking, the right- hand side of 4 can be ill-detined for certain x E X and a E A if either [(H x) or f( xr, a) are del-nefi, or if f(H x) = O. For our analysis, however, it suffices to impose (4) on 8R only when the right- hand side is well-defined.

---

50
50
60
40
40
Actal cozt
Actual cost
Actual cast
#0-
100
200
20
20
10
10
0
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
Target cost
Target cost
Target cost
(a) AntCircle
(b) HopperVelocity
(c) DroneRun
Figure 2: Relations between target safety cost return G and actual safety cost return (T(T) of pretrained CDT policies (red lines). Blue dotted lines represent y = Target reward returns fixed with the reward returns of the best trajectories included in the offline dataset. Observe that CDT policies suffer from unsuccessful misalignment between actual returns and target returns: (a) constraint violation, (b) excessively conservative behavior, and (c) both.
In the inference phase, a user specifies target reward return R and target safety cost return G the beginning of the episode and iteratively update the target returns for the next time step by R&+1- = R& Tn and G++1 G. 9r with R; R and G₁ G. Since the target returns play critical roles in the CDT framework, we explicitly add them the notations ा to emphasize the dependence on the pair of target returns == (R,, G); that is, let us denote ā (a x)) and define Z to be the set of all that feasible. Crucially, since CDT is a variant of RCSL that extends DT to constrained RL settings, the mathematical discussions are also true with CDT by replacing R with z, by defining f( 2) in or po
G') in (5). for example.
Safety issues of CDT policies. Ideally, we desire to align actual returns with target returns; that is, J, (TT.)) R R and (T.)) = G for 2 (R, G). This is why the target reward return R is typically set to be the maximum return included in the offline dataset, while the target safety cost return G is set to the safety threshold. Unfortunately, however, the actual returns are not necessarily aligned with the correct target returns. As evidence, Figure (isowss shows the empirical relations between target returns and actual returns of CDT policies. Specifically, actual returns may differ from corresponding target returns, and their differences vary depending the tasks pre-trained CDT models.
5 Theoretical Relations Between Target and Actual Returns
In Figure 2 while observe discrepancies between the target and actual returns, there seem to be some relations that can be captured using data. Our goal here is to theoretically understand when and how closely the CDT policy "Ø achieves the target returns, z. Unfortunately, however, given the architecture and learning complexity of CDTs, analyses without any assumptions; hence, we first list several necessary assumptions.
almost impossible conduct such theoretical
Assumption 1 -Near-deterministic transition). Let . Pr(r Sr'. S g' s, a. } be the corresponding density function. There exist deterministic maps .. s'(,)), and small constants Cg. Cs, 5 2 such that P. (. 8, a) S 6g for all ||g g(s, a) > 6 and P(s' s'sss s, S e, for all E S and A.
(r, g) denote pair of reward and safety cost.
Also, let (q s, a) aq
Assumption D is more general than that used in Brandfonbrener et al. because I) ours for multiobjective settings and 2) we consider d-neighborhood rather than exact equality (i.e.. 8 0). Especially, the second extension is beneficial since we can analyze theoretical properties of CDT policies optimized based continuous reward and safety cost, whereas Brandfonbrener et al. [9] effectively limits the scope of application to the problems with discrete rewards. This is a significant extension because safe RL problems typically require the agent deal with safety constraints with continuous safety cost functions and thresholds. Moreover, similar (near- Jdeterministic assumptions
common in notable safe RL literature [5] 10 (33)
We then make assumptions about the conditional probability density function of the behavior return; that is, of defined in 4 With slight extension from R to we assume the following three conditions on f(z IT), with fixed to value of interest.
Assumption 2 (Initial coverage). 7 = f(z 11) > 0.
5

---

Assumption 3 (Boundedness), C₂ == zex f(z x) < cc.
Assumption 4 (Continuity). cz (=(6) == sup2' =/= $26, 526 NeX If(z'
x) {(z
2x) co is small.
Finally, we assume the expressiveness and regularity of the regularized model (P, ②) in to control the behavior of the MLE, 9. The following assumptions are fairly standard and borrowed from Van der Vaart [52]: therefore, for case understanding, we will make informal assumptions below. See Appendix D.3 for the formal presentations of these assumptions.
Assumption 5 (Soft realizabiliity, informal). There exists 6 O such that 5g and TT are close to each other regarding the KL divergence and ((0") is small. See Assumption 14 for a formal version.
Assumption 6 (Regularity, informal). P and 4 are 'regular' enough for to be asymptotically normal. See Assumption T5 for formal version.
Finally, we present a theorem that characterizes the relation between target and actual returns. Theorem 1 (Relation between target and actual returns). For any policy æ, let us define J (T) = Jg(T)) Also, let T denote the policy obtained by the algorithm, which is characterized by a set of target returns z = (R, G). Recall that n is the number of trajectories contained in the offline have
dataset. Then, under Assumptions
((T
Vn F(z)
< + OP
(6)
where E(x) is small bias function and F [0, HJ GP(O, k), whose precise definitions are given in Theorems fand respectively. Here, (-(.) (-) the probabilistic small-o notation, i.e., b,, op(a,, implies lim,,> ((b,,/a., > 0, Ve O.
R23 is a sample path of Gaussian process
See Appendix E for its formal statement and complete proof. Intuitively, the difference between the target and actual returns is decomposed into an unbiased Gaussian process term H²F(z)/ /n, a small bias term c==), and asymptotically negligible term (r/1/v/) Remark 1 (Smoothness). Examining the explicit form of the covariance function k( reveals that F()) is smooth (under suitable conditions). Specifically, the smoothness of F is known to be
closely matches that of & (Corollary in (13). ). For more details, see Remark 9
is important to clarify the role of our assumptions. While Assumptions I 6 are necessary for the rigorous analysis in Theorem 1, we designed the PLS framework itself to be more general meta- algorithm. Our experiments show that PLS remains robust and safe even when these assumptions, such as near-deterministic transitions, do not strictly hold. This suggests the core framework is applicable well beyond the conditions required for the theoretical guarantees.
Provably Lifetime Safe Reinforcement Learning
We finally present Provably Lifetime Safe Reinforcement Learning (PLS), simple yet powerful approach that advances safe RL toward the longstanding goal of end-to-end safety.
As illustrated in Figure PLS begins with offline policy learning from a pre-collected dataset. Since RL agents are most prone to violating safety constraints during the early phases of learning, this offline learning step is particularly beneficial for ensuring lifetime safety. Also, a key idea behind PLS is the use of constrained RCSL (c.g., CDT) for this offline policy learning step. This approach yields a return- conuu--coditioned policy that enables control over both reward and safety performance through a few significant parameters. In the case of single safety constraint, all we have to do is optimize wwo-dimensional target return vector. Therefore, this method offers several advantages, including computational efficiency and enhanced controllability of policy behavior.
Hereinafter, we suppose there is a pre-trained policy obtained by constrained RCSL. For simplicity, denote such return-conditioned policy मх characterized by target reward and safety cost returns 2 (R, G) while omitting the neural network parameters A.
6.1 Characterizing Reward and Safety Cost Returns via Gaussian Processes
Guided by Theorem we employ GPs to model the mapping from a target return vector = (R, G) to the actual returns (Tx) == (J+(Tx). (T))). We formulate this as supervised learning problem

---

with the dataset (Tx, ))1 where 21, :2 E Z is a sequence of target returns. For tractability, we discretize the search space, yielding a finite candidate set 2 with cardinality |2]. While collecting such data, we sequentially choose the next target returns 2 E 2 that maximize the actual reward return (T=) subject to the safety constraint (i.e., (T())) - b). The measured returns
are assumed to be perturbed by i.i.d. Gaussian noise for sampled inputs Zx = [z]...., xN S Z. Thus, for E {r,g] (the symbol is used as a wildcard), we model the noise- peseeppeturbed observations ) + wo.j with Wo.j ~ (0, ve), for all j E [N].
by Yo. (T
A GP stochastic process that is fully specified by a mean function and a kernel. We model the reward and safety cost returns with separate GPs:
(x=) GP( (z, and (T() ~ GP(M, (2,
(7)
where Mo (z) is mean function and (z, æ)) is covariance function for o E {r. g}. In principle, Jr ) and (Tx) may be correlated (i.c., off-diagonal elements in & is non-zero in Theorem D. but ignore these cross-come cross-oorrelations and learn each GP independently for simplicity.
Then, given the previous inputs N = ( ZN] and observations yo.N = [yo.1,... Yo N. we can analytically compute a GP posterior characterized by the the mean (o,N(z) = (o, (T.,n)((K...)) v²T,v)= No and variance (2)
+ "ko,N) where (z, = [2), z),....,.... (zN, =)] and KO,N is the positive definite kernel matrix [z,(z, 8) =., and In E RNNN is the identify matrix. Finally, we assume that Jg(Tz) ) is T--ipschitz continuous with respect some distance metric d(::,) in 2. This assumption is rather mild and is automatically satisfied by many commonly-used kernels [45 48)
(z, 2) (z,,N(2T((K,,
6.2 Safe Exploration and Optimization of Target Returns
Our current goal is to find the optimal pair of target returns (R. G) that maximizes (Tz)
while guarantceing the satisfaction of the safety constraint (i.c., (T²) S b) according to GP-based inferences. For this purpose, we optimistically sample the next target returns z while pessimistically ensuring the satisfaction of the safety constraint, as conducted in Sui et al. [49].
A key advantage of using GPs is that we can estimate the uncertainty of the actual returns Jr and Jg. To guarantee, high probability, both constraint satisfaction and reward maximiza- tion, for each function o E {r. gr. we construct confidence interval defined as Ro,v(2) == [Mo,N-1(z) + Co.,N N where No,N E IR,, is a positive scalar that balances exploration and exploitation. These coefficients a, and 0g crucial in the performance of PLS, and principled choices for these coefficients have been extensively studied in the Bayesian optimization literature (c.g.. 12 45)). Thus, following Srinivas et al. 45). we define
Or.,j = Rg.j = V 2 logg (6A))
(8)
where A E [o, 1] is the allowed failure probability, and 11 in 88 is the circle ratio, not policy.
To expand the set of feasible target returns x while satisfying the safety constraint, we use alternative confidence intervals Av( (z) = (2) n.g (z) with Az( = 10. b] so that Av are sequentially contained in A for all N. We thus define an upper bound un (z) = max (z) and lower bound of xx = min Ax (z), respectively. Note that uy is monotonically non-increasing and &
monotonically non-decreasing, with respect to N.
Safe exploration. Using the GP upper confidence bound, we construct the set of safe target returns by VN UzEVx 1 (=' E Z uN(=) + L- d(zz z²) < b). At each iteration, PLS computes a set of that are likely to increase the number of candidates for safe target returns. The agent thus picks with the highest uncertainty while satisfying the safety constraint with high probability; that is,
2 = argmax (uN(2) - (x(()) with EN = (2 E VN : (x(2)) > 0),
Fv
where (2) = 2 z VN quantifies the potential enlargement of the current safe set after obtaining a new sample z.
(x) L- d(2, < 6}1. Intuitively, ex(-) optimistically
Reward maximization. Safe exploration is terminated under the condition max (uv (z) (x)) S, where R4 is a tolerance parameter. After fully exploring the set of safe target
7

---

returns, we turn to maximizing J+(+) under the safety constraint. Concretely, we choose the next target returns optimistically within the pessimistically constructed set of safe target returns by
N
argmax arg.ax(4r,, O,, or,n(2)).
(10)
=EV/x
6.3 Theoretical Guarantees Safety and Near-optimality
We provide theoretical results on the overall properties of PLS. We will make an assumption and then present two theorems on safety and near-optimality. The assumption below is fairly mild in practice, because we can easily ensure that the relurn-conditioned policy meets the safety constraint by conservatively choosing small target returns, R and G. See Appendix [orr the full proofs.
Assumption 7 (Initial safe set), There exists singleton seed set Zg that is known to satisfy the safety constraint; that is, for all 2 E Zo. Jn(TT) ) S b holds.
Theorem (Safety guarantee). At every iteration j. suppose that Ogg is set as in ( and the target
returns 2j are chosen within y/. Then, Jg(TT, < holds - i.e. the safety constraint is satisfied- for all j 2 0, with probability of least A.
Intuitively, because PLS samples the next target returns 2 so that the GP upper bound u(z) is smaller than the threshold b, the true value (=²) is guaranteed be smaller than b with high probability under proper assumptions. Moreover, since PLS learns the return- conni-condiiooned policy offfine, Theorem 2 leads to an end-to-end safety guarantee, ensuring that the constraint is satisfied from learning to operation, with at least a high probability.
Theorem (Near- optemalpti Set ar,y as in for all j 2 0. Let x* denote the optimal feasible target returns. For any & 2 0, define N₃ the smallest positive integer N satisfying
Co
SE
(11)
where C. = 1/logg1 + 2, Then, PLS a near-optimal z such that: (T²) 2 Jr (m²z)) E
(12)
with a probability at least 1 - A, after collecting N₃ GP observations for reward maximization.
Theorem 3 characterizes the online sample complexity of PLS. Following the analysis of Sui et al. [48], we can show that the safe exploration phase expands the estimated safe set until it contains the optimal target return vector x" after most N₁ 2 GP iterations. Consequently, Theorem 3 thus implies that PLS will find a near-optimal target return vector z using at most w (N, + N2.) trajectories, where E Z4 is the number of trajectories used for sample approximations of J, and J, for each GP update. Because PLS optimizes only the two-dimensional target return vector (i.e., R and G), it requires far fewer online interactions than conventional online safe RL algorithms, which is an essential advantage in safety-critical settings where every interaction is costly or risky.
7 Experiments
We conduct empirical experiments for evaluating our PLS in multiple continuous robot locomo- tion tasks designed for safe RL. We adopt Bullet-Safety- [19 and fffety-Gymnasium (2 benchmarks and implement our PLS and baseline algorithms using OSRL and DSRL libraries [6
Experimental details are deferred to Appendix K
Metrics. Our evaluation metrics are reward return and safety cost return, respectively normalized by
- and G Gnormalizea (T) = ((T) Recall that A((T) and G(Tr) are defined
Raonmalized (TT) = Riase, Feess.-R''.. RR'm.. as the evaluated cumulative reward and safety cost that are obtained by a policy π. In the above definitions, Ra R and R are the maximum and minimum cumulative rewards of the trajectories in the offline dataset D. Note that we call a policy safe Gnommaiizaa S 1.
Baselines.
We compare PLS against the following six baseline algorithms: BCQ-Lag, BEAR-
CPQ, COptiDICE, CDT, and CCAC. BCQ-Lag and BEAR-Lag are both L.grangian- -base methods that apply PID-Lagrangian 47 to BCQ [6] and BEAR [28) respectively. CPQ [57] is an offline safe
8

---

Table I: Experimental result with the safety cost threshold b = 20. The mean and standard deviation over 5 runs for each algorithm are shown. Reward and cost are normalized. Bold: Safe agents whose normalized cost is smaller than 1. Red: Unsafe agents. Blue: Safe agent with the highest reward.
Task
Metric
BCQ-Lag
BEAR-Lag
CPQ
COpiDICT
CDT
CCAC
PLS
Reward
0.79 0.05
0.01 0.01
0.63
0.01
0.72 1
0.22
0.00
0.78 0.06
Ant- Run
Safety cost
5.52 0.67
0.12 = 0.13
0.00 0.00
0.79 0.42
0.90 = 0.12
0.00
1:00=00 0.00
0.77 = 0.10
Reward f
0.59 0.18
0.58 # 0.24
0.00 = 0.00
0.16: 0.13
0.47 = 0.00
0.62
0.13
0.41 = 0.01
Ant-Circle
Safety cost I
2.28 + 1.50
3.37 = 1.71
0.00 = 0.00
2,98 + 3.55
2.23 = 0.00
1.24 = 0.55
0.77 = 0.05
Car Can-CCincle
Rewanl 1
0.65 0.19
0,76 1. 0.12
0.70 0.03
0.48 0.04
0.73 1 0.01
0.72
0.03
0.72 1. 0.01
Safety cost 4
2.17 1.10
2.74 +
0.01 0.07
1.85 1.48
0.87
0.29
0.88
Reward f
0.63 0.11
-0.03 0.02
0.19 0.01
0.69 +
0.57 = 0.00
0.82
0.05
0.59 = 0.00
Drone Run
Safety cost
3,91 2.02
0.00 = 0.00
0.00 0.00
3.48 0.19
0.34 = 0.29
7.62 0.37
0.50 = 0.44
Drone-Cirele
Reward 1
0.69 1 0.05
0.82 1. 0.06
-0.26 1. 0.01
0.22 0.10
0.60 1 0.00
0.37
0.14
0.59 1. 0.00
Safety cost L
1.92 0.64
3.58 + 0.74
0.14 0.39
0.68 0.46
1.12 + 0.06
0.74
0.24
0.90 0.08
Reward 1
1.00 0.01
-1.01 0.00
-1.01 +: 0.00
1.00 0.01
0.97 = 0.00
0.68
0.34
0,98 = 0.00
Ant. Velocity
Safety cost
3.22 0.60
0.00
0.00 0.00
6.60 1.07
0.60
0.21
=
Walker2d
Reward
0.78 0.00
0.89 1: 0.04
-0.02 1: 0.03
0.13 0.01
0.80 1: 0.00
0.81
0.07
0.79 d: 0.00
-Velacity
Safety cost |.
0.44 0.32
7.60 + 2.89
0.00 0.00
1,75 0.31
0.01 + 0.04
6.37
0.95
0.000 0.00
HalfChoctah
Reward 1
1.03 + 0.03
0.98 = 0.03
0.22 =: 0.33
0.63 + 0.01
0.96 = 0.03
0.84 = 0.01
0.99 = 0.00
VVelocity
Safety cost |
27.00 + 8.76
12:35 8.63
0.28 0.23
0.00 0,00
0.03 = 0.13
1.36 0.19
0.15 =: 0.19
Hopper VVelccity
Reward 1
0.85 0.22
0.36 4:
0.20 0.00
0.14 0.10
0.17
0.09
0.83
Safety cost
8.48 2.75
10.39 3.79
3.06 0.07
0.34 0.42
0.12 0.26
1.79 1.52
0.42 + 0.10
RL algorithm that regards safe acatin pairs. COptiDICE [3T], a member of DIstribution Correction Estimation (DICE) family, is specifically designed for offline safe RL and directly estimates the stationary distribution correction of the optimal policy in terms of reward returns under safety constraints. CDT [37] is a DT-based algorithm that learns a policy conditioned on the target returns, as discussed in Section 2 as a preliminary. Finally, CCAC (22) is a recent proposed offline safe RL algorithm that models the relationship between state-action distributions and safety constraints and then leverages this relationship to regularize critics and policy learning. We use offline safe-RL algorithms as basclines because standard online approaches often violate safety constraints during training and optimize objectives that diverge from ours. Although some safe exploration algorithms share similar goals, they rely on strong assumptions- succ known and deterministic transition dynamics [51 or access to an emergency reset policy 44 [54] that do not hold in our experimental setting.
actions as unsafe and learns the reward critic using only
Implementation of PLS. We use CDT 37 for offline policy learning as a constrained RCSL algorithm. The neural network configurations or hyperparameters for PLS are the same as the CDT used baseline. The key difference lies in how target returns are determined. In the baseline CDT, as a typical choice, we set the target reward return to the maximum reward return in the dataset and the target safety cost return to the threshold. In contrast, PLS employs GPs with radial basis function kernels to optimize the target returns for maximizing the reward under the safety constraint.
Main results. Table I summarrize our experimental results under a safety cost threshold of b = 20. Additional results, including Table for b 40, are provided in Appendix K Notably, PLS the only method that satisfies the safety constraint in every task. In contrast, every baseline algorithm violates the safety constraint in at least one task, which implies that a policy violating constraints could potentially persist in unsafe behavior in an actual environment. Moreover, PLS achieves the highest reward return in most tasks, which demonstrates its its superior overall performance in terms of reward and safety. In summary, while baseline methods suffer from either safety constraint violations or poor reward returns, PLS consistently delivers a balanced performance.
Computational cost. Although GPs are known to be computationally expensive, PLS only needs to optimize target returns two dimensions, = (R, G). Because the amount of training data for the GPs is fairly small until convergence (see also Figure 3 in Appendix [ their computational overhead not problematic. Consequently, the main source of computational cost in PLS stems from offline policy learning. Since PLS can adapt to multiple thresholds using a single policy by appropriately choosing target returns, it typically incurs lower overall computational cost than baseline algorithms (c.g., CPQ. COptiDICE), which require training separate policy for each threshold.

---

Safe exploration. As shown in Figure i Appendix K PLS successfully ensures safety not only after convergence but also while exploring target returns, which consistent with Theorem 2 In some cases, however, maintaining safety beyond the initial deployment can still pose a challenge in practice. Because our guarantee is prohabilistic and constructing accurate GP models not always feasible, a small number of unsafe deployments may occur.
8 Conclusion
We propose PLS as a solution to a longstanding goal in safe RL: achicving end-to-end safety from learning to operation. PLS consists of two key components: (1) offline policy learning via RCSL and (2) safe deployment that carefully optimizes target returns on which the pre-trained policy is conditioned. The relationship between target and actual returns is modeled using GPs, an approach justified by our theoretical analyses. We also provide theoretical guarantees on safety and near- optimality, and empirically demonstrate the effectiveness of PLS in safe RL benchmark tasks.
Limitations. Our work has several limitations that open avenues for future research. First, while PLS guarantees near-optimal target returns, as established in Theorem [ this does not directly translate into achieving a ncar-optimal policy. Second, our current framework does not update the policy network with new online data; that is, the policy is fixed after the initial offline training phase. Extending PLS to an offine-to-online setting where the policy continually learns while preserving safety guarantees is a crucial next step. Finally, while our experiments demonstrate strong performance, further evaluation in more complex and highly stochastic environments is needed to fully assess the practical robustness of our theoretical assumptions and the scalability of the approach.
References
[1] J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In International Conference on Machine Learning (ICML), pages 22-31, 2017.
[2] M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu. Safe reinforcement learning via shielding. In AAAI Conference Artificial Intelligence (AAAI), 2018.
[3] E. Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.
[4] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in Al safety, arXiv preprint
2016.
[5] F. Berkenkamp, M. Turchetta, A. Schoellig, and A. Krause. Safe model-based reinforcement learning with stability guarantees. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[6] S. Bhatnagar and K. Lakshmanan. An online actor-critic algorithm with function approximation for constrained Markov decision processes. Journal of Optimization Theory and Applications, 153(3);688-700)) 2012.
[7] K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine. Training diffusion models with reinforcement learning. In International Conference on Learning Representations (ICLR), 2024.
[8] V. S. Borkar. An actor-critic algorithm for constrained markov decision processes. Systems & 2005.
control letters, 54(3):207-213, 2005.
[9] D. Brandfonbrener, A. Bietti, J. Buckman, R. Laroche, and J. Bruna. When does return- conditioned supervised learning work for offline reinforcement learning? Advances in Neural Information Processing Systems (NeurIPS), 35:1542- 1553,, 2022.
[10] I. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in Neural Information Processing Systems (NeurIPS), 14:15084-15097 2021.
[11] R. Cheng, G. Orosz, R. M. Murray, and J. W. Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAÍ conference artificial intelligence (AAAI), volume 33, pages 3387-3395, 2019.
10

---

[12] S. R. Chowdhury and A. Gopalan. On kernelized multi-armed bandits. In International Conference on Machine Learning (ICML), pages
2017.
[13] N. Da Costa, M. Pförtner, L. Da Costa, and Hennig. Sample path regularity of Gaussian processes from the covariance kernel. arXiv preprint arXiv:2312. 2023.
[14] S. Emmons, B. Eysenbach. I, Kostrikov, and S. Levine. RvS: What is essential for offline RL via supervised learning? In International Conference on Learning Representations (ICLR), 2021.
[15] J. Fu, M. Norouzi. O. Nachum, G. Tucker, A. Novikov, M. Yang. M. R. Zhang, Y. Chen, A. Kumar, C. Paduraru, et al. Benchmarks for deep off-policy evaluation. In International Conference on Learning Representations (ICLR), 2021.
[16] S. Fujimoto, D. Meger, and D. Precup. Off-po deep reinforcement learning without explo- ration. In International Conference on Machine Learning (ICML), pages 2052-2062, 2019.
[17] N. Fulton and A. Platzer. Safe reinforcement learning via formal methods: Toward safe control through proof and learning. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
[18] J. Garcia and F. Fernández. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research (JMLR), 16(1): (6()):147--148, 2015.
[19] S. Gronauer. Bullet-safety-gye gym: A framework for constrained reinforcement learning. Technical report, mediaTUM, 2022.
[20] S. Gu, L. Yang, Y. Du, G. Chen, F. Walter, J. Wang, and A. Knoll. A review of safe reinforcement learning: Methods, theory and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[21] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang. X. Bi, et al. Deepseek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv 2025.
preprint
[22] 7. Guo, W. Zhou, S. Wang, and W. Li. Constraint-conditioned actor-critic for offline safe reinforcement learning. International Conference on Learning Representations (ICLR), 2025.
[23] B. Hambly, R. Xu, and H. Yang. Recent advances in reinforcement learning in finance. Mathematical Finance, 3333)3337-503 2023.
[24] K.-C. Hsu, A. Z. Ren, D. P. Nguyen, A. Majumdar, and J. F. Fisac. Sim-to-lab-to-real. Safe reinforcement learning with shielding and generalization guarantees. Artificial Intelligence, 314: 103811, 2023.
[25] N. Hunt, N. Fulton, S. Magliacane, T. N. Hoang, S. Das, and A. Solar-Lezama. Verifiably safe exploration for end-to-end reinforcement learning. Proceedings of the 24th International Conference on Hybrid Systems: Computation and Control, pages 1-11, 2021.
[26] J. Ji, B. Zhang, J. Zhou, X. Pan, W. Huang, R. Sun, Y. Geng, Y. Zhong, J. Dai, and Y. Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[27] H. Krasowski, J. Thumm, M. Müller, L. Schäfer, X. Wang, and M. Althoff. Provably safe reinforcement learning: Conceptual analysis, survey, and benchmarking. arXiv preprint arXiv:2205.067: 066500 2022.
[28] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
arxiv preprint [29] A. Kumar, X. B. Peng, and S. Levine. Reward-conditioned policies. 1xXiv:1912.13465 2019.
[30] H. Le, C. Volohhn,, and Y. Yue. Batch policy learning under constraints. International Conference Machine Learning (ICML), pages 3703-3712, 2019.

---

[31] J. Lee, C. Paduraru, D. Mankowitz, N. Heess, D. Precup, K.-E. Kim, and A. Guez. COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference on Learning Representations (ICI.R), 2021.
[32] S. Levine, C. Finn, T. Darrell, al. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research (JMLR), 17(1):133--1333, 2016.
[33] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
[34] Y. Li, Y. Wen, D. Tao, and K. Guan. Transforming cooling optimization for green data center via deep reinforcement learning. IEEE transactions on cybernetics, 50(5):2002-200 2019.
[35] Q. Lin, B. Tang, Z. Wu, C. Yu, S. Mao, Q. Xie, X. Wang, and D. Wang. Safe offline rein- forcement learning with real-time budget constraints. In International Conference on Machine Learning (ICML), pages 21 2117-21152. PMLR, 2023.
[36] Z. Liu, Z. Guo, H. Lin, Y. Yao, J. Zhu, Z. Cen, H. Hu, W. Yu, T. Zhang, Tan, et al. Datasets and benchmarks for offline safe reinforcement learning, arXiv preprint arXiv:2306.093000 2023.
[37] Z. Liu, Z. Guo, Yao, Z. Cen, W. Yu, T. Zhang, and D. Zhao. Constrained decision transformer for offline safe reinforcement learning. In International Conference on Machine Learning (ICML), 2023.
[38] L. Ouyang, Wu, X. Jiang, D. Almeida, C. Wainwright, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[39] S. Paternain, M. Calvo-Fullana, L. F. Chamon, and A. Ribeiro. Safe policies for reinforcement learning via primal-dual methods. arXiv preprint 1rXiv:1911.09101, 2019.
[40] R. F. Prudencio, M. R. Maximo, and E. L. Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[41] A. Radford. Improving language understanding by generative pre-training. OpenAl, 2018.
[42] H. Satija, S. Thomas, J. Pineau, and R. Laroche. Multi-objective SPIBB: Seldonian offline policy improvement with safety constraints in finite MDPs. In Advances Neural Information Processing Systems, volume 34, 2021.
[43] J. Schmidhuber. Reinforcement learning upside down: Don't predict rewards-just map them to 2019.
actions. arXiv preprint
[44] A. Sootla, A. Coven- J. Wang, and H. Bou Ammar. Enhancing safe exploration using safety state augmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[45] N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In International Conference on Machine Learning (ICML), 2010.
[46] R. K. Srivastava, P. Shyam, F. Mutz, W. Jaskowski, and J. Schmidhuber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv:1912.02877 2019.
[47] A. Stooke, Achiam, and P. Abbeel. Responsive safety in reinforcement learning by PID Lagrangian methods. In International Conference on Machine Learning (ICML), 2020.
[48] Y. Sui, A. Gotovos, J. W. Burdick, and A. Krause. Safe exploration for optimization with Gaussian processes. In International Conference on Machine Learning (ICML), 2015.
[49] Y. Sui, V. Zhuang, J. W. Burdick, and Y. Yue. Stagewise safe Bayesian optimization with Gaussian processes. In International Conference Machine Learning (ICML), 2018.
12

---

[50] G. Thomas, Y. Luo, and T. Ma. Safe reinforcement learning by imagining the near future. Advances Neural Information Processing Systems, 34:13859-13869 2021.
[51] M. Turchetta, F. Berkenkamp, and A. Krause. Safe exploration finite Markov decision processes with Gaussian processes. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
[52] A. W. Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
[53] A. Wachi and Y. Sui. Safe reinforcement learning in constrained Markov decision processes. In International Conference on Machine Learning (ICML), 2020.
[54] A. Wachi, W. Hashimoto, X. Shen, and Hashimoto. Safe exploration in reinforcement learning: A generalized formulation and algorithms. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[55] A. Wachi, X. Shen, and Y. Sui. survey of constraint formulations in safe reinforcement learning. In International Joint Conference Artificial Intelligence (IJCAI), pages 8262-8271, 2024.
[56] R. Wu, Y. Zhang, Z. Yang, and Z. Wang. Offline constrained multi-objective reinforcement learning via pessimistic dual value iteration. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[57] H. Xu, X. Zhan, and X. Zhu. Constraints penalized Q-learning for safe offline reinforcement learning. In AAAI Conference on Artificial Intelligence (AAAI), 2022.
[58] L.. Yang and M. Wang. Reinforcement learning feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning (ICML), 2020.
[59] Y. Yao, Z. Liu, Z. Cen, J. Zhu, W. Yu, T. Zhang, and D. Zhao. Constraint-conditionee policy optimization for versatile safe reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 36:12555- 12568, 2023.
[60] C. Yu, Liu, S. Nemati, and G. Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.
13

---

Appendix
A
Nomenclature
For readability, we present the main variables and functions below as nomenclature table.
Symbol
Description
de
Action time step f. Safety threshold for the cumulative safety cost. Conditional probability density function of the behavior return.
b
J(D)
Safety cost function, bounded in [0, 1].
g
G
The target safety return.
(T))
The observed cumulative safety cost for trajectory т. The fixed, finite length of each episode (horizon). pair of reward and safety returns, (J, (T), (=)) The expected cumulative reward return for policy The expected cumulative safety cost return for policy π.
H
((x)
(T
(T(
k(5))
Covariance (kernel) function for Gaussian Process. Lipschitz constant.
The number of trajectories in the offline dataset D. The number Gaussian Process (GP) observations or iterations. A parametric model of conditional action probability densities. State transition probability function, P Transition kernel associated with (Prr,g). Reward function, bounded in [0, 1]. The target reward return.
N
S x A - A(S).
Pr
r
R
R(())
The observed cumulative reward for trajectory т. State at time step I.
r
The context at time step A pair of target returns, G).
Parameter balancing exploration and exploitation in Bayesian optimization. The retur-condiitoned behavior policy used to generate the offline dataset D.
8
A
The allowed failure probability for safety guarantees (e.g.. 1 - A).
E(2)
A small bias function in the theoretical analysis of returns. Initial coverage, defined (=)(a1).
(())
Mean function for Gaussian Process.
A policy that maps context action distribution, :''' A(A).
T
A return-conditioned policy characterized by target returns 2. A trajectory, (&,, & ..., {
0
0
The parameters of the neural network policy model.
The maximum likelihood estimate of the model parameters 0.
An episode or trajectory, {
&
A tuple at time defined as (8t, a,, 9t)
C
A tolerance parameter for terminating safe exploration.
The number of trajectories used for sample approximations of returns for each GP update.
Broader Impacts
We believe that our proposed approach PLS plays significant role in enhancing the benefits associated with reinforcement learning while concurrently working to minimize any potential negative side effects. However, must be acknowledged that any reinforcement learning algorithm, regardless of its design or intended purpose, is intrinsically susceptible to abuse, and we must remain cognizant of the fact that the fundamental concept underlying PLS can be manipulated or misused in ways that might ultimately render reinforcement learning systems less safe.
14

---

C Pseudo Code of PLS
For completeness, will present pseudo code of our PLS.
Algorithm Provably Lifetime Safe Reinforcement Learning (PLS)
1: Input: Pre-collected dataset D, safety threshold b, safe singleton set Zo. Lipschitz constant L
2:
3: // Offline policy Learning (safe with probability 1)
4: Train a return ceturn-conditioned policy иs from D via constrained RCSL
5:
6: // Safe exploration (safe with high probability)
7: Initialize Vo with Zo
8: for N = 1,...,,.,,.. ,.. Ni do
9: VNv - U2ENN-1 { E (z) + L 5
ex(2) + (2' E \' (g. L. d(z,2') s b]]
11:
EN (z VN (z) > 0)
12: IN + argmax EEEy Ey (o,,N(z) (z,)(z))
13: Update GPs using the reward and safety cost observations (TTx)) and Jg (TxN)
end for
15:
// Reward maximization (safe with high probability)
17: for N = N₁ + +1 1....., N₂ N₂ do
18:
NN - U>> E UzEr,, Z (g,N(z) + L ((2zz2)) bb}
19: NN +- argmax (z)
20: Update GPs using the reward and safety cost observations J, (TT and Jz(TZN)
21: end for
22:
23: // Operation (safe with high probability)
24: while true do
Continue to use nn as target returns for long-term operation.
26: end while
D Preliminaries of Theoretical Analyses
As more general formulation of the problem. define multi-objective MDP characterized by m reward functions, where m an arbitrary positive integer. Our theoretical analyses in the main paper are a specific case of m 2 compared to those we will present the following.
D.1 Multi-objective Reinforcement Learning
Episodes are sequences of states, actions, and rewards E ((s,, de, r:) (S x A R where H 2 0 is time horizon and m 2 is the number of reward dimensions. The 1-th context dt of an episode refers to the partial history
($1 01, .i...., St-1, 0r-1: 1t)
(13)
for 1 St be the set of all i-th contexts and X' = - X, the sets of all contexts steps
where let s = 1 be dummy state. Let X; == (S A R S
H.
With fixed initial state and transition kernel Pr S A + A(R" x S), we consider the Markov decision process (MDP) M - AS, H, s1, Pr) Pr) Under M. every (context-dependent) policy ग. X + A(A) identifies a probability distribution pa on E\t such that a, ~ T and St+1) (s, az) for all
Assumption 8 (Bounded reward). For any policies n, we have -ammmosst surely 0 0$ ri I 1 for 1 t t & H and < S m.
OOur analysis can be easily extended to sı being stochastic.
15

---

Assumption 9 (Near-determinittic transition), There exist deterministic maps f(:,-), ((()))) and small constants e,, Es. 6 2 0 such that, if (r, s') (r)
the reward density pr(r''s, a) = are Pr(r I ss a H well-defined and bounded by 6r outside the S-neighborhood of F(s,a), i.e., supr:]I--((sa))ll.. =( -8 Pr (r|s, a) S Cr. and
2. the successor state s' coincides with 8' a) with probability of at least 1 - Es.
for all 8 and a E A.
(1P3)n be collection of Let B: + A(A) be behavior policy and D { i..d. copies of episodes generated by 3.
Assumption 10 (Reward-indepeeddet behavior). The behavior action distribution ((xe), Ie E X, is conditionally independent of the past rewards {rn - given the past states and actions x (rr T==1
1f-1
Let J (T) denote the multi-dimensional policy value of n
(TT) (m E [ R
(14)
where R = Σ/= Tr denotes the return of episode and the superscript ग of E" signifies the dependency on P
The aforementioned setting leads to constrained RL problems where a policy aims to maximize one dimension of the policy value J1 (TT much as possible while controlling the other dimensions to satisfy constraints J& (T) S 6y with certain threshold 6І. R. for S S m. More specifically, T1 and respectively correspond to and g in the main paper.
D.2 Return-condittoned supervised learning
Return-conditioned supervised learning (RCSL) is a methodology of offline reinforcement learning that aims estimating the return- -ttun-cooditioned behavior (RCB) policy (R(a(z) == P (a, a|ı; x, R R), the action distribution conditioned on the return R E [0, H|TT as well as the context I X. According to the Bayes' rule, the RCB policy SR X' ((A) is written as the snportance-weighte behavior policy
d3n(a x)= f(R x, d3(a 1x), f(R x)
(15)
where J(R 1)) AR P R xi =)) and SRR x) = aJR respectively denote the conditional probability density functions of the behavior return
2t = 0t a)
Return-based importance weighting (15) favors the actions that led the target return over those that did not. Hence, intuitively, it is expected that 8R achieves
J (BR) R R.
(16)
This is the case under suitable assumptions. Thus solve multi objei-o reinforcement learning with RCSL by setting R to a desired value.
We assume the following conditions on f(R] a)), with R fixed to value of interest.
Assumption 11 (Initial coverage). 1).. f(R $1) > 0.
Assumption 12 (Boundedness). Cn supzex {R x) < co.
Assumption 13 (Continuity). cn( = supre N Rll 526, TEx [(R' (x) f(R(z) < oo is small.
"We abuse the notation r < r for r' E 1 to imply the multi dimensional inequality, i.e., ry S r/ for
all 1 j <
*Strictly speaking, the RHS of 15 may be ill-netined for some x E X' and a E A either f [R or
[R|r, a) are del-defined, or f(R 2) 0. However, it is sufficient for our analysis to impose [5 on 3R only
if the RHS well-defined.
16

---

D.3 Decision transformers
Decision transformer (DT) is an implementation of RCSL. More specifically, it seen as a regularized maximum likelihood estimation (MLE) method
H Σ
i argmin liee
(0) xr. In (a
R 000)
(17)
nH
where := (a z, Ree is parametric model of conditional probability densities, typically constructed with the transformer architecture, and (() is a penalty term representing inductive biases, both explicit and implicit, in the procedure of parameter optimization. Here, ag 2 (c) and
R(() are the t-th action, the f-th context, and the return of the i-th episode E DD respectively. The output of decision transformer is then given by " where R denotes the policy associated with po(- R). Note that the original DT is for single-dimensional reward function, we presented
(17) by extending it to multi-dimensional settings.
We introduce some notation and conditions on the probabilistic model P and the penalty Q. Let us define regularized risk of relative 3R by
R&(()) = E Unir([[ ( (,,, + +()),
(18)
dissimilarity 8 and Tw, a expectation
where PKL(-1) denotes the Kullback-Leible divergence.
Assumption 14 (Soft realizability). cp, minøge (+) is small.
Remark 2. Assumption 14 is relaxation of standard realizability condition. That is, have ép, = 0 if 8 is realizable in P without penalty, i.e., there exists 0g E @ such that Te. R = 3R and (00)
Assumption 15 (Regularity). The following conditions are met.
i) θ is a compact subset of R4, d21 1.
ii) R. (0) admits unique minimizer 6* in the interior set t
iii) R.g. (0) is twice differentiable 0* with Hessian Ig- V2 > 0.
iv) The one-s stochastic gradient (a(a,, R) vol- in R) +(0)) is locally bounded in expectation as
(aalaalzer R2) 10]12]] <0
E Unit[[]
sup [
(19)
for every sufficiently small ball θ in O.
v) 0 O almost surely.
Remark 3. At first glance, ii) the unique existence of 0* and iii) the positive definiteness of the Hessian seem restrictive for peeemarraetrizedd models, including transformers. However, we note that these conditions may be enforced by adding a tiny, strongly convex penalty to (()).
Remark 4. Similarly, å 6 can be also enforced by adding barrier function such (()) K Rhngg ( (dist(f, 11,,8, Rd O)/h), where h > 0 and K
respectively suitably small and large constants, dist F) = inføeg ||0 0'll2- and @binge max {0, 1 - -t).
E
Error analysis
Our goal here to understand when and how closely the output of decision transformer, TO.R achieves the target return, R. The following theorem summarizes our theoretical results, answering the above question.
17

---

Theorem 4. Under the assumptions of Theorems 5 to
we have
2
(( R F
S < + op
(20)
where F H| + R is a sample path of a Gaussian process with mean zemo and a(R) = 20 2Cm(H² (o))H(HHe))))Hmmmm + / is small bias function, where CR = max m 1}
and G, + C. Here, 0p(-) is the probabilistic small-o notation, i.e, bn op(an) signifies limm- PP Pllnn/un c) for all >> 0.
Remark 5. Theorem in the main paper is special case of Theorem of m = 2, which is presented slightly informal manner.
To derive Theorem consider the bias-variance decomposition
J( R J(BR) R [T RR) ((RR) + ((T.RR) [TT
(21)
RCSL
MLE
variance of MLE
and evaluate each term in RHS with Theorems respectively, through Appendices F.1 and
H.2
E.1
Bias of RCSL
The following theorem gives an upper bound the first bias term, showing that it is negligible under suitable conditions, such as the near-determinism of the transition and the regularity of the return density. The proof is deferred to Appendix F
Theorem 5. Suppose Assumptions
g to 13 hold.
Then,
2CR (H²E -P c²()(())
[1((8)) Rl I
(22)
1J
where e = + Eg and CR = max {CR,1] 1}.
few remarks follow order. First, we compare our result to previous one.
Remark 6. Theorem 5 can be considered as a complementary extension the previous result [9]. In particular, our result is applicable when the return density f(R[s1) is bounded away from 0 and 0o, while Theorem of [9] is not. On the contrary, Theorem [9] is applicable when there is nonzero probability of exactly R = ft while our result is not since [(R(s1) = oc.
Remark Our result also extends Theorem in Brandfonbrener et al. 9 in allowing the transition kernel Pr include small additive noises in the reward, i.e., 5 >
Below is generalization of (22 that is useful to understand what constitutes the upper bound.
Remark 8. Taking a closer look at the proof of Theorem we can conclude
2CR (H?E+ δн) ""= HcR(())
[J(R)) Rl S
(23)
R
where 6, is the additive noise tolerance specific to the t-th transition. In other words, the contributions of these additive errors to the bias of RCSL depends largely on whether they are in the terminal step (t = H) or not.
we have Assumption 9 with å = 0, Assumption 13 is automatically satisfied with cn(0) = 0 and Assumption 10 unnecessary, resulting in the following rather simplified corollary.
Corollary 1. Suppose Assumptions
g
and 12 hold with å = 0. Then,
2CnH2
[J(Bn) Rl S
(24)
nR
Besides, Assumption 12 can be replaced with stronger variant of Assumption 13
18

---

Corollary 2. Suppose Assumptions 8 11 hold. Also assume the Hölder continuity of f(12z)
|f(R' [r) (000-((RR): f(Rxx) s K |R
Rl
R, E H), x X.
(25)
Then,
2(K 11) (0)
[[](3R) Bl S
(26)
Proof. It directly follows from that CR S K + 1 and cn()) - (28) 2K8 See Lemma 3 for the argument on bounding CR.
E.2 Bias and variance of MLE
The following theorem shows that the bias of MLE in 21) negligible if mild realizability condition is met. The proof is deferred to Appendix
Theorem 6. Suppose Assumption
14 holds. Then,
nd
J(3n)l S H2
fP 2
(27)
V
Moreover, the following theorem characterizes the asymptotic distribution of the variance of MLE in 21 The proofs are deferred to Appendix H
Let us introduce the gradient covariance matrix
Vo R Uniff Vo(arl R) R) R E Raxd
(28)
and the normalized policy Jacobian
1 (R) H
E
Unifi( [QT ae Vo ln pa R) R) E R
(29)
where Q (1x, a)= E [R x, a) Rm is the m-dimensional action value function.
Theorem 7. Suppose Assumption holds. Then, we have
J5(TT.R) (r - RR)] JEim),
GP(O, k)
(30)
H
in the limit of + 00, where k. [0, H) 10, HITh Raxm is the covariance function given by
k: R. R = Uo. (R) Uo. (R)).
(31)
Remark The differentiability of sample paths of the limit process F(.) GP(O, k) known to
be (roughly) the same as the differentiability of the covariance function k( (Corollary in 13]
which, according to (31. is governed by that of Ua- (.). smooth. With straightforward calculation, one can further see that Uq- (-) is smooth if, under some mild regularity conditions, the probabilistic model P is smooth in terms the associated policy To. n and the gradient Vo inpola./zz R)lo=o as functions of the target return R.
F-)) other words, F (-) is smooth if Ug- (-) is
F Proof of Theorem 5
Consider the weighted error function given by
(xx)) ((R) V (x.)lla
(32)
where V(x;) == SR D
K- Th [z,] is the value function of 3R and V (xx)= = R- h- ra is the
target value function. suffices for the proof of Theorem to establish suitable bound on ((x1) since, by Assumption
Φ(21)
Φ(x1)
J R
(33)
[(R(r))
19

---

To this end, we will make use of Pr: Pr:Sx.A x AR" x S), the near- deaarmettriiiistic component of Py such that
dPr(r, (() E ((,,a) (Pr(,,,'sss)),
(34)
(s,
where T(s a) = Bxc (f(8, ()) (8((s c R x S is the image of the near-deterministic transition and Bo0 (r, 5) = (r' E R 1| 11pn-rlla -rl 26 is the eoel centered at with radius 8. Let also P, IE, fTT 1BT be probability distributions and expectation operators identical to P. E, P", ,," respectively, except that the transition kernel Pr is replaced with Pr under the hood.
Now, for 1 v t S H - 1, we can bound ((x+) in terms of ((x++1).
Lemma 1. Suppose Assumptions to 10 12 and 13 hold. Then, for all E 24 with 1 -1SH-1
we have
v(x+)= S E. (0( [xa] + Hen(O) + H.n.
(35)
Proof. Let f(R,x ar) = f R ( 12 at). Note that (R Te, ax) = [[ oil is
well-defined for all 24 E Xr and a, E A by Assumptions 12 and 13 Thus, the claim follows from
x(x+) = ((R( (())))
< ((R)aa)) /
((0++1)) - ((t++))
(o,, Te, 8141
2 ((R)))) / V((et)))) (x(+2))
afssaan (6,1 cHCn
(( - (((et2) (((
f(Rla,, ( (a, S411(2)) + cHCR
((LL+)) ((x--1) ||-0-
f(R] 21 a )dp (an Te, s4,1) + 2cHCR
/ | (2
(a, r 841 11) + HcR(0) + 2eHOR f(Rlx(+1) (a,
/ (es+))dTe (ach))d 1z)) + Hen(s) + 2cHCn
where (a) is shown by Jensen's inequality with V (xx) - (xx) = S (z+4 V (x++1) [zrl.
(b) shown by Assumption g implying [|V(z) (z)) x I H, Assumption 12 and Lemma and, (c) shown by 15). (d) shown by Assumption 12 and evaluating f(R(a, ae)
R IT, oz)
I [
Pr (re, 944151,, a.) with Lemma and (e) shown by Lemma
Finally, the proof of Theorem 5 is concluded by dealing with the boundary term Φ(ZH).
Lemma 2. Suppose Assumptions
8 to 10 and 13 hold. For all aH E XH. we have
s(xn) S 2eHOn + 25CR.
(36)
Proof. Similarly as the proof of Lemma 1 we have
((n)) S [ - ((r-i)) ((enn)))--Ve)))) [(R)an (dloa rnlan) + eHCn-
We evaluate the RHS above by separating the domain of integral into two: where an E Admm {a A (f(sH,aH)) (*H)llo S d) and ii) where aи & Adem- For the case i), have (( (((H)))-V(0aA (()++))) S 1| - F(sH,an)ll + +||0(sn,an) ((n)) 28
20

---

and therefore, by Assumption 12 the integral restricted to Adm is bounded with 2dCR. For the case ii), note that J(R as) Pr (an))sH is well-defined by Assumption with | (xH) f(HH, ax) > 5. Thus, we have
S
V(E++1) (Em+1)) (RRan)dPPe (an, rula))
(8)
((*H+1)= V (* {( an ( (an, ralæn)
az Aumi
Jan A-Aam
V(TH+1) V (I pr (xH)lsH au (ap, (HlT)))
(b)
x H 1 Hc,
where (a) follows from desired result.
15 and (b) from Assumption
Combining both cases, we arrive at the
G Proof of Theorem
For simplicity, let T == RT. By the performance difference lemma (Lemma 6 we have
H
- ((3R)
Σ
E
R
(
- Bn(Zz))
(37)
where RHS is further bounded by
(a)
H
R (x()))))) (R(I))llvV
(38)
H 2 E PR
- (
(39)
Unif
(b)
H E
(40)
Unif[[]
(0)
2 112
BR Uni (XL((B(()))lm((()))
(41)
(4) H cp.0.
(42)
Here, (a) is owing to the boundedness of the Q-function 0 (x) Н, (b) is to Pinsker's
incquality, (c) is to Jensen's ss and (d) is to Assumption 14
H Proof of Theorem
Note that 0 is the M-estimator 12 associated with the criterion function
Ma (a x R) = In (a(a R)
Po- (a|x, R) (00) + ((0)).
(43)
Also note that Ma is locally bounded in the sense that, for every e2-ball U in O with a sufficiently small radius p 0,
E Unitte
sup Mo(a (2,, f)
(44)
Leeu
8 < BuUniH
Mo, (leaaalaa Pk) + ssup || (or, R)122 nEU
(45)
E PV
sup [EE
VoMo(a. m)]2 < <00,
(46)
21

---

where 9o is the center of U. Here, the first inequality follows from Ma(-1-) = (-1-) + fo
0o) "/ 1011-1)6+1100-111 (-o--)d, while the second inequality follows from that E U--ffit( [Ma [Te f)] S
and Jensen's 's inequality. This, with Assumption T5 i,ii,, allows us to use Theorem 5.14 in (22 and
obtain the consistency of MLE: & - 0" Furthermore, with Assumption 15 jii-v), is possible to use Theorem 5.23 in [52] and have the asymptotic normality
vn (0 (0) - NN,, Vg. Vo.
(47)
Finally, we apply the functional delta method (Theorem 20.8 in [22) on 0 and the mapping + (T) The desired result follows from calculating the derivative
VoJJ(TO,nn) Σ FRR
[Q) TT az) Vo Lo R) H22
(48)
according to the policy gradient theorem (Corollary 3
I Lemmas
Lemma 3. Suppose 25
holds. Then, we have Assumption 72 with CR S K +
Proof. Let B (R 1)n J0, HI and note that p supReN RR' - RRloo 1. Then, by the assumption, we have
/ [( 2 K} ((R K.
((S(R()
(49)
Rearranging the terms, we get the desired result.
Lemma 4. Let = + c.. Then, under Assumption g we have
[Pr(s,a) - Pr(s, Se
(50)
for all s E S and a E A.
Proof. is shown by (r)() Pr(s, a)
/ d d (prr- -Pr) (r, (1,,))
TV
sup F
Pr (r, E ()).s))
Pr (Ir ((s. 8 s Pr Pr # (3(
S6
where (a) follows from taking E = (s, (b) from the union bound, and (c) from Assumption 9
Lemma 5. Suppose Assumptions and hold.
Then, for all a+41 E X such that (re, 8t41)
ist. at). we have
j(R xr, a.) f(R(2141) S CR()).
(51)
22

---

Proof. Recall that f(R(zr..) az) = [ 1 az),
where 2/41
(I,, ae Ty 11). Now, the claim shown by
f(R ar) -((((xe+1)
/ ( (ictii)--((RR(aaaa dPr(r)
/ [S(R- - + rrelart))) [R 04 14
sup
(f(R r/ + ( ((
||rl-rel 526 $26
cn(()),
where (a) follows from Assumption 10 and 8/4 (8,, ax) 8141 almost surely, (b) from
||r f(s,, loo å and ||r/ f(s,, at) oo ò almost surely, and (c) from Assumption I3
Lemma 6. We have
11
J( = Σ Ex tm1
(1, T(x)) (" x (x))),,
(52)
where (x) E ト -= r2 2, a) is the action value function of n.
Proof. We may write Q" x'))) = (u) (w) 1Q (z. a)]. Now, observe
H
J(T') = ET [r:]
(53)
and
((x) Q" (²1,, 1ET (x1,
(54)
Σ E" (x,, Q (()41)),
(55)
where the last equality is due to Q" (T = 0. Taking the difference, we see
"
J(x) ( =
Σ
E=
- T (
(56)
t=
IQ (I,, Q (xt, x xx+)))
(57)
where the last equality follows from Q az), E [re Q" (2141, ((
Corollary 3. Suppose Assumption holds. Let no : X A(A) be a policy associated with Rdd, whose score function to (o(a)z) = Vo lu pa (a/z) is
parametrized density po po(alz), bounded in the sense Ee (supereu |l (a]x)]la] < for some U being a neighborhood of 0. Then, have
VoJ VOJ( = Σ T QT (1.
(58)
Proof. Let > 0 and fix À E Rdd arbitrarily. Set x aptwA and mp. and let be the base measure A relative to which pe (als)) is defined. Now, divide both sides of (52) by and take the
23

---

limit w + 0 to obtain
11
y QTy @ (Zie, a)
(o-twa(aaaz) - po(a(zt)) de(a)
A VDJ
Σ
lim T
(59)
t=l
( (o(d))))dd
ET
(60)
where the last equality is owing to the interchange of the expectation and the limit enabled by the dominated convergence theorem. Now, the desired result is shown since À is arbitrary.
J Proofs of Theorems and
Lemma 7, Pick E (0, 1) and set Co.j = ( for E (r.g). Then,
[Tx (=))(z) S Coos Oz)(z) Vz E
Vj v121
(61)
holds with a probability at least 1 A.
Proof. See Lemma 5.1 and its proof in Srinivas et al. 45
U
Lemma 8. Pick A (0, 10.1) and set Coo.j 2 2 (2TT2 for E {r,g)- Then, the following inequality holds:
N
Jo -((
No,N
a
(62)
log(1 + vo
1=1
with probability at least
A. where N is the number of iterations in the reward maximization
phase.
Proof.
This lemma directly follows from Lemma 5.4 in Srinivas al. 45
J.1 Proof of Theorem
Proof.
PLS chooses the next target returns z such that
Ug. (z) L d(z, x²) b.
(63)
By Lemma and the Lipschitz continuity, we have
Ug (g) L- L- x') 2 (Tx) + L. d(dzz)
(64)
(Tx).
(65)
Therefore, we obtained the desired theorem.
J.2 Proof Theorem
Proof. We first define an one-step reachability operator with certain margin E IR as
(x) = Y E Z 3z' E {,,)(zz)) + + Ld(=', < b].
(66)
Then, we can obtain the following reachable set after N iterations:
(Z.) 2g (2g ...))
(67)
N times
Here, the optimal target return z" in this paper can now be defined as
argmax (zz)).
(68)
= (2n)
Based on Theorem in Sui et [49] it guaranteed that 1) the safe exploration phase in PLS fully expands the predicted safe set (with some margin and 2) Ç-optimal target return vector x exists

---

within the safe set, after at most N₁ GP samples. Note that N₁ is defined as the smallest positive integer satisfying
M C+(126 (2o) +1)
(69)
9.N, Sg.Nr
5²
where C+ E R.+ is a positive constant.
The following proof mostly follows from that of Theorem 2 in Sui et al. 49 but there are differences in how to construct the confidence intervals. Specifically, for the compatibility with Theorem [I] we cannot assume that the functions are endowed with reproducing kernel Hilbert space (RKHS), which leads to a different bound terms of optimality.
The reward maximization phase in PLS chooses the next sample using the upper confidence bound in terms of reward within the fully expanded safe region. Thus, by the Cauchy-Schwand inequality, have
Σ
(J. J, (Tzz)))
S N
(J((T2)) Jr (Tz,))
(70)
By combining the above inequality with Lemma
we have
2
Σ (J((T))) (aao))-J(((T] (Tx,)))
S N
(71)
log(1 + 1o
16N
12
log
(72)
log(1 + Vr 2)
64
Given N₂ be the smallest positive integer N such that
E,,
12
4 V N llog(1 + 1z
log
(73)
6A
we then have
Σ - SE
(74)
N
The LHS of (74) represents the average regret. Thus, there exists & E Z in the samples such that
(T²) 2 (Tx)))
K
Experiment Details and Additional Results
K.1
Computational Resources
Our experiments were conducted in workstation with Intel(R) Xeon(R) Silver4316 CPUs@2.30GHz and NVIDIA A100-SXM4-80GI GPUs
K.2 Hyperparameters
We use the OSRL library for implementing most of the baseline algorithm. We leverage the default hyperparameters used in the OSRL library for the baselines. For CCAC, we use the authors' implementation"] For baselines, we use Gaussian policies with mean vectors given as the outputs of neural networks, and with variances that are separate learnable parameters. The policy networks and Q networks for all experiments have two hidden layers with ReLU activation functions. The Ky, K2 and Kn D are the PID parameters [47 that control the Lagrangian multiplier for the Lagrangian- algorithms (i.e., BCQ-Lag and BEAR-Lag). We use the same 10 gradient steps and rollout length which is the maximum episode length for CDT and baselines for fair comparison. Specifically, we set the rollout length to 500 for Ant-Circle, 200 for Ant-Run, 300 for Car-Circle and Drone-Circle. 200 for Drone-Run Run, and 1000 for Velocity. The safe cost thresholds for baselines are 20 and 40 across all the tasks. The hyperparameters used in the experiments are shown in Table 3
https::/gittub.cco DEPEBUDDDDEPE-LEEEE
25

---

Table 3: Hyperparameters for BCQ-Lag, BEAR Lag, CPQ. COptiDICE, and CCAC.
Parameler
BCQ-Lag
BEAR-Lag
CPQ
COptiDICE
CCAC
Actor hidden size Critic hidden size
[256, 256]
[256. 256]
[512, 512, 64, 512, 512]
VAE hidden size
400]
[400, 400]
[400, 400]
KK)
[0.1. 0.003. 0.001]
[0.1. 0.003, 0.001]
512,
Batch size
512
512
512
512
2048 (Velocity)
Actor learning rate
1.0e-3
1.0e-3
1.0c-4
1.0c-4
1.0c-4
Critic learning rate
1.0e-3
1.0e-3
1.0e-3
1.0c-4
1.0c-3
Moreover, we will present hyperparameters specifically used for the CDT and PLS that are based on return-conditioned supervised learning, in Table The experimental settings are same as the original authors' implementation of CDT.
Table 4: Hyperparameters common for CDT and PLS.
Parameter
All tasks
Number of layers
Number of attention heads
Embedding dimension
128
Batch size
2048
Context length K
10
Learning rate
0.0001
Droupout
0.1
Adam betas
(0.9, 0.999)
Grad norm clip
0.25
We now summarize the hyperparameters related to GPs in safe exploration and reward maximization phases in PLS. We set the number of episodes for each policy evaluation as w 20 for all tasks. We use GPs with radial basis function (RBF) kernels: one for the reward and one for the safety cost. We set the lengthscales of the reward as 50 for Sullet-Safety-Gyee tasks and 100 for Safety-Gymnasium Velocity tasks. The length-scales for the safety cost is set to be 5.0 for all tasks. While variances for the reward are 1.0 for mullet-Safety-Gyye tasks and 100 for Dafety--Gmmasinee
Velocity tasks, those for the safety cost are 1.0 for all tasks. Finally, following Turchetta et al. [5]] or Sui et al. [49) we set the Lipschitz constant L =
Other important experimental settings include how to set initial safe set Zo associated with Assumption 7 Tables 5 and summarize our experimental settings regarding the initial safe set of target returns.
Table 5: Safe target return range (Zn) for PLS "Bulltt-afety -affety-Gyy....
Parameter Ant-Circle
Ant-Run
Car-Cirele
Drone-Circle
Drone-Run
Reward
[250, 300] [0, 5]
[700, 750]
[400, 475]
[700, 720]
[400, 450]
Safety
[0, 5]
[0, 5]
[0, 5]
[0, 5]
K.3 Additional Experimental Results
We present additional experimental results for a different threshold 40 Table as for PLS and CDT, the refurn-conditioned policy in Table iis is same as that in Table The only
Note that,
26

---

Table Safe target return range (Z,) for PLS Gyriet--Gymmssimmm Velocity).
Parameter
Ant
HalfCheetah
Hopper
Walker2d
Reward
[2000, 2300]
[200, 2300]
[1200, 1500]
[2000, 2400]
Safety [0. 5]
[0, 5]
[0, 5] [0, 5]
Table 7: Evaluation results for the case with the safety cost threshold 40. We computed the mean and standard deviation by running each algorithm five times. Reward and cost normalized: thus, the normalized cost limit is 1.0. Bold: Safe agents whose normalized cost is smaller than 1, Red: Unsafe agents. Blue: Safe agent with the highest reward.
Task
Metric
BCQ-Lag
BEAR-Lag
CPQ
COptiDICE
CDT
CCAC
PLS
Reward f
0.76 0.14
0.02 0.01
0.63 0.05
0,72 + 0.0.3
0.02
0.01
Ant-Run
Safety
2.34 0.61
0.05 0.03
0.00 + 0.00
0.56 + 0.34
1,10 0.000
0.00
0.00
0.54 0.09
Ant-Cirele
Reward
0.78 0.16
0.63 = 0.25
0.00 1: 0.00
0.17 2 0.14
0.53 1 0.00
0.62
0.14
0.55
0.00
Safety cost
2.54 0.87
2.15 1. 1,38
0.00 1 0.00
2,50 1. 2.81
0.79 1 0.00
1.13 0.44
0.82 0.00
Car- Car-Cincle
Reward f
0.79 0.10
0.84 = 0.09
0.73 + 0.03
0.49 = 0.04
0,80 1 0.00
0.77 = 0.02
0.80
0.02
Safory cost
1.58 0.38
1.75 = 0.37
0.86 + 0.04
1.44 = 0.72
0.05
0.86 0.04
0.93 0.06
Drone-Run
Reward f
0.68 0.12
0.87 +
0.19 0.10
0.69 0.02
+
0.57 0.00
Safety cost
2.34 0.64
3,04 1 0.61
2.41 0.34
1.64 0.10
0.89 1 0.11
1.73 0.01
0.91
0.09
Drune -Crcle
Reward
0.92 0.05
0.78 4: 0.06
-0.27 = 0.01
0.28 =: 0.03
0.69 1 0.00
0.16
0.27
0.68
0.01
Safety cost
2,31 0,24
1.69 L. 0.31
0.20 1. 0.67
0.29 _. 0.24
1.00 1 0.00
0.71
0.49
0.96
0.03
Reward f
1.01 0.01
-1.01 + 0.00
-1.01 + 0.00
1.00 + 0.01
0.97 + 0.01
0.60
0.39
0.99
0.00
Ant-Veluee Vellociy
Safoty cost
2.25 0.29
0.00 = 0.00
0.00 + 0.00
3.35 = 0.74
+ 0.44
0.29 0.68
0.49
0.05
Walker2d
Reward 1
0.78 0.00
-0.01 0.00
0.13 0.01
1
0.84
0.02
VVelacity
Safety
0.30
0.13
4.05 1. 1.31
0.00 0.00
0.90 0.10
0.00 1. 0.00
3.49 0.43
0.00 0.00
HallCheetah
Reward
1.04
0.02
0.98 :: 0.04
0.01 1: 0.22
0.63 : 0.01
0.97 1 0.03
0.85
0.01
1,00
0.01
-Velocity
Safety cost L
14.10 + 3.46
6.34 = 5.46
0.10 - 0.11
0.00 = 0.00
0.05 + 0.11
1.22 = 0.09
0.01
0.00
Hopper
Reward f
0.85 - 0.19
0.40 + 0.21
0.23 + 0.00
0.05 + 0.07
0.67 + 0.03
0.60
0.17
0.84 0.00
Vellcity
Safety cost
5.30 3.85
2.75 0.04
0.46 0.17
0.60
0.63
difference regarding PLS between Tables and 7 optimization algorithm.
is the target returns as a result of our target returns
Observe that the experimental results in Table exhiibit similar tendency to those in Table More specifically, in both cases of = 20 and = 40, PLS is the only method that satisfies the safety constraint in all tasks, while every baseline algorithm violates the safety constraint in at least one task. Moreover, PLS obtains the highest reward return in most tasks, which demonstrates its higher performance in terms of reward and safety.
In addition, we provide Figure 3 to show how our PLS explores target returns z. Please observe that PLS guarantees safety in most of policy deployment. Moreover, even if safety constraint is violated, PLS quickly recovers to meet the safety requirement.
K.4 Online Sample Efficiency
A key advantage of PLS is its sample efficiency during the online optimization phase. Unlike methods that require fine-tuning a high- dithemmnnniioal policy network, PLS only optimizes a -io-ddiensiioall target return vector (R, G). This significantly reduces the number of required online interactions. Our experiments show that PLS typically converges within at most 20 GP iterations. With 20 rollout episodes per iteration for evaluation, this amounts to a total of approximately 400 online episodes, a number substantially lower than what is typically required for standard policy fine-tuning.
To demonstrate this benefit, we compare PLS against two standard offline-to-onlin fine-tuning base- lines: CDT-FT (S), which uses small budget of 400 episodes, and CDT-FT (L), which uses large budget of 40, 000 episodes. As shown in Table 8 while standard fine-tuning can eventually achieve comparable rewards, it incurs a substantial number of safety violations during the learning process. In contrast, PLS achieves strong performance while maintaining safety throughout, highlighting its suitability for safety-critical applications where online interactions are costly and risky.
27

---

1.7
g.4
66
66
(a) HopperVelocins
(b) AntRun
(c) AntVelocity
00
Banadan
(d) HalfCheetah Velocciy
(c) CarCircle
(f) DroneCirele
Figure 3: Experimental results on how our PLS ensures the satisfaction of the safety constraint while
obtaining new GP observations. Black dotted lines represent the normalized safety threshold.
Table 8: Comparison of PLS and fine-tuning (FT) baselines. PLS achieves comparable final perfor-
mance to CDT-FT but with significantly fewer safety violations during online adaptation.
Task
Method
Final Reward 1
Final Safety Cost
Safety Violations during Training
PLS
0.78 + 0.06
0.77 + 0.10
+
Ant-Run
CDT-FT (S)
0.75 + 0.08
0.80 1 0.12
125 1 20
CDTFT (L)
0.80 = 0.02
0.90 = 0.12
5368 + 490
0.41 0.01
0.77 + 0.05
+ 98 = 15
Ant-Circle
CDT-FT (S)
0,40 = 0.02
0.81 + 0.06
CDDFF (I.)
0.47 + 0.00
1.23 + 0.00
10051 + 1290
PLS
0.01
0.88 + 0.09
+ 110 + 18
Car-Cirele
CDT-FT (S)
0.71 0.03
0.90 + 0.11
CDT-FT (L)
0.73 0.01
0.98 + 0.12
16023 + 2309
PLS
0.59 0.00
0.50 + 0.44
3 145 25
Drone-Run
CDT-FT (S)
0.58 0.02
0.55 +
CDT-F (L.
0.59 + 0.00
0.82 + 0.05
2400 + 479
PLS
0.59 0.00
0.90 + 0.08
Drone-Cirele
CDT-FT (S)
0.59 + 0.01
0.92 + 0.09
-
CDT-FT (L)
0.60 + 0.00
0.37 + 0.14
3080 + 2746
PLS
0.98 0.00
0.82 + 0.19
+ 130 22
Ant-Vel
CDT-FT (S)
0.97 + 0.02
+ 0.21
CDT-FT (L)
0.68 = 0.34
0.97 = 0.00
17010 + 3589
PLS
0.79 0.00
0.00 + 0.00
Walker2d-Ve Vel
CDT-FT
0.75 0.04
+ 0.01 0.81 + 0.07
CDT-FT (L)
0.80 0.00
9810 + 2830
PLS
0.99 0.00
0.15 0.19
+ 160 30 2801 + 1828
HalfChectah- Vel
CDT-FT CDT-FT (L)
0,98 0.02
0.18 + 0.20
0.96 0.03
+ 0.13
PLS
0.83 0.01
0.42 0.10
2 - 115 24
Hopper-
CDT-FT (S)
0.82 0.03
0.45 + 0.12
CDT-FT (I.)
0.84 0.06
0.82 + 0.26
12790 2589
28

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state the main claims of this paper in both the abstract and introduc- tion. Especially, we write the "Our contributions" paragraph at the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: This paper discusses limitations in Section g
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

We explicitly list the assumptions in the main paper and then provide the full and formal proofs in Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provided the details of our experiments in the main paper and appendix. Also, we submit the source code as supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
30

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide the source code supplementary material. We do not any new data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cor
mublicc/qudees/Codeuubbii www.ssiiinooooiylo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtps/cccppplgguuuiiaeeeaec Stteee///ooSSSSeissSioica ssissssliccyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper specifies all the training and test details the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We provide the experimental results by computing mean and standard devia-
tions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
31

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This paper provides information on computational resources in Appendix K.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ate::/nneuiippee httpc//pbbccc/t/hcccciiiaaaallad
Answer: [Yes]
Justification: All the authors of this paper have carefully reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper discusses broader impacts in Appendix B
Guidelines:
The answer NA means that there is no societal impact of the work performed.
32

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks because we do not release models or datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We properly mention the existing assets used in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
33

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
34

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We do not use LLMs as an important, original, or non-standard component of the core methods in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
35


Reviews:
[{"review_id": "r3cWkIHEep", "reviewer": "Reviewer_hNWo", "summary": "This paper introduces ​Provably Lifetime Safe RL (PLS)​, a novel method addressing the critical challenge of ensuring policy safety throughout both learning and deployment phases in safe reinforcement learning. PLS integrates ​offline policy learning​ via CDT and safe online deployment​ using Gaussian Processes. Theoretically, the authors justify GP usage and establish safety guarantees during deployment.", "questions": "Regarding Section 6.2, could you please provide a more intuitive explanation and discussion of the design choices presented?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:​​\nTo ensure safety while maximizing reward throughout the entire process from learning to execution, the policy is formulated as a return-conditioned policy, where the return comprises both reward and cost. This policy is trained offline via supervised learning. During inference, Gaussian Processes are effectively employed to model the relationship between the target return and the actual return. This model is then used to determine the optimal target return that guides the policy toward safe and high-reward behavior.\n\n​This paper unifies offline safe RL and Bayesian optimization for end-to-end safety, addressing a fundamental gap in safe RL literature. The core innovation of this work lies in the integration of offline reinforcement learning with Gaussian Processes, offering a new design that bridges data-driven policy learning and uncertainty-aware return optimization.\n\nThe empirical results are compelling, demonstrating the effectiveness of the proposed method.\n\nWeaknesses:​​\n\nThis paper relies a series of assumptions, such as near-deterministic transitions (Assumption 1) and initial safe target returns (Assumption 7), which may not hold in highly stochastic environments. The performance of proposed method hinges on offline dataset coverage and quality; distributional shifts may degrade GP predictions.\n\nThe theoretical sections use overly complex notation, which hampers readability. Additionally, some notations are reused for different concepts, for example, $\\pi$ is used to denote both the policy and circle ratio. This reuse can lead to confusion and should be avoided. Simplifying and standardizing the notation would greatly enhance the clarity of the presentation.", "was_revised": true, "final_justification": "The authors clarification indeed addresses my concerns. The justification that PLS as a ​meta-algorithm​ and highlight its robustness beyond initial theoretical assumptions.\n\nI think it's essential for the ​key arguments​ regarding the theoretical assumptions (especially the nature of Assumptions and the meta-algorithm flexibility) are incorporated into the paper's main text or relevant sections, ensuring readers benefit from these clarifications without needing the rebuttal.", "rebuttal": "We deeply appreciate the reviewer hNWo for helpful and thoughtful comments and questions. We will first address the comments in Weaknesses and then answer the Question.\n\n> **[Weakness 1]** This paper relies a series of assumptions, such as near-deterministic transitions (Assumption 1) and initial safe target returns (Assumption 7), which may not hold in highly stochastic environments. The performance of proposed method hinges on offline dataset coverage and quality; distributional shifts may degrade GP predictions.\n\nThank you for raising this important point. We agree that assumptions such as near-deterministic transitions (Assumption 1) and the existence of an initial safe target return (Assumption 7) may not hold in highly stochastic or low-coverage settings. However, Assumptions 1-6 are introduced to conduct our theoretical analysis particularly for modeling Constrained Decision Transformers (CDTs), but they are not inherent to the general structure of PLS.\n\nIn particular, while Assumption 1 (near-deterministic transitions) is strong, it is more general than the one used in Brandfonbrener et al. [9], where exact deterministic transitions (i.e., $\\delta = 0$) are assumed. Our formulation allows for a $\\delta$-neighborhood around deterministic dynamics, which permits small stochastic perturbations and thus covers a broader class of environments. Moreover, similar (near-)deterministic assumptions are common in the safe RL literature, as seen in:\n\n- Berkenkamp et al. \"Safe model-based reinforcement learning with stability guarantees.\" NeurIPS, 2017.\n \n- Wachi and Sui. \"Safe reinforcement learning in constrained Markov decision processes.\" ICML, 2020.\n\n- Thomas et al. \"Safe reinforcement learning by imagining the near future.\" NeurIPS. 2021.\n\nOur PLS is designed as a meta-algorithm and can be empirically paired with any return-conditioned offline RL where performance can be approximated by a Gaussian Process (GP) as a function of target returns. If future policy architectures relax or eliminate assumptions like near-determinism, PLS can incorporate them directly and may benefit from improved generalization and theoretical guarantees.\n\nRegarding offline dataset coverage and distribution mismatch, we acknowledge that these factors can affect GP prediction accuracy. However, a key feature of PLS is that it actively learns the discrepancy between target and actual returns during online deployment, using uncertainty-aware safe exploration and reward maximization. This mitigates the risk of unsafe behavior due to model miscalibration or distributional shifts.\n\n> **[Weakness 2]** The theoretical sections use overly complex notation, which hampers readability. Additionally, some notations are reused for different concepts, for example, $\\pi$ is used to denote both the policy and circle ratio. This reuse can lead to confusion and should be avoided. Simplifying and standardizing the notation would greatly enhance the clarity of the presentation.\n\nThank you for highlighting the notational ambiguity. We acknowledge the reuse of the symbol $\\pi$ and have included a disclaimer regarding this on line 257 of the original submission. Nonetheless, we agree that the reuse may still be confusing. In the camera-ready version, we will revise the notation to avoid this overlap and improve clarity. \n\nWe believe precise notation is essential for conveying the theoretical content accurately, and we have already attempted to manage the overall complexity by moving detailed mathematical notations to the appendix. If there are specific notations that felt unclear, we would be grateful for your suggestions and happy to incorporate them. In the camera-ready version, we will include a nomenclature table in the appendix. For example,\n\n| Symbol | Meaning | Section |\n|--------|---------|------|\n| $\\pi$   | context-dependent policy | 3 |\n| $\\pi_\\text{circle}$   | circle ratio ($\\approx$ 3.14) |6 |\n| $\\beta_R$ | return-conditioned behavior policy | 4 |\n| $b$     | safety budget | 3 |\n| $\\vdots$ | $\\vdots$ | $\\vdots$ |\n| $R, G$    | target reward / cost | 3 |\n\n> **[Question]** Regarding Section 6.2, could you please provide a more intuitive explanation and discussion of the design choices presented?\n\nOur goal in Section 6.2 is to find the best target returns $z = (R,G)$ that maximize reward while satisfying the safety constraint. We achieve this through a two-phase process guided by GP uncertainty estimates:\n\n- Pessimism for safety: To ensure that selected targets are safe, we evaluate the safety cost using the GP’s upper confidence bound. This pessimistic estimate accounts for worst-case scenarios. A new target return is considered only if this worst-case cost remains below the safety threshold $b$. \n\n- Optimism for reward: To encourage discovering high-performing policies, we use the GP’s upper confidence bound for the reward. This optimistic estimate helps explore targets with the highest potential reward return.\n\nIn the safe exploration phase, we first explore the most uncertain but still verifiably safe target returns to efficiently expand the safe region. Also, in the reward maximization phase, we switch to selecting targets that maximize the optimistic reward estimate within the safe region."}, {"review_id": "M8EfoQhp8S", "reviewer": "Reviewer_XNH5", "summary": "This paper studies a mix of lifelong learning with safe reinforcement learning. The paper proposes Provably Lifetime Safe Reinforcement Learning (PLS), with the aim of ensuring safety throughout the entire RL process from learning to deployment. PLS combines offline safe RL using return-conditioned supervised learning (specifically Constrained Decision Transformer) with online safe policy deployment. The main goal is to find the optimal pair of target returns $\\boldsymbol{z}=(R, G)$ that maximizes $J_r\\left(\\pi_{\\boldsymbol{z}}\\right)$ while guaranteeing the satisfaction of the safety constraint (i.e., $J_g\\left(\\pi_{\\boldsymbol{z}}\\right) \\leq b$ ) according to GP-based inferences. For this purpose, they optimistically sample the next target returns $\\boldsymbol{z}$ while pessimistically ensuring the satisfaction of the safety constraint. The key innovation is optimizing target returns (reward and safety cost) using Gaussian Processes (GPs) during deployment, rather than using fixed target returns. The authors provide theoretical analysis showing the relationship between target and actual returns follows a Gaussian process with a small bias term. They prove that PLS guarantees safety with high probability throughout the process and finds near-optimal target returns. Experimentally, PLS is evaluated on continuous robot locomotion tasks from the Bullet-Safety-Gym and Safety-Gymnasium benchmarks. Results show PLS is the only method that satisfies safety constraints across all tasks while achieving competitive rewards. The method requires fewer online interactions than conventional online safe RL due to optimizing only two-dimensional target returns. The approach assumes access to an offline dataset and initial safe target returns, leveraging GPs' uncertainty estimates for safe exploration.", "questions": "I have the following questions:\n\n1. Even though this paper tries to set itself apart from the existing safe RL works, there are some similarities. For example, in assumption 7, line 278 there exists the safe set. This basically aids in the intial exploration which is reflected in Theorem 2. How is this approach different from other existing safe RL works?\n2. Where is the pessimistic exploration showing up? It seems to me from section 6.2 lines 263-268 that to conduct safe exploration, similar to Srinivas et al. you are also using optimistic GP UCB?\n3. The termination condition also requires more explanation. Safe exploration is terminated under the condition $\\max\\_{\\boldsymbol{z} \\in E_N}\\left(u\\_N(\\boldsymbol{z})-\\right.$ $\\left.\\ell_N(\\boldsymbol{z})\\right) \\leq \\zeta$, where $\\zeta \\in \\mathbb{R}_{+}$is a tolerance parameter. How is this tolerance parameter chosen? Shouldn't this be a problem-specific parameter that should depend on the hardness of the problem?\n4. Why did you choose to use CDT as the base offline RL algorithm? Have you experimented with other return-conditioned methods, and would the GP-based optimization still be effective with different underlying policies?\n5. Your experiments show some violations of safety constraints during exploration (Figure 3), despite theoretical high-probability guarantees. Can you elaborate on the sources of this gap and whether tighter bounds or different GP configurations could reduce these violations?", "limitations": "See weakness and questions section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. The paper provides a rigorous theoretical analysis establishing the relationship between target and actual returns through Theorem 1, showing it follows a Gaussian process with explicit bias terms. The safety guarantees (Theorem 2) and near-optimality results (Theorem 3) are formally proven with clear assumptions (Assumptions 1 to 4, lines 170-186).\n2. To me, the combination of offline constrained safe  RL with online GP-based optimization is well-motivated (but the exact novelty is not clear). This tries to address a current limitation of existing safe RL paradigms, where online methods struggle with initial safety and offline methods suffer from distribution mismatch.\n3. The experiments cover 9 different tasks across two benchmarks, with consistent results showing PLS as the only method maintaining safety across all tasks while achieving competitive rewards. The computational efficiency advantage is also demonstrated.\n\nWeaknesses\n\n1. The assumptions 1-4 are standard and prima facie seem to stem from other offline RL and safe RL literature. It is not clear what the key additional requirement is needed for this CSRL setting.\n2. Theorem 3 only guarantees near-optimal target returns, not a near-optimal policy. This gap between optimizing target returns and achieving optimal policy performance is not addressed.\n3. Going back to the assumptions, the theoretical analysis relies on restrictive assumptions, including near-deterministic transitions (Assumption 1), specific conditions on return density functions, and soft realizability. Not only may these not hold in many practical scenarios, but even their own empirical setting doesn't satisfy this.\n4. The GP bandit motivation of Srinivas et al. (2014) specifically tackled high high-dimensional setting. However, their approach is limited to single safety constraints (two-dimensional target returns). Extension to multiple constraints or high-dimensional return spaces would face computational challenges and may require fundamentally different approaches.", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer’s encouraging comments. The thoughtful comments of Reviewer XNH5 are valuable and help us improve the manuscript's quality. We will answer the Questions first and then address the comments in Weakness.\n\n### [Question 1] Differences from other existing safe RL works \n\nThank you for the thoughtful question. The key difference lies in how the input space or initial safe set is defined. In most existing methods, the safe set is defined over high-dimensional states or state-action pairs, which makes their construction and verification challenging. In contrast, our method assumes a safe set over target returns. This space is low-dimensional (typically two-dimensional) and allows direct control over the behavior of a return-conditioned policy. By setting the target returns to low values for both reward and safety cost, the policy tends to output generally safe actions, so identifying target returns that belong to the initial safe set is not difficult.\n\n### [Question 2] Pessimistic exploration\n\nThank you for the helpful question. Because the safety constraint is represented as $J_g(\\pi) \\le b$, incorporating the upper bound of the safety cost function is pessimistic. By constructing $Y_N$ using the upper bound of the safety function, we aim to explore the safety cost function within the pessimistically identified safe region. We acknowledge that this may not have been sufficiently clear in the current text and will add further clarification in the camera-ready version.\n\n### [Question 3] Termination condition\n\nThank you for the important question. We agree that the choice of the tolerance parameter plays an important role and may be problem-specific. In our experiments, however, we found that using a small fixed value exhibited stable and effective performance across all evaluated tasks. While this approach may not be universally optimal, it provided a practical and consistent stopping criterion in our settings. We will add further explanation and discussion of this choice in the camera-ready version.\n\n### [Question 4] CDT as the base offline RL algorithm\n\nThank you for this insightful question. One of our main contributions in this paper lies in the theoretical analysis presented in Section 5, which is specifically developed for the (constrained or multi-objective) Decision Transformer. These theoretical results do not directly extend to arbitrary return-conditioned methods. Therefore, our choice of CDT was primarily motivated by the need to align the theoretical framework with the empirical implementation. That said, as the reviewer suggests, we believe that our PLS can also be empirically effective when combined with other return-conditioned policies, and exploring this generality would be an interesting direction for future work.\n\n### [Question 5] Some safety violations during exploration\n\nThank you for the thoughtful question. Crucially, our safety guarantee in Theorem 2 is *probabilistic*. This means that a small number of constraint violations are theoretically permissible and expected in rare cases. The primary source of the gap is the approximation error between the GP model and the (complex and noisy) true safety cost return. Also, the finite number of rollout evaluations introduces noise that can affect the GP's predictions.\n\nAs the reviewer suggests, it is possible to reduce the number of violations. For example, using more conservative confidence bounds (e.g., increasing $\\alpha_g$) or carefully tuning the GP kernel and its hyper-parameters could lead to more cautious exploration and better alignment with the safety constraints, at the cost of exploration efficiency. We will include a discussion of these points in the camera-ready version.\n\n### [Weakness 1] Key additional requirement needed for our RCSL setting\n\nAssumption 2 (initial coverage) is conceptually similar to the concentrability condition often used in offline RL to ensure that important target distributions are well-supported by the dataset.\nAssumptions 1, 3, and 4, on the other hand, are more specific to the analysis of CDT in our setting. These assumptions help us characterize the relationship between the target returns and the actual returns, which is critical for ensuring that the GP model used in PLS poses sufficient accuracy. In particular, they are used to show that the CDT policy is approximately unbiased, which supports our theoretical guarantees. We will add more clarifications in Appendix C.2.\n\n### [Weakness 2] Theorem 3 only guarantees near-optimal target returns, not a near-optimal policy\n\nThank you for the insightful feedback! As noted in our Limitations paragraph, our PLS does not guarantee the optimality of the resulting policy. This gap stems from the limitations of the underlying RCSL framework, which currently lacks a theoretical characterization of how closely it approximates optimal policies. While PLS benefits from provable safety and efficient target return optimization, it also inherits this fundamental gap from RCSL. Bridging this gap would require a deeper theoretical understanding of how well RCSL approximates optimal policies under given target returns, and we consider that this direction is highly valuable for future work.\n\n### [Weakness 3] Restrictive assumptions \n\nThank you for pointing this out. As discussed earlier, some of the assumptions (i.e., Assumptions 1, 3, 4, and 5) are used solely to bound the bias in the RCSL model and ensure that the GP approximation remains sufficiently accurate. Importantly, our theoretical results still hold even when the GP is biased; that is, this is captured by the bias term $\\varepsilon(z)$ in Theorem 1, which quantifies how deviations from ideal assumptions affect the predictive accuracy.\n\nIn particular, while Assumption 1 (near-deterministic transitions) is  strong, it is more general than the one used in Brandfonbrener et al. [9], where exact deterministic transitions (i.e., $\\delta = 0$) are assumed. Our formulation allows for a $\\delta$-neighborhood around deterministic dynamics, which permits small stochastic perturbations and thus covers a broader class of environments. Moreover, similar (near-)deterministic assumptions are common in the safe RL literature, as seen in:\n\n- Berkenkamp et al. \"Safe model-based reinforcement learning with stability guarantees.\" NeurIPS, 2017.\n \n- Wachi and Sui. \"Safe reinforcement learning in constrained Markov decision processes.\" ICML, 2020.\n\n- Thomas et al. \"Safe reinforcement learning by imagining the near future.\" NeurIPS. 2021.\n\nA similar reasoning applies to Assumption 6, which ensures the asymptotic normality of the policy parameter estimator. Even when asymptotic normality does not strictly hold, the approximate consistency\n$|J(\\pi_{\\hat\\theta,z}) - z| \\le \\varepsilon(z) + o_P(1)$\ncan be established under weaker conditions (e.g., Theorem 5.14 in Van der Vaart [51]). This level of consistency is sufficient for characterizing the safe set used in Assumption 7.\n\n\n### [Weakness 4] High-dimensional or multiple constraints setting\n\nWe agree with the reviewer that Srinivas et al. (2014) specifically tackled a high-dimensional setting. However, safe exploration literature, such as Sui et al. (2015), Berkenkamp et al. (2016), and Turcehtta et al. (2016), has applied extensions of GP-UCB primarily in low-dimensional input spaces, at least empirically: \n\n- Sui et al. \"Safe exploration for optimization with Gaussian processes.\" ICML. 2015.\n\n- Turchetta et al. \"Safe exploration in finite Markov decision processes with Gaussian processes.\" NeurIPS. 2016.\n\nIn our case, PLS is designed for sequential decision-making problems with a single safety constraint, which covers a broad class of safe RL problems encountered in practice. \nAs the reviewer notes, however, extending to multiple constraints or high-dimensional return spaces presents important and non-trivial challenges. Our current PLS method may face computational limitations in such settings, and addressing them may require fundamentally different algorithmic designs. We will include additional discussion of these limitations and future directions in the camera-ready version."}, {"review_id": "PZ1T4RdSRL", "reviewer": "Reviewer_njCE", "summary": "This paper presents Provably Lifetime Safe RL. The method begins by training a policy / agent in the offline safe RL setting, i.e. learning from a static dataset of state transitions, rewards, and costs under some behavior policy. It then introduces a notion of approximating target returns (again, both rewards and costs) using Gaussian Processes to allow for further optimization via (1) safe exploration, i.e. the finding of an increasingly expansive set of safe trajectories and then (2) reward maximization. The paper provides theoretical analysis and justification for the proposed approach.", "questions": "1. Why would alpha_r be equal to alpha_g in (5)? This seems to imply that exploring to increase performance is the same as safety? There is a 2-step scheme for exploration that controls for this somewhat; i.e. a kind of \"safe set\" is identified first before reward maximization. I am just wondering if, perhaps the allowed failure probabilities can be tuned for safe exploration versus reward maximization (i.e. it is not ok to fail during safe exploration, but it might slightly more ok to fail during reward maximization). In this case, perhaps Delta with a subscript for each phase?\n\n2. GPs have nice properties, many of which this paper takes advantage of. For example, smoothness, continuity, etc, as well as being nonparametric. However, they also have issues. One is that - if one wants to preserve certain theoretical properties - then computational and scaling issues arise as the size of the data grows. The authors seem to have addressed this, for example by assuming k is diagonal, i.e. uncorrelated covariance function. This is somewhat related to the fact that GPs are very sensitive kernel selection and have been found to in fact be poor at uncertainty quantification (which seems to be a motivation of this paper, offline RL in general, and the notion of going from a static dataset to online inference). See Wilson, Andrew, and Ryan Adams. \"Gaussian process kernels for pattern discovery and extrapolation.\" In International conference on machine learning, pp. 1067-1075. PMLR, 2013.For example, deep ensembles may not be amenable to the same theoretical treatment but they simply \"work better\". This comment digresses somewhat from the main contributions of the paper but the choice of GPs (and perhaps the tricks used to make this work in practice) could be described in more detail.\n\n3. The experimental results appear to be incomplete. This appears to be from Bullet Safety Gym and a few tasks in Safety Gym. Why not every task in Safety Gym and Bullet Safety Gym? And why not Metadrive, a la the following papers (below) and many others?\n\n4. In section 6.1, where do the target return vectors come from? This is often a practical problem with return-conditioned RL, because either the task or reward (cost) functions have to be amendable to knowing a priori what a \"good\" target. The desired reward per state plus an appropriate episode length might need to be known, for example. The paper states that the next target returns are chosen sequentially that maximize reward return subject to safety constraint. How? To be clear, this question is about the process of collecting (generating) data used for fitting the two GPs. How does one know that this process is safe, for example? NOTE: this question is addressed in section 6.2 but I will leave the comment here because some of the questions are still valid; that is, the problem must still be amenable to the notion of safe exploration in section 6.2 so both the task structure and cost function formulation must be appropriate. Note also a separate question about baselines; perhaps these questions are all related.\n\n5. Is the 'H' in the definition of candidate thresholds (line 116) the same H that defines episode length? It is unclear so far why this would be the case. Also, the notation here is a bit strange. It implies a *set* of candidate thresholds but this does not appear to be a finite set. Is the optimization problem in (1) for an infinite set of constraints?\n\n6. Line 92-93, the episode length is fixed for each episode; does this mean all episodes have the same length? Or what is \"fixed\" here?\n\nOther comments and questions\nThe abstract states, \"while cautiously optimizing a limited set of parameters, known as target returns\". This is a bit confusing. Is part of (for example) the policy network frozen such that only a subset of parameters are updated during deployment? I do not think this is the case; rather, the policy is optimized relative to a limited set of *objectives*.\n\nLine 75-77 states, \"offline safe RL still suffers from a central difficulty: learned policies often become either unsafe or overly conservative, largely due to the intrinsic challenges of off-policy evaluation (OPE) in stateful environments\". There are certainly challenges in OPE, and they are related to some of the issues in Offline RL, but they are not exactly 1:1. OPE has to do with performance evaluation when one only has access to data. Another way of saying this is that OPE is a kind of *solution* to some of the challenges in offline RL, but an alternative is to train from offline data and then test in in the environment. The latter approach has challenges due to OOD, the dataset used for training being insufficient, etc. In fact, one of the grand challenges is dealing with narrow or biased data distributions; OPE itself can suffer from this. This is not a fundamental issue with the paper but is a bit of a strange representation of the problem.\n\nIs a context-dependent policy Markovian?\n\nThis is mostly clear from the context but the paper might be improved by mentioning that everything is in terms of undiscounted returns (or costs).\n\nMany works have gone to great lengths to try to predict returns under a policy - this is essentially one of the pillars of RL. In fact, one small piece of the decision transformer approach is to use it as a critic. GP seems like an interesting approach given that almost every work out there is parametric, and in fact involves some type of deep learning framework.\n\n\n\nYinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with feasibility-guided diffusion model. arXiv preprint arXiv:2401.10700, 2024.\nKoirala, Prajwal, Zhanhong Jiang, Soumik Sarkar, and Cody Fleming. \"Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning.\" arXiv preprint arXiv:2412.08794 (2024).", "limitations": "While the paper does address limitations (and these limitations are indeed interesting avenues of research), perhaps the questions above and other reviewers' questions will assist the authors in identifying other potentially significant limitations.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is relevant, i.e. it addresses known gaps or challenges in offline RL, and offline safe RL specifically, in particular the leap from the dataset to the environment and the various issues that arise due to this transition. \n- The paper is rigorous theoretically.\n- The paper leverages several existing ideas and approaches in clever ways, including pulling together GPs with deep RL.\n\nWeaknesses:\n- The assumptions may not match reality, or to what extent do they match reality?\n- While interesting, the use of GPs seems to be more theoretically motivated than for performance or computational reasons. \n- There is not really an \"algorithm\", although it is somewhat made clear during the text.\n- There is no ablation. The framework starts with CDT, a very strong baseline. Can any approach that first trains a policy and then allows it to explore afterwards -- would this not, by definition, result in performance that is no worse than CDT?", "was_revised": true, "final_justification": "This is a nice paper. Other reviewers rated it higher and appreciated the theoretical contribution perhaps more than I did. In the end, one has to choose a ranking/score, and there is nothing between weak reject and weak accept; if there were, I would probably choose that. As it is, I will leave my score as-is but obviously fully support the other reviewers.", "rebuttal": "We deeply appreciate the reviewer’s thoughtful feedback. The thoughtful comments of Reviewer njCE are valuable and help us improve the manuscript's quality. We will first answer the Questions and address the comments in Weaknesses.\n\n### [Question 1] Setting of $\\alpha$\n\nThe primary reason we set $\\alpha_r = \\alpha_g$ is to clearly convey our key ideas while avoiding complex notations. We agree that using separate failure probabilities is a valuable extension for practical applications and will discuss this in the final version.\n\n### [Question 2] Challenges of GPs\n\nWe thank the reviewer for constructive feedback! While we acknowledge the general challenges with GPs regarding scaling and uncertainty quantification, these concerns are substantially mitigated in our work for two key reasons:\n\n- The primary reason computational and scaling issues are not a concern is that PLS does not operate on the high-dimensional state or state-action spaces. Instead, it optimizes only a two-dimensional target return vector $(R,G)$. The amount of data required for the GP to model this low-dimensional space is very small, making the computational cost negligible.\n\n- Another reason for choosing GPs over alternatives like deep ensembles is that they provide a principled way to quantify uncertainty. This is not just a nice property; it is the fundamental mechanism that allows us to provide the high-probability safety and near-optimality guarantees (Theorems 2 and 3) that are central to our paper's contribution. While a deep ensembles is a nice function approximator, they do not usually offer this same level of provable guarantee for safe exploration.\n\nAs the reviewer correctly inferred, our implementation makes a simplifying assumption that the reward and safety cost returns are uncorrelated, allowing us to model them with independent GPs. We agree that the practical details of our GP configuration could be described more thoroughly. In the final version of the paper, we will include a more detailed discussion on our kernel selection and provide a clearer justification.\n\n### [Question 3] Empirical experiment\n\nOur primary objective in the empirical evaluation was to conduct a rigorous and direct comparison against the current state-of-the-art method for *versatile* offline safe RL, which is Constraint-Conditioned Actor-Critic (CCAC) [Guo et al., 2025]. To conduct the most direct and fair comparison, we have included all the environments used in the original CCAC paper. This allows us to clearly demonstrate the advantages of our proposed method relative to the most relevant and competitive baseline.\n\n- Zijian et al. \"Constraint-conditioned actor-critic for offline safe reinforcement learning.\" ICLR. 2025.\n\nWhile we acknowledge that other benchmarks like Metadrive exist, our focus was on providing a targeted empirical evaluation that supports our main claims. Crucially, let us highlight that **a core contribution of our work also lies in the theoretical analysis**, where we establish a provable framework for end-to-end safety. The experiments are designed to validate this theoretical foundation and demonstrate its practical effectiveness, rather than to exhaustively cover every possible benchmark. We believe our chosen set of experiments successfully achieves this goal.\n\n### [Question 4] Target returns in Section 6.2\n\nThe target return vectors are selected through a safe exploration process described in Section 6.2. We begin with a conservative initial set (low reward and safety cost targets), which empirically produces safe behavior. These initial rollouts form the seed data for GP modeling.\n\nAs proved in Theorem 1, the relationship between target and actual returns can be modeled as a GP with bounded bias. This theoretical result justifies our use of GP-based uncertainty to guide exploration. New target returns are selected only if the GP’s worst-case predicted cost remains below the safety threshold, which ensures all data collection is performed under high-probability safety guarantees (Theorem 2).\n\n### [Question 5] $H$ in the definition of candidate thresholds\n\nYes, $H$ is the episode length. We assume the safety threshold $b$ is bounded in $[0,H]$, as the maximum cumulative safety cost is $H$, for avoiding trivial cases. The optimization in (1) finds a single, *versatile* policy, $\\pi$, adaptable to a continuous range of safety thresholds, $b$. This contrasts with training a unique policy for each specific threshold. We will make the problem statement more explicit in the revision.\n\n### [Question 6] Fixed episode length\n\nYes, in our formal problem statement, \"fixed\" means that all episodes are defined to have the same constant length, $H$. This is a standard assumption for finite-horizon CMDPs. In practice, we handle variable-length episodes by treating $H$ as the maximum episode length and padding shorter trajectories. This is a standard technique that works seamlessly with return-conditioned sequence models like the CDT.\n\n### [Weakness 1] The assumptions may not match reality\n\nWe agree that theoretical assumptions inevitably introduce some gap between theory and practice. In our case, the assumptions, such as near-deterministic transitions and sufficient data coverage, are adopted for rigorous guarantees while still capturing important characteristics of many real-world safe RL scenarios.\n\nWhile these conditions may not fully hold in all cases, they are loosely aligned with common assumptions in safe RL literature (e.g., smooth dynamics in simulated robotics). For instance, deterministic state transition is a popular assumption in safe RL literature, as being represented by\n\n- Wachi and Sui. \"Safe reinforcement learning in constrained Markov decision processes.\" ICML, 2020.\n\n- Thomas et al. \"Safe reinforcement learning by imagining the near future.\" NeurIPS. 2021.\n\nThat said, we acknowledge that relaxing these assumptions is a valuable direction for future work. We will revise the paper to clarify the practical implications of each assumption.\n\n### [Weakness 2] While interesting, the use of GPs seems to be more theoretically motivated than for performance or computational reasons.\n\nWhile our use of GPs is indeed theoretically motivated, we view this as a key strength of our framework, not a limitation or weakness. The ability of GPs to provide calibrated uncertainty estimates is what enables PLS to reason about safety in a principled way. This is essential for enjoying high-probability safety guarantees and for enabling reliable safe exploration under constraints. Our goal is not simply to achieve SOTA performance on benchmarks. Rather, we believe that for the safe RL community, especially with a view towards real-world, safety-critical applications, it is more important to build methods that are reliable and well-understood. **Our primary contribution is a framework where we can explain why and how safety is guaranteed.**\n\n### [Weakness 3, 4] There is no algorithm/ablation\n\nThank you for the comments. We would like to highlight that a pseudo-code of PLS has been provided in Appendix B.\nAlso, it may seem to be a natural outcome that PLS outperforms the baseline CDT. Our contribution, however, is not simply improving performance, but rather being the first to propose a end-to-end safe RL framework that captures the relationship between target returns and actual returns and applies this to safe online optimization. Crucially, our claims are supported both theoretically and empirically.\n\nWe first theoretically demonstrate that the difference between target and actual returns follows a GP (Theorem 1) . This is a non-trivial insight that provides the theoretical foundation for using a GP-based optimization approach. Based on this theoretical insight, we proposed PLS to optimize target returns using GPs. Through experiments, we empirically demonstrated that this framework performs better than baselines.\n\nA simpler search or optimization method could not provide such provable safety guarantees. We believe the value of our work lies in presenting a new approach for safe RL that is validated by both theoretical insight and empirical results.\n\n### [Other Comments]\n\n**Confusing sentence in Abstract.** Thank you for flagging this ambiguity. We optimize the target return inputs $(R,G)$ to steer a fixed policy, maximizing rewards under safety constraints. We will clarify this in the abstract with your suggested text: \"...while cautiously optimizing a limited set of inputs, known as target returns, that condition the policy's behavior.\"\n\n**OPE and distribution mismatch.**\nOur intention on lines 75-77 was to convey that the difficulty in learning a good policy offline is associated with the difficulty of accurately evaluating the long-term performance from a static dataset. The inability to perform reliable OPE, primarily due to the distributional shift, often leads to overly conservative or unsafe policies. We agree with the reviewer that it would be a more precise representation to frame this issue as a result of distribution mismatch. In the final version of the paper, we will revise the sentence to more accurately reflect this nuance.\n\n**Context-dependent policy.** A context-dependent policy defined in the paper is not necessarily Markovian. We define a context-dependent policy as a more broader class than a state-dependent policy to conduct theoretical analyses of RL policies based on Transformer and the mixture of multiple such policies.\n\n**Undiscounted returns.** Thank you for helpful suggestions. We will add explanations that undiscounted returns or safety costs are considered.\n\n**Interesting approach.** Thank you for this encouraging feedback and for highlighting the novelty of our approach! Our choice of a GP, while non-standard, was directly motivated by our theoretical analysis (Theorem 1). Its ability to provide principled uncertainty bounds is what enables the provable safety guarantees that are central to our work."}, {"review_id": "HxcZgP54BN", "reviewer": "Reviewer_A6AP", "summary": "The paper studies an important problem of learning safe policies offline, continuing to adapt online while maintaining safety upon deployment. The paper frames safety with constrained Markov decision processes and proposes using Constrained Decision Transformers (CDT). CDTs are used to train policies that are conditioned on the reward and cost returns, allowing policies to maintain different levels of constraint satisfaction “zero-shot”. The proposed method then uses safe Bayesian optimization techniques to gradually increase the budget, expanding the set of safe policies while maintaining safety. \n\nThe paper gives formal safety guarantees, as well as optimality guarantees w.r.t. an optimal policy that can be extracted from the offline dataset. The proposed method (“PLS”) is then evaluated against existing baselines in safe offline RL literature, showing significant improvement in terms safety and performance.", "questions": "* How many seeds were used in the experiments? Could you add error bars to figure 3?", "limitations": "As discussed above, I believe that the main limitation of the paper is that it tackles a problem in which the offline dataset has high-coverage of the MDP. While the paper makes a solid contribution on this problem, I believe a more crucial (and realistic) challenge for offline-to-online algorithms, especially when safety is a concern, is when the offline dataset is limited, thus online adaptation _of the policy_ is required.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n* The connection between (budget-conditioned) CDTs to safe Bayesian optimization is novel and interesting.\n* Identifying the bias error between the target reward/budget to the those measured online (Figure 2) is interesting, using a GP to regress the error is correct, and could transfer to similar problems like safe sim-to-real transfer.\n* The paper is presented very well.\n* Experiments are comprehensive, showing improvement over existing methods.\n* The theory seems solid and novel in some aspects.\n\n## Weaknesses \n\n* The proposed method does not in fact continue training the policy with online data. Offline data is generally limited, and online data, being expensive, would not be discarded. Therefore, a more reasonable problem setting would consider the case where the policy adapts/learns while data is collected (similar to [3]). The paper addresses the problem of scalable online end-to-end safety _only_ when the offline dataset has high-enough coverage, meaning that one can extract a good enough policy from it, without ever improving it online. Unfortunately this is generally not the case in practice, with many algorithms that try to address precisely this challenge [1, 2]. I believe this point should be made more clear in the presentation of the paper. While this is mentioned to some extent in the conclusions section, making this limitation more explicit would make the contribution more transparent and will improve the paper.\n\n* The assumption on good data coverage (Assumption 2) might be unrealistic. While this is a fair assumption to make when one theoretically studies the problem, in the context of safety, lack of data coverage, or “distribution mismatch issues between the offline data and the actual environment”, can be detrimental. The paper does not touch upon this limitation.\n\n* In practice, cost and reward return measurements can be quite noisy on real hardware. Assumption 1 covers this case theoretically, however this could be quite limiting, requiring a significant amount of online measurements to fit the GP.\n\n* minor: the method relies on CDTs, making it less practical in problems that require other learning techniques that leverage offline data (e.g. the recent advancements in imitation learning and diffusion/flow-matching models).\n\n[1] Ball, Philip J., Laura Smith, Ilya Kostrikov, and Sergey Levine. \"Efficient online reinforcement learning with offline data.\" In International Conference on Machine Learning, pp. 1577-1594. PMLR, 2023.\n\n[2] Xiao, Wei, Jiacheng Liu, Zifeng Zhuang, Runze Suo, Shangke Lyu, and Donglin Wang. \"Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only.\" arXiv preprint arXiv:2505.16856 (2025).\n\n[3] As, Yarden, Bhavya Sukhija, Lenart Treven, Carmelo Sferrazza, Stelian Coros, and Andreas Krause. \"ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning.\" arXiv preprint arXiv:2410.09486 (2024).", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer’s encouraging comments. The thoughtful comments of Reviewer A6AP are valuable and help us improve the manuscript's quality. We will answer the Questions and then reply to the comments in Weaknesses.\n\n### [Question] How many seeds were used in the experiments? Could you add error bars to figure 3?\n\nWe used five random seeds for each algorithm and environment in our experiments. Figure 3 in the current manuscript presents a representative result from a single seed to illustrate the safe exploration behavior over GP iterations. In the camera-ready version, we will also add a new figure showing averaged performance (reward and safety cost) with standard deviations across multiple runs, as suggested by the reviewer.\n\n### [Weakness 1] Offline-to-Online RL\n\nWe thank the reviewer for highlighting this important point. We agree that our current method does not update the policy network with online data. This was a deliberate design choice to achieve the primary objective of this work: to propose an end-to-end safe RL algorithm with provable theoretical guarantees.\n\nOur main goal is to ensure the safety of a policy throughout the entire process, from learning to long-term operation. The current problem formulation, which first learns a return-conditioned policy offline and then cautiously optimizes low-dimensional target returns online, is crucial for this goal. This approach allows us to provide high-probability safety guarantees (Theorem 2) and near-optimality guarantees for the target returns (Theorem 3), addressing a longstanding challenge in safe RL.\n\nHowever, we agree that a natural and valuable extension would be to incorporate online policy updates. Our current framework can serve as a foundation for such an offline-to-online approach. It would be an important future research direction to investigate how to perform continual learning with new online data while still preserving the theoretical safety guarantees. As suggested, we will make this limitation and the potential for future extensions more explicit in the final version of the paper to improve its transparency.\n\n\n### [Weakness 2] Assumption on good data coverage\n\nThank you for raising this important point regarding the practicality of Assumption 2. While Assumption 2 is necessary to provide the initial theoretical grounding for the policy trained offline, we agree that assuming good data coverage may be a strong requirement and that a mismatch between the offline data and the online environment is a central challenge in offline RL.\n\nFrom the perspective of safety guarantee, we would like to clarify that Assumption 2 is only required at the points of interest: the initial target return $z_0$ and the subsequent points $z_N$ selected during early stages of exploration. In practice, by choosing $z_0$ from a region well-covered by the offline dataset, and due to the uncertainty-based selection in PLS, early exploration remains local. This limits reliance on the assumption to neighborhoods where the offline data is dense.\n\nMore crucially, our PLS is specifically designed to mitigate the bad effects of this exact \"distribution mismatch\" during deployment. Specifically, in safe online optimization phase, the role of GPs is precisely to learn the discrepancy between the target returns and the returns achieved online, effectively correcting for the distribution mismatch. \n\nOur PLS uses the GP models to cautiously and safely optimize the policy's behavior in the online environment, with high-probability safety guarantees (Theorem 2). Therefore, rather than ignoring this limitation, our work is a direct attempt to bridge the gap from a potentially limited offline dataset to safe and effective online operation.\n\n\n### [Weakness 3] Noisy measurements on real hardware\n\nWe agree that reward and safety cost return measurements on real hardware can be noisy, and this can affect the number of online samples required to fit an accurate GP model. While our theoretical framework covers this case through Assumption 1 and the noise-aware nature of GPs, we designed our PLS method specifically to be sample-efficient during the online phase. The key to this efficiency lies in the reduced dimensionality of the optimization problem. Unlike conventional RL methods that optimize a high-dimensional policy network, PLS only optimizes a two-dimensional target return vector $z = (R,G)$. As stated on lines 297-299 in the paper, because we are fitting a GP to this simple two-dimensional input space, PLS requires much fewer online interactions than conventional online safe RL algorithms. This low-dimensional optimization significantly reduces the number of online measurements needed to learn the relation between target and actual returns, even in the presence of noise.\n\n### [Weakness 4, minor] Dependence on CDTs\n\nThank you for the helpful comment. We agree that our method currently builds on CDTs, which may limit direct applicability in settings favoring alternative offline learning approaches such as imitation learning or diffusion/flow-matching models. However, our framework is grounded in the more general RCSL paradigm, and CDT was chosen as a representative and well-established instance. The structure of CDT enables us to develop a theoretically justified, end-to-end safe RL method. Such theoretical guarantees are currently challenging to establish for alternative paradigms, which typically lack this level of analytical tractability."}]

Author rebuttal:

Meta-review: The paper proposes a method called Provably Lifetime Safe RL (PLS) to ensure a reinforcement learning policy is safe throughout its entire lifecycle, from training to deployment. 

### Key Strengths:
- The combination of offline RL with online safe optimization using Gaussian Processes was seen as novel and well-motivated.
- The paper's main strength was its rigorous theoretical analysis, providing provable safety guarantees, which addresses a significant gap in safe RL.
- Empirically, the PLS method consistently outperformed baselines by successfully maintaining safety across all experiments while achieving competitive rewards.

### Weakness:
- Several reviewers noted that the theoretical guarantees rely on strong assumptions (e.g., near-deterministic environments) that may not hold in real-world scenarios.
- The most significant point of contention was the limited number of experiments. One reviewer (njCE) rated the paper "borderline reject" primarily because the evaluation wasn't extensive enough, even though the theory was solid.
- Some reviewers found the theoretical sections and notation complex and hard to read.
- The method only optimizes the target returns online, not the policy network itself, which limits its ability to adapt from new online data.

### Author Rebuttal:
The authors provided a comprehensive rebuttal that successfully addressed most concerns.

### Conclusion:
The final consensus among the reviewers and the AC was positive
Decision: Accept (poster)