Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
Authors: Keru Chen
Honghao Wei
Zhigang Deng
Sen Lin
Venue: Accepted by TMLR ()

Abstract: High costs and risks involved in extensive environmental interactions hinder the practical application of current online safe reinforcement learning (RL) methods. Inspired by recent successes in offline-to-online (O2O) RL, it is crucial to explore whether offline safe RL can be leveraged to facilitate faster and safer online learning, a direction that has yet to be fully investigated. To fill this gap, we first show that naively applying existing O2O algorithms from standard RL would not work well in safe RL due to two unique challenges: \emph{erroneous Q-estimations}, resulted from offline-online objective mismatch and offline cost sparsity, and \emph{Lagrangian mismatch}, resulted from difficulties in aligning Lagrange multipliers between offline and online policies. To address these challenges, we introduce \textbf{Marvel}, the first policy-finetuning based framework for O2O safe RL, comprising two key components that work in concert: \emph{Value Pre-Alignment} to align the learned Q-functions with the online objective before finetuning, and \emph{Adaptive PID Control} to effectively adjust the Lagrange multipliers during  finetuning. Extensive experiments  demonstrate the superior performance of Marvel over related baselines.

Published in Transactions on Machine Learning Research (01/2026)
Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
Keru Chen
kchen234 Casu. edu
School of Electrical, Compuler and Eneryy Engineering Arizona State University
Honghao Wei
honghao Brnnghao.weiiussee wwwiwws..eeee edu
School of Electrical Engineering and Computer Science Woshington State University
Zhigang Deng
zdengg @central uh. edu
Department of Computer Science University of Houston
Sen Lin'
slin50@central. uh. edu
Department of Computer Science University of Housion
Reviewed on OpenReview: https: openreview. net/ forum? id= 1S07umL.Fp
Abstract
High costs and risks involved in extensive environmental interactions hinder the practical application current online safe reinforcement learning (RL) methods. Inspired by recent successes offine-to-online (O20) RI, is crucial to explore whether offline safe RI. can be leveraged to facilitate faster and safer online learning, direction that has yet to be fully investigated. To fill this gap, first show that naively applying existing 020 algorithms from standard RL would not work well in safe RI. due to two unique challenges: erroneous Q-estimations, resulted from offline-online objective mismatch and offline cost sparsity, and Lagrangian mismalch, resulted from difficulties in aligning Lagrange multipliers between offline and online policies. To address these challenges, we introduce Marvel, the first policy-finetuning based framework for 020 safe RI, comprising two key components that work in concert: Value Pre-Aligmeent objective before finetuning, and Adaplive PID Control to effectively adjust the Lagrange multipliers during finetuning, Extensive experiments demonstrate the superior performance of Marvel over related basclines.
align the learned Q-functions with the online
1 Introduction
prioritizes not only reward maximization but also the adherence to safety constraints, enhancing its applicability in real-world scenarios. For instance, an autonomous vehicle must reach its destination without excceding a fuel limit. However, solving safe online RL from scratch in fields such as robotics (Brunke et al.) 2022 Kiran et al.] 2021] and
Safe reinforcement learning (safe RL) Gu ct al. 2022 Garcia & Fernándezi 2015)
healthcare (Yu et al. 2021 Qayyum et al.) 2020) often prohibitive, due to the high risks and costs of extensive environment interactions. address this, offline safe RT. (Zheng al. 2024 Ray et al. 2019 has been introducod, cnabling learning safe policies from static datasct without online interactions. Yet, offline
CCorreponding author.

---

Published in Transactions on Machine Learning Research (01/2026)
safe RL faces its own limitations (Ghosh et al. 2022): it typically shows limited performance, relies on the quality of the offline dataset, and suffers from the impact of out-of-distribution (OOD) actions, restricting its effectiveness across varying scenarios.
The pretraining-an and-finttnning paradigm is a well-known strategy in the fields of computer vision and nat- ural language processing, for enabling fast online learning based on offline pretrained models. Following a similar line, oflline-to-online RL (020 RL) in the unconstrained setting (Nair et al. 2020 Wang et al. 2024 Zhang et al. 2023b) and imitation learning (Yue et al. 2024 Ross et al., 2011) have recently gained promi- nence. These approaches utilize policies (including Q-functions) derived from offline learning, along with offline datasets, to expedite online finctuning, which effectively avoids extensive environmental interactions in training policies from scratch. Note that in offline-lo-omleee but rather to obtain a safe warm start from offline data and perform limited online finetuning under safety constraints. Thus motivated, a key insight is leveraging the pretraining-aue -aaaaiiinng-n---nnunnnnng paradigm can also potentially facilitate more efficient and practical online safe RL, which however has not been fully explored in the literature. To fill this gap, we seek to answer the following question: Can we design an effeclive offtine-to to-tne-ttoone online approach for safe RL to address the limitations of both online and offline safe RL, thereby enabling fast online safe policy learning?
safe RL, the goal is not large-scale pretraining
BallCircle
BaliCircle Visualizatior
BeliCircle Cost Visualization
700
Start From Scratch Marvel(ours)
600
150
500
125
Reward
-10
400
Warm Start From Scratch
Cost
100
100
166
300
Morve
75
CarRunn Reward Visualiztioon
Car Run Cost Visualization
200
50
100
25
0
0
-50
0
100
200 Steps
300
400
500
100
200
300
400
500
-0
Steps
392 -153
-90
100
(a)
(b)
Figure "Steps" on the x-axis represent the number of policy gradient updates (i.c., optimizer updates). For each update, the agent interacts with the environment for episodes. This convention is followed in the subsequent figures. In (a), we evaluate these methods in BallCircle from Bullet Safety Gym (Gronauer 2022), with cost limit of 20. As shown, while "Warm Start" begins with a reasonably good initial policy, it performs poorly and overly conservatively, even worse than "From Scratch" where the policy and Q-functions are initialize randomly. This implies that directly finetuning the pretrained policy and Q-functions may hinder online learning. In contrast, "Marvel" achieves impressive results, finding a policy with much higher return in just few online steps while adhering to the cost limit. In (b), t-SNE visualization of state in the environment, reduced to 2D space. Each point represents a stale, with rewards uniformly distributed across the space, while costs are sparse, appearing as isolated points clusters, reflecting their limited association with states.
However, achieving this is highly nontrivial, and simply applying existing 020 algorithms in conventional RL would not work well here due unique challenges in safe RL. In Fig. D (a). 'Warm Start' refers to using the oflline pretrained policy and Q-networks directly to initialize an online safe RL algorithm. 'From Scratch' refers to purely online safe RL training. As illustrated, directly finetuning the offline pretrained policy and Q-functions by using standard online safe RL often results in suboptimal performance and, in some cases, complete training failures.
The reasons behind this phenomenon are as follows: a) Erroneous Q-estimations resulting from objective mismatch and offline cost sparsity. To avoid explorations beyond offline data and reduce the extrapolation errors, offline safe RL algorithms typically introduce additional regularizations in the objective function push up the cost estimates of OOD actions, e.g., VOCE (Guan et al. 2024 and CPQ (Xu et al. 2022
2

---

Published in Transactions on Machine Learning Research (01/2026)
leading to different overall objective from standard online safe RL. More critically, the majority of state- actions in offline datasets for sale RL usually are safe with wero cost (Fig. [ (b)), resulting in pretrained cost Q-function that predicts extremely low cost for most in-distribution (LND) state-actions. By erroneously giving high values for OOD state-actions and low values for IND state-actions, the pretrained cost Q-function will conservatively force the online finetuning to stay in the stale-action space similar to offline dataset and be reluctant to explore (e.g., cost of "Warm Start" in Fig. D.. b) Mismatch of Lagrange multipliers. Many online safe RL algorithms (Chow et al. 2018a] AAhiaa al., 2017) solve the constrained optimization problem based on the primal-dual approach, which requires synchronous update of Lagrange multipliers. Nonetheless, initial values for these multipliers that match the offline policies cannot be obtained from offline safe RL precisely. Using traditional dual ascent methods to update the Lagrange multipliers may result in slow online learning even with accurately estimated Q-functions, ultimately degrading the performance of the learned policy. In this work, we seek to design an effective 020 framework for safe RL by addressing these two challenges above.
The main contribution of this work is the WarM-stArt safe Reinforcement learning with Value prE- aLignment (Marvel) framework, consisting of two key components: Value Pre-Alignment (VPA) and Adaptive PID Control (aPID). VPA re-aligns pretrained Q-functions to closely match true Q-values of the offline policy under the online learning objective by re-evaluating the offline policy based on offline data alone. To further compensate VPA and address the multiplier mismatch problem, aPID quickly adapts Lagrange multipliers based on cost violations instead of directly finding the best initial multipliers. Experi- mental results demonstrate Marvel's ability to quickly learn a safe, high-reward policy with minimal online interactions, marking it as a pioneering solution in O20 safe RL that integrates seamlessly with existing offline and online safe RL methods.
2 Preliminaries
Constrained Markov Decision Process. We consider a standard constrained Markov Decision Process (CMDP) (Altman 2021), defined by a tuple (S, A, T. R, C, 17, Cx).). Here S is the state space, A is the action space, ST:SAxx + [0. 1] is the transition probability function, R: S + [0, Rmax] is the reward function, and C : S x A + [0, Cmax] is the cost function. y E [0. 1) is the discount factor, represents the initial state distribution, and eth is the cost. threshold that sets the limit on cumulative costs for the policy. A policy π. S + P(A) is mapping from states to a probability distribution over actions, where n (als)) denotes the probability of selecting action in state s. Let the policy Ty be parameterized by A. Given a policy n, its cumulative reward is defined as R(T)- = Er-r 12 "r((s, dt)], where = (5o, ao, $1, a1....) is trajectory induced by policy n, and the expectation is taken over the distribution of trajectories. Similarly, its cumulative cost is defined as C(T) Eywr 1E 7c((st, at).. The reward Q-function for a policy π is defined as: (=, a) = Rraz EE 7'r(s,, ae) 80 = 8, 40 a]. Similarly, the cost Q-function Q (s, a) is defined as the expected cumulative cost starting from the same state-action pair (s, a) and thereafter following policy Q7 (s, a) EE, IE 7'((st,a)) so = s, ag a). In CMDP, the goal is to find an optimal policy x² that maximizes R(TT), subject to the constraint that the cunulative cost C'(T)) does not exceed a predefined threshold cas- This can be formulated as the following:
max, R(T), s.t. C(T) S ceh-
(1)
To solve this, a common approach is to apply the Lagrangian relaxation method (Ray et al. 2019) by introducing a Lagrange multiplier X to enforce the cost constraint, leading to the following primal-dual formulation:
minx20 max, [m(=) X(C(=) - -cw)],
(2)
which can be solved by iteratively updating policy π and x. X is updated with the learning rate ax:
Aet1 = >e + ax(((m) - cx.)
Online Safe RL. Primal-dual based algorithms have shown great effectiveness and superior performance in the literature for online safe RL. Without loss of generality, we consider SAC-lag (Ray et al. 2019) as the online algorithm, which integrates the widely used SAC algorithm (Haarnoja et al. 2018) with the Lagrange multiplier method. More specifically, SAC minimizes the following objectives for the actor (policy) and the
3

---

Published in Transactions on Machine Learning Research (01/2026)
critic (Q-function), respectively:
CEAC (0) and, a>xa(11)) La (alog -((a,))))))(((aa ((s,
(4)
c& (w) - (a.a
E (Q((, =((,)())²)
(5)
where v(r, = r >Ba (2(s, d':w") - a log ('))s'),, Q(s, w)w) is parameterized by w, Q(s, ((s,a;w') w²) is the larget reward Q-function paramelerized by w', d represents the data distribution in the replay busler, and a > 0 is some constant. To be applied in online safe RL, SAC-lag adapts SAC by using the Lagrangian method, such that:
(3)C(0) -a log (a(as)) (Q(s, AQ=(s,)))]-
The optimization of the Q-functions for both reward and cost in SAC-lag is with Eq. in SAC.
Offline Safe RL. Offline safe RL algorithms typically push up the cost estimations of OOD actions to avoid exploration beyond the offline dataset D. Considering the comprehensive performance across various environments, in this paper we consider the SOTA Lagrangian- CPQ Xu et al.] 2022). More specifically, CPQ first generates OOD actions via conditional variational autoencoder (CVAE). The cost of the generated OOD actions is increased by minimizing the following loss function for cost critic (Q--function):
algorithm for offline learning, namely
CoPo Cq. (w.) En (Qo(s, - (Ea 12o(s' d(w&)))) V. - (Q (Q,(s,; a;.)]
where D represents distribution of (s,a,s') s') in the offline dataset. (c(s,a;wa) is parameterized by wes De(s,a;ad is the larget cost Q-function parameterized by w/ represents the distribution of OOD ac- tions generated by the CVAE. Additionally, to ensure both constraint safety and in-distribution safety, CPQ updates the reward Q- Q-function using only state-action pairs that satisfy the cost threshold
LQ (w) ED (Q(s, w) (r
((Qaass a'; ww.) 1)Q(s's ( [)
where I(.) the indicator function. The policy loss function is defined as: L CCPQ (0) -ED (ee a;w.) < (s) a, w]]] Similarly, when maximizing the reward, the policy only con- siders state-action pairs that meet the safety constraints. By assigning higher cost to OOD actions, CPQ mitigates the OOD problem while meeting safety constraints.
020 Safe RL. To the best of our knowledge, Guided Online Distillation (Li et al. 2024 is the only work studying 020 safe RL, which leverages a large-scale DT based on GPT-2 (Radford et al. 2019) as a guide policy to accelerate online learning, by following the idea of Jump-start RL (Uchendu et al. 2023). However, how to achieve fast safe online learning by finctuning pretrained policy is still not clear due to the mismatch of Lagrange multipliers and erroneous Q-estimations from objective mismatch and offline cost. sparsity.
In this work, we seek to enable faster and safer policy learning by finctuning policics and Q-functions pretrained via oflline safe RL, using slandard online safe RL methods. Our approach does not rely on large models and is intended as an initial step toward promoting further research on policy-finetunigg based 020 safe RL. In principle, any offline safe RL algorithm that outputs a policy and Q-functions can be used in the offline pretraining stage. More details of related work are provided in Section
3
020 Safe RL with Value Pre- AlegAliigment
As shown in Fig. [ naively finetuning the offline policy for safe RL would not work and the finetuned policy shows clear "inertia" in improving its performance: within long period after online finetuning starts, its cost stays far below the limit, but its reward is quite low and not improving. This implics that such a strategy automatically "inherits" the conservatism from oflline safe RL and is reluctant to actively explore to fully utilize the safe gap below the cost limit. In this section, we delve into this failure of naive finetuning, which points to two unique challenges for policy finctuning in 020 safe RL, i.c., erroncous offline Q-estimations and Lagrange multiplier mismatch. To address these, we propose a framework for 020 safe RL, i.., warM-stArt safe RL with Value prF-alignment (Marvel).
4

---

Published in Transactions on Machine Learning Research (01/2026)
3.1 Pre-Finetune Phase
Challenge I: Erroneous Q-estimations resulting from objective mismatch and offline cost sparsity. By learn- ing from fixed dataset only, offline safe RL typically suffers from large extrapolation errors for OOD actions beyond the support of the dataset. A general principle to handle this is to penalize the reward/cos estima- tions for the OOD actions in such way that risky explorations outside the dataset are discouraged. The Q-functions can be optimized as follows:
Q: En (Q(s.a) - (r yBa ( + - P.s
(6)
Qei min ED (Qa(s,a) - (c yEu 110 a'))) the Pe(s,
(7)
Here Teval denotes policy under evaluation. 16 (000)) and w. Pa(s, (oon) are the penalty terms. For instance, penalties are introduced in VOCE (Guan et al. 2024) to minimize the reward Q-values and maximize the cost Q-values for OOD actions. CPQ (Xu actions during Q-function and policy updates. In contrast, the optimization of Q-functions in online safe RL is standard without any penalty terms, i.e, 1 in Eq. and Eq. 7
2022) increases the perecived cost of OOD
Obviously, Offlinc and online safe RL have distinct objectives for Q-functions, meaning pretrained Q-functions may not accurately estimate values for state-action pairs encountered during online interactions. As result, offline policies tend to act overly conservatively, exploring only low-cost regions during online finetuning. This limitation actually arises from the joint effect (i) sparse offline costs leading the pretrained cost critic Qe to assign low costs to IND actions-and (ii) offline pessimism that assigns high costs OOD actions through conservative extrapolation. Under Lagrangian update, the policy thus prefers to stay within the low-cost IND region and avoids OOD regions that the critic deems high-cost, regions contain high rewards. However, effective online learning requires identifying state action pairs with both high rewards and low costs, which necessitates exploring areas with potentially higher costs. This objective mismatch is even more pronounced in 020 safe RL due to the additional cost Q-function.
when those
Solution: Value Premmnigm. To address the first challenge, a naive approach to reevaluate the offline policy no in online environments using Monte Carlo simulations, which however introduces additional interaction costs. Motivated by the recent advances in Off-Policy Evaluation (OPE) (Cehara al. 2022) borrow the idea from Fitted Q Evaluation (Hao et al. 2021 to align the offline Q-functions with the online learning objectives for the offline policy, by using the offline dataset before online policy finetuning. Starting from the pretrained Q-functions, we seek to minimize:
C8 CVPA (w) = En (o(s,a) - -7 -1 VPA PPAlog logoo + +r))
(8)
cØ. (wo) = ED (Qo(e,a) (1
a YPA log (g + +c) ) 27
(9)
where w denotes parameters of Q function and Q is target Q function. Here the entropy terms can result in both higher rewards and costs for state-action pairs with high entropy, where pretrained policy is 'uncertain! Offline policy remains unchanged during VPA to preserve knowledge extracted from offline data.
Assume that the mismatch between the offline policy-induced distributions of the learned policy To and the d_o(a.0)
offline behavior policy μ is bounded by constant < oo, i.e, max ("(s,a) C.C. Let Qu. QK, and
TTe be Q-function obtained from offline phasc, Q-function after K itcrations in VPA and the optimal (fixed- point solution) Q-lunction for reward no, respectively (similarly, Q0,0; Qr,K, and Q for cost Q-functions). We can have following result justify effectiveness of VPA:
Theorem 1. Assume that the reward r and cost are bounded by 1 for all (s, E S x A. With probability at least. 1 5, the Q-estimation errors in VPA can be bounded in the weighted I-z-orr [|. |2,avo under the
5

---

Published in Transactions on Machine Learning Research (01/2026)
state-action distribution induced by policy No!
VCe TTH 112,070 1 1-7
11Qx
+ 1p11100 - -TTO 12.a0o,
(10)
VCec
1|0,,K - 112,00 S 1-7 + x 1Q0,0 - @02112..0m
(11)
where and é are the approrimation errors. It decreases the size of the dataset increases while increases with the values of a VPA and a. A respectively.
The full proof and the specific expression é are provided in AppendixHl As shown in Theorem I the overall estimation error consists of two terms, i.c., the statistical error in the first term and the iteration error the second term. With larger dataset and more expressive function approximator, the first term vanishes, and the overall error is dominated by the iteration error, which decays exponentially with training steps K. This indicates that VPA can yield increasingly accurate Q-function estimates. Moreover, since reward optimization dominates policy updates at the carly stage of online finetuning when Lagrange multiplier is small, a larger uncertainly in reward Qeestimations is preferred to facilitate more online exploration. To this end, Theorem 1 further implies that a larger a VPA should be selected in Eq. g for reward Q-values, whereas a PPA in Eq. should be smaller to ensure a more accurate estimation of cost Q-values. While the theory suggests improved Q estimations for IND and OOD actions, our empirical findings show estimations for OOD actions after VPA tends to be overoptimistic. This discrepancy is likely due to imperfect data coverage and function approximation errors. Interestingly, while such overestimation is problematic and avoided for offline RL without online explorations, it is indeed beneficial for 020 RL during early phase of online finetuning, which can encourage more active explorations beyond the offline dataset. While Luo ct al. (024)) considered a similar analysis for unconstrained RI, we focus on the safe RL sctting and further introduce VPA, a pre-finetuning procedure incorporating an entropy term to mitigate erroneous Q-value estimations.
Table Spearman's rank correlation coefficients of Q-value and Qo-aalue in BallCirele and CarRun. "Random" refers to rollouts starting from randomly initialized state-action pairs (potentially OOD), while "Dataset" refers to rollouts from state-action pairs sampled from the offline dataset.
BallCircle
CarRun
VPA
Random
Dataset
Random
Dataset
Before
-0.2387
-0.3852
-0.1143
-0.5078
Q Q-value
After
0.5661
0,8278
-0.0125
0.8314
Before
-0.2521
0.1725
-0.2431
-0.4327
Qe-value
A After
0.3579
0.8252
0.1254
0.4937
To empirically characterize the performance of VPA in correcting Q-estimations, we leverage Spearman's 's rank correlation coefficient, which measures the strength and direction of a monotonic relationship between two ranked variables. The reason that relative ranking of Q-values are more important than absolute values for policy update. We provide more details of Spearman's rank correlation coefficient in Appendix G.I Specifically, given dataset collected by the offline policy in the environment, learned reward/cost Q-values before and after VPA with that of estimated actual return via Monte Carlo simulations. A large Spearman's 's rank correlation cocfficient implics that distribution of learned Q-values is more aligned with the distribution of true Q-values. As shown in Table 1 il is evident that coefficient increases significantly after VPA for both reward and cost Q-values, no matter offline policy rolls out from a soen statc-action pair in offline dataset or from a randomly selected OOD state-action pair. This clearly demonstrates effectiveness of VPA in aligning the pretrained Q-functions.
compare the ranking of
3.2 Finetune Phase
Challenge II: Lagrange multiplier mismatch. Conventional value-based online safe RL relies on updating Lagrange multipliers alongside the policy and Q-fucctins during training, to push the overall cost below

---

Published in Transactions on Machine Learning Research (01/2026)
the limit while balancing between maximizing the reward and minimizing constraint violations. Although the policy and Q-functions can benefit. from offline pretraining for warm start, offline safe RT. algorithms like CPQ (Xu et al. 2022) and BEAR-lag (Ray et al. 2019) cannot accurately estimate Lagrange multipliers with regularizing strengths matching the cost of the offinn policy, c.g., a small multiplier is not powerful enough to push down the policy cost, while a large multiplier prevents active exploration of high-reward state-action pairs. For instance, in BallCircle, the offline Lagrange multiplier value obtained using the BEAR-lag algorithm is approximately 1500, whereas during online finetuning, the SAC-lag requires a value of only about 0.65. The gap between these values precludes the direct use of offinn Lagrange multipliers. Improper initialization can lead to extensive constraint violations or training stagnation, an issue we term as the Lagrange multiplier mismatch.
BailCircle
CarCircle
360
VRA wo init
VSA wio init
VRA w/o In
120
VRA wrO Ini.
wi
80
VMA
340
VPA
VPA init
320
100
Reward
Reward
300
Cost
80
40
280
50
260
40
20
240
230
300
100
Steps
Steps
Steps
Steps
Figure 2: Comparison of online finetuning performance after VPA with two different initial values of the Lagrange multiplier. In 'VPA w/o init', the initial value is set to 0, whereas we initialize Lagrange multipliers with a good value found empirically (0.65 in BallCirele and 0.5 in CarRun) in 'VPA w/ init'. The multiplier is then updated using the standard dual ascent method.
On the other hand, VPA promotes active exploration of high-reward statc-actions by optimistically esti- mating rewards and pessimisticall) estimating costs, it inevilably increases the risk of exploring high cost state-actions, which in turn amplifies the need for appropriate Lagrange multipliers to quickly reduce the constraint violations.
Figure 2 compares online finetuning performance BallCircle when using VPA with: (1) a well-initialized Lagrange multiplier vs. (2) zero initialization, both with standard dual ascent updates. A good initial multiplier enables effective cost control, while poor initialization leads to significant constraint violations and slower convergence. The results also imply that although VPA aligns the distributions of Q-values, it may introduce high costs for online finetuning, which can be addressed with an appropriate initial Lagrange multiplier.
Solution: Adaptive PID Control. Finding a good initial value of the Lagrange multiplier can address the mismatch problem and mitigate the risk brought by VPA. However, obtaining such a value by manual tuning is challenging, and no theory provides a reliable prediction. We therefore adapt the multiplier rapidly and effectively by using a PID-style controller that has shown strong empirical performance in online safe RL (Stooke et al. 2020 Yuan et al. 2022 Zhou et al. 2021), and we tailor it to the 020 setting.
We use discrete, position-form PID update. Let the discounted total episode cost be at training iteration 4, the limit be Cεk, and the violation be 6, Q Ga-- Define an EMA-smoothed violation & = a (1-op))c with smoothing factor ap E 1). Define an smoothed cost & = a.t-1+ (1- with smoothing factor aa E 1), and derivative buffer of length E N. The integral state and the derivative signal follow the implementation:
= max (0, I + Kiet),
D. max(0, ē Ee-a).
The multiplier is then
A max (0, K2 E T + Ka Di).
(12)
7

---

Published in Transactions on Machine Learning Research (01/2026)
This position-form matches the code path where the proportional term uses ét, the integral term uses et: and the derivative term uses a lagged difference of smoothed costs D.. The integral channel provides the same role as dual ascent.
VPA relaxes offiine conservatism and increases exploration of high-return state-action pairs during early finetuning, which can raise violations. A larger control strength early stages reduces violations rapidly, and a smaller control strength near the limit stabilizes learning and suppresses oscillations. This motivates adaptive gain schedule.
Gains are adapted from short buffer of smoothed costs. Let the buffer contain the last n values of é that are also used to form the derivative buffer. Define the average C₁ = 3/4 Em G+11-s and the standard
deviation or V Σ - (c+11 0.)2. With coefficients a, 3, y O, the update is
4 Cth
Kp + clip
Kp. + a
Kp.iins K Kp.max)
(13)
4
c Cth
K; + clip
K
1+ B.
Ki,min Ki,nax
(14)
Kd + clip
Tt Ka. 1 11
Kd,mins Kd,max
(15)
C
This schedule increases (Kp: K,) when C₁ > Cτh to raise low-frequency gain and accelerate correction, and decreases (Kp, K2) when & < Cah to avoid overrcaction and improve stability. The schedule increases K₃ when or is large to provide additional damping against short-horizon volatility. Ka tends to platcau naturally once or stabilizes. The minimum and maximum bounds of each PID gain are set to 0.1 and 10 times its initial value, respectively. Since Ka is non-decreasing by design, we set Kd,min only keep notation consistent with (Kp.iiu K.
Appendix D.3.2 demonstrates that these parameters are easy to tune but also robust in their own selection (e.g., same across all environments in the experiments).
In a nutshell, combining VPA and aPID leads to our proposed framework Marvel: 1) Given the pretrained policy and Q-functions from offline learning, Marvel first applies VPA align the pretrained Q-functions for both reward and cost using the offline dataset; 2) Marvel next utilizes Lagrangian algorithms to further finetune both the pretrained policy and aligned Q-functions, by using aPID to update the Lagrange multipliers. We present the algorithmic framework of Marvel in Appendix A
online safe RL
Here VPA and aPID work in concert: aPID addresses the Lagrange multiplier mismatch problem and quickly pushes down the potential high cost resulted by VPA, whereas VPA facilitates active exploration of high- reward state-action pairs and the usage of the pretrained policy as a warm-start for fast online finetuning with aPID control.
4 Experiments
The objective of O20 safe RL to utilize offline data to enable fast online finetuning, i.e., maximizing rewards while satisfying the cost threshold, with minimal interactions with the environment. Therefore, an important evaluation criterion for effective 020 safe RI. methods is whether the method can quickly obtain a good policy with high rewards and low safety violations only after limited number of online interactions, which is the setup considered in this paper. Due to the space limit, we delegate the experimental details and some additional results to Appendix D and Appendix G
4.1 Evaluation Setup
Benchmarks. We consider the DSRL benchmark (Liu et al.) 2023b) and select ten environments from the Bullet Safcty Gym (Gronauer) 2022) and Safcty Gymnasium (Ji et al. 2023): BallRun, BallCircle, CarRun, CarCirele, HalfCheetah, AnrCle AntRun, DroneCircle, Hopper, and Swimmer (results for the last four are in Appendix D.I.p The cost. threshold is set to be 20 in these environments. As mentioned earlier in
8

---

Published in Transactions on Machine Learning Research (01/2026)
Section 2 we choose CPQ and SAC-lag as base algorithms in our proposed framework Marvel for offiine training and online finetuning, respectively, due to the effectiveness and representativeness of them. Each experiment was conducted using five random seeds, and the results were averaged to generate the final learning curves. We use a dataset that includes data provided by DSRL (Liu et al. 2024 and random data generaled by a random policy to control the quality of the offline dataset.
Baselines. While Guided Online Distillation (Li al. 2024) is the only work studying 020 safe RL, its usage of large pretrained model leads to unfair comparison with standard RL frameworks using typically small-scale policy networks. In this work, compare Marvel with JSRL (Uchendu et al.) 2023), as Guided Online Distillation mainly follows this approach except using DT as the pretrained policy. Besides, we further adapt some SOTA approaches in O20 RL to 020 safe RL, including SO2 (Zhang et al. 2024) and PEX (Zhang et al. 2023a),, and a Warm Start approach as baselines. SO2 improves Q-value estimation through Perturbed Value Updates, JSRI, and PEX utilize offline pretrained policies for exploration, and Warm Start directly finetunes the policy and Q-networks from offiine safe RL, without modifications. We compare with online learning from scratch, namely From Scratch, i.e., SAC-lag, which can achieve strong performance but require large number of online interactions. For completeness, we also evaluate the performance of Cal-QL (Nakamoto et al.) 2023) and SAC-Lag with an offline replay buffer across four representative environments: BallCirele, BallRun, CarCircle, and CarRun. The corresponding results are presented in Fig. G Given the interaction and network update settings in our experiments, we compare our implementation of SAC-lag with the standard FSRL library (Liu et al. 2024 in Appendix D.5 and the results show that their performance is highly comparable.
More importantly, aPID is used to update Lagrange multipliers in all these bascline methods (instead of truditional dual ascent), which in fact already improves the performance of these methods compared to their original designs. These baselines provide a meaningful comparison to demonstrate the effectiveness of Marvel, but also indicate the need of addressing the two challenges simultaneously for O20 safe RL. We provide the detailed descriptions of cach bascline algorithm in Appendix C Additionally, we provide a comparison with CDT (Liu el al. 2023c), using a smaller decision transformer (around 600k parameters) than the one in (Li et al. 2024). in Appendix D.1. and also demonstrate Marvel's superiority over baselines with large models.
Table 2: The "offline" column represents the performance of the offline safe RL pre-training, We present the performance at 20% of the total training steps during the finetuning process, as well as the performance after all the training steps are completed, e.g. "100 steps r/c' and "500 steps r/c' for reward and cost, respectively. Cost values greater than 20 are shown in gray. Bold numbers represent the highest reward while satisfying the cost threshold. The plots are in Appendix D
Environment
Marvel (ours)
From Scratch
Warm Start
SO2
JSRL
PEX
Offline
BallCirele
100 steps. t/e
580.3 603.9 19.8
19.2
170.8 88.9
54.7 0.0
54.4 / 0.0
137.5
0.0 119.7
168.0 / 11.0
500 stops r/c
241.7 / 10.9
176.6 18.5
58.4 / 2.0
1.6 165.4
-4.3 / 39.9
BallRun
steps t/c
276.0 / 5.8
1055.9 75.4
346.3 22.1
338.6 19.9
-120.5 / 37.2
270.3 42.9
262.0 / 3.0
500 steps: t/c
306.6 / 5.5
315.2 5.6
132.2 23.0
286.8 / 12.1
174.0 / 92.1
-555.6 / 88.8
CarCirele
100 steps: r/c
330.6
18.8
49.1 / 16.4
73.2 71.1
69.3 17.6
2.4 36.3
-0.5 / 69.4
265.0 / 14.0
500 steps r/c
341.3/ 19.5
115.2 / 12.7
141.6 / 24.1
115.9 / 9.1
1.1 / 130.0
-12.6 / 142.9
CarRun
100 steps r/c
537.9
17.00
337.1 / 74.3
383.9 /644
510.2 / 17.3
-186.4
140.8
-183.6 / 112.5
644.0 / 72.0
300 steps: t/c
3/1..5 18.2
293.5 / 7.2
391.4 / 16.9
522.2 / 16.2
126.6 / 128.2
153.0 / /8.8
AntCircle
800 steps: x/c
4.0 / 0.0
1.2 / 0.0
1.1 / 0.0 1.3 /
2.4 0.0 3.7 /
0.0 / 0.0
1.3 / 0.2
4.0 / 39.0
4000 stepa: xfc
5.6 /
1.5 / 0.0
0.6 / 1.9
/ 2.3
HalfCheotah
800 stops r/c steps:
1343.6 / 19.2
39.3 / 0.0
1605.9 / 180.5 974.1 15.4
1100.5 / 27.1
0.1 / 0.0
-70.6 0.0
113.0 / 17.0
1544.8 20.0
1074.3 28.8
559.4 / 23.9
27.6 /.8
-135.6 / 0.2
BallCirele (azAA)))))
100 steps r/e
621.1
19.2
170.8 / 88.9
174.5 / 101.9
-9.8 / 135.1
/ 150.2
24.5 / 129.4
394.0 / 19.0
300 steps: r/c
552.3 15.2
241.7 10.9
440.2 100.0
71.8 16.6
10.5 163.0
-13.6 5.8
Rum (KKAR.AN
steps: t/c
110.3
60.0
397.1 71.3
398.7 38.1
191.1 / 1.3
125.0 8/12
/ 15.2
270.0 / 17.0
300 steps: t/c
510.5
19.2
293.5 / 7.2
204.6 / 50.1
-59.6 0.0
31.9 113.6
200.6
34.0
9

---

Published in Transactions on Machine Learning Research (01/2026)
4.2 Main Results
As shown in Table 2 Marvel demonstrates better or comparable performance compared to all baselines consistently across all environments, i.c., achicving the higher return while koeping the cost below the threshold. In partieular, as shown in Fig. + Marvel demonstrates superior stabilily compared to other baseline algorithms. Note that key aspect of safe RL is ensuring safety, which requires keeping the cost below the predefined threshold while maximizing reward. Marvel achieves this balance effectively, as its cost remains below the threshold in all environments. In stark contrast, the naive warm start method proves largely ineffective, often causing performance drop or stagnation during training. Without aligning the Q- estimations, both JSRL and PEX struggles a lot to improve during online learning and fails to control the cost. While SO2 mitigates the inaccuracies of Q-estimations related to 020 RL, it does so only to a limitod extent and cannot maintain its performance consistently across different environments, even though aPID has already been used to boost its performance. On the other hand, the fact that SO2 performs better than other baselines further indicates 1) the great potentials of enabling fast and safe online learning through policy finetuning (compared to using the pretrained policy only as guide policy as in JSRL and PEX) and 2) the need of correcting pretrained Q-estimations before online finetuning.
More importantly, Marvel demonstrates the ability to find a good and safe online policy very quickly with minimal online interactions, as shown in Fig. and Fig. 5 (shown in Appendix D). For instance, in the BallCircle and CarCircle environments, near-optimal performance can be achieved with merely 20 steps of gradient updates. By leveraging offline policies and aligned Q-functions, Marvel rapidly explores high- reward regions while addressing the overly conservative nature of oflline pretrained policies. Although initial finetuning may cause a cost spike (Fig. ! aPID) effectively reduces costs by guiding the policy towards low- cost statc-actions in high-reward regions. Also, Marvel achieves the best max reward for a given cumulative cost, illustrated in Fig. 16 demonstrating superior performance while incurring fewer violations in the environment." " Notably, the same aPID parameters are used across all environments, showcasing its robustness and adaptability,
Compatibility of Marvel: In 020 safe RL, compatibility with different offline safe RL methods is es- sential. Given the non-interactive nature of offline training and the potential unavailability of algorithms due to privacy concerns, this compatibility becomes even more critical compared to online algorithms. Our design of Marvel naturally fits a variely of offline safe RL methods and only requires a pretrained pol- icy and Q-functions. To further verify this, the last two rows of table 2 show the training process using BEAR+ Lagrangian (BEAR-lag) (Kumar et al. 2019) in the offline phasc and SAC-lag in online finetuning. Note that BEAR-lag was not specilically designed for offline safe RL, but rather it incorporates the Lagrange multiplier into offline RL. In BallCircle, Marvel achieves the highest reward while satisfying cost constraints, and in the CarRun enviromment, it also outperformed others while maintaining cost below the threshold. This highlights the flexibility of our algorithm across different offline safe RL methods.
4.3 Ablation Studies
We conduct experiments in various setups to better understand Marvel. As shown in Fig. 3 the performance is best when both VPA and aPIl) are used. In contrast, if only VPA is used with traditional dual ascent during online finetuning, it significantly slows down safe online learning and takes a much longer time to reduce the cost. If only aPID is applied without VPA, the learning performance is very similar to naive policy finetuning, which struggles to improve due to the erroncous Q-estimations. We also evaluate the effectiveness of adaptive control in aPID, by comparing the performance between Marvel (VPA+aPID) and Marvel with aPID replaced by PID (VPA+PID). In Fig, 3 the training curve for cost exhibits significant fluctuations without using aPID. More critically, when the cost is close to the limit, PID cannot reduce its control strength. As a result, even if on average the cost of VPA-PID is close to the threshold, it is very frequent that the real-time cost exceeds the limit substantially, which is in fact not safe. aPID can adjust À more precisely, which directly affects how the policy balances reward and cost at every timestep. In contrast, a delayed or inaccurate adjustment of the Lagrange multiplier alters the relative weighting of reward and cost, thereby degrading reward performance, as observed in the CarCircle and HalfCheetah plots (Appendix D.2p
10

---

Published in Transactions on Machine Learning Research (01/2026)
BallCircle
CarRun
1000
175
600
80
VRK+RD
800
150
Reward
500
VRI
Reward
600
125
200
Cost
100
300
40
200
20
-200
-400
25_
0
100
200
300
400
500
100
206
300
400
500
100
200
300
400
500
100
200
300
400
500
Steps
Steps
Steps
Steps
CarCircle
HalfCheetah
350
2000
250
300
200
Reward
200
250
00
Reward
1000
500
Cosl
150
150
100
100
1000
0
100
200
300
400
500
100
200
300
400
500
1500
1000
2000
3000 Steps
4000
5000
1000
2000
3000
4000
5000
Steps
Steps
Steps
Figure 3: 'aPID' refers to adaptive PID applied only during online finetuning: 'VPA' refers to VPA is only used pre-finetuning. 'VPA+PID' uses standard PID and VPA, while 'VPA+ combines VPA with adaptive PID. VPA+aPID shows the best learning efficiency, stability, and convergence.
Table 3: Performance comparison between VPA+aPID (which is standard Marvel), VPA, and Warm Start when agent have different performance of offline policy. Note that although the second line shows Marvel exceeding the cost threshold, the violation is minor and can be corrected with just a few update steps.
Environment
VPA+aPID
VPA
Warm Start
Offline
100 steps: r/c
580.3 / 19.2
579.0 / 18.4
54.4 / 0.0
166.0 11.0
500 steps: r/c
610.1 / 19.6
511.3 / 16.9
176.6 / 18.5
BallCircle
100 steps: r/c
543.6 20.5
576.8 92.1
430.7 / 13.5
446.0 / 6.0
500 steps: r/c
605.4 20.9
400.1 24.3
269.7 17.4
100 steps: r/c
283.8
20.0
320.1 / 49.7
107.3 8.8
145.0 17.0
500 steps: T/c
327.2
19.5
306.3 / 21.9
176.4 20.8
CarCircle
100 steps: r/C
330.6
18.8
260.6 / 19.6
73.2
71.1
265.0 / 14.0
500 steps: r/c
341.3
19.5
283.1 / 17.2
141.6
24.1
In practical 020 safe RL training, the quality of offline datasets can vary significantly, making essential to evaluate whether Marvel maintains strong performance under different pretrained policies and Q-networks. As shown in Table 3 we train offline policies (and corresponding Q-networks) with varying quality lev- els. Marvel rapidly achieves near-optimal performance (with only few online interaction steps, shown in Figure 9 regardless of the quality of the offline dataset or the effectiveness of the pretrained policy. This demonstrates the robustness of Marvel to variations in offline data quality. Furthermore, the algorithm consistently enforces cost constraints while improving rewards, thereby accommodating pretrained policies and Q-networks with diverse performance levels.
Regarding the necessity of each component of Marvel, Table illustrates the importance of applying VPA both the Q and Qe networks initialized from offline pre-trained models, as well as the contribution of the entropy term. Finetuning the pretrained Q-networks also outperforms learning new Q-networks from scratch
11

---

Published in Transactions on Machine Learning Research (01/2026)
in VPA, which implies that the pretrained Q-networks, although not accurate, can still provide meaningful prior knowledge.
The training curves corresponding to Table 3 and Table 4 are presented in Appendix D.2 which further validating Marvel's robustness and design choices.: 1) Through extensive experiments across diverse envi- ronments, we confirm that the algorithm consistently maintains cost. constraints while improving rewards across offline datasets of varying quality, which consequently accommodates offline pretrained policies and Q-networks with different performance levels. 2) The training dynamics confirm the rationality of VPA's design, particularly the necessity of updating all Q-networks in conjunction with the entropy term. 3) Pa- rameter sensitivity analyses in Appendix D.3 and Appendix D.4 reveal acceptable performance variance despite the introduced hyper-paramnters. Furthermore, we verify correctness of our baseline implementation by demonstrating comparable performance to the standard library FSRL (Liu et al., 2024 under identical experimental settings, including interaction steps and policy update froquencies.
Offline
Qc
Entropy Pre-trained Q
Performance
603.9 / 19.8
511.3 / 16.9
166.0 / 11.0
V
364.7 11.2
64.1 5.3
1.2 51.0
Table 4: Ablation study of VPA on BallCircle. "Offline" shows the initial policy/Q values. "Entropy" indicates hee entropy regularization is used. "Q"/"Qe" columns denote the largets of VPA. "Pre- trained Q" shows if VPA starts from an offline Q network. "Performance" reports final reward and cost.
5 Conclusion
020 safe RL has great potential to put safe RL on the ground in real-world applications, by leveraging offline learning to facilitate fast online safe learning. In this paper, we proposed the first policy-finctunnng basod framework, namely Marvel, for 020 safe RL. In particular, by showing that naive finetuning would not work well, identified two unique challenges in 020 safe RL, i.e., the erroncous Q-estimations and Lagrangian mismatch. To address these challenges, Marvel consisted of two key designs: 1) value pre-alignment to correct the Q-estimations before online finetuning, and 2) adaptive PID control to dynamically change the control parameters so as to rapidly and appropriately control the cost. Extensive experiments demonstrate the superiority of Marvel over multiple basclincs. More importantly, Marvel compatible to a varicty of oflline and online safe RL approaches, making it very practically appealing. We hope our work bridges the gap between offline and online safe RL, distinct from unconstrained RL, and enhances the efficiency of online safe RL, paving the way for practical applications.
Acknowledgement
We thank the anonymous reviewers and the Action Editor for their constructive feodback and helpful sug- gestions, which greatly improved the quality and clarity of this paper.
References
Joshua Achiam, David Held, Aviv Tamar, and Picter Abbeel. Constrained policy optimization. In Interna- tional conference on machine learning, pp. 22 31. PMLR, 2017.
Alekh Agarwal, Nan Jiang, Sham Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Depl., UW Seatile, Sentlle, WA, USA, Tech. Rep, 32:96, 2019.
Fitan Altman. Constrained Markov decision processes. Routledge, 2021.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Aaron D Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control barrier functions: Theory and applications. In 2019 18th European control conference (ECC), pp. 3420 3431. IEEE, 2019.
Somil Bansal, Mo Chen, Sylvia Herbert, and Claire J Tomlin. Hamilton-jacobi reachability: A brief overview and recent advances. In 2017 IEEE 56th Annual Conference Decision and Control (CDC), pp. 2242 2253. IEEE, 2017.
Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-base based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5(1):411-444, 2022.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Ar- avind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084 15097, 2021.
Jason Choi, Fernando Castanoda, Claire J Tomlin, and Koushil Sreenath. Reinforcement learning for safety- critical control under model uncertainly, using control Iyapunov functions and control barrier functions. arXiv preprint ar arXi::2004.074 07084,, 2020.
Yinlam Chow, Mohammad Ghavanzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167):1-51, 2018a.
Yinlam Chow, Ofir Nachum, Edgar Duznan, and Mohammad Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. Advances in neural information processing systems, 31, 2018b.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052 2062. PMLR, 2019.
Javier Gareia, and Fernando Fernández. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437 1480, 2015.
Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline rl policies should be trained to be adaptive. In International Conference on Machine Learning, pp. 7513 7530. PMLR, 2022.
Sven Gronauer. Bullet-safoty-gym: framework for constrained reinforcement lcarning. 2022.
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330 2022.
Jiayi Guan, Guang Chen, Jiaming Ji, Long Yang, Zhijun Li, al. Voce: Variational optimization with conservative estimation for offline safe reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXie preprint ar 1ef:::812.059000 2018.
Botao Hao, Xiang Ji, Yaqi Duan, Hao Lu, Csaba Szepesvari, and Mengdi Wang. Bootstrapping fitted g-evaluation for off-policy inference. In International Conference on Machine Learning, pp. 4074 4084.
PMLR, 2021.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Jaaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: unified safe reinforcement learning benchmark. In Thérty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track,
2023. URL
13

---

Published in Transactions on Machine Learning Research (01/2026)
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey, IEFE Transactions on Intelligent Transportation Systems, 23(6):4909 4926, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off pooicy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Fung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pp. 6120 6130. PMLR, 2021.
Jongmin Lee, Cosmin Paduraru, Danicl J Mankowitz, Nicolas Heess, Doina Procup, Kco-Eung Kim, and Arthur Guez. Coptidice: Offine constrained reinforcement learning via stationary distribution correction estimation. arXiv preprint ar arxi::204. 08957,, 2022a.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Ofline-too-nlinee reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pp.
PMLR, 2022b.
Jinning Li, Xinyi Liu, Banghua Zhu, Jiantao Jiao, Masayoshi Tomizuka, Chen Tang, and Wei Zhan. Guided online distillation: Promoting safe reinforcement learning by offline demonstration. In 2024 IEEE Inter- national Conference on Robolics and Aulomalion (ICRA), pp. 7447-7454. IEEE, 2024.
Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-point policy optimization under constraints. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 4940-4941 2020.
Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Yihang Yao, Hanjiang Hu, and Ding Zhao. Towards robust and safe reinforcement learning with benign off-policy data. In International Conference on Machine Learning, pp. 21586 21610. PMLR, 2023a.
Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datascts and benchmarks for offiine safe reinforcement learning. arXiv preprint ar Xrx:222006.00 00000,00.093,,, 2023b.
Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. In International Conference on Machine Learn- ing, pp. 11611221630. PMLR, 2023c.
Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, and Ding Zhao. Datasets and benchmarks for offline safe reinforcement learning. Journal of Data-centric Machine Learning Research, 2024.
Yao Lu, Karol Hausman, Yevgen Chebotar, Mengyuan Yan, Eric Jang, Alexander Herzog, Ted Xiao, Alex Irpan, Mohi Khansari, Dmitry Kalashnikov, et al. Aw-opt: Learning robotic skills with imitation andrein- forcement at scale. In Conference Robot Learning, pp. 1078 1088. PMLR, 2022.
Qin-Wen Luo, Ming-Kun Kun Xic, Yewen Wang, and Sheng-Jun Huang. Optimistic critic reconstruction and constrained fine-tuning for general oflline-to-oline rl. Advances in Neural Information Processing Systems, 2024.
Ashvin Nair, Bob McGrew, Marcin Andrychowiez, Wojeiech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEFE international conference robotics and automation (ICRA), pp. 6292 6299. IEEE, 2018
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint "Xix:2006.09335 2020.
Mitsuhiko Nakamoto, Yucxiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsca Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pro-training for efficient online fine-tuning. 2023. URL
https://arxia wwt.v/orgggbbbaeeeee / 05479
14

---

Published in Transactions on Machine Learning Research (01/2026)
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Aduances in Newral Information Processing Systems, 36, 2024.
Adnan Qayyum, Junaid Qadir, Muhammad Bilal, and Ala. Al-Fuqaha. Secure and robust machine learning for healthcare: A survey. IEEE Reviews in Biomedical Engineering, 14:156-180, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 188):9, 2019.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and 2017.
demonstrations. arXiv preprint
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint (XX:::910001100 7(1):2, 2019.
Jie Ren, Peter Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. Advances in neural information processing systems, 32, 2019.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured pre- diction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Tim GJ Rudner, Cong Lu, Michael A Osborne, Yarin Gal, and Yee Teh. On pathologies in kl-regularized reinforcement learning from expert demonstrations. Advances in Neural Information Processing Systems, 14:28376-28389) 2021.
John Schulman. Trust region policy optimization. arXiv preprint ar orXic:1502.05/4 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint Xrv:: 1xXxx:107.000 707.06347, 2017.
Adam Stooke, Joshua Achiam, and Pieter Abbecl. Responsive safety in reinforcement learning by pid
lagrangian methods. In International Conference on Machine Learning, pp. 9133 9143. PMLR, 2020.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074 2018.
Tkechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. In International Conference on Machine Learning, pp. 34556-34583. PMLR, 2023.
Masatoshi Uchara, Chengehun Shi, and Nathan Kallus. A review of off-policy evaluation in reinforccment learning. arXio preprint 2r1::222 211:2222. 2022.
Kim P Wabersich, Andrew Taylor, Jason J Choi, Koushil Sreenath, Claire J Tomlin, Aaron D Ames, and Melanie N Zeilinger. Data-driven safety filters: Hamilton-jacobl reachability, control barrier functions, and predictive methods for uncertain systems. IEEE Control Systems Magazine, 43 2023.
Shenzhi Wang, Qisen Yang, Jiawei Gao, Matthieu Lin, Hao Chen, Liwei Wu, Ning Jia, Shiji Song, and Gao Huang. Train once, get a family: State-adaptive balances for offline-t-oollnee reinforcement learning. Advances in Newral Information Processing Systems, 36, 2024.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized lelannnnng for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8753-8760, 2022.
Long Yang, Jiaming Ji, Juntao Dai, Yu Zhang, Pengfei Li, and Gang Pan. Cup: A conservative update policy algorithm for safe reinforcement learning. XiXiu preprint ar Xix::2200..0 07565, 2022.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. arXiv preprint 0rXi::2000.0312,, 2020.
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey.
ACM Computing Surveys (CSUR), 55(1):1-33 2021.
Dongjie Yu, Haitong Ma, Shengbo Li, and Jianyu Chen. Reachability constrained reinforcement learning. In International conference on machine learning, pp. 25636 25655. PMLR, 2022.
Zhaocong Yuan, Adam W Hall, Siqi Zhou, Lukas Brunke, Melissa Greeff, Jacopo Panerati, and Angela P Schoellig. Safe-control-gym: A unified benchmark suite for safe learning- based control and reinforcement learning in robotics. IEEE Robolics and Aulomation Lellers, 7(4):11142 12111999 2022.
Sheng Yue, Xingyuan Hua, Ju Ren, Sen Lin, Junshan Zhang, and Yaoxue Zhang. Ollic: Imitation learning from oflline pretraining to online finetuning. arXiv preprinl 2xxi:2405.177777 2024.
Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging ofline-to-online reinforcement learning. arXiv preprint 2r 00935, 2023a.
Linrui Zhang, Li Shen, Long Yang, Shixiang Chen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Penalized proximal policy optimization for safe reinforcement learning. arXiv preprint Xii::2205111814 2022.
Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 33:15338-15349, 2020.
Yinmin Zhang, Jie Liu, Chuming Li, Yazhe Niu, Yaodong Yang, Yu Liu, and Wanli Ouyang. A perspective of q-value estimation offine-to-online reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16908- 16916, 2024.
Ziqi Zhang, Xiao Xiong, Zileng Zhuang, Jinxin Liu, and Donglin Wang. Sera: Sample efficient reward reinforcement learning. arXiv preprint arXiv:2310. 19805, 2023b.
augmentation in
Yinan Zheng, Jianxiong Li, Dongjee Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with foasibility-uidded diffusion model. arXiv preprint (01.:22/1.100000, 2024.
Zhehua Zhou, Ozgur S Oguz, Marion Leibold, and Martin Buss. Learning low-dimensional representation of a safe region for safe reinforcement learning on dynamical systems. IEEE Transactions on Neural Networks and Learning Systems, 34(5):2513 2527, 2021.
Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforcement learning fine-tuning need not retain offline data, 2024. ICLR 2025.
16

---

Published in Transactions on Machine Learning Research (01/2026)
A Overview of Algorithm
Algorithm Marvel
1: Input: Offline dataset Doss, environment &; offline RL loss functions {Lofs Quoe Loff h and on-
line RL loss functions (Lon, Lcse [.0):: offline learning rates (no off "QQ. 11/ no online learning rates (1Q 'ie ) PID hyper-parameters and initial controller state (Ao, To, Dn).
2: % These loss functions define generic aetor-critte safe RL algorithm: Los and L are the objective functions for learning the reward critic Q and the cost critic Qe, respectively; L is the policy optimiza- tion objective. Marvel treats them as modular components, allowing any safe RL algorithm to be plugged into the framework.
% In our implementation, we instantiate the offline losses using CPQ (Xu al,, 2022) and the on- line losses using SAC-lag (Ray et al. 2019), but Marvel is compatible with any actor critic safe RL formulation.
4: Output: Trained policy parameters 0, critic parameters o, de, online replay buffer Don, and final La- grange multiplier X.
5: Initialize network parameters o, de, 0; initialize Don + 0 set (A,1, + (No. Lu Do).
while in offline training phase do Sample miníbatch (s, of V. LoL
r,r, a,r,c, Doff
Qg
8: + 6-11 11Q
de - de - NQ.
off
V.. LoLLff
Quee
10: 0 - 8 - n V Lass
11: end while
12: % VPA with offiine dolasel
13: while in VPA phase do
14: for cach VPA step do
r,,,,,,))) Doss 16: Update Q by Eq. S update Qu by Eq. 9
15: Sample transitions
17: end for
18:
end while
19: while in online training phase do
20: Roll out one episode in & with policy тe, collect transitions (s;, ai, Ti, Cts 8841) and append to Don
21: Compute total episode cost e, and update EMA-smoothed cost é
22:
for each update step do Sample minibatch (s, a, c, 8') Don
23:
24:
Φ - ② - 1Q VoLon
Q=oe
25:
Φe de 12 V
Vo. Lon
26:
0 - 0 - T V Lg
27:
Update A by Eq. 12 with input &
28:
Update PID parameters by Eq. 15
29:
end for end while
30:
B Related Work
Online Safe RL. Online safe RL approaches can be generally divided into several categorics. The first calegory includes primal-dual based methods, such as PDO (Chow el al. 2018a), which combines PPO (Schulman et al. 2017) with the Lagrange multiplier method to obtain a policy that satisfies safety con- straints. CPPO-PID (Stooke et al., 2020) combines PID control with Lagrangian methods dampon cost oscillations. Similar Lagrangian-bar based methods are applied in conjunction with other unconstrained safe RL algorithms, such as TRPO-lag, PPO-lag, and SAC-lag. CPO AAchiam al. 2017 inherits from TRPO
17

---

Published in Transactions on Machine Learning Research (01/2026)
(Schulman 2015 : optimizing with the Lagrange multiplier method within the trust region. CUP Yang
et al. 2022) extends CPO by incorporating the generalized advantage estimator. In comparison, RCPO (Tessler et al. 2018) uses different update rates for the primal and dual variables. Two-stage iterative methods have also been developed for online safe RL, c.g., PCPO (Yang et al. 2020) and FOCOPS (Zhang
et al. 2020). Besides the primal-dual based methods, primal methods, which are also known as Lyapunov
methods, have been leveragod some studies for online safe RL. For instance, IPO (Liu et al.] 2020) uses logarithmic barrier functions. P30 (Zhang et al., 2022) employs an exact penalty function to derive an equivalent unconstrained objective and restrict policy updates within the trust region. (Chow et al., 2018b) leverages Lyapunov functions to handle constraints, which contains two parts, safe policy iteration and safe value iteration. Additionally, some studies (Wabersich et al., 2023 Choi et al. 2020) borrow techniques from the control theory, such as HJ reachability (Bansal el. al. 2017 Yu el. al. 2022 and control barrier functions [Ames et al. 2019), to ensure state-wise zero costs.
Offline Safe RL. Offiine safe RL soeks to learn a safe policy from static datasets without online environ- mental interactions. Similar to online safe RL,, Lagrangian methods can still be applied here, by adapting offline unconstrained RL algorithms like BCQ (Fujimoto et al.) 2019) and BEAR (Kunar et al. 2019) to the
safe RL sctting. CPQ (Xu et al.) 2022) uses a VAE to detect OOD Renn et al.) 2019) actions and penalizes them in terms of cost. COptiDICE [Lee et al. 2022a) extends OptiDICE (Lee et al. 2021) by adding safety constraints and derives a safe policy through the stationary distribution of the optimal policy. FISOR (Zheng et al. 2024) decouples the process of satisfying safety constraints from maximizing rewards and employs a diffusion model as the policy. VOCE (Guan et al., 2024) estimates Q-values of both cost and reward in a pessimistic way, mitigating extrapolation errors caused by OOD actions. Decision transformer (DT) Chen
ct al. 2021 has also been applied to safe RI., leading to constrained decision transformer (Liu et al. 2023c
020 Unconstrained RL. 020 RL has recently attracted much attention in the unconstrained case, where a policy pretrained on an offline dataset is used to assist online policy learning, e.g., through finetuning or serving as a guide policy. Morc specifically, (Hester et al., 2018 Nair et al., 2018 Rajeswaran ct al.) 2017) and (Rudner et al., 2021) explore various combinations of oflline demonst ration data with online learning. The core idea is that pure offline RL often struggles with limited performance due to heavy reliance on dataset quality. However, if interaction with the environment is allowed, pffline pretrained policy can be finetuned for improved performance. However, naive implementation of this process often leads to suboptimal performance (Nair et al. 2020 Uchendu et al. 2023) AWAC (Nair et al. 2020) prioritizes actions with high advantage estimates, while AW-Opt (Lu et al. 2022) builds on AWAC by applying positive sample filtering and using hybrid actor-critic exploration during online finetuning. (Lee et al. 2022b) finctunes the
pretrained policy by balancing the offline and online datasets. FamO20 (Wang el al. 2024) trains a family of policies using a universal model and then employs a balance model to select the most suitable policy for each state. Cal-QL (Nakamoto al.) 2024) constrains the updates the Q-network during online finetuning to prevent underestimation of the Q-values. SO2 (Zhang et al. 2024) improves Q-value estimation by updating Q-values more frequently and using noise-angmented actions. Instead of directly finetuning the pretrained policy, Jump-start RL (Uchendu et al. 2023) and PEX (Zhang et al. 2023a) follows another direction to leverage the offline policy, by using it to guide the update of the online policy during online learning. Zhouu et al.] (2024) propose to finetune an offline policy without retaining the offline data. Their method first warms up training by interacting with the environment using the offline policy before performing any policy updates.
C Details on Baselines
Considering the characteristics of safe RL, which requires keeping the cost below certain threshold, not all 020 unconstrained RL algorithms are suitable for 020 safe RL. For instance, AWAC (Nair et al., 2020). ; which maximizes the advantage function, has not yet been applied in the safe RL context. We compare Marvel with the following baselines:
SO2 Zhang et al. 2024). By analyzing Q-value estimation in offline to online transitions, the SO2 algorithm achieves more accurate Q- Qavalue estimation through Perturbed Value Update and by increasing the frequency of Q-value updates.
18

---

Published in Transactions on Machine Learning Research (01/2026)
JSRL (Uchendu et al. 2023
JSRL employs offiine pretrained policy the exploration policy and policy under training during the online phase as the target policy. Initially, the exploration policy is used, followed by the target policy during online interaction to facilitate curriculum learning. To adapt to the safe RL setting, we update the Lagrange multipliers using the aPID method when updating the target policy.
PEX (Zhang al., 2023a). Similar to JSRI,, PEX uses an offline pretrained policy and a policy under training during the online phase for online interaction. However, PEX selects one of the actions based on the Q-networks's value estimation of actions chosen by the two policies. To meet the safe RL requirements concerning cost, like the modifications to JSRL, use the aPID method to update the Lagrange multipliers.
Cal-QL (Nakamoto et al.) 2023 This method fine-tunes policies using a better-calibrated Q-function. We employ the aPID method to updale the Lagrange multipliers.
SAC with an offline replay buffer. This baseline naively incorporates offline data into the online replay buffer. As with our modifications before, we use the aPID method to update the Lagrange multipliers.
Warm Start. We directly utilize the policy, Q-network, and Qe-network networks obtained from offline safe RL without any modifications (no VPA and aPID), and apply online safe RL algorithms for finetuning,
We selected SO2, JSRL, PEX and Warm Start as baselines because they represent prominent methods in 020 RI, and adapting them to the safe RL context provides a meaningful comparison. Including these baselines allows us to demonstrate the effectiveness Marvel in a fair and relevant context.
D More Experimental Results
D.1 More experiments
In Fig. we provide training curve of Marvel and bascline algorithms in BallRun, BallCircle, CarRun, CarCirele, HallCheetah, AniCircle. In environments like BallCircle and CarCirele, Marvel finds a good policy within less than 15 steps, dominating baseline methods in both performance and speed. Note that all approaches indeed start from the offline policy and Q-functions, i.c., the same point at step 0 (ignored in all figures).
In Fig. 5 we additionally provide experimental results in more environments, including DroneCircle, AntRun, Hopper, and Swimmer. The results indicate that our proposed Marvel algorithm achieves competitive performance across these settings.
In Fig. 6 Cal-QL, which fine-tunes policies using a better-calibrated Q-function, demonstrates reasonable performance. However, since it was not originally designed for safe reinforcement learning, its direct appli- cation in this setting is inappropriate, since it fails to fully address the issues of erroneous Q-estimations and Lagrangian mismatch, as illustrated before. Similarly, the naive use of offline data, SAC with an offline replay buffer, fails to achieve satisfactory outcomes, as its incurred cost substantially exceeds the predefined threshold. This violation of safely constraints highlights the algorithm's inability to succeed under such conditions.
In Table 5 We conpare our Marvel with CDT et al., 2023c). Due to CDT's design for offline settings, we use the replay buffer to apply CDT in the online setting, which represented as "CDT-fimettne" in the table. It can be seen that CDT, having more parameters, generally performs better than CPQ in the offline phase. However, since the finetuning process was not specifically designed, performance stagnation and even degradation occur during finetuning.
D.2
More ablations
D.2.1
VPA on Q vs Qc vs Both and if VPA need entropy term
Fig. 7 presents more detailed ablation experiments, including whether VPA ncods to be applied both the Q-network and Qc-network, as well as whether the entropy term should be added VPA. By comparing VPA(Q), VPA(Qc), and VPA(Q+Qe)) we can observe that applying VPA solely to the Qe-network results
19

---

Published in Transactions on Machine Learning Research (01/2026)
BallCircle
BallRun
2200
150
Rowera
Dewardd
Cost
200
50
20-
-2500
250
400
500
200
402
500
Sieps
Stegs
Steps
Steps
CarCircle
CarRun
500
375
200
Rewerd
200
Roward
200
ACD
200
400
25
-500
300
300
Steps
AntCircle
HalfCheetah
30
230
230
1.000
230
Bewarrs
Dewward
1:000
130
A00
-500
1000
2013
2000
4033
100.5
/002
2000
40020
1000
2000 Steps
2000
4000
0
1000
2020
3000
4000
Steps
From Scratch
S02
PEX
Warm Start
JSRL
Marvel (ours)
Figure Performance comparison between Marvel and bascline methods in multiple environments. It is clear that Marvel can quickly find high-return policy while keeping the cost below the limit.
DroneCircle
220
200
Reward
250
Reward
120
200
2000
2000
4000
5050
1000
2002
3020 Stops
4000
$200
000
1200
Stops
Hopper
Swimmer
300
3000
Jen
Cost
150
400
100
-20
-30
2005 3000
Stops
4200
2006
Stops
3500
1000
5006
Steos
Steps
From Scratch
SO2
PEX
Warm Start
JSRL
Marvel (ours)
Figure 5: We provide experiments more environments.
in very poor performance during online finetuning. For example, in the BallCircle environment, results similar to naive finetuning shown in Fig. [ were observed. On the other hand, applying VPA only to the Q-network leads to significant instability during finetuning (e.g., large error bands in BallCircle, CarCircle,
20

---

Published in Transactions on Machine Learning Research (01/2026)
BaliCircle
BallRun
20
Rowerd
Boward
600
600
400
20
200
Steps
Stegp
Steps
Steps
CarCircle
CarRun
500
175
190
M w
m
Reward
Reward
11/1
600
600
50
200
500
203
500
100
30C
500
100
100
500
Marvel (ours)
Cal-QL
SAC-lag offline buffer
Figure 6: This figure presents performance comparison between Marvel, Cal-QL, and SAC with an offline replay buffer. As shown, Marvel achieves the best overall performance.
Table 5: Comparison of Marvel and CDT
Offline
Online
Environment
CPQ
CDT
Marvel (ours)
CDT-finetune
BallCircle
166.0
11.0
521.7 / 18.3
603.9 / 19.8
468.7
9.3
BallRun
262.0 / 3.0
448.5 / 48.5
306.6 5.5
449.0
50.3
CarCircle
265.0 14.0
328.7 / 18.1
341.3 / 19.5
320.2
23.1
CarRun
544.0
72.0
553.9 / 25.1
547.5 / 18.2
542.0
23.3
AntCircle
4.0
39.0
221.6
/ 54.7
5.6 / 0.9
161.2
36.4
HalfCheetah
113.0 / 17.0
1358.0 / 24.3
1544.8
20.0
1536.7 / 3.9
and HallCheetah) and poor performance in terms cost (e.g., the cost curve CarRun shows a sharp increase beyond the cost threshold). This occurs because if only the reward is optimistically estimated while the cost is pessimistically ovetestimated, it causes the agent to neglect the cost during exploration, adversely affecting finetuning performance. The experiments demonstrate that applying VPA both the Q-network and Qc-network simultancously has the best results, which aligns with the motivation discussed in Section 3 Comparing VPA(Q+Qe) with VPA(Q+Qe) with entropy, it is evident that optimistically estimating both reward and cost, while aligning with the pretrained policy, proves to be effective.
D.2.2 Finetune Q-networks vs train new Q-networks in VPA
In Marvel, VPA finetunes the offline pretrained Q-networks. Fig. illustrates the training curves when, instcad of finetuning the pretrained Q-networks, the Q-networks are retrained from scratch during the VPA phase and subsequently finetuned online. As shown, finetuning the pretrained Q-nelworks achieves better performance. This is because, although the pretrained Q functions may be inaccurate, they still provide meaningful prior knowledge from the offline dataset and serve as a valuable starting point for Q function finetuning. This would generally speed up the learning and lead to a better local optima compared learning from scratch based the offline data from random initial point. Moreover, considering the limited number
21

---

Published in Transactions on Machine Learning Research (01/2026)
BalliCrrle
CarRun
500
220
602
640
Reward
500
60
Reward
420
110
Cost
200
-200
visib
20
-40
100
100
303
430
920
100
200
300
405
520
202
232
400
300
200
320
420
Steps
Steps
Stegs
Steps
CarCircle
HalfCheetah
200
2020
300
250
1530
123
Reward
110
73
-50
25
50
Staps
Stops
Stops
Stops
Figure 7: In the figure, VPA(Q), VPA(Qc), and VPA(Q+Qcc) represent applying VPA the Q-network, the Qe-network, and both simultaneously, without using the entropy term. This corresponds to setting and de to 0 in Eq. and Eq. g Conversely, VPA(Q+Qe) with entropy indicates that the entropy term is used in VPA, meaning 0% and a are non-zero. In all experiments represented by the curves, we employed aPID.
of steps allowed in VPA for efficiency, directly learning completely forgoes the knowledge learned offline and can fail to find good Q estimations.
BallCircle Rewards
BallCircle Costs
700
175
000
150
500
125
Reward
Cost
100
AM
200
100
1
VPA(tinetune)
VPAffrom scratch)
100
300
Steps
Steps
Figure 8: In the figure, "VPA (finetune)' refers to finetuning the oflline pretrained Q-net during the VPA phase, while "VPA (from scratch) refers to training new Q-networks from scratch during the VPA phase.
D.2.3 Marvel's Ability to Improve Pretrained Policies at Different Performance Levels
regardless of the quality of the offline dataset or the performance of the pretrained policy, Marvel is able to quickly achieve optimal performance with only few online interaction steps. This highlights the robustness of the Marvel algorithm to variations in the quality of the oflline dataset.
As shown in Fig. g
D.3 Parameter sensitivity of aPID
D.3.1
PID
The SAC-lag algorithm in (Luu et al.] 2024) utilizes PID control, with PID parameters carefully optimized. However, if their provided parameters used directly under the environmental settings, policy updates,
22

---

Published in Transactions on Machine Learning Research (01/2026)
BallCirrle-llow
BallCircle-high
Warn Stat. una
700
500
Start
Cost
Beward
#20
Cost
800
20
200
300
420
203
3003
200
200
400
320
200
420
310
Steps
Steps
Steps
Steps
CarCirclellw
CarCircle-high
320
300
125
73
50
50
25
22-
Stops
Stops
Stops
Steps
Figure 9: In the figure, "high' and "low" represent the different performance levels of offiine pretrained policies resulting from varying quality in the offline dataset. These policies are then fineluned online. The results demonstrate that the Marvel algorithm is robust to both different offline dataset qualities and pretrained policy performances.
and Q-network update configurations of this paper, the performance is suboptimal. Fig. 10 presents a comparison, showing that when the PID parameters from the FSRL library are applied, the performance of online finetuning is significantly degraded. indeed significantly outperforms the implementation of PID provided by FSRL. More importantly, even with inappropriate PID parameters, aPID effectively boosts performance, achieving higher rewards while maintaining more stable cost levels.
is clear that the implementation of PID in our paper
D.3.2 Parameters in aPID
a, 8, and 7 are the parameters used in aPID adjust the PID parameters. These parameters enhance the robustness of the initial settings for the PID parameters while being inherently robust themselves. Although our method aPID introduces more parameters, this is very common for adaptive algorithms in order to control the adapiation during the learning procedure. Fig. 11 illustrates the performance under various combinations of a, 8, and y, with values ranging from 0.01 to 0.5. All curves achieve similar performance in terms of reward and cost by the end training. This demonstrates that these paramoters are both casy to tune and robust. their selection.
D.4 Parameter sensitivity of VPA
The selection of a and a follows a similar approach to the selection a in SAC. These values need to be empirically determined based on the evaluation results the pretrained policy, the entropy of the policy, and the scale of the Q-values provided by the Q networks. parameters do not cause the entropy term to dominate the Q-value update process. The tuning process involves starting with small values, such as those in the range of 1 10-5, Considering that the entropy value is typically negative single-digit number, the upper limit for a and should generally be around x 10-1, For relatively conservative offline pretrained policies, larger values of a and ae may be more suitable.
crucial ensure that the values of these
To demonstrate the robustness of the chosen a and ae, we scaled the values provided in this paper by a factor of five, ranging from 1 x 1 to x 10-5, As shown in Fig. 12 the choice of different a and a values has minimal impact the final performance.
23

---

Published in Transactions on Machine Learning Research (01/2026)
BallCircle Rewards
BallCircle Costs
700
600
80
400
nm
Cost
Our PID
Our aFID
FSAL
FSRL aPID
Steps
Steps
CarRun Rewards
CarRun Costs
800
140
120
100
Reward
400
Cost
80
200
60
40-
20
-200
100
200
300
400
500
100
260
300
405
500
Steps
Steps
Figure 10: "Our PID' and "Our aPID' refer to using the PID and aPID parameters proposed in this paper for adjusting the Lagrange multipliers, respectively. Similarly, "FSRL PID' and "FSRL aPID' represent the parameters provided by the FSRL library for the same purpose.
D.5 Correctness of our implementation of SAC-lag
The primary goal of 020 safe RL algorithms is to achieve competitive performance with minimal en- vironment interactions and in the shortest time by leveraging oflline information to accelerate online learning. In contrast, the algorithm (Liu et al., 2024 (and other similar online algorithms) achieves higher performance but relies on significantly more interactions. For example, the BallCircle environment, (Liu et al.) 2024) utilized 1.5 million environment interactions, whereas our method required only 120,000 interactions (with average of 600 interactions per gradient update). This significant reduction highlights the efficiency of our approach, particularly in resource-constrained and safety-critical settings where the number of online interactions is strictly limited.
To validate the correctness of our implementation of SAC-lag, we conducted additional experiments compar- ing it to the SAC-lag implementation provided by the FSRL library under the same experimental settings (including both environment interaction steps and policy update frequencies). The results, presented Fig. 13 show that both implementations demonstrate similar performance in terms of reward and cost. This validates the correctness of our implementation and ensures its reliability as bascline for comparisons in our study.
D.6 Without VPA but with good initial values of the Lagrangian multipliers
As shown Fig. 14 when VPA is not used and the Lagrange multipliers are updated using dual ascent (as described in Eq. 3 even with appropriately chosen initial values for the Lagrange multipliers ("Warm Start
24

---

Published in Transactions on Machine Learning Research (01/2026)
BallCircle
700
80
600
60
Reward
M n
500
Cost
40
400
20
300
0
100
200
300
400
500
100
200
300
400
500
Steps
Steps
[0.05, 0.05, 0.05]
[0.01, 0.01, 0.01]
[0.1, 0.05, 0.05)
[0.5, 0.5, 0.5]
[0.01, 0.05, 0.05]
Figure 11: The three numbers within "[]" represent the three parameters for adaptively adjusting Kp, K and Kd, namely a, and n.
BallCircle
700
80
600
60
Reward
Cost
500
40
400
20
300
0
100
200
300
400
500
100
200
300
400
500
Steps
Steps
[1e-3, 5e-4]
[2e-4, 1e-4]
[5e-3, 1e-3]
Figure 12: The two numbers inside "[]' represent the values of and a. used in VPA, described in Eq.
and Eq. 9
w/ lag init"), the performance, while better than initializing with zero W"arm Start"), still falls short of achieving optimal results.
D.7 Additional Baseline: Distilling Q-values of the Behavior Policy
To further evaluate the effectiveness Marvel, we include an additional baseline that distills the Q-values of the policy which generated the offline datasct. In offline RI, the datasct is necessarily produced by a single policy; it can be mixture of trajectories collected from multiple behavior policies of varying quality. To align with such mixed dataset, first. use behavior cloning (BC) to obtain policy consistent with the
25

---

Published in Transactions on Machine Learning Research (01/2026)
BallCircle
350
300
120
250
100
Reward
200
Cost
80
WWw no Mi
150
60
100
40
50
20
Our SAC-lag
FSRL SAC-lag
0
0k
25k
50k
75k
100k
125k
0k
25k
50k
75k
100k
125k
Steps
Steps
Figure 13: "Our SAC-lag" refers the SAC-lag algorithm implemented in this paper, while "FSRL SAC- lag" represents the SAC-lag algorithm provided by the FSRL library. Using the same environment settings (including interaction steps) and update frequencies in this paper, the results from the FSRL library are shown to be similar to ours.
CarRun
1000
120
100
800
Reward
600
80
Cost
60
400
40
200
20
MY
0
100
200
300
400
500
100
200
300
400
500
Steps
Steps
Marvel
Warm Start
Warm Start w/ lag init
Figure 14: In the figure, "Marvel" and "Warm Start" follow the legend defined Fig. 1 "Warm Start w/ lag init" represents the approach of empirically selecting appropriate initial values for the Lagrange multipliers and performing online finetuning.
oflline data distribution. We then perform off-policy evaluation to distill the Q-values of this cloned policy and use the resulting critics to initialize online finetuning.
However, our experiments show that this approach fails to keep the cost below the safety threshold during online finetuning. Because the BC policy directly imitates actions from the oflline dataset, it inevitably learns unsafe action patterns present the data. When the distilled critics are subsequently used for online updates, these unsafe patterns remain and cannot be corrected even with the adaptive control provided by aPID. As result, the policy cannot maintain safety and exhibits performance significantly worse than MARVEL.
26

---

Published in Transactions on Machine Learning Research (01/2026)
BaliCircle
.75
720
.000
Reward
600
60
Reward
.25
020
600
400
20
320
200 350
400
500
150
20C
200 Stepp
420
503
200 320
400
2010 2013
Steps
Stops
Steps
Figure 15: Comparison between Marvel and the additional baseline that distills Q-values of the behavior policy. The BC-distilled baseline fails to control the cost below the threshold due to inherited unsafe patterns from the offline dataset, leading to high constraint violations and lower returns, whereas MARVEL effectively maintains safcty and achieves superior performance.
E
More Analysis of Marvel
Similar to the analysis presented in (Liu et al. 2023a). this section introduces alternative way to evaluate safe RL performance beyond training curves, as shown in Fig. 16 The cumulative cost represents the total cost accumulated from all environment interactions up to given timestep during training, while the max reward denotes the highest reward achieved up to that timestep. The relationship between these two metries reflects the algorithm's is ability to achieve maximum reward performance under a certain amount of cost incurred in the environment.
The figure shows that Marvel achieves the best max reward for given cumulative cost. Moreover, when targeting specific performance level (i.e., reward), Marvel requires the least cunulative cost. This further highlights Marvel's superior performance from another perspective.
HalfCheetah
BallCircle
CarRun
1500
MBD
Nax (Meourr
Mox Rewerri
400
200
500
200
Cumubative Cost
Cumuletive Cost
Cumulative Cost
From Scratch
S02
PEX
Warm Start
JSRL
Marvel (ours)
Figure 16: The legends in this figure follow the conventions of this paper and illustrate the relationship betweon cumulative cost and maximum reward.
F
Additional PID Design Details
For completeness we provide additional details on the PID controller usod in Marvel and clarify how cach component corresponds to our implementation. We employ discrete, position-forn PID update. Let e denote the total episode cost at training iteration t and ct the cost limit. The instantaneous violation is
e, = Ct Cth.
EMA-smoothed proportional signal. exponentially smoothed violation:
To reduce variance, the proportional channel operates on an
é = a + ap)e,.
where ap [0, 1) is the smoothing factor.
27

---

Published in Transactions on Machine Learning Research (01/2026)
EMA-smoothed cost for the derivative channel. We first compute smoothed version of the episode cost,
= a= + + (1-aa)e
with smoothing factor aa (0, This low-pass filtering mitigates the high variance of raw episode costs.
Lagged derivative approximation. Instead of estimating the derivative by immediate finite differences, we compute lagged derivative using a delay window of length d:
D. max (0, 6 --d).
This corresponds exactly to our implementation, where we maintain a buffer the past smoothed costs and take their difference. Such a delay-basod derivative approximation common in discrete PID controllers, it substantially reduces sensitivity to noise while retaining responsiveness sustained increases in cost.
Anti-windup mechanism. The integral state is updated as
I max (0. I,-1 + K,et);
which prevents the integral term from accumulating negative values and ensures stability. This clipping-basel anti-windup mechanism the same as the implementation.
Final position-form update. The Lagrange multiplier used by the policy optimization step is computed
\ = Kpē I + KaD;).
Thus the proportional lerm uses the smoothed violation εt, the integral term accumulates the raw violation et; and the derivative term is lagged difference of smoothed costs. This matches the code path in our implementation and avoids introducing second integrator.
These design choices (EMA smoothing, lagged derivative, and anti-windup) follow standard practice in discrete PID control and were found to improve robustness and stability in our reinforcement learning setting.
G Experimental Details
We conduct all experiments in single Nvidia 4080 GPU,
G.1 Spearman's rank correlation coefficients
We aim to explore the effect of VPA on the distribution of the Q and Qc networks. Specifically, for different stale-action pairs, we need to analyze the true Q and Qe values versus the predicted values from the Q and Qe networks. To achieve this, we choose to use Spearman's rank correlation coefficiett, which allows us quantify the ranking accuracy of the and Qc values over sequence of state-action pairs.
Spearman's 's rank correlation coefficient, denoted P, is a non-parametric measure of the strength and direction of the association between two ranked variables. It evaluates how well the relationship between two variables can be described using monotonic function, rather than assuming a linear relationship. This makes it particularly useful in our case, where we are more concerned with the rank ordering predicted versus true Q and Qc values rather than their exact numerical differences.
Mathematically, Spearman's rank correlation coefficient given by:
d p 1 n(n² -1)
(16)
where d, is the difference between the ranks of the corresponding values of the two variables (in this case, the true and predicted Q Qe values) for each state-action pair. the number of state-action pairs.
28

---

Published in Transactions on Machine Learning Research (01/2026)
Spearman's coefficient ranges from -1 to 1, where p 1 indicates perfect positive rank correlation (i.e., the predicted Q and Qe values perfectly match the rank of the true values) p -1 indicates perfect negative rank correlation, p = 0 indicates no correlation between the ranks of the predicted and true values.
By applying this measure, we can rigorously assess how well the Q and Qe networks preserve the relative rankings the true values across various state-action pairs, thus quantifying the alignment between the predicted and true distributions.
G.2 Estimation of Q-values and Qc-values through Monte Carlo Simulations in Table
The valuus represent the expected cumulative reward from a given state when following a specific policy, while the Qe-values represent the expected cumulative cost, These values are estimated through Monte Carlo (MC) simulations, making them accurate because the simulations explicitly capture the sequential interactions of the agent with the environment under the given policy.
For the MC simulations, we use the pre-trained policy derived from the training phase. Each simulation starts from a selected initial state. The number interaction steps with the environment depends on the specific settings of the environment. For instance, in the BallCircle environment, the maximum number of steps is 200. total of 10 Monte Carlo simulations are performed, and at each timestep, we record both the reward and the cost. To compute the true Q-values and Qc-values, the recorded rewards and costs are averaged cumulatively across all steps in the episodes.
Regarding the choice of the initial state, the term "dataset" refers to selecting the initial state from the offline dataset used during VPA, whereas "random" indicates that the initial state is chosen randomly. This approach ensures a diverse evaluation and enhances the robustness the estimated values.
G.3 Experimental Setup
In Table B we present the specific ppper-parrmeters used in the experiments. Table 7 lists the configurations of the environments used the experiments.
H Proof of Theorem
We first define a modified VPA Bellman operator with entropy regularization as follows:
VVPAQ(s, = r(s, a) + "B [Q(s', (0sss'''') a') a <(a'(s))
(17)
d'ma(116")
where a denotes the entropy regularization term a VPA or a VPA depending on the context. We use a for simplicity in following proof. We can have the following result to characterize the contraction property of TVPA, To is the policy obtained in the offline phase which the fixed policy in VPA and also the initial policy of online finetune.
Considering the similarity in the derivations for Q and Qc we will proceed with the derivation for Q, and the derivation for the cost Q-function Qe follows in the same manner. Assumption (Concentrability). 3C < co such that
do (s, SC
max (x,a)ESxA du (s,a)
where (s, denotes the distribution induced by the behavior policy μ that generates the offline dataset. For notational simplicity, refer to the behavior policy's 's slale-action distribution as μ, i.e., d*, p.
distribution induced by the offline policy To, and d' a) is the
Theorem 2 (Contraction Property). The VPA Bellman operator TVPA is n-contraction under the L2 norm weighted by the state-action distribution d
| VPA Q2 22112,0m S <1ll11 Q2 Q2122.00.
(18)
29

---

Published in Transactions on Machine Learning Research (01/2026)
Hyper-parameter
Value
Policy Learning Rate
5e-5
Q-network Learning Rate
3e-5
Qc-network Learning Rate
8e-5
Lagrangian Learning Rate
lc-4
SAC-lag:
5e-3
VPA Entropy Coefficient : a
le-3
VPA Entropy Coefficient ae
5e-4
aPID: Kp
lc-4
aPID: Ki
le-5
aPID: Kd
1e-5
aPID:
0.05
aPID:
0.05
aPID: ?
0.05
Batch Size
256
MLP hidden layer size
[256, 256]
discount
0.99
5e-2
replay buffer size
1e6
Table 6: Experiment hyper-parameters
Environment
Episode length
Cost threshold
BallCircle
200
20
BallRun
100
20
CarCircle
200
20
CarRun
200
20
AntCCrcle
500
20
AntRun
200
20
DroneCircle
200
20
HalfCheetah
1000
20
Hopper
1000
20
Swimmer
1000
20
Table 7: Enviroument setup
Here, E 10 is the discount factor, and
|| I|a,a0o is defined as:
[| (Bie.))aaTo [f(s (1)
30

---

Published in Transactions on Machine Learning Research (01/2026)
Proof. Let A Q1 Q2-
Note that the entropy terms cancel when taking the difference between TVPA Q1
and TVPA Q2, we have:
IT Q1 TVVA 221..22
= (ssa)dd
[(7
(10))
S 7 vdtto a.0) (() [d]]]
(by Jensen's inequality)
72 Bie ( (A)s, a
= 11
Taking the square root of both sides gives:
ITVPAQ1 TVPAQ2ll2.s S111Q1 Q2112,000
The next result will show that the Q-value function is bounded under the conditions considered in this paper. Lemma (Bounded Value Function). Under the conditions that the marimum reward max |r| < and (ax |log (a(as)) B, the value function of VPA satisfies:
1 1 aB
(s, S 1-7
(19)
Proof. We start from the entropy-augmenteed Bellman equation:
Q (s, a(s,a) + (( 1Q" d'ma)(-)d")
a log (ala'ss')]
Taking absolute values and applying triangle inequality:
(s) S E
y' (Ir(s;, algg
7 + aB)
1=0
1 aB
7
As a direct consequence of the pointwise boundedness of QTa, we can also derive an upper bound on its Le norm under the state-action distribution d Recall that the bound Lemma [ is given in terms of the absolute value, i.e., 1Q (s, a)] S 1+a8 for all (s,a) S A. Since the norm is defined the supremum of the absolute value over the domain, we have
I-y
|| sup (s (s, 1+aB (s,a)
1-7
Moreover, for any measurable function f and any distribution d, holds that ||/ S || Therefore, we obtain:
1 + 1
IlQ 122,70 & 1100 -
(20)
1 1-7
This result provides a useful uniform bound in the L2 sense, which will be instrumental in the subsequent generalization and convergence analysis.
31

---

Published in Transactions on Machine Learning Research (01/2026)
H.1 Final Convergence Bound for VPA
are now ready to quantify how the Q function produced by K steps of VPA evaluation deviates from the fixed fixed-point solution under the offiine policy To.
We begin by briefly reviewing Theorem I As the number of iterations K increases, the estimated Q-function QK progressively moves away from the initaaliztooon Qo, and converges toward the fixed-point solution Q under the offline policy Tg- The closeness between the two is measured in the weighted Lz-norm || 1 which denotes the expected squared error with respect the state-action distribution induced by policy mo.
(a,a) S C and Lemma 1 after K iterations, with probability at least 1- Based on (a, (=(o.)
vC² ||Qk - 12,,000 S 1-7 + +,* 1*11100
t 112,000.
(21)
where ē 22(1 + ing( (F188)
+2 220d VPA 8 (0, 1) the confidence level parameter, F denotes the function
class used to approximate Q-values, and [D] is the number of samples the dataset. The term de VPA represents the inherent Bellman evaluation error under policy vo using the VPA operator. The parameter a corresponds to the entropy coefficient in the VPA Bellman operator, and B = max(s,a) |oogTo To s)] denotes the maximum policy entropy. The constant C comes from the concentrability assumption, and E (0, 1) is the standard discount factor in Markov decision processes.
Proof. We decompose the error at iteration & using triangle inequality:
||Qx 12,000 < Q1 V Qx- Qu-1lla.00 + ITT -
Approximation error
(II) Iteration
Term (I): Based on the concentrability,
(1) S -TPAA
At each iteration k; the VPA algorithm performs a supervised regression to update the Q-function estimate Qk. using regression targets constructed from the previous iterate Qx-1 under an VPA Bellman operator. Given the oflline dataset D, the Q-function is updated one step to minimize the following objective:
Qk + arg min QEF
Σ
1((s - yi|2
where denotes the function class used to approximate Q-functions, and the target value ye incorporates an entropy penalty defined as:
+ (Barmoot-le)) (Q (Q,=1(6,,a))) a log mo(alls()))
The training objective socks to minimize the squared deviation between the predicted Q-values and the soft Bellman target computed from Qu-1 at each iteration.
+ (see Lemma and the boundedness assumptions Iral s max log (a(a))) s B, we obtain:
To bound (gal, using ||Qx-1lloo
Mnl S |ra| 7 (11ox + aB)
1 + aB aB
$ 1 11
1-7
a
<2
2V.
1 1-y
32

---

Published in Transactions on Machine Learning Research (01/2026)
To control the one-step regression error, apply the least squares generalization bound (Lemma 11) from (Agarwal et al. 2019). which provides a high-probability upper bound on the expected squared error between the regression output Qx and the target computed from Qk-1-
Specifically, given that (1) the supervised regression target Yi is uniformly bounded, i... 1y S 2V (2) the function class used to fit Qs is finite or has bounded covering number (3) the sample inputs (8i, a:) are drawn i.d. from the offline dataset, we can get the following bound with probability least 6:
1|Qx QU- 2212 logl (o/(
20d7 VPA
[D]
VPA denotes the approximation error between the best function in F and the Bellman target under where of policy no-
Substituting
L+oB into the bound above: 1-7
22(1 + (B))
1|Qx - TVPA TVPAQx-1ll2. 1
+ 20d VPA Vł.
D - 7)
Thus, we can get the bound of approximation error (I):
(I) VoIQu - VPA
V Q0-1112 S Vo..
Term (II):Based on Theorem we have:
V 1 12.dr0
-TPA- 112 57114u-1 - 122
Combining both (I) and (II) term:
||Qx - 12 S +V 2T 112,a
Let Qo be the Q-function learned offline, which is the starting point for VPA. Unroll the recursion:
||Q - QT 12, S v 112,
S ( - Q 12,,00)))
- (/=0|1,,0
S +C
+
Q 2,d
K-1
$ VCE Σ
* * - - 12.
VCE
(1 11 + x1100 - 12,000
7
VCE
1- 7 "k $11Qo T 112,000
33

---

Published in Transactions on Machine Learning Research (01/2026)
Limitation
While Marvel performs well in most environments, it does not exhibit the same effectiveness in certain scenarios, such as in the AntRun environment. As depicted in Fig. 5 during finetuning, Marvel does not significantly improve cost and reward metrics. Consequently, aPID evidently does not function optimally in these settings. This suggests that further enhancements are needed for VPA to increase the agent's 's ex- ploratory behavior during online finetuning. Combined with aPID'ss ' efficient control over costs, this approach could achieve optimal performance with minimal interaction with the environment, thus minimizing the time required. In addition, it would be interesting to investigate strategies that explicitly re-initialize resct Ka when the environment undergoes new phases of non-stationarity.
34


Reviews:
[{"review_id": "ci9cW1uB1c", "reviewer": "Reviewer_5PFa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes offline-to-online (O2O) \"safe\" RL with Value Pre-Alignment to initialise the Q-functions ($Q$ and $Q_c$) w.r.t. the learned offline policy and use them with adaptive PID (aPID) to effectively adjust and learn the Lagrange multiplier $\\lambda$. The contribution comes with theoretical evidence (e.g. Theorem 1) and a large number of experimental supports, but my major concern is that the PID control part (and thus its adaptation rules) are incorrectly derived, although I value the authors' ideas and efforts.", "claims_and_evidence_yn": "No", "claims_explanation": "The authors have demonstrated the performance of the proposed approach with a large number of experiments including ablation studies, correctness checks and sensitivity analysis. The bound of the VPA Q-functions from the fixed points are given as a Theorem whose proof seems reasonable. However, as mentioned above, the PID control part (and thus its adaptation rules) are incorrectly derived. Their PID control (12) is *not* a PID control but a variant, so there is a lack of evidence that supports the design of the PID adaptation and their PID control part itself.", "audience_yn": "Yes", "audience_explanation": "This work particularly tackles the difficulty in O2O safe RL, which embraces the topics of RL such as offline RL, safe RL and offline and online transfer. The PID technique is quite control-engineering-oriented but has already been also studied in ML community (Stooke et al., 2020) in relevant works on safe RL. So, TMLR's audience, particularly in the fields of RL, shall be interested in the findings of the paper if the authors have corrected the error in their PID control explained below.", "requested_changes": "**[critical/main concerns on the PID control and its adaptation rules]**\n\n1. First of all, PID control (12) is *not* a PID control but a variant. Specifically, denoting the error $e_t := C(\\pi_t) - c_{th}$, we can view the Lagrangian update (3), $\\lambda_{t+\\Delta t} = \\lambda_t + \\alpha_\\lambda e_t$, as an Euler-discretisation of the Integral control $\\lambda_t = k_i \\int_0^t e_\\tau d\\tau$, with $\\alpha_\\lambda = \\Delta t \\cdot k_i$, as the time derivative satisfies $(\\lambda_{t+\\Delta t} - \\lambda_t) /\\Delta t \\approx \\dot \\lambda_{\\Delta t + t}$ $= k_i e_{t+\\Delta t}$ for small $\\Delta t > 0$. That is, $K_p$ in (12) must be the integral gain $k_i$ and there is no motivation of using $K_i$ in (12) which is actually a \"double\" integral gain that is not a part of PID control and rarely introduced to controllers for stability reason. In fact, the authors should've not started their derivation from (12) which is a mixture of discrete and continuous time. What Stooke et al. (2020) in the reference showed in their work is to propose the P and D gains attached to the Integral part \"$k_i \\int_0^t e_\\tau d\\tau$\" (Secs 4.2 and 4.3), so that the PID-control-oriented Lagrange update (when discretised) must have $k_p$, $k_i$ and $k_d$ terms associated with $e_t$ and its first- and second-order differences $\\Delta e_t$ and $\\Delta^2 e_t$, respectively. *In summary, the derivation needs to be corrected, and the authors have not introduced the $k_d$-part associated with $\\Delta^2 e_t$ but the seemingly unnecessary double integrator (the $K_i$ part); the introduction of the double integrator has no motivation, so I believe that $K_i$-term contributes no significant effect on the learning performance.*\n\n2. Since the PID part has not been designed properly, neither has \"adaptive\" PID (aPID) been, although aPID dramatically improves the learning performance. Specifically, (1) the design of the $K_i$-adaptation (14) does not make sense; (2) the $K_p$-adaptation (13) in fact corresponds to the adaptation rule of the integral term $k_i$ that is however introduced to eliminate the steady state (i.e., final) error, rather than speeding up the convergence, so the adaptation rule (13) is an unprincipled heuristic; (3) the $K_d$-rule (15) corresponds to $k_p$-adaptation rule, hence it is reasonable to design it in the way (13) and (14) are designed, but it has not done. Rather, $K_d$-rule (15) depends on the std dev $\\sigma_c$ (proportionally) and the average cost $\\bar c$ (inv. proportionally); it is not adaptive in the sense that it is always increasing as both $\\sigma_c$ and $\\bar c$ are always positive. I suspect that the $K_d$ gain will always increase from its minimum $K_{d_{min}}$ to maximum $K_{d_{max}}$, probably very rapidly at the early stage. Have the authors checked the trajectories of $K_d$?\n\nBecause of those two critical concerns, I lean to suggest a \"reject\", although I appreciate a numerous number of experiments to verify the performance, unless there is a significant improvement/revision including the derivations of the PID control and its adaptation rule and the experiments corresponding to PID/aPID control. Other less significant concerns regarding PID control are as follows.\n\n3. I believe the clipping thresholds (bounds) of $K_p$, $K_i$ and $K_d$ are critical. The authors mentioned that the hyper-params $\\alpha$, $\\beta$ and $\\gamma$ are easy to tune and the same across all environments (as shown in the Appendix), but how did authors tune the bounding parameters (e.g. $K_{p_{min}}$ and $K_{d_{max}}$). Are they also the same across the environments?\n\n4. A limitation of the PID approach in principle is that the sampling time $\\Delta t$ is sufficiently small, but there are many other tasks where $\\Delta t$ is not so or is even not defined (e.g. the game of Go). The authors are encouraged to mention this somewhere, e.g., Conclusions or Limitations in Appendix H.\n\n**[Minor Concerns/Corrections on Theorem 1]**\n\n1. $\\delta$ in the phrase \"with probability at least $1 - \\delta$\" affects the approximation error $\\tilde \\epsilon$ according to the proof but has not mentioned in Theorem 1. What is the implication of that dependancy? It'd be great if the authors mention it or add some explanation about $\\delta$ within Theorem 1 or the next paragraph.\n\n2. On pp. 5, \"Theorem 1 further implies that a larger $\\alpha^{VPA}$ should be selected in Eq. 8 for reward Q-values, whereas $\\alpha^{VPA}_c$ in Eq. 9 should be smaller to ensure a more accurate estimation of cost Q-values …\" — why? It may need some additional explanations… What if $\\alpha^{VPA}_c$ is large? Does the behaviour become more conservative?\n\n3. In Theorem 1, \"$\\epsilon$ and $\\tilde \\epsilon_c$ are the approximation errors that decrease the size of the dataset while increasing with the values of $\\alpha^{VPA}$ and $\\alpha^{VPA}_c$, respectively.\" — some errata in English? “decrease with the size of the dataset”?\n\n**[Other Minor Concerns]**\n\n1. It seems obvious that there is relevancy between $L_{Q_c}^{CPO}$ on pp. 4 and (6) and (7), among the $\\psi$-terms, especially between $\\mathcal{P}(\\cdot)$ in (7) and $Q_c(\\cdot)$ in $L_{Q_c}^{CPO}$. Can you explain it to me and if possible, briefly in the manuscript?\n\n2. On pp. 5, \"The sparsity of oﬄine cost leads to a pretrained cost Q-function that predicts low costs for IND state-actions, further limiting exploration during finetuning.\" — I don’t understand this… if the costs are low, then how it is limiting explorations? In my opinion, low costs increases possibility of constraints satisfaction, so more chance of exploring the region as the agent recognises those regions as being safe.\n\n**[Other Minor Corrections]**\n\n1. The authors have used different notations to represent the input distributions, namely, in Sec. 2, $s \\sim d$ and $(s, a, s') \\sim d$ in Online Safe RL subsection, and $D$ in Offline Safe RL part; both $D$ and $\\mathcal{D}$ in Sec. 3. There would have been some intention of using those different notations, but they seem too much and somewhat unnecessary. I suggest to use the same and/or consistent notations. E.g. right below (9), the authors have mentioned that $(s, a, s') \\sim D$ for offline data; it'd be good to explain when the notation $D$ appears for the first time in Sec. 2; I don't understand why $\\mathcal{D}$ is introduced in (6) and (7) rather than $D$ is used.\n\n2. Right below (15), what is the cost $c_i$ exactly? $Q_c(s, a)$?\n\n3. What is $\\mathcal{L}_2$ in Eqs. (8) and (9)? If it meant an L-2 norm, why don’t we employ the standard notation $||\\cdot||_2$; otherwise, we need a definition of it at least.\n\n4. In Algorithm 1 (Appendix A), what is the final policy? Just argmax of the Q? For clarity, it’d be great to mention the final policy within the algorithm, although it is mentioned in Sec. 4.1. The same for offline RL loop.\n\n5. pp. 10: \"To further verify this, the last two columns of table 2 show …\" —  change “columns” to “rows”", "broader_impact_concerns": "The paper deals with \"safe\" RL, which is applicable to safety-critical domains, but safety is not guaranteed during the learning phase. The offline RL may compensate this, but during the exploratory online phase, there are certain cases and chances of inevitably violating the safety margin as it is evidenced by the experimental results. So, a special care is needed when the proposed method is applied to safety-critical domains as the safety is still not guaranteed during online learning, especially at the early stage.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3JKJeSyLtx", "reviewer": "Reviewer_rko7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the problem setting of offline-to-online safe RL, which requires trading off between reward maximization and constraint satisfaction in a way that most efficiently makes use of both online samples and a pre-existing offline dataset. The paper identifies two potential challenges in applying existing methods to the offline-to-online problem: 1) inaccurate Q-value and cost predictions, and 2) inappropriately initialized Lagrange multipliers for the online fine-tuning period. It proposes a method to mitigate both of these issues by applying a policy evaluation phase during offline learning and using a PID controller to adapt the lagrange multipliers during the online phase.\n\n**Strengths**\n\n- The paper successfully isolates two independent challenges in adapting existing methods to the offline-to-online setting and conducts ablations to show that both of these issues must be addressed to obtain optimal performance.\n\n- The proposed PID controller for the Lagrange multiplier is a creative idea for efficiently recovering from inappropriate multiplier values.\n\n- Learning a policy from offline data that is efficient to fine-tune is a challenge that is relatively under-studied in RL, and this paper takes an interesting perspective on the challenge by also considering the additional complexity of added cost constraints.\n\n- The proposed method is relatively simple (aside from the number of additional hyperparameters it introduces, more on that later) and the paper provides a clear story to explain why it works. \n\n**Weaknesses**\n\n- The introduction poses offline to online RL as a \"pretraining to finetuning\" paradigm, but it is not exactly comparable to the traditional pretraining setting which uses a much larger and more diverse dataset for pretraining and then finetunes on a subset of that dataset.\n\n- The mismatch between the objective of offline safe RL vs offline-to-online safe RL suggests that offline safe RL methods should trivially be worse than any method which assumes access to a fine-tuning period. This makes the comparison between the proposed approach and offline RL methods feel unfair.\n\n- Value mismatch is a natural consequence of these different objectives; offline RL methods are deliberately pessimistic about the values of off-policy actions to reduce the risk of overestimation / bootstrapping errors. Inaccurate Q-estimation is a feature of such methods, not a bug! When using the offline phase as a 'warm-up', it is obvious that one does not need to be as conservative about off-policy action values and costs. Additionally, how conservative one wants to be will now depend on the fine-tuning budget allotted to the learner.\n\n- The proposed 'value alignment' phase is just on-policy value estimation of the data collecting policy. It seems like what the paper's results are saying is just that if you're going to fine-tune afterwards, you don't need to do any fancy offline RL methods and can just estimate the Q-values.\n\n- The Lagrange multiplier adaptation method introduces by my count at least 9 additional hyperparameters (alpha/beta/gamma and kmin/max for each K), which makes me question how robust the observed gains are. I'm not convinced by the empirical evaluations on two environments that the aPID is improving performance at a rate that is \"worth\" the added complexity.\n\n- The BallCircle comparison in Figure 1 just seems like none of the methods are actually learning (warm start gets less reward than MARVEL off the bat).\n\n- In Figure 3, the PID controller obtains similar costs as the aPID controller, but worse rewards, which seems odd to me as I would have expected the Lagrange multiplier to have roughly symmetric effects on reward and cost.\n\n- Empirical evaluations show that the adapted baseline methods dramatically underperform the offline learning baseline in almost every environment. This is surprising to me as I would have assumed that the O2O methods were explicitly designed to outperform an offline baseline, and makes me suspect that these methods were not given a fair shot in the evaluations (perhaps due to insufficient hyperparameter tuning.\n\n- I would have liked a clearer sense of how good the data collection policy that generated the offline examples was, and how much the benefits of MARVEL over from scratch / warm start / offline depend on the offline data collection policy. I would assume that the value alignment phase depends heavily on the original policy being reasonably good.", "claims_and_evidence_yn": "No", "claims_explanation": "The paper does not provide sufficiently convincing and clear evidence to support its conclusions in its current form. In particular:\n\n1. It's not clear how one should budget the offline RL pretraining and value alignment phases, or why policy evaluation is such a better starting point for online fine-tuning than offline methods. This seems like something that should have already turned up in the literature if it were a robust result.\n\n2. The number of hyperparameters involved in the aPID controller make me suspicious of the relatively small gains it appears to produce in the experimental results section.\n\n3. In general, the number of environments used, particularly for ablations, is fairly small and non-diverse, increasing the risk of the results being an inadvertent artifact of differentially motivated tuning of hyperparameters.", "audience_yn": "Yes", "audience_explanation": "In general, I think many people are interested in understanding how best to leverage offline data to facilitate sample efficient online learning. Ensuring that the learner also satisfies a cost constraint is similarly desirable in several problem settings.", "requested_changes": "The paper requires two major changes before I think it can be said to have rigorously shown that the proposed method is meaningfully better than a naive combination of existing approaches.\n\n1. The study of value alignment should demonstrate a clearer trade-off between effectiveness of the purely offline policy (which will be more conservative) and the value pre-aligned policy (which may be more vulnerable to going out of distribution but will also be a more accurate estimate of the true Q-value). One additional useful baseline would be to simply distill on the q-values of the policy which generated the data and see whether training on that from scratch does better than the slightly less direct value alignment phase which also aims to accurately estimate the q-values of the current policy. The benefits of this approach should also be stress-tested against poorly-performing initial policies in deterministic environments, where a good offline learning algorithm should be able to stitch together trajectories and learn a better initial policy than what would be obtained by policy evaluation.\n\n2. The benefits of the aPID controller need to be decisively shown to outperform what would be expected from tuning the additional degrees of freedom in the hyperparameter set.", "broader_impact_concerns": "N/A", "additional_comments": "One follow-up question I have about the value alignment phase is whether applying an offline RL method prior to the policy evaluation objective used for value function alignment even matters. If it does matter, does that mean that the policy evaluation phase needs to be limited in duration to avoid overwriting too much of the conservatism that was baked in originally? This is an important ablation that I would want to see in the paper, ideally demonstrating the sensitivity of the final performance to the length of the value alignment phase.", "extra_scores": {}, "extra_text": {}}, {"review_id": "woi6mywNTq", "reviewer": "Reviewer_nZsR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an effective method for offline-to-online finetuning in the context of safe RL, where in addition to maximizing a reward, an agent needs to stay within a certain cost budget, which measures an important constraint on the policy. The authors identify core challenges in handling the offline-to-online transfer, pessimistic values, and wrong Lagrangians, and introduce methods to address these.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall, the paper is well written and easy to follow. Some of the writing is a bit dense, for example I would encourage the authors to use enumerate as they frequently use numbered list which are inlined in text. This is a small issue though. The acronym of the method also feels extremely forced and is effectively unrelated to the name of the method. While the authors are free to pick any name of course, something a bit more aligned with the actual title might make it more memorable and easy for others to look for.", "audience_yn": "Yes", "audience_explanation": "Overall, the paper seems reasonable and (mostly) complete to me. The discussed issues seem of general relevance to the (sub)community, and the solutions fit the issues.", "requested_changes": "The problem discussion that motivate the warm start method is reminiscent of Zhou et al. (https://openreview.net/forum?id=HN0CYZbAPw), even though the solution is different. I actually somewhat prefer the solution suggested here. I suggest the authors briefly acknowledge and discuss the related work. It would also be interesting if retaining the offline buffer helps in some sense in the safe RL setting, which would justify keeping it around even better.\n\nThe PID controller introduction seems somewhat orthogonal to the offline-to-online setting. While I appreciate the difficulty of estimating the Lagrange multipliers from offline data, the PID method would probably work in both the offline-to-online and the pure online setting. I think it would be great if the authors experimented with the PID method as well in the standard pure online setting, as this might be of independent interest to researchers focused on that area.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)