Off-Policy Safe Reinforcement Learning with Cost-Constrained Optimistic Exploration
Authors: Guopeng Li
Matthijs T. J. Spaan
Julian F. P. Kooij
Venue: ICLR 2026 Poster (2026)
Keywords: constrained reinforcement learning
safe reinforcement learning
safe exploration
epistemic uncertainty quantification
Primary area: reinforcement learning
TL;DR: An off-policy primal-dual safe reinforcemen learning approach with cost-bounded optimistic exploration.

Abstract: When safety is formulated as a limit of cumulative cost, safe reinforcement learning (RL) aims to learn policies that maximize return subject to the cost constraint in data collection and deployment. Off-policy safe RL methods, although offering high sample efficiency, suffer from constraint violations due to cost-agnostic exploration and estimation bias in cumulative cost. To address this issue, we propose Constrained Optimistic eXploration Q-learning (COX-Q), an off-policy safe RL algorithm that integrates cost-bounded online exploration and conservative offline distributional value learning. First, we introduce a novel cost-constrained optimistic exploration strategy that resolves gradient conflicts between reward and cost in the action space and adaptively adjusts the trust region to control the training cost. Second, we adopt truncated quantile critics to stabilize the cost value learning. Quantile critics also quantify epistemic uncertainty to guide exploration. Experiments on safe velocity, safe navigation, and autonomous driving tasks demonstrate that COX-Q achieves high sample efficiency, competitive test safety performance, and controlled data collection cost. The results highlight COX-Q as a promising RL method for safety-critical applications.

Published as a conference paper at ICLR 2026
OFF-POLICY SAFE REINFORCEMENT LEARNING WITH CONSTRAINED OPTIMISTIC EXPLORATION
Guopeng Li
Matthijs T. J. Spaan
Faculty of Mechanical Engineering Delft University of Technology Delft, the Netherlands g. 1i 5etuudeftt.t
Faculty of Electrical Engineering, Mathematics and Computer Science
Delft University of Technology
Delft, the Netherlands
m. v.
Julian F. P. Kooij
Faculty of Mechanical Engineering Delft University of Technology Delft, the Netherlands
j@tudefft.n.
ABSTRACT
When safety is formulated as a limit of cumulative cost, safe reinforcement learn- ing (RL) aims to learn policies that maximize return subject to the cost constraint in data collection and deployment. Off-policy safe RL methods, although offer- ing high sample efficiency, suffer from constraint violations due cost-agnostic exploration and estimation bias in cumulative cost. To address this issue, we pro- pose Constrained Optimistic eXploration Q-learnin (COX-Q), an off-policy safe RL algorithm that integrates cost-bounded online exploration and conservative of- fline distributional value learning. First, we introduce novel cost-constrained optimistic exploration strategy that resolves gradient conflicts between reward and cost in the action space and adaptively adjusts the trust region to control the training cost. Second, we adopt truncated quantile crities to stabilize the cost value learning. Quantile critics also quantify epistemic uncertainty to guide ex- ploration. Experiments safe velocity, safe navigation, and autonomous driv- ing tasks demonstrate that COX-Q achieves high sample efficiency, competitive test safety performance, and controlled data collection cost. The results highlight COX-Q -XX-QQee as a promising RL method for safety-critical applications.
INTRODUCTION
Many real-world decision-making tasks have safety requirements. For example, robots must harm humans (Luo et al.) 2025), and autonomous vehicles must avoid collisions (Feng et al. 2023). Such concerns motivate safe reinforcement learning (RL), which commonly formulates the problem as constrained Markov decision process (CMDP) (Altman) 2021). In such setting, the agent aims to maximize the return while keeping the cumulative safety cost below threshold. The growing interest in the deployment of RL has led to increased attention to safe RL (Brunke et al. 2022).
Collecting data directly from the environment is imperative for many RL applications due to the limited fidelity of simulation the need for tonomous driving in mixed traffic (Chen al. 2024) and healtheare advising (Gottesman et al., 2019) tasks, agents must collect data safely in the real world. Therefore, sample for safe RL, as it directly determines the cost data collection.
interactions. For example, in au-
sample efficiency critical
Off-policy RL has higher sample efficiency than on-policy methods through experience replay (Chen et al. 2021) and active exploration (Ladosz et al. 2022). However, applying off-policy methods to safe RL faces substantial challenges. First, the underestimation bias in the cumulative cost of- ten leads to unsafe policies (Wu et al., 2024). Second, exploration in off-policy RL lacks cost constraints. The agent be misled into risky areas, causing uncontrolled data collection costs.

---

Published as a conference paper at ICLR 2026
Therefore, existing safe RL methods are predominantly on-policy (Gu et al. 2024b). Off-policy approaches struggle to satisfy cost constraints in data collection and deployment, as shown in the OmniSafe benchmark (Ji et al. 2024). These issues highlight critical knowledge gap:
How can off-policy safe RL maintain high data efficiency and meanwhile achieve robust constraint satisfaction in both data collection and deployment, through cost-constrained exploration and reli- able value learning?
To address this challenge, we propose Constrained Optimistic eXploration Q-learning (COX-Q), an off-policy primal-dual safe RL algorithm. COX-Q integrates a novel cost-bounded optimistic ex- ploration strategy with conservative value learning based mixed quantile crities. COX-Q demon- strates competitive performance on various safe RL benchmarks, showcasing its effectiveness for safety-critical applications.
RELATED WORK
This section provides concise overview of related work contextualize the core contributions of this study. We first clarify some key terminologies and define the scope of the overview. Safe RL. broad concept that involves a wide range of methodologies, such Control Barrier Functions (CBFs) (Chen et al. 2024), reachability methods (Ganai et al. (023). We focus on the formulation of safety constraints on cumulative costs, and address it within the constrained RL framework (Alt- man 2021). Additionally, this overview comprises only model-free safe RL methods. Model-basee methods (e.g., Safe Dreamer (Huang et al.] 2023)) are not included due fundamental differences. Related methods are grouped into on-policy and off- off-policy categories.
Most existing safe RL methods are on-policy, as sharing the behaviour and target policies allows each update to directly enforce constraint satisfaction through adjusted gradients or trust region techniques. On-policy approaches include first-order methods such as FOCOPS (Zhang et al. 2020) and CUP (Yang et al.] 20222. 2,2,) as well as second-order methods like CPO (Achiam et al. 2017 and RCPO (Tessler et al. 2018). Other variants include the PID-Lagrangian method (Stooke et al. 2020). risk-aware scheduling methods such Saute RL (Sootla et al. (2022a) and PPOSimmer (Sootla et al. 2022b), and the early terminated MDP formulation (Sun et al. 202T). These methods and their variants have demonstrated strong empirical performance in many safe RL benchmarks. For a comprehensive review, we refer readers to (Gu et al. 2024b).
In contrast, off- off-policy safe RL is less studied. Most approaches adopt primal-dual methods like Lagrangian and PID-Lagrangian (Stooke et al., (020), but suffer from poor safety performance due to the underestimation bias in cost values, often leading to constraint violations. To mitigate this, conservative cost estimators have been proposed. For example, Worst-Case SAC (WCSAC) (Yang al. 202I) penalizes underestimated costs to improve constraint satisfaction. CAL (Wu et al.) 2024 further accelerates training using local policy convexification and the augmented Lagrangian method, achieving strong safety and sample efficiency using high update-to-date (UTD) ratio. In terms of exploration, Gao et al. (2025) proposed the so-called MICE to address the underestimation of cost. The key idea is to use a memory-based intrinsic cost around unsafe states so the cost critic conservatively overestimates risk. Although the original implementation is for on-policy methods, in principle, the idea can be adopted to off-policy approaches. A recent study by McCarthy et al.) (2025) incorporates optimistic actor-critic (OAC) (Ciosek et al., 2019) into off-policy safe RL. The resulting ORAC algorithm actively explores regions with potentially higher reward and lower cost. While ORAC shows robust safety performance in tests, the authors state, it does not enforce cost constraints in data collection. How to realize cost-compliant exploration remains an open challenge.
In summary, a key gap in off-policy safe RL is the lack of principled cost-constrained exploration strategy integrated with conservative value learning. Our approach addresses this challenge from both theoretical and practical aspects.
PROBLEM FORMULATION
Consider CMDP defined by
S R is the state space. For state s; S, an agent controlled by a policy a ~ T(. (--ss) takes an action a, in the action space C R", then the next state follows p(s at). The agent receives reward re R. and pays non-negative cost
2

---

Published as a conference paper at ICLR 2026
C E R* The distribution of the initial state is po(su). y (0, is the discount factor shared by the cumulative reward Z and cost Ze, which are both random variables:
00
2 (st, 01) = T
(5 ar) =
7
C+++11
(1)
The state-action value functions (Q-functions) capture the expected return and cost for the policy: (st, a4) = Ez[2, ae)); (s,, az) = Ez (st. ac)]
(2)
In this setting, safe RL considers a constrained optimization problem:
max E. Bapp (Q7 (s, s.t. Ex--p,.-- Q2 (s. s d.
(3)
where Par the state density function of æ, and is cost threshold that should not be exceeded to ensure safety. The primal-dual approach constructs the following dual form, updating the policy π and Lagrangian multiplier À iteratively:
min Eerp, a-on -
(4)
arg mig À (d A>0
(5)
In summary, for safe RL, have two factors that impact exploration and policy learning:
A cost limit d divides (s, a) into safe (Qr S d) and unsafe (Q. > d) regions.
Two objectives, (5, a) for the return and Q (s, a) for the cumulative cost.
It is also useful to note that d is the cost limit for both data collection (training) and tests. This requirement is naturally satisfied for on-policy methods, but not for off-policy methods. Next, we introduce the proposed COX-Q algorithm in detail.
COST-CONSTRAINED OPTIMISTIC EXPLORATION
Off-policy RI. for continuous control tasks can use Optimistic Actor-Critic (OAC) (Ciosek et al, 2019) for active exploration. In single-objective RL, OAC first estimates optimistic upper bound of Q-value QLB from ensemble of critics, then maximizes this objective under a KL diver- gence constraint (trust region). the target policy is ((rr, Er), then the OAC exploration policy N(E, Σx) is given by the following proposition in the original paper (Cossek et al. 2019):
Σr V. TUQUUB (s,a))l= a))==pr
#E = = PT 28 x
EE Er,
(6)
V QUB a) ()
where 5 is threshold value for the KL-divergence between the target and the exploration policies, which is hyperparameter. For convenience, we can rewrite the displacement μs of the mean action in equation in terms of the total gradient Sr and the step length 7 follows:
28
PA = PE - PT = Ergg where 17 = =V Ergi
9t = VaQUB a)la=ur
(7)
For safe RI., the exploration policy is expected to fully explore safe regions, keep the number of visits to unsafe regions below the cost limit, and prevent any objective (return or cost) from dominating the exploration. Therefore, we propose the Cost-Constrained Optimistic eXploration (COX) strategy,
which extends the single-objective OAC (Ciosek al. 2019) multi-objective safe RL settings. In principle, COX exploration sequentially determines (1) the effective exploration direction g" and (2) the safe exploration step length 17' , to replace gi and 7) in equation 7 respectively. All theories this section are based on the assumption of Gaussian policies, which are compatible with most mainstream off-policy RI methods, such Soft Actor-Critic (SAC) (Haarnoja et al. 2018).
3

---

Published as a conference paper at ICLR 2026
4.1 POLICY-MGDA FOR EXPLORATION GRADIENT CONFLICT RESOLUTION
Since safe RL involves two objectives, we first determine the aligned, non-conflicting exploration direction g" in terms of these objective gradients. Omitting superscript π. we denote:
9r V.Q.B (s, a>l== ge = Va (s.a)la=ar, Im V VaQumass a)la=wr>
(s,
(8)
where superscripts "UB" and "LB" represent estimated optimistic upper and lower bound, respec- tively. Note that the dual form in equation[]aan favours higher reward and lower cost.
In safe regions: Within safe regions (Q" a) < d), the KKT condition of equation indicates that the constraint is not activated. The exploration considers the return along, thus g gr.
In unsafe regions: Within unsafe regions, the gradient for the overall objective according to equa- tion 4is 9r Age. However, this naive should not be used directly for g" since we further want to ensure that both return and cost are improving:
AQL (8, - g. م.
of Erg. I and P ( ME) = g. = 11 gTTT:: 2
one of the conditions equation is violated, we say that the exploration gradients conflict. For example, if gr dominates the exploration in unsafe regions, then the agent may be misled deeper towards the unsafe side. Note that 7) is non-ne non-negative, so whether gradients conflict and the magnitude of the conflict is measured by E-metric:
(g.99)) g Εイクソ
(10)
This metric is in the action space, so the covariance matrix of the policy is included. This is different from multi-task learning that uses the direct inner product in the model parameter space ZZang & Yang, 2021). To resolve exploration gradient conflicts, we extend the Multiple Gradient Descent Algorithm (MGDA) (Désidéri (012)) to the action space, forming the so-called Policy-MGDA. We first define gradient space (a "hyper-cone") in which both conditions of equation hold:
k= (g 6r Er 0, U. (-9en 9) 2 0}.
(11)
For two gradient vectors, such a K always exists except for degraded or co-linear cases. Then we find the optimal 9" that best aligns with the original direction 9. Xge w.r.t -mmtric:
g"
arg min "EK
[u (gr )l
(12)
Lemma We denote Irow Dr Age and the following Gram-scalars and multipliers:
SecUr + 8rcVe
He -8 +
8 Dc
Sij =
9j) U/ = (gt, 9i)
Pr
(13)
Scr - s?
Scy
s7c
Then the optimal solution for equation 12 is:
graw
if E K
Ur
graw
gr
i < and Ve S
g"
Syr
(14)
graw
ge
ifu, 0 and ve
Sec
Braw
Prgr Page
v and U. >>
The proof is in Appendix A.1 policy-MGDA operates in the action space during the online data collection stage, which makes fundamentally different from existing gradient manipulation methods operating in the offline model
g" is the aligned exploration direction in unsafe regions. Note that
update stage (Gu et al. 2024a Chow et 2021
Liu al. 2022)
4.2 ADAPTIVE STEP LENGTH FOR EXPLORATION COST CONTROL
Given the exploration gradient g¹, now determine the step length 77', which controls the data collection cost. Both the microscopic single-step exploration and the macroscopic training progress considered.
4

---

Published as a conference paper at ICLR 2026
For each exploration step, the original single-objective OAC does not involve the cost constraint in equation 3 To address this issue, we explicitly bound the cost expectation by adjusting the step
length 77 Given the exploration direction g", the threshold of non-negative violation along this direction is the hinge:
<n) = AQQ (d - Qa [rlym )") (d ))
(15)
Then we can formulate the following bi-level optimization problem:
arg max 44"
s.t. S nj S " <(n1"
min ó(E).
(16)
oSeSn
This means that, once the full exploration step length makes the mean cost exceed d, we choose the maximum 17" the trust region to ensure the cost constraint violation ((rr) is or minimized.
Lemma gm is defined in equation g We further denote:
(0m 9 E'r
r= = d Qmeen
(17)
Then the optimal solution for equation 16 is:
7
if <
1f"
0
if >> and < 0;
(18)
min(n.r/s
if and
The proof is given in Appendix A.2 0 around the optimum, s So, the oscillating sign of g" makes 17" jump between t7, manifesting as pure extra action noise. To address this issue, we further adaptively adjust d, thus maximum step length v,, based on a near-on-policy cost a recent replay buffer Brecest:
Nevertheless, equation 18 is not always valid. When g" tends
arg min x (d EecE Broca G3) 04658
(19)
As result, the exploration cost is governed by d. The adaptive step length tends to fully utilize the budget in safe regions, while remaining conservative unsafe regions. By using the two lemmas get the adjusted exploration direction g" and the step length 17".. Inserting them back into equation 7 gives the final COX exploration policy.
above,
It should be noted that this section assumes value estimation is accurate, particularly for costs. If the critics cannot provide reliable cost estimates due to the lack of data or function approximation errors, especially in the early stage of training, the data-collection cost is not effectively controlled. Plausible improvements include iusing classical methods, such as reachability analysis (Ganai et al.,
2023). or combining COX with model-based RL, such SafetyDreamee (Huang al. (023).
So far, we have explained the "COX-" part, including the effective exploration direction and the adaptive step length under cost constraints. Next, we introduce the "-Q" part about distributional value learning and uncertainty quantification.
5 DISTRIBUTIONAL VALUE LEARNING AND UNCERTAINTY QUANTIFICATION
Due to the sparsity of cost and sometimes sparse goal-reaching reward, learning the tails of return and cost distributions is crucial. Therefore, quantile critics are often used in safe RL, for example, in distributional WCSAC (Yang et al. 2023) or ORAC (McCarthy et al.) 2025). The objective function equation indicates that the Bellman update favours overestimation bias of return and underestimation bias of cost (Wu al. 2024). Moreover, stabilizing the value learning and reducing its gradient variance is important for constraint satisfaction. Considering all these requirements, we
adopt Truncated Quantile Critics (TQC) (Kuznetsov al. 2020).
TQC follows Quantile Regression RL (Dabney et al. 2018). Each independent critic learns the re- turn distribution by certain number of evenly distributed quantiles. TQC mixes and sorts quantiles from all critics, and then truncates the top & atoms to mitigate the overestimation bias. Specific to safe RL,, truncate the top k atoms for reward and the bottom k; atoms for cost critics. The mixed

---

Published as a conference paper at ICLR 2026
atoms provide low-variance gradients stabilize the learning, and the number of truncated atoms controls biases with high flexibility.
Another advantage of TQC is that quantifying distribution-level epistemic uncertainty is straightfor- ward. Suppose that we have N cost critics and N reward critics, each critic predicts M quantiles,
denoted as (x) ( a). representing the quantile function value at level Tm (m 0.5)/M. The overall confidence bounds are estimated by computing per-quantile bounds first, and then aggregat- ing them using Conditional Value at Risk (CVaR) (Rockafellar et al. 2000):
M Σ
1
(m, a) =mm,oss, a) - 6,0m,e(s,a) QL (s)
(s.
(20)
M vn Maa
M
Pmn. (8, Pm, (8, a) 3, 2
Σ (8,
(21)
M
ma
Here Pm../c and mmr are quantile-wise mean and standard deviation across N critics, respec- tively. The two hyperparameters, B, and Se, adjust the aggressiveness of exploration. For cost, use the a head quantiles only, which determines CVaR. A smaller a means more risk-aversion in policy learning, like in WCSAC| et al. (2021). For return, consider the full distribution to compute the optimistic upper bound. Inserting equation 20 and equation 21 into equation gives the corresponding gradients.
Combining COX and TQC-based conservative learning yields the full COX-Q algorithm. It ad- dresses both unconstrained exploration and stable cost value learning in one integrated framework. The implementation is based on SAC (Haarnoja et al. 2018). and keeps the ALM (essentially an enhanced constraint violation penalty) used in CAL (Wu et al. 2024) and ORAC (McCarthy al.]
2025). The pseudo-code of COX-Q is provided in Appendix
6
EXPERIMENTS
We compare COX-Q against off-policy and on-policy baselines three safe RL benchmarks:
Safe Velocity Safe Velocity is a set of velocity-constraine dense-reward robot locomotion tasks. The agent moves alone in the environment. It has immediate binary cost signals: exceeding the velocity threshold incurs a cost of 1, otherwise O. The episode cost limit(for 1000 steps) is 5. We select four robots, hopper, walker2d, ant, and humanoid, which share the same reward structure. For faster training, experiments are run in Brax (Freeman et al. 2021). Detailed environment settings explained in Appendix C.
Safe Navigation Safe Navigation requires robot to reach goal or complete a specific task, while avoiding static and moving hazards. The observation is Lidar points. Safe navigation has a small, dense progress reward and a large, sparse goal-reaching reward, along with sparsely activated costs. We select tasks: SafetyPointGoul2, SaffetyPinn/utton22, SafetyCarBution/ SafetyCarButton2, and SafetyPointPushl. where the suffix "2" denotes the highest difficulty level. In general, controlling a car agent is more difficult than a point agent because a car cannot rotate but steer to the desired location, and button tasks are more challenging than goal tasks. We set d = 10. Detailed descriptions about the environment and costs are provided in Appendix
SMARTS safe autonomous driving In autonomous driving, the vehicle interacts with other road users in a closed-loop manner, making it substantially challenging. We use the SMARTS simula- tion platform (Zhou al., 2020). Three scenarios with intensive vehicle interactions are selected: Overtaking a two-lane highway, Intersection without traffic lights, and T-junction without traf- fie lights. In the last two scenarios, the vehicle needs to execute an unprotected left turn and lane change sequentially. The reward includes a small distance progress towards the goal and big bonus the vehicle reaches the goal. The cost is 10 if the vehicle collides, drives off-road, or violates traf- fie rules severely (drives into the opposite direction). If the vehicle fails to reach the goal in 60 the episode terminates (marked as a timeout). More details are provided in Appendix C.3] including a discussion about the reward and cost design that might be useful for some interested readers.

---

Published as a conference paper at ICLR 2026
Baselines Selected baselines include representative on on-policy and recent oftte-of-the-are off off-po methods. For on- on-policy baselines, we select one from each the categories introduced Section 2 They are CUP Yang et al., 2022), RCPO (Tessler et al. 2018), PPOStmmerPID (Sootla et al. 2022b), and CPPOPID (Stooke et al. 2020). For Safe Navigation tasks, we replace CPPOPID with
TRPOPID because its performance advantage, shown in Omnisafe (Ji al. (024).
Off-policy baselines are more relevant to our proposed method. We choose: (1) SACLag- UCB imple- ments conservative double-Q cost learning Hasselt (2010) augment the SACLag (Ji al. 2024). which uses SAC with the Lagrangian- which uses the conservative cost learning and ALM (Luenberger et al. 1984). The update- to data (UTD) ratio is set as 1 to avoid being over-conservative in our sparse-cos tasks; (3) Distributional WCSAC Yang et al. (023). which uses quantile cost critics and a conservative, risk-averse actor objective based on CaRR; (4) MRACCMc et al. (205)) recent method which combines WC- SAC and the ALM CAL, and further adds risk-averse optimistic exploration towards the low-cost side. Details about the implementations of baselines are explained in Appendix D
method (Stooke et al. 2020) (2) CAL (Wu et al.
2024).
For Safe Velocity and Safe Navigation, we conduct 10 runs with the same 10 random seeds for all methods. For the autonomous driving benchmark, we only select CPPOPID, SACLag, CAL, and ORAC as baselines, and only train the policy once with the same random seed due to the long training time. The code is available via: lttpps//gittub.comm htt.b/ccoooo/mamiiiiiimoa
6.1 RESULTS ON SAFE VELOCITY
hopper
walker2d
ant
humanoid
return (k)
2-
1.0
0.0
1.0
0.0
100
100
100
100
TestipCost
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
2
TrainingEpCoos
100
100
80
50
0.
0-
0.0
env onv_stop (million)
cnv stop (million)
inv_stop (million)
env_stop (million)
SACLag-IC8
CAL(UTD=1)
WCSAC
GRAC
COX-Q (ours)
hopper
walker2d
ant
humanoid
Test return
2
1000
100
100
100
Testip
50
0.5 (million)
1.0
0.0
1.0
0.0
1.0
(million)
siv_step (millien)
(million)
CUP
RCPO
PPOSiimeer1D
CPPOND
CDP-Q (ours)
Figure 1: COX-Q Q v.s. off-policy (top) and on-policy (bottom) baselines. TrainingEpCost for data collection, which is expected to stay near or below the threshold throughout the training. Note that training and test costs are identical for on-policy baselines.
The results on the Safe Velocity benchmark are presented in Figure I Overall, COX-Q -QX demonstrates superior sample efficiency, achieves high cumulative returns, and has nearly-zero test costs fast, while keeping data collection costs below the predefined budget. More specifically: (1) COX-Q exhibits a clear advantage in data efficiency over on-policy baselines. Its ooflpo nature and the truncation mechanism in TQC enable test cost significantly lower than the budget, property
7

---

Published as a conference paper at ICLR 2026
that on-policy methods do not have. (2) By comparing COX-Q against SACUCB-PID and CAL, observe that distributional RL has higher sample efficiency than point-value based baselines, (3) The cost-constrained exploration and step-length auto-tuning efffettvvlyy regulate the data-collection cost, especially the middle and late training phases. This is evidenced by the smooth and horizontal (near the threshold) training cost profiles of COX-Q in all tasks. In contrast, baseline methods incur higher training costs on one or more tasks due to unregulated optimistic exploration. For humanoid, no baseline policies can make the robot walk fast enough reach the unsafe boundary, so both training and test costs are near-zero. These observations highlight the key strengths of COX-Q: high data efficiency, improved safety, and controlled training costs for exploration. Next, we assess its performance in exploration-challnngee
environments.
6.2 RESULTS ON SAFE NAVIGATION
Due to the sparse goal-reaching rewards and costs Safe Navigation, truncating too many atoms can suppress the learning progress. Therefore, preserve the mixed quantiles in COX-Q but do not apply truncation. Instead, we use the estimated CVaR-based upper bound of cost to update the actor and Lagrangian multiplier, same in Worst-Case SAC (Yang al. 202T).
SafetyPointGoal2
SafetyPointButton2
SafetyCarButton1
SafetyCarButton2
SafetyPolntPush2
25
25
0
Test
-25
1.0
TestEpoost
40
40
40
TrainingEpCoss
40
20
20
20
20
20-
0
EstimateBias
0.0
0.0
0.0
-2.5
-2.5
-2.5
-2.5
-25
5.0
5.0
5.0
5.0
5.0
0
0
env_step (million)
env_step (million)
env_step (million)
env_step (million)
step (million)
SACLag-UCB
CAL
WCSAC
ORAC
COX-Q (ours)
Figure 2: Benchmark of COX-Q against off-policy baselines on safe navigation tasks (episode cost limit 10). The bottom figure the cost value estimation bias, computed from cost critic outputs and the recorded trajectories in the evaluation phase. Below O means underestimation.
The results summarized in Figure 2 For on-policy baselines, the results provided in the Appendix[E E In general, COX-Q achieves on-par or higher returns than basclines. The advantage is significant in more challenging tasks CarButtonl, CarButton2, and PointPushl. The training and test costs both converge below the limit. The estimation bias of COX-Q consistently converges to 0 with training, while all baselines are either coer-conservative or unstable. This observation indicates that the mixed quantiles significantly improve value learning.
6.3 ABLATION STUDIES ON SAFE VELOCITY AND SAFE NAVIGATION
Next, we evaluate the contribution and role of each module of COX-Q. Two variants are compared: (1) with TQC only, without exploration; (2) TOC ORAC-style exploration.
The results on 4 selected tasks are shown in Figure 3 In all tasks, all ablation models' returns remain higher than baselines, which indicates that the TQC contributes mainly improved returns. In Safe
8

---

Published as a conference paper at ICLR 2026
walker2d
SafetyCarButtonn1
SafetyCarButten22
Test return (k)
Test return
20
0.0
0.2
0.4
0.0
0.2
0.4
-10
0.0
0.5
1.0
0.0
0.5
1.0
TestEpCast
20
Test5pCost
40
40
20
20-
1000
100
0.0
0.5
1.0
0.0
0.5
1.0
TrainingEptes
TrainingEpCor
.5-
50
0.2
0.4
stop (million)
(million)
TQC,
CoX-Q (ours)
TQC, exploration
CoX-Q (ours)
TOC ORAC
TQC ORAC
Figure 3: Ablations Safe Velocity and Safe Navigation.
Velocity, the exploration strategy significantly influences training costs. Both ORAC and COX- Q exploration increase training cost, but the proposed cost-constrained mechanism well controls training costs below the given budget. However, in Safe Navigation, COX-Q and ORAC exploration exhibit close performances to the TQC-only baseline. This highlights two properties of the task:
(1) Gradient conficc in the action space is weak in Safe Navigation due to the sparse obstacles. Therefore, ORAC and COX-Q become identical in most cases. In Appendix [ the analysis shows that the ratio of triggered gradient conflicts in the first 200K steps is below 10%, and even below 2% for PointPushl. (2) Cost value learning in Safe Navigation is highly biased due to the sparsity of cost signals. As shown by the bottom row of Figure the cost is underestimated during the early training stage. Correspondingly, the cost constraint violations for training and testing are triggered. This important observation indicates that, for constrained RL with sparse costs, the underestimation bias in the cumulative cost the major bottleneck, rather than the exploration mechanism. Possible improvements include inducing Hindsight Experience Replay (HER) (Andrychowicz et al., (017) or prioritized experience replay (Schaul et al.) 2015) to better estimate the cost objective.
6.4 EVALUATION ON SMARTS SAFE AUTONOMOUS DRIVING
Finally, we evaluate the effectiveness of COX-Q in more challenging autonomous driving tasks, in which surrounding vehicles have closed-loop interactions with the controlled RL agent. Au- tonomous driving is a typical safety-critical task. We set a nearly-zero cost limit (0.01), same as in SafetyDreamer's MetaDrive task (Huang et al. 2023). The vehicle stays in "unsafe" regions (the cumulative cost is above 0,01) during data collection and aims to minimize the test cost as much as possible, thus intentionally increasing the frequency of exploration gradient conflict and the pro- portion of constrained exploration. Because the policy stays in unsafe regions during training, we do not add the step length auto-tuning in equation 19 avoid the exploration converging to zero, which will make COX-Q the same as original TQC. Additionally, in this benchmark, we use the TQC-based ORAC focus on the differences caused by the exploration mechanism. After 512K steps of training, we run 2000 episodes with stochastic initial states to obtain the test performance.
The test performance is presented in Table T and the number of unsafe events (collisions and off- road) during training is listed in Table 2 Overall, COX-Q achieves the best safety performance in tests without incurring significant excessive exploration cost or exhibiting over-conservative driving behaviours (time-out). Moreover, compared to ORAC, COX-Q significantly reduces both unsafe events during data collection and timeouts during testing. This observation indicates that resolving conflicting gradients in direction that simultaneously reduces cost and improves reward can effec- tively maximize return while avoiding being over-conservative safety performance of all methods in overtaking is relatively worse. The reason is that SMARTS uses an instantaneous lane change model from SUMO (Krajzewicz et al. 2012), making collision avoidance inherently hard due to the lack of warning (e.g., turn signals).
Another notable point is that the

---

Published as a conference paper at ICLR 2026
Table I: Test safety performance SMARTS (512K steps, 2000 stochastic runs)
Scenario
Metric
CPPOPID
SACLag
CAL
TQC-ORAC
COX-Q (ours)
Collision
331
194
186
97
99
Off-road
96
7
3
Overtaking
Rule violation
Timeout
887
Collision
183
33
23
18
12
Off-road
2
Intersection
Rule violation
Timeout
Collision
195
55
36
28
21
Off-road
91
2
0
5
0
T-junction
Rule violation
Timeout
Table 2: Number of unsafe events in data collection (512K steps, excluding the initial 5120 steps)
Scenario
CPPOPID
SACLag
CAL
TQC-ORAC
COX-Q (ours)
Overtaking
3697
1570
1544
3215
1665
Intersection
4969
1755
739
3589
1123
T-junction
5513
1965
1675
3837
1794
7 CONCLUSIONS
This paper proposes an off-policy primal-dual safe RL method, constrained optimistic exploration Q-lcarning, involving a cost-constrained optimistic exploration strategy and TQC-based conserva- tive value learning. The proposed COX-Q is evaluated in three representative safe RL benchmarks. The results demonstrate that COX-Q has significantly higher data efficiency than on-policy base- lines. When the exploration gradient conflict between reward and cost is significant (Safe Velocity and SMARTS), COX-Q shows superior safe performance in tests, while effectively controlling ex- ploration cost in data collection. When the exploration gradient confliet is weak, the bias in cost estimation is high due to sparse cost signals (Safe Navigation), COX-Q is par better with the state-of-the-ar method. posed method can be used in complex environments with large neural networks. In conclusion, COX-Q a promising solution to RI. applications with data efficiency and safety concerns.
addition, the autonomous driving experiment showcases that the pro-
The major limitation of this study is the reliability of quantified epistemic uncertainty. TQC mixes quantiles from all crities and learns the entire return distribution. Therefore, the diversity of critics for nearly Out-Of-Distribun dients for all critics. Implementing improved methods such as diverse ensemble projection (Zanger) et al. 2023) random priors (Osband al. 2018) to enhance the quality of epistemic uncertainty quantification is a potential future research direction. Another future research direction is how to effectively implement COX in sparse-cost tasks such SafeNavigation. A key step is to use, e.g., HER (Andrychowicz et al. (017) or prioritized experience replay (Schaul et al. 2015) to robustify the cost-critic learning.
Limitations
samples might be suppressed due to highly correlated gra-
This work has received funding from the European Union's Horizon 2020 Research and Innovation program under Grant Agreement No. 964505 (E-pi).
Acknowledgments
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning. pp. 22-31. PMLR, 2017.
Eitan Altman. Constrained Markov decision processes. Routledge, 2021.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience play. Advances in neural information processing systems, 30, 2017.
Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela Schoellig, Safe learning in robotics: From learning-based control to safe reinforcement learning, Annual Review of Control, Robotics, and Autonomous Systems, ((1):411-444) 2022.
Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End- to-end autonomous driving: Challenges and frontiers. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without model. arXiv preprint arXiv:2101.05983 2021.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duzmazz and Mohammad Ghavamzadeh. Safe policy learning for continuous control. In Conference Robot Learning, pp. 801-821. PMLR, 2021.
Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems, 32, 2019.
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Jean-Antoine Désidéri. Muldipte-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, 350(5-6):313-315 2012.
Shuo Feng, Haowei Sun, Xintao Yan, Haojie Zhu, Zhengxia Zou, Shengyin Shen, and Henry X Liu. Dense reinforcement learning for safety validation of autonomous vehicles. Nature, 615(7953): 620-627, 2023.
Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106. 13281, 2021.
Milan Ganai, Zheng Gong, Chenning Yu, Sylvia Herbert, and Sieun Gao. Iterative reachability estimation for safe reinforcement learning. Advances in Neural Information Processing Systems, 36:69764-69797, 2023.
Shiqing Gao, Jiaxin Ding, Luoyi Fu, and Xinbing Wang. Controlling underestimation bias in con- strained reinforcement learning for safe exploration. In Proceedings of the International Confer- ence on Machine Learning, 2025.
Thomas Gillespie. Fundamentals of vehicle dynamics. SAE international, 2021.
Omer Gottesman, Fredrik Johansson, Matthieu Komorowski, Aldo Faisal, David Sontag, Finale Deshi-Velez, and Leo Anthony Celi. Guidelines for reinforcement learning healthcare. Nature medicine, 25(1):16-18. 2019.
Shangding Gu, Bilgehan Sel. Yuhao Ding, Lu Wang. Qingwei Lin, Ming Jin, and Alois Knoll. Balance reward and safety optimization for safe reinforcement learning: A perspective of gradient manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 21099-21 21999-11006 106, 2024a.
11

---

Published as a conference paper at ICLR 2026
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theories and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024b.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. Pmlr, 2018.
Hado Hasselt. Double q-learning. Advances in neural information processing systems, 23, 2010.
Weidong Huang, Jiaming Ji. Chunhe Xia, Borong Zhang, and Yaodong Yang. Safedreamer: Safe reinforcement learning with world models. arXiv preprint arXiv:2307.07176. 2023.
Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yi- fan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learn- ing benchmark. Advances in Neural Information Processing Systems, 36:18964-18993 2023.
Jiaming Ji. Jiayi Zhou, Borong Zhang, Juntao Dai, Xuchai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. Journal of Machine Learning Research, 25(285):1 2024.
Daniel Krajzewicz, Jakob Erdmann, Michael Behrisch, Laura Bicker, et al. Recent development and applications of sumo-simulation of urban mobility. International journal on advances systems and measurements, 5(3&4):123 128-138, 2012.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overesti- mation bias with truncated mixture of continuous distributional quantile critics. In International conference on machine learning, pp. 5556-5566. PMLR, 2020.
Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1-22, 2022.
Quanyi Li, Zhenghao Peng, Lan Feng. Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 45(3):3461-3477 2022.
Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Con- strained variational policy optimization for safe reinforcement learning. International Confer- ence on Machine Learning, pp. 13644-13668. PMLR, 2022.
David G Luenberger, Yinyu Ye, et al. Linear and nonlinear programming, volume 2. Springer, 1984.
Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning. Science Robotics, 10<105)eaads5033, 2025.
James McCarthy, Radu Marinescu, Elizabeth Daly, and Ivana Dusparic. Optimistic exploration for risk-averse constrained reinforcement learning. arXiv preprint arXiv:2507.0873 087999 2025.
Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2980-2987. IEEE, 2023.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. Advances neural information processing systems, 31, 2018.
R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2:21-42, 2000.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay, arXiv preprint arxiv: I51: 1.09952 2015.
Aivar Sootla, Alexander I Gowen-Rivers Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmenta- tion. In International Conference on Machine Learning, pp. 20423-20443 PMLR, 2022a.
12

---

Published as a conference paper at ICLR 2026
Aivar Sootla, Alexander I Cowen Rivers, Jun Wang, and Haitham Bou Ammar. Effects of safety state augmentation on safe exploration. arXiv preprint arXiv:2206.02677 02677,,,..00 2022b.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp. 9133-9143. PMLR, 2020.
Hao Sun, Ziping Xu, Meng Fang, Zhenghao Peng. Jiadong Guo, Bo Dai, and Bolei Zhou. Safe exploration by solving early terminated mdp. arXiv preprint urXiv:2107.04200, 2021.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv: /805. /074, 2018.
Chen Wang, Yuanchang Xie, Helai Huang, and Pan Liu. A review of surrogate safety measures and their applications in connected and automated vehicles safety modeling, Accident Analysis & Prevention, 157:106157, 2021.
Zifan Wu, Tang, Qian Lin, Chao Yu, Shangqin Mao, Qianlong Xie, Xingxing Wang, and Dong Wang. Off-policy primal-dual safe reinforcement learning. In ICLR, 2024.
Long Yang, Jiaming Ji, Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang, and Gang Pan. Constrained update projection approach to safe policy optimization. Advances in Neural Information Processing Systems, 15:9111-91221 2022.
Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. Wesac: Worst-case soft actor critic for canttacconttrained reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10639-10646, 2021.
Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. Safety-constrained reinforcement learning with distributional safety critic. Machine Learning, 112(3):859-887, 2023.
Moritz Zanger, Wendelin Böhmer, and Matthijs TJ Spaan. Diverse projection ensembles for distributional reinforcement learning. arXiv preprint arXiv:2306.07/24, 2023.
Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 33:15338-1534, 2020.
Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE transactions knowledge and data engineering, 34(12)55566-5600 2021.
Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Mont-
gomery Alban, Iman Fadakar, Zheng Chen, al. Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving. arXiv preprint 27Xiv22010.09776, 2020.
13

---

Published as a conference paper at ICLR 2026
A PROOFS OF THE TWO LEMMAS
A.1
LEMMA-1
The solution of equation and equation 12 is given follows. For simplicity, we denote 91 Dr-
92 -cc. and Σ En here.
Decompose 9t. - Us g1 with gs S and (g),g1)s , 0. Constraints depend only on 9s. Therefore, it suffices to solve in S and then add back 91-
Let S (g.
This becomes 2D problem. We next derive the KKT conditions. With the inequalities eı (u)
- (91. uiy I O and (2( = -99 u) I 0, we add two non-negative multipliers to form the
Lagrangian:
[(u, (1, = 2 uu - 1e + (-(1,, u)s) + p2( ((2 4/xz)
(A.1)
Then the stationarity is:
TuC=I(u 91) -1 - 12Σ = u = 9t + 1191 + P292
(A.2)
The primal feasibility gives:
",,u,>= 2 0, (92, u/y 2
(A.3)
The complementary slackness gives
Pi (9 u/y = 0, Pz u/y: O.
(A.4)
So, we define the so-called G-Gram scalars and target correlations as:
8ij = (gi Fy)x,
U2 = (gis 0t)x,
(A.5)
and plug stationarity into the constraints:
$11
812
M1
01
(A.6)
821
822
[22
U2
Because the Gram matrix is apparently SPD if 91 and 92 are not co-linear, the solution is unique whenever both constraints are active. For the degenerated co-linear cases, we assume that g1 ag2. a > 0, then K is a half-space, then the solution a direct projection:
u = 9e min(0, )91-
(A.7)
$11
If 0, constraints reduce to (g), u)r 0 (hyper-plane):
g" u St
$11 91,
(A.8)
and a is trivial.
For non-degenerate cases, we apply the optimal active set A = c2(4)) There are four possibilities:
(1) No constraint active: Then A1 #2 = 0, g" = Dt.
(2) Only ei (u) active: Set μl2 = From (91, u) = 0, we get:
U1
p1
u gt
811 91
(A.9)
811
(3) Only (2(u) active: Similar to the previous case, we have:
u" St
$22 92-
(A.10)
(4) Both boundaries active: Then we solve equation A.6
That gives:
P1 U1 812
H2 det G
812
811
det G 811822 S 20 0 (A.11)
det G
='= St #191 #292.
(A.12)
Replace ΦI and 92 by gr and -ec respectively, then the proof of Lemma is done.
14

---

Published as a conference paper at ICLR 2026
A.2 LEMMA-2
The solution of equation is derived based on two cases;
Case A: O, which means moving along the does not increase the expected cost. The hinge (()) is non-increasing w.r.t. 1). Therefore, minimal violation is achieved by taking the largest trust region:
= TKL
(A.13)
Case B: O, which means moving along the u" increase the expected cost. Then check the feasibility of a zero-violation set on the ray {n 17 S r). If r I 0. then the zero-violation set is empty [0, nku] and the hinge increases with r,. Therefore, the minimizer is trivial 17" = 0. If r 2 0, simply take the boundary as the zero-violation set:
min(nx.. :
(A.14)
Case C: which means the hinge becomes constant. this case. I 2 0, every 1) E [0, 7Kl] optimal. If < O. violation is unavoidable. We 17" by rule for conservativeness.
Combining the three cases above gives the complete proof of Lemma
B IMPLEMENTATION DETAILS OF COX-Q
Algorithm COX-Q based on SAC, with optional Augmented Lagrangian Method (ALM)
Input and initialization:
policy network Te(s). N reward quantile critic networks (gv.. 1.N N cost quantile critic networks o9 i=1, with M quantile heads.
replay buffer D, truncation parameters and ki exploration optimism parameters B, and 8 cost limit d, maximum trust region size VKL (6), Lagrangian multiplier X,
risk-level CVaR a repeat
Observe State St. if use COX then
Compute the target policy (HT,, Er) = (s;))
Compute QUR Q!. Qmeen from critics using equation and equation 21 Compute their gradients Des Jmn Jer
if Qmuean in safe area then compute g"
Age
else
Compute aligned exploration gradient g" using equation 14
end if
Compute adjusted step length 17" using equation 18
Compute action shift μa using OAC formula from 17" and g" select action a; = clip(ue Glowers Glower, aupper), where N N(us, 2,)
else
select action a; (sip(uo(sE) + C, Olower, Oupper), where E.)
end
Execute 0r, observe next state 8 reward rt and cost Ca
Store the transition (#r, dt; (r,, in D
critic/actor update then Execute TQC or Worst-Case SAC updates, with optional ALM (used by default) end if if 7/K update then Sample recent N, transitions from D, compute the average cost
Update 5 using equation
end until Convergence
15

---

Published as a conference paper at ICLR 2026
In the pseudo-code of Algorithm [ the updates of critics the same as the original TQC (Kuznesovv et al., 2020). The actor update involves the ALM proposed buenebrerg et al. (1984) and introduced in safe RL by Wu et al. (2024). ALM alters the optimization objective of the actor
by the following equations:
m Bexpe,anat( Quean max, (s--pea-at(1s)
XQQ d) o QQU d) if
À 2d E(QUC))
(s) Qmem))
otherwise
(B.1)
The added quadratic term helps conform to cost constraints and move the optimization direction towards the cost limit, which can accelerate the learning process. In our studies, we use = 10 for all tasks. This ALM used for CAL, ORAC, and COX-Q in all experiments, in their original paper.
In addition, off-policy safe RL needs to set the cap on Q-values d in an "on-policy" approach, instead of directly using the test episode costs as in on-policy methods. This is explained in the paper of CVPO (L.iu al. 2022), using the following formula:
1-77
d depisode T(1 7)
(B.2)
in which the episode length. In all off off-policy methods used in this study, we use this formula to convert the episode cost limit to the limit Q
C DESCRIPTION OF THE THREE SAFE RL ENVIRONMENTS
C.I SAPETYVELOCITY-V
hopper
walker2d
humanoid
Figure C.1: The four selected robots in Velocityvl benchmark.
For the selected 4 robots, their configurations are shown in Figure C.I They share the same reward structure as follows:
wh Thealth
Tvelocity We Tairl
(C.1)
which Thealth is a binary reward. If the robot keeps upright, get +1 reward: otherwise, get 0 and terminate the episode. Fvelocity is reward equal to the moving velocity along a given direction. Teul is the control cost penalty, measuring how much torques are applied to the joints. wh. wa and W. are three positive weights. Cost is binary. For hopper and walker2d, if the velocity along the +x axis exceeds the threshold, the cost is +1; otherwise, O. For ant and humanoid, if the velocity along any direction exceeds the threshold, the cost is +1; otherwise, 0. The episodie cost limit is set to 25, as recommended in the original paper (Zhang et al. 2020). The weight coefficients, velocity thresholds, and the dimensionality of action spaces for different robots are listed in Table C.I All inplementations are based on the Brax (Freeman et al. 2021), using the same parameters (e.g. velocity thresholds) as in Sefeet-Gyymasiumm (Ji al.] 2023). Brax supports fully parallelized simulations on GPU, so it can save lot of time for training. The default "generalized" backend is used for simulation.
C.2 SAFE NAVIGATION IN SAFETY-GYMNASIUN
In Safe Navigation, the name of a task is composed of two parts. "-Point-" or "-Car-" in the middle indicates what is the type robot used, as shown on the top of Figure C.2, Point is a simple robot that has two actuators, one for rotation and the other for forward/hackward movement. Car is
16

---

Published as a conference paper at ICLR 2026
Table C.I: Weight coefficients and velocity threshold for Safety Velocity-vl
ROBOT
(wh, Wes we)
velocity threshold
Action dimension
hopper
(1, 0.001)
0.7402
walker2d
(1, 0.000)
2.3415
6
ant
1, 0.5)
2.6222
humanoid
(5, 1.25,, 0.1)
1.4119
17
more complex robot that can move in two dimensions. is equipped with two independently driven parallel wheels and freely rotating wheel. Both steering and forward/backward motion require coordinated control the two drive wheels, imposing more complex control dynamics. Both robots equipped with 2D Lidars perceive the environment. Their action dimensionalities are both 2.
The last part of the name indicates the type of task and its difficulty level. Three example tasks are shown at the bottom of Figure C.2
Goal2: The robot needs to reach a goal position (green pillar) while avoiding touching
hazard pitfalls (blue circles) or move fragile vases (while cubes).
Button2: The robot needs reach the correct button (orange spheres) among 4 buttons, while avoiding touching blue-cirele pitfalls or being hit by the moving gremlins (purple cubes moving in circle).
Push/: The robot needs push the yellow object the green goal position while avoiding blue pitfalls and the tall pillar.
Point
Goal-2
Button-2
Push-1
Figure C.2: The robots and the tasks in the safe navigation benchmark.
The reward and cost designs complicated, depending each specific task. We refer the readers to the public webpage of the Safery-Gymmasium for more details: https://safet- gymnasium. paaedthhedoss.ooeee//leeeeeeiiie
http:/aate.aaaiigitiooo..oit html
Additionally, to accelerate the learning process, the simulation time step is modified to 2.5 times the original value, according the paper CVPO (Liu al. (2022) and CAL (Wu al. (0244). While ORAC (McCarthy et al. 20255) does not release its code, the final reward performance implies that they probably used the same simulation settings. We therefore also keep the modification.
C.3 SMARTS AUTONOMOUS DRIVING
SMARTS is scalable RL training platform for autonomous driving (Zhou et al. 2020), providing closed-loop simulation in diverse traffic scenarios. In this paper, we control an ego vehicle (red) to drive through the scenario. The ego vehicle has two actions: accelerations (between + ms -2) and steering rate (between 1.5 radss for intersections and +0.7 for highways). Then the vehicle's motion is controlled by bicycle model (Gillespie) 2021). In the simulation, the vehicle
17

---

Published as a conference paper at ICLR 2026
only change its actions every 0.25 avoid oscillating trajectories. Note that settings more realistic than the original SMARTS. their default action spaces, the ego vehicle has infinite acceleration and can completely stop from the highest speed in 0.1s.
The three scenarios are illustrated in Figure C.3 For the intersection and the T-junction, the ego vehicle needs first pass an unsignalized to the right lane to reach the goal. For highway over-taking, the leading vehicle slow, and other vehicles can change their lanes arbitrarily. The ego vehicle needs overtake the slow vehicle and reach the goal on the same lane. All surrounding traffic vehicles are controlled by a set of predefined driving models with a distribution of inner parameters, providing diverse interactions.
and execute unprotected left turn, then change
Intersection
T T-junction
Highway Overtaking
Figure C.3: The three autonomous driving scenarios in SMARTS benchmarks. Arrows the entering lane of the ego vehicle, and triangles are goal positions. Highlighted green lanes are the "on-route" areas for the ego vehicle. White boxes are surrounding traffic vehicles.
The reward and cost design follows the minimalist principle:
(C.2) The first term is the travelled distance (in meters) within one decision step s0.25s). The second term +30 reaching the goal. The cost is 0 when staying safe. When collisions, off-road, driving on the wrong side of the road, or off-route happen, the cost is +10. The first three situations also trigger the termination of the episode.
R ddistance + Tgoal-
We hereby give short discussion about reward and cost design that might be useful for inter- ested readers. We actually tried many other different designs, but this simplest version works the best. The observed issues of other settings are summarized below:
Do not terminate the episode when an unsafe event happens: This is similar to the method used in Safety Dreamer's MetaDrive task (Huang al. (023). However, in intersection and T-junction scenarios, due the complexity of the road layout, the replay buffer is filled with meaningless, unsafe cases in the early stage of training. For example, when the ego vehicle drives off-road, it may stay there for a long time until the episode ends. This severely hinders policy learning.
Assign different costs to different unsafe events: Many RL studies on autonomous driving tasks, e.g., MetaDrive (Li et al. (2022), give higher penalty for severe events like col- lisions, and a smaller penalty for traffic rule violations. In our trials, we found that the agent tends to do "reward hacking" in such settings. For example, the vehicle will choose to drive off-road to get a lower penalty instead of learning how to avoid collisions. This -acking particularly severe when the vehicle needs solve the final potential collision, as is our case (restricting the acceleration and steering rate).
a series of actions to
Use risk field Surrogate Safety Measures (SSMs) costs: Using SSMs (Wang al. 202T), such as Time-to-Collsste (TTC) or risk field, to shape the reward is also a widely- used technique in RL-based autonomous driving. Our trials found that using TTC and
18

---

Published as a conference paper at ICLR 2026
the capsule risk field can indeed accelerate learning in the early stage. However, the final performance is worse than our simplest setting. One the possible reasons could be that these SSMs add inductive biases to safety. They focus on of unsafe (potential collision) cases. This may restrict the exploration power of RL. The simple end-oriented costs, in contrast, encourage exploring diverse and better solutions.
several specific types
Both policy and critic networks by default use the WayFormer (Nayakanti et al., (023) structure. For reward and cost critics, they share the torso and different MLP heads give multiple predictions of returns. Their network structures are briefly illustrated in Figure
Way Former Block
Action
+
Ego Vehicle State
Reward / cost network
Other Agents Trajactories
PLP-
Q-N
Maps
WayFomer Block
Policy network
Ego Vahicle State
Action Distribution
Other Agents Trajectories
Contraller
Figure C.4: The policy and critic network structure for SMARTS.
D HYPERPARAMETER SETTINGS
For on-policy baselines, we use the same IM step hyperparameter settings recommended by the OmniSafe benchmark platform (Ji al. 2024) for all experiments. Details provided their public webpage https:/ggttub..oo//PP KKK-S wwt.:oo-///.i/Aooeeeetttte Aiin-nineeet//omiiaat We did some modifications to make the training faster, which are available the open-source code.
For COX-Q, the implementation is based on SAC (Haarnoja et al., 2018). The shared parameters listed in Table D.I and the environment-speciff parameters listed in Table D.2 SACLag- UCB was implemented based on the SACLag method provided in OmniSafe. For CAL (Wu et al., 2024). use the same hyperparameters in the original paper, except for the randomized ensemble technique and the UTD ratio (1 in our experiments). It is useful to note that the original CAL paper uses UTD=20 in their experiments with densified cost signals (e.g., the Safe Velocity experiments, the cost is not 0/1 but the real-time speed). While in our 0/1 sparse cost settings, using high UTD will quickly make the policy over-conservative training, thus suppressing the maximization of return. This explains why we choose UTD=1.
The code of ORAC (MeCarthy et al. 2025) not available yet. For safe navigation tasks, we use the recommended hyperparameters in the ORAC paper in our implementation. While for Safe Velocity and SMARTS, we did not find a proper set of hyperprrameters for the original ORAC. The performance is quite unstable. Therefore, choose modify ORAC based quantile crities implementation. For all off-policy methods, we use the same discount factor and episode length listed in Table D.2 for consistency.
To accelerate the training for Safe Velocity and SMARTS, we use high number of parallel envi- ronments and lower offline update frequency, as recommended by Brax (Freeman al. 2021
19

---

Published as a conference paper at ICLR 2026
Table D.1: Shared oft-policy parameters
Parameters
Value)
Policy learning rate
3e-4
Critic learning rate
3e-4
Entropy learning rate
3e-4
Batch size
256
Maximum step length T)KL.
Tau
0.005
Convexification c in ALM
10
Number of cost critics
Number of reward critics
Table D.2: Environment-specifee off off-policy parameters
Parameters
Safe Velocity
Safe Navigation
SMARTS
Episode length
1000
400
240
discount factor
0.99
0,975
0.975
Episode cost limit
25
10
0.01
Number of parallel envs
64
128
Gradient steps
64
Policy update steps
64
Lagrangian initial value
1
0.001
Lagrangian learning rate
3c-4
5c-4
3e-4
Step length auto-tuning learning rate
le-4
le-4
NA
Initial steps
10240
5000
5120
Buffer size
1024000
1000000
512000
Policy network
256
256 2
Wayformer
Critic network
256 5
256
Wayformer
Layer Normalization
False
False
NA
Entropy auto-tuning
True
Falsc
True
Number of quantiles M
25
32
25
Truncation (kr, ke)
(2, 5)
(0, 0)
(1, 0)
Optimism (3., Be)
(4, 3)
(3, 3)
(3,
Cost CVaR a
13
16
Target update frequency
64
SUPPLEMENTARY RESULTS
The performance of COX-Q against on-policy baselines safe navigation tasks is presented in Figure E.1 Although they adhere to the cost constraints, the returns are significantly lower than off off-policy baselines.
Figure E.2 gives the percentage of triggered exploration gradient conflicts for the first 200K steps in the safe navigation benchmark using COX-Q. We see that the reward and cost objectives rarely conflict with each other (<. 10%); Additionally, as shown in Figure the policy starts from unsafe regions in the early stage of training, which will adjust the exploration step length to a small value. Therefore, the differences between ORAC and COX-Q are small. We hereby give two possible explanations: First, just like conventional multi-task learning. the gradient conflicts often happen between two loss functions with significantly different scales. However, for safe navigation, both reward and cost are on the same scale (0-30). Second, as both reward and cost are sparse signals (or least highly skewed), most exploration gradients near zero, making it highly stochastic.
20

---

Published as a conference paper at ICLR 2026
SafetyPolntGGol2
SafetyPointButton2
SafetyCarButton1
SafetyCarButton2
SafetyPolntPushl
25
20
Test
0
-25
10
-25
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1,0
0.0
0.5
1.0
TestEppost
40-
40
40
40
40
20
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
_ste (million)
env_step (million)
(million)
env_step (million)
snv__tep (million)
CUP
RCPO
PPOD
TRPOPID
COX-Q (ours)
Figure E.1: Training curves on-policy baselines and COX-Q for safe navigation tasks
Exploration gradient confiict ratio in the first 200K steps (Mean Std)
PointButton2
PointGoal2
CarButton2
PointPush1
Figure E.2: Exploration gradient conflict analysis for safe navigation tasks
F THE USE OF LARGE LANGUAGE MODELS (LLMS)
LLMs are used for polishing writing only, such selecting proper words.
21


Reviews:
[{"review_id": "Qvv00LsbzQ", "reviewer": "Reviewer_cj7m", "summary": "The paper tackles a key gap in off-policy safe RL: optimistic exploration improves sample efficiency but often violates cost constraints, because reward and cost gradients point in different directions and cost critics are biased. The authors propose a primal-dual off-policy algorithm that performs cost-constrained optimistic exploration, and adaptively shrinks the exploration step.", "questions": "Please see weaknesses.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper clearly identifies the off-policy safety pain point, i.e., optimism without cost control, and a concrete mechanism of policy-MGDA + cost-aware step-length selection to make OAC-style exploration safe.\n2. Both key steps are derived with closed-form solutions. The SMARTS results, though single-seeded, are non-trivial and show the method still works with large networks.", "weaknesses": "1. Safety still depends on cost-critic accuracy. The whole cost-bounded exploration, c.f. Sec. 4.2., assumes the lower/mean cost estimates are reliable. In sparse-cost navigation the paper shows underestimation, and COX-Q behaves like ORAC. A discussion for robustifying the cost critic is missing. I am curious how the authors would further improve this.\n2. The lemmas give local, per-step solutions, but there is no theorem that the whole off-policy procedure respects the CMDP constraint under function approximation and replay. However the abstract emphasizes controlled data-collection cost. I believe this gap should be made explicit.\n3. Some baselines are slightly weakened. CAL is run with UTD=1 but CAL's strength is precisely high UTD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6qv5I4T3cO", "reviewer": "Reviewer_EnSs", "summary": "The paper proposes COX-Q, an off-policy safe reinforcement learning algorithm that achieves efficient and safety-aware exploration by integrating optimism, gradient conflict resolution, and uncertainty estimation. Building upon the optimistic actor-critic framework, COX-Q introduces a cost-constrained optimistic exploration strategy that balances reward maximization with constraint satisfaction. It employs a Policy-MGDA mechanism to align reward and cost gradients, ensuring exploration occurs only in directions that improve performance without increasing risk, and an adaptive step-length controller to prevent constraint violations during updates. Additionally, COX-Q leverages Truncated Quantile Critics (TQC) to obtain uncertainty-aware and bias-reduced value estimates.", "questions": "My questions pertain to the weaknesses highlighted above:\n\n1. How does the proposed gradient resolution method for optimistic exploration differ from existing approaches in the literature?\n\n2. In the context of safe exploration, preventing constraint violations during early learning is critical. While overestimation of cost can promote safe exploration, it may also lead to overly conservative behavior. How does the proposed method better balance safety and exploration compared to existing baselines such as MICE [1]?\n\nReference\n[1] Gao et al., Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Exploration (2025)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles an important and timely problem by addressing the challenges associated with implementing off-policy reinforcement learning algorithms in the context of safe RL, where maintaining safety during exploration remains a key concern.\n\n2. The proposed framework integrates gradient conflict resolution with mechanisms to mitigate value estimation biases, effectively addressing both reward overestimation and cost underestimation—two critical issues that commonly affect off-policy safe RL methods.\n\n3. The paper presents a comprehensive empirical evaluation across diverse domains, including locomotion, navigation, and autonomous driving tasks, and benchmarks the proposed method against strong and relevant baselines.", "weaknesses": "[W1] Relevant Comparisons are Missing\n\nThe concept of gradient manipulation or resolution is not entirely new in Safe RL, as several works [1–3] have proposed similar approaches. To convincingly demonstrate the advantages of the gradient resolution method presented in this paper, it is essential to compare against these baselines. While some of these methods report results in the context of on-policy RL, their underlying ideas can be readily adapted to off-policy settings.\n\nRecent work [4] addresses the problem of cost underestimation, which is also a focus of our method. Therefore, it represents an important baseline for evaluating the effectiveness of our approach in mitigating cost underestimation.\n\n[W2] Lack of Ablations\n\nThe proposed method comprises two key components: (i) cost-constrained optimistic exploration via Gradient Resolution, and (ii) mitigation of value estimation bias via Truncated Quantile Critics (TQC). The paper would benefit significantly from ablation studies that quantify the contribution of each component individually. Specifically, it would be valuable to demonstrate:\n\n1. the performance of gradient resolution alone compared to prior works [1–3], and\n\n2. the effect of addressing value estimation bias via TQC compared to [4].\n\nSuch ablations would clearly establish the incremental value of each component and provide a stronger empirical justification for the proposed approach.\n\nReferences\n[1] Gu et al., Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation (2024)\n[2] Chow et al., Safe Policy Learning for Continuous Control (2020)\n[3] Liu et al., Constrained Variational Policy Optimization for Safe Reinforcement Learning (2022)\n[4] Gao et al., Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Exploration (2025)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sohFC9wdqD", "reviewer": "Reviewer_ARdC", "summary": "This paper introduces an off-policy primal–dual safe reinforcement learning (RL) algorithm named COX-Q, aiming to improve sample efficiency and reduce estimation bias. The method integrates a cost-constrained optimistic exploration strategy to adjust the trust region for safe exploration. Additionally, the authors propose to mitigate estimation bias through the use of truncated quantile critics.", "questions": "Given the major conceptual and presentation issues identified above, I do not have specific technical questions for the authors. The paper requires substantial clarification and restructuring.", "limitations": "", "rating": null, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "The empirical evaluations are comprehensive, with three applications.", "weaknesses": "1. The presentation of the theoretical results is weak. The paper does not clearly state the assumptions underlying the proposed method, leaving it unclear under what conditions the approach is valid or applicable. The policy’s action distribution is simply assumed to be Gaussian without justification. Moreover, Lemma 1 and Lemma 2 are rather trivial and do not appear to offer substantial theoretical contribution. If these are presented as lemmas, one would expect a main theorem or stronger result to follow—otherwise, the theoretical section lacks depth.  \n2. Even if the lemmas are accepted, the authors fail to explain how these results contribute to improving sample efficiency or mitigating estimation bias. While the empirical evaluations suggest some performance gains, it remains unclear under what circumstances the proposed method is effective. The framework does not appear to generalize well.  \n3. Although the experimental coverage is broad, the presentation quality is poor. For instance, no quantitative results are provided to support the claim of reduced estimation bias. In addition, the explanations are poorly organized and often lack logical coherence, which makes the empirical section difficult to follow.\n4. There is no pseudo code for clarification and implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)