Safety-Polarized and Prioritized Reinforcement Learning
Authors: Ke Fan
Jinpeng Zhang
Xuefeng Zhang
Yunze Wu
Jingyu Cao
Yuan Zhou
Jianzhu Ma
Venue: ICML 2025 poster (2025)
Keywords: Safe Reinforcement Learning
Primary area: reinforcement_learning

Abstract: Motivated by the first priority of safety in many real-world applications, we propose \textsc{MaxSafe}, a chance-constrained bi-level optimization framework for safe reinforcement learning. \textsc{MaxSafe} first minimizes the unsafe probability and then maximizes the return among the safest policies. We provide a tailored Q-learning algorithm for the \textsc{MaxSafe} objective, featuring a novel learning process for \emph{optimal action masks} with theoretical convergence guarantees. To enable the application of our algorithm to large-scale experiments, we introduce two key techniques: \emph{safety polarization} and \emph{safety prioritized experience replay}. Safety polarization generalizes the optimal action masking by polarizing the Q-function, which assigns low values to unsafe state-action pairs, effectively discouraging their selection. In parallel, safety prioritized experience replay enhances the learning of optimal action masks by prioritizing samples based on temporal-difference (TD) errors derived from our proposed state-action reachability estimation functions. This approach efficiently addresses the challenges posed by sparse cost signals.  Experiments on diverse autonomous driving and safe control tasks show that our methods achieve near-maximal safety and an optimal reward-safety trade-off.

Pafety-Pollrized and Prioritized Reinforcement Learning
Ke Fan
Jinpeng Zhang Xuefeng Zhang?
Yunze Wu Jingyu Cao
Yuan Zhou 415 Jianzhu Ma
Abstract
1. Introduction
Safety is a critical bottleneck for deploying reinforcement learning (RL) algorithms real- real-world applications due to the catastrophic consequences of unsafe decisions, such as crashes in autonomous driving (Leurent & Mercat, 2019), In such scenarios, safety takes precedence over all other objectives, and RL algorithms must prioritize achieving maximal safety (Gu et al., 2024; Garefa & Fernández, 2015). A widely adopted framework for safe RL is the Constrained Markov Decision Process (CMDP) (Altman, 1999; Achiam et al., 2017; Tessler et al., 2019; Ray al., 2019; Yang et al., 2022), which enforces safety by constraining the expected cumulative safety cost to remain below a predefined budget. While this framework provides a practical way to manage safety constraints, it is inherently limited to ensuring a user- specified safety level rather than learning the optimal safety cost. Furthermore, CMDP-based algorithms typically rely on dense cost signals to guide the agent's behavior, making them ill-su for scenarios where safety costs sparse but highly consequential. To address these limitations, work focuses on developing RL algorithms that maximize cumulative rewards while ensuring the unsafe probability is minimized.
Motivated by the first priority of safety in many real-world applications, we propose MAXS AFF, a chance-constrained bi-level optimization frame- work for safe reinforcement learning. MAXSAFE first minimizes the unsafe probability and then maximizes the return among the safest policies. We provide tailored Q-learning algorithm for the MAXS AFE objective, featuring novel learning process for optimal action masks with theoretical convergence guarantees. To enable the applica- tion of our algorithm to large-scale experiments, we introduce two key techniques: safety polar- ization and safety prioritized experience replay. Safety polarization generalizes the optimal action masking by polarizing the Q-function, which as- signs low values to unsafe state-action pairs, ef- fectively discouraging their selection. In parallel, safety prioritized experience replay enhances the learning of optimal action masks by prioritizing samples based on cemporal-difference (TD) errors derived from proposed state-action reacha- bility estimation functions. This approach effi- ciently addresses the challenges posed by sparse cost signals. Experiments on diverse autonomous driving and safe control tasks show that our meth- ods achieve near- maaimaaximal safety and an optimal reward-safety trade-off.
There are two primary methodologies in the literature for providing frequent safety guidance to learning agents: ac- tion correction and action masking, Action correction modi- fies unsafe actions by either replacing them with safe ones based on predefined backup policy or projecting them back to the safe set (Alshiekh et al., 2017; Zhang et al., 2023). However, this approach often leads to suboptimal re- wards due to the over-conservatism of the backup policy or projection. In contrast, action masking excludes unsafe ac- tions, allowing agents to explore safe options and potentially achieve optimal rewards while maintaining maximal safety, Despite its promise, designing effective action-maskinn pro- cedures remains challenging. For instance, Srinivasan et al. (2020) masks actions with unsafe probabilities exceeding a fixed threshold e. However, this uniform, state-agnostic threshold may fail differentiate actions with varying un- safe probabilities, limiting the policy's 's ability to learn the safest actions and compromising safety performance.
Eqqual contribution 'Department of Mathematical Sciences, Tsinghua University, Beijing, China IInstitute for Artificial Intel- ligence, Peking University. Beijing, China 'Department of Com- puter Science and Engineering, Shanghai Jiao Tong University, Shanghai, China 'Yau Mathematical Sciences Center, Tsinghua University, Beijing, China 'Beijing Institute of Mathematical Sciences and Applications, Beijing, China "Department of Elec- tronic Engineering, Tsinghua University. Beijing, China 'Institute for AI Industry Research, Tsinghua University, Beijing, China. Correspondence to: Yuan Zhou <'uannzhouu shai-zzooue @ tsinggua.nccu wdu.aeeduuu...... Jianzhu Ma <majianzhu @ tsinghua.cdu.cn..
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this work, we propose a novel level optimization framework, namely MAXSAFE for the
bi-

---

Safety-Polarized and Prioritized Reinforcement Learning
maximal-safety RL problem. In our framework, we mask the actions where the estimated unsafe probability is above state-dependent safety threshold ((s) that is learned from the interaction with the environment. Our estimated unsafe probability is the undiscounted safety cost of the optimal policy, which is more aligned with the definition of the unsafe probability compared to the discounted cost used in Srinivasan et al. (2020). This undiscounted safety cost is learned by our proposed state-action dependent reacha- bility estimation (namely the SA-REF backward induction), which is adapted from the techniques in Ganai et al. (2023). We show theoretically that our SA-REF backward induction converges to the unsafe probability of the optimal policy (Theorem 4.3). We then show that jointly learning the policy and the action mask rules (including the unsafe probability and the state-dependent safety threshold) yields the optimal policy (Theorem 4.4).
sible set, thereby enhancing safe policy optimization, as in RCRL (Yu et al., 2022a) and RESPO (Ganai et al., 2023).
Action-correection-asse Safe RL. The first line of works replace the detected unsafe action to a safe one instructed by a backup policy, e.g. a shielding policy via formal meth- ods (Alshiekh et al., 2017; Anderson et al., 2020), a re- covery policy (Thananjeyan et al., 2020) that recovers the agent back to safe states, an intervention policy (Wagener et al., 2021), safety editor policy (Yu et al., 2022b), etc. Another line of works project the unsafe action back to the safe set, c.g., the projection based on control barrier function (Cheng et al., 2019) with known dynamics, the Unrolling Safety Layer (Zhang et al., 2023), the Reduced Policy Optimization method (Ding et al., 2023), the Barrier Certificate method (Yang et al., 2023), etc. In general, the performance of the above methods might be sub-optimal due to the over-conservative nature of the backup policy or the projection operation.
To enable applications in large-scale safety-critical RL tasks, we propose a soft masking technique called safety polar- ization, based on a pre-selected polarization function fpol This technique reduces the Q-function values used for action selection under unsafe state-action pairs, with the decrement in Q-value determined by the unsafe probability and state- dependent threshold. To better address the sparsity of safety cost signals, we incorporate techniques inspired by Prior- itized Experience Replay (PER) in Q-learning (Schulman et al., 2017), originally designed for sparse reward signals. Specifically, we use the TD error of our proposed SA-REF to prioritize samples in the replay buffer. This prioritiza- tion enables the algorithm to focus safety-critical transi- tions. It enhances the learning of unsafe probabilities and improves the safety performance of the algorithm. We con- duct extensive experiments on autonomous driving and safe control tasks, demonstrating that our proposed algorithms, SPOM and SPOM PER, achieve superior safety and the best reward-safety trade-off among state-of-the-ar safe RL methods (Section 6).
Action Masking for Safe RL. A large body of works use specific assumptions or prior knowledge to build action masks (Krasowski et al., 2023) for safe RL. Fulton & Platzer (2018; 2019) construct the action masks based on theorem proving of differential dynamic logic specifications (Platzer, 2008), which require the knowledge of the system dynam- ics. Kalweit et al. (2020) propose Constrained Q Q-Learning where the action masking is done via a set of one-step cost constraints that query the dynamics to check whether the next state inside the constraints. Huang & Ontanón (2022) give a more detailed analysis of the effects of action mask- ing under the context of policy gradient algorithms. For specific tasks, there are works studying action masking for autonomous driving (Mirchevska et al., 2018; Brosowsky et al., 2021; Krasowski et al., 2022) and traffic light con- trol (Muller & Sabatelli, 2022) to ensure safety. Most of the above works require domain knowledge that not al- ways accessible. In terms of learning approach, the typical example is the Safety Q-functions for RL (SafeQ) proposed in Srinivasan et al. (2020) and also in Tan et al. (2024), which is then extended to other settings, e.g., safe explo- ration (Bharadhwaj et al., 2021). As we will show, SafeQ might fail to achieve the minimal unsafe probability.
2. Related Work
Constrained RL by CMDP. The classical framework ad- dressing safety constraints in RL is the CMDP (Altman, 1999), which aims to maximize the expected reward while keeping the expected cost below a predetermined threshold. Primal-dual approaches use Lagrangian relaxation to trans- form the original constrained optimization problem into unconstrained one, as in PPOLag (Ray et al., 2019) and RCPO (Tessler et al., 2019). Trust region methods perform local policy improvements within the constrained region, such as CPO (Achiam et al., 2017), FOCOPS (Zhang et al., 2020), and CUP (Yang et al., 2022). Recent studies have incorporated Hamilton-Jacobi reachability (Bansal et al., 2017) into the CMDP framework to identify the largest fea-
Hamilton-Jacobi Reachability and Reachability Estima- tion Function. Early works employ Hamilton-Jaobi(HH)) reachability value functions to assess state feasibility, re- lying on known system dynamics and numerical meth- ods (Ganai et al., 2024). Some studies approximate un- known dynamics using Gaussian Processes (Zhao et al., 2023) or symbolic regression (Wang & Zhu, 2024). Once reachability is computed, the state space is partitioned into feasible/ifeaaibbee regions to guide policy optimiza- tion (Zheng al., 2024). Other works focus on formal safety verification of DRL systems (Dong et al., 2024). However,
2

---

Safety-Polarized and Prioritized Reinforcement Learning
HJ-based value functions are not well-suited for stochastic MDPs during RL training. To address this, reachability es- timation function(REF) estimates unsafe probabilities via backward reduction. Our work extends REF to the state- action level, enabling state-dependent action masking to reduce safety violations.
ing state, meaning that the episode will terminate once the agent enters this region. This assumption imposes stricter safety constraint, motivated by the critical importance of safety in practical applications. The overall goal of our pro- posed MAXSAFF framework consists of chance-constrained bi-level objective which maintains unsafe probability as low as possible while maximizing the return:
Prioritized Experience Replay for Off-policy RL. Prior- itized Experience Replay (PER) widely adopted tech- nique in RL., designed to address the inefficiency of uniform sampling in replay buffers. Initially introduced by Schul- man et al. (2017), PER assigns priorities to transitions based on their TD errors, enabling agents to learn more effectively from iigh-impact experiences. Variants of PER have been explored, such as annealing the prioritization exponent over time (Horgan et al., 2018) and incorporating multi-step re- turns into the prioritization process (Hessel et al., 2018) to improve stability and learning efficiency. In our work, rather than applying prioritization to the learning of the Q-function, we tailor the prioritization process to the learning of our pro- function, addressing the challenges posed by sparse and critical cost signals.
max Ess (80)]
s.t. T E arg min
Pr Somop.
[s,, T 8t E S.).
(T.
We assume that there is sufficiently large policy space with minimum unsafe probability, e.g., zero unsafe probability. Such environments are common in autonomous driving and robotic control, for instance there are many possible driving policies that allow a vehicle to operate safely on the road without collisions. Also, this assumption is already explored in the literature, e.g., Ganai et al. (2023), which assumes the existence of safest policies with zero cost.
posed
Let nT" be an optimal policy of MAXS AFE objective (2) and let ((s) denote the unsafe probability at state s under n², which in fact minimal,
3. Preliminaries
In safe RL, most of the algorithms are designed un- der the formulation of the Constrained Markov Decision Process (CMDPs), which is defined as a tuple M = (S,A,P, P,T.P, N, 7, p). where S is the state space, A is the action space, P S A - A(S)) is the transition prob- ability function, : S A - TR is the reward function, c: S x A + RR+ is the cost function, 7 E (0, 1) is the dis- count factor and p is the initial state distribution. Typically, the reward value function and the cost value function are de- fined respectively as V.7T (s) P) En ('((,, at)] and (s) = E, Ee 7f((st,a)) a.),, where т = {+o = 8, dg 81, ".. ~ (T, P) denotes a trajectory starting from under the given policy æ and transition function P. The goal is to find policy π: S - A(A) that maximizes the expected discounted cumulative reward while ensuring that the cost value function remains below predefined safety budget d. Formally:
((s) = Pr T~
Es ET::T: E Salso = s).
(3)
Then we define the optimal action mask Cc(s) for MAXSAFE as the set of safest actions at state 8
Cç(s) (b Al
Pr TNoTT P)
Bst T Su = 8,
d0 bj S ((5)
(4)
For tabular MDPs, Cs(s) is optimal in the sense that, if we know (s). we can define the Bellman operator B₂
RISIMI + R || with optimal action masking Cc.
(tQ(s.a)=r(s... + +E max d'ECc(s))
Q(s',a') (5)
which can be iteratively applied to find an optimal policy under MAXSAFE framework, as shown in the following lemma:
max E p V (so)
Lemma 3.1. The following results hold:
s.t. E, o VVT (so) < d.
(1) (Contraction). The Bellman operator (5) is ? contraction and thus has unique fixed point, which is denoted by Q22
The CMDP framework ensures agents operate within pected safety budget, but it may fall short in safeta- scenarios where a single unsafe action can cause catas- trophic failure. This necessitates a stricter safety-oriented framework.
(2) (Safety Optimality). For all E S, we define n [(())= == arg max (s) Q2(ss, a) to be the corresponding pol- icy induced by Q²- Then the safety of nद is optimal: के E arg min, ( E T s E &.)).
We introduce S, S to be the set of unsafe states. In this paper, we assume that any unsafe state s. S, is an absorb-
(3) (Reward Optimality). Reward of nr is optimal among the safe policies: Eso~p(V* (V*)(s) = Esgep V (sa))
(BB denotes the set of probability distributions over set B.
3

---

Safety-Polarized and Prioritized Reinforcement Learning
The proof of Lemma 3.1 can be found in Appendix B.I. Grounded by Lemma 3.1, in the following section, we focus on designing an algorithm that learns (s(s) (5) on the fly which then guides the Q- Q-function to select the action with optimal reward among the safest actions and thus ensure maximal safety and reward to solve our MAXSAFE objective (2).
Theorem 4.3. (Optimal SA-REF backward induction) For tabular MDPs, with tho = 0, the update
01t1 mass E Ss) min Ve(s', a') a')}
a'EAA
(10)
will converge to the optimal SA-REF o' as +o0.
The proof of Theorem 4.3 is deferred to Appendix B.3.
4. Learning the Optimal Action Masks
In this section, we focus on learning state-dependent optimal action masks (s). Ganai et al. (2023) proposed reacha- bility estimation function (REF) to capture the probability of constraint violation at state s under a given policy. In our work, we extend the definition of REF to be state-action dependent. As shown in Section 4.1, this extended REF can serve as an estimate of the unsafe probability for a given state-action pair. It is then utilized to construct our action masks, forming the foundation of our learning solution to the MaxSafe objective (2).
4.2. Safe Q-Learning with Optimal Action Masks
With Theorem 4.3, we provide an update of thy the finite- sample form and incorporate this update with the Q-learning algorithm to obtain safe Q-learning algorithm with theo- retical convergence- torpreencceto-ooimmltyy guarantees.
The s update, starting from thu = 0, goes as follows
thet1 1 az) (1 (1) ae)+
(11)
Br max{1l(s E Sul. min (e d')},
a'EA
where 0- < B, S 1 is the step size, each a, ~ TT(-(s)))) is sampled from a behavior policy gy. Following Equation (9) we then define the learned optimal action masks as
4.1. State-Action Reachability Estimation Functions
Definition 4.1. Given a policy we define the state-action reachability estimation function (SA-REF) vith S x A +
[0, 1] as follows,
CG (s) = {6 A () S (((s)),
(12)
t (8,a) a)= = B max #EET
I[se E &Su] 50 = =,
where for a small enough constant K > 0, (((s) == minge.A the a) + K. Here x is needed in order to tol- erate the stochastic approximation error during the learning process. Note that the threshold function (e(s) is state- dependent, which is the key for the action masks Ca, to optimality.
(6)
(s, represents the unsafe probability at state 8 and action under the current policy n, that is, o (s,
Prys(s,P) p) [3g E 8 Su/so- 8, ao = a]. Expressing in the form of Equation (6) provides convenient way to compute x= (s, a) via backward induction.
We now propose our safe Q-learning update with optimal action masks:
Lemma 4.2. We have the following Bellman backup for the SA-REF :
Qc+1(s;, (1- (s,,
Br(r(s), +7 max ('EC)
Qe(sc+1, a')),
" (s,a) max{[[(s E Sul,
(7)
(13)
Eg Py)a((1 (1s) 1 (s', a'),
where the difference from the original Q-lerrnigg algorithm is that the action maximizing Q at the next state s' is chosen only from the learned optimal action mask during iteration as Cc, (s'). Now the learned policy is of the form
The proof of Lemma 4.2 is in Appendix B.3. For conve- nience, we define the optimal SA-REF 30" : x²" where T²' is our optimal policy of MAXSAFE. Based on our SA-REF formulation, we can rewrite the definition in Equation (3) as
Tee (s) == arg max Qs(s. aEC..
(14)
<(s) Ea-x" (s) [w* (s, a)] = min dh (s, a)
(8)
aEA
The following convergence analysis is based on the fact that Ca, (s') will be the same as the optimal Cs()') for large enough and after that the Q-update (13) will stably converge.
since n² reaches the minimum unsafe probability, and the optimal action mask becomes
Theorem 4.4. For tabular MDPs, suppose the following conditions hold:
Cs(s) = (b E A Jه (s. < ((s)
min 16 a). (9)
aeA
Therefore, to find the optimal action mask Cç, we would like to learn the function vh* Fortunately, this can be done via backward induction by mimicking x* to take actions with the minimum unsafe probability as follows.
(1) each state-action pair (8, a) is infinitely visited;
(2) the step size sequence 1.e satisfies < Be S
and Eizo :: +00, E120 B3 < +00;

---

Safety-Polarized and Prioritized Reinforcement Learning
Algorithm Safety-Polarized Optimal action Masks with Prioritized Experience Replay (SPOM_PER)
5. Deep Q-Learning with Safety Polarization and Safety Prioritized Experience Replay
Require: a polarization function fpot. safety prioritiza- tion exponent a, importance sampling exponent 0
In this section, mainly focus on how we combine modern deep Q-learning algorithms to solve practical safety-critical RL tasks. In Section 5.1, we focus on a practical implemen- tation that transfers our proposed masking strategy using the polarization function. Section 5.2, we adopt a tech- nique inspired by prioritized experience replay to further help maximize safety in long-horizon sparse cost signal scenarios.
2: Initialize: safety prioritized replay buffer D, Q-
network Q, vi-network v,, along with target networks Q and z
3: for each time step do
4: Take action at based on ((8) (cf. Equation (17)) combined with any exploration strategy, e.g., 6- greedy
5: Store the collected sample (st, Te, 81+1) into D with maximal priority Pt maxict.
5.1. Safety Polarization for Q-functions
6: for each update step do
To fully implement our learned action masking in deep RI algorithm, we define the gating operator
7:
Sample a mini-batch of data (st, as, Tú, 11+1) from
replay buffer according to their priorities ~ P(i)
(cf. Equation (19))
0, x S "
(T(x)
8: Compute importance sampling weight based wy (ef. Equation (20))
1. otherwise.
Our action masking Cç. (s) defined in Equation 12 can be viewed as adding -oo to Q2 at the masked action while adding O to Qr at other action. Specifically, for actions where (s) (1b(s, a))
9: Compute
0,41
arg max ((s++1:0) + Spol(1 (1- + ($1+1, aEA
we penalize Q. during training, We implement the following class of polarization function combine with the learned Q-function. Formally, we define the polarization function class as follows:
10:
Compute Q targets y = r. + (Q(+(11(( apt1)
11:
Minimize MSE loss between Q(s;, ai) and y
12:
Compute _ = arg m (8)+11
Definition 5.1. Define the polarization fittriitaionuunctiinn class Fpol as the set of all polarization functions fpol [0, 1] +
Compute d targets
[-oc, 0) which is monotonically increasing function satis- fying fpol (1) = 0, (pa) == lim fpo -00
(5++ y max [I[s; E T6),
and TD errors & = y (s,, a:)
Examples of polarization functions include e- -oo(o))) 1- for a constant value 0. Together with the gating operator r, and our learned optimal action mask Cz, (s), we derive our learned MaxSafe policy can be rewritten
14:
Minimize weighted MSE loss between (sss, a.)
and y based on we (cf. Equation (20))
15:
Update transition priority p; based on 3D-TD errors
16:
Update target networks Q and e
ग (s) = arg max (s( aEC₂(#)
17:
end for
18: end for
arg max aEA
(Qi(s, + - ((()-
(16)
During implementation, we further get rid of the hard mask- ing rule induced by the gating operator I, and compose the Q-function Q: with the SA-REF uh, through f follows:
(3) K > 0 is small enough to identify the gap between (s) and A \, Cc(s), i.e. Ab e Cç(s), A A Cs))),
(s(s) = arg max Qe a) + f(1 ve(s, (17) aEA
3 + 2x < (8,e),
(15)
Intuitively, a state-action pair (s, a) is measured to be unsafe, (x a) will close to 0, then f(1 - (, a)) will be close -o and the action is likely not to be chosen state s since Qr a) + fpa (1 (s,s is very low. Therefore the actions below the state-dependent thresh- old (s)) are more likely to be selected. The choice of the polarization function fpol provides a balance between explo- ration and safety: we focus more on safety if fpo converges faster to -o as x + 0+,, we encourages more exploration otherwise.
then the the update in Equation (11) will converge to the optimal SA-REF 1b",, and our safe Q-learning update in Equation (13) will converge Q² Thus, the learned pol- icy (s)ss) will converge an optimal policy nदद under our MAXSAFE objective (2).
The proof of Theorem 4.4 is deferred Appendix B.4.
5

---

Safety-Polarized and Prioritized Reinforcement Learning
5.2. Safety Prioritized Experience Replay
5.3. Practical Implementation
Prioritized Experience Replay (PER) enhances off-policy RL by assigning priorities to transitions based their TD errors, emphasizing those with the highest learning potential. This approach improves sample efficiency and accelerates convergence, particularly in scenarios with sparse reward signals. In our work, to address the challenge of sparse cost signals, we leverage PER to enhance the learning of the y network. The motivation behind this approach is similar to that of PER in traditional Q-learning, where transitions with larger TD errors are prioritized to accelerate learning. Furthermore, importance sampling is employed to mitigate the distribution shift introduced by the prioritization process.
For large-scale tasks with discrete action space A based on DQN (Mnih et al., 2015), we use neural networks Q and & to approximate the optimal Q-function and the SA-REF, respectively (the corresponding target networks are Q and b). We select the Sigmoid function as the activation for the output layer of the v-nntwork to ensure that its output remains bounded within the range [o, 1]. The pscudo-code is presented in Algorithm 1.
When computing the targets for Q and db updates, use a double---learming-style implementation (Van Hasselt et al., 2015). Specifically, at lines 9 and 10 of Algorithm 1, the target for Q is computed by querying the action o which combines the max Q-value with the safety polarization func- tion penalty at the next state S1+1- This is done using the current policy æ, composed of the current Q and v. Sim- ilarly, the target for tp (computed at lines 12 and 13 of Algorithm 1) follows the same principle. Note that for the eb target at line 12, we multiply R by the discount factor to reduce the fong-term variance in v backward induction. This approach is analogous to the multiplication of Q by in DQN.
Specifically, we introduce a prioritization technique tailored for the network. The priority of transition i is deter- mined by the TD error, defined as
6, = [[ss, E S,l, min (. Ve(ss a:),
aEA
(18)
where the priority p; is calculated as p; = =16,1 + e. Here, is a small positive constant added to ensure that transitions are not excluded from sampling when their error becomes zero. The sampling probability for each transition is then computed using the prioritization heuristic:
For the polarization function fpol used in our experiments, we select 10 (og(())) which demonstrates the best empirical performance. Additionally, we conduct an ablation study to analyze the effects of different polarization functions (see Section 6.3).
= P
P(1)
(19)
ΣN
PE
where a E [o. 1] controls the degree of prioritization. Unlike traditional PER in Q-learning, our prioritization scheme is explicitly tied to the learned & network, reflecting the specific requirements of safety-critical scenarios.
6. Experiments
6.1. Experiment Setup
Benchmarks. Our evaluation adopts the following four tasks: TwoWay, Merge, Roundabout, and Intersection. These tasks are from the highway-env environment (I (Leurent, 2018; Leurent & Mercat, 2019), designed for simulated au- tonomous driving with diverse objectives that require intri- cate behaviors to safely achieve the corresponding goals. Additionally, our evaluation includes classical safe control tasks such as Adaptive Cruise Control (ACC) (Anderson et al., 2020) and Circle (Achiam et al., 2017). We highlight that the cost functions used for the CMDP-based methods are designed as = 1 when a crash happens, and 0 oth- erwise. More details about the environments, e.g., state spaces, action spaces, reward functions, etc., can be found in Appendix C.I.
Since safety prioritized experience replay introduces sam- pling bias by altering the transition distribution, we correct this bias using imporance-samplinnn (IS) weights during the s update. The IS weight for a sampled transition i is defined as:
1 1 N
wi
P(1)
(20)
where N is the size of the replay buffer, and E [] controls the degree of correction. Then the update of our vb network will become
(1 az) (1 we wrr (81, w,B, max{ E Su], min (e(st41,a))) d')}. d'eA
Baselines. Our base unconstrained RL is DQN which, in our implementation, uses the double Q-learning tech- nique (Van Hasselt et al., 2015) by default. Other safe RL baselines include the following:
(21) experience replay, our
By incorporating approach effectively enhances the learning of the & network, particularly in environments with long- long-horizon tasks and sparse cost signals. This integration improves both safety and sample efficiency.
Reward Shaping. The reward is shaped to -10 when
6

---

Safety-Polarized and Prioritized Reinforvement Learning
Table SPOMPPPPE achieves the best safety-reward tradeoff in terms of SWU score, while SPOM performs the best among the remaining safe-RL RL baselines.
SWU Score 1
TwoWay
Merge
Roundabout
Intersection
ACC
Circle
Overall
SPOM_PER (ours)
0.98
0.94
1.22
0.68
1.07
0.96
0.98
SPOM (ours)
0.88
0.95
0,40
0.96
0.87
0.37
0.74
SafeQ
0.73
0.84
0.63
0.75
0.50
0.15
0.60
Recovery
0.34
0.66
0.81
0.66
0.85
0.25
0.60
RCDQN
0,41
0.81
0.48
0.57
0.46
0.14
0.48
RewsDQN
0.33
0.92
0.48
0,50
0.47
0.11
0.47
DQN
0.37
0.56
0,48
0.58
0.46
0.13
0.43
205
Dprecee
2.
0.3
0.00
Skop
Margs
dabbut
Interastion
0.4-
044
0.5
6.4
0.
3.0
165
340
245
Figure Training curves SPOM, SPOM_PER and baselines the four tasks from highway-env The æraxis represents the number steps, and the y-axis: the first row is episode reward (higher is better); the second row is safety measured in crash rate (lower is better). All experiments are run over 6 random seeds and the shaded areas are 95% confidence interval.
a crash occurs, which is a strong penalty, and we label the reward-shaped DQN variant as RewsDQN.
common CMDP-based RL is built on an actor-critic frame- work. For completeness and clarity of presentation, we add additional CMDP- approaches RESPO (Ganai et al., 2023), PPOLag (Ray et al., 2019), which use Proximal Pol- icy Optimization (PPO) (Schulman et al., 2017) as the base RI algorithm, in Appendix C.3, where we find that they fail to optimize under our environments since the cost signal corresponding to a crash is very sparse. The detailed imple- mentations and hyperparameters of these baselines and our algorithm SPOM and its safety prioritized experience replay version SPOM PER are provided in Appendix C.2. Eval- uation Metrie. To clearly quantify the trade-off between rewards and safety, we follow Yu et al. (2022b) and com- pute the seigtt-weighted--util utiilty (SWU) score as the final evaluation metric. The SWU score is defined as:
Direct e-Masking, This is the SafeQ (Srinivasan et al., 2020) as discussed before, which masks out the actions whose estimated unsafe probability by a safety critic is greater than The original actor-critic based algorithm in Srinivasan et al. (2020) is modified be DQN- based.
Recovery RL. Originated from Thananjeyan et al. (2020), labeled Recovery here, this corresponds to the action correction approach, where task policy selects action, but if the estimated unsafe probability from safety critic is greater than s, the action will be corrected by a learned recovery policy. The task policy and recovery policy are implemented via DQN.
CMDP-based RL. We choose the reward constrained approach in Tessler et al. (2019) which constrains the reward through tiplier and e the cost signal corresponding to crash. Then use DQN to learn upon this constrained re- ward. We denote this baseline by RCDON.
SWU == min (1. UnsafcRateTargee
Utility
UnsafeRate
Utility BaseRL
Ac, where À the Lagrange mul-
where BaseRL refers the unconstrained RL algorithm DQN in experiments. We choose the Utility be the episode reward, and UnsafeRate is the crash rate. The UnsafeRate Target is chosen as the minimal crash rate among the compared methods for each environment, since we aim at achieving the maximal safety. To reduce variance, Utility, UnsafeRate and UnsafeRate are averaged
For SafeQ and Recovery, the parameter& E is chosen to be 0.1 throughout experiments. Note that the above baselines are implemented via DQN for fair competition, while the

---

Safety-Polarized and Prioritized Reinforcement Learning
lel
Acc
1e2
Circle
8-
7-
poisode Reward
6-
5
SPoM
SPOM PERlous)
RowSDON
3-
DON
2-
RcbON
Recovery
SaleQ
0.2
0.4
0.8
0.0
0.4
0.8
Step
1e6
Step
1e6
Acc
16-1
Circle
SPOM)uu))
8-
Palliors)
BeweDON
8
DON
Crash Rate
REDON
6-
Recovery SafeO
Crash
2
0.4
0.2
0.4
0.6
1.0
Step
1e6
Step
1e6
Figure Training curves SPOM, SPOM PER and baselines the classical safe control tasks: ACC Circle. The r-axis represents the number of steps, and the y-axis: the first row is episode reward (higher is better); the second row is safety measured in crash rate (lower better).
over the last 1 training steps (Yu et al., 2022b).
SPOM PER relatively small in these tasks, which we at- tribute to the shorter episode lengths in the highway environ- ment, reducing the impact of the prioritization mechanism. Compared to baselines, SafeQ performs moderately well in Merge and Intersection but struggles to achieve competitive SWU scores in other tasks, failing to effectively balance safety and rewards. Methods like Recovery and RCDQN exhibit more conservative behavior, achieving lower crash rates the cost of significantly reduced rewards. DQN and RewsDQN, on the other hand, fail manage safety effectivelyy resulting in much lower SWU scores across all tasks.
6.2. Main Results
First, our proposed algorithms, SPOM PER and SPOM, consistently achieve the best performance across all tasks, demonstrating their effectiveness in balancing safety and rewards. As shown in Table SPOM PER achieves the highest average SWU score of 0.90, followed by SPOM with score of 0.79, outperforming all other baselines by significant margin.
For the classical safe control tasks, ACC and Circle, SPOM_PER, thanks to the prioritization mechanism in the replay buffer for the learning of the & network, achieves significantly lower crash rates compared other methods while ensuring convergence to higher episode rewards. In comparison, SPOM also performs competitively in ACC and Circle but slightly lags behind SPOM PER in terms of crash rate reduction. Baselines such as Recovery perform reasonably well in ACC, but their inherent conservativeness leads to suboptimal rewards, and they struggle significantly in Circle.
Overall, the results validate that SPOM PER and SPOM are highly effective in optimizing the trade-off between safety and rewards, with SPOM excelling in long- horizon tasks and maintaining the highest overall SWU score.
6.3. Ablation Studies
We conduct ablation studies on the usage of different po- larized function fpo and apply the optimal action masks directly in Intersection and TwoWay. The results shown in Figure 3. Observe that, the direct optimal action masking "OAM" is significantly conservative in Intersection, while in TwoWay, its crash rate has little improvement but reward is lower. This shows the hurt brought by optimal action masking on an immature & network in the early stage of training, which is also the reason of suboptimality of the strong polarization effect given by "xp",, since masking is
For the four tasks from highway-env, both SPOM and SPOM_PEL demonstrate leading performance. SPOM PER achieves the highest SWU scores in Merge and Roundabout, while SPOM leads in TwoWay and In- tersection. The performance gap between SPOM and
*Code for the experiments is available at https: thub. com Ftt//FrankSiioo wat.esaiinaaaaaaaaaaa.aa wtw.aaattrraaaaaaaaaaaaaaaaaeeee Sat./aafffttt/ Pt..et..yy............
8

---

Safety-Polarized and Prioritized Reinforvement Learning
Intersection
1e1
TwoWay
Episode Reward
Episode Reward
4-
2.0
3
10log
0.8
log
2
xP
OAM
0.2
0.2
0.4
0.8
1.0
Step
le5
Step
1e5
1e-1
Intersection
1e-1
TwoWay
Crash Rate
Crash Rate
2-
8.0
0.2
0.4
0.6
1.0
Step
le5
Step
1e5
Figure 3. Ablation studies, where "log", "xp" represent using polarization function log(æ) and 1 xt respectively, "10 log" is our default choice log(z), and "OAM" means applying optimal action masks directly. The z-axis represents the number of steps, and the y-axis: the first row is episode reward (higher is better); the second row is safety measured in crash rate (lower is better).
special case of safety polarization. Finally, for "log" its po- larization effect is weak and thus cannot be safe enough like our default "10 log". The same phenomena can be observed in other tasks. Full details are presented in Appendix C.3.
lage Science and Technology City Key Technology funding, and Wuxi Research Institute of Applied Technologies.
References
Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained policy optimization. In International Conference on Ma- chine Learning (ICML), 2017.
7. Conclusions
We propose MAXSAFE, novel chance-constrained bi-level optimization framework for safe RL, motivated by the prior- itization of safety in real-world applications. We address the MAXS AFE objective by learning optimal action masks and introduce the technique of safety polarization as practical generalization optimal action masks. Additionally, we propose -afoty-prioritized experience replay, designed accelerate the learning of optimal action masks, especially when cost signals are sparse. Extensive experiments demon- strate that our method achieves an optimal trade-off between reward and safety, delivering near-maximal safety.
Alshiekh, M., Bloem, R., Ehlers, R., Könighofer, B.,
Niekum, S., and Topeu, U. Safe reinforcement learning via shielding. In AAAI Conference Artificial Intelli- gence (AAAI). 2017.
Altman, E. Constrained Markov Decision Processes. Chap- man and Hall/CRC, 1999.
Anderson, G., Verma, A., Dillig, I., and Chaudhuri, S. Neu- rosymbolic reinforcement learning with formally verified exploration. In Neural Information Processing Systems (NeurIPS), 2020.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Bansal, S., Chen, M., Herbert, S., and Tomlin, C. J. Hamilton-jacobe reachability: brief overview and re- cent advances. In IEEE Conference on Decision and Control (CDC),
Acknowledgments
Bharadhwaj. H., Kumar, A., Rhinchart, N. Levine, S., Shkurti, F., and Garg, A. Conservative safety critics for exploration. In International Conference on Learning Representations (ICLR), 2021.
This work was supported in part by the National Natural Science Foundation of China Grant 52494974, China's Vil-

---

Safety-Polarized and Prioritized Reinforcement Learning
Borkar, V. S. Stochastic Approximation: A Dynamical Systems Viewpoint. Cambridge University Press, 2008.
Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., van Hasselt, H., and Silver, D. Distributed priori- tized experience replay. In International Conference on Learning Representations (ICLR), 2018.
Brosowsky, M., Keck, F., Ketterer, J.. Isele, T., Slicter, D., and Zöl J. M. Safe deep reinforcement learning for adaptive cruise control by imposing state-sper sppece-seiiicc safe sets. In IEEE Intelligent Vehicles Symposium (IV), 2021.
Huang, S. and Ontanón, S. A closer look at invalid action
masking in policy gradient algorithms. In International Florida Artificial Intelligence Research Society Confer- ence (FLAIRS), 2022.
Cheng, R., Orosz, G., Murray, R. M., and Burdick, J. W. End-to-end safe reinforcement learning through barrier functions for safety-critical critical continuous control tasks. In AAAI Conference on Artificial Intelligence (AAAT), 2019.
Kalweit, G., Huegle, M., Werling, M., and Boedecker, J. Deep constrained q-learning, 2020. URL https: a rexiv.org/ae 2003.000322000 09398.
Ding, S., Wang, J., Du, Y., and Shi, Y. Reduced policy optimization for continuous control with hard constraints. In Neural Information Processing Systems (NeurIPS), 2023.
Krasowski, H., Zhang, Y., and Althoff, M. Safe reinforce- ment learning for urban driving using invariably safe braking sets, In International Conference on Intelligent Transportation Systems (ITSC), 2022.
Dong, Y., Zhao, X., Wang, S., and Huang. X. Reachability verification based reliability assessment for deep rein- forcement learning controlled roboties and autonomous systems. IEEE Robotics and Automation Letters, 9(4): 3299-33000 -306, 2024.
Krasowski, H., Thumm, J., Müller, M., Schäfer, L.,, Wang, X., and Althoff, M. Provably safe reinforcement learning: Conceptual analysis, survey, and benchmarking. Trans- actions on Machine Learning Research, 2023.
Fulton, N. and Platzer, A. Safe reinforcement learning via formal methods: Toward safe control through proof and learning. In AAAI Conference on Artificial Intelligence (AAAI), 2018
Leurent, E. An environment for autonomous driving decision-making, 2018. URL https://github. com/eleurent/ silleurenn/highwwy-eeet
Fulton, N. and Platzer, A. Verifiably safe off-model rein- forcement learning. In International Conference on Tools and Algorithms for Construction and Analysis of Systems (TACAS), 2019.
Leurent, E. and Mercat, J. Social attention for autonomous Decision-making in dense traffic. In Machine Learning for Autonomous Driving Workshop at NewrIPS, 2019.
Mirchevska, B., Pek, C., Werling, M., Althoff, M., and Boedecker, J. High-level decision making for safe and reasonable autonomous lane changing using reinforce- ment learning. In International Conference on Intelligent Transportation Systems (ITSC), 2018.
Ganai, M., Gong, 7., Yu, C., Herbert, S. I.., and Gao, S. Iter- ative reachability estimation for safe reinforcement learn- ing. In Neural Information Processing Systems (NeurIPS), 2023.
Ganai, M., Gao, S., and Herbert, S. L. Hamilton-jacobi reachability in reinforcement learning: A survey. IEEE Open Journal of Control Systems, 3:310-324, 2024.
Mnih, V., Kavukeuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M. A., Fidje- land, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. Human-level control through deep reinforcement learning. Nature, 518:529-533, 2015.
Gareía, J. and Fernández, F. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16:1437 16:144771480, 2015.
Gu, S., Yang, L., Du, Y., Chen, G., Walter, F., Wang, J., Yang, Y., and Knoll, A. A review of safe reinforcement learning: Methods, theory and applications. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 46 (12):1 1216-11235 2024.
Muller, A. and Sabatelli, M. Safe and psychologically pleas- ant traffic signal control with reinforcement learning us- ing action masking. In International Conference on Intel- ligent Transportation Systems (ITSC), 2022.
Platzer, A. Differential dynamic logic for hybrid systems.
Hessel, M., Modayil, J., van Hasselt, H., Schaul, T., Ostro- vski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
Journal of Automated Reasoning, 41:143-189, 2008
Puterman, M. L. Markov Decision Processes: Discrete
Stochastic Dynamic Programming. John Wiley and Sons, Inc., USA, 1st edition, 1994, ISBN 0471619779.
10

---

Safety-Polarized and Prioritized Reinforcement Learning
Ray, A., Achiam, J., and Amodei, D. Benchmarking safe ploration in deep reinforcement learning. In International Conference Machine Learning (ICML), 2019.
Yu, H., Xu, W., and Zhang, H. Towards safe reinforcement learning with safety editor policy. In Neural Information Processing Systems (NeurIPS), 2022b.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algo- rithms, 2017. URL https://arxx org/ abs/ 1707.06347
Zhang. L.,, Zhang, Q., Shen, L.,, Yuan, B., Wang. X., and Tao, D. Evaluating model-free reinforcement learning toward safety-critical tasks. In AAAI Conference on Artificial Intelligence (AAAI), 2023.
Zhang, Y., Vuong, Q., and Ross, K. First order constrained optimization in policy space. Neural Information Pro- cessing Systems (NeurIPS), 2020.
Srinivasan, K. P., Eysenbach, B., Ha, S., Tan, J.. and Finn, C. Learning to be safe: Deep with safety critic, 2020.
URL https::p////. /ttp://aaavvv...//eeeeeeee..aela org/ abs/ 14603.
Zhao, W., He, T., and Liu, C. Probabilistic safeguard for re- inforcement learning using safety index guided gaussian process models. In Learning for Dynamics and Control Conference (L4DC), 2023.
Tan, D. C., McCarthy, R., Acero, F., Delfaki, A. M., Li, Z.,, and Kanoulas, D. Safe value functions: Learned critics as hard safety constraints. In IEEE International Conference on Automation Science and Engineering, 2024.
Zheng. Y., Li, Yu, D., Yang, Y., Li, E., Zhan, X., and Liu, J. Safe offline reinforcement learning with feasibility- guided diffusion model. International Conference on Learning Representations (ICLR), 2024.
Tessler, C., Mankowitz, J., and Mannor, Reward con- strained policy optimization. In International Conference on Learning Representations (ICLR), 2019.
Thananjeyan. B., Balakrishna, A., Nair, S., Luo, M., Srini- vasan, K. P. Hwang, M.. Gonzalez, E., Ibarz, J., Finn, C., and Goldberg, K. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 6:4915-4922, 2020.
Van Hasselt, H., Guez, A., and Silver, D. Deep reinforce- ment learning with double q-learning In AAAI Confer- ence on Artificial Intelligence (AAAI), 2015.
Wagener, N., Boots, B., and Cheng, C.-A. Safe reinforce- ment learning using advantage-based intervention. In International Conference on Machine Learning (ICML), 2021.
Wang, Y. and Zhu, H. Safe exploration reinforcement learning by reachability analysis over learned models. In Gurfinkel, A. and Ganesh, (eds.), Computer Aided Verification (CAV), volume 14683 of Lecture Notes in Computer Science, pp. 190-213. Springer, Cham, 2024.
Yang, I.,, Ji, J., Dai, Zhang, I.., Zhou, B., Li, P., Yang, Y., and Pan, G. Constrained update projection approach to safe policy optimization. In Neural Information Process- ing Systems (NeurIPS), 2022.
Yang, Y., Jiang, Y., Liu, Y., Chen, J., and Li, S. E. Model- free safe reinforcement learning through neural barrier certificate. IEEE Robotics and Automation Letters, 8(3): 1295-1302, 2023.
Yu, D., Ma, H., Li, S., and Chen, J. Reachability constrained reinforcement learning. In International Conference on Machine Learning (ICML), 2022a.

---

Safety-Polarized and Prioritized Reinforvement Learning
A. Why State-agnostic Masking Threshold is not Enough
Unsafe Rate
Ours
Qo. 0.9
S2
0.40
SafeQ(0.28)
SafeQ(0.36)
dg 0.8
0.28
0.35
0.32
a1, 0.8
S3
S4
So
0.30
a₁ 0.6
0.0
0.2
0.4
0.6
0.8
1.0
1e4
(a)
(b)
Figure 4. (a) The MDP structure: the reward is always Taking action [ao, 01) transitions the with probability y or to a fixed unsafe state (termination) with probability - blue arrows indicate transitions to the pointed state regardless of action. (b) Learning curves of unsafe rates using our algorithm and SafeQ over 10 seeds. The æ-axis represents the number of steps. Lower is better.
Our proposed optimal action masking (state-dependent) enjoys a convergence guarantee as shown above. contrast, we will show that the direct E-masking method used by SafeQ. which masks actions with an estimated unsafe probability exceeding fixed, state-agnostic threshold E, may fail achieve the minimum unsafe probability Emin even when is min- To be more specific, we write SafeQ in the form of Q-learning and use the SA-REF to be its unsafe probability estimator
Q4+1(5 + (1 ())
Be(r: + max d'ECr()))))
Qr(s(+1,d)).
(22)
where (s) = {b E ()) S o) is the action mask based on e, and the tp-update of SafeQ
(141(5 + (1 (e) az)+ B. max(T[se (. (((+4))))),
(23)
where (s) arg (s) Qr(s, a) is the learned policy by SafeQ.
Our bad example is based on the intrinsic shortcoming of SafeQ: if 3a * be C. (s) while Qr(s, = (s, 6), then SafeQ policy it could not identify which action better. The MDP structure of the bad example presented Figure 4(a) where 80 the fixed initial state and the reward function set to 0. Clearly, the optimal policy is ==(s)) = ag and T (s1) dg. achieving the minimum unsafe probability Emin 0.28. The sub-optimal policy Taub(So) ag and Taub (ao|s1) = Tsub (a (s1) = has unsafe probability 0.32. We run our method and SafeQ in this example and find that, setting
0.28 for SafeQ (the SafeQ(0.28) in Figure 4(b)), it only learns the sub-optimal policy suub with unsafe probability 0.32. This is because at state the unsafe probability of executing action or is smaller than Emin 0.28 and the action mask C=(s1) with = = 0.28 could not distinguish the difference between and since Qt = 0 by our MDP. Thus, SafeQ may only uniformly choose actions at 81- The same argument holds for larger E, e.g., E = 0.36 (SafeQ(0.36) in Figure 4(b)) which is the unsafe probability policy =(sg) = ag, (s1) = a1 contrast, our safe Q-learning with optimal action masks converges to minimum unsafe probability of 0.28, which aligns with our Theorem 4.4.
12

---

Safety-Polarized and Prioritized Reinforcement Learning
B. Theortical Proofs
B.1. Proofs of Lemma 3.1
Proof: For the contraction property, v(s,a) A, we have
(s, - ((Q2(s,a)
(r(8,a) + +BEeNP((bb.)
Q1(6',a'- r((8,) - -7Bor max d'ECc(o')
Q2(0', a')
max d'ECc('')
Q1(6',
22(6',a²)
max a'EC((s")
max a'ECc(o')
Q2 - -Q2(s', (e)6s,d)]]
|m Q1 max Q2 I max Q1 Q2
max o'ECc(o))
Qi(s'' Q2(5',a)) ,d) 8'a'
$ y max
and thus
1B,Q1 B,Q2llo S <11Q1= -22loe
where [0 == max., IQ(s, a)] is the Loc norm. Therefore B& is a 7-contraction (with respect to the -ormm).).
Let Q7 be the unique fixed point of B3 and क (5) == arg (c)))(((()) Qī(s, a) be the corresponding policy induced by Q For the safety optimality of .द. since (s) always chooses actions among the optimal action mask Cç (s), Vs S, which masks the action with non-minimal unsafe probability, the unsafe probability of मद is the same n² and thus safety optimal, i.e.
केे E mig
Pr [s; ET:8; Sy). somp. (T.
n
For the reward optimality, this follows from the fact that x" will only select actions in Cc(s) since n² E arg min, Pr [Est E T st E S.]. Therefore the Bellman backup of Q the same as that of Q?, which T~(,,P)
implies that
(s, (,
As result, के is reward optimal.
B.2. Proof of Lemma 4.2
Proof: First, if s E S., then (s, a) = by definition and the result holds clearly. Therefore we assume & S,.. Let T= {+0 = 00 = a, ".. 8t, oz, ... } ~ (T,, P) be a sampling trajectory starting from state sp and action do a using policy æ with transition function P. Then we have
$0 (8, Ey-(H.P) max Tise E S.]] 80 = 8, 5,00 REET
= a)
max L[se E S.]] 51 = = d) a.EM
E...P(lsa)) (Tr(T,P)
-gssp(lsa) Nz (s'
where = {s) = 8' = d'....,,,
st.... ,t....,,t....................... (T,, P) is the sampling trajectory starting from state s' and action a', This
completes the proof.
13

---

Safety-Polarized and Prioritized Reinforvement Learning
B.3. Proof of Theorem 4.3
Proof: We convert the update of th to an equivalent undiscounted Q-function update follows. Consider the following undiscounted infinite horizon MDP M,, == (S,A, Fur P,P) where
-1,
E Su,
ru(8,a)= - 0, otherwise.
Let (s, -om (, a), where
N-1 l
()(s,)) = Ez Σ (u(s so s, 0,00 =
Since S. consists of absorbing states, (s) > -00, VT, Clearly, Pry Est E T E Sulso aj
-2, a). Then o0 (s, a) = (T a) and 30' (s, (2, a), where Q (8, a) is the optimal undiscounted Q-value in Mu. Furthermore,
( 0) = max [I[s E Sul, P( min (ss))'a') d'Ed
max d')l =: LQu,(s,a). Qu.++i(s,a) as) P( d'AA
According to Theorem 7.3. 10. of Puterman (1994), starting from Qu,0 = 0, the iterative update Qu,++I LQN,, converges monotonically Qú, which is equivalent to the convergence of the to a/ starting from voo 0. In this way we complete the proof.
B.4. Proof of Theorem 4.4
Proof. First consider the update,
Ve+1(Se, 0.) (())))(ssss + Be max{I[st E Sul, min (s)+),a'))
a'EA
(24)
= We at) + 3 (( 8t, a:) thesss at) + Me+1)
where F: ( + IR S||A| is the backup operator defined by
F(158,0) a) == max S.), P min v/ol, (bd',a'))
(25)
d'dd
and the noise term
M41 max{I[s e Sul, min (s)41,)d)) max{I[s; E Sul, ( min velss' ,')}.
(26)
d'EA
d'eA
The associated ordinary differential equation (ODE), with initial condition (00) tho = 0, is
F(b) tb
(27)
The following lemma is borrowed from Borkar (2008), Page 127, Theorem 4. Lemma B.1. (Borkar, 2008). Consider stochastic approximation algorithm
X, Met1l, t20 0,
where X, E Rd, F: Rd + Rd. Let the associated ODE be *(t) = F(x(t)) xxt) starting from z00) = Xo. Define the
Troe -norm || X| = max,eld] [x,]. Assume:
14

---

Safety-Polarized and Prioritized Reinforvement Learning
(A1) (Non-expansiveness of F). VX, Y E R2, [F(X) & < || Ylloo
(A2) (Boundedness). sup12o Xe < +oo almost surely.
(A3) (Diminishing step sizes). The sequence (fr) satisfies < Br S and E120 B: B== +00, Eizo 89 +00.
(A4) (Martingale difference noise). Let F, be the o-algebru generated by (Xo, Mo....,,y, Me). There exists > 0 such that Vt, E|M [Fo] and M| Il Fil S K )
Under the above assumptions, if H = {X E Rd F(X) = } is non-empry, then X, + X a single point in H (depending on Xol-
Claim, The sb backup operator F in Equation (25) non-expansive.
Proof of Claim. This is due to, for any 15 I [|SA|| and for any E S a A.
F F(1; 8, a) sa)
max{Ilss Sul. Sul.Berr(lson min (ss. max{I(ss E Sul, P(iie.) min o(s' ,')) a'EA
a'EA
Byr-P(llan) min w5s'' aa) ) Berm(() min ((s', a') a'EA"
d'EA (b')'aa)) &661, a')
max S'EE
|| 001oo.
Claim, the is bounded: Vt 2 0, 1/vello0 S.
Proof of Claim, We prove this by induction. First t/n = 0 and the claim holds at / = 0. Assume now the claim holds at f. For t + 1, (s, a) # (st, ar) then [1/ a)l = (v)(s, a)l s i; (s, = (st, ar), then, since 0 < 3r S 1 we
have,
Wet1 (st, (1 (s,,,) + & max{I[se E Sul, min (e(4441, a') d'eA
$ - 5,) + =
Therefore || < 1 and then the claim holds for all / 2 0.
Claim, The noise term M,+1 in Equation (26) satisfies: Vt, M[+ = and |||| 12 FFl s 4.
Proof of Claim. First, using (s, a)l and considering whether s E Su or not, one easily that
B[M~1 [..]
max (T[se Sul, min the (8 a'EA
max [[st E Sul,
min (i a²) a a'EA
0.
Second, again by |ov $
E Ml
maxx Sul, min (8 - max {T/or E Sa), Per a'EA
min (s ((6',,)))] |] LaeA
[[ max E Sul, min d'Ed ve(s))1,d)) 2
< 2E., ImP(laad)
+ max [ss Sul, mnd)) min We a0) o0]}] Laee
<
This proves the claim.
15

---

Safety-Polarized and Prioritized Reinforvement Learning
As result, the de update in Equation (24) satisfies the Assumption (A1)-(A4) in Lemma B.I. Therefore, according the Lemma B.1, the converges single point H {F( 1/). Since (0(0) 0, our ODE in Equation (27) converges
to 1 by Theorem 4.3, Therefore the & update in Equation (24) will converge: 1/: + x," starting from vho = 0.
Next, we consider the Q update. We expand the Equation (13) as follows:
Q1+1 ax) Qc uz) + Be (r tr max Qc(+++1,a) - Qe(st, an)
aEce, (3112)
-Qo(se; ur) + Be r7 (ECe($111)
max
a)( - Qr((s,
+7 max (5112) Qr(sc+1, -7
max aEC((S111)
Qr(3E+1,a)
(28)
= -Q((st, 0r) + Br rr + YEs
max (CEC((8))
Qc(5',a) a) - -Q((s, ar)
+y max (+++1)
Qc(8++1, - >'IEs'
max (cEC(()))
(r(s',a)) a0]
+ aEC; (5441)
Qc(S++1, -7 max
(c(
where we recall that
=c(s)= = (bEA ( S ((s) = min vt (ss
acA
and Cs, (s) is the set of minimizers of (1t,
(s) = (bEAl (8, S ((()) = min
aEA (he a) + k):
where & > 0 a small enough constant satisfying v/ (s. + 2 < (s, e) for any 8 E S, Cc(ss and E C(((ss)).
Since t/e converges to 2²* t + +00, we have for any e 0, there exists some T > 0 such that when T. [s a) v (s) < e S A. Taking 2. for any b (s(s) and A Cc(s). we have
+c (s,e) 2x+ (1,,e) + 2c x == (s. (s + n (s,
(29)
Claim. for any b E (s), we have b E CG. (s), and for any e E A \ Cc(s)) we have e E A C
Proof of Claim. Since ((s) = v0 (s, b). we also have (s, E [((s) - e, ((s) + c]. Then
(s)) min (s, +5= min (b b) + ((s) e+ = ((()==(()+++++ +c, a
bEC((a)
where the second cquality is due to Equation (29) and the final equality is due to = S. Thus (e( b) s ((s)+e S Ge(s).
This shows that Cx(s) (s) C C2, (s). On the other hand, again by Equation (29), (s,ee > b)) 5 2 min (s +5= ((()). This shows that A \ (s) C AAY \ Ca (s).
Therefore C₁. (8) will be the same as (s) after t> T. Taking t > T, the Equation (28) becomes
Qe+1(s,, a:) Qe(s,, ac) + 3, (. +
Qe(s',a) a)] (r(s)aat)
max <cEC((o))
(30)
+ Y aEC2(5141)
max
Qc - 7E
max aECc(a")
Qr(s),a)))
whose convergence to Q7 is guaranteed by classical Q-learning theory (Borkar, 2008). We complete the proof.
16

---

Safety-Polarized and Prioritized Reinforcement Learning
(a) TwoWay
(b) Merge
(c) ACC
(d) Roundabout
(c)
Intersection
Circle
Figure 5. The visualization of our benchmarks.
C. Experiment Details
C.1. Benchmarks
In highway-env (Leurent, 2018), the state space can either be of kinetics type, which describes list of nearby vehicles by set of location and velocity features, or be of time-to-colle co-elloocollision type, which represents the predicted time-to-collision of observed vehicles on the same road as the ego car. In Two and Merge, we select the cone-to-collision type for state space, while Roundabout and Intersection, we select the kinetics type. The action space consists of discrete meta actions, which can change the target lane and speed that are used as setpoints for some low-level controllers, so that the ego automatically follow the road the desired speed. The rewards encourage agents move fast and achieve their corresponding goals, which are:
TwoWay. The goal drive two-way lane as fast as possible with incoming traffic.
Merge. The goal of ego car is to maintain high speed while making room for the incoming cars the access ramp so that they can merge into the traffic safely.
Roundabout. Facing the complex traffic flow, the ego need to pass roundabout safely following a pre-defined route with high speed.
Intersection. In the dense traffic, the ego car should cross a four-way intersection while avoiding crashing with other cars.
For the other two safe control tasks, give their detailed environment setup here:
ACC. (Anderson al., 2020) The ego aims to follow lead closely possible without crashing into it. The state S consists of the relative distance d, the speed of ego ue and the speed of lead car ey. There three actions: -1 (decelerate), 0 (idle), 1 (accelerate). The acceleration Auy of the lead car follows from a Gaussian distribution Au; ~ (10, 0.2). We do not allow the lead car and ego car to backward. The lead car is taken as reference. When < -00 (staying too far) or d 2 0 (crashing), the episode terminated and agent receives crash reward 0; otherwise the reward r 10 The maximal episode length 100.
Circle. (Achiam et al., 2017; Wagener et al., 2021) point robot gets rewards for fast circular movement, but is constrained to stay inside the area restricted by two walls which narrower than the target circle. The state
17

---

Safety-Polarized and Prioritized Reinforvement Learning
Table Hyperparameters for all DQN-based algorithms
Algorithm
Parameter
Value
optimizer
Adam
discount factor
0.99
Q-network learning rate
10
batch size
64
Shared
update every
imitial e-greedy exploration rate
< decay
0.995
< min
0.01
number of random seeds
SA-REF d learning rate
5- 10-4
SPOM (ours)
polarization function
fpo(z) 10 log(())
priority exponent
0.6
SPOM PER (ours)
importance sampling exponent
0.4
safety critic Qrisk learning rate Erisk
5 10
SafeQ and Recovery
Yrisk
0.99
RCDQN
Lagrange multiplier learning rate
0.001
RewsDQN
shaped crash reward
Table Hyperparameters for PPO-based algorithms
Algorithm
Parameter
Value
optimizer
Adam
discount factor
0.99
Shared
learning rates of actor and critic
3 10
GAE parameter
0.97
clip ratio
0.2
REF learning rate Lagrange multiplier learning rate
1 10
RESPO
5 10
PPOLag
Lagrange multiplier learning rate 0.001
= (x, y, ct 1), where (x. y) is the xy-coordinate of the agent, and i are the speeds at x-direction and y- direction respectively. The action space consists of one idle action and the following eight directions for acceleration: (cos(k:r44) sin(kn/ ())) = 0, ......77 7. After choosing an action a, the corresponding acceleration vector is Aconst O, where Oconst 0 is constant acceleration scalar. The maximal episode length is 100. The rewards encourage eircular movement of radius R and the unsafe state set S,, restricts agent to stay inside the region |ar S Tmaxx
S = {s = (2, Ú) E |x]]
S xmax )
(.,ū) (-y,z)
if & Sus otherwise.
a(s
1+ (2),8)11) R
For experiments, we take Aconst Imax 2.5 and R² 5.
C.2. Algorithms
1. IMPLEMENTATION
The original SafeQ (Srinivasan et al., 2020) and Recovery RL (Thananjeyan et al., 2020) are based on actor-critic frameworks. Here we give detailed descriptions on their adaption the DQN implementations.
18

---

Safety-Polarized and Prioritized Reinforcement Learning
SafeQ. (Srinivasan al., 2020) A Q-network Q and a safety critic Qrisk are required with their corresponding target networks Q and Qrsk The safety critic Qrisk is used predict the future discounted risk for safety violation, with discounted risk factor Trisk (0, 1). SafeQ performs direct -rakkngg
*(s) = arg max (s,,a): acCe, (8)
where Cevia (8) E EA| Qrisk b) Erisk and Erisk 0 is fixed state-agnostic threshold. The target of safety critic is computed through
iisk I[s E S.] (1 I[s E Sriak Drisk (1 ($(+1)),
to estimate the unsafe risk under current masked policy म, and the target of Q-network is also based on π:
y + Q854
Recovery. (Thananjeyan et al., 2020) Here we also require a Q-network and a safety critic Qrisk- But the final policy is composed of a task policy T trask and a recovery policy Trees where
task (s) = arg max ((8.a), a), (s) arg min (sisk(s,
and then is obtained via an intervention-base scheme: if the action a task chosen by the task policy Trask has unsafe risk greater than Erisk. then it will be overtaken by the recovery policy Tree which will choose the action with minimal risk:
a task = Teack(();
Qrisk (8, a task) Erisks
π (8)
(s),
otherwise.
Note that according to Thananjeyan et al. (2020), the task policy is trained on the task buffer (se, of task Te, st+1), and the Q target is computed by
y =r r. Q((++ (1ast((8441)) ));
while the safety critic Qrisk is trained on the real buffer (st, des Te, s141) and the Qrisk target is
Vi risk = 1ss E S.) + (1 I[s E 8.l)) Trisk Drick (0 n (8(41)),
in order to estimate the unsafe risk under current policy æ. Since we focus on the persistent safety, for both SafeQ and Recovery, we will take Yrisk = y the discount factor of the MDP.
RCDQN. Following Tessler et al. (2019), we constrains the reward through r r- Ae, where À is the Lagrange multiplier and c the cost signal corresponding to crash, and then we use DQN learn upon this constrained reward. Note that the Lagrange multiplier is updated via the on-policy samples, which relates to the safety of current policy more closely. RewsDQN. This reward-shaping- based DQN tries inform the agent to avoid crashing via strong penalty when crash occurs. The shaped crash reward is set to be -10 uniformly.
For completeness, we also choose some of the Proximal Policy Optimization (PPO) (Schulman al., 2017) based based algorithms under CMDP framework to our tasks. We include the following representative ones:
RESPO. Ganai et al. (2023) propose the Reachability Estimation Safe Policy Optimization, where state-dependent reachability estimation function (REF) is learned to estimate the unsafe probability under current policy. Then RESPO optimizes the rewards in safe regions while maintaining safety through Lagrangian methods, and optimizes the costs in unsafe regions.
PPOLag, (Ray et al., 2019) A classical primal-dual method, which uses a Lagrangian relaxation to transfer the original CMDD--based constrained optimization problem to an unconstrained one and then adapts to PPO.
PPOBarrier. (Yang et al., 2023) An extension of PPO that incorporates control-theoretic barrier certificates to ensure policy safety by enforcing constraint satisfaction during both training and execution.
19

---

Safety-Polarized and Prioritized Reinforvement Learning
Table The SWU scores of algorithms,, including RESPO PPOLag.
SWU Scoret
TwoWay
Merge
Roundabout
Intersection
ACC
Circle
Overall
3POM_PER (ours)
0.98
0.94
1.04
0.68
1.07
0.96
0.95
SPOM (ours)
0.88
0.95
0.34
0.96
0.87
0.37
0.73
SafeQ
0.73
0.84
0.54
0.75
0.50
0.15
0.59
Recovery
0.34
0.66
0.69
0.66
0.85
0.25
0.58
RCDQN
0,41
0.81
0,41
0.57
0.46
0.14
0.47
RewsDQN
0.33
0.92
0.41
0,50
0.47
0.11
0.46
DQN
0.37
0.56
0.41
0.58
0.46
0.13
0.42
RESPO
0.06
0.09
0.09
0.11
0.06
0.01
0.07
PPOLag
0.06
0.03
0.13
0.11
0.11
0.02
0.08
C.2.2. HYPERPARAMETERS
Network Architectures. In Roundabout and Intersection, to deal with the kinetics type of state, which contains a list of features about nearby vehicles, we use the ego-attention-base architecture proposed by Leurent & Mercat (2019), which can handle the permutation of the list input and thus is suitable for these two tasks.
In TwoWay and Merge, and Circle, we also use two-layer MLP, with 256 hidden units.
two-layer multi-layer perceptron (MLP) networks, with hidden layer of size 128. In ACC
Parameter Settings for DQN-based Algorithms. The exploration strategy is unified to be e-greedy. In highway-en we use replay buffers of size 15000 and hard update for target networks with hard update interval 512, following the default configuration Leurent & Mercat (2019); while ACC and Circle, we use the replay buffers of size 10a and soft update with soft update coefficient 10-3 and interval 1, to ensure stable and fast learning. Other hyperparameters are unified across tasks and can be found in Table 2.
Parameter Settings for PPO-based Algorithms. According to Ganai al. (2023), RESPO should maintain that the Lagrange multiplier learning < the REF learning rate < the actor and critic learning rates, in order to ensure a stable convergence to a local optimum, and so we use their default parameters. For PPOLag, the Lagrange multiplier learning rate set to be larger than that of RESPO, to bias more towards constraint violation in the original primal-dual formulation. The detailed hyperparameters are listed in Table 3.
C.3. Full Results
C.3.1. MAIN EXPERIMENTS
The full training curves of all baselines (DQN-based and PPO-based) and SPOM, SPOM PER are provided in Figure and 7. We find that RESPO and PPOLag fail to optimize under our environments since the cost signal corresponding to a crash is very sparse. The full SWU scores are presented Table where SPOM PER achieves the highest overall SWU score of 0.90, and SPOM secures the second-best score of 0.79. Both algorithms demonstrate significant improvement over baselines, with SPOM PER achieving over 50% improvement compared to most baselines.
We also list the average crash rates and episode rewards over the last and per task. in Table 5. SPOM_PER along with SPOM achieves the lowest or one of the lowest crash rates across almost all tasks while maintaining competitive or superior episode rewards.
TO training steps and all random seeds, per algorithm
Comparison with direct e-masking approach SafeQ: Across all tasks, SPOM PER and SPOM outperform SafeQ both in terms of crash rates and rewards. This again validates our insight that state-agnostic masking threshold is not enough to ensure maximal safety.
Comparison with aetion-correccion-baaee method Recovery: In ACC, SPOM PER achieves the best SWU score of 1.07, outperforming Recovery, which sacrifices rewards to reduce crash rates. Similarly, in Circle, SPOM PER and SPOM achieve significantly lower crash rates than Recovery while maintaining higher rewards. In Intersection, Recovery trains to be safe faster but converges to lower rewards compared to SPO..PER. For other tasks, SPOM PER and SPOM achieve a better trade-off between safety and rewards while being safer than Recovery.
20

---

Safety-Polarized and Prioritized Reinforcement Learning
Table 5. The crash rates and episode rewards (mean and std), averaged over the last To training steps and six random seeds, of all compared algorithms. SPOM achieves the lowest or one of the lowest crash rates while maintaining competitive episode rewards.
TwoWay
Merge
Roundabout
Intersection
ACC
Circle
SPOM PER (ours)
0.073+0011
0.043+00008
0.055++00277
0.086=+0.025
0.163+0011
0.038=00010
SPOM (ours)
0.085
0.042=00006
1..17
0.069
0.200
0. 0.1011
SafeQ
0.104
0.050=0.007
0.090
0.091
0..01
0.255
Recovery
0.134:10016
0.063
0.062:10.33
0.06810.02
0.205
0.119
Crash Rate 4
RCDQN
0.176=#0001
0.052=0.006
0. 0.120==0.029
0. 0.113=00008
0.350=0.000 009
0.268=00018
RewsDQN
0.209
0.042=00.08
0.122:00067
0.130=0.027
0.344=00018
0.350=00044
DQN
0. 0.198
0.074=0.014
0. 1.04:
0.118=00012
0,351
0.288=00035
RESPO
10.39
0.228::0.247
0.047 10.104
0.400.10.100
0.932 10.062
1..097
PPOLag
0.509
0.426
0.140 14.4
0.355 009
0.729
0.148 1..022
SPOM PER (ours)
10.124=0037
1.833=0.035
0.985=00324
4,433=00.00
78.393=0.558
279.432=2866
SPOM (ours)
10.520#0.389
1.823=00008
0.947=+0218
4.978
78.337
279.4322:2.655
SafeQ
10.636=0.166
1.929=00003
0.799
5.115=0.112
73.013=0.551
268.774-44.755
Recovery
6.392-+0.279
1.897=00.08
0.637=t0413
3.208
78.296-0:520
216.606+911 606--91.921
Episode Reward f
RCDQN
10.217=0.567
1.935=0.049
0.8 200256
4.844=20.118
73.026=0.818
273.679 373.679=1.008
RewsDQN
9.775
1.764=20.075
0.827
4.932:10.286
72.571 11.084
209.573
DQN
10.284
1.919=00040
0.789
5.125
73.278=10077
270.404
RESPO
6.099
0.900=0.354
0.071 103
3.294=1
22.396 7.88
7.179
PPOLag
4.860+3.870
0.535=0.600
0.298=00352
3.024=11812
37.752=22766
25.753=4.506
Comparison with CMDP- bbsee approach RCDQN, RESPO, and PPOLag: The DQN-based RCDQN has stable perfor- mance but fails to guide the policy effectively in terms of safety due to sparse cost signals. The PPO-based algorithms, RESPO and PPOLag, suffer from high variance and fail optimize either safety or rewards effectively. In highway-env, their performance shows inconsistency between crash rates and rewards, and in ACC and Circle, they converge prematurely to suboptimal solutions. Overall, CMDP-based methods struggle with sparse cost signals our tasks, leading to subpar performance compared to SPOM and SPOM.
Comparison with reward-shaae -wwaadd-happngg approach RewsDQN: While RewsDQN achieves competitive crash rates in some tasks like Merge, its episode rewards are significantly lower. In Circle, RewsDQN fails to achieve comparable crash rates and rewards as SPOM_PEL PER and SPOM, demonstrating the difficulty of balancing safety through reward shaping alone.
Note on vanilla DQN: Vanilla DQN can reduce crash rates to certain level due to early termination in tasks. However, its performance in balancing crash rates and rewards falls far behind SPOM_PER and SPOM, which consistently achieve better safety and reward trade-offs.
C.3.2. ADDITIONAL ABLATION STUDIES
We provide the training curves of our ablation studies for the remaining four tasks in Figure 8.
Once again, we can find the over-conservative phenomena of direct optimal action masking "OAM" Merge and Roundabout (the first two rows in Figure 8), where "OAM" shares similar crash rates as others while it significantly sacrifices the rewards to achieve the same level of safety. This shows the hurt brought by optimal action masking on an immature & network in the early stage of training, which is also the reason of suboptimality the strong polarization effect given by "xp". And the weak polarization effect given by "log" cannot ensure to safe enough. But in ACC and Circle, all ablation algorithms have competitive safety and reward performance (the last two rows in Figure 8).
C.3.3. ADDITIONAL RESULTS COMPARED TO PPOBARRIER
We present the training curves the four highway benchmarks, comparing our methods against the bascline PPOBarrier, in Figure 9.
21

---

Safety-Polarized and Prioritized Reinforcement Learning
TwoWay
TwoWay
12
10
Episode Reward
8-
Crash Rate
0.6
6-
0.4
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
Step
1e5
Step
1e5
Merge
Merge
2.0
Episode Reward
1.5
Crash Rate
0.6
1.0
0.5
0.2
0.0
0.0
-0.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Step
Step
1e5
Roundabout
Roundabout
1.50
Episode Reward
1.25
1.00
Crash Rate
0.3
0.75
0.50
0.1
0.25
KCDON
0.00
Rewibok
-0.5
0.1
5ako
0.2
0.6
1.0
0.2
0.4
0.6
1.0
Step
1e5
Step
1e5
Figure 6. Training all environments. Each shows the episode reward (left) and crash (middle) for specific enviromment. The rightmost panel in the third row shows the legend shared across all plots. Higher reward and lower crash rate are better.
22

---

Safety-Polarized and Prioritized Reinforcement Learning
Intersection
Intersection
0.6
Episode Reward
0.5
Crash Rate
3-
0.2
1-
0.1
0.0
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
Step
1e5
Step
1e5
1e2
Circle
1e-1
Circle
8-
3.0
Episode Reward
2.5
Crash Rate
2.0
1.5
1.0
0.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Step
1e6
Step
1e6
1e1
Acc
1e:1
Acc
10
Episode Reward
8
Crash Rate
6
4
KODON
2
Rewibok
Sako
0.0
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Step
1e6
Step
1e6
Figure 7. All training curves in Intersection, ACC, Circle. The z-axis represents the number of steps, and the y-axis: the first episode reward (higher is better); the second row is safety measured in crash rate (lower is better).
23

---

Safety-Polarized and Prioritized Reinforcement Learning
Merge
Roundabout
1.5
2.0
Episode Reward
Episode Reward
1.5
1.0
1.0
10log
log
0.5
xp
OAM
0.0
0.2
0.6
0.0
2.0
2.5
3.0
Step
1e5
Step
1e4
Merge
1e-1
Roundabout
Crash Rate
Crash Rate
2-
1
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.5
1.5 Step
2.0
2.5
3.0
Step
1e4
1e1
ACC
1e2
Circle
Episode Reward
Episode Reward
7
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Step
1e6
Step
1e6
ACC
1e-1
Circle
1.0
Crash Rate
Crash Rate
0.6
0.2
8.0
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Step
1e6
Step
1e6
Figure Training of other ablation studies, where "log", "xp" represent using polarization function log(æ)) and respectively, "10 log" our default choice 10- log(z), and "OAM" means applying optimal action masks directly. The æ-axis represents the numher of steps, and the y-axis: the odd rows are episode reward (higher better); the even rows are safety measured crash rate (lower is better).
24

---

Safety-Polarized and Prioritized Reinforcement Learning
TwoWay
Merge
2.00
Episode Reward
1.50
1.25
Episode
0.75
0.5
PERlours)
SPOM PER(ours) PPOBamier
PPOBarier
1e5
Step
Step
TwoWay
Merge
SPOMoours) PERlours)
SPOM|ours]
0.8
SPOM PERlburs)
0.8
Crash Rate
Cresh Rate
0.4
0.4-
02-
0.0
-0.2
Step
Step
Roundabout
Intersection
144
PPOBarrer
1.-
Episode Reward
SPOMiours] SPOM PERiOUrS)
0.6
Episode
2-
PPOBaarreer
02-
-02-
0.0
0.2
0.4
0.6
0.0
1.0
0.0
0.2
0.4
0.8
0.0
1.0
Step
165
Step
1e5
Roundabout
Intersection
SPOMlours)
0.5
SPOM|
SPOM PERCours)
SPOW PERlours)
PPO
0.4
#Potiamer
Crash Rate
0.3
0.1
0.0
0.0
-011
0.2
0.0
0.2
0.8
165
Step
165
Figure Training curves compared PPOBarrier on four highway tasks.
25


Reviews:
[{"review_id": "r6AQ1pzQvx", "reviewer": "Reviewer_BcoJ", "summary": "This work introduce a chance-constrained bi-level optimization framework called MaxSafe, for the\nmaximal-safety RL problem.\nMaxSafe first minimizes the unsafe probability and then maximizes the return among the safest policies.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude to the reviewer for the valuable feedback and constructive comments. To better quantify the trade-off between reward and safety, we adopt the SWU score introduced by Yu, H., Xu, W., and Zhang, H. in Towards Safe Reinforcement Learning with a Safety Editor Policy (NeurIPS 2022b) as our final evaluation metric to compare the performance of different algorithms. The following sections address the two main questions raised by the reviewer.\n\nQ1: The soundness of our assumption that there exists a sufficiently large policy space with minimal unsafe probability.\n\nWe assume that in the environments we consider, there exists a sufficiently large policy space where the probability of unsafe events is minimal. Such environments are common in autonomous driving and robotic control. For instance, there are many possible driving policies that allow a vehicle to operate safely on the road without collisions, and a sufficiently large set of policies that enable a robot to navigate without falling or colliding with obstacles—indicating that a large policy space with minimal unsafe probability indeed exists in there environments. This assumption aligns well with many practical scenarios. Building on this assumption, our MAXSAFE policy selects policies from this space and aims to optimize reward performance while ensuring safety.\n\nQ2: The proposed methods SPOM_PER and SPOM do not always guarantee the lowest crash rate.\n\nIn our experiments, we use the SWU score to evaluate the trade-off between safety and reward. As shown in Table 1 of our paper, SPOM_PER and SPOM achieve the best overall performance across all six benchmark environments. Although the Recovery baseline achieves the lowest crash rates in Roundabout and Intersection, its reward performance is significantly worse, indicating that it suffers from an overly conservative policy in these two environments. In contrast, our SPOM_PER and SPOM attain the highest SWU scores across all benchmarks. Also, the results demonstrate that applying Prioritized Experience Replay (PER) to our proposed SA-REF effectively reduces safety violations while preserving strong reward performance.\n\nWe sincerely appreciate the reviewer’s constructive feedback, which has been invaluable in improving the clarity and rigor of our work. We hope our detailed responses have addressed your concerns, and we welcome any further suggestions to strengthen our paper.", "claims_and_evidence": "1) The authors assume that there is sufficiently large policy space with minimum unsafe probability.\nAnd justify their assumption by saying that it has already been explored in\nthe literature (e.g., Ganai et al., 2023).\n\nHowever I'd say that this assumption depends very much on the environment at hand. \nWhat class of environments would justify such an assumption?", "methods_and_evaluation_criteria": "Yes, the experimental evaluation is rather thorough, with several relevant RL environments considered, and different related approaches compared.", "theoretical_claims": "The theoretical results appears to be sound. Proofs are provided in the appendix, but I did not check them.\n\nThe authors claim that \"Extensive experiments demonstrate that our method achieves an optimal trade-off between reward and safety, delivering near-maximal safety.\"\nIs this meant in terms of the SWU metrics introduced in Sec. 6?", "experimental_designs_or_analyses": "The authors present experiments on autonomous driving and safe control tasks, demonstrating that the proposed algorithms, SPOM and SPOM PER, achieve the best reward-safety trade-off (as computed by safety-weighted-utility) among state-of-the-art safe RL methods.", "relation_to_broader_scientific_literature": "The state of the art is discussed in depth in the related work section.\nIn particular the authors refer to (Ganai et al., 2023) and their reachability estimation function (REF) to capture the probability of constraint violation at state s under a given policy. In this work, the authors extend the definition of REF to be state-action dependent.", "essential_references_not_discussed": "Nothing relevant not discussed to my knowledge.", "other_strengths_and_weaknesses": "1) The paper makes both theoretical (Sec. 4) and practical (Sec. 5) contributions.\nIn Sec. 4 the authors define a methodology to obtain optimal policies with provably minimal safety violations.\nThen, in Sec. 5 they adapt the methodology to an implementation with deep RL, by using polarization functions and safety prioritized experience replay.\n\n2) As a weakness, the experimental evaluation does not appear to be conclusive, this might be due in part to the fact that the trade-off between safety and performance is elusive.\nAs it appears from Fig. 1, the methods here proposed, SPOM_PER and SPOM, not always guarantee the lowest crash rate.\n\n3) Also, related to the point made above, it is not clear how much justified is the assumption that there is sufficiently large policy space with minimum unsafe probability.\nConsider a stochastic exploration task, where there is a non-zero probability of ending up in an unsafe state. It might be that no policy is zero unsafe, as there is always some non-zero chance of ending up in a non-safe state. It looks like in such case the method described in the paper would fail.", "other_comments_or_suggestions": "Nope", "questions_for_authors": "Could the authors answer points (2) and (3) in Strengths And Weaknesses?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rpQJQOiQ0t", "reviewer": "Reviewer_dGwp", "summary": "The paper has a clear motivation to improve the safe RL through action masking. To avoid directly applying infinity to Q, the paper introduces the polarization functions. To improve the learning of REF, the paper uses prioritized learning. The result shows significant improvement in both reward and safetyness compared to the baselines.\n\n## update after rebuttal\n\nIt seems that the authors have addressed most of the other reviewers' concerns well. I am happy to raise my score to 5.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the reviewer for the positive recognition of our work.\n\nRegarding the concern about why we did not choose Safety-Gym as our benchmark suite, the primary reason is that our setting fundamentally differs from that of Safety-Gym. In our framework, safety is treated as the top priority. Accordingly, we assume that an episode terminates immediately upon the execution of any unsafe action (i.e., entering into any unsafe region). In contrast, Safety-Gym employs a scalar cost signal and allows safety violations as long as the expected cost-to-go remains within a predefined budget. For example, the “circle” environment in Safety-Gym’s safe navigation task is conceptually similar to the “circle” environment in our benchmarks. However, the underlying assumptions are different: in Safety-Gym, safety is treated as a constraint, and episodes continue even when temporary violations occur, as long as the total cost remains within the safety budget. In contrast, our formulation assumes strict safety—any safety violation immediately terminates the episode. This setting better reflects scenarios where safety is non-negotiable and takes precedence over performance. The mathematical comparison between these two setups is provided in the Preliminaries section of our paper, and we refer the reviewer to that part for detailed formulation. As a result of our strict safety assumption, the cost signal in our setting only appears at the end of an episode, making it significantly more sparse. To address this challenge and improve learning efficiency under such sparse cost feedback, we propose the use of Prioritized Experience Replay (PER) in our SPOM_PER method. This design decision is also motivated by the need to balance reward maximization with strong safety guarantees.\n\nWe truly appreciate the reviewer’s positive feedback and encouragement. We hope our responses have addressed your concerns, and we welcome any further suggestions to improve the paper.", "claims_and_evidence": "Yes, the claim made by the submission is supported clearly. There are clear improvements in the experiments.", "methods_and_evaluation_criteria": "The proposed method is on the right direction to tackle the key challenge of safe RL - balance between 2 objectives.\n\nThe evaluation criteria seems correct to me. Hopefully, the author can add Safety RL Gym as an additional benchmark, but I am totally okay without it.", "theoretical_claims": "I checked the covergence proof and there is no issue.", "experimental_designs_or_analyses": "The proposed experiment seems a new experiment setting. I double check and the settings look good overall, including the metrics and the reward design.", "relation_to_broader_scientific_literature": "The key contribution of the paper is to introduce the polarization function, which I feel is the correct way of learning action masked policy. \n\nTypically, people do a mixture of 2 modes - optimizing an objective and minimizing the cost - by adding them together as 1 objective. These approaches often suffer from sensitive Lagrangian hyperparameter or other issue to balance between the 2 modes.\n\nThe paper made crystal clear improvement by directly injecting the safe signal into the learning of Q function, which I feel to be the correct way to ensure safetyness inherently for learning a policy. The introduction of polarization function also feels novel and tackles the right key challenge part to learn the correct Q function.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The paper is written very well. It is overall very easy to read, easy to grasp the key idea of the author.", "other_comments_or_suggestions": "I wish the impact statement can have more text. Since this is about Safe RL, any improvement may have a positive impact to the society if the algorithm will be deployed at scale in real world.", "questions_for_authors": "1. Is there any particular reason why safety gym is not included as a benchmark? Or why do the authors choose to run the methods on a new benchmark?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e0PAFQDRL6", "reviewer": "Reviewer_urEV", "summary": "The paper proposed MAXSAFE, a safe RL algorithm which aims to maximize the return while minimizing (reduce to zero) the probability of visiting an unsafe state. MAXSAFE is extended based on Q-learning algorithm, and thus is applicable to discrete action MDP problems. The major contribution of this paper is that it outlines the proposed process in masking unsafe actions from the Q-safety critic and provide theoretical guarantee in tabular MDP setting.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the reviewer for the valuable feedback of our work.\n\nQ1: The soundness of our assumption that there exists a sufficiently large policy space with minimal unsafe probability.\n\nWe assume that in the environments we consider, there exists a sufficiently large policy space in which the probability of unsafe events is minimal. Such environments are common in autonomous driving and robotic control. For instance, there are many feasible driving policies that allow a vehicle to operate safely on the road without collisions, as well as a sufficiently large set of policies enabling a robot to navigate without falling or colliding with obstacles—indicating that a large policy space with minimal unsafe probability indeed exists in these environments. This assumption aligns well with many practical scenarios. Building on this assumption, our MAXSAFE policy selects from within this space and aims to optimize reward performance while ensuring safety.\n\nQ2: The proposed method is limited to discrete action spaces.\n\nOur action masking method is currently only suitable for RL tasks with discrete action spaces, as simple action masking strategies may not perform well in continuous settings. We view this as an interesting direction for future work, which would require more advanced techniques to better optimize the trade-off between safety and performance.\n\nNevertheless, the proposed safety Prioritized Experience Replay (safety PER) is a general approach and can also be applied to the learning of the cost critic. To validate this, we run experiments on two environments—SafetyCarCircle and SafetyPointGoal—from the Omnisafe benchmark, https://github.com/PKU-Alignment/omnisafe. The results suggest that the simple SAC-Lag algorithm can benefit from being combined with safety PER to achieve better performance. We use 6 random seeds for evaluation, and we provide both the training curves and the mean/std of episode return and episode cost. The results are available at:https://anonymous.4open.science/r/per_test/results.png, https://anonymous.4open.science/r/per_test/training_curve.png.\n\nQ3: Is a SAC-Lagrangian baseline (modified to handle discrete actions) with different trade-off settings a better baseline?\n\nWe include a variety of off-policy baseline RL algorithms. Since our tasks involve discrete action spaces, we adopt DQN-based off-policy methods. For example, our baseline RCDQN is an off-policy algorithm, and all algorithms with names ending in “DQN” are off-policy methods designed for discrete action settings.\n\nQ4: Usefulness of the SWU score.\n\nWe adopt the evaluation setting from previous literature to demonstrate the trade-off between safety and reward. In our paper, we also report the training results for episode reward and crash rate in **Table 5**.\n\nQ5: Choice of benchmarks\n\nRegarding the concern about why we do not use Safety-Gym as our benchmark suite, the primary reason is that our setting fundamentally differs from that of Safety-Gym. In our framework, safety is treated as the highest priority. Specifically, we assume that an episode terminates immediately upon the execution of any unsafe action (i.e., entering an unsafe region). In contrast, Safety-Gym employs a scalar cost signal and allows temporary safety violations as long as the expected cumulative cost remains within a predefined budget. For example, the “circle” environment in Safety-Gym’s safe navigation task is conceptually similar to the “circle” environment in our benchmarks. However, the underlying assumptions differ: in Safety-Gym, safety is treated as a constraint, and episodes continue despite violations, provided the overall cost remains within budget. In contrast, our formulation enforces strict safety—any violation leads to immediate episode termination. This setting better reflects real-world scenarios where safety is non-negotiable and must take precedence over performance. A formal comparison of the two formulations is provided in the **Preliminaries** section of our paper. We refer the reviewer to that section for detailed definitions. Due to our strict safety assumption, the cost signal in our setting only appears at the end of an episode, resulting in significantly sparser cost feedback. To address this challenge and improve learning efficiency, we introduce Prioritized Experience Replay (PER) in our SPOM_PER method. This design is motivated by the need to balance reward maximization with strong safety guarantees.\n\nWe hope our responses have addressed your concerns and we welcome any further suggestions to improve the paper. If you find our clarifications satisfactory, we would greatly appreciate it if you could consider increasing your score accordingly.", "claims_and_evidence": "1. One assumption is that the policy space is sufficiently large that unsafe probability can be minimized to near zero. Although the paper cited a prior work which also makes this assumption, I'd argue that this is not generally applicable to most safe RL problems where the transition dynamics can be stochastic and hence it might not be feasible to reduce the probability of visiting unsafe states to near zero.  \n\n2. The proposed method is limited to discrete action setting but most of the safe RL problems and benchmarks handle continuous action in general.", "methods_and_evaluation_criteria": "1. The CMDP-based RL selected is based on RCPO (Tessler et al., 2019) which (a) is an on-policy algorithm, (b) dynamically adjusts the safety-reward tradeoff coefficient $\\lambda$ based on the specified level of safety. These two features are not present in MAXSAFE, which is an off-policy algorithm and targets to absolutely minimizes the probability of visiting unsafe states. This makes me think that this selection of baseline might not be valid, considering that on-policy algorithm may need more samples to converge and RCPO also needs to learn the optimal tradeoff. Perhaps an SAC-Lagrangian baseline (modified to handle discrete actions) with different tradeoff setting is a better baseline?\n\n2. I have a little bit of doubt whether condensing both the safety and reward performance figure to a single evaluation metric SWU is useful. It seems to overly simplify the tradeoff into one single number without any statistical confidence value. Perhaps using the more conventional way, e.g. report reward and safety separately with both average and stdev, would be better.", "theoretical_claims": "The theoretical part in the main paper mainly includes the steps to derive the steps to learn the optimal action masks, which has some resemblance to the cited baseline SafeQ (Srinivasan et al., 2020). The steps look sound, do note that I did not carefully check the detailed step-by-step proof in the appendix.", "experimental_designs_or_analyses": "Most of the safe RL literature uses the safety gym / gymnasium or bullet gym for experiments and the chosen environments in this paper: highway-environments and ACC are a little unfamiliar to me.  \n\nPerhaps running experiments in these well-tested benchmark domains would be easier for reviewers to assess.", "relation_to_broader_scientific_literature": "MAXSAFE is relevant to the safe RL work as it provides an algorithm which can learn a safe policy while optimizing reward, although it has some inherent restrictions which might limit its applicability to general safe RL setting.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Using Prioritized Experience Replay is an interesting strength of MAXSAFE as it enhances its capability in handling sparse and catastrophic safety violation setting.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please refer to earlier sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G47Un6ShN6", "reviewer": "Reviewer_bCNL", "summary": "Main findings:\n- The paper introduces MAXSAFE, a chance-constrained bi-level optimization framework for safe reinforcement learning to achieve hard-constraint satisfaction and near-zero costs in sparse cost setting. Particularly, MAXSAFE minimizes the unsafe probability and maximizes the return under safe policies. \n\nMain results:\n- This paper tests their algorithm on diverse autonomous driving and safe control tasks. The MAXSAFE achieves comparable reward-safety trade-off on both benchmarks. \n\nAlgorithmic ideas:\n- The algorithm of this paper contains Q-learning + reachability estimation function (REF) + safety polarization + priority replay experience (PER). In particular, this paper extends REF to state-action REF (SA-REF) and constructs optimal action masks based on polarization function to get rid of the gating operator. Besides, this paper utilizes the TD error of REF for PER to address the sparse cost setting.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the reviewer for the thorough reading of our paper. We address your valuable questions in the following responses.\n\nQ1: Sparse cost challenge\n\nIn our setup, episodes terminate immediately upon safety violations, as we treat safety as a hard constraint. The learner receives no cost feedback until a violation occurs, resulting in at most one non-zero cost per episode—what we refer to as sparse safety cost. This sparsity makes it difficult to learn safety estimations, as the agent must infer early indicators of unsafe outcomes without intermediate cost signals. These challenges motivate our use of PER, which prioritizes unsafe transitions to improve policy safety.\n\nQ2: Question about proving $\\gamma$-contraction\n\nA brief proof here: https://anonymous.4open.science/r/rebuttal-6740/proof_details.png.\n\nQ3: Two more hard-constraint baselines\n\nWe conduct experiments with PPOBarrier on highway benchmarks and find it performs conservatively, with consistently low rewards across all tasks. In contrast, our SPOM and SPOM_PER achieve a better tradeoff. Detailed SWU scores, crash rates, and episode rewards are provided in https://anonymous.4open.science/r/rebuttal-6740/ppo_barrier_table.png, along with training curves in https://anonymous.4open.science/r/rebuttal-6740/ppo_barrier_training_curve.png. As for the second referenced work, its setting assumes access to analytical violation functions, while our SA-REF is learned without any domain-specific knowledge.\n\nQ4: Literature on HJ reachability and REF\n\nWe will include all relevant literature on HJ reachability and REF in a new section of our Related Works. Briefly, early works employed HJ reachability value functions to assess state feasibility, relying on known system dynamics and numerical methods. Some studies approximate unknown dynamics using Gaussian Processes (Zhang, W. et al. 2023)or symbolic regression (Wang, Y. et al. 2024). Once reachability is computed, the state space is partitioned into feasible/infeasible regions to guide policy optimization (Zheng, Y. et al. 2024). Other works focus on formal safety verification of DRL systems (Dony, Y. et al. 2024). However, HJ-based value functions are not well-suited for stochastic MDPs during RL training. To address this, REF estimates unsafe probabilities via backward reduction (Ganai, M. et al. 2024). Our work extends REF to the state-action level, enabling state-dependent action masking to reduce safety violations.\n\nQ5: Literature on hard-constraint safe RL\n\nThanks for the suggestions. We indeed discuss hard-constrained safe RL in our related works. Based on policy optimization strategies, we categorize prior works into: action correction and action masking. Action correction modifies actions after they are proposed (e.g., via projection), while action masking alters the action distribution directly. The papers mentioned by the reviewer largely fit within this framework and will be incorporated into our Related Works. For instance, Tan,D. et al. 2024 uses value functions as control barrier functions in a shielding setup, aligning with action correction. Yang,Y. et al. 2023 learns certificates that softly penalize unsafe actions, guiding policies toward safer regions—similar in spirit to action masking. Suttle, W. et al. 2024 introduces truncated sampling from state-dependent safe action sets, which we view as a general form of action masking. We will include all the literature in our paper for completeness.\n\nQ6: How does the “off-policy” affect the theoretical analysis?\n\nWe assume that actions are sampled from a behavior policy $\\pi_b$ in **Section 4.2**, which is a common assumption in Q-learning to ensure sufficient exploration of the state-action space. Our theoretical analysis follows the classical Q-learning framework: we prove the $\\gamma$-contraction property of our operator and demonstrate convergence in tabular MDPs. This directly parallels traditional Q-learning theory.\n\nQ7：Gating operator\n\nThe gating operator in Section 5.1 represents our hard masking rule. In **Theorem 4.4**, we prove its convergence in tabular MDPs. When extending this method to deep Q-learning, we found that using Equation (17) as the objective yields better empirical performance. We report the ablation results in **Section 6.3**, where ‘OAM’ refers to Optimal Action Masking, alongside other polarization function choices.\n\nQ8: Key terms\n\n•Maximal safety: A policy that achieves the lowest probability of safety violations.\n\n•Safety polarization: The use of our polarization function to mask actions with relatively high SA-REF values.\n\n•REF: can be regarded as the probability for the current state to reach unsafe regions under the current policy.\n\nWe hope our responses have addressed your concerns and we welcome any further suggestions to improve the paper. If you find our clarifications satisfactory, we would greatly appreciate it if you could consider increasing your score accordingly.", "claims_and_evidence": "This paper claims (1) highly consequential cost and (2) sparse cost settings as a challenge. I agree that claim (1) is important, which is equivalent to the generally studied hard-constraint problem. However, I am confused why sparse cost is a challenge. For the reward, we have sparse reward settings because we want to maximize the cumulative reward and sparse reward settings indicate the environment cannot give valid reward signals. As for the cost, sparse cost settings indicate most of the cases are safe, which enlarges the feasible region of safety. This setting actually simplifies the problem in my view. \n- Full cost -> No feasible region; Dense cost -> small feasible region; Sparse cost -> Large feasible region; No cost -> Full feasible region (common RL).", "methods_and_evaluation_criteria": "1.\tYes, the proposed methods and evaluation criteria make sense for the problem. However, the benchmark is a little old. For autonomous driving, here is a survey about open-source simulators. The highway-env is categorized into “driving policy simulator” in reference [1]. However, lots of newer simulators from 2020 to 2023 are also listed.\n2.\t[1] Li, Y., Yuan, W., Zhang, S., Yan, W., Shen, Q., Wang, C., & Yang, M. (2024). Choose your simulator wisely: A review on open-source simulators for autonomous driving. IEEE Transactions on Intelligent Vehicles. https://ieeexplore.ieee.org/abstract/document/10461065.", "theoretical_claims": "In Appendix B.1., |max⁡ Q_1-max⁡ Q_2 |≤max |Q_1-Q_2 | is not obvious. I think it is better to give more details or hints.", "experimental_designs_or_analyses": "The baselines for comparison in experimental designs do not satisfy the soundness/validity. This paper claims on sparse cost and high consequential (safety critical) tasks. For this kind of tasks, lots of hard-constraint methods have been proposed in safety-critical RL. However, most of the baselines (except for RESPPO in Appendix) in this paper are soft-constraint RL methods, whose objective is to satisfy the constraint in expectation rather than hard constraint. I think this paper should add some hard-constraint RL methods (such as [1] and [2]) for comparison. \n\n---\n[1] Yang, Y., Jiang, Y., Liu, Y., Chen, J., & Li, S. E. (2023). Model-free safe reinforcement learning through neural barrier certificate. IEEE Robotics and Automation Letters, 8(3), 1295-1302. Code link: https://github.com/jjyyxx/srlnbc\n\n[2] Zhao, W., He, T., & Liu, C. (2023, June). Probabilistic safeguard for reinforcement learning using safety index guided gaussian process models. In Learning for Dynamics and Control Conference (pp. 783-796). PMLR.", "relation_to_broader_scientific_literature": "1.\tThis paper introduces CMDP and Action-correction-based Safe RL, which are two types of baselines in experiment. Besides, Action Masking for Safe RL is also introduced as the foundation for their method.\n2.\tPrioritized Experience Replay: This paper introduces PER, which is the foundation for their proposed Safety PER.", "essential_references_not_discussed": "Yes, there are lots of related works that are essential for this paper but are not cited/discussed in this paper.\n\n1. Hamilton-Jacobi reachability and reachability estimation function (REF) in Safe RL. As the most important contribution of this paper, these concepts are not introduced or discussed until they propose SA-REF. For example, some literatures include but not limited to [1], [2], [3], [4], and [5]. It is better to discuss this topic in related work and then explain the concept in Preliminaries.\n---\n[1] Ganai, M., Gao, S., & Herbert, S. (2024). Hamilton-jacobi reachability in reinforcement learning: A survey. IEEE Open Journal of Control Systems.\n\n[2] Wang, Y., & Zhu, H. (2024, July). Safe Exploration in Reinforcement Learning by Reachability Analysis over Learned Models. In International Conference on Computer Aided Verification (pp. 232-255). Cham: Springer Nature Switzerland.\n\n[3] Zhu, K., Lan, F., Zhao, W., & Zhang, T. (2024). Safe Multi-Agent Reinforcement Learning via Approximate Hamilton-Jacobi Reachability. Journal of Intelligent & Robotic Systems, 111(1), 7.\n\n[4] Dong, Y., Zhao, X., Wang, S., & Huang, X. (2024). Reachability Verification Based Reliability Assessment for Deep Reinforcement Learning Controlled Robotics and Autonomous Systems. IEEE Robotics and Automation Letters, 9(4), 3299-3306.\n\n[5] Zheng, Y., Li, J., Yu, D., Yang, Y., Li, S. E., Zhan, X., & Liu, J. (2024). Safe offline reinforcement learning with feasibility-guided diffusion model. arXiv preprint arXiv:2401.10700.\n\n2. Hard-constraint RL: By now, many researches have studied hard-constraint RL (safety-critical RL; near-zero constraint; persistently safe)\n---\n[1] Yang, Y., Jiang, Y., Liu, Y., Chen, J., & Li, S. E. (2023). Model-free safe reinforcement learning through neural barrier certificate. IEEE Robotics and Automation Letters, 8(3), 1295-1302.\n \n[2] Zhao, W., He, T., Li, F., & Liu, C. (2024). Implicit Safe Set Algorithm for Provably Safe Reinforcement Learning. arXiv preprint arXiv:2405.02754.\n\n[3] Suttle, W., Sharma, V. K., Kosaraju, K. C., Seetharaman, S., Liu, J., Gupta, V., & Sadler, B. M. (2024, April). Sampling-based safe reinforcement learning for nonlinear dynamical systems. In International Conference on Artificial Intelligence and Statistics (pp. 4420-4428). PMLR. \n\n[4] Zhao, W., He, T., & Liu, C. (2023, June). Probabilistic safeguard for reinforcement learning using safety index guided gaussian process models. In Learning for Dynamics and Control Conference (pp. 783-796). PMLR. \n\n[5] Wei, H., Liu, X., & Ying, L. (2024, March). Safe reinforcement learning with instantaneous constraints: the role of aggressive exploration. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 19, pp. 21708-21716).\n\n[6] Gu, S., Sel, B., Ding, Y., Wang, L., Lin, Q., Knoll, A., & Jin, M. (2025). Safe and balanced: A framework for constrained multi-objective reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence.\n\n[7] Tan, D. C., McCarthy, R., Acero, F., Delfaki, A. M., Li, Z., & Kanoulas, D. (2024, August). Safe Value Functions: Learned Critics as Hard Safety Constraints. In 2024 IEEE 20th International Conference on Automation Science and Engineering (CASE) (pp. 2441-2448). IEEE.", "other_strengths_and_weaknesses": "1.\tStrengths\n- It extends REF to state-action based, which is more detailed for reinforcement learning.\n- The Theoretical proofs are in detail.\n- The appendix is very detailed.\n\n2.\tWeaknesses\n- Although Theoretical proofs are in detail, there is a lack of explanation of the derivation process. \n- The sparse cost settings are not convincing. Sparse cost settings indicate that most of the cases are safe, which enlarges the feasible region of safety. This setting actually simplifies the problem.", "other_comments_or_suggestions": "1.\tSome key terms are not explained, at least not clearly, such as “maximal safety”, “safety polarization”, “REF” .\n2.\tBetter to add a section about “Hamilton-Jacobi reachability and reachability estimation function (REF) in Safe RL” in related work.\n3.\tAdd a section about hard-constraint safe RL in related work.\n4.\tExplain the formulation of reachability and REF in Preliminaries\n5.\tThe space in line 101-102 at Page 2 is incorrect.", "questions_for_authors": "Q1: How does the “off-policy” affect the theoretical analysis since the analysis procedure follows just one policy π but off-policy RL has different behavior and target policies? (RESPPO is an on-policy algorithm, so it has no such problem.)\n\nQ2: What is the purpose of introducing the gating operator in Section 5.1. and then getting rid of it in Equation (17)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The MAXSAFE framework introduces a promising chance-constrained bi-level optimization approach for safe reinforcement learning with novel components including optimal action masks, safety polarization, and safety prioritized experience replay. While reviewers noted concerns about the policy space assumption (BcoJ, urEV) and baseline selection methodology (bCNL), the authors provided adequate justifications and implemented improvements including additional baselines and statistical validation. The paper focuses on specific application domains where the assumptions are reasonable, and the SWU score demonstrates strong overall performance in reward-safety trade-offs. The current restriction to discrete action spaces represents a reasonable scope limitation rather than a fundamental flaw. Though one reviewer maintained a weak reject position (urEV), another recommended acceptance (dGwp), and a third moved to weak accept after rebuttals (bCNL). On balance, the paper's novel bi-level optimization approach makes a valuable contribution to safe reinforcement learning despite limitations that could be addressed in future work.
Decision: Accept (poster)