Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking
Authors: Hanna Krasowski
Jakob Thumm
Marlon Müller
Lukas Schäfer
Xiao Wang
Matthias Althoff
Venue: Accepted by TMLR ()

Abstract: Ensuring the safety of reinforcement learning (RL) algorithms is crucial to unlock their potential for many real-world tasks. However, vanilla RL and most safe RL approaches do not guarantee safety. In recent years, several methods have been proposed to provide hard safety guarantees for RL, which is essential for applications where unsafe actions could have disastrous consequences. Nevertheless, there is no comprehensive comparison of these provably safe RL methods. Therefore, we introduce a categorization of existing provably safe RL methods, present the conceptual foundations for both continuous and discrete action spaces, and empirically benchmark existing methods. We categorize the methods based on how they adapt the action: action replacement, action projection, and action masking. Our experiments on an inverted pendulum and a quadrotor stabilization task indicate that action replacement is the best-performing approach for these applications despite its comparatively simple realization. Furthermore, adding a reward penalty, every time the safety verification is engaged, improved training performance in our experiments. Finally, we provide practical guidance on selecting provably safe RL approaches depending on the safety specification, RL algorithm, and type of action space.

Published in Transactions on Machine Learning Research (11/2023)
Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking
Hanna Krasowski
mnnna..rasouski@tmm.. de jakob.thumm thumm &tamn de marlon marlon.maeller-eee mueller de sokas.schaefees Olum. de ziao. wwng de althoff@tum. de
Jakob Thumm
Marlon Müller
Lukas Schäfer
Xiao Wang
Matthias Althoff
School of Computation, Information and Technology
Technical University of Munich
Reviewed on OpenReview: https: // openreview. net/ forun? id= mcNDexbna0
Abstract
Ensuring the safety of reinforcement learning (RL) algorithms is crucial to unlock their potential for many real-world tasks. However, vanilla RL and most safe RL approaches do not guarantee safety. recent years, several methods have been proposed to provide hard safety guarantees for RL, which is essential for applications where unsafe actions could have disastrous consequences. Nevertheless, there provably safe RL methods. Therefore, we introduce categorization of existing provably safe RL methods, present the conceptual foundations for both continuous and discrete action spaces, and empirically benchmark existing methods. We categorize the methods based on how they adapt the action: action replacement, action projection, and action masking. Our experiments on an inverted pendulum and a quadrotor stabilization task indicate that action replacement the best-performing approach for these applications despite its comparatively simple realization. Furthermore, adding a reward penalty, every time the safety verification is engaged, improved training performance in experiments. Finally, provide practical guidance on selecting provably safe RI. approaches depending on the safety specification, RL algorithm, and type of action space.
comprehensive comparison of these
1 Introduction
Reinforcement learning (RL) contributes many recent advancements in challenging research fields such robotics (El-Shamouty al., 2020; Zhao et al., 2020), autonomous systems (Kiran et al., 2022; Ye et al., 2021), and games (Mnih et al., 2013; Silver et al., 2017). A vanilla RL agent typically explores randomly and executes undesired actions multiple times to learn how to achieve the highest possible reward. However, safety is important for many applications. Therefore, safe RL emerged where the learning process is adapted such that the agent considers safety aspects next to performance during training and/or operation (García & Fernández, 2015). There are different degrees of how safety is considered for safe RL approaches. First, some approaches only incorporate safety aspects without formal guarantees. Here, the agent chooses safe actions with higher probability, e.g., by adding reward component that indicates risk or adapting the exploration such that the agent follows safe heuristic. Second, some approaches provide probabilistic safety guarantees, e.g., by using probabilistic models for safe actions (Könighofer et al., 2021). Still, hard safety guarantees for RL agents are necessary whenever failures are disastrous and need to be avoided at all costs during training and deployment. Such safety-critical applications include autonomous driving, human-robot. collaboration,
FEqual contribution.

---

Published in Transactions on Machine Learning Research (11/2023)
energy grids. We refer to this third subcategory of safe RL methods providing hard safety guarantees for both training and operation as provably safe RL.
We provide for the first time a consistent conceptual framework for provably safe RL in both continuous and discrete action spaces, a comprehensive literature survey, and a comparison between provably safe RL approaches on two widely used control benchmarks. The characteristic difference between provably safe RL approaches is how they adapt the actions of the agent. Therefore, we propose classifying them into three categorics: action replacement, action projection, and action masking. Our proposed categorization of provably safe RL provides a concise presentation of the research field, supports researchers implementing provably safe RL through clear terminology and a comprehensive literature review, and outlines ideas for future research within the three categories. Furthermore, we evaluate the methods experimentally. Our three main findings of our experiments were that all provably safe RL methods were indeed safe, that action replacement performed best on average over five tested RI. algorithms, and that adding a penalty to the reward when using the safety function further improved performance.
Our contributions are fourfold. First, we introduce comprehensive classification of provably safe RL methods and their formal description. This categorization allows us to compare and benchmark the effects of choosing a specific type of action modification on the ability of agents to learn. Second, we propose the first formulation of action masking for continuous action spaces. Third, we provide a structured and comprehensive survey of previous provably safe RL works and assign them to the three categories. Finally, we are the first to evaluate the performance of all three provably safe RL methods on two common control benchmarks. This comparison provides insights into the strengths and weaknesses of the different provably safe RL approaches and allows to provide advice on selecting the best-suited provably safe RL approach for specific problem independent of the safety verification method used.
The remainder of this paper is structured as follows. First, we briefly review the historical development of safe RL and provably safe RI. in Section 1.1. We describe preliminary concepts in Section 2 and introduce our proposed categorization. Then, we show how the related provably safe RL literature fits our categorization in Section 3. Section 4 compares the different provably safe RL categories experimentally on a two-dimensional (2D) quadrotor stabilization task. Section 5 discusses the results of our experimental evaluation and the practical considerations following them. Finally, we conclude this work in Section 6.
1.1 Evolution towards provably safe RL
The notion of risk and safety in RL is discussed at least since the 1990s (Heger, 1994). The reasons for combining safety and RL were to focus the learning relevant or safe regions and improve the convergence speed. Thus, the field of safe RL started developing, and in 2015, Garcia & Fernández (2015) were the first to cluster safe RL. They provide two high-level categories: approaches that modify the optimization criterion and approaches that modify the exploration. Since 2015, significant advances in model-free RL and the increased applicability of deep RT. changed the research focus of safe RT. Notably, the higher efficiency of free deep RL made its real-world application tangible and amplified the need for formal guarantees in safe RL. This is also apparent from the survey by Brunke et al. (2022), who investigated recent developments at the intersection of control and learning for safe robotics. As a goal of the broader field of safe learning for control, they identify methods with as little as possible system knowledge while ensuring formal safcty guarantees (Brunke et al., 2022, Fig. 4). Among existing safe RL approaches, provably safe RL research is a growing field located at this frontier as it provides hard safety guarantees during both learning and deployment. While a few papers mentioned Brunke et al. (2022) are part of provably safe RI, it is not a focus of their work. In the following paragraphs, we use the common classification by safety specification type, which can be soft constrainis, probabilistic guarandees, and hard guarantees, to locate provably safe RL in the field of safe RL.
Soft constraints Approaches with soft constraints consider safety directly in their optimization objective so that the agent can explore all actions and states regardless of safety. Thus, these methods can be unsafe during training, especially initially, but usually converge to a safer policy without formal safety guarantocs after sufficiently many training steps. The simplest way to inform an RL agent about safety constraints is through its reward function. Despite its elegance, the reward function approach has many potential pitfalls.
2

---

Published in Transactions on Machine Learning Research (11/2023)
First, the reward function might be ill-defined, either from manual tuning when learned from human input. When manually defined, the reward function might overlook certain features or fine details, leading to a hackable reward (Skalse et al., 2022) from which the agent learns an unsafe behavior. Learning the reward function from human feedback (Christiano et al., 2017) is also error-prone because communicating safety constraints alongside performance metrics hard for sparse, nonlinear, conditional, or seldom occurring constraints. Second, even if the reward function defined correctly, the trained policy is not guaranteed to be safe, e.g., it was shown by Packer et al. (2018) that RL agents struggle with out-of-distribution states during deployment. Third, the agent might learn to perform actions safely but ignore the task objective due to goal misgeneralization (Langosco et al., 2022). Still, the majority of safe RL research considers safety aspects as soft constraints, so we provide a short overview of soft constraint methods in the following paragraph.
To reduce the burden of manual reward specification, a recent line of work formalizes the task and its safcty specifications as a temporal logic formula. Then, the temporal logic formula is transformed into the RL reward either by directly using the robust measure associated with the formula as the reward (Aksaray et al., 2016; Li et al, 2017; Varnai & Dimarogonas, 2020) or by transforming the temporal logic formula into an automaton that generates the reward (Camacho et al., 2019; Hahn et al., 2019; Cai et al., 2021; Hasanbeig et al., 2022: Alur et al., 2023). For some algorithms, it can even be ensured that the policy converges to the optimal policy, which maximally satisfies the temporal logic specification (Alur et al., 2022; Yang et al., 2022). Another way to inform an RL agent about safety than through the reward is by formulating a constrained optimization problem. Many recent advances have been made in constrained RL (Altman, 1998; Achiam et al., 2017; Stooke et al., 2020), for which the policy aims to maximize the reward while satisfying user-defined specifications. The specifications can be formulated as constraint functions (Chow et al., 2018; Stooke et al., 2020; Yang et al., 2020; Marvi & Kiumarsi, 2021) or as temporal logic formulas (De Giacomo et al., 2021; Hasanbeig et al., 2019a;b; 2020). The main advantage of soft constraint methods over probabilistic or hard constraint methods is that no explicit model of the agent dynamics or the environment is required as the agent learns the safety aspects through experience. Thus, such safe RL methods have a high potential in non-critical settings, where unsale actions do not cause major damage.
Probabilistic guarantees Probablistically safe RL, approaches rely on probabilistic models or synthe- size a model from sampled data. Here, the action and state space can be restricted based on probabili- ties. Nonetheless, unsafe actions are sometimes not detected and might occur occasionally. Several works (Turchetta et al., 2016; Berkenkamp et al, 2017; Mannueci et al., 2018) try to determine the maximal set of safe states by starting from an often user-defined conservative set and extending it with the gathered lcarning experience. Other methods (Könighofer et al., 2021; Thananjeyan et al., 2021; Dalal et al., 2018; Zanon & Gros, 2021; Yang et al., 2021; Gillula & Tomlin, 2013) are based on formulating probabilistic models that identify the probability of safety for an action. In general, approaches that rely on probabilistic methods are especially applicable one cannot bound measurement errors, modeling errors, and disturbances by sets.
Hard guarantees Provably safe RL features hard safety guarantees, which are fulfilled by integrating prior system knowledge into the learning process. Here, the agent only explores safe actions and only reaches states fulfilling the safety specifications. Provably safe RL already fulfills the given safety specifications during the learning process, which is essential when training fine-tuning agents safety-critical tasks in the physical world. Thus, we exclude approaches that only verify learned policies (Bastani et al., 2018; Schmidt et al., 2021) from our survey. We locus on model-free RL algorithms that do not explicitly learn or use a model of the system dynamics to optimize the policy. Generally, deploying learned controllers in the physical world became increasingly realistic recent years, and thus, the need for provably safe RL grew, and more provably safe RL approaches were developed. With this work, we aim to structure and provide practical insights into this growing field.
3

---

Published in Transactions on Machine Learning Research (11/2023)
2 Conceptual analysis
We introduce three provably safe RL classes by providing their formal description in one comprehensive conceptual framework. This framework clarifies the differences between the three classes and eases the following literature review and benchmarking.
Markov decision process The RL agent learns on a Markov decision process (MDP) that is described by the tuple (8,A,T,7,1) Hereby, we assume that the set of states S is fully observable with bounded precision. Partially observable MDPs can be handled using methods like particle filtering (Sunberg & Kochenderfor, 2018) and are not further discussed in this work. The action space A and state space S can be continuous discrete. T(s, als,a,s)) is the transition function, which in the discrete case returns the probability that the transition from state s to state s' occurs by taking action a. In the continuous case, T(s, a, s') denotes the probability density function of the transition. We assume that the transition function is stationary over time. For each transition, the agent receives a reward r: S x A - R from the environment. The discount factor 0 < y < 1 weights the relevance of future rewards. The policy or value function that the action learns can be optimized for an infinite or finite episode horizon p.
Safety of a system For provably safe RL, it is required that the safety of states and actions is verifiable. Otherwise, no formal claims about the safety of a system can be made. Thus, we first introduce the set of safe states S. S S containing all states for which all safety specifications are fulfilled.. For verifying the safety of actions, we use safety function gp: :p:S. S x A + 10
1, if (s, a) is verified safe 0, otherwise.
((.
(1)
Conceptually, this is mainly done by apr-approxinnttin the set of states that are reachable by taking action a in state 8, and then validating if the reachable set of states is a subset of S, and if all these reachable states are verified safe until the episode horizon p. In other words, for each of these reachable states, there exists at least one action that keeps the system within S, until episode termination. To formalize this concept, we define the set of provably safe actions As(s) = {alp(s, a) = =1 for a given state s. The set of provably safe actions (s) is subset of all safe actions² (s)s), ie., Ap(s) C (s)s) A. The safe action set (s) includes all safe actions, while the provably safe action set Ap(s) only includes actions that are verified as safe by the safety function a(s.a). In other words, the safety function possibly returns that an action is unsafe, which is indeed safe, while it never predicts truly unsafe actions to be safe. Moreover, we define the set of provably safe states based on the safety function: S.p = {s]3a E A, ppss a) = 1] S Sg. Consequently, for a verified state-action tuple all reachable next states s' are in S.p. All provably safe RL approaches rely on the availability of provably safe actions and states to achieve a provably safe system:
Lel the system be inilinted in a provably safe state By ESp. Then, there exists a sequence of provably safe actions that ensures s E Sg at all times until the episode horizon p.
Proposition 1
The proof can be casily obtained from the definitions by induction as ((s, a) = s' E ^ AA + V. Note that if the episode horizon is finite, the last state of an episode is verified safe if it is contained in S.. A provably safe action must not necessarily exist for this last state.
We define S, relatively broad since the safety specifications are usually task-specific and can take various forms such as stability, not entering a lime-invariant or Lime-varying unsafe set, and temporal logic specifica- tions. Depending on the safety specification and system under consideration, different verification mots are applicable and encoded in the safety function (ps,, a). We discuss the concrete verification methods used by previous works in Section 3. Although spss, a) only verifies safety for a given state and action a, it may take more than the next. state into account. To make this more graspable, we shortly explain two concepts for
The state space often augmented from the state space for classical control purposes to state space that includes other safety-relevant dimensions. 2Noee that. "taking action" commonly considered to be part of the action space, most often with the action (0.....''
4

---

Published in Transactions on Machine Learning Research (11/2023)
Mask
#t++
next state
Environment
Agent
Avp
Ft
reward
0x
Environment
Agent
Og
action
Action replaeement/projcciioon
safe action
(a) Structure action replacement
projection
(b) Structure action masking
A
A
A action set
Ay
As sufe action
a
Ap provably safe action
replaced/rroeeteed action
(c) Action replacement
(d) Action projection
Action masking
Figure 1: Structure of the three types of provably safe RL methods. The post-posed action replacement or projection methods (a) alter unsafe actions before sending them to the environment. In contrast, preemptive action masking approaches (b) allow the agent to only choose from the safe action space and, therefore, only output safe actions to the enviromment. Figures (c-e) highlight the differences belween the three approaches in the action space. Here, action replacement (c) replaces unsafe actions with actions from the safe action space, action projection (d) projects unsafe actions to the closest safe action, and action masking (e) lets the agent choose solely from the safe action set.
calculating the provably safe states set S.p. For our benchmarks in Section 4, we compute a control invariant set as Sp. Thus, the online verification only consists of checking that all reachable states are contained in this control invariant set. Another concept is predicting the reachable states until a specified time horizon while starting from and applying a for the first time step and an action sequence for the consecutive time steps. The verification checks that all reachable states are in & and that either the reachable set the prediction horizon is contained in safe terminal set or the prediction horizon is the episode horizon. Tf the verification succeeds, we know that E S.p.
Provably safe RL relies on model knowledge to provide safety guarantees, i.e., conformant model that covers the salety-relevant system and environment dynamics. Hereby, the verification process can use an abstraction of the real system dynamics as long as is conformant (Rochm et al., 2019; Liu et al., 2023) to the real system, i.e., over-approximates both aleatoric and epistemic uncertainties and covers all relevant safety aspects. This eases efficient verification, as the complexity of the abstraction is usually significantly lower than the complexity of the underlying MDP. In systems where such safety model is unavailable, provably safe RL is not applicable, and only non-provably safe approaches, as discussed in Section can be used. In practice, the safety specifications are often weakened to legal or passive safety. Hereby, inevitable safety violations caused by other agents are not considered to be the fault of the agent and are, therefore, not considered unsafe. Examples of proving legal safety have been presented for autonomous driving (Pek et al., 2020) and robotics (Bouraine al., 2012).
There are multiple ways ensure provable safety for RL systems, which we summarize in three categories: action replacement, where the safety method replaces all unsafe actions from the agent with safe actions, action projection, which projects unsafe actions onto the safe action space, and action masking, where the agent can only choose actions from the safe action space. We choose this categorization, as it represents the three main approaches found in the literature lo modily actions and thereby ensure safety for RL. Action replacement and action projection alter the action after the agent returns it. In contrast, action masking lets the agent exclusively choose from the safe action space. Figure displays the basic concept of these mothods. The following subsections describe the concept, mathematical formalization, and practical implications of each approach.
5

---

Published in Transactions on Machine Learning Research (11/2023)
2.1 Action replacement
The first approach to ensure the safely of actions is to replace any unsafe action returned by the agent with a safe action before its execution. The first step of action replacement is to evaluate the safety of the suggested action a E A using p(s, a). If the action sampled from the policy (Taas) is not verified as safe, it is replaced with a provably safe replacement action = <b(s,, where sb * - Aup is called replacement function. Following this procedure, it guaranteed that only safe actions a with
a = fa~ntaaa T(ass), if ((s =1
(2)
(b))). otherwise
are executed. We discuss how this action replacement alters the MDP in the Appendix and additionally refer the interested reader to Hunt et al. (2021).
There are two general replacement functions found in the literature, sampling and failsafe. In sampling, the replacement function (sample uniformly samples random action from Ap(s). The other approach is to use a failsafe controller (ttiisafe as replacement action, which could also stem from human leedback. In time-critical and complex scenarios, where building As(s) online becomes too time-consuming, Vaaalsafe is the only fcasible option.
2.2 Action projection
In contrast to action replacement, where the replacement action is not necessarily related to the action of the agent, action projection returns the closest provably safe action with respect to the original action and some distance function. For this, we define the optimization problem
ā arg min dist (a, à)
(3)
à subject to a(s,a)= 1,
where dist(-) describes an arbitrary distance function, c.g., a p-norm. Note, that it might not be possible to define such a distance function, especially in discrete action spaces. The constraints are often defined explicitly by n constraint functions fi(a, s) S O, Vi E I....,n that confine the next state to the set of provably safe states, i.., s' E S... The optimization problem (3) minimizes the alteration of the actions while satisfying the safety specification, which is usually expressed through constraints for the optimization problem. Following Proposition 1, the optimization problem in (3) must always be feasible.
The most prominent ways to formulate the safety constraints for action projection are based on control barrier functions (CBFs) or robust model predictive control (MPC). For the first method, the constraints are defined by CBFs (Wieland & Allgöwer, 2007) that translate state constraints to control input constraints. We formulate the CBFs according to Taylor et al. (2020) as it is an intuitive formulation for RL. Consider nonlinear control-affie system
s= m(s) b(s) a.
(4)
where 8 E S C N is the continuous state with N dimensions, and à E A C RM is the continuous control input with M dimensions and m(s) and b(s) are locally Lipschitz continuous functions. Then, the function h is a CBF there exists an extended class function a such that (Wabersich et al., 2023, Eq. 10)
Vh(s)(m( b(s)a) 2 a(h(s))
(5)
If the system dynamics are not exactly known, the nominal model can be extended with bounded disturbances d to model the unknown system dynamics:
$ = + ā +d. d.
(6)
To reduce consorvatism, disturbances can be modeled as state and input-depended d(s,, ä), and the maximal occurred disturbance can be learned from data as presented Taylor et al. (2021). The limitation to control- affine systems makes formulating the constrained optimization problem efficient, e.g., for Fuclidean norm
6

---

Published in Transactions on Machine Learning Research (11/2023)
the distance function, (3) results in quadratic program. A downside of using CBFs is that h(s) is not trivial to find, especially in environments with dynamic obstacles.
For the second common projection method, we formulate the optimization problem with MPC according to Wabersich Zeilinger (2021). There, the constraint in (3) is satisfied we find an action sequence that steers the system from the current state 8 into the safe terminal set M. within a finite prediction horizon I. N while respecting input and state constraints, which reflect the safety specification (Wabersich & Zeilinger, 2021, Eq. 5):
arg min à subject to
dist (a, a)
(7)
81+1 g(s), ai), Su = s,
VI E { 1} :11- E Sa
81 E M,
1I E (0, (O...., 1- 1) a: E.E.A
== do.
where sı and a are the predicted state and action 1 steps ahead of the current time step. The function ( is obtained by time discretizing a smooth contimuous-time nonlinear system, whose dynamics are governed by = f(s, The safe terminal set M S S is control invariant, i.c., after the agent has entered M, the associated invariance--nforcinng controller keeps the agent inside this set indefinitely. If the optimization problem is solvable, à is executed. If not solvable, the control sequence from the previous state is used backup plan until the safe terminal set is reached or the optimization problem is solvable again (Schürmann et al., 2018). For perturbed systems of the form å [(s, a, d) with a bounded disturbance d, robust MPC schemes, e.g., Schürmann et al. (2018), have to be employed and output-feedhack MPC schemes, e.g., Gruber & Althoff (2021), account for measurement uncertainties. Similar to the CBF approach, conservatism can be reduced by learning the disturbance bounds from data (Hewing et al., 2020). Note that, for an environment with dynamic obstacles, the safe terminal set can be Lime-dependent, and we are unaware of a straightforward integration where Proposition still holds.
2.3
Action masking
The two previous approaches modify unsafe actions from the agent. In action masking, we do not allow the agent to output an unsafe action in the first place (preemptive method). Hereby, a mask is added to the agent so that it can only choose from actions in the provably safe action set. In addition to Proposition 1, action masking in practice requires an efficient function n)s) S - P(A), where P denotes the powerset, that determines sufficiently large set of provably safe actions Aup C A for a given state 8. The policy function π is informed by the function (s)) and the action selection is adapted such that only actions from Ao can be selected:
(8) If (s)) can only verify one or few actions efficiently, the agent cannot Iearn properly because the agent cannot explore different actions and find the optimal one among them. Ideally, the function 17(8) achieves A. Ag.
a a~ a~)(aa) 8) Anp.
The action masking approaches for discrete and continuous action spaces are not casily transferable into each other, and will therefore be discussed separately in this subsection. For discrete actions, the safety of each action is typically verified in each state using pls, a) and all verified safe actions are added to (s).). i.e., 7) iterates over all actions for the current state with vess, a) to identify (s)s). Intuitively, the discrete action mask is an informed drop-out out layer added at the end of the policy network. We define the resulting safe policy n. (als) based on Huang & Ontañón (2022, Fq. 1) as
π (a(s)
π Tm (als)) = ((8 a) Ea (a 8)
We omitted (7) that 81, &1, ai a 1 are decision variables of optimization problem improve the readability.
7

---

Published in Transactions on Machine Learning Research (11/2023)
The integration of masking in specific learning algorithm is not trivial. The effects on policy optimization methods are discussed in Krasowski et al. (2020); Huang & Ontañón (2022). For RT. algorithms that learn the Q-function, we exemplary discuss the effects of discrete action masking for deep Q-network (DQN) (Mnih et al., 2013), which is most commonly used for Q-learning with discrete actions. During exploration with action masking, the agent samples its actions uniformly from A.p. When the agent exploits the Q-function, it chooses only the best action among Aug i.e.; arg max Q(s, a). The temporal difference error for updating the Q-function Q(s, a) is (Mnih et al., 2013, Eq. 3)
r(s, + max Q( -(())))saa))))
(10)
where the action in the next state is a' E Aop in contrast to the vanilla temporal difference error where the maximum Q-value for the next state is searched among actions from A. Using the adapted temporal difference error in (10), the learning updates are performed only with Q-values of actions relevant in the next state instead of the full action space.
To comprehensively compare the different provably safe RL approaches discrete and continuous action spaces in Section 4, we propose a simple formulation for continuous masking since there is no existing approach. We formulate this form of continuous action masking as a transformation of the action of agents to the provable safe action set. Our approach requires both A and App to be axis-aligned boxes with the same center. We propose to transform the action space A into An by applying the transformation
ā ā= (a min(A)) max max min() max A)) min(A)
+ min(App
(11)
to the actions a E A where min(-) and max(-) return a vector containing the minimal and maximal value of the given set in each dimension respectively, and all operations are evaluated clement-wise. For example, given a two-dimensional continuous action space A [0, 1] x [-1, then min(A) = [0, -1]_ Note that the representation of Ap as axis-aligned box centered in A can be ander-approximative and, thus, lead to conservative behavior. To overcome this limitation, more complex set representations, such as the zonotopes (Althoff et al., 2021), for the action spaces (A and Ap) in combination with solving an optimization problem that maximizes the size of App could be investigated. A less sophisticated yet possibly effective approach is searching for a good latent interval representation of and transformation to Ag by applying principal component analysis to set of Ap for different states as pre-computing step. Since the action spaces for RL are defined a priori, there must always be a valid transformation from a to à E Ap and such that the operation time-invariant for all state-action pairs. In the next section, we discuss the effect of the three provably safe RL approaches on the policy distribution and exploration.
2.4 Impact on the distribution of actions
The three previously presented provably safe RL methods have different effects on the resulting distribution of actions, illustrated for one-dimensional continuous action space and probabilistic policy in Figure 2. For action projection, all actions that are not verified safe & As are projected to the boundary of the provably safe action set DAy. Therefore, actions on DAs are disproportionately explored compared to the interior of Ap. A similar effect can occur in action replacement depending on the replacement strategy, e.g,, with (s)),, the failsafe action is explored more often. However, the random sampling strategy Usample( (s) is used, as shown in Figure 2 (a), the likelihood of all safe actions being explored increases equally. The sampling strategy, therefore, fosters exploration and discourages exploitation, as all non-provably safe actions lead to uniformly distributed exploration in the safe action space. The distribution of actions for both action replacement and projection differs from the distribution of the current policy, which might be problematic for on on-policy algorithms. In action masking, we only map the exploration from A to Ap. Thus, the exploration strategy is not affected by action masking. In this aspect, our approach in (11) is conceptually similar to action normalization, which is commonly used in RL (Sutton & Barto, 2018, Ch. 16.8)
8

---

Figure 2: Idealized impact of the provably safe RL methods on the probability density function of the RL policy for a single action in a given state. The probability density function of the original RL policy π (als) and the provably safe policy æ (āls) are depicted as the gray area and the blue line respectively. Figure (a) shows action replacement with the random sampling strategy. Figure (b) displays action projection, where the vertical arrows are scaled Dirac delta distributions that stem from the fact that the unsafe parts Published in Transactions on Machine Learning Research (11/2023)
(aass)
(als)
n (als)
Aup
A
Aup
A
Aup
A
(a) Action replacement
(b) Action projection
(c) Action masking
of the original policy distribution are projected the boundary of Ap. Figure (c) depicts our proposed implementation of continuous action masking.
2.5 Learning tuples
When changing the RT. action, the training of the agent can be conducted with four possible learning tuples:
naine learning based on the action a returned by the policy network of the agent and the reward r(8, a²) corresponding to the executed action at, which we denote by the tuple (s, a, 8 r(s,, a²)). This ensures that the agent is updated according to its current policy. Learning with the original action a should benefit on on-policy learning, where the policy is updated based on experience collected using the most recent policy.
adoption penalty is naive with penalty r"(s, a, a²) = r(s, a*) + "'penalty (s, a, a²) if an unsafe action was selected, which we denote by the tuple (8, (8,a, 8', r" (8, a, a))). In action projection, the penalty Tpuaalyy can include a term that is proportional to the projection distance dist(a, a²), as proposed by Wang (2022).
safe action learning based on the safe and possibly adapted action and the corresponding reward, denoted by the tuple (s, a², s', a r(s, a))) By using the safe action tuple, we are correctly rewarding the agent for the actually performed transition. However, this requires updating the agent with an action that did not stem from its current policy π (a]ss) which is an expected behavior in off- off-policy but not on-policy learning. So, the safe action tuple might be a better fit. for off-policy than for on-policy R..
both in case the RL agent proposes an unsafe action, both the adaption penalty and the safe action luples are used for learning.
In all cases, the next state s' and reward are the true state and reward received from the environment after executing the safe action a
Action masking is always paired with the naive or adaption penalty learning tuple in the literature. The adaption penally is related to the reduction of the action space due lo masking or a safety component in the reward function, and not a sparse reward as for action projection and action replacement. For discrete action masking, the naive and the safe action tuples are oquivalent since the agent only allowed choose provably safe actions (see Figure 1). For continuous action masking, using the safe action tuple with the transformed action leads to inconsistencies in learning because every action is transformed if A, # A.

---

Published in Transactions on Machine Learning Research (11/2023)
3 Literature review
In this section, we summarize previous works in provably safe RL and assign them to the proposed categories. To identify the related literature, we used the search string TITLE-ABS (TTLL-ABS""reiinoorceeee ("reinforcement learning") AND
TITLE (learning) AND (TITLE(Safe*) OR TITLE(verif*) OR TITLE AND LIMIT-TO (LANGUAGE, "English")4 for the Scopus" and IEEEXplore search engine, which lod to 620 papers already removing duplicates. Then, we screened papers by title and abstracts to identify 160 seemingly relevant papers, After close inspection, we identified 47 of these 160 papers as provably safe RL works. We give a condensed overview of all application-inependeent provably safe RL works in Table 1, and cluster all 47 provably safe RL works in Table 2 by their application. Some approaches in Table 1 are presented for unbounded disturbance. In such a setting, hard safety guarantees are generally not achievable. Still, we include approaches that would be provably safe with the assumption that the disturbance is bounded.
OR TITLE(shield*)]
Action replacement One of the earliest provably safe RL works is Alshiekh et al. (2018), which con- structed a so-called safety shield from linear temporal logic formulas. For that, they construct the verifi- cation function v(ss by converting the linear temporal logic formulas into an automaton, on which they perform model checking. The advantage of online model chocking is that lincar temporal logic constraints can be guaranteed for general nonlinear systems, and the online complexity is linear in the number of discrete states. However, the method is only applicable to small discrete state spaces, constructing the automa- ton offline has exponential complexity in the number of discrete states, and the online checking complexity also increases exponentially with the formula length (Baier & Katoen, 2008). In Alshickh et al. (2018), the agent outputs n ranked actions, which are all checked for safety using (088, a). The shield executes the highest-ranked safe action or replaces the action with (ssamlle(s if none of the n actions is safe. They update the agent with the safe action learning tuple but also propose that the both learning tuple can be used to obtain additional training information. Similarly to Alshiekh et al. (2018), Könighofer el al. (2020) show that both probabilistic and deterministic shields increase the sample efficiency and performance for both action replacement and masking methods.
Akametalu et al. (2014) propose action replacement based on Hamilton-Jacoii-ssaan Isaacss reachability analysis, which was later extended by Fisac et al. (2019) to a general safe RI. framework. They determine S using Hamilton-Jacobii-saan
reachability analysis given bounded system disturbances. Safety is guaranteed by replacing any learned action on the border of S. with (ssslssfe(s) stemming from an Hamilton-Jacobe Isaacs optimal controller to guide the system back inside the safe set. Hamillonn-accoiiTsaacc reachability analysis can verify reach-avoid problems with arbitrary non-convex sets (Wabersich et al., 2023) and disturbances that stem from compact set. However, constructingg the sale set scales exponentially in complexity with the number of state dimensions, which makes Hamilton-Jacobb-Lsaa Isaacs reachability analysis infeasible for systems with more than four state dimonsions (Chen & Tomlin, 2018). Fisac et al. (2019) arguc that replacing the unsafe action with the action that maximizes the distance to the unsafe set increases performance in uncertain real-world environments compared to action projection. However, Hamitton-acabii-Taan
approaches suffer from the curse of dimensionality and are, therefore, only feasible for systems with specific characteristics (Herbert et al., 2021).
Shao et al. (2021) use trajectory safeguard based on set-based reachability analysis for pps, Set-based reachability analysis is applicable to reach-avoid problems for general nonlinear systems with uncertainties in the initial state, system dynamics, and imput disturbances as long as they stem from a compact set, see, e.g. Althoff et al. (2021). Set-based reachability analysis has polynomial complexity in the state dimension for most set representations, as discussed by Althoff et al. (2021). However, compared to HamiltonJaccbii Isaacs reachability analysis, set-based reachability analysis cannot handle arbitrary non-convex sets but depends on specific set representations. Shao et al. (2021) sample n new actions randomly in the vicinity of the action if the action is unsafe. They then execute the closest safe action to the original (unsafe) action. If none of the n new actions is safe, failsafe action {/iaissate(s) (s) is executed. Shao et al. (2021) train their agent
4 DDoummentation at http://schcmmm.ccevcccc..... com /tdp://ddddd//mmtttttt 6 sccopus.com
6 ieeee.r
10

---

Published in Transactions on Machine Learning Research (11/2023)
Table 1: Comparison of application-indeeendeett provably safe RL approaches.
Reference
Verification Method
Space
Learning
Environment
Tuple
State
Action
Action replacement
Akamotalu et al. (2014)
reachability analysis
cont.
cont.
special RL alg.
1D quadrotor [stoch.],
cart-pole [stoch.]
Fisac et al. (2019)
HJI reachability analysis
cont.
cont.
N/A
1D quadrotor (stoch.)
Alshiekh al. (2018)
model checking automaton constructed from LTL
disc.
disc.
safe action
Grid world [stoch.]
Könighofer et al. (2020)
model checking automaton constructed from LTT.
disc.
disc.
adaption penalty
ACC' [stoch.]
Anderson et al. (2020)
robust control invariant set
cont.
cont.
N/A
pendulum [det.], reach- avoid [det.), others [dct.]
Hunt al. (2021)
theorem proving of dC formu- las
disc.
disc.
naice
Grid world [stoch.]
Bastani (2021)
MPC
cont.
cont.
deployment only
bicyele [det.], cart-pole [det.]
Shao et al. (2021)
set-based reachability analysis
cont.
cont.
našce
3D quadrotor [det.],
highway driving [det.] 3D quadrotor [stoch.],
Selim et al. (2022b)
aet-based reachability analysis
cont.
cont.
adaption penalty
mobile robot [stoch.]
Action projection
Pham et al. (2018)
verification of affine constraints actions
cont.
cont.
adaption penalty
manipulator [det.]
Cheng et al. (2019)
CBF"
cont.
safe action
ACC [stoch.], pendulum [stoch.] manipulator [dett.]
Li al. (2019a)
CBP synthesized from LTT,
cont.
adaption penalty
Gros et al. (2020)
MPC
cont.
cont. naive
2D LTI" system [stoch.]
Wabersich & Zeilinger (2021)
MPC
cont.
cont.
quadrotor [stoch.]. pendulum (stnch.)
Marvi Klumarei (2022)
cont.
cont.
adaption penalty naive
2D system [det.]
Selim et al. (2022a)
aet-baaed reachability analyala
cont.
cont.
3D quadrotor [stoch.].
mobile robot [stoch.]
Kochdumper et al. (2023)
set-based reachability analysis
cont.
adaption penalty
3D quadrotor [stuch.]
Action masking
Fulton & Platzer (2018)
theorem proving of formulas cont.
disc. naive
ACC [det.]
Fulton & Platzer (2019)
theorem proving of dC formulas verification affine equality
cont.
disc.
naive
ACC [stoch.]
Huang & Ontañóó (2022)
disc.
disc.
naive
Grid world [N/A]
coustraints for actions
This study
set-based reachability analysis
cont.
cont.
naine
pendulum [det.).
2D
quadrotor [stoch.
Abbreviations: sstoch.: stochastic environment model, det.: deterministic environment model, Saaamilton-Jacob-Teaaaaaa (HJT), Slneaa temporal logic (L.T.), "adaptive cruise control (ACC), ddifferential dynamic logic (dC), model predictive control (MPC), "control barrier function (CBF), flinear time-invariant (LTT).
on the naive learning tuple. Selim et al. (2022b) also verify the safety of actions with set-based reachability analysis. They propose an informed replacement for (b(s such that the reachable set the controlled system is pushed away from the unsafe set $ S, The authors further propose a method to account. for unknown system dynamics using a so-called black-box reachability analysis. They use the adaption penalty lcarning tuple and showcase that their approach achieves provable safety in three use cases.
Hunt et al. (2021) build the verification function p(s, a) using theorem proving of differential dynamic logic formulas. Using (pss, a), they determine (s(s) for discrete action spaces and use (haalsate for replacement. They further show how provably safe end-to-end learning can be accomplished using controller and model monitors. They train the RL agent using the naive learning tuple on a drone example. The work of Anderson et al. (2020) proposes define & as a robust control invariant set and construct the safety function (p(s, a) based on worst-case linear model of the system dynamics. A further notable work is Bastani (2021), which proposes model predictive shield alongside the trained policy, which uses Phtaisatt (s) for action replacement.
11

---

Published in Transactions on Machine Learning Research (11/2023)
Table 2: Overview of applications in provably safe RL.
Action Replacement
Action Projection
Action Masking
Acrial Vehicles
12 Alkametalu et al. (2014);
Wabersich & Zeillnggr (2021);
N/A
Shyamsundar et (2017):
Selim al. (2022a); I Kochdumper al. (2023)
FFissc et al. (2019);
Anderson et al. (2020);
Harris & Schaub (2020);
Shao et al. (2021);
Selim al. (2022b);
Nazmy et al. (2022)
Autonomous
Chen et (2020):
Cheng al. (2019);
Fulton Platzer (2018);
Driving
Könighofrr al. (2020);
Wang (2022);
Mirchevskka et (2018);
(hha et al. (2021);
Hailemichael al. (2022b);
Fulton & Platzer (2019);
Chen et al. (2022a);
Hailemichmel et al. (2022a);
KKrsswwski et al. (2020);
LLe & Kwon (2022);
Kochdumper et al. (2023)
Brosowsky et al. (2021);
1 Wang et al. (20233);
I Krasowssi al. (2022)
Evans et al. (2023)
Power Systems
Cousters et al. (2023)
Eichelbeckk et al. (2022);
Trabas & Zhang (2022)
1 Chen et al. (2022b);
Zhang et al. (2023);
YYu et al. (2023)
Robotic
Thumm & Althoff (2022)
Phamm et al. (2018):
N/A
Manipulation
" et al. (2019s)
Control
Akametalu et al. (2014));
Cheng al. (2019);
N/A
Benchmarks
Auderson et al. (2020):
Gros et al. (2020);
Bastani (2021);
Wabersich & Zeilinger (2021);
Shao et al. (2021)
Marvi & Kiumansi (2022)
Grid World
Alshiekh et al. (2018);
N/A
Huang & Outañóó (2022)
Games
Hunt et al. (2021)
Miscellaneous
Active euspension: (2019b);
Mobile robot:
Computing networks:
Sellm et sl. (2022x):
Seetanadi et al. (2020);
Compating networks:
Engine emission:
Traffic signal:
"Wang et al. (2022);
Norouzl et al. (2023)
Miilller & Sabatelli (2022)
Mobile robot:
"Selim al. (2022b)
Note: Studics using high-fidelity simulators are markod with 1, and # indicates physical experiments. Addi- tionally, papers occur multiple times in case they demonstrate their approach for different applications.
The most popular method by publications is action replacement. This is also visible from the large variety of application-specife approaches, e.g., for aerial vehicles (Shyansundar et al., 2017; Harris & Schaub, 2020; Nazmy et al., 2022), autonomous driving (Chen et al., 2020; 2022a; Lee & Kwon, 2022; Wang et al., 2023; Evans al., 2023), power systems (Ceusters et al., 2023), robotic manipulation (Thumm & Althoff, 2022), active suspension systems (Li et al., 2019b), and traffic engineering in computing networks (Wang et al., 2022). In partieular, Ceusters et al. (2023) compare fail-safe action replacement and sampling-basse action replacement. They observe that both methods have higher initial performance than the unsafe RLL baseline, and fail-safe action replacement leads to better performance than the sampling- based version.
Action projection Research on action projection is usually conducted on continuous action and state spaces. The main differentiating factor between studies in this category is the specification of the optimization problem for the projection. To begin with, the work of Pham al. (2018) guarantees safety using a differentiable constrained optinization layer called OptLayer. Their approach is restricted to quadratic programming problems, so the system model and constraints have to be linear. Despite these limitations, they show the effectiveness of their approach on collision avoidance task with simple robotic manipulator.
Cheng el al. (2019) specify the safety constraint sp8 a) of the optimization problem via CBFs. Thus, the optimization problem in (3) becomes a quadratic program. Theoretically, the CBF approach is applicable to general control-affine systems with disturbances from a compact set for reach-avoid specifications. However, finding a CBF is not trivial, and synthesizing them can be exponential in the system dimension (Ames et al., 2019). To solve (3) more efficiently online, Cheng et al. (2019) add neural network to the approach
12

---

Published in Transactions on Machine Learning Research (11/2023)
in Section 2.2, which approximates the correction due to the CBF. The action is then shifted by the approximated value prior to optimization. This shift improves the implementation efficiency while still guaranteeing safety, as the action is often already safe after the shift, and no optimization problem needs to be solved. Their safe learning with CBFs shows faster convergence speed than vanilla RL when learning on a pendulum and a car following task. Li el al. (2019a) propose a method to construct a contun CBF from an automaton, which is defined by linear temporal logic formulas. They further construct a guiding reward from the given automaton to improve the learning performance. The proposed approach is capable of learning a high-dimensional cooperative manipulation task safely. The authors of Marvi & Kiumarsi (2022) define a different problem, where the system model is assumed to be deterministic but unknown. They learn an optimal controller and the system dynamics iteratively while decreasing the conservativeness of their CBF in each iteralion. The approach is provably safe for linear lime-invarianl (LTT) systems without disturbances.
Gros et al. (2020) and Wabersich & Zeilinger (2021) implement the optimization problem as a robust MPC problem as defined in (7). MPC is applicable to reach-avoid problems with measurement and stale dis- turbances. However, when controlling high-speed systems, robust MPC is often limited to linear systems (Zeilinger et al., 2014). Gros et al. (2020) mainly discuss how the learning update has to be adapted if action projection is used for different RL algorithms. For Q-learning, they find that no adaption of the learning algorithm is necessary when the naine tuple is used. For policy gradient methods, they argue that the projection must also be included in the gradient for stable learning (Gros et al., 2020, Sec. 3). One downside of the robust MPC formulation of Gros et al. (2020) and Wabersich & Zeilinger (2021) is that dynamic constraints originating from moving obstacles or persons in the environment are not trivial to integrate. They approximate the dynamics of the system with a Gaussian process (GP) so that hard safety guarantees are impossible to prove. However, they could guarantee hard safety specifications if they would assume deterministic system dynamics with bounded disturbance as the aforementioned approaches do. Gros et al. (2020) evaluate their approach on a simple 2D LTI system, and Wabersich & Zeilinger (2021) show the efficacy of their approach on a pendulum and a quadrotor task.
Contrary to their previous work in Selim et al. (2022b), Selim et al. (2022a) propose to solve an optimization problem to find the closest safe action instcad of using an informed replacement. They again use set- based reachability analysis to construct (p(s.a) a). They test their approach a quadrotor and mobile robot benchmark. Kochdumper et al. (2023) utilize set-based reachability analysis to verify actions in ypss, a). They formulate the projection for a parameterization of the action space and arrive at mixed-integer quadratic problem with polynomial constraints. Their approach achieves provable safety for nonlinear systems with bounded disturbances, and they demonstrate their approach on two quadrotor tasks, autonomous driving on highways, and a physical FITENTH car.
Next to the conceptual approaches, action projection algorithms are also specifically proposed for many cyber- physical systems, such autonomous driving (Wang, 2022; Hailemichael et al., 2022a;b), power systems (Eichelbeck et al., 2022; Chen et al., 2022b; Zhang et al., 2023; Yu et al., 2023), and engine emission control (Norouzi et al., 2023). Wang (2022) compares the deployment of a discrete action masking approach with her continuous action projection approach. The goal-rea performance is lower for the discrete action masking approach. However, this could be due to the coarse discretization of the action space in three actions.
Action masking To the best of our knowledge, the existing literature considers action masking only for discrete action spaces. The work Huang & Ontañón (2022) analyzes the effect of discrete action masking on the policy gradient algorithm in RL, but they assume that As is known, which is typically only the case in game and grid world environments.
The two main works investigating action masking are Fulton & Platzer (2018) and Fulton & Platzer (2019). They construct controller and model monitors based on theorem proving of differential dynamic logic speci- fications, see Platzer (2008). The controller monitor is used to build the mask (7),, and the model monitor verifies if the underlying system model is correct based on previous transitions. In each state, the agent can choose from the set of actions that the controller monitor verified as safe. Identifying the correct system can be challenging, thus an approach to automatically generate candidates is introduced well. Their approach
13

---

Published in Transactions on Machine Learning Research (11/2023)
is provably safe if the initial model is correct (Fulton & Platzer, 2018) or multiple models are given, from which at least one is correct (Fulton & Platzer, 2019) for all times. They validate their provably safe action masking on adaptive cruise control tasks.
In addition to the works mentioned above, there are works investigating action masking for the specific application of autonomous driving (Mirchevska et al., 2018; Krasowski et al., 2020; Brosowsky et al., 2021; Krasowski et al., 2022), power systems (Tabas & Zhang, 2022), adaptive routing in computing networks (Seetanadi et al., 2020), and urban traffic signal control (Müller & Sabatelli, 2022). The only application- specific approach that compares action masking with other provably safe RL approaches is Brosowsky et al. (2021). They observe that their masking approach converges slightly faster than action projection.
4 Experimental comparison
In this section, we evaluate the performance of the three provably safe RL classes and the four learning tuples introduced in Section For our comparison, we select inverted pendulum and 2D quadrotor stabilization task?, these benchmarks are commonly evaluated in related works presented in Table 1. The provably safe state set Sp is the same for all three approaches and, therefore, comparable. We add system disturbances to the benchmarks to make them more realistic and show that the provably safe RL approaches can handle disturbances sampled from a compact disturbance set. Despite their low dimensionality, our results are likely transferable to real-world systems since real-world complexity is often reduced practice by using lower-dimensional abstract models and an additional disturbance term. Conformance checking techniques (Roehm al., 2019; Liu al., 2023) can then guarantee that the abstract model incorporates recorded real-world behaviors of the system.
The algorithms shown in this section are action replacement with (sampie(s), action projection using a alfine constraints, and action masking. We compare each configuration on ten random seeds and five common RL algorithms contimuons Twin Delayed Deep Deterministic policy gradient algorithm (TD3) (Fujimoto et al., 2018), continuous soft actor-critic (SAC) (Haarnoja et al., 2018), discrete DQN (Mnih et al., 2013), and continuous and discrete proximal policy optimization (PPO) (Schulman al., 2017).
4.1 Environments
We compare the provably safe RL approaches on an inverted pendulum and a 2D quadrotor stabilization task.
Inverted pendulum The state of the pendulum is defined as s = [0,0] and follows the dynamics
8 =
(12)
4 sin (0) + mlz
where a is the one-dimensional action, is gravity, m is the mass of the pendulum, / its length, and friction and damping are ignored. We discretize the dynamics using the explicit Euler method. The actions are bounded by |a| < 3-1 The desired equilibrium state is OT.. The observation and reward are identical to the OpenAl Gym Pendulum- V0 environment.
2D quadrotor The quadrotor in our experiments can only fly in the x-2-planc and rotate around the y- axis with angle 0. The state of the system is defined as = [x, 2. s. a, 0 0] and the action as a [01, 0z]
"Our implementation is available at CodeOrean: doi.org/ /eoorr/11.2444/0 21..443300..220022......
9209121.v1
8 All implementations based baable-baselines (Raffin al., 2021).
9 Available at: gymnasium. wommasium.aramma.rrgg/oooo
14

---

Published in Transactions on Machine Learning Research (11/2023)
The system dynamics
& & a ls sin
0
0
8 =
W1
(13)
g + a ls cos 0
W2
duO d 0 + no02
are based on Mitchell et al. (2019), where w1, w2 represent the system disturbance, and k, do, d1, and no are constant parameters (sce Table 4). We lincarize the dynamics using first-order Taylor expansion at the equilibrium point s* [0, 0, 0, 0] and obtain the discrete-time system for the linearized dynamics. We sample the disturbance w = [w], w2] uniformly, independent, and identically distributed from a compact
disturbance set W R The actions range from amin - _ -
to amax [1.5 + k. 7 The
reward is defined as r(s, a) = exp (- ||s s-112 0.01 a-min(A)
2 max (A) (in
4.2 Computation of the safe state set
To oblain a possibly large set of provably safe states Sip and provably safe controller for our enviromments, use the scalable approach for computing robust control invariant sets of nonlinear systems presented in Schäfer et al. (2023): for every state 50 E S C S,, there exists provably safe action āo E A so that 81 g (o. do wo) E Sp with a bounded disturbance wo W C R where W has O dimensions. Hence, ( ap) = 1 for every 80 E S.p. In this work, we assume the disturbance to be constant in between sampling times. Note that we use the obtained provable safe controller for the failsafe replacement function tailsaff (s). The algorithm in Schäfer et al. (2023) provides an explicit representation of S which enables a fair comparison of our provably safe RL implementations. To retrieve Ap from S. at given state s, we first convert S from generator reprosentation, which is used in Schäfer et al. (2023), into halfspacc representation, i.e., S.p- Cs S 4), using the open-source toolbox CORA (Althoff, 2015). We evaluate the safety function given the state 8y E Sp and an action E A by computing the reachable set R(k + 1) at the next time step, which encloses the states that are reachable for all wn E W (Althoff, 2015). The reachable set can be represented as a zonotope, ie., R(k + 1) = [8841/$K41 = c GB, [3) S 1}. If R(k + 1) S Sp, the action a is verified as safe, ie., 18 ax+1) = 1, which holds if and only if (Schürmann et al., 2020, Theorem 2)
Ce | < 9,
(14)
where the absolute value is applied elementwise and 1 denotes a vector full of ones of appropriate dimension. The approach of Schäfer et al. (2023) allows us to compute S,p for high-dimensional nonlinear systems. However, the conversion to halfspace representation is computationallyy too expensive for higher dimensional systems. Therefore, we plan to develop generator-based versions of our provably safe RL methods in future work to mitigate this shortcoming.
4.3 Results
The 2D quadrotor task is the main comparison environment in this work as it is more complex and shows the differences between provably safe RL approaches clearer than the inverted pendulum task. We evaluate the differences between the provably safe RL algorithms in Figure 3 and the effect of different learning tuples in Figure 4. All training runs on all individual algorithms and environments are presented in the Appendix, including comparison between the (s)mppesss) and (saatsato replacement function.
Comparison of provably safe RL algorithms The safety violation evaluation of the basclines in Fig- ure 3d shows that the baseline algorithms fail to guarantee safety during training in the 2D quadrotor stabilization task. All provably safe RI. algorithms guarantee safety as expected. Between the baselines, TD3 converges significantly faster than all other algorithms.
Figures 3a and 3b show the performance of the three provably safe RL categories (a) action replacement using saampte (8), (b) action projection, and (c) action masking together, with the baselines averaged over all
15

---

Published in Transactions on Machine Learning Research (11/2023)
Raseline
Replacement
-0.1
Intervention ralee
0.8
Projuctio
Reward
0.6
Masking
0.4
0.2
10
16
Steps (in 10*)
Steps (in 10
(a) Reward using provably safe RL algorithms.
(b) Intervention rate of provably safe RL algorithms.
Prosebly safe approccchhh
(coat.) baseline
-0.1
Salety Violation
0.8
PPO (disc. becelue
Reward
0.6
haseline
Feudline
DQN baseline
0.4
0.2
0
4
6
12 Steps (in 10*)
10
14
16
18
20
2
4
8
10
12
14
16
18
20
Steps (in 10*)
(c) Reward using basclines.
(d) Safety violation rate of basclines.
Figure 3: Training curves for the 2D quadrotor benchmark. Top:
Comparison of the three provably safe RL classes and the unsafe benchmarks averaged over all algorithms trained with the naive tuple. Bottom: Comparison of benchmark algorithms TD3, SAC, DQN, continuous and discrete PPO. All training runs were conducted on ten random soods per algorithm. The left column depicts the reward. The right column shows the safety violations for the baselines, and the safety intervention rate for the provably safe RL algorithms.
five RL implementations and trained using the naive learning tuple. For a better comparison of the reward curves in Figure 3a, added dashed green line that indicates the final training reward averaged over all five RL baselines and ten random seeds. The reward comparison shows that action replacement performs better than action projection and masking on average.
We also compare the intervention rate the three safety mechanisms. For action replacement and projection, our intervention rate metric indicates the share RL steps per episode which the safety function altered the action. For action masking, the intervention rate compares the average volume of the provably safe action set over episode with the volume of the provably safe action set the equilibrium point of the system, e.g., Vpg.odes Va..ll/bria. Figure 3b shows that action replacement relies significantly less the safety mechanism than projection and masking. Generally, we report that lower intervention rate often coincides with higher reward.
Comparison of learning tuples We evaluate the impact of different learning tuples the performance and intervention rates averaged over all five RL algorithms in Figure 4. When action masking is used, only safe actions can be sampled, i.c., only the naive tuple is meaningful; so we omit action masking from this evaluation. For both action replacement and projection, the adaplion penally luple leads to the highest performance and lowest safety intervention rate, even outperforming the average over the baselines. In action projection, the naive tuple performs significantly worse than action replacement. The safe action and both tuples seem to be only beneficial when using action projection and decrease performance when using action replacement.
16

---

Published in Transactions on Machine Learning Research (11/2023)
Repbaermet (atoçtion penalty
-0.1
Intervention rate
0.8
Rephaement (sal udion)
Reward
Replacemeut (both)
0.2
10
12
14
16
18
20
Steps (in 10*)
Steps (in 10*)
(a) Reward using action replacement.
(b) Intervention rate of action replacement.
Projection (naive)
Projectoon (alaption penalty
-0.1
Intervention rate
0.8
Projection (safs action)
Reward
0.6
Projection (both)
0.4
0.2
0
2
4
8
10
12
14
16
18
20
12 Steps (in 10*)
8
10
14
16
18
20
Steps (in 10*)
(c) Reward using action projection.
(d) Intervention rate of action projection.
Evaluation of the training tuples for the 2D quadrotor averaged over the five RL algorithms TD3, SAC, DQN, contimuus and discrete PPO on ten random seeds per algorithm. The left column depicts the reward and the right column the safety intervention rate. The top row shows the different learning tuples for action replacement and the bottom row for action projection.
Figure 4:
5 Discussion
Our experiments confirm the theoretical statement that provably safe RL methods are always safe when Proposition 1 holds. In the two tested environments, the investigated RL baselines show non-zero safety violations during training, even after convergence. Subsequently, we discuss five statements resulting from the experiments, provide intuitions for implementing provably safe RL, summarize the limitations, and identify future research directions in provably safe RI.
Selecting provably safe R.L approach First, we want summarize our experience with the three provably safe RL classes on the two investigated benchmarks. Action replacement was the easiest method to implement for contimuus action spaces. It shows very good performance and low intervention rates. Hereby, using failsafe action for replacement with an adaption penalty is simple to implement, and was among the best-performing methods in our experiments. Still, the random sampling of safe actions might outperform the failsafe action. So, if sampling from the safe action space is readily available, e.g., in discrete action spaces, a failsafe controller is unnecessary. Action projection tends to be problematic in practice due to small numerical errors, resulting infeasible optimization problems. We, therefore, have reuse previous optimization results, e.g., as in Schürmann al. (2018) for robust MPC, use failsafe controller if the optimization problem is not solvable. Together with the higher intervention rates compared to action replacement and the complex implementation, our experience. However, if one alrcady has CBF or MPC formulation, it might be the most suitable solution. Action masking is particularly easy to implement for discrete action spaces and performs well in that setting. However, for action masking in continuous action spaces, there no efficient and general algorithm yet that can handle safe action spaces significantly diverging from an axis-aligned box. Generally, the different approaches can also be combined to some extent. For example, the optimization problem for action projection becomes infeasible, failsafe controller be used.
would not recommend action projection based
17

---

Published in Transactions on Machine Learning Research (11/2023)
Selecting learning tuple The adaption penalty learning tuple performed best, especially when using action projection. In our experiments, a simple constant reward penalty already improved the performance. Other environments may require more careful reward tuming, or the adaption penalty tuple could fail alto- gether due to reward hacking (Skalse et al., 2022) or goal misgencralization (Langosco et al., 2022). The results Figure 4 further show that, using the sale action á in the training luple, i.e. configurations both and safe action, benefits the performance of action projection mothods but impairs the training with action replacement. This effect can result from the fact that action replacement alters the action more than action projection, leading to a lower likelihood that the altered action stems from the RL policy. The evaluations in the Appendix show that this effect is prominent when using PPO. This on-policy algorithm assumes that the current batch of training data stems from the current policy. Hence, we would recommend using the adaption penally tuple when possible and only using safe action in combination with off-policy methods.
Convergence The training of provably safe RL. agents converges similarly fast or faster than the basclines in our experiments. In contrast, the performance (measured by the reward) at the beginning of the training is better for provably safe RL agents. One reason for the faster convergence is the exploration setting. Since App S A, the provably safe agents learn in usually smaller action space than the baselines, which can accelcrate training. However, in some cases, the bascline agents might be better informed about the environment dynamics by exploring unsafe actions. Generally, the verification method should aim for A = Ag such that the provably safe agents can explore the full safe action space. Another reason for convergence differences can be that action replacement and action projection potentially correct the action after the forward pass through the policy. Thus, the gradient calculation might need correction as well. So far, there is only little theoretical work on this (Hunt et al., 2021; Gros et al., 2020); and it is unclear if, in practice, a correction of the gradient is necessary or if using the adaption penalty tuple is sufficient. Furthermore, the change in the distribution of the actions can impact the exploration strategy, as discussed in Section 2.4.
Computational complexity The computational complexity of the three approaches highly depends on the scenario-specific implementation. For action projection, the main implementation challenge is to guar- anlee that the optimization problem is always feasible. If the optimization problem can be formulated as a quadratic program, the computational complexity is polynomial, as shown by Vavasis (2001). On the contrary, the computational complexity of action replacement and action masking depends highly the algorithm that identifies the safety of actions. For discrete action masking, the computational complexity apparently scales linearly with the total number of actions O[Al). For action replacement, we only need a single safe action, so in the ideal case, e.g,, using a failsafe controller, the computational complexity is constant with respect to the total number of actions. Suppose an action replacement approach needs to determine the entire set of safe actions, it obviously has the same compulational complexity as action mask- ing with respect to the number of actions. The computational complexity for identifying the continuous safe action space depends on the task-specific implementation. One possibility is to compute the safe action space using based reachability analysis, where we want to point the interested reader to Althoff et al. (2021) for different approaches.
Online vs. offline implementation Online and offline have two notions in provably safe RL: online vs. offline safety verification and online vs. offline RL. The safety function usually needs to be evaluated online since, for continuous state spaces, pre-computing the safe action set for all states is often not feasible. Thus, the computational complexity of the safety function is important for real-time applications, as discussed in the previous paragraph. If the state and action space are discrete, it can be possible to pre-compute the safe actions offinn (Alshiekh et al., 2018; Huang & Ontañón, 2022). Generally, safety is only guaranteed if the safety function is integrated between the agent and environment to correct actions (sce Figure 1). In this study, we compare online on-policy and off-policy RL algorithms (Levine et al., 2020) since they are used for existing provably safe RL research. Still, provably safe RL can also be used for offline RL where the safety function would be integrated during deployment and most likely also during the data gathering phase if this phase is conducted in safety-critical environment. However, more specific advice on offline provably safe RL. needs to be substantiated with experimental evaluations and, thus, is a topic for future research.
18

---

Published in Transactions on Machine Learning Research (11/2023)
Limitations of provably safe RL There are limitations of provably safe RL that follow from the concep- tual analysis in Section 2. Most importantly, safety has to be verifiable, L.e, there must be safety function ((8, a), which complies with Proposition 1. For this safety function, system knowledge is necessary, and especially for systems with a high number of continuous state variables, the safety function is potentially complex to compute. Additionally, safety guarantees are strongly tied to the safely function. If the safely function provides complex guarantees (e.g., ensures temporal logic specifications), it is usually computation- ally more expensive than for simpler guarantees (e.g,, system stays within safe state set). Second, safety can only be decided if the state of the system is correctly observed within noise bounds. Thus, for a provably safe autonomous system, the perception module also needs to be verified such that it provides observations that are correct within the noise bounds. Third, there is often a trade-off between safety and performance since, for many lasks, these two objectives are only partially aligned. For example, if an automated vehicle drives faster, it reaches its destination earlier, but collisions are more difficult to avoid at higher speeds. Since provably safe RL ensures safe behavior, there is no such trade-off as safety is always prioritized over perfor- mance. Thus, the safety function p(s, a) should not be too conservative since the agent would only perform trivial safe actions e.g., standing still at the side of the road forever. Lastly, comparing provably safe RL approaches is challenging we need to define a safety function yfs a) that is efficiently usable by different approaches. Furthermore, the notion of safety is usually application-specfft so different application-specife approaches are hard to compare. We provide the first comparison of provably safe RL on two common continuous stabilization tasks, but further research is necessary to make more substantial claims about the most promising provably safe RL approaches.
Future research based on proposed taxonomy Most action projection approaches discussed in Sec- tion 3 project the RL action on the border of App. In our experiments, we encountered two negative side effects related to this action projection implementation: First, the projection to the border of A, often leads to a relatively small Ag in the next RL step, quickly resulting in a very small set Ap if the RL agent proposes a few unsafe actions after each other. Second, small numerical errors can cause unsafe actions and must be considered in the safety verification. Therefore, future action projection research should investigate objective functions for (3) that achieve a more robust behavior while still depending on the action the RL agent proposed, e.g., projecting the action not to the border but by learnable margin inside the provably safe action set. Action masking is a promising technique but has mainly been used with discrete action spaces in grid world environments and games, c.g., the Atari benchmark (Huang & Ontañón, 2022). Our proposed contimuous action masking approach only applies to specific environments and performed well for the pendulum but showed mixed results for the 2D quadrotor. Thus, future research should investigate ways to extend continuous action masking to general convex or non-convex representations of Aup to improve its applicability to more complex benchmarks and reduce the conservativeness of Ag.. Additionally, it should be investigated if the agent should be informed about the reduction of the action space in action masking. This could result in improved convergence and an agent that is more aware of the action mask, similar to the effect of the adaption penalty tuple for action replacement and action projection. The ovaluation of the considered benchmarks shows that action replacement performs better than action projection and masking, as discussed previously. However, it still unclear how important the replacement strategy (ss)) is for the convergence and performance of the agent, especially when applied to more complex tasks. Thus, future action replacement research should empirically and theoretically investigate this question.
Improving the applicability of provably safe RL Despite the promising previous work discussed in Section 3, there are only few works on high-dimensional nonlinear systems and limited real-world ap- plications. We suggest five major factors where future research would improve applicability. First, some approaches need to be computationally more efficient to be real-world applicable. The computational effi- ciency of verification methods is especially relevant and should be improved, as discussed previously. Second, we observe that the learning tuple used has significant influence on the performance of the agent for some RL algorithms. Also, there needs to be more theoretical research on how provably safe RL approaches influ- ence convergence to an optimal policy. More empirical and theoretical research on the effects of provably safe RL and its learning tuples for convergence is desirable. Third, common benchmarks are necessary to evaluate new provably safe RL approaches. Additionally, the three action correction strategies should be compared more complex benchmarks to clarify if our observations can be extended to them. Such benchmarking
19

---

Published in Transactions on Machine Learning Research (11/2023)
would make research provably safe RL more comparable, case starting research provably safe RL, and provide more evidence to decide on the best-suited provably safe RL. approach. Fourth, recent work shows a low variety of safety specifications, mainly comprising stabilization and reach-avoid specifications. On the contrary, real-world safety is more complex, c.g., traffic rules such as waiting at a red light and safely but quickly moving al a green light. Finally, provably sale RL requires expert knowledge of verification methods. Future research could mitigate this through modular and automatic approaches, where fewer engineering decisions are necessary and more parameters are tuned automatically. With these advances, provably safe RL could bring the best elements of RL and formal specifications together towards RL methods that require as little expert knowledge as necessary and provide formal guarantees for complex safety specifications to achieve reliable and trustworthy cyber-physical systems.
6 Conclusion
In conclusion, we categorize provably safe RL methods to structure the literature from a machine learning perspective. We present provably safe RL methods from conceptual perspective and discuss necessary assumptions. Our proposed categorization into action replacement, action projection, and action masking supports researchers in comparing their works and provides valuable insights into the selection process of provably safe RL methods. The comparison of four implementations of provably safe RL on a 2D quadrotor and an inverted pendulum stabilization benchmark provides further insights into the best-suited method for different lasks. We further present practical recommendations for selecting a provably safe RL approach and learning tuple, which will be valuable for researchers who are new to RL or formal methods. Lastly, discussed in Section 5, our proposed taxonomy and experimental evaluation yield multiple promising future research directions.
Acknowledgments
The authors gratefully acknowledge the partial financial support of this work by the rescarch training group ConVeY, funded by the German Research Foundation under grant GRK 2428, by the project TRAITS under grant number 011S21087, funded by the German Federal Ministry of Education and Research, by the Horizon 2020 EU Framework Project CONCERT under grant number 101016007, by the project justITSELF funded by the European Research Council (ERC) under grant agreement number 817629, and by the German Federal Ministry for Economics Affairs and Climate Action project VaF under grant number KK5135901
References
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proc. of the Int. Conf. on Machine Learning (ICML), pp. 22-31, 2017.
Anayo K. Akametalu, Shahab Kaynama, Jaime F. Fisac, Melanie N. Zeilinger, Jeremy H. Gillula, and Claire J. Tomlin. Reachability-baseed safe learning with Gaussian processes. In Proc. of the IEFE Conf. on Decision and Control (CDC), pp. 1424 1431, 2014.
Derya Aksaray, Austin Jones, Zhaodan Kong, Mac Schwager, and Calin Belta. Q-learning for robust satis- faction of signal temporal logic specifications. In Proc. of the IEEE Conf. Decision and Control (CDC), pp. 1565-6570,, 2016.
Mohammed Alshickh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Nickum, and Ufuk Topcu. Safe reinforcement learning via shielding. In Proc. of the AAAI Couf. on Artificial Intelligence (AAAT), pp. 2669 2678, 2018
Matthias Althoff. An introduction to CORA 2015. In Proc. of the Workshop on Applied Verification for Continuous and Hybrid Systems, pp. 120-151, 2015.
Matthias Althoff, Goran Frehse, and Antoine Girard. Set propagation techniques for reachability analysis. Annual Review of Control, Robolies, and Autonomous Systems, 4(1):369 395, 2021.
20

---

Published in Transactions on Machine Learning Research (11/2023)
Eitan Altman. Constrained Markov decision processes with total cost criteria: Lagrangian approach and dual linear program. Malhematical Methods of Operalions Research, 48(3):387-417, 1998.
Rajeev Alur, Suguman Bansal, Osbert Bastani, and Kishor Jothimurugan. A Framework for Transforming Specifications in Reinforcement Learning, pp. 604 624. Springer Nature Switzerland, 2022.
Rajoev Alur, Osbert Bastani, Kishor Jothimurugan, Matco Perez, Fabio Somenzi, and Ashutosh Trivedi. Policy synthesis and reinforcement learning for discounted LTL. In Proc. of the Int. Conf. on Compuler Aided Verification (CAV), pp. 415-435, 2023.
Aaron D. Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control barrier functions: Theory and applications. In Proc. of the European Control Conference (ECC), pp. -420- 2019.
Greg Anderson, Abhinav Verma, Isil Dillig, and Swarat Chaudhuri. Neurosymbolic reinforcement learning with formally verified exploration. In Proc. of the Int. Conf. on Neural Information Processing Systems (NearIPS), volume 33, pp. 6172-6183, 2020.
Christel Baier and Joost-Pieter Katoen. Principles of Model Checking. MIT press, 2008.
Osbert Bastani. Safe reinforcement learning with nonlinear dynamics via model predictive shiekding. In Proc. of the American Control Conf. (ACC), pp. 3488 3488-3494, 2021.
Osbert Bastani, Yewen Pu, and Armando Solar- Verifiable reinforcement learning via policy ex- traction. In Proc. of the Int. Conf. Neural Information Processing Systems (NeurIPS), pp. 2499-2509, 2018.
Felix Berkenkamp, Angela P. Schoollig, Matteo Turchetta, and Andreas Krause. Safe model-based rein- forcement learning with stability guarantees. In Proc. of the Inl. Conf. on Neural Information Processing Systems (NeurIPS), pp. 908 918, 2017.
Sara Bouraine, Thierry Fraichard, and Hassen Salhi. Provably safe navigation for mobile robots with lim- ited field-of-views in unknown dynamic environments. In Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 174-179, 2012.
Mathis Brosowsky, Florian Keck, Jakob Ketterer, Simon Isele, Daniel Slieter, and Marius Zöllner. Safe deep reinforcement learning for adaptive cruise control by imposing state-specific safe sets. In Proc. of the IEEE Intelligent Vehicles Symp. (IV), pp. 488-495 2021.
Lukas Brunke, Mclissa Greeff, Adam W. Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela Schoellig. Safe learning in robotics: From learning-basee control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5:411-444, 2022.
Mingyu Cai, Mohammadhosein Hasanbeig, Shaoping Xiao, Alessandro Abate, and Zhen Kan. Modular deep reinforcement learning for continuous motion planning with temporal logic. IEEE Robotics and Automation Letters, ((4):7973-7996)) 2021.
Alberto Camacho, Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, and Sheila A Mcllraith. LTL. and beyond: Formal languages for reward function specification in reinforcement learning, In Proc. of the Int. Joint Conf. Artificial Intelligence (IJCAI), pp. 6065-6073, 2019.
Glenn Ceusters, Luis Ramirez Camargo, Rüdiger Franke, Ann Nowé, and Maarten Messagie. Safe reinforce- ment learning for multi-energy management systems with known constraint functions. Energy and AI, 12, 2023.
Dong Chen, Longshong Jiang, Yue Wang, and Zhaojian Li. Autonomous driving using safe reinforcement learning by incorporating a based human lane-changing decision model. In Proc. of the American Control Conf. (ACC), pp. 4355 4361, 2020.
21

---

Published in Transactions on Machine Learning Research (11/2023)
Mo Chen and Claire J. Tomlin. Hamilton-Jacobi reachability: Some recent theoretical advances and applica- tions in unmanned airspace management. Annual Review of Control, Robolics, and Aulonomous Systems, 1(1):333 358, 2018.
Shengduo Chen, Yaowei Sun, Dachuan Li, Qiang Wang, Qi Hao, and Joseph Sifakis. Runtime safety assurance for learning-enabled control of autonomous driving vehicles. In Proc. of the IEEE Int. Conf. on Robolics and Automation (ICRA), pp. 8978 8984, 2022a.
Yize Chen, Yuanyuan Shi, Daniel Arnold, and Sean Peisert. SAVER: Safe learning- baani controller for rcal-time voltage regulation. In Proc. of the IEEE Power and Energy Society General Meeting (PESGM), pp. 1-5, 2022b.
Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proc. of the AAAT Conf. on Artificial Intelligence (AAAI), pp. 3387 3395, 2019.
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzzan and Mohammad Ghavamzadeh. A Lyapunov-basod approach to safe reinforcement learning. In Proc. of the Ind. Conf. on Newral Information Processing Systems (NeurIPS), pp.
2018.
Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep rein- forcement learning from human preferences. In Proc. of the Int. Conf. on Neural Information Processing Systems (NeurIPS), 2017.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv, abs/1801.0, 2018
Giuseppe De Giacomo, Luca locchi, Marco Favorito, and Fabio Patrizi. Foundations for restraining bolts: Reinforcement learning with L.Tf/LLLFf restraining specifications. In Proc. of the Int. Conf. on Automated Planming and Scheduling (ICAPS), pp. 128 136, 2021.
Michael Eichelbeck, Hannah Markgraf, and Matthias Althoff. ceatingency-coostrranne economic dispatch with safe reinforcement learning. In Proc. of the IEEE Int. Conf. Machine Learning and Applications (ICMLA), pp. 597 602, 2022.
Mohamed El-Shamouty, Xinyang Wu, Shanqi Yang, Marcel Albus, and Marco F. Huber. Towards safe human-robot collaboration using deep reinforcement learning. In Proc. of the IEEE Inl. Conf. on Robolics and Automation (ICRA), pp. 4899 4905, 2020.
Benjamin D. Evans, Hendrik W. Jordaan, and Herman A. Engelbrecht. Safe reinforcement learning for high-speed autonomous racing. Cognitive Robotics, 3:107 126, 2023.
Jaime F. Fisac, Anayo K. Akametalu, Melanie N. Zeilinger, Shahab Kaynama, Jeremy Gillula, and Claire J. Tomlin. A general safety framework for learning based control in uncertain robotic systems. IEEE Trans- actions on Automatic Control, 64(7):2737-22 2752, 2019.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proc. of the Int. Conf. on Machine Learning (ICML), pp. 2587-2601, 2018.
Nathan Fulton and André Platzer. Safe reinforcement learning via formal methods: Toward safe control through proof and learning. In Proc. of the AAAI Conf. on Artificial Intelligence (AAAI), pp. 6485-6492. 2018.
Nathan Fulton and André Platzer. Verifiably safe off-model reinforcement learning. In Proc. of the Int. Conf. on Tools and Algorithms for the Construction and Analysis of Systems (TACAS), pp. 413 430, 2019.
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(44))1437-1448) 2015.
22

---

Published in Transactions on Machine Learning Research (11/2023)
Jeremy H. Gillula and Claire J. Tomlin. Reducing conservativeness in safety guarantees by learning dis- turbances online: Iterated guaranteed safe online learning. Robolics: Science and Systems, 8(1):81-88 88, 2013.
Scbastien Gros, Mario Zanon, and Alberto Bemporad. Safe reinforcement learning via projection on a safe set: How to achieve oplimality? PTAC-PapersOnliine, 53(2))8076 800800666888,,,, 2020.
Felix Gruber and Matthias Althoff. Scalable robust output feedback MPC of linear sampled-data systems. Proc. of the IEEE Conf. on Decision and Control (CDC), pp. 2563-2570, 2021.
Tuomas Haarnoja, Aurick Zhou, Picter Abbecl, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proc. of the Iul. Conf. on Machine Learning (ICML), pp. 1861-1870, 2018.
Ernst Moritz Hahn, Matco Perez, Sven Schewe, Fabio Somenzi, Ashutosh Trivedi, and Dominik Wojtezak. Omega-regular objectives in model-free reinforcement learning. In Proc. of the Inl. Conf. on Tools and Algorithms for the Construction and Analysis of Systems (TACAS), pp. 395-412, 2019.
Habtamu Hailemichael, Beshah Ayalew, Lindsey Kerbel, Andrej Ivanco, and Keith Loiselle. Safety filtering for reinforcement learning-based adaptive cruise control. IFAC-PapersOnniine, 55(24):149 154, 2022a.
Habtamu Hailemichael, Beshah Ayalew, Lindsey Kerbel, Andrej Ivanco, and Keith Loiselle. Safe reinforce- ment learning for an energy-efficient driver assistance system. In IPAC-PapersOnnline volume 55:37, pp. 615 620, 2022b.
Andrew Harris and Hanspeter Schaub. Spacocraft command and control with safety guarantees using shielded deep reinforcement learning. In ATAA Scilech 2020 Forum, volume 1, 2020.
Mohammadhosein Hasanbeig, Yiannis Kantaros, Alessandro Abate, Daniel Kroening, George J. Pappas, and Insup Loc. Reinforcement learning for temporal logic control synthesis with probabilistic satisfaction guarantees. In Proc. of the IEEE Conf. on Decision und Control (CDC), pp. 5338 5343, 2019a.
Mohammadhosein Hasanbeig, Daniel Kroening, and Alessandro Abate, Towards verifiable and safe model- free reinforcement learning. In Proc. of the Workshop on Artificial Intelligence and Formal Verification, Logic, Aulomata, and Synthesis, pp. 1 2019b.
Mohammadhosein Hasanbeig, Alessandro Abate, and Daniel Kroening. Cautious reinforcement learning with logical constraints. In Proc. of the Int. Conf. Autonomous Agents and Multi Agent Systems (AAMAS). pp. 483 491, 2020.
Mohammadhosein Hasanbeig, Daniel Kroening, and Alessandro Abate. LCRL: Certified policy syuthesis via ongically-oonstranne reinforcement. learning. In Proc. of the Inl. Conf. on Quantilative. Evaluation of Systems (QEST), pp.
2022.
Matthias Heger. Consideration of risk in reinforcement lcarning. In Proc. of the Int. Conf. on Machine Learning (ICML), pp. 105 111, 1994.
Sylvia Herbert, Jason J. Choi, Suvansh Sanjeev, Marsalis Gibson, Koushil Sreenath, and Claire J. Tomlin. Scalable learning of safety guarantees for autonomous systems using Hamilton-Jacoi reachability. In Proc. of the IEEE Inf. Conf. on Robotics and Automation (ICRA), pp. 5914 5920, 2021.
Lukas Hewing, Kim P. Wabersich, Marcel Menner, and Melanie N. Zeilinger. Learning based model predictive control: Toward safe learning in control. Annual Review of Control, Roboties, and Autonomous Systems, 3(1):269 296, 2020.
Shengyi Huang and Santiago Ontañón. A closer look invalid action masking in policy gradient algorithms. The Tnl. Florida Artificial Intelligence Research, Sociely Conf. Proc. (FLAIRS), 35, 2022.
23

---

Published in Transactions on Machine Learning Research (11/2023)
Nathan Hunt, Nathan Fulton, Sara Magliacane, Trong Nghia Hoang, Subhro Das, and Armando Solar- Lezama. Verifiably safe exploration for end-to-end reinforcement learning. In Proc. of the Tnt. Conf. Hybrid Systems: Computation and Control (HSCC), pp. 11, 2021.
B. Ravi Kiran, Tbrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A.AI Sallab, Senthil Yogamani, and Patrick Perez. Deep reinforcement learning for autonomous driving: A survey, IEEE Transactions Intelligent Transportation Systems, 23(6):4909,4996, 2022.
Niklas Kochdumper, Hanna Krasowski, Xiao Wang, Stanley Bak, and Matthias Althoff. Provably safe reinforcement learning via action projection using reachability analysis and polynomial zonotopes. IEEE Open Journal of Control Systems, 2:79-92, 2023.
Bettina Könighofer, Florian Lorber, Nils Jansen, and Roderick Bloem. Shield synthesis for reinforcement learning. In Leveraging Applications of Formal Methods, Verification and Volidation: Verification Princi- ples, pp. 290 306, 2020.
Bettina Könighofer, Julian Rudolf, Alexander Palmisano, Martin Tappler, and Roderick Bloem. Online shielding for stochastic systems. In Proc. of the NASA Formal Methods (NFM), pp. 231 248, 2021.
Hanna Krasowski, Xiao Wang, and Matthias Althoff. Safe reinforcement learning for autonomous lanc changing using set-based prediction. In Proc. of the IEEE Inl. Intelligent Transportation Systems Conf. (ITSC), pp. 1-7, 2020.
Hanna Krasowski, Yinqiang Zhang, and Matthias AlthofT. Safe reinforcement learning for urban driving using invariably safe braking sets. In Proc. of the IEEE Int. Intelligent Transportation Systems Conf. (ITSC), pp. 2407 2414, 2022.
Lauro Langosco Di Langosco, Jack Koch, Lee D. Sharkey, Jacob Pfau, and David Krueger. Goal misgen- eralization in deep reinforcementt learning. In Proc. of the Inl. Conf. on Machine Learning (ICML), pp. 12004-12019, 2022.
Dongsu Lee and Minhae Kwon. ADAS-RL: Safety learning approach for stable autonomous driving, ICT Express, 8(3):479-45) 483, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arxi::2005. 01644, 2020.
Xiao Li, Cristian-loan Vasile, and Calin Belta. Reinforcement learning with temporal logic rewards. In Proc. of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), pp. 3834 3839, 2017.
Xiao Li, Zachary Serlin, Guang Yang, and Calin Belta. A formal methods approach to interpretable rein- forcement learning for robotic planning. Science Roboties, 4(37), 2019a.
Zhaojian Li, Tianshu Chu, and Uroš Kalabié. Dynamics-enabled safe deep reinforcement learning: Case study on active suspension control. In Proc. of the IEEE Conf. on Control Technology and Applications (CCTA), pp. 585 591, 2019b.
Zemin Eitan Liu, Quan Zhou, Yanfci Li, Shijin Shuai, and Hongming Xu. Safe deep reinforcement learning- based constrained optimal control scheme for HEV energy management. IEEE Transactions on Trans- portation Electrification, (3):42778-427) -4977, 2023.
Tommaso Mannueci, Erik-Jan van Kampen, Cornelis de Visser, and Qiping Chu. Safe exploration algorithms for reinforcement lcarning controllers. IEEE Transactions on Neural Networks and Learning Systems, 29 (4):1069 1081, 2018.
Zahra Marvi and Bahare Kiumarsi. Safe reinforcement learning: A control barrier function optimization approach. International Journal of Robust and Nonlinear Coutrol, (6)):19923-194 2021.
Zahra Marvi and Bahare Kiumarsi. Reinforcement learning with safety and stability guarantees during exploration for linear systems. IEFE Open Journal of Control Systems, 1332--344 2022.
24

---

Published in Transactions on Machine Learning Research (11/2023)
Branka Mirchevska, Christian Pek, Moritz Werling, Matthias Althoff, and Joschka Boedecker. High-level decision making for safe and reasonable autonomous lane changing using reinforcement learning. In Proc. of the IEEE Int. Intelligent Transportation Systems Conf. (ITSC), pp. 2156 2162, 2018.
Tan M. Mitchell, Jacob Budzis, and Andriy Bolyachevets. Invariant, viability and discriminating kernel under-approximation via zonotope scaling. In Proc. of the Int. Conf. on Hybrid Systems: Computation and Control (HSCC), pp. 268-269, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv, abs 1312.55 2013.
Arthur Müller and Matthia Sabatelli. Safe and psychologically pleasant traffic signal control with rein- forcement learning using action masking. Proc. of the IEEE Conf. on Intelligent Transportation Systems (ITSC), pp. 951-958, 2022.
Islam Nazmy, Andrew Harris, Morteza Lahijanian, and Hanspeter Schaub. Shielded deep reinforcement learning for multi-sensor spacecraft imaging. In Proc. of the American Control Conf. (ACC), pp. 1808- 1813, 2022.
Armin Norouzi, Sacid Shahpouri, David Gordon, Mahdi Shahbakhti, and Charles Robert Koch. Safe deep reinforcement learning in diesel engine emission control. Proc. of the Institution of Mechanical Engineers. Part I: Journal of Systems and Control Engineering, 2023.
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning, 2018.
Christian Pek, Stefanie Manzinger, Markus Koschi, and Matthias Althoff. Using online verification to prevent autonomous vehicles from causing accidents. Nature Machine Intelligence, 2(9):518 528, 2020.
Tu-Hoa Pham, Giovanni De Magistris, and Ryuki Tachibana. OptLayer practical constrained optimization for deep reinforcement learning in the real world. In Proc. of the IEEE Int. Conf. Robotics and Aulomation (ICRA), pp. 6236 6243, 2018.
André Platzer. Differential dynamic logic for hybrid systems. Journal of Automated Reasoning, 41(2): 143-189, 2008.
Antonin Rallin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-Baselines Reliable reinforcement learning implementations. Journal of Machine Learning Re- search, 22(268):1 8, 2021.
Hendrik Roehm, Jens Oehlerking, Matthias Woehrle, and Matthias Althoff. Model conformance for cyber- physical systems: A survey. ACM Transactions on Cyber-Physical Systems, 3(3): 26, 2019.
Lukas Schäfer, Felix Gruber, and Matthias Althoff. Scalable computation of robust control invariant sets of nonlinear systems. IEEE Transactions on Aulomatic Control, (early acces):1 15, 2023.
Lukas M. Schmidt, Georgios D. Kontes, Axel Plinge, and Christopher Mutschler. Can you trust your autonomous car? interpretable and verifiably safe reinforcement learning, In Proc. of the IEEE Intelligent Vehicles Symp. (IV), pp. 171-178, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arxiv, abs/1707.0, 2017.
Bastian Schürmann, Niklas Kochdumper, and Matthias Althoff. Reachset model predictive control for disturbed nonlinear systems. In Proc. of the IEEE Conf. on Decision and Control (CDC), pp. 3463 3470, 2018.
Bastian Schürmann, Riccardo Vignali, Maria Prandini, and Matthias Althoff. Set-based control for disturbed piecewise affine systems with state and actuation constraints. Nonlinear Analysis: Hybrid Systems, 36 (Art. 100826), 2020.
25

---

Published in Transactions on Machine Learning Research (11/2023)
Gautham Nayak Sectanadi, Karl-Erik Årzén, and Martina Maggio. Adaptive routing with guaranteed delay bounds using safe reinforcement learning. In ACM Inl. Conf. Proc. Series, pp. 169,160, 2020.
Mahmoud Selim, Amr Alanwar, M. Watheq EI-Kharashi, Hazem M. Abbas, and Karl H. Johansson. Sale reinforcement learning using data-driven predictive control. In Proc. of the Int. Conf. on Communications, Signal Processing, and their Applictiinns (ICCSPA), pp. 1-6, 2022a.
Mahmoud Selim, Amr Alanwar, Shreyas Kousik, Grace Gao, Marco Pavone, and Karl H. Johansson. Safe reinforcement learning using black-box reachability analysis. IEEE Robotics and Automation Letters, 7 14):00 2022b.
Yilei Simon Shao, Chao Chen, Shreyas Kousik, and Ram Vasudevan. Reachability-basee tra trajectory safeguard (RTS):: A safe and fast reinforcement learning safety layer for continuous control. IEEE Robotics and Aulomation Letters, 6(2):3663 3670, 2021.
Suhas Shyamsundar, Tommaso Mannucci, and Erik-Jan Van Kampen. Reinforcement learning based algo- rithm with safety handling and risk perception. In Proc. of the IEEE Symp. Series Computational Intelligence (SSCT), pp. 1-7, 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, Toannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George Van Den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550(7676) :54 (54-666)):54359)) 2017.
Joar Max Viktor Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. In Proc. of the Jnt. Conf. on Neural Information Processing Systems (NeurIPS), 2022.
Adam Stooke, Joshua Achiam, and Picter Abbeel. Responsive safcty in reinforcement learning by PID Lagrangian methods. In Proc. of the Ind. Conf. on Machine Learning (ICML), pp. 9133 9143, 2020.
Zachary Sunberg and Mykel Kochenderfer. Online algorithms for POMDPs with continuous state, action, and observation spaces. Proc. of the Int. Conf. on Automated Planning and Scheduling (ICAPS), 28(1): 259-263, 2018.
Richard S. Sutton and Andrew G. Barto. Reinforcement Leaning: An Introduction. Bradford Book, 2nd edition, 2018.
Daniel Tabas and Baosen Zhang. Computationally efficient safe reinforcement lcarning for power systems. In Proc. of the American Control Conf. (ACC), pp. 3303 3310, 2022.
Andrew Taylor, Andrew Singletary, Yisong Yue, and Aaron D. Ames. Learning for safety-critical control with control barrier functions. In Proc. of the Conf. on Learning for Dynamics and Control, pp. 708 717, 2020.
Andrew J. Taylor, Andrew Singletary, Yisong Yue, and Aaron D. Ames. control barrier perspective on episodic learning via projection-to-stant safcty. IEEE Control Systems Letters, 533):1119-1024, 2021.
Brijen Thananjcyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery RL: Safe reinforcement learning with learned recovery zones. IEEF Robolies and Audomation Lellers, 6(3):4915 04149999-22--- 2021.
Jakob Thumm and Matthias Althoff. Provably safe deep reinforcement. learning for robotic manipulation in human environments. In Proc. of the IEEE Int. Conf. Robotics and Automation (ICRA), pp. 2022.
Mattco Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite Markov dccision processes with Gaussian processes. In Proc. of the Int. Conf. on Neural Information Processing Systems (NeurIPS), pp. 4312 24220, 2016.
26

---

Published in Transactions on Machine Learning Research (11/2023)
Peter Varnai and Dimos V. Dimarogonas. On robustness metries for learning STL tasks. In Proc. of the American Control Conf. (ACC), pp. 5394-5399) 2020.
Stephen A. Vavasis. Complezity Theory: Quadratic Programming. Springer, Boston, MA., 2001.
Kim P. Wabersich and Melanie N. Zeilinger. predictive safety filter for learning-basedd control of constrained nonlinear dynamical systems. Automatica, 129(1):109597 109614, 2021.
Kim P. Wabersich, Andrew Taylor, Jason Choi, Koushil Sreenath, Claire J. Tomlin, Aaron D. Ames, and Melanie N. Zeilinger. Data-driven safety filters: Hamilton-Jacobi reachability, control barrier functions, and predictive methods for uncertain systems. IEEE Control Systems Magazine, 43(5):137-177, 2023.
Chengyu Wang, Luhan Wang, Zhaoming Lu, Xinghe Chu, Zhengrui Shi, Jiayin Deng, Tianyang Su, Guochu Shou, and Xiangming Wen. SRI-TR2: A safe reinforcement learning based trajectory tracker framework. IEEE Transactions on Intelligent Transportation Systems, 24(6):5775-55) 5780, 2023.
Linghao Wang, Miao Wang, and Yujun Zhang. A safe training approach for deep reinforcement learning- based traffic engineering. In Proc. of the IEEE Int. Conf. on Communicattinss (ICC), pp. 1450 1455, 2022.
Xiao Wang. Ensuring safety of baaring--based motion planners using control barrier functions. IEEE Robotics and Automation Letters, ((2):4773-478)) 2022.
Peter Wieland and Frank Allgöwer. Constructive safety using control barrier functions. IFAC Proc. Volumes, 40(12))462-467, 2007.
Cambridge Yang, Michael L. Littman, and Michael Carbin, On the (in)tractability of reinforcement learning for L.T. objectives. Proc. of the Trl. Joint Conf. on Artificial Intelligence (IJCAT), pp. 3650-366000 2022.
Chenchen Yang, Jing Liu, Haiying Sun, Junfeng Sun, Xiang Chen, and Lipeng Zhang, Safe reinforcement learning for CPSs via formal modeling and verification. In IEEE Int. Joint Conf. on Neural Networks Proc. (IJCNN), pp. 1-8, 2021.
policy optimization. In Proc. of the Int. Conf. on Learning Representations (ICLR), pp. 1-21, 2020. Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Projection-based constrained
Fei Ye, Shen Zhang, Pin Wang, and Ching Yao Chan. A survey of deep reinforcement learning algorithms for motion planning and control of autonomous vehicles. In Proc. of the IEEE Intelligent Vehicles Sympp (IV), pp. 1073-101 1080, 2021.
Peipei Yu, Hongcai Zhang, Yonghua Song, Hongxun Hui, and Ge Chen. District cooling system control for providing operating reserve basod on safe deep reinforcement learning. IEEE Transactions on Power Systems, pp. 13, 2023.
Mario Zanon and Sebastien Gros. Safe reinforcement learning using robust MPC. IEEE Transactions on Automatic Control, 66(8):3638 3652, 2021.
Melanie N. Zeilinger, Davide M. Raimondo, Alexander Domahidi, Manfred Morari, and Colin N. Jones. On real-time robust model predictive control. Automatica, 50(3):683 694, 2014.
Jin Zhang, Yuxiang Guan, Liang Che, and Mohammad Shahidchpour. Ev charging command fast allocation approach based on deep reinforcement learning with safety modules. IEEE Transactions on Smart Grid, 2023.
Wenshuai Zhao, Jorge Pena Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: A survey. In Proc. of the IEEE Symp. Series on Computational Intelligence (SSCI), pp. 737-744, 2020.
27

---

Published in Transactions on Machine Learning Research (11/2023)
A
Appendix
MDP modification with action replacement
Action replacement alters the MDP on which the agent learns. Hunt et al. (2021) discuss this modification for discrete action spaces and uniformly sampling from the safe action space. We generalize this discussion to using any replacement function and continuous action spaces. To this end, we define {b(s) so that it randomly replacement policy π, (ā(s) with Eaca)(e) Tr (āls)
samples the replacement action according the discrete case, and (a)(2) Tr, (ä s) dā = 1 for the continuous case, and vä As(() n,, (āls) 2 0. In the example of uniform sampling from (s), the replacement policy n. (ā)s) 1//V,, where (a,(s) is the volume of (s) By replacing unsafe actions, the transition function of the MDP changes to
for
as, 8'), if ((8,)) =1 (T)s, otherwise,
(15)
T (8, = Σ n, (als) T(s,ū,
(16)
(EAp(a)
The reward function of the MDP changes accordingly to
(x(8, a) r(s,a), f(((s,)==
(17)
(s),,,,
otherwise,
(s) E n. ((..).
(18)
(x)
In the continuous case, we get s') by marginalizing the transition probability density function over (s):
Tx(8,8') / n (āls) T(s ā, s s')d (x)
(19)
Analogously, we have that
(r(8) Tr (ā)s))(ss a).. Ap(a)
(20)
Environment parameters
We provide an overview of all environment--peeific parameters in Table and Table
Table 3: Environment parameters of the pendulum.
Parameter
Value
Gravity g
9.81 -
Mass m
1kg
Length
m
Hyperparameters for learning algorithms
We specify the hyperparameters for all learning algorithms (see Table 5 for PPO, Table for TD3, Table 7 for DQN, and Table for SAC) that are different from the Stable Baselines3 (Raffin al., 2021) default values. Additionally, the code for the experiments is available the CodeOcean capsule 2oi.or//122443//0000022 to reproduce results.
28

---

Published in Transactions on Machine Learning Research (11/2023)
Table 4: Environment parameters of the 2D quadrotor.
Parameter
Value
Gravity g
9.81 ms
k
11/kg
do
70
17
no W
55
[--01,0011,[0011.1111111
Table 5: Hyperparamoters for PPO.
Parameter
Pendulum
2D quadrotor
Learning rate
1 10
5 x 10
Discount factor 7
0.98
0.999
Steps per update Optimization epochs
2048
512
20
30
Minibatch size
16
128
Max gradient clipping
0.9
0.5
Entropy coefficient
1 x 10 10-3
2 10
Value function coefficient
0.015
0.5
Clipping range
0.3
0.1
Generalized advantage estimation À
0.8
0.92
Activation function
ReLU
ReLU
Hidden layers
2
2
Neurons per layer
32
Training steps
60k
200k
Table 6: Hyperparameters for TD3.
Parameter
Pendulum
2D quadrotor
Learning
3.5 x 10-3
x 10
Replay buffer size Discount factor ? Initial exploration steps Steps between model updates
1 x 10*
x 10
0.98
0.98
10 10
100
256
Gradient steps model update Minibatch size per gradient step Soft update coefficient
256
10
512
512
5 10
5 x 10-3
Ganssian smoothing noise a
0.2
0.12
Activation function
ReLU
ReL.U
Hidden layers
2
Neurons per layer
32
64
Training steps
60k
200k
29

---

Published in Transactions on Machine Learning Research (11/2023)
Table 7: Hyperparameters for DQN.
Parameter
Pendulum
quadrotor
Learning rate
x 10-3
x 10-4
Replay buffer size Discount factor y Initial exploration steps Steps between model updates Gradient steps per model update Minibatch size gradient step Maximum for gradient clipping Update frequency larget network Initial exploration probability Linear interpolation steps of
5 10*
x 10
0.95
0.999 99
500
100
8
512
64
10
100
x 10
x 10a
1.0
0.137
6 x 10
x 10*
Final exploration probability Activation function
0.1
0.004
tanh
tanh
Hidden layers
2
Neurons per layer
32
64
Training sleps
60k
200k
Table 8: Hyperparameters for SAC.
Parameter
Pendulum
2D quadrotor
Learning
3 10-4
10-4
Replay buffer size Discount factor Y Initial exploration steps Steps between model updates Gradient steps per model update Minibatch size per gradient step Entropy coefficient
1 x 10
5 x 10
0.99
0.98
100
1000
1
32
256
512
learned
x 10
Soft update coefficient
5 x 10-3
x 10
Activation function
ReLU
ReLU
Hidden layers
2
Neurons per layer
32
64
Training steps
60k
200k
30

---

Published in Transactions on Machine Learning Research (11/2023)
Full evaluation
In this section, we present all training results of the five RL algorithms TD3, SAC, DQN, and PPO continuous and discrete. We compare these algorithms on the inverted pendulum and 2D quadrotor environment on ten random seeds. The tested algorithms are action replacement with (sample and 4faiilaate (x), action projection, and action masking. When action replacement is used with a failsafe controller, we omit safe action and both because for discrete action spaces, the failsafe controller might use an action that is not the discrete action space. This is due to the failsafe controller proposing actions from the continuous action space.
First, we present the effect of the learning tuples on the on-policy algorithm PPO in Figure 5. This compar- ison clearly shows the negative effect of the safe action tuple on the training performance of PPO. Figure 6 depicts the aggregated training results for the pendulum as proviously discussed for the 2D quadrotor in Figures 3, and 5. Figures to 10 depict how the reward and intervention rate evolve during training for all investigated configurations. Finally, the Tables and 10 show statistical results for deploying the learned models for the two benchmarks.
Replacenent (taiss)
Replacement PPO (alapion penalty)
-0.1
0.8
Replacement (safcaction)
Reward
Intervention
0.6
Replacement PPO (beth)
0.4
14 16
8 10
12
20
Steps 10*)
Steps (in 10*)
(a) Reward of action replacement PPO.
(b) Intervention rate of action replacement PPO.
Figure 5: Evaluation of the training tuples for the 2D quadrotor averaged over the continuous and discrete PPO training runs using action replacement with ten random seeds each. The left column depicts the reward and the right column the safety intervention rate.
31

---

Published in Transactions on Machine Learning Research (11/2023)
Provably
-01
Safety Violation
PPO (disc) baseline
Reward
0.6
DQN bootline
huaciie
0.2
-10
0
5
Steps 10*)
Steps (in 10*)
(a) Reward of baselines.
(b) Safety violation rate of baselines.
Busclite
-01
Intervention Rate
0.8
Prejection
Masking
Reward
0.6
0.1
0.2
-10
Steps 10a)
Steps 10
(c) Reward of provably safe RL algorithms.
(d) Intervention rate of provably safe RL algorithms.
Replacemen (naive)
Replasemen Galaptioe penaly)
0.8
Replacement
Replacemen (bodh
Reward
Intervention
0.4
0.2
Steps (in 10a)
Steps (in 10a)
(c) Reward of action replacement.
(f) Intervention rate of action replacement.
Projection
-0.1
Pepeztion (ateption pesalty)
Projection (safe
Reward
Intervention
0.6
Projection
0.4
0.2
-10
Steps (in 10*)
Steps (in 10
(g) Reward of action projection.
(h) Intervention rate of action projection.
Replacemen PPO
0.1
Intervention Rate
0.8
Replacement PPO (adaption pesslty)
Replacement PPO (edl action)
Replasement PPO
Rewnrd
0.6
0.4
0.2
-10
0
0
2
5
Steps (n 1
Steps (in 10
(1) Reward of action replacement PPO.
(i) Intervention rate of action replacement PPO.
Figure 6: Evaluation the training tuples for the pendulum averaged over ten random seeds each. The left column depicts the reward and the right column the safety intervention rate. We would like to refer the reader to Figures 3, and for the corresponding 2D quadrotor results.
32

---

Published in Transactions on Machine Learning Research (11/2023)
Failsafe (naive)
Failsafe (adaption penalty)
Sample (naive)
Sample (adaption penalty)
Sample (safe action)
Sample (both)
Project (naive)
Project (adaption penalty)
Project (safe action)
Project (both)
Mask (naive)
Bascline
-0.1
-0.1
Reward
Reward
-10
-10
1
3
4
0
4
Steps (in 10*)
Steps (in 10*)
(a) TD3.
(b) SAC.
-0.1
-0.1
Reward
Reward
-1
-10
-10
Steps (in 10*)
Steps (in 10
(c) PPO continuous.
(d) PPO discrete.
-0.1
Reward
-1
-10
1
3
5
Steps (in 10
(e) DQN.
Figure 7: Pendulum: Average reward and standard deviation per training step for TD3, SAC, DQN, PPO discrete, and PPO continuous. For each configuration, len training runs with different random seeds were conducted. Each subplot contains implemented variants. Note that the reward for the adaption penalty variants is still and the adaption penalty y* is not included in the curves for better comparability.
33

---

Published in Transactions on Machine Learning Research (11/2023)
Failsafe (naive)
Failsafe (adaption penalty)
Sample (naive)
Sample (adaption penalty)
Sample (safe action)
Sample (both)
Project (naive)
Project (adaption penalty)
Project (safe action)
Project (both)
Mask (naive)
Bascline
-0.1
-0.1
Roward
Reward
2
4
10
12
14
16
18
20
0
2
6
8
10
12
14
16
18
20
Steps
10*)
Steps (in 10*)
(a) TD3.
(b) SAC.C
-0.1
-0.1
Reward
Reward
12
18
14
Steps (in 10
Steps (in 10*)
(c) PPO continuous.
(d) PPO discrete.
-0.1
Reward
4
6
10
12
14
16
18
20
Steps (in 10*)
(e) DQN.
Figure 8: 2D quadrotor: Average reward and standard deviation per training step for TD3, SAC, DQN, PPO discrete, and PPO continuous. For each configuration, ten training runs with different random seeds were conducted. Each subplot contains all implemented variants. Note that the reward for the adaption penalty variants is still and the adaption penalty y* is not included in the curves for better comparability.
34

---

Published in Transactions on Machine Learning Research (11/2023)
Failsafé (naive)
Project (naive)
Mask (naive)
Failsafe (adaption penalty)
Project (adaption penalty)
Sample (naive)
Project (safe action)
Sample (udaption penalty)
Project (both)
Sample (safe action)
Sample (both)
1
Intervention Rate
Intervention Rate
0.8
0.8
Intervention
0.6
0.4
0.4
0.2
0.2
0
Steps (in 10*)
Steps (in 10*)
Steps 10*)
(a) Replacement TD3.
(b) Projection TD3.
(c) Masking TD3.
Interventton Rate
Intervention Rate
Intervention Rate
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
2
3
5
6
2
3
5
2
4
5
Steps (in 10*)
Steps (in 10
Steps (in 10*)
(d) Replacement SAC.
(e) Projection SAC.
(f) Masking SAC.
Intervention Rate
Intervention Rate
Intervention Rate
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
3
6
2
4
5
0
2
3
5
Steps (in 1
Steps (in 10
Steps (in 10a)
(g) Replacement PPO cont.
(h) Projection PPO cont.
(i) Masking PPO cont.
Intervention Rate
Intervention Rate
0.8
0.8
Intervention
0.6
0.6
0.4
0.4
0.2
0.2
Steps (in 10a)
Steps (in 10
Steps 10
Replacement PPO disc.
(k) Projection PPO disc.
(I) Masking PPO disc.
Intervention Rate
Intervention Rate
0.8
0.8
0.8
Intervention
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0
0
0
3
6
2
4
5
0
2
3
5
Steps (in 10
Steps (in 10*)
Steps (in 10a)
(m) Replacement DQN.
(n) Projection DQN.
(a) Masking DQN.
Figure 9: Pendulum: Intervention rate for TD3, SAC, PPO discrete, PPO continuous, and DQN.
35

---

Published in Transactions on Machine Learning Research (11/2023)
Failsafe (naive)
Project (naive)
Mask (naive)
Failsafe (adaption penalty)
Project (adaption penalty)
Sample (naive)
Project (safc action)
Sample (adaption penalty)
Project (both)
Sample (safe action)
Sample (hoth)
Intervention Rate
Intervention Rate
0.8
Intervention
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0
0
6
8
10
16 18
4
6
10
12
14
16
18
20
2
4
12
14
16
20
Steps (in 10*)
Steps (in 10*)
Steps 10*)
(a) Replacement TD3.
(b) Projection TD3.
(c) Masking TD3.
Intervention Rate
Intervention Rate
0.8
0.8
0.8
Intervention
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
8
10
16
18
16
12
14
20
Steps (in 104)
Steps (in 10*)
Steps (in 10*)
(d) Replacement SAC.
(c) Projection SAC.
(f) Masking SAC.
Intervention Rate
Intervention Rate
Intervention Rate
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
4
6
8
10
12
14
16
18
20
4
6
8
10
12
14
16
18
20
2
4
6
10
12
14
16
18
20
Steps (in 10*)
Steps (in 10*)
Steps (in 10*)
(g) Replacement PPO cont.
(h) Projection PPO cont.
(i)
Masking PPO cont.
Intervention Rate
Intervention Rate
0.8
Intervention
0.6
0.4
0.4
0.4
0.2
0.2
0
8
10
16
18
4
6
14 16
4
12
14
20
Steps (in 104)
Steps (in 104)
Steps
10*)
(j)
Replacement PPO
(k)
Projection PPO disc.
(I) Masking PPO
1
Intervention Rate
Intervention Rate
0.8
0.8
0.8
Intervention
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0
8
10 14
16
18
14 16
4
12 14
20
Steps (in 104)
Steps (in 10*)
Steps (in 10*)
(m) Replacement DQN.
(n) Projection DQN.
(o) Masking DQN.
Figure 10: 2D quadrotor: Intervention rate for TD3, SAC, PPO discrete, PPO continuous, and DQN.
36

---

Published in Transactions on Machine Learning Research (11/2023)
Table 9: Mean and standard deviation of 30 pendulum deployment episodes.
Reward
Intervention Rate
Safety Violation
Approach
MEAN
STO. DEV.
MEAN
STO. DEV.
MEAN
SID. DEV.
PPO (continuous)
PROJECTION (SAPRACTION)
-1.14
0.42
0.76
0.29
0.00
0.00
PROJECTION (ADAPTION PENAITY)
-0.07
0.07
0.00
0.00
0.00
0.00
PROJECTION (BoTU)
-0.07
0.07
0.00
0.00
0.00
0.00
PROJECTION (NAIVE)
-0.06
0.07
0,00
0.00
0.00
0.00
SAMPLR [SAFEACTION]
-0.13
0.16
0.01
0.02
0.00
0.00
SAMPLE (ADAPIIONPINAATT)
-0.07
0.07
0.00
0.00
0.00
0.00
SAMPLE (BoTH)
-0.07
0.07
0.00
0.00
0.00
0.00
SAMPER (NAIVE)
-0.07
0.07
0.00
0.00
0.00
0.00
FARSAPE (ADAPTIONPENAIE)
-0.07
0.07
0.00
0.00
0.00
0.00
FAILAFPE (NAIVE)
-0.07
0.07
0.00
0.00
0.00
0.00
BASELINE (NAIVE)
-0.06
0.07
0.00
0.00
MASKING (NAIVE)
-0.07
0.07
0.00
0.00
0.00
0.00
PPO (discrete)
PROJECTION (SAPEACTION)
-0.52
0.55
0.28
0.42
0.00
0.00
PROJECTION (ANAPTIONPESALIT)
-0.14
0.12
0.00
0.00
0.00
0.00
PROJECTION (BOTH)
-0.09
0.09
0.00
0.00
0.00
0.00
PROJECTION (NAIVE)
-0.15
0.27
0,03
0.00
0.00
SAMPLE (SAFEACTION)
-0.09
0.08
0.00
0.00
0.00
0.00
SAMPLR (ADAPTIOPPINAATT)
-0.13
0.16
0.00
0.00
0.00
0.00
SAMPLE (BoTn)
-0.10
0.08
0.00
0.00
0.00
0.00
SAMPLE (NAIVE)
-0.09
0.08
0.00
0.00
0.00
0.00
FAILSAFE (ADAPTIONPENALTT)
-0.09
0.07
0.00
0.00
0.00
0.00
PAI.SAPR (NAIVe)
-0.08
0.07
0.00
0.00
0.00
0.00
BASELINE (NAIVE)
-0.08
0.07
0.00
0.00
MASKING (NAIVE)
-0.07
0.07
0.00
0.00
0.00
0.00
TD3
PROJECTION (SAFEACTION)
-0.07
0.07
0.00
0.00
0.00
0.00
PROJECTION (ADAPTIONPENAITY)
-0.08
0.08
01.00
0.00
0.00
0.00
PROJECTION (BOTH)
-0.07
0.07
0.00
0.00
0.00
0.00
PROJECTION (NAIve)
-0.07
0.07
0.00
0.00
0.00
0.00
SAMPLE (SAFEACTION)
-0.07
0.07
0.00
0.00
0.00
0.00
SAMPLE (ADAPTIONPENNALTY)
40.09
0.07
0,00
0.00
0.00
0.00
SAMPLE (Born)
-0.09
0.07
0.00
0.00
0.00
0.00
SAMPLE (NAIVE)
40.07
0.07
0,00
0.00
0.00
FAILSAPE (ADAPTIONPENALTT)
-0.09
0.07
0.00
0.00
0.00
0.00
PAILSAPR (NaIve)
-0.07
0.07
0.00
0.00
0.00
0.00
BASELINE (NAIVE)
-0.07
0.07
0.00
0.00
MASKING (NAIVE)
-0.07
0.07
0.00
0.00
0.00
0.00
DQN
PROJECTION (SAPRACTION)
-0.07
0.07
0.00
0.00
0.00
0.00
PROJECTION (ADAPTIONPESALTT)
-0.07
0.07
0.00
0.00
0.00
0.00
PROJECTION (BOTH)
-0.07
0.08
0.00
0.00
0.00
0.00
PROJECTION (NAIVE)
-0.07
0.07
0.00
0.00
0.00
0.00
SAMPLE (SAFEACTION]
-0.07
0.07
0.00
0.00
0.00
SAMPLE (ADAPIIONPPENALTY)
40.09
0.07
0.00
0.00
0.00
0.00
SAMPLR (BOTH)
-0.07
0.08
0.00
0.00
0.00
0.00
SAMPLE (NAIVE)
-0.07
0.07
0.00
0.00
0.00
0.00
PAILSAPE (ADAPTIONPENAATT)
-0.07
0.07
0.00
0.00
0.00
0.00
PAILSAPK (NAIVE)
-0.07
0.07
0.00
0.00
0.00
0.00
BASELINE (NAIVE)
-0.07
0.07
0.00
0.00
MASKING (NAIVE)
-0.07
0.07
0.00
0.00
0.00
SAC
PROJECTION (NAIVR)
-0.08
0.07
0.00
0.00
0.00
0.00
PROJECTION ALDAPTIOPENAATT
-0.10
0.09
0.00
0.00
0.00
0.00
PROJECTION (SAFEACTIOK)
-0.08
0.07
0.00
0.00
0.00
0.00
PROJECTION (BOTR)
-0.10
0.08
0.00
0.00
0.00
SAMPLE (NAIVE)
-0.08
0.07
0.00
0.00
0.00
0.00
SAMPLE (ADAPTIONPENNATT)
-0.10
0.08
0.00
0.00
0.00
0.00
SAMPLR (SAFFACTIOO]
-0.08
0.07
0.00
0.00
0.00
0.00
SAMPLE (Bors)
-0.09
0.08
0.00
0.00
0.00
0.00
FAILAAEE (NAIVE)
-0.08
0.07
0.00
0.00
0.00
0.00
FAISAFPE (ADAPTIONPENAATT))
-0.09
0.09
0.00
0..0
0.00
0.00
BASELINE (NAIVE)
-0.11
0.09
0.00
0.00
MASKING (NAIVE)
-0.08
0.07
0.00
0.00
0.00
0.00
Note: indicates that there is no intervention rate for the baselines as they don't implement a safety verification.
37

---

Published in Transactions on Machine Learning Research (11/2023)
Table 10: Mean and standard deviation of 30 2D Quadrotor deployment episodes.
Reward
Intervention Rate
Safety Violation
Approach
MEAN
STO. DEV.
MEAN
STO. DEV.
MEAN
SID. DEV.
PPO (continuous)
PROJECTION (SAPRACTION)
-0.44
0.00
0.68
0.00
0.00
0.00
PROJECTION (ADAPTION PENAITY)
-0.33
0.07
0.44
0.05
0.00
0.00
PROJECTION (BoTu)
-0.39
0.07
0.57
0.13
0.00
0.00
PROJECTION (NAIVE)
-0.31
0.10
0.47
0:25
0.00
0.00
SAMPLR [SAFEACTION]
-0.43
0.00
0.86
0.00
0.00
0.00
SAMPLR (ADAPTIONPINAATT)
-0.25
0.12
0.10
0.12
0.00
0.00
SAMPLE (BoTH)
-0.36
0.03
0.41
0.20
0.00
0.00
SAMPER (NAIVE)
-0.39
0.06
0.23
0.13
0.00
0.00
FARSAPE (ADAPTION
-0.31
0.11
0.08
0.06
0.00
0.00
FAILAFPE (NAIVE)
-0.27
0.11
0.27
0:29
0.00
0.00
BASELINE (NAIVE)
.0.86
0.01
0.94
0.01
MASKING (NAIVE)
-0.43
0.09
0.57
0.28
0.00
0.00
PPO (discrete)
PROJECTION (SAPEACTION)
-0.41
0.01
0.74
0:21
0.00
0.00
PROJECTION (ANAPTIONPESALTT)
-0.24
0.13
0.02
0.02
0.00
0.00
PROJECTION (BOTH)
-0.35
0.11
0.16
0.14
0.00
0.00
PROJECTION (NAIVE)
40.34
0.14
0.46
0.00
0.00
SAMPLE (SAFEACTION)
-0.42
0.02
0.82
0.01
0.00
0.00
SAMPLR (ADAPTIOPPINARTT)
-0.11
0.10
0.00
0.00
0.00
0.00
SAMPLE (BoTn)
-0.38
0.09
0.32
0.18
0.00
0.00
SAMPLE (NAIVE)
-0.13
0.16
0.02
0.03
0.00
0.00
FAILSAFE (ADAPTIONPENALTT)
-0.19
0.15
0.01
0.03
0.00
0.00
PAI.SAPR (NAIVe)
-0.34
0.13
0.41
0.21
0.00
0.00
BASELINE (NAIVE)
0.03
0.00
0.00
MASKING (NAIVE)
-0.25
0.14
0.28
0.21
0.00
0.00
TD3
PROJECTION (SAFEACTION)
-0.21
0.03
0.28
0.00
0.00
PROJECTION (ADAPTIONPENAITY)
-0.22
0.03
01.26
0.10
0.00
0.00
PROJECTION (BOTH)
-0.21
0.04
0.28
0.00
0.00
PROJECTION (NaIve)
-0.25
0.05
0.26
0.17
0.00
0.00
SAMPLE (SAFEACTION)
-0.18
0.03
0.07
0.01
0.00
0.00
SAMPLE (ADAPTIONPENNALTY)
-0.21
0.04
0.13
0.05
0.00
0.00
SAMPLE (Born)
-0.21
0.06
0.06
0.03
0.00
0.00
SAMPLE (NAIVE)
40.19
0.04
0.12
0.00
0.00
FAILSAPE (ADAPTIONPENALTT)
-0.20
0.03
0.05
0.02
0.00
0.00
PAILSAPR (NaIve)
-0.26
0.09
0.05
0.02
0.00
0.00
BASELINE (NAIVE)
-0.90
0.05
0.95
0.02
MASKING (NAIVE)
-0.16
0.03
0.03
0.03
0.00
0.00
DQN
PROJECTION (SAPRACTION)
-0.06
0.02
0.00
0.01
0.00
0.00
PROJECTION (ADAPTIONPESAATT)
-0.05
0.00
0.00
0.00
0.00
0.00
PROJECTION (BOTH)
-0.05
0.01
0.00
0.00
0.00
0.00
PROJECTION (NAIVE)
-0.09
0.06
0.12
0.24
0.00
0.00
SAMPLE (SAFEACTION]
-0.07
0.01
0.01
0.00
0.00
SAMPLE (ADAPIIONIPENALTY)
-0.06
0.03
0.00
0.00
0.00
0.00
SAMPLR (BOTH)
-0.07
0.02
0.00
0.00
0.00
0.00
SAMPLE (NAIVE)
-0.05
0.01
0.00
0.00
0.00
0.00
PAILSAPE (ADAPTIONPENAATT)
-0.06
0.02
0.02
0.04
0.00
0.00
PAILSAPK (NAIVe)
-0.07
0.03
0.10
0.10
0.00
0.00
BASELINE (NAIVE)
-0.21
0.37
0.20
0.40
MASKING (NAIVE)
40.15
0.16
0.14
0.00
0.00
SAC
PROJECTION (NAIVR)
-0.20
0.01
0.52
0.02
0.00
0.00
PROJECTION ALDAPIINPENALIT
-0.19
0.00
0.49
0.01
0.00
0.00
PROJECTION (SAFEACTIOK)
-0.19
0.00
0.419
0.01
0.00
0.00
PROJECTION (BOTR)
40.20
0.01
0.49
0.00
0.00
SAMPLE (NAIVE)
-0.15
0.01
0.05
0.00
0.00
0.00
SAMPLE (ADAPTIONPENNATT)
-0.15
0.01
0.05
0.00
0.00
0.00
SAMPLR (SAFFACTIOO)
-0.15
0.01
0.05
0.00
0.00
0.00
SAMPLE (Born)
-0.16
0.02
0.05
0.00
0.00
0.00
FAILAAEE (NAIVe)
-0.21
0.01
0.17
0.00
0.00
FAISAFPE (ADAPTIONPENAATT))
-0.17
0.01
0.04
0.00
0.00
0.00
BASELINE (NAIVE)
-0.88
0.02
0.96
0.03
MASKING (NAIVE)
-0.11
0.02
0.00
0.00
0.00
0.00
Note: indicates that there no intervention rate for the baselines as they don't implement a safety verification.
38


Reviews:
[{"review_id": "nVT9ECYlCZ", "reviewer": "Reviewer_xEjt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- This survey paper is clear and easy to read, and has detailed coverage of several different prior works. \n- The paper attempts to provide unifying perspective on prior works by categorizing them into different groups based on how safety is ensured, and through experiments in consistent environments that provide perspectives on relative strengths/weaknesses of these different classes of safe RL algorithms\n- The overall summaries of prior works in terms of algorithm designs and applications in the two big tables are helpful in distilling the differences at a glance. \n\nWeaknesses\n\n- the paper does not provide a lot of insights beyond what prior works already cover (in particular other surveys like Brunke et al. 2022) and the there isn't a lot of interesting conceptual  comparisons beyond the three broad categories. \n\n- the paper does not provide any comparisons of how the theoretical guarantees in the prior works differ from each other, and how they can be viewed under a unified lens. Which papers have better guarantees under certain constraints of the environment needs to come out more clearly. Right now, the comparisons are mostly at the level of algorithm design.\n\n- the motivation of the paper is about practical considerations of why provably safe RL is necessary for real world deployments, but the experimental comparisons are very limiting. I am not sure why a 1D pendulum environment, and a 2D quadrotor environment provide insights for safe RL algorithms that would have the possibility of translating to real world systems (that are typically high dimensional both in terms of observation space and action space) For examples of such environments (in simulation), refer to Fig 5 of prior survey paper Brunke et al. 2022\n\nBroadly, given the above limitations, and with existing prior survey+comparison papers like Brunke et al. 2022, this paper does not provide any interesting insights that are likely to be helpful in developing safe RL algorithms of the future.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a survey of provably safe reinforcement learning algorithms, and an experimental comparison of some prior approaches on simple simulated control environments. The paper categorizes prior works under three categories: action replacement, action projection, and action masking depending on how safe behavior is guaranteed by the algorithm. The experiments evaluate these different types of algorithms and attempts to provide consolidated insights into which type of algorithm might be suited for a particular type of problem. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the detailed list of weaknesses above. In summary:\n\n1) is it possible to provide any comparisons of how the theoretical guarantees in the prior works differ from each other, and how they can be viewed under a unified lens?\n\n2) is it possible to provide the experimental comparisons in more realistic (simulated / real world) environments, for example the ones described in Brunke et al. 2022?", "broader_impact_concerns": "None. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "muAfIvx3dh", "reviewer": "Reviewer_CFAn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The authors provide a thorough literature review and classification of the existing work on safe RL algorithms.  This helps summarize the entire field, develop insights for implementing the algorithms in new domains, and potential ideas for modifying existing methods to fit the given application.\n- Based on the experimental results, on page 14 the authors provide a succinct summary on their experience implementing provably safe RL algorithms to the domains considered in the paper. This helps provide actionable insight for implementing the algorithms in practice.\n\n### Weaknesses\n- The experimental results seem limited and hard to derive statistical significance on the comparison between the different methods.\n- The discussion on implementing action masking in continuous state-action spaces is under-developed and could be expanded to more general convex action spaces.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### High Level Overview\n\nReinforcement learning has been used in different areas and applications such as robotics, autonomous systems, and games.  However, in safety-critical applications it is important to design and develop RL algorithms with safety guarantees.  For example, consider using an RL algorithm for autonomous driving. Whether the algorithm is trained using simulations or with historical data, the trained policy must abide by road constraints and ensure safety of the vehicle and its surroundings.  Vanilla RL algorithms are typically set-up to explore potentially unsafe regions focused solely on reward maximization.  In contrast, safe RL algorithms limit the exploration and execution to only consider “safe” parts of the state-action space.  At a high level, the authors here provide a survey of “provably safe” RL algorithms, where provably safe refers to the fact that safety constraints are satisfied both during training and also during test time.  They introduce a categorization of existing provably safe RL methods into three types: action replacement, action projection, and action masking.  Lastly, they provide experimental evaluation to confirm that existing algorithms from the safe RL field indeed satisfy the safety constraints, and provide insights into which algorithm framework should be used in a given application area.\n\n### Main Model\n\nRL is build upon the framework of an MDP, defined as a tuple $(S,A,T,r,\\gamma)$ where $S$ is set of states, $A$ set of actions, $T$ the transition distribution, $r$ the rewards, and $\\gamma$ the discount factor.  For provably safe RL it is required that safety of state-action pairs are verifiable, hence the authors assume there is a safety function $\\phi(s,a)$ which outputs $1$ if the state-action pair is verified to be safe, and zero otherwise.  It is clear that executing $\\phi(s,a)$ or even the knowledge of such a function requires some prior knowledge.  However, in settings without some knowledge of the dynamics that provably safe RL is not feasible.  Lastly, the authors make the following key assumption that there is at least one provably safe initial state, and that for all safe states there exists at least one safe action.\n\n### Summary\n\nAction Replacement: The first approach the authors discuss to ensure safety is to replace any unsafe action outputted by the agent with a safe one before it gets executed.  Indeed, if the policy $\\pi$ proposes an action $a$, then if $\\phi(s,a) = 1$ then $a$ is taken.  Otherwise, the authors assume there is a replacement policy $\\psi(s)$ which always guarantees to output a provably safe action, and so $\\psi(s)$ is instead taken.  Note that $\\psi(s)$ can either be sampled randomly from the safe set, or using a failsafe controller.\n\nAction Projection: The second approach in the literature is action projection. Here, if the action proposed by the policy $\\pi$ is unsafe, then the actual action taken solves the following optimization problem, whereby you minimize $d(a, \\tilde{a})$ subject to $\\phi(s,\\tilde{a}) = 1$, i.e. project the action $a$ to the set of feasible actions and find the closest point.  This can be formulated using either control barrier functions or using additional domain knowledge.\n\nAction Masking: The last approach limits the action space $A$ that is fed into the RL algorithm.  Indeed, assume you are given a function $\\eta(s)$ that outputs a set of provably safe actions.  Then the policy $\\pi$ which is learned is informed by both $\\eta(s)$ (dictating the actions available in that state) as well as the current state $s$.  While this is straightforward in discrete action spaces, the authors additionally outline its implementation with continuous action spaces.\n\n### Experimental Results\n\nLastly, the authors complement their literature review with experimental results.  They compare “unsafe” baseline algorithms to their “safe” counterparts on two tasks (a pendulum and quadrotor task).  The results highlight that the safe algorithms indeed satisfy the safety constraints both during training and during test.  However, they additionally notice that action replacement is the most stable and high-performing method.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Requested Changes\n- The authors should provide a discussion on the distinction between implementing these algorithms in an offline versus an online setting. This is briefly discussed on page 5 but not elsewhere.  There are important considerations between the feedback, adjusting for importance sampled weights for the behaviour policy, etc, which should be included.\n- The experimental results seem to provide no statistical insights between the performance of the different algorithms (especially with the inclusion of the confidence intervals).  This makes it difficult to appreciate one of the key findings that action replacement is the most stable and high-performing method (when in reality the performance is statistically indistinguishable).  Potentially this could be ameliorated by running more experiments on different areas.  However, as written in Section 4.3 it is difficult to derive any actionable insights form the experiments other than safe RL methods are indeed safe.  The first paragraph in page 14 is helpful to this, and could be used as a guide for the rest of the section.\n- Action masking for continuous state-aciton pairs seems under-developed. The authors only consider a straightforward implementation of it, without any extensions to arbitrary convex action spaces, or tools for potentially computing a \"safe\" bounding box within the safe set $A_\\phi$ using tools from optimization for generating non-axis parallel transformations.\n\n### Questions\n- What is the reason for the performance gap on the figure in page 13 between “safe” and baseline algorithms? Does this imply that the optimal policy does not necessarily follow a safe trajectory? Issues in convergence for the provably safe RL algorithms?\n- Do you have any thoughts on extensions for action masking to other continuous action space?\n\n### Minor Comments\n- Language in the first paragraph is a bit over-the-top and could be toned down\n- Third paragraph on page one is a nice succinct summary of the main contributions in the paper\n- Under Section 1.1 should mention that consider “provable” as only “hard” safety constraints and highlight that it means safe during both learning and execution\n- Loved the diagram in Figure 1 - helped highlight the different approaches and taxonomy of safe RL algorithms proposed\n- Page 3 under “safety of system” can highlight how can incorporate aggregate or cost constraints by augmenting the state space (e.g. considering budget)\n- Safe action set A_s(s) seems poorly defined and used a couple places throughout the paper. Recognize that you are trying to highlight that the safety function $\\phi$ doesn’t need to necessarily be perfect and allow for some type 2 errors (i.e. the state is safe but it is predicted by $\\phi$ to be unsafe) but no type 1 errors.  Think that this could be clarified and explained a bit more, since the notation is used later in in the discussion with the taxonomy\n- First paragraph under section 2.1 discussing the evaluation of $\\phi$ or over-approximating the set of safe states could be moved to the previous section\n- Top of page 7 in equation (9) should be defined by $\\eta(s)$\n- Enjoyed discussion on page 7 starting with “Since the action spaces for RL …”\n- Results in Table 2 seemed very interesting, highlighting real-world high fidelity simulations. Would have enjoyed some more discussion into the practical insights from these papers in section 3\n- First paragraph on page 14 was great - provided nice actionable advice\n", "broader_impact_concerns": "The work provides a general purpose summary of proposed RL algorithms for safety-critical applications.  The authors provide no new algorithms or implementations, mostly summarize existing algorithms and ideas in the field.  Lastly, the authors mention that researchers utilizing safe RL algorithms should adhere to ethical standards within their particular context.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nw12xEqKVY", "reviewer": "Reviewer_9Fsx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n   * Overall goal of identifying different action selection methods is good\n   * Clear presentation of taxonomy; seems potentially useful as well. E.g., pointing out the relative lack of work on action masking in some applications and for continuous actions\n   * Discussion of learning tuples illuminates different design choices and potential consequences\n   * Methodology for lit review seems good\n\n\nWeaknesses\n   * Experimental section is quite weak\n      * Overall goal is either too strong (“The theory and experiments confirm that provably safe RL methods are always safe, while the baselines still violate the safety results even after the reward converged.”) or unclear\n      * Small sample size so can’t give strong recommendations\n         * Not a lot of algorithms evaluated\n         * Not a lot of environments\n         * Hard to make claims about the categories because only a few algorithms are evaluated; claims are only particular to the algorithms\n         * Not many seeds used\n      * Hard to interpret results of fig 3\n         * “A notable result of our evaluation is that using the adapted action…” -> unclear to me from plots\n         * Error bars are large\n         * Too much going on, potentially too many colours\n   * Relatively weak discussion of safety in the related work\n      * Safety in the context of this work is with respect to the objective function, but it is often difficult in practice to get that objective function\n      * Should talk about reward learning (https://arxiv.org/abs/1706.03741), reward hacking (https://openreview.net/forum?id=yb3HOXO3lX2), goal misgeneralization (https://arxiv.org/abs/2105.14111)\n   * Missing a discussion of the limitations of the safe RL framework\n      * For example, can the constraint function take into account safe and unsafe trajectories since the constraint is only for a given state and action?\n      * Maybe whether a state is desirable or not depends upon the states that have previously been visited; is this accounted for in the MDP formulation?\n   * Scattered discussion of the benefits of this taxonomy\n      * Not clear that there are enough clear and strong future research directions identified. E.g., should there be more action masking work, and why?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a taxonomy of action selection methods for provably safe RL, comprehensively categorizing existing methods according to this taxonomy. In particular, the types of methods are action replacement, action projection, and action masking. The paper also discusses the different design choices for learning that must be made in light of each action selection method. The experimental results compare certain action selection methods. The paper also discusses practical insights that could prove useful for implementing these methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The following are the most significant requested changes.\n   * Strengthen experimental section in line with my comments above. Either a more thorough experimental evaluation of action selection methods overall or a more focused evaluation of the properties of a few select action selection methods could work. \n   * Need a more organized discussion of the benefits of this taxonomy for future work\n\nAt the moment, I am leaning towards rejection of the paper unless the above are addressed. Some other issues below are lower priority, but which would greatly improve the paper in my opinion.\n\n   * Improve discussion of related work on safe RL in line with my comments above\n   * Include a discussion of the limitations of the present safe RL framework\n   * Learning tuples should maybe be its own separate section since it’s relevant for all of the action selection methods\n   * Hard to compare rewards of baselines w/ provably safe methods in fig 2; maybe group the lines different or make a separate plot\n\n\nEDIT: All of my concerns have now been addressed by the authors and I have changed my score", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I am recommending that the paper be accepted with minor revisions. The authors addressed several initial shortcomings pointed by reviewers, and improved the presentation and organization of the paper. For completeness, I suggest that authors cite and contextualize a few more papers from RL + logic and formal verification communities, since provable safety is a core requirement in this survey. Here are a few examples:

LTL and Beyond: Formal Languages for Reward Function Specification in Reinforcement Learning. Camacho et al, IJCAI '19

LTL Realizability via Safety and Reachability Games. Camacho et al, IJCAI '18

A Symbolic Approach to Safety LTL Synthesis. Zhu et al. HVC '17

Policy Synthesis and Reinforcement Learning for Discounted LTL*. Alur et al. CAV '23

Reinforcement Learning With Temporal Logic Rewards. Li et al. IROS '17

Q-Learning for Robust Satisfaction of Signal Temporal Logic Specifications. Aksaray et al. CDC '16

On the (In)Tractability of Reinforcement Learning for LTL Objectives. Yang et al. IJCAI '22 

There are multiple other papers that one could cite here, but I'm hoping the authors will manage to present this literature in a more complete way. Other than that, the paper is ready to be accepted. Thank you to the authors for addressing all the issues that the reviewers brought forth.
Decision: Accept (poster)