Title: Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Safety Exploration
Abstract: Safe exploration is essential for the practical use of reinforcement learning (RL) in many real-world scenarios. In this paper, we present a generalized safe exploration (GSE) problem as a unified formulation of common safe exploration problems. We then propose a solution of the GSE problem in the form of a meta-algorithm for safe exploration, MASE, which combines an unconstrained RL algorithm with an uncertainty quantifier to guarantee safety in the current episode while properly penalizing unsafe explorations before actual safety violation to discourage them in future episodes. The advantage of MASE is that we can optimize a policy while guaranteeing with a high probability that no safety constraint will be violated under proper assumptions. Specifically, we present two variants of MASE with different constructions of the uncertainty quantifier: one based on generalized linear models with theoretical guarantees of safety and near-optimality, and another that combines a Gaussian process to ensure safety with a deep RL algorithm to maximize the reward. Finally, we demonstrate that our proposed algorithm achieves better performance than state-of-the-art algorithms on grid-world and Safety Gym benchmarks without violating any safety constraints, even during training.

Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms

Abstract

Safe exploration is essential for the practical use of reinforcement learning (RL) in many real-world scenarios. In this paper, we present a generalized safe exploration ( GSE ) problem as a unified formulation of common safe exploration problems. We then propose a solution of the GSE problem in the form of a meta-algorithm for safe exploration, MASE , which combines an unconstrained RL algorithm with an uncertainty quantifier to guarantee safety in the current episode while properly penalizing unsafe explorations before actual safety violation to discourage them in future episodes. The advantage of MASE is that we can optimize a policy while guaranteeing with a high probability that no safety constraint will be violated under proper assumptions. Specifically, we present two variants of MASE with different constructions of the uncertainty quantifier: one based on generalized linear models with theoretical guarantees of safety and near-optimality, and another that combines a Gaussian process to ensure safety with a deep RL algorithm to maximize the reward. Finally, we demonstrate that our proposed algorithm achieves better performance than state-of-the-art algorithms on grid-world and Safety Gym benchmarks without violating any safety constraints, even during training.

1 Introduction

Safe reinforcement learning (RL) is a promising paradigm that enables policy optimizations for safety-critical decision-making problems (e.g., autonomous driving, healthcare, and robotics), where it is necessary to incorporate safety requirements to prevent RL policies from posing risks to humans or objects [14]. As a result, safe exploration has received significant attention in recent years as a crucial issue for ensuring the safety of RL during both the learning and execution phases [6].

Safe exploration in RL has typically been addressed by formulating a constrained RL problem in which the policy optimization is subject to safety constraints [9, 18]. While there have been many attempts under different types of constraint representations (e.g., expected cumulative cost [3], CVaR [29]), satisfying constraints almost surely or with high probability received less attention to date. Imagine safety-critical applications such as planetary exploration where even a single constraint violation may result in catastrophic failure. NASA's engineers hope Mars rovers to ensure safety at least with high probability [8]; thus, constraint satisfaction 'on average' does not fit their purpose.

While several algorithms have addressed this problem with this stricter notion of safety, there are several formulations in terms of how the constraints are represented, including cumulative [32], state [39], and instantaneous constraints [41], which respectively correspond to Problems 1, 2, and 3

as we will discuss shortly in Section 2. Unfortunately, there has been limited discussion on the relationships between these approaches, making it challenging for researchers to acquire a systematic understanding of the field as a whole. If a generalized problem were to be formulated, then the research community could pool their efforts to develop suitable algorithms.

A closer examination of existing algorithms that span the entire theory-to-practice spectrum reveals several areas for improvement. Practical algorithms using deep RL (e.g., [32],[39],[40]) may provide satisfactory performance after convergence, but do not usually guarantee safety during training. In contrast, theoretical studies (e.g., [4], [41]) that guarantee safety with high probability during training often have limitations, such as relying on strong assumptions (e.g., known state transition) or experiencing decreased performance in complex environments. In summary, many algorithms have been proposed in various safe RL formulations, but the creation of a safe exploration algorithm that is both practically useful and supported by theoretical foundations remains an open problem.

Contributions. We first present a generalized safe exploration ( GSE ) problem and prove its generality compared with existing safe exploration problems. By taking advantage of the tractable form of the safety constraint in the GSE problem, we establish a meta-algorithm for safe exploration, MASE . This algorithm employs an uncertainty quantifier for a high-probability guarantee that the safety constraints are not violated and penalizes the agent before safety violation, under the assumption that the agent has access to an 'emergency stop' authority. Our MASE is both practically useful and theoretically well-founded, which allows us to optimize a policy via an arbitrary RL algorithm under the high-probability safety guarantee, even during training. We then provide two specific variants of MASE with different uncertainty quantifiers. One is based on generalized linear models (GLMs), for which we theoretically provide high-probability guarantees of safety and near-optimality. The other is more practical, combining a Gaussian process (GP, [27]) to ensure safety with a deep RL algorithm to maximize the reward. Finally, we show that MASE performs better than state-of-the-art algorithms on the grid-world and Safety Gym [28] without violating any safety constraints, even during training.

2 Preliminaries

Definitions. We consider an episodic safe RL problem in a constrained Markov decision process (CMDP, [3]), M = ⟨ S , A , H, P , r, g, s 1 ⟩ , where S is a state space, A is an action space, H ∈ Z > 0 is a (fixed) length of each episode, P : S × A × S → [0 , 1] is a state transition probability, r : S × A → [0 , 1] is a reward function, g : S × A → [0 , 1] is a safety (cost) function, and s 1 ∈ S is an initial state. At each discrete time step, with a given (fully-observable) state s , the agent selects an action a with respect to its policy π : S → A , receiving the new state s ′ , reward r , and safety cost g . Though we assume a deterministic policy, our core ideas can be extended to stochastic policy settings. Given a policy π , the value and action-value functions in a state s at time h are respectively defined as

and Q π r,h ( s, a ) := E π [ ∑ H h ′ = h γ h ′ r r ( s h ′ , a h ′ ) | s h = s, a h = a ] , where the expectation E π is taken over the random state-action sequence { ( s h ′ , a h ′ ) } H h ′ = h induced by the policy π . Additionally, γ r ∈ (0 , 1] is a discount factor for the reward function. In the remainder of this paper, we define V max := 1 -γ H r 1 -γ r and let T h : ( S × A → R ) → ( S × A → R ) denote the Bellman update operator T h ( Q )( s, a ) := E [ r ( s h , a h ) + γ r V Q ( s h +1 ) | s h = s, a h = a ] , where V Q ( s ) := max a ∈A Q ( s, a ) .

Three common safe RL problems. We tackle safe RL problems where constraints must be satisfied almost surely, even during training. While such problems have garnered attention in the research community, there are several types of formulations, and their relations are yet to be fully investigated.

One of the most popular formulations for safe RL problems involves maximizing V π r := V π r, 1 ( s 1 ) under the constraint that the cumulative cost is less than a threshold, which is described as follows:

Problem 1 (Almost surely safe RL with cumulative constraint [32]) .

where ξ 1 ∈ R ≥ 0 is a constant representing a threshold, and γ g ∈ (0 , 1] is a discount factor for g .

Observe that, the expectation is not taken regarding the safety constraint in Problem 1. This problem was studied in [32], which is stricter than the conventional one where the expectation is taken with respect to the cumulative safety cost function (i.e., E π [ ∑ H h =1 γ h g g ( s h , a h ) ] ≤ ξ 1 ).

Another popular formulation involves leveraging the state constraints so that safety corresponds to avoiding visits to a set of unsafe states. This type of formulation has been widely adopted by previous studies on safe-critical robotics tasks [38-40, 45], which is written as follows:

Problem 2 (Safe RL with state constraints) .

where ξ 2 ∈ R ≥ 0 is a threshold, I ( · ) is the indicator function, and S unsafe ⊂ S is a set of unsafe states.

Finally, some existing studies formulate safe RL problems via an instantaneous constraint, attempting to ensure safety even during the learning stage while aiming for extremely safety-critical applications such as planetary exploration [42] or healthcare [36]. Such studies typically require the agent to satisfy the following instantaneous safety constraint at every time step.

Problem 3 (Safe RL with instantaneous constraints) .

where ξ 3 ∈ R ≥ 0 is a time-invariant safety threshold.

3 Problem Formulation

This paper also requires an agent to optimize a policy under a safety constraint, as in the three common safe RL problems. We seek to find the optimal policy π ⋆ : S → A of the following problem, which will hereinafter be referred to as the 'generalized' safe exploration ( GSE ) problem:

Problem 4 ( GSE problem) . Let b h ∈ R denote a time-varying threshold.

This constraint is instantaneous, which requires the agent to learn a policy without a single constraint violation not only after convergence but also during training. We assume that the threshold is myopically known; that is, b h is known at time h , but unknown before that. Crucially, at every time step h , since s h is a fully observable state and the agent's policy is deterministic, we will use a simplified inequality represented as g ( s h , a h ) ≤ b h in the rest of this paper. This constraint is akin to that in Problem 3, with the difference that the safety threshold is time-varying.

Importance of the GSE problem. Though our problem may not seem relevant to Problems 1 and 2, we will shortly present and prove a theorem on the relationship between the GSE problem and the three common safe RL problems.

Theorem 3.1. Problems 1, 2, and 3 can be transformed into the GSE problem (i.e., Problem 4).

See Appendix B for the proof. In other words, the feasible policy space in the GSE problem can be identical to those in the other three problems by properly defining the safety cost function g and threshold b h . Crucially, Problem 1 is a special case of the GSE problem with b h = η h for all h , where η h +1 = γ -1 g · ( η h -g ( s h , a h )) with η 0 = ξ 1 . It is particularly beneficial to convert Problems 1 and 2, which have additive constraint structures, to the GSE problem, which has an instantaneous constraint. The accurate estimation of the cumulative safety value in Problems 1 and 2 is difficult because they depend on the trajectories induced by a policy. Dealing with the instantaneous constraint in the GSE problem is easier, both theoretically and empirically. Also, especially when the environment is time-varying (e.g., there are moving obstacles), the GSE problem is more useful than Problem 3.

Typical CMDP formulations with expected cumulative (safety) cost are out of the scope of the GSE problem. In such problems, the safety notion is milder; hence, although many advanced deep RL algorithms have been actively proposed that perform well in complex environments after convergence,

their performance in terms of safety during learning is usually low, as reported by Stooke et al. [33] or Wachi et al. [43]. Risk-constrained MDPs are also important safe RL problems that are not covered by the GSE problem; they have been widely studied by representing risk as a constraint on some conditional value-at-risk [11] or using chance constraints [24, 26]. 1

Difficulties and Assumptions. Theorem 3.1 insists that the GSE problem covers a wide range of safe RL formulations and is worth solving, but the problem is intractable without assumptions. We now discuss the difficulties in solving the GSE problem, and then list the assumptions in this paper.

The biggest difficulty with the GSE problem lies in the fact that there may be no viable safe action given the current state s h , safety cost g , and threshold b h . When b h = 0 . 1 and g ( s h , a ) = 0 . 5 , ∀ a ∈ A , the agent has no viable action for ensuring safety. The agent needs to guarantee safety, even during training, where little environmental information is available; hence, it is significant for the agent to avoid such situations where there is no action that guarantees safety. Another difficulty is related to the regularity of the safety cost function and the strictness of the safety constraint. In this paper, the safety cost function is unknown a priori.; thus, when the safety cost does not exhibit any regularity, the agent can neither infer the safety of decisions nor guarantee safety almost surely.

To address the first difficulty mentioned above, we use Assumptions 3.2 and 3.3.

Assumption 3.2 (Safety margin) . There exists ζ ∈ R > 0 such that Pr[ g ( s h , a h ) ≤ b h -ζ | P , π ⋆ ] = 1 , ∀ h ∈ [1 , H ] .

Assumption 3.3 (Emergency stop action) . Let ̂ a be an emergency stop action such that P ( s 1 | s, ̂ a ) = 1 for all s ∈ S . The agent is allowed to execute the emergency stop action and reset the environment if and only if the probability of guaranteed safety is not sufficiently high.

Assumption 3.2 is mild; this is similar to the Slater condition, which is widely adopted in the CMDP literature [13, 25]. We consider Assumption 3.3 is also natural for safety-critical applications because it is usually better to guarantee safety, even with human interventions, if the agent requires help in emergency cases. In some applications (e.g., the agent is in a hazardous environment), however, emergency stop actions should often be avoided because of the expensive cost of human intervention. In such cases, the agent needs to learn a reset policy allowing them to return to the initial state as in Eysenbach et al. [15], rather than asking for human help, which we will leave to future work.

As for the second difficulty, we assume that the safety cost function belongs to a class where uncertainty can be estimated and guarantee the satisfaction of the safety constraint with high probability. We present an assumption regarding an important notion called an uncertainty quantifier:

Assumption 3.4 ( δ -uncertainty quantifier) . Let µ : S×A → R denote the estimated mean function of safety. There exists a δ -uncertainty quantifier Γ : S ×A → R such that | g ( s, a ) -µ ( s, a ) | ≤ Γ( s, a ) for all ( s, a ) ∈ S × A , with a probability of at least 1 -δ .

4 Method

We propose MASE for the GSE problem, which combines an unconstrained RL algorithm with additional mechanisms for addressing the safety constraints. The pseudo-code is provided in Algorithm 1, and a conceptual illustration can be seen in Figure 1.

The most notable feature of MASE is that safety is guaranteed via the δ -uncertainty quantifier and the emergency stop action (lines 3 -9 ). The δ -uncertainty quantifier is particularly useful because we can guarantee that the confidence bound contains the true safety cost function, that is, g ( s, a ) ∈ [ µ ( s, a ) ± Γ( s, a ) ] for all s ∈ S and a ∈ A . This means that, if the agent chooses actions such that µ ( s h , a h ) + Γ( s h , a h ) ≤ b h , then g ( s h , a h ) ≤ b h holds with a probability of at least 1 -δ . Regarding the first difficulty mentioned in Section 3, it is crucial that there is at least one safe action. Thus, at every time step h , the agent computes a set of actions that are considered to satisfy the safety constraints with a probability at least 1 -δ given the state s h and threshold b h . This is represented as

Whenever the agent identifies that at least one action will guarantee safety, the agent is required to choose an action within A + h (line 3 ). The emergency stop action is executed if and only if there is no

Algorithm 1 Meta-Algorithm for Safe Exploration ( MASE )

̸

viable action satisfying the safety constraint (i.e., A + h +1 = ∅ ); that is, the agent is allowed to execute ̂ a and start a new episode from an initial safe state (lines 6 -9 ). The safety cost is upper-bounded by 1 because g ∈ [0 , 1] by definition. Note that MASE proactively avoids unsafe actions by selecting the emergency stop action to take beforehand ; this is in contrast to Sun et al. [37], whose method terminates the episode immediately after the agent has already violated a safety constraint.

When safety is guaranteed in the manner described above, the question remains as to how to obtain a policy that maximizes the expected cumulative reward. As such, we first convert the original CMDP M to the following unconstrained MDP

The changes from M lie in the action space and the reward function, as well as in the absence of the safety cost function. First, the action space is augmented so that the agent can execute the emergency stop action, ̂ a . The second modification concerns the reward function. When executing the emergency stop action ̂ a , the agent is penalized as its sacrifice so that the same situation will not occur in future episodes; hence, we modify the reward function as follows:

where c ∈ R > 0 is a positive scalar representing a penalty for performing the emergency stop. This penalty is assigned to the state-action pair ( s h , a h ) that placed the agent into the undesirable situation at time step h +1 , represented as A + h +1 = ∅ (see Figure 1).

To show that MASE is a reasonable safe RL algorithm, we express the following intuitions. Consider the ideal situation in which the safety cost function is accurately estimated for any state-action pairs; that is, Γ( s, a ) = 0 for all ( s, a ) . In this case, all emergency stop actions are properly executed, and the safety constraint will be violated at the next time step if the agent executes other actions. It is reasonable for the agent to receive a penalty of ̂ r ( s, a ) = -∞ because this state-action pair surely causes a safety violation without the emergency stop action. Unfortunately, however, the safety

s

h

s

h

+1

s

h

+1

s

h

+2

a

h

a

h

+1

×

×

̂

r

≪

0

̂

r

=

r

No viable action (i.e.,

A

+

h

+1

=

∅

)

a

h

̂

a

cost is uncertain and the agent conservatively executes ̂ a although there are still actions satisfying the safety constraint, especially in the early phase of training; hence, we increase or reduce the penalty according to the magnitude of uncertainty in (1) to avoid an excessively large penalty.

We must carefully consider the fact that the quality of information regarding the modified reward function ̂ r is uneven in the replay buffer D . Specifically, in the early phase of training, the δ -

uncertainty quantifier is loose. Hence, the emergency stop action is likely to be executed even if viable actions remain; that is, the agent will receive unnecessary penalties. In contrast, the emergency stop actions in later phases are executed with confidence, as reflected by the tight δ -uncertainty quantifier. Thus, as in line 15 , we rewrite the replay buffer D while updating Γ depending on the model in terms of the safety cost function (as for specific methods to update Γ , see Sections 5 and 6).

Connections to shielding methods. The notion of the emergency stop action is akin to shielding [2, 20] which has been actively studied in various problem settings including partially-observable environments [10] or multi-agent settings [23]. Thus, MASE can be regarded as a variant of shielding methods (especially, preemptive shielding in [2]) that is specialized for the GSE problem. On the other hand, MASE does not only block unsafe actions but also provides proper penalties for executing the emergency stop actions based on the uncertainty quantifier, which leads to rigorous theoretical guarantees presented shortly. Such theoretical advantages can be enjoyed in many safe RL problems because of the wide applicability of the GSE problem backed by Theorem 3.1.

Advantages of MASE . Though certain existing algorithms for Problem 3 (i.e., the closest problem to the GSE problem) theoretically guarantee safety during learning, several strong assumptions are needed, such as a known and deterministic state transition and regular safety function as in [41] and a known feature mapping function that is linear with respect to transition kernels, reward, and safety as in [5]. Such algorithms have little affinity with deep RL; thus, their actual performance in complex environments tends to be poor. In contrast, MASE is compatible with any advanced RL algorithms, which can also handle various constraint formulations while maintaining the safety guarantee.

Validity of MASE . We conclude this section by presenting the following two theorems to show that our MASE produces reasonable operations in solving the GSE problem.

Theorem 4.1. Under Assumption 3.4, MASE guarantees safety with a probability of at least 1 -δ .

Theorem 4.2. Assume that the safety cost function is estimated for any state-action pairs with an accuracy of better than ζ 2 ; that is, Γ( s, a ) ≤ ζ 2 for all ( s, a ) . Set c ∈ R to be a sufficiently large scalar such that c > ζV max 2 γ r H . Then, the optimal policy in ̂ M is identical to that in M .

See Appendix D for the proofs. Unfortunately, obtaining a δ -uncertainty qualifier that works in general cases is highly challenging. To develop a feasible model for the uncertainty quantification, we assume that the safety cost can be modeled via a GLM in Section 5 and via a GP in Section 6.

5 A Provable Algorithm under Generalized Linear CMDP Assumptions

In this section, we focus on CMDPs with generalized linear structures and analyze the theoretical properties of MASE . Specifically, we provide a provable algorithm to use a class of GLMs denoted as F for modeling Q ⋆ r,h := Q π ⋆ r,h and g , and then provide theoretical results on safety and optimality.

5.1 Generalized Linear CMDPs

We extend the assumption in Wang et al. [44] from unconstrained MDPs to CMDPs settings. Our assumption is based on GLMs as with [44] that makes a strictly weaker assumption than their Linear MDP assumption [19, 46]. As preliminaries, we first list the necessary definitions and assumptions.

Definition 5.1 (GLMs) . Let d ∈ Z > 0 be a feature dimension and let B d := { x ∈ R d : ∥ x ∥ 2 ≤ 1 } be the l 2 ball in R d . For a known feature mapping function ϕ : S × A → B d and a known link function f : [ -1 , 1] → [ -1 , 1] , the class of generalized linear model is denoted as F := { ( s, a ) → f ( ⟨ ϕ s,a , θ ⟩ ) : θ ∈ B d } where ϕ s,a := ϕ ( s, a ) .

Assumption 5.2 (Regular link function) . The link function f ( · ) is twice differentiable and is either monotonically increasing or decreasing. Furthermore, there exist absolute constants 0 < κ < κ < ∞ and M < ∞ such that κ < | f ′ ( x ) | < κ and | f ′′ ( x ) | < M for all | x | ≤ 1 .

This assumption on the regular link function is standard in previous studies (e.g., [16], [21]). Linear and logistic models are the special cases of the GLM where the link functions are defined as f ( x ) = x and f ( x ) = 1 / (1 + e -x ) . In both cases, the link functions satisfy Assumption 5.2.

We finally make the assumption of generalized linear CMDPs (GL-CMDPs), which extends the notion of the optimistic closure for unconstrained MDP settings in Wang et al. [44].

Assumption 5.3 (GL-CMDP) . For any 1 ≤ h < H and u ∈ F up, we have T h ( u ) ∈ F and g ∈ F .

Recall that T h is the Bellman update operator. In Assumption 5.3, with a positive semi-definite matrix A ∈ R d × d ≻ 0 and a fixed positive constant α max ∈ R > 0 , we define

√

where ∥ x ∥ A := x ⊤ A x is the matrix Mahalanobis seminorm, and ∥ A ∥ op is the matrix operator norm. For simplicity, we suppose the same link functions for the Q-function and the safety cost function, but it is acceptable to use different link functions. Note that Assumption 5.3 is a more general assumption than Amani et al. [5] that assumes linear transition kernel, reward, and safety cost functions or Wachi et al. [43] that assumes a known transition and GLMs in terms of reward and safety cost functions.

5.2 GLM-MASE Algorithm

We introduce an algorithm GLM-MASE under Assumptions 5.2 and 5.3. Hereinafter, we explicitly denote the episode for each variable. For example, we let s ( t ) h or a ( t ) h denote a state or action at the time step h of episode t . We also let ˜ ϕ ( t ) h := ϕ ( s ( t ) h , a ( t ) h ) for more concise notations.

Uncertainty quantifiers. To actualize MASE in the generalized linear CMDP settings, we first need to consider how to obtain the δ -uncertainty quantifier in terms of the safety cost function. Since we assume g ∈ F , we can define the δ -uncertainty quantifier based on the existing studies on GLMs, especially in the field of multi-armed bandit [22, 16]. Based on Assumptions 5.2 and 5.3, we now provide a lemma regarding the δ -uncertainty quantifier on safety.

Lemma 5.4. Suppose Assumptions 5.2 and 5.3 hold. Set δ = 1 . With a universal constant

TH √

where I is the identity matrix. Let ̂ θ g h,t ∈ R d be the ridge estimate, which is computed by ̂ θ g h,t := arg min ∥ θ ∥ 2 ≤ 1 ∑ τ ≤ t ( g ( s ( τ ) h , a ( τ ) h ) -f ( ⟨ ˜ ϕ ( τ ) h , θ ⟩ ) ) 2 . Then, the following inequality holds

for all ( s, a ) ∈ S × A , with a probability at least 1 -δ .

For the purpose of qualifying uncertainty in GLMs, the weighted l 2 -norm of ϕ (i.e., ∥ ϕ s,a ∥ Λ -1 h,t ) plays an important role. Because we assume that the Q-function and safety cost function share the same feature, we have a similar lemma on the uncertainty quantifier regarding the Q-function as follows:

Lemma 5.5. Suppose Assumptions 5.2 and 5.3 hold. Let ̂ θ Q h,t ∈ R d denote the ridge estimate; that is, ̂ θ Q h,t := arg min ∥ θ ∥ 2 ≤ 1 ∑ τ ≤ t ( y ( τ ) h -f ( ⟨ ˜ ϕ ( τ ) h , θ ⟩ ) ) 2 , where y ( τ ) h := r ( s ( τ ) h , a ( τ ) h ) + max a ′ ∈A ̂ Q ( τ ) r,h +1 ( s ( τ ) h +1 , a ′ ) for all τ ≤ t with

that is initialized with ̂ Q (0) r,h = 0 for all h ≤ H and ̂ Q ( t ) r,H +1 = 0 for all 1 ≤ t ≤ T . Then, with a universal constant C Q / g ∈ R > 0 , the following inequalities holds

for all ( s, a ) ∈ S × A , with a probability at least 1 -δ .

Note that Γ( s ( t ) h , a ( t ) h ) is the δ -uncertainty quantifier with respect to the safety cost function. One of the biggest advantages of the generalized linear CMDPs is that the magnitude of uncertainty for the Q-function is proportional to that for the safety cost function. Hence, by exploring the Q-function

based on the optimism in the face of the uncertainty principle [7, 35], the safety cost function is also explored simultaneously, which contributes to the efficient exploration of state-action spaces.

Integration into MASE . The GLM-MASE is an algorithm to integrate the δ -uncertainty quantifiers inferred by the GLM into the MASE sequence. Detailed pseudo code is presented in Appendix E.

To deal with the safety constraint, GLM-MASE leverages the upper bound inferred by the GLM; that is, for all h and t , the agent takes only actions that satisfy

By Lemma 5.4, such state-action pairs satisfy the safety constraint, i.e. g ( s ( t ) h , s ( t ) h ) ≤ b h , for all h and t , with a probability at least 1 -δ . If there is no action satisfying the safety constraint (i.e., A + h = ∅ ), the emergency stop action ̂ a is taken, and then the agent receives a penalty defined in (1).

As for policy optimization, we follow the optimism in the face of the uncertainty principle. Specifically, the policy π is optimized so that the upper-confidence bound of the Q-function characterized by ̂ r is maximized; that is, for any state s ∈ S , the policy is computed as follows:

Intuitively, this equation enables us to 1) solve the exploration and exploitation dilemma by incorporating the optimistic estimates of the Q-function and 2) make the agent avoid generating trajectories to violate the safety constraint via the modified reward function.

Theoretical results. We now provide two theorems regarding safety and near-optimality. For both theorems, see Appendix E for the proofs.

Theorem 5.6. Suppose the assumptions in Lemma 5.4 hold. Then, the GLM-MASE satisfies g ( s ( t ) h , a ( t ) h ) ≤ b h for all t ∈ [1 , T ] and h ∈ [1 , H ] , with a probability at least 1 -δ .

Theorem 5.7. Suppose the assumptions in Lemmas 5.4 and 5.5 hold. Let C 1 and C 2 be positive, universal constants. Also, with a sufficiently large T , let t ⋆ denote the smallest integer satisfying λ min (Σ) tH -C 1 √ tHd -C 2 √ tH ln δ -1 ≥ 2 C g · ζ -1 , where λ min (Σ) is the minimum eigenvalue of the second moment matrix Σ . Then, the policy π ( t ) obtained by GLM-MASE at episode t satisfies

with probability at least 1 -δ .

Theorem 5.6 shows that the GLM-MASE guarantees safety with high probability for every time step and episode, which is a variant of Theorem 4.1 under the generalized linear CMDP assumption and corresponding δ -uncertainty quantifier. Theorem 5.7 demonstrates the agent's ability to act near-optimally after a sufficiently large number of episodes. The proof is based on the following idea. After t ⋆ episodes, the safety cost function and the Q-function are estimated with an accuracy better than ζ 2 . Then, based on Theorem 4.2, the optimal policy in ̂ M is identical to that in M ; thus, the agent achieves a near-optimal policy by leveraging the (well-estimated) optimistic Q-function.

6 A Practical Algorithm

Though we established an algorithm backed by theory under the generalized linear CMDP assumption in Section 5, it is often challenging to obtain proper feature mapping functions in complicated environments. Thus, in this section, we propose a more practical algorithm combining a GP-based estimator to guarantee safety with unconstrained deep RL algorithms to maximize the reward.

Guaranteeing safety via GPs. As shown in the previous sections, the δ -uncertainty quantifier plays a critical role in MASE . To qualify the uncertainty in terms of the safety cost function g , we consider modeling it as a GP: g ( z ) ∼ GP ( µ ( z ) , k ( z , z ′ )) , where z := [ s, a ] , µ ( z ) is a mean function, and k ( z , z ′ ) is a covariance function. The posterior distribution over g ( · , · ) is computed based on n ∈ Z > 0 observations at state-action pairs ( z 1 , z 2 , . . . , z n ) with safety measurements y n := { y 1 , y 2 , . . . , y n } , where y n := g ( z n ) + N n and N n ∼ N (0 , ω 2 ) is zero-mean Gaussian noise

with a standard deviation of ω ∈ R ≥ 0 . We consider episodic RL problems, and so n ≈ tH + h for episode t and time step h , although the equality does not hold because of the episode cutoffs. Using the past measurements, the posterior mean, variance, and covariance are computed analytically as µ n ( z ) = k ⊤ n ( z )( K n + ω 2 I ) -1 y n , σ n ( z ) = k n ( z , z ) , and k n ( z , z ′ ) = k ( z , z ′ ) -k ⊤ n ( z )( K n + ω 2 I ) -1 k n ( z ′ ) , where k n ( z ) = [ k ( z 1 , z ) , . . . , k ( z n , z )] ⊤ and K n is the positive definite kernel matrix. We now present a theorem on the safety guarantee.

Theorem 6.1. Assume ∥ g ∥ 2 k ≤ B and N n ≤ ω for all n ≥ 1 . Set β 1 / 2 n := B +4 ω √ ν n +1+ln(1 /δ ) and construct the δ -uncertainty quantifier by

where ν n is the information capacity associated with kernel k . Then, MASE based on (2) satisfies the safety constraint g ( s ( t ) h , a ( t ) h ) ≤ b h for all t and h with a probability of at least 1 -δ .

See Appendix F for the proofs. Theorem 6.1 guarantees that the safety constraint is satisfied by combining the GP-based δ -uncertainty quantifier in (2) and the emergency stop action.

Maximizing reward via deep RL. The remaining task is to optimize the policy via the modified reward function ̂ r in (1), whereby the agent is penalized for emergency stop actions. This problem is decoupled from the safety constraint and can be solved as the following unconstrained RL problem:

There are many excellent algorithms for solving (3) such as trust region policy optimization (TRPO, [31]) and twin delayed deep deterministic policy gradient (TD3, [17]). One of the key benefits of our MASE is such compatibility with a broad range of unconstrained (deep) RL algorithms.

7 Experiments

We conduct two experiments. The first is on Safety Gym [28], where an agent must maximize the expected cumulative reward under a safety constraint with additive structures as in Problems 1 and 2. The safety cost function g is binary (i.e., 1 for an unsafe state-action pair and 0 otherwise), and the safety threshold is set to ξ 1 = 20 . The reason for choosing Safety Gym is that this benchmark is complex and elaborate, and has been used to evaluate a variety of excellent algorithms. The second is a grid world where a safety constraint is instantaneous as in Problem 3. Due to the page limit, we present the settings and results of the grid-world experiment in Appendix H.

To solve the Safety Gym tasks, we implement the practical algorithm presented in Section 6 as follows. First, we convert the problem into a GSE problem by defining b h := γ -1 g · ( ξ 1 -∑ h -1 h ′ =0 γ h ′ g g ( s h ′ , a h ′ )) and enforcing the safety constraint represented as g ( s h , a h ) ≤ b h for every time step h in each episode. Second, to infer the safety cost, we use deep GP to conduct training and inference, as in [30] when dealing with high-dimensional input spaces. Finally, as for the policy optimization in ̂ M , we leverage the TRPO algorithm. We delegate other details to Appendix G.

Baselines and metrics. We use the following four algorithms as baselines. The first is TRPO, which is a safety-agnostic deep RL algorithm that purely optimizes a policy without safety consideration. The second and third are CPO [1] and TRPO-Lagrangian [28], which are well-known algorithms for solving CMDPs. The final algorithm is Sauté RL [32], which is a recent, state-of-the-art algorithm for solving safe RL problems where constraints must be satisfied almost surely. We employ the following three metrics to evaluate our MASE and the aforementioned four baselines: 1) the expected cumulative reward, 2) the expected cumulative safety, and 3) the maximum cumulative safety. We execute each algorithm with five random seeds and compute the means and confidence intervals.

Results. The experimental results are summarized in Figure 2. The figures show that TRPO, TRPO-Lagrangian, and CPO successfully learn the policies, but violate the safety constraints during training and even after convergence. Sauté RL is much safer than those three algorithms, but the safety constraint is not satisfied in some episodes, and the performance of the policy significantly deteriorates in terms of the cumulative reward during training. Our MASE obtains better policies in a smaller number of samples compared with Sauté RL, while also satisfying the safety constraints with respect to both the average and the worst-case. Note that, after convergence, the policy obtained by MASE performs worse than those obtained by the baseline algorithms in terms of reward, as shown in

0

100

200

300

400

500

Epoch

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

TRPO

TRPO Lag

CPO

Saute TRPO

MASE

(a) Average episode return.

0

100

200

300

400

500

Epoch

0

10

20

30

40

50

TRPO

TRPO Lag

CPO

Saute TRPO

MASE

Safety threshold

(b) Average episode safety.

0

100

200

300

400

500

Epoch

0

20

40

60

80

100

120

140

TRPO

TRPO Lag

CPO

Saute TRPO

MASE

Safety threshold

(c) Maximum episode safety.

0

100

200

300

400

500

Epoch

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

TRPO

TRPO Lag

CPO

Saute TRPO

MASE

(d) Average episode return.

0

100

200

300

400

500

Epoch

0

5

10

15

20

25

30

35

40

TRPO

TRPO Lag

CPO

Saute TRPO

MASE

Safety threshold

(e) Average episode safety.

0

100

200

300

400

500

Epoch

0

20

40

60

80

100

120

140

160

TRPO

TRPO Lag

CPO

Saute TRPO

MASE

Safety threshold

(f) Maximum episode safety.

TRPO

TRPO Lag

CPO

Saute TRPO

MASE

2.70

2.75

2.80

2.85

2.90

2.95

3.00

(a) Average episode return.

TRPO

TRPO Lag

CPO

Saute TRPO

MASE

5

10

15

20

25

Safety threshold

(b) Average episode safety.

TRPO

TRPO Lag

CPO

Saute TRPO

MASE

20

40

60

80

100

Safety threshold

(c) Maximum episode safety.

8 Conclusion

In this article, we first introduced the GSE problem and proved that it is more general than three common safe RL problems. We then proposed MASE to optimize a policy under safety constraints that allow the agent to execute an emergency stop action at the sacrifice of a penalty based on the δ -uncertainty qualifier. As a specific instance of MASE , we first presented GLM-MASE to theoretically guarantee the near-optimality and safety of the acquired policy under generalized linear CMDP assumptions. Finally, we provided a practical MASE and empirically evaluated its performance in comparison with several baselines on the Safety Gym and grid-world.

Acknowledgments and Disclosure of Funding

We would like to thank the anonymous reviewers for their helpful comments. This work is partially supported by JST CREST JPMJCR201 and by JSPS KAKENHI Grant 21K14184.

References

Achiam, J., Held, D., Tamar, A., and Abbeel, P. (2017). Constrained policy optimization. In International Conference on Machine Learning (ICML) .

Alshiekh, M., Bloem, R., Ehlers, R., Könighofer, B., Niekum, S., and Topcu, U. (2018). Safe reinforcement learning via shielding. In AAAI Conference on Artificial Intelligence (AAAI) .

Altman, E. (1999). Constrained Markov decision processes , volume 7. CRC Press.

Amani, S., Alizadeh, M., and Thrampoulidis, C. (2019). Linear stochastic bandits under safety constraints. In Neural Information Processing Systems (NeurIPS) .

Amani, S., Thrampoulidis, C., and Yang, L. (2021). Safe reinforcement learning with linear function approximation. In International Conference on Machine Learning (ICML) .

Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 .

Auer, P. and Ortner, R. (2007). Logarithmic online regret bounds for undiscounted reinforcement learning. In Neural Information Processing Systems (NeurIPS) .

Bajracharya, M., Maimone, M. W., and Helmick, D. (2008). Autonomy for mars rovers: Past, present, and future. Computer , 41(12):44-50.

Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S., Panerati, J., and Schoellig, A. P. (2022). Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems , 5:411-444.

Carr, S., Jansen, N., Junges, S., and Topcu, U. (2023). Safe reinforcement learning via shielding under partial observability. In AAAI Conference on Artificial Intelligence .

Chow, Y., Ghavamzadeh, M., Janson, L., and Pavone, M. (2017). Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research (JMLR) , 18(1):60706120.

Chowdhury, S. R. and Gopalan, A. (2017). On kernelized multi-armed bandits. In International Conference on Machine Learning (ICML) .

Ding, D., Zhang, K., Basar, T., and Jovanovic, M. (2020). Natural policy gradient primal-dual method for constrained Markov decision processes. In Neural Information Processing Systems (NeurIPS) .

Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., and Hester, T. (2021). Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning .

Eysenbach, B., Gu, S., Ibarz, J., and Levine, S. (2018). Leave no trace: Learning to reset for safe and autonomous reinforcement learning. In International Conference on Learning Representations (ICLR) .

Filippi, S., Cappe, O., Garivier, A., and Szepesvári, C. (2010). Parametric bandits: The generalized linear case. In Neural Information Processing Systems (NeurIPS) .

Fujimoto, S., van Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICML) .

Garcıa, J. and Fernández, F. (2015). A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research (JMLR) , 16(1):1437-1480.

Jin, C., Yang, Z., Wang, Z., and Jordan, M. I. (2020). Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory (COLT) .

Könighofer, B., Bloem, R., Junges, S., Jansen, N., and Serban, A. (2020). Safe reinforcement learning using probabilistic shields. In International Conference on Concurrency Theory: CONCUR .

Li, L., Chu, W., Langford, J., and Schapire, R. E. (2010). A contextual-bandit approach to personalized news article recommendation. In International Conference on World Wide Web (WWW) .

Li, L., Lu, Y., and Zhou, D. (2017). Provably optimal algorithms for generalized linear contextual bandits. In International Conference on Machine Learning (ICML) , pages 2071-2080.

Melcer, D., Amato, C., and Tripakis, S. (2022). Shield decentralization for safe multi-agent reinforcement learning. In Neural Information Processing Systems (NeurIPS) .

Ono, M., Pavone, M., Kuwata, Y., and Balaram, J. (2015). Chance-constrained dynamic programming with application to risk-aware robotic space exploration. Autonomous Robots , 39(4):555-571.

Paternain, S., Chamon, L., Calvo-Fullana, M., and Ribeiro, A. (2019). Constrained reinforcement learning has zero duality gap. Neural Information Processing Systems (NeurIPS) .

Pfrommer, S., Gautam, T., Zhou, A., and Sojoudi, S. (2022). Safe reinforcement learning with chance-constrained model predictive control. In Learning for Dynamics and Control Conference (L4DC) , pages 291-303.

Rasmussen, C. E. (2003). Gaussian processes in machine learning. In Summer school on machine learning , pages 63-71. Springer.

Ray, A., Achiam, J., and Amodei, D. (2019). Benchmarking safe exploration in deep reinforcement learning . OpenAI.

Rockafellar, R. T., Uryasev, S., et al. (2000). Optimization of conditional value-at-risk. Journal of risk , 2:21-42.

Salimbeni, H. and Deisenroth, M. (2017). Doubly stochastic variational inference for deep Gaussian processes. In Neural Information Processing Systems (NeurIPS) .

Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015). Trust region policy optimization. In International Conference on Machine Learning (ICML) .

Sootla, A., Cowen-Rivers, A. I., Jafferjee, T., Wang, Z., Mguni, D. H., Wang, J., and Ammar, H. (2022). Sauté RL: Almost surely safe reinforcement learning using state augmentation. In International Conference on Machine Learning (ICML) .

Stooke, A., Achiam, J., and Abbeel, P. (2020). Responsive safety in reinforcement learning by pid Lagrangian methods. In International Conference on Machine Learning (ICML) , pages 9133-9143.

Strehl, A. L. and Littman, M. L. (2005). A theoretical analysis of model-based interval estimation. In International Conference on Machine Learning (ICML) .

Strehl, A. L. and Littman, M. L. (2008). An analysis of model-based interval estimation for Markov decision processes. Journal of Computer and System Sciences , 74(8):1309-1331.

Sui, Y., Gotovos, A., Burdick, J. W., and Krause, A. (2015). Safe exploration for optimization with Gaussian processes. In International Conference on Machine Learning (ICML) .

Sun, H., Xu, Z., Fang, M., Peng, Z., Guo, J., Dai, B., and Zhou, B. (2021). Safe exploration by solving early terminated MDP. arXiv preprint arXiv:2107.04200 .

Thananjeyan, B., Balakrishna, A., Nair, S., Luo, M., Srinivasan, K., Hwang, M., Gonzalez, J. E., Ibarz, J., Finn, C., and Goldberg, K. (2021). Recovery RL: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters , 6(3):4915-4922.

Thomas, G., Luo, Y., and Ma, T. (2021). Safe reinforcement learning by imagining the near future. In Neural Information Processing Systems (NeurIPS) .

Turchetta, M., Kolobov, A., Shah, S., Krause, A., and Agarwal, A. (2020). Safe reinforcement learning via curriculum induction. In Neural Information Processing Systems (NeurIPS) , volume 33, pages 12151-12162.

Wachi, A. and Sui, Y. (2020). Safe reinforcement learning in constrained Markov decision processes. In International Conference on Machine Learning (ICML) .

Wachi, A., Sui, Y., Yue, Y ., and Ono, M. (2018). Safe exploration and optimization of constrained MDPs using Gaussian processes. In AAAI Conference on Artificial Intelligence (AAAI) .

Wachi, A., Wei, Y., and Sui, Y. (2021). Safe policy optimization with local generalized linear function approximations. In Neural Information Processing Systems (NeurIPS) .

Wang, Y., Wang, R., Du, S. S., and Krishnamurthy, A. (2020). Optimism in reinforcement learning with generalized linear function approximation. In International Conference on Learning Representations (ICLR) .

Wang, Y., Zhan, S. S., Jiao, R., Wang, Z., Jin, W., Yang, Z., Wang, Z., Huang, C., and Zhu, Q. (2023). Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. In International Conference on Machine Learning , pages 36593-36604. PMLR.

Yang, L. and Wang, M. (2019). Sample-optimal parametric Q-learning using linearly additive features. In International Conference on Machine Learning (ICML) .

By (5), we have

In summary, by introducing the new variable η h , Problem 1 is rewritten to the following problem:

The aforementioned problem (8) is a special case of the GSE problem with b h := η h . Therefore, we obtain the desired lemma.

Appendices

A Limitations and Potential Negative Societal Impacts

We will first discuss limitations and potential negative societal impacts regarding our work.

A.1 Limitations

One of the major limitations of this study is that emergency stop actions are allowed for agents. Emergency stop actions should often be avoided because of the expensive cost of human intervention in many applications (e.g., the agent is in a hazardous or remote environment). In future work, we will investigate an algorithm that requires the agent to learn a reset policy allowing them to return to the initial state as in [15], rather than asking for human intervention via emergency stop actions.

Another limitation is how to construct the uncertainty quantifier. In our experiment, because we used a computationally inexpensive deep GP algorithm [30] and the uncertainty quantifier is updated at the end of the episode (see Line 13 in Algorithm 1), the computational time of the GP part was much smaller than the RL part in our experiment settings. However, since GP is generally a computationally expensive algorithm, GP can be a computational bottleneck in some cases.

A.2 Potential Negative Societal Impacts

We believe that safety is an essential requirement for applying RL in many real problems. While we have not found any potential negative societal impact of our proposed method, we must remain aware that RL algorithms are vulnerable to misuse and ours is no exception.

B Proof of Theorem 3.1

We first present lemmas regarding the relationship between the GSE problem and Problems 1, 2, and 3. After that, we present the proof for the Theorem 3.1 in Appendix B.4 by combining them.

B.1 Relationship between the GSE problem and Problem 1

Lemma B.1. Problem 1 can be transformed into the GSE problem.

Proof. We first utilize a safety state augmentation technique presented in Sootla et al. [32] by defining a new variable η h such that

This new variable η h means the remaining safety budget associated with the discount factor γ g , which is updated as follows:

By (4), the necessary and sufficient condition for satisfying the constraint in Problem 1 is

B.2 Relationship between the GSE problem and Problem 2

Lemma B.2. Problem 2 can be transformed into the GSE problem.

Proof. The following chain of inequalities holds:

Because we assume Markov property, we simply define the safety cost function g : S × A → R as

We now set

then the Problem 2 can be transformed into

Finally, we obtained the desired lemma.

B.3 Relationship between the GSE problem and Problem 3

Lemma B.3. Problem 3 can be transformed into the GSE problem.

Proof. Set b h = ξ 3 for all h . Then, the GSE problem is identical to Problem 3.

B.4 Summary

Proof. Combining Lemma B.1, B.2, and B.3, we obtain the desired Theorem 3.1.

C Connections to Safe RL Problems with Chance Constraints

As a strongly related formulation to Problem 2, policy optimization under joint chance-constraints has been studied especially in the field of control theory such as Ono et al. [24] and Pfrommer et al. [26], which is written as

Problem 5 (Safe RL with joint chance constraints) . Let ξ 5 ∈ R ≥ 0 be a constant representing a safety threshold. Also, let S unsafe ⊂ S denote a set of unsafe states. Find the optimal policy π ⋆ such that

Lemma C.1. Problem 2 is a conservative approximation of Problem 5.

Proof. This lemma mostly follows from Theorem 1 in Ono et al. [24]. Regarding the constraint in Problem 5, we have the following chain of equations:

In the first step, we used Boole's inequality (i.e., Pr[ A ∪ B ] ≤ Pr[ A ] + Pr[ B ] ). The final term in (10) is the LHS of the constraint in Problem 2, which implies that Problem 2 is a conservative approximation of Problem 5. Therefore, the GSE problem is also a conservative approximation of Problem 5.

Corollary C.2. Suppose we solve the GSE problem by properly defining the safety function g ( · , · ) and the threshold b h . Then, the obtained policy is a conservative solution of Problem 5.

Remark C.3. It is extremely challenging to directly solve the Problem 5 characterized by joint chance-constraints without approximation, as discussed in Ono et al. [24]. Practically, it would be a promising and realistic approach to solve Problem 5 by converting it into the GSE problem.

More detailed explanations for the aforementioned remark are as follows. It is extremely challenging to directly solve the Problem 5 characterized by joint chance-constraints. Most of the previous work does not directly deal with this type of constraint and uses some approximations or assumptions. For example, Pfrommer et al. [26] assume a known linear time-invariant dynamics. Also, Ono et al. [24] approximate the joint chance constraint as in the above procedure and obtain

This is a conservative approximation with an additive structure, which is easier to solve than the original joint chance constraint. Ono et al. [24] deals with the above constraints with additive safety structure. By additionally transforming the conservatively-approximated problem into the GSE problem, the problem would become easier to handle because the safety constraint is instantaneous.

D Proof of Theorems 4.1 and 4.2

D.1 Proof of Theorem 4.1

Proof. Recall that, at every time step h , the MASE chooses actions satisfying

By definition of the δ -uncertainty quantifier, the actions that are conservatively chosen based on (11) also satisfy the safety constraint, with a probability of at least 1 -δ ; that is,

In addition, when there is no action satisfying (11), the emergency stop action is executed; that is, safety is guaranteed with a probability of 1 . Hence, the desired theorem is now obtained.

D.2 Proof of Theorem 4.2

Proof. Assumption 3.2 implies that there exists a policy that satisfies a more conservative safety constraint written as

By combining (13) and the assumption Γ( s, a ) ≤ 1 2 ζ, ∀ ( s, a ) ∈ S × A , we have

Algorithm 2 GLM-MASE

which guarantees that there exists a policy that conservatively satisfies the safety constraint via the δ -uncertainty quantifier Γ( · , · ) at every time step h , with a probability of at least 1 -δ .

When we set c to be a sufficiently large scalar such that c > ζ 2 γ r H V max , the penalty ̂ r satisfies

This means that, when the constraint violation happens even a single time, the value by a policy obtained in ̂ M becomes negative because max s V π r ( s ) ≤ V max .

Under Assumption 3.2, after convergence, the optimal policy in ̂ M will not violate the safety constraint, and thus the emergency stop action ̂ a will not be executed. In this case, the modified (unconstrained) MDP ̂ M is identical to the original CMDP M . Therefore, we now obtain the desired theorem.

E Supplementary materials regarding GLM-MASE

E.1 Pseudo-code for GLM-MASE

We first present the pseudo-code for GLM-MASE in Algorithm 2.

E.2 Proofs of Lemmas 5.4 and 5.5

Proof. See Lemma 1 (and Lemma 7) in Wang et al. [44].

E.3 Preliminary Lemmas

Lemma E.1. Suppose the assumptions in Lemma 5.4 and 5.5 hold. Let C 1 and C 2 be positive, universal constants. Also, with a sufficiently large T , let t ⋆ denote the smallest integer satisfying

where λ min (Σ) is the minimum eigenvalue of the second moment matrix Σ . Then, we have

Proof. By Proposition 1 of Li et al. [22],

By combining the aforementioned inequality with (15), we have

Using the definition of λ max (Λ -1 h,t ) = 1 λ min (Λ h,t ) , the following chain of equations hold for all t ∈ [ t ⋆ , T ] and h ∈ [1 , H ] :

Therefore, we have the desired lemma.

E.4 Proof of Theorem 5.6

Proof. By definition, the GLM-MASE chooses actions satisfying

By Lemma 5.4, the actions that are conservatively chosen based on (17) also satisfy the actual safety constraint, with a probability at least 1 -δ ; that is,

In addition, when there is no action satisfying (17), the emergency stop action is executed where no unsafe action will be executed. Hence, the desired theorem is now obtained.

E.5 Proof of Theorem 5.7

By Assumption 3.2, the optimal policy π ⋆ satisfies

Thus, the set of state-action pairs that are potentially visited by π ⋆ are written as

which satisfies the following chain of inequalities:

The state and action spaces in the last line represent the set of state-action pairs that may be visited by the policy obtained by the MASE algorithm. We used Lemma 5.4 in the first line and Γ( s, a ) < ζ 2 in the second line.

By Lemma 8 in Strehl and Littman [34], the total regret can be decomposed into two parts as follows:

where the first term is the regret under the condition that the confidence bound based on δ -uncertainty quantifier successfully contains the true safety function. Also, the second term is the regret under the opposite condition, which occurs with a probability δ .

The first term in (20) is upper-bounded based on Wang et al. [44] as follows:

As for the second term in (20), set δ = 1 TH and then we have

In summary, the regret can be upper bounded by ˜ O ( H √ d 3 ( T -t ⋆ )) . We now obtain the desired theorem.

F Proof of Theorem 6.1

Lemma F.1 ( δ -uncertainty quantifier) . Assume ∥ g ∥ 2 k ≤ B and N n ≤ ω for all n ≥ 1 . Set

with β 1 / 2 n := B +4 ω √ ν n +1+ln δ -1 , where ν n is the information capacity associated with kernel k . Then, Γ is a δ -uncertainty quantifier.

Proof. Recall the assumption that ∥ g ∥ 2 k ≤ B and N n ≤ ω , ∀ n ≥ 1 . Also, set

By Theorem 2 in Chowdhury and Gopalan [12], we have

for all n ≥ 1 , with probability at least 1 -δ . Now, define

then we interpret that Γ : S × A → R is a δ -uncertainty quantifier based on GP.

F.1 Proof of Theorem 6.1

Proof. Based on Lemma F.1, when there is at least one safe action (i.e., A + h = ∅ ), the satisfaction of the safety constraint is guaranteed based on the δ -uncertainty quantifier with a probability at least 1 -δ . Also, if there is no safe action (i.e., A + h = ∅ ), the emergency stop action is executed. In both cases, MASE guarantees the satisfaction of the safety constraint with probability at least 1 -δ .

̸

G Details of Safety-Gym Experiment

We present the details regarding our experiments using Safety-Gym. Our experimental setting is based on Sootla et al. [32], which is slightly different from the original Safety Gym in that the obstacles (i.e., unsafe region) are replaced deterministically so that the environment is solvable and there is a viable solution. In this experiment, we used a machine with Intel(R) Xeon(R) Silver 4210 CPU, 128GB RAM, and NVIDIA A100 GPU. For a fair comparison, we basically used the same hyper-parameter as in Sootla et al. [32], which is summarized in Table 1.

In our experiment, when the agent identified that there was no safe action based on the GP-based uncertainty quantifier, we simply terminated the current episode (i.e., resetting) immediately after the emergency stop action and started the new episode. The frequency of the emergency stop actions is shown in Table 2.

The emergency stop action is a variant of so-called resetting actions that are common in episodic RL settings, which prevent the agent from exploring the state-action spaces since the uncertainty quantifier is sometimes quite conservative. We consider that this is the reason why the reward performance of our MASE is worse than other methods (e.g., TRPO-Lagrangian, CPO). However, because we require the agent to solve more difficult problems where safety is guaranteed at every time step and episode, we consider that this result is reasonable to some extent. Though it is better for an algorithm for such a severe safety constraint to have a comparable performance as CPO, we will leave it for future work.

We also conducted an experiment to compare the performance of MASE with the early-terminated MDP (ET-MDP, [37]) algorithm. The ET-MDP is an algorithm to execute emergency stop actions immediately after safety constraints are violated. Figure 4 shows the experimental results. The ET-MDP and MASE exhibit similar learning curves on the average episode reward and average episode safety. However, while ET-MDP violated the safety constraint in most episodes (i.e., almost

0

100

200

300

400

500

Epoch

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

TRPO

TRPO Lag

CPO

ET MDP

Saute TRPO

MASE

(a) Average episode return.

0

100

200

300

400

500

Epoch

0

10

20

30

40

50

TRPO

TRPO Lag

CPO

ET MDP

Saute TRPO

MASE

Safety threshold

(b) Average episode safety.

0

100

200

300

400

500

Epoch

0

20

40

60

80

100

120

140

TRPO

TRPO Lag

CPO

ET MDP

Saute TRPO

MASE

Safety threshold

(c) Maximum episode safety.

0

100

200

300

400

500

Epoch

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

TRPO

TRPO Lag

CPO

ET MDP

Saute TRPO

MASE

(d) Average episode return.

0

100

200

300

400

500

Epoch

0

5

10

15

20

25

30

35

40

TRPO

TRPO Lag

CPO

ET MDP

Saute TRPO

MASE

Safety threshold

(e) Average episode safety.

0

100

200

300

400

500

Epoch

0

20

40

60

80

100

120

140

160

TRPO

TRPO Lag

CPO

ET MDP

Saute TRPO

MASE

Safety threshold

(f) Maximum episode safety.

all episodes are terminated after an unsafe action is executed), MASE did not violate any safety constraint.

H Grid-world Experiment

We also conduct an experiment using the grid-world problem as in Wachi and Sui [41]. Experimental settings are based on their original implementation ( https://github.com/akifumi-wachi-4/ safe_near_optimal_mdp ). We consider a 20 × 20 square grid in which reward and safety functions are randomly generated. There are two types regarding the safety threshold: one is time-invariant as in [41] and the other is time-variant as in the GSE problem.

We run SNO-MDP [41] and MASE in 100 randomly generated environments, and we compute the reward collected by the algorithms and count the number of episodes in which the safety constraint is violated at least once. The reward is normalized with respect to that by SNO-MDP.

The experimental results are shown in Table 3. When the safety threshold is time-invariant, MASE behaves identically with SNO-MDP; thus, the performance of the MASE is comparable with that of SNO-MDP. When the safety threshold is time-variant, SNO-MDP cannot deal with it by nature; hence, the safety constraint is not satisfied in most of the episodes. In contrast, our MASE satisfies the safety constraint in every episode, which also contributes to the larger reward.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the safe RL problem with a generalized stepwise safety chance (probability) constraint, essential for many safety-critical systems with RL. The authors propose a meta-algorithm to solve this problem (MASE), by combining unconstrained RL with an uncertainty quantifier to guarantee safety with probability. They study two variants for MASE, one for the linear model and the other for GP. Experiments on grid-world and safety gym show that MASE with GSE formulation achieves SOTA compared to the baselines.\n\nSTRENGTHS:\nThe paper is well-written and easy to follow. The proposed GSE problem is more general than the CMDP formulation with an additive expectation constraint. Especially, the GSE problem is an important problem to study for many real-world safety-critical systems, such as autonomous cars and cars. \nThe proposed method looks sound and correct to me. \nThe MASE algorithm essentially builds on Assumption 3.4 (uncertainty quantifier) which could also be a potential bottleneck or limitation, but the author did a good job by proposing a general linear model and GP for it. \nThe experimental results show better results in terms of safety violations, compared to other CMDP-based approaches.\n\nWEAKNESSES:\n1. The author may want to at least discuss this ICML paper in their related work. The hard safety chance constraint is highly relevant to the GSE problem in this paper, although it is indeed using different approaches to solve the problem. \nWang, Y., Zhan, S. S., Jiao, R., Wang, Z., Jin, W., Yang, Z., ... & Zhu, Q. (2022). Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments. arXiv preprint arXiv:2209.15090.\n\n2. The authors should discuss the limitations of this work\n\n3. In Method, MASE needs to compute a safe action set, how complex this computation is? In order to compute it, what kind of assumptions of the underlying environments do you acquire?\n\nQUESTIONS:\n1. Page 3 Lines 96 to 98, although I can understand what the authors mean, we should always keep in mind that s_h (a_h) are random variables, you cannot let a random variable be less than a value. It has to be in Pr() format, even if the probability is 1.\n\nLIMITATIONS:\nI would like to see the authors' opinions on the limitations in their responses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a novel safe RL algorithm for safety constraints with probability one. First, the authors present a problem formulation that can be used to derive different common safe RL formulation (state constraints, accumulated constraints etc). Then the authors leverage a sophisticated technique to reshape the reward accounting for safety information. In particular, they predict if the following actions will be safe by learning the action safety state similar to safety shield techniques. Their shields are learning using Gaussian processes and offer safety predictions with uncertainty estimates.`\n\nSTRENGTHS:\n1. The algorithm is technically sound, novel and quite interesting as an idea.\n1. The authors present an extensive analysis for the case of generalized linear CMPDs. The theoretical results seem to be correct, but I have a few minor concerns regarding the statements \n2. The numerical results are impressive but shown only for one environment.\n\nWEAKNESSES:\n1. The statement of Theorem 3.1 reads there are instances of the GSE problem that are not equivalent and cannot be transformed into Problem 1, 2 or 3. This is not shown. \n1. The fact that Problem 2 can be transformed into the GSE problem does not mean the GSE problem is more general than Problem 2. If the GSE problem can be transformed to Problem 2 as well, then they are equivalent. Furthermore, if the accumulated cost is used as a state as in Lemma A.1, then the problems cannot truly be equivalent. I think the authors should rephrase these results and make more accurate claims. \n1. The method does remind me of safety layer techniques [Dalal 18] and [28] with a more sophisticated reward-shaping approach. Can the authors provide a short discussion on the relation to these papers?\n1. The numerical results can be improved \n     * I recommend adding boxplots to the simulation results to see the distributions of the traces. For instance the boxplots of the trajectories for the final epoch as in [28]\n     * I recommend providing more epochs in the experiments. I suspect that the algorithms generally achieve similar performance, but the algorithms with probability one constraint simply converge slower.\n\n* [Dalal 18] Dalal, G., Dvijotham, K., Vecerik, M., Hester, T., Paduraru, C., & Tassa, Y. (2018). Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757.\n* [Yu 22] Yu, Haonan, Wei Xu, and Haichao Zhang. \"Towards safe reinforcement learning with a safety editor policy.\" Advances in Neural Information Processing Systems 35 (2022): 2608-2621.\n\nQUESTIONS:\n1. Please provide details on the safety gym environment. This does not seem like a standard one. \n1. How the algorithm scales with a number of epochs/states? My concern is that the GP is not the most scalable model for RL\n\nLIMITATIONS:\nThere are two limitations of the approach: 1) the scalability of the algorithm and 2) the restrictiveness of the problem definition. While the authors discuss the latter limitation and show that this problem formulation is important, I didn’t find a discussion on scalability.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper considers the generalized safe exploration problem (problem 4) and compares it to the other safe exploration problems in the literature, leading to Thm 3.1 that concludes that it is more general than the others. The authors then introduce MASE, a meta-algorithm for safe exploration, that attempts to solve the GSE problem with the ability to execute an emergency stop \"beforehand\" as opposed to others that have done it afterwards. Section 6 then presents a more practical algorithm using GP models, which is then used in section 7 to compare the new approach with several baselines.\n\nSTRENGTHS:\nThe paper is well presented, with several technical results about the safety analysis, culminating in Thms 5.6 and 5.7 which show that GLM-MASE guarantees safety with high probability for every time step.\n\nSeveral numerical experiments are presented to compare with unconstrained and SOA constrained RL algorithms.\n\nWEAKNESSES:\nThe development in the paper assumes access to an emergency stop action that enables the agent to avoid violating a safety constraint. This assumptions seems difficult to achieve in practice as for many agents of interest, stopping is not a safe state. Further, the action would typically be state dependent, requiring at a minimum a emergency action policy (rather than action). Finding either action or policy seems non trivial and could possibly be overly simplifying most problems of interest.\n\nGiven the comment about [33] at the bottom of page 4, I would have expected to see a comparison of these approaches to imposing an action to avoid an unsafe state before/after a training epoch in the numerical results. If that is included in section 7, suggest highlight that discussion point more.\n\nThe footnote on page 3 makes reference to this being a conservative approximation of that in safe RL problems with chance constraints, and consigns the discussion Appendix B. It would seem like more clarification than that is needed here. Also, the more recent work by Pavone in this area will be of interest:\n* Lucas Janson, Edward Schmerling, and Marco Pavone, “Monte Carlo motion planning for robot trajectory optimization under uncertainty.” In Robotics Research, pages 343–361. Springer, 2018.\n* Anirudha Majumdar and Marco Pavone “How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Robotics,” https://doi.org/10.48550/arXiv.1710.11040\n\nFig 2 shows that MASE satisfies the constraints, but if I understand the plot correctly, this is achieved with very conservative margins (constraint at 20, values are typical ≤ 5). This is similar to Saute TRPO, but perhaps suggest why the performance (episode return) is so weak compared to the unconstrained solutions (that don’t violate the constraints by much). Is there a way to trade off  this conservatism to achieve better performance?\n\nVery hard to see the frequency with which TRPO Lagrangian violates the constraints given the lines/colors on fig 2.\n\nQUESTIONS:\nDoes the comment about safe RL problems with chance constraints suggest that there is a better answer available already? The text says that problem 5 in App B is hard to solve, but presumably the authors of [20] and [22] did so? Can their results be compared to the ones here?\n\nLIMITATIONS:\nSee little discussion of this point in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of safe reinforcement learning, in particular safe exploration. In a nutshell, the authors provide an algorithm that is supposed to go beyond standard 'safety' measures in safe RL, such as the constrained MDP setting. There, an agent is bound to satisfy an additional (expected) cost constraint. Here, the authors postulate that an agent must satisfy a constraint almost surely or with high probability. As a key feature of their approach, the authors assume that a so-called  'emergency stop action' is available that allows the agent to always have a fallback. The authors provide a general framework in the form of an algorithm, prove its theoretical guarantees, and evaluate the method on a set of standard benchmarks.\n\nSTRENGTHS:\nThe authors tackle a very important problem, safe exploration in RL. Moreover, they identify correctly that the standard constrained RL (or constrained MDP) setting is generally insufficient to ensure safety during exploration. In principle, the constrained RL setting provides just an incentive to act safely during training, and even after training, safety depends on an expectation to satisfy a safety constraint. Generally, the paper is well-written and easy to follow.\n\nWEAKNESSES:\nI like the paper in general, but in its current state cannot be accepted, in my opinion. The reason is a severe lack of related work. Most importantly, the authors seem unaware of a flavor of safe RL that is often referred to as 'shielding.' In these settings, a so-called shield 'blocks' unsafe actions according to some pre-defined safety measure. [1] was the first paper to introduce this aspect, [2] introduced shields for almost-sure properties in partially observable environments, [3] provides shields that satisfy a property with a certain probability, [4] provides a shielding mechanism for multi-agent settings. There are many more relevant works. The general shielding framework depends on varying assumptions, but the general procedure looks like the MASE algorithm. Note that safety during exploration/training is the most important motivation for shielding. I encourage the authors to thoroughly compare these (and more) works to explain the contribution and novelty better.  \n\n[1] Alshiekh et al.: Safe Reinforcement Learning via Shielding. AAAI 2018\n\n[2] Carr et al.: Safe Reinforcement Learning via Shielding for POMDPs. AAAI 2023\n\n[3] Jansen et al.: Safe Reinforcement Learning Using Probabilistic Shields. CONCUR 2020\n\n[4] Melcer et al.: Shield Decentralization for Safe Multi-Agent Reinforcement Learning. NeurIPS 2022\n\nMoreover, I am not convinced by the experimental evaluation. The emergency action seems central to the approach, but I fail to see how it has been integrated into the standard benchmarks. Then, how often is it called during training by the agent? How does it impede an agent's exploration rate? \n\n\nminor comments\n\nl92: In the definition of a policy, it seems to be non-stochastic. In such a multi-objective setting, it might be beneficial to use stochastic policies. Have you considered this? \n\nl 125/126: in a probabilistic setting, there might be states with a high probability of violating a safety constraint but do not already violate it. It could be interesting to consider such information in the value function.\n\nQUESTIONS:\n1. Compare your approach to the shielding framework.\n\n2. How is the emergency action affecting the agent, please discuss in line with the experiments. \n\n3. How is the 'blocking' of actions realized for an RL agent on a technical level?\n\nLIMITATIONS:\nThe limitations are not properly addressed, see my earlier comments on the evaluation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}