Title: Flipping-based Policy for Chance-Constrained Markov Decision Processes
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement learning

Chance constraints

Stochastic policy
Abstract: Safe reinforcement learning (RL) is a promising approach for many real-world decision-making problems where ensuring safety is a critical necessity. In safe RL research, while expected cumulative safety constraints (ECSCs) are typically the first choices, chance constraints are often more pragmatic for incorporating safety under uncertainties. This paper proposes a \textit{flipping-based policy} for Chance-Constrained Markov Decision Processes (CCMDPs). The flipping-based policy selects the next action by tossing a potentially distorted coin between two action candidates. The probability of the flip and the two action candidates vary depending on the state. We establish a Bellman equation for CCMDPs and further prove the existence of a flipping-based policy within the optimal solution sets. Since solving the problem with joint chance constraints is challenging in practice, we then prove that joint chance constraints can be approximated into Expected Cumulative Safety Constraints (ECSCs) and that there exists a flipping-based policy in the optimal solution sets for constrained MDPs with ECSCs. As a specific instance of practical implementations, we present a framework for adapting constrained policy optimization to train a flipping-based policy. This framework can be applied to other safe RL algorithms. We demonstrate that the flipping-based policy can improve the performance of the existing safe RL algorithms under the same limits of safety constraints on Safety Gym benchmarks.

Flipping-based Policy for Chance-Constrained Markov Decision Processes

Abstract

Safe reinforcement learning (RL) is a promising approach for many real-world decision-making problems where ensuring safety is a critical necessity. In safe RL research, while expected cumulative safety constraints (ECSCs) are typically the first choices, chance constraints are often more pragmatic for incorporating safety under uncertainties. This paper proposes a flipping-based policy for ChanceConstrained Markov Decision Processes (CCMDPs). The flipping-based policy selects the next action by tossing a potentially distorted coin between two action candidates. The probability of the flip and the two action candidates vary depending on the state. We establish a Bellman equation for CCMDPs and further prove the existence of a flipping-based policy within the optimal solution sets. Since solving the problem with joint chance constraints is challenging in practice, we then prove that joint chance constraints can be approximated into Expected Cumulative Safety Constraints (ECSCs) and that there exists a flipping-based policy in the optimal solution sets for constrained MDPs with ECSCs. As a specific instance of practical implementations, we present a framework for adapting constrained policy optimization to train a flipping-based policy. This framework can be applied to other safe RL algorithms. We demonstrate that the flipping-based policy can improve the performance of the existing safe RL algorithms under the same limits of safety constraints on Safety Gym benchmarks.

1 Introduction

In safety-critical decision-making problems, such as healthcare, economics, and autonomous driving, it is fundamentally necessary to consider safety requirements in the operation of physical systems to avoid posing risks to humans or other objects [14, 19, 43]. Thus, safe reinforcement learning (RL), which incorporates safety in learning problems [19], has recently received significant attention for ensuring the safety of learned policies during the operation phases. Safe RL is typically addressed by formulating a constrained RL problem in which the policy is optimized subject to safety constraints [1, 15, 32, 49]. The safety constraints have various types of representations (e.g., expected cumulative safety constraint [4, 5, 7], instantaneous hard constraint [36, 45], almost surely safe [9, 44], joint chance constraint [29-31]). In many real applications, such as drone trajectory planning [40] and

Problem

CCRL

Problem

PMO

Share the same

optimal value

Theorem 1

Problem

FPO

Flipping-based policy

Share the same

optimal value

Theorem 2

Problem

ECRL

Conservative approximation

Theorem 5

•

Flipping-based policy → Optimality

Theorem 4

•

Existing safe RL algorithm → Optimal flipping-

based policy

Theorem 6, Theorem 7

planetary exploration [8], safety requirements must be satisfied at least with high probability for a finite time mission, where joint chance constraint is the desirable representation [33, 43].

Related work. The optimal policy for RL without constraints or with hard constraints is deterministic policy [10, 20, 39]. Introducing stochasticity into the policy can facilitate exploration [11, 38] and fundamentally alter the optimization process during training [16, 21, 37], affecting how policy gradients are computed and how the agent learns to make decisions. It has been shown that the optimal policy for a Markov decision process with expected cumulative safety constraints is always stochastic when the state and action spaces are countable [3]. Policy-splitting method has been proposed to optimize the stochastic policy for safe RL with finite state and action spaces [12]. In [35], an algorithm was proposed to compute a stochastic policy that outperforms a deterministic policy under chance constraints, given a known dynamical model. In more general settings for safe reinforcement learning, such as with uncountable state and action spaces, the theoretical foundation regarding whether and how a stochastic policy can outperform a deterministic policy under chance constraints remains an open problem. Developing practical algorithms to obtain optimal stochastic policies with chance constraints requires further investigation.

Contributions. We present a Bellman equation for CCMDPs and prove that a flipping-based policy archives the optimality for CCMDPs. Flipping-based policy selects the next action by tossing a potentially distorted coin between two action candidates where the flip probability and the two candidates depend on the state. While solving the problem with joint chance constraints is computationally challenging, the problem with the Expected Cumulative Safe Constraints (ECSCs) can be effectively solved by many existing safe RL algorithms, such as Constrained Policy Optimization (CPO, [1]). Thus, we establish a theory of conservatively approximating the joint chance constraints by ECSCs. We further show that a flipping-based policy achieves optimality for MDP with ECSCs. Leveraging the existing safe RL algorithms to obtain a conservative approximation of the optimal flipping-based policy with chance constraints is possible. Specifically, we present a framework for adapting CPO to train a flipping-based policy using existing safe RL algorithms. Finally, we show that our proposed flipping-based policy can improve the performance of the existing safe RL algorithms under the same limits of safety constraints on Safety Gym benchmarks. Figure 1 summarizes the main contributions.

2 Preliminaries: Markov Decision Process

A standard Markov decision process (MDP) is defined as a tuple, ⟨S , A , r, T , µ 0 ⟩ . Here, S is the set of states, A is the set of actions, r : S × A → R is the reward function. This paper considers the general case with state and action sets in finite-dimension Euclidean space, which can be continuous or discrete. Let B ( · ) be the Borel σ -algebra on a metric space and M ( · ) be the set of all probability measures defined on the corresponding Borel space. The state transition model T : S × A → M ( S ) specifies a probability measure of a successor state s + defined on B ( S ) conditioned on a pair of state and action, ( s , a ) ∈ S × A , at the previous step. Specifically, we use p ( ·| s , a ) to define a conditional probability density associated with the state transition model T ( s , a ) . Finally, µ 0 is the distribution of the initial state s 0 ∈ S . A stationary policy κ : S → M ( A ) is a map from states to probability measures on ( A , B ( A )) . We use π ( ·| s ) to define a conditional probability density associated with κ ( s ) , which specifies the stationary policy . Define a trajectory in the infinite horizon by τ ∞ := { s 0 , a 0 , s 1 , a 1 , ..., s k , a k , ... } . An initial state s 0 and a stationary policy π defines a unique probability measure Pr π s 0 , ∞ on the set ( S × A ) ∞ of the trajectory τ ∞

[22]. The expectation associated with Pr π s 0 , ∞ is defined as E τ ∞ ∼ Pr π s 0 , ∞ . Given a policy π ∈ Π , the value function at an initial state s 0 = s is defined by V π ( s ) := E τ ∞ ∼ Pr π s 0 , ∞ { R ( τ ∞ ) | s 0 = s } with R ( τ ∞ ) := ∑ ∞ k =0 γ k r ( s k , a k ) , where γ ∈ (0 , 1) as the discount factor. Also, the action-value function is defined as Q π ( s , a ) := r ( s , a ) + γ E τ ∞ ∼ Pr π s 0 , ∞ { V π ( s + ) | s 0 = s , a 0 = a } .

3 Flipping-based Policy with Chance Constraints

A constrained Markov decision process (CMDP) is an MDP equipped with constraints restricting the set of policies. Let S be the 'safe' region of the state specified by a continuous function g : S → R in the following way: S := { s ∈ S : g ( s ) ≤ 0 } . Let T ∈ N + be the episode length. As suggested in [30, 31], the following joint chance constraints is imposed:

where α ∈ [0 , 1) denotes a safety threshold regarding the probability of the agent going to an unsafe region and [ T ] := { 1 , ..., T } is the index set. The left side of the chance constraint (1) is a conditional probability, specifying the probability of having states of future T steps in the safe region when s k is inside the safe region. When the system is involved with unbounded uncertainty w , it is impossible to ensure the safety with a given probability level in infinite-time scale [18]. Instead, ensuring safety in a future finite time when the current state is within the safety region is reasonable and practical [20]. This paper calls the MDP equipped with chance constraint (1) as Chance Constrained Markov decision processes (CCMDPs). It refers to the problem with almost surely safe constraint when α = 0 . The set of feasible stationary policies for a CCMDP is defined by Π α := { π ∈ Π : ∀ s k ∈ S , (1) holds } . Chance constrained reinforcement learning ( CCRL ) for a CCMDP is to seek an optimal constrained stationary policy by solving

Define optimal solution set of Problem CCRL by Π ⋆ α := { π ∈ Π α : V π ( s ) = max π ∈ Π α V π ( s ) } . Let π ⋆ α ∈ Π ⋆ α be an optimal solution of Problem CCRL . Associated with π ⋆ α , we denote V ⋆ α ( s ) := V π ⋆ α ( s ) and Q ⋆ α ( s , a ) := Q π ⋆ α ( s , a ) for the optimal value and value-action functions.

Define a function P ⋆ ( s , a ) := Pr π ⋆ α s , ∞ { s k + i ∈ S , ∀ i ∈ [ T ] | s k = s , a k = a } . The continuity of P ⋆ ( s , a ) is guaranteed under mild conditions giving as Assumptions 1 and 2 (pp. 78-79 of [25]). Besides, the upper semicontinuity of Q ⋆ α ( s , a ) is from Assumption 1.

Assumption 1. Suppose that A is compact and r ( s , a ) is continuous 1 on S×A . Besides, assume that the state transition model T can be equivalently described by s + = f ( s , a , w ) , where w ∈ W ⊆ R s is a random variable and f ( · ) is a continuous function on S × A × W . The probability density function is p W ( w ) .

Assumption 1 is natural since it only requires that the reward function is continuous and the state transition can be specified by a state space model with a continuous state equation, which is general in many applications. We do not require f ( · ) to be available.

Assumption 2. The constraint function g ( · ) is continuous. For every s ∈ S and a ∈ A , we have

Assumptions 1 and 2 are essentially assuming the regularities of g and T ( s , a ) , which is not a strong assumption. With P ⋆ ( s , a ) , we define a probability measure optimization problem ( PMO ) by

We have the following theorem for the optimal constrained stationary policy π ⋆ α of Problem CCRL :

Theorem 1. The optimal value of Problem PMO equals V ⋆ α ( s ) for any s ∈ S . The probability measure µ ⋆ α associated with π ⋆ α ( ·| s ) is an optimal solution of Problem PMO for any s ∈ S .

The proof is based on the following idea. After showing that V ⋆ α ( s ) is not larger than the optimal value of Problem PMO , we then prove that V ⋆ α ( s ) can only equal to the optimal value of Problem PMO by contradiction. See Appendix B for the proof details. From Theorem 1, we know that the solution of Problem PMO gives the probability measure associated with the action's probability distribution π ⋆ α ( ·| s ) given by the optimal stationary policy, which is Bellman equation for CCMDPs.

Problem PMO is difficult to solve since we must optimize a probability measure, an infinitedimensional variable. We further reduce Problem PMO into the following flipping-based policy optimization problem ( FPO ):

We have the following theorem for Problem FPO :

Theorem 2. Suppose that Assumptions 1 and 2 hold. The optimal objective value of Problem FPO equals to V ⋆ α ( s ) for every s ∈ S . Let the solution of Problem FPO be z ⋆ α ( s ) := ( a ⋆ (1) ( s ) , a ⋆ (2) ( s ) , w ⋆ ( s ) ) . Define a stationary policy ˜ π w α ( ·| s ) that gives a discrete binary distribution for each s , taking a = a ⋆ (1) ( s ) with probability w ⋆ ( s ) and a = a ⋆ (2) ( s ) with probability 1 -w ⋆ ( s ) . The policy ˜ π w α ( ·| s ) is an optimal stationary policy with chance constraint, namely, ˜ π w α ( ·| s ) ∈ Π ⋆ α .

The proof is based on the following idea. We first show that Problem PMO has an optimal solution that is a discrete probability measure (Proposition 1 in Appendix C). We then apply the supporting hyperplane theorem and Caratheodory's theorem to show further that the discrete probability measure can be focused on two points. See Appendix C for the proof details. Theorem 2 simplifies the optimizing of the policy in a probability measure space for each s into an optimization problem in finite-dimensional vector space. An optimal stationary policy gives a discrete binary distribution for each state s . This paper calls the stationary policy with discrete binary distribution as fllipping-based policy since it is similar to the process of random coin flipping, taking a = a ⋆ (1) ( s ) with probability w ⋆ ( s ) and a = a ⋆ (2) ( s ) with probability 1 -w ⋆ ( s ) . Wesummarize one condition that the deterministic policy is enough for the optimality of Problem CCRL in Theorem 3. See Appendix D for the proof.

Theorem 3. Suppose that Assumptions 1 and 2 hold. There exists a deterministic policy that achieves the optimality of Problem CCRL when α = 0 .

4 Practical Implementation of Flipping-based Policy

This section introduces the practical implementation of flipping-based policy. Obtaining the optimal flipping-based policy for CCMDP is intractable due to the curse of dimensionality [38] and joint chance constraints [43]. The parametrization can tackle the curse of dimensionality. The issue by joint chance constraint is resolved by conservative approximation. The common conservative approximation for joint chance constraint is the linear combination of instantaneous chance constraints. We further show that it is possible to find an expected cumulative safety constraint to conservatively approximate the joint chance constraint, which enables Constrained Policy Optimization (CPO) proposed in [1] to find a conservative approximation of Problem CCRL 's optimal flipping-based policy. We show the optimal and finite-sample safety of the flipping-based policy for MDP with the expected cumulative safety constraint.

4.1 Extensions to Other Safety Constraints

Except for the joint chance constraints, several other formulations of safety constraints exist, such as expected cumulative [6] and instantaneous constraints [46]. We extend the optimality of flippingbased policy to other safety constraints to show the generality of our result, which may stimulate further study of designing flipping-based policy for other safe RL formulations.

We introduce the extension of the flipping-based policy to MDP with a single expected cumulative safety constraint. The problem is formulated by

where γ unsafe ∈ (0 , 1) is the discount factor and I ( z ) defines an indicator function with I ( z ) = 1 if z > 0 and I ( z ) = 0 otherwise. The following theorem for Problem ECRL holds:

Theorem 4. A flipping-based policy exists in the optimal solution set of Problem ECRL .

See Appendix E for the proof. The proof follows the same pattern of Theorem 2. We first construct the Bellman recursion with the expected cumulative safety constraint and then prove the existence of a flipping-based policy as optimal policy. The optimality of flipping-based policy can also be extended to the safety constraint function with an additive structure in a finite horizon, written by ∑ T i =1 Pr π θ s , ∞ { s i ∈ S | s 0 = s } . This safety constraint refers to affine chance constraints [13]. We summarize the extension to problems with affine chance constraints in Appendix J.

Remark 1. Theorem 4 can be extended to a more general case where the cumulative safety constraint is not limited to an indicator function but can be any Lipschitz continuous function, thereby broadening the applicability of our theory to more practical scenarios.

4.2 Conservative Approximation of Joint Chance Constraint

We resolve the curse of dimensionality by searching for the optimal policy within a set Π θ ⊆ Π of parametrized policies with parameters θ ∈ Θ ⊂ R n θ , for example, neural networks of a fixed architecture. Here, we use π θ to specify a policy parametrized by θ . If the assumption of the existence of the universal approximator holds, we can approximate the optimal flipping-based policy by using a neural network with state s as input and z ⋆ α ( s ) as output. Another representation of the flipping-based policy is using Gaussian mixture distribution, written by π ( ·| s ) = w ( s ) N ( ¯ a (1) ( s ) , Σ (1) ( s ) ) +(1 -w ( s )) N ( ¯ a (2) ( s ) , Σ (2) ( s ) ) . The output is ¯ a (1) ( s ) , ¯ a (2) ( s ) , w ( s ) , Σ (1) ( s ) , and Σ (2) ( s ) .

If we have w ( s ) = w ⋆ ( s ) , ¯ a (1) ( s ) = a ⋆ (1) ( s ) , and ¯ a (2) ( s ) = a ⋆ (2) ( s ) for every s , the flipping-based policy using Gaussian mixture distribution can approximate the flipping-based policy with binary distribution when the covariances Σ (1) ( s ) and Σ (2) ( s ) vanish for every s [35]. To simplify the implementation, we can use the neural network that outputs z ⋆ α ( s ) and achieve the random search by adding a small Gaussian noise on a ⋆ (1) ( s ) and a ⋆ (2) ( s ) during implementation.

Rewrite the joint chance constraint (1) with s 0 = s for parametrized policy by

In local parametrized policy search ( LPS ) for CCMDPs, θ is updated by solving

Here, D ( · ) denotes a similarity metric between two policies, such as Kullback-Leibler (KL) divergence, and δ > 0 is a step size. The updated policy π θ k +1 is parametrized by the solution θ k +1 of Problem LPS . The updating process considers the joint chance constraint. Problem LPS is challenging to solve directly due to joint chance constraint. Since MDP with the expected discounted safety constraint can be solved by the existing safe RL algorithm (e.g. CPO). Thus, by introducing the conservative approximation of joint chance constraint, we enable the existing safe RL algorithm to obtain a conservatively approximate solution of Problem CCRL . First, we introduce the formal definition of the conservative approximation:

Definition 1. A function C : Θ ×S → R is called a conservative approximation of joint chance constraint (3) if we have

Let H unsafe ( θ , s ) := E τ ∞ ∼ Pr π θ s 0 , ∞ {∑ ∞ i =1 γ i unsafe I ( g ( s i )) | s 0 = s } be a value function for unsafety starting with state s . We have theorem of conservative approximation of joint chance constraint:

Theorem 5. Suppose that Θ and S are compact and Assumption 2 holds. Define a function C unsafe ( θ , s ) by C unsafe ( θ , s ) := H unsafe ( θ , s ) -α. There exist large enough γ unsafe and small enough T such that C unsafe ( θ , s ) ≤ 0 is a conservative approximation of (3) .

The proof of Theorem 5 is summarized in Appendix F. We formulate a conservative approximation of Problem LPS ( CLPS ) as follows:

By Theorem 5, the optimal solution of Problem CLPS is a feasible solution of Problem LPS and thus the corresponding parametric policy is within Π ⋆ α . We have a remark on Theorem 5 as follows:

Remark 2. By the same procedure of proving Theorem 5, we can show that Problem ECRL is a conservative approximation of Problem CCRL .

4.3 Practical Algorithms

Then, we present a practical way to train the optimal flipping-based policy using existing tools in the infrastructural frameworks for safe reinforcement learning research, such as OmniSafe [24]. The provided tools can train the deterministic or Gaussian distribution-based stochastic policies. We take the parameterized deterministic policies as an example, which is specified by π d θ . The parameter θ is within a compact set Θ . We write the reinforcement learning of parameterized deterministic policy ( PDPRL ) for CCMDPs by

Here, the constraint function F d ( θ ) is defined by

Let J ∗ α and Θ ∗ α be the optimal value and optimal solution set of Problem PDPRL . Different from the previous discussions, we here consider the expectation of the initial state s 0 instead of considering the problem for each s 0 . The reason is that the provided tools in OmniSafe, for example, CPO [1] and PCPO [48], address the problems in which the reward functions consider the expectation of the initial state. We extend the previous results of the flipping-based policy to this case.

Let B ( Θ ) be the Borel σ -algebra ( σ -field) on Θ ⊂ R n θ with Euclidean distance. Let ν ∈ M ( Θ ) be a probability measure on ( Θ , B ( Θ )) . With the above notation, associated with Problem PDPRL , a reinforcement learning of parameterized stochastic policy ( PSPRL ) is formulated as:

Let M α ( Θ ) := { µ ∈ M ( Θ ) : ∫ Θ F d ( θ ) d ν ≥ 1 -α } be the feasible set of Problem PSPRL . The optimal objective value and the optimal solution set of Problem PSPRL are J ∗ α := max ν ∈ M α ( Θ ) ∫ Θ J ( θ ) d ν and A α := { µ ∈ M α ( Θ ) : ∫ Θ J ( θ ) d ν = J ∗ α } . A probability measure ν ∗ α ∈ A α is called an optimal probability measure for Problem PSPRL . Define V m := { ν m ∈ [0 , 1] 2 : ∑ 2 i =1 ν m ( i ) = 1 } . Let ζ m := ( ν m (1) , ν m (2) , θ (1) , θ (2) ) be a variable in the set Z m := V m × Θ 2 . Consider an optimization problem on ζ m , reinforcement learning of parameterized flipping-based policy ( PFPRL ), written as

Define Z m ,α := { ζ m ∈ Z m : ∑ 2 i =1 F d ( θ ( i ) ) ν m ( i ) ≥ 1 -α } as the feasible set of Problem PFPRL . In addition, define the optimal objective value and optimal solution set of Problem PFPRL by J w α := min { ∑ 2 i =1 J ( θ ( i ) ) ν m ( i ) : ζ m ∈ Z m ,α } and D α := { ζ m ∈ Z m ,α : ∑ 2 i =1 J ( θ ( i ) ) ν m ( i ) = J w α } , respectively. We have Theorem 6 for parameterized flipping-based policy.

Theorem 6. The optimal values of Problems PFPRL and PSPRL are equal, J ∗ α = J w α . If J ∗ α is a strictly convex function of α on an interval ( α, α ) , then, ∀ α ∈ ( α, α ) , J ∗ α > J ∗ α holds.

See Appendix G for the proof. Note that Theorem 6 clarifies the existence of a parameterized flipping-based policy achieving optimality and the condition under which it performs better than deterministic policies.

Remark 3. Theorems 6 and 2 have different results on the flipping probability. Theorem 2 claims a state-dependent flipping probability while the flipping probability in Theorem 6 is fixed. The

Algorithm 1 General training algorithm for flipping-based policy

1: Set a positive integer S ∈ N + and obtain the sample set Z S = { ˜ α i } S i =1 by randomly sampling ˜ α i ∈ [0 , 1] , ∀ i ∈ [ S ] according to uniform distribution

2: Optimize a policy parameter ˜ θ i for each ˜ α i via an existing safe RL algorithm

3: Solving linear program LP and obtain ( ν s ( j ∗ 1 ) , ν s ( j ∗ 2 ) , ˜ θ j ∗ 1 , ˜ θ j ∗ 2 )

Algorithm 2 Flipping-based policy implementation

1: Observe the state s k at time step k = 0 , 1 , 2 , ...

2: Randomly generate a number κ from [0 , 1] obeying uniform distribution

3: If κ ≤ ν s ( j ∗ 1 ) , generate a k by π d ˜ θ j ∗ 1 . Otherwise, generate a k implement π d ˜ θ j ∗ 2

distinction arises from a subtle difference between Problem CCRL and Problem PSPRL . In Problem CCRL , the optimal policy for each state is derived based on a revised Bellman equation, ensuring that the joint chance constraint is satisfied pointwise for every state. On the other hand, Problem PSPRL focuses on the expectation of the joint chance constraint, evaluated over the probability distribution of the initial state. This formulation eliminates the need for pointwise satisfaction across the state space, causing the state-dependent nature of the constraint to disappear.

There is no existing tool to solve Problem PFPRL and we can only apply them to solve Problem PDPRL for any given α. Let Z S = { ˜ α i } S i =1 , ˜ α i ∈ [0 , 1] , ∀ i ∈ [ S ] be a set of probability levels. For each ˜ α i , define ˜ θ i be the optimal solution of Problem PDPRL with α = ˜ α i . Consider the linear program ( LP ):

Define the optimal objective value and optimal solution set ˜ D α ( Z S ) of Problem LP by ˜ J k α ( Z S ) and ˜ D α ( Z S ) , respectively. The optimal flipping-based policy is characterized by ( ν s ( j ∗ 1 ) , ν s ( j ∗ 2 ) , ˜ θ j ∗ 1 , ˜ θ j ∗ 2 ) , where j ∗ 1 and j ∗ 2 are the index for the non-zero elements of the optimal solution of linear program LP The following theorem holds for ˜ J k α ( Z S ) and ˜ D α ( Z S ) .

Theorem 7. There exists an optimal solution in ˜ D α ( Z S ) such that the number of non-zero elements does not exceed two. Besides, if ˜ α i is extracted independently and identically (uniform distribution), as S →∞ , we have ˜ J k α ( Z S ) →J ∗ α with probability 1.

See Appendix H for the proof. Theorem 6 shows that there exists an optimal solution to Problem PSPRL that is a linear combination of two deterministic policies. Theorem 7 clarifies that we could obtain an approximate flipping-based policy to Problem PSPRL by optimizing the linear combination of multiple trained optimal deterministic policies. One is the linear combination of two policies among all possible linear combinations. The above conclusions can be extended to the Gaussian distribution-based stochastic policies. Besides, the above conclusions still hold after replacing the chance constraint with the expected cumulative safe constraint in CPO and PCPO. We summarize a general algorithm for approximately training the flipping-based policy based on the existing safe RL algorithms in Algorithm 1. With ( ν s ( j ∗ 1 ) , ν s ( j ∗ 2 ) , ˜ θ j ∗ 1 , ˜ θ j ∗ 2 ) , we implement the flipping-based policy by Algorithm 2. In the practical implementation, the weight is constant instead of a function of the initial state since Problems PDPRL and PSPRL consider the expectation of the initial state. Besides, ˜ α i in (1) is replaced by cost limit when using CPO or PCPO to obtain a conservative approximation of (3).

4.4 Safety with Finite Samples

The update by solving Problem CLPS is difficult to implement practically since the evaluation of the constraint function H unsafe ( θ , s ) ≤ α is necessary to clarify whether a policy π is feasible,

which is challenging in high-dimensional cases. Here, we apply the surrogate functions proposed in [1] to replace the objective and constraints of Problem CLPS . With a probability α s ∈ [0 , α ) , the CPO-based approximation of Problem CLPS ( CPOS ) is written by

Note that the optimal solution θ k +1 of Problem CPOS may differ from the one of Problem CLPS . Proposition 2 of [1] gives the upper bound of CPO update constraint violation. The upper bound depends on the values of the step size δ , probability level α s , discount factor γ unsafe , and the maximal expected risk defined by η θ k +1 α s := max s ini ∈ S E a ∼ π θ k +1 { I ( g ( s + )) } . The upper bound can be written √

by H unsafe ( θ k +1 , s ) ≤ α s + 2 δγ unsafe η θ k +1 α s (1 -γ unsafe ) 2 . By choosing sufficiently small step size δ, discount factor γ unsafe , and probability level α s , it is able to ensure that H unsafe ( θ k +1 , s ) ≤ α.

In practical implementation, the exact value of E s ∼ π θ k , a ∼ π θ { I ( g ( s + )) } is unavailable and samples of s ini ∼ π θ k and a ∼ π θ are used to approximate the CPO update. The data set is defined by D N := { ( s ( i ) , a ( i ) , s + , ( i ) ) } N i =1 , where N ∈ N + is the sample number and s + , ( i ) is a sample of successor with previous state s ( i ) and action a ( i ) . Instead of directly solving Problem CPOS , the following sample average approximate of Problem CPOS ( S -CPOS ) is solved:

Here, ˜ H loc unsafe ( θ k , s , γ unsafe , D N ) := H unsafe ( θ k , s )+ 1 (1 -γ unsafe ) N ∑ N i =1 I ( g ( s + , ( i ) )) and ˜ α s ∈ [0 , α s ) is a probability level. The extraction of sample set D N is random, and thus the optimal solution ˜ θ ˜ α s ( s , θ k , D N ) of Problem S -CPOS is a random variable due to the independence on the sample set D N . We need to investigate the probability that ˜ θ ˜ α s ( s , θ k , D N ) admits a feasible policy for Problem CCRL . We have Theorem 8 for the safety with finite sample number. See Appendix I for the proof.

Theorem 8. Suppose that the step size δ > 0 and α s ∈ [0 , α ) are adjusted to ensure that H unsafe ( θ k +1 , s ) ≤ α. There exist T and γ unsafe such that, if ˜ α s ∈ [0 , α s ) , γ unsafe > γ unsafe , and T < T , such that ˜ θ ˜ α s ( s , θ k , D N ) admits a feasible policy for Problem CCRL with a probability larger than 1 -exp { -2 N ( α s -˜ α s ) 2 (1 -γ unsafe ) 2 } .

Note that Theorems 5 and 8 only show the existence of the parameters for safety but do not show an explicit way to choose γ unsafe for specified T . If a conservative safety is desired for practical applications, we recommend using a γ unsafe close to 1 .

5 Experiments

5.1 Numerical Example

We conduct a numerical example to illustrate how the flipping-based policy outperforms the deterministic policy in CCMDPs. The numerical example considers driving a point from the initial point (0 , 0) to the goal (15 , 15) with the probability of entering dangerous regions smaller than a required value. The uncertainties come from the disturbances to the implemented actions. The metric for evaluating the performance is the cumulative inverse distance to the goal, a reward function. Due to the page limit, we summarize the details of the model and heuristic method for obtaining the neural network-based policy in Appendix K. Figure 2 (a) shows trajectories by the deterministic policy in one thousand simulations with mean reward as 0 . 8667 and violation probability as 17% . The red trajectories have intersections with the dangerous region. The deterministic policy led to a sideway in front of the dangerous regions since crossing the middle space violates the violation probability constraint. Figure 2 (b) shows trajectories by flipping-based policy. The mean reward was reduced to 1 . 8259 while the violation probability is 17% , the same as the deterministic policy. The reason is that the flipping-based policy sometimes took the risk of crossing the middle space to improve the mean reward. To balance the violation probability, the sideway root taken by the flipping-based policy was

a)

Trajectoriesby deterministicpolicy

(b)Trajectories by fipping-based policy

(c)Reward v.s.violation probability

20

20

2.5

15

15

2

Meanreward

10

10

1.5

Original

Flipping-based

0

0.5

0

5

10

15

20

0

5

10

15

20

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

C

Violationprobabilitya

60

80

100

120

140

160

180

Cost Limit

5

10

15

20

25

30

Reward

(a) CPO

Original Mean

Flipping-based Mean

60

80

100

120

140

160

180

Cost Limit

5

10

15

20

25

30

(b) PCPO

Original Mean

Flipping-based Mean

more conservative than the deterministic policy. Figure 2 (c) gives the profile of the mean reward along with the violation probability. Until around 22% , the flipping-based policy outperformed the deterministic policy since the violation probability of crossing the middle space is larger than that, and the deterministic policy can cross it. The profile in Figure 2 has a convex shape until 22% . Theorem 6 points out that the strict convexity implies the better reward performance of the flipping-based policy.

5.2 Safety Gym

We conduct experiments on Safety Gym [34], where an agent must maximize the expected cumulative reward under a safety constraint with additive structures. The reason for choosing Safety Gym is that this benchmark is complex and elaborate and has been used to evaluate various excellent algorithms. The infrastructural framework for performing safe RL algorithms is OmniSafe [24]. The proposed method has been validated in two environments: PointGoal2 and CarGoal2. Four algorithms are used as baselines. The first is CPO [1], a well-known algorithm for solving CMDPs. The other three algorithms are PCPO [48], P3O [50], and CUP [47], recent algorithms that achieve superior performance compared to CPO. Due to space limitations, we only present the experimental results of the test processes for CPO and PCPO on PointGoal2. The details of the experimental setup, all four algorithms' training process results on PointGoal2 and their test process results on CarGoal2 are provided in Appendix L.

Baselines and metrics. We implement the practical algorithms presented in Section 4.3 to obtain the flipping-based policy. We modified Algorithm 1 to train the flipping-based policy based on CPO and PCPO. In Algorithm 1, the sample set Z S consists of samples of violation probabilities. Since CPO and PCPO consider the expected cumulative safety constraints, the sample set Z S includes the samples of cost limits of the expected cumulative safety constraints. Instead of using the training

(a) CPO

Original (data)

(b) PCPO

Original (data)

;Original (fitted)

Original (fitted)

Prob.

0.4

口

Flipping (fitted)

Flipping (data)

rob.

0.4

8

00

口

口

口

Flipping (fitted)

Flipping (data)

P

0.2

P

0.2

口

0

0

口

Cost Limit

150

100

口

50

o

口

0

10

T

20

30

Cost Limit

150

100

50

只

0

10

T

20

30

process, we compared the performance of the testing process, where we implemented the trained policy with new randomly generated initial states and goal points and evaluated the expectations of the reward and cost for each trained policy. We investigate whether the expected reward of each baseline under the same expected cost limit can be improved by transforming the policy into a flipping-based policy without any other changes. We employ the expected cumulative reward and the expected cumulative safety as metrics to evaluate the flipping-based policy and the aforementioned baselines. We execute CPO and PCPO with five random seeds and compute the means and confidence intervals.

Results. The experimental results are summarized in Figure 3. As shown in the figure, for CPO and PCPO, the expected reward increases as the expected cost limit rises, and it exhibits convexity at some intervals. At intervals with convexity, the flipping-based policy significantly increases the expected reward. While at intervals without convexity, the flipping-based policy does not increase the expected reward. The above observation fits Theorem 6. From the experiments including more details in Appendix L, we also observe that our flipping-based policy can generally enhance existing safe RL algorithms, although the degree of improvement depends on the original algorithm's performance. Essentially, the flipping mechanism is a linear combination of a performance policy (risky but highperforming) and a safety policy (safe but lower-performing) designed to increase the reward while maintaining the required level of risk. One concern regarding the results in Figure 3 is that the flipping-based policy introduces broader confidence intervals. In theory, however, the flipping-based policy does not increase the size of the confidence intervals. This is demonstrated in the numerical example in Section 5.1, where the policy achieves solutions closer to the optimal ones as outlined in Theorem 2. The practical implementation described in Section 4.3, however, may experience broader confidence intervals due to the presence of two sources of Gaussian noise. It is possible to mitigate this issue by reducing the degree of stochasticity in the policy, for instance, by using smaller variances for the Gaussian noise. This adjustment would not negatively impact the performance in terms of mean reward and cost.

On the other hand, we summarize the results of the relation between expected cumulative safety and the violation probability in Figure 4. Expected cumulative safety and violation probability follows a linear causality, indicating that the flipping-based policy outperforms the deterministic policy under joint chance constraint. With the same expected cumulative safety, a larger T introduces a larger violation probability, which validates Theorem 5.

6 Conclusions

In this article, we first introduce the Bellman equation for CCMDP and prove that a flipping-based polity exists that achieves optimality. We then proposed practical implementation of approximately training the flipping-based policy for CCMDP. Conservative approximations of joint chance constraints were presented. Specifically, we introduced a framework for adapting Constrained Policy Optimization (CPO) to train a flipping-based policy. This framework can be easily adapted to other safe RL algorithms. Finally, we demonstrated that a flipping-based policy can improve the performance of safe RL algorithms under the same safety constraints limits on the Safety Gym benchmark.

References

Achiam, J., Held, D., Tamra, A., and Abbeel, P. (2017). Constrained policy optimization. Proceedings of the 34th International Conference on Machine Learning , PMLR 70: 22-31.

Alshiekh, M., Bloem, R., Ehlers, R., Konighofer, B., Niekum, S., and Topcu, U. (2018). Safe reinforcement learning via shielding. Proceedings of the AAAI conference on Artificial Intelligence , 32(1).

Altman, E. (1995). Constrained Markov Decision Processes , RR-2574, INRIA.

Amani, S., Alizadeh, M., and Thrampoulidis, C. (2019). Linear stochastic bandits under safety constraints. Advances in Neural Information Processing Systems , 32: 9256-9266.

Amani, S., Thrampoulidis, C., and Yang, L. (2021). Safe reinforcement learning with linear function approximation. Proceedings of the 38th International Conference on Machine Learning , PMLR 139: 243-253.

Arnob, G., Zhou, X, and Shroff, N. (2022). Provably efficient model-free constrained RL with linear function approximation. Advances in Neural Information Processing Systems 35: 1330313315.

As, Y., Usmanova, I., Curi, S., and Krause, A. (2022). Constrained policy optimization via Bayesian world models. 2022 International Conference on Learning Representations .

Bajracharya, M., Maimone, M. W., and Helmick, D. (2008). Autonomy for mars rovers: Past, present, and future. Computer , 41(12): 44-50.

Bennett, A., Misra, D., and Kallus, N. (2023). Provable safe reinforcement learning with binary feedback. Proceedings of the 26th International Conference on Artificial Intelligence and Statistics , PMLR 206:10871-10900.

Bertsekas, D.P. and Tsitsiklis, J.N. (2008). Introduction to Probability, Second Edition , Athena Scientific, Belmont, Massachusetts.

Bravo, M. and Faure, M. (2015). Reinforcement learning with restrictions on the action set. SIAM Journal on Control and Optimization , 53(1): 287-312.

Chen, H., Lam, H., Li, F., and Meisami, A. (2020). Constrained reinforcement learning via policy splitting. Proceedings of The 12th Asian Conference on Machine Learning , 129:209-224.

Cui, Y., Liu, J., and Pang, J.S. (2022). Nonconvex and nonsmooth approaches for affine chanceconstrained stochastic programs. Set-Valued and Variational Analysis , 30: 1149-1211.

Dulac-Arnold, G., Levine, N., Mankowitz, D.J., Li, J., Paduraru, C., Gowal, S., and Hester, T. (2021). Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning , 110: 2419-2468.

Ding, D., Zhang, K., Basar, T., and Jovanovic, M. (2020). Natural policy gradient primal-dual method for constrained Markov decision processes. Advances in Neural Information Processing Systems , 33: 8378-8390.

Ding, Y., Zhang, J., and Lavaei, J. (2022). On the global optimum convergence of momentumbased policy gradient. Proceedings of the 25th International Conference on Artificial Intelligence and Statistics , PMLR 151: 1910-1934.

Eckhoff, J. (1993). Helly, Radon, and Caratheodory type theorems. Handbook of Convex Geometry , A: 389-448.

Gao, Y., Johansson, K.H., and Xie, L. (2021). Computing probabilistic controlled variant sets. IEEE Transactions on Automatic Control , 66(7): 3138-3151.

Garcia, J. and Fernandez, F. (2015). A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 16(1):1437-1480.

Gros, G. and Zanon, M. (2020). Data-driven economic NMPC using reinforcement learning. IEEE Transactions on Automatic Control , 65(2): 636-648.

Gu, S., Sel, B., Ding, Y., Wang, L., Lin, Q., Jin, M., and Knoll, A. (2024). Balance reward and safety optimization for safe reinforcement learning: A perspective of gradient manipulation. Proceedings of the AAAI Conference on Artificial Intelligence , 38(19): 21099-21106.

Hernandez-Lerma, O. and Lasserre, J. (1996). Discrete-Time Markov Control Processes: Basic Optimality Criteria , Springer.

Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association , 58: 13-30.

Ji, J., Zhou, J., Zhang, B., Dai, J., and et al (2023). Omnisafe: An infrastructure for accelerating safe reinforcement learning research, arXiv preprint arXiv:2305.09304 .

Kibzun, A., and Kan, Y. (1997). Stochastic Programming Problems with Probability and Quantile Functions. Journal of the Operational Research Society , 48: 846-856.

Lasserre, J. (2010). Moments, Positive Polynomials and Their Applications , Imperial College Press, London.

Luenberger, D.G. (1969). Optimization by Vector Space Methods , John Wiley & Sons, New York.

Melcer, D., Amato, C., and Tripakis, S. (2022). Shield decentralization for safe multi-agent reinforcement learning. Advances in Neural Information Processing Systems , 35: 13367-13379.

Mowbray, M. and et al (2022). Safe chance constrained reinforcement learning for batch process control. Computers & chemical engineering , 157:107630.

Ono, M. (2012). Joint chance-constrained model predictive control with probabilistic resolvability. Proceedings of 2012 American Control Conference (ACC) , 435-441.

Ono, M., Pavone, M., Kuwata, Y., and Balaram, J. (2015). Chance-constrained dynamic programming with application to risk-aware robotic space exploration. Autonomous robots , 39(4): 555-571.

Paternain, S., Chaman, L., Calvo-Fullana, M., and Ribeiro, A. (2019). Constrained reinforcement learning has zero duality gap. Advances in Neural Information Processing Systems , 32.

Pfrommer, S., Gautam, T., Zhou, A, and Sojoudi, S. (2022). Safe reinforcement learning with chance-constrained model predictive control. Proceedings of the 4th Annual Learning for Dynamics and Control Conference , PMLR 168: 391-303.

Ray, A., Achiam, J., and Amodei, D. (2019). Benchmarking safe exploration in deep reinforcement learning . OpenAI.

Shen, X. and Ito, S. (2024). Approximate Methods for Solving Chance-Constrained Linear Programs in Probability Measure Space. Journal of Optimization Theory and Applications , 200: 150-177.

Shi, M., Liang, Y., and Shroff, N. (2023). A near-optimal algorithm for safe reinforcement learning under instantaneous hard constraints. Proceedings of the 40th International Conference on Machine Learning , PMLR 202:31243-31268.

Schulman, J., Levine, S., Abbeel, P., Jordan, M., and P. Moritz, P. (2015). Trust region policy optimization. Proceedings of the 32nd International Conference on Machine Learning , PMLR 37:1889-1897.

Sutton, R.S. and Barto, A.G. (2018). Reinforcement Learning: An Introduction, Second Edition , MIT Press, Cambridge, MA.

Seel, K., Gros, S., and Gravdahl, J.T. (2023). Combining Q-learning and deterministic policy gradient for learning-based MPC. Proceedings of the 62th IEEE Conference on Control (CDC) , 610-617.

Thorp, A.J., Lew, T., Oishi, M.M.K., and Pavone, M. (2022). Data-driven chance constrained control using kernel distribution embeddings. Proceedings of Machine Learning Research , 144: 1-13.

Wachi, A. and Sui, Y. (2020). Safe reinforcement learning in constrained Markov decision processes. Proceedings of the 37th International Conference on Machine Learning (ICML) , PMLR 119: 9797-9806.

Wachi, A., Hashimoto, W., Shen, X., and Hashimoto, K. (2023). Safe exploration in reinforcement learning: a generalized formulation and algorithms. Advances in Neural Information Processing Systems , 36: 29252-29272.

Wachi, A., Shen, X., and Sui, Y. (2024). A survey of constraint formulations in safe reinforcement learning. In 2024 International Joint Conference on Artificial Intelligence .

Wang, Y., Zhan, S., Jiao, R., Wang, Z., Jin, W., Yang, Z., Wang, Z., Huang, C., and Zhu, Q. (2023). Enforcing hard constraints with soft barriers: safe reinforcement learning in unknown stochastic environments. Proceedings of the 40th International Conference on Machine Learning , PMLR 202:36593-36604.

Wei, H., Liu, X., and Ying, L. (2024). Safe reinforcement learning with instantaneous constraints: the role of aggressive exploration. Proceedings of the AAAI Conference on Artificial Intelligence . 38(19): 21708-21716.

Xiong, N., Du, Y., and Huang, L. (2023). Provably safe reinforcement learning with step-wise violation constraints. Advances in Neural Information Processing Systems , 36(2366):5434154353.

Yang, L., Ji, J., Dai, J., Zhang, L., Zhou, B., Li, P., Yang, Y ., and Pan, G. (2022). Constrained update projection approach to safe policy optimization. Advances in Neural Information Processing Systems , 35(662):9111-9124.

Yang, T.Y., Rosca, J., Narasimhan, K., and Ramadge, P.J., (2020). Projection-based constrained policy optimization. 2020 International Conference on Learning Representations .

Ying, D., Ding, Y., and Lavaei, J. (2022). A dual approach to constrained Markov decision processes with entropy regularization. Proceedings of the 25th International Conference on Artificial Intelligence and Statistics , PMLR 151: 1887-1909.

Zhang, L., Shen, L., Yang, L., Chen, S., Wang, X., Yuan, B., and Tao, D. (2022). Penalized proximal policy optimization for safe reinforcement learning. In 2022 International Joint Conference on Artificial Intelligence .

Appendix

A Limitations and Potential Negative Societal Impacts

Limitations. The practical algorithm (Algorithm 1) for training the flipping-based policy is adaptable to any safe RL algorithms. However, it has the following limitations. First, there is a gap in the scenarios with non-smooth functions. The results should be extended to more practical scenarios with non-smooth functions. Second, training a couple of policies is required to find an optimal combination for each cost limit. We could not figure out the convexity after getting enough pairs of expected rewards and costs. Third, the probability of the flip in the practical algorithm is not state-dependent. Although it achieves the optimality of the parameterized flipping-based policy when considering the expectation of the initial state, optimality by Theorem 2 has not yet been achieved. Future work should focus on designing a practical algorithm to obtain the flipping-based policy, for example, training a neural network to take action candidates and the probability of flip as output and the state as input. If we could develop an efficient algorithm to learn the flipping-based policy given by Theorem 2, there is no need to consider the tradeoff between performance and computational complexity.

Potential negative societal impacts. We believe that safety is an essential requirement for applying reinforcement learning (RL) in many real-world problems. While we have not identified any potential negative societal impacts of our proposed method, we must acknowledge that RL algorithms, including ours, are vulnerable to misuse. It is crucial to remain vigilant about the ethical implications and potential risks associated with their application.

B Proof of Theorem 1

The proof sketch is summarized as follows:

(a) Show that V ⋆ α ( s ) is not larger than the optimal value of Problem PMO ;

(b) Assume that V ⋆ α ( s ) is smaller than the optimal value of Problem PMO ;

(c) From (b), we can construct a better policy than which implies that V ⋆ α ( s ) is not the optimal value function;

(d) Since (c) contradicts the fact that V ⋆ α ( s ) is the optimal value function, V ⋆ α ( s ) cannot be smaller than the optimal value of Problem PMO and only equality holds. From the equality, we prove Theorem 1.

Following the above sketch, the proof of Theorem 1 is as follows:

Proof of Theorem 1. For a state s , let µ ∗ α ∈ M ( A ) be the solution of Problem PMO and the associated probability density function is p ∗ α ( · ) . Note that π ⋆ α is a stationary policy and π ⋆ α ∈ Π α . Thus, the probability measure associated with π ⋆ α ( ·| s ) is a feasible solution of Problem PMO . By the definitions of V ⋆ α ( s ) and Q ⋆ α ( s , a ) , we have

Suppose V ⋆ α ( s ) < E a ∼ p ∗ α { Q ⋆ α ( s , a ) } . Since µ ∗ α is a feasible solution of Problem PMO , we have

Thus, by implementing µ ∗ α when the state is s , the probability of having s k + i ∈ S , ∀ i ∈ [ L ] is larger than 1 -α.

Construct a new policy ˜ π ∗ α by

The state is s : ˜ π ∗ α = p ∗ α ( · ) ;

The state is not s : ˜ π ∗ α = π ⋆ α .

Share same optimal value

Theorem 2

Optimal value

Theorem 1

Optimizing discrete distribution

Share same optimal value

Proposition 1

Share same optimal value

Proposition 2

Linear program

•

Proposition 1

•

Supporting hyperplane theorem

•

Caratheodory's theorem

Same problem when

Share same optimal value

Proposition 2

Combine these two

points and obtain

Due to (5) and π ⋆ α ∈ Π α , we have that ˜ π ∗ α ∈ Π α since ˜ π ∗ α satisfies the chance constraint (1). Therefore, we have

Note that (6) contradicts with the fact that π ⋆ α is the optimal stationary policy, which completes the proof.

C Proof of Theorem 2

To help understand the proof of Theorem 2, we illustrate the proof sketch by Figure 5.

Let L ∈ N + be a positive integer and [ L ] := { 1 , ..., L } be the set of the index. Consider the augmented space A L and define an element of S L by C L = ( a (1) , ..., a ( L ) ) . For an arbitrarily given C L , we define a set of discrete probability measures by

The set C L becomes a sample space with finite samples if it is equipped with a discrete probability measure µ d ,L ∈ U d ,L , where the i -th element µ d ,L ( i ) denotes the probability of taking decision a ( i ) , i.e., µ d ,L ( a ( i ) ) = µ d ,L ( i ) , i ∈ [ L ] . In this way, µ d ,L and C L essentially defines a finite linear combination of Dirac measures. We then define a reduced problem of Problem PMO as follows:

Define ˜ U α ( s , L ) := { ( µ d ,L , C L ) : ∑ L i =1 P ⋆ ( s , a ( i ) ) µ d ,L ( i ) ≥ 1 -α } as the feasible set of Problem ˜ B α ( s , L ) . Since the constraint function ∑ L i =1 P ⋆ ( s , a ( i ) ) µ d ,L ( i ) : U d ,L ×A L → R is continuous, and its domain U d ,L × A L is compact, we have the feasible set ˜ U α ( s , L ) of Problem ˜ B α ( s , L ) is also a compact set. As a result, Problem ˜ B α ( s , L ) 's optimal solution exists. We have the following proposition for the relationship between Problems PMO and ˜ B α ( s , L ) :

Proposition 1. Suppose that Assumptions 1 and 2 hold. Then, there exists a finite and positive integer L < ∞ such that makes each optimal solution of Problem ˜ B α ( s , L ) be an optimal solution of Problem PMO .

Proof of Proposition 1. By Assumption 1, We know that A is compact and Q ⋆ ( s , a ) is continuous on S × A (from the one that r ( s , a ) is continuous on S × A ). Besides, by Assumption 2, we know that P ⋆ ( s , a ) is continuous on S × A [25]. Then, the conclusion of Proposition 1 can be directly obtained by applying Theorem 1.3 of [26].

For L = 1 , Problem ˜ B α ( s , L ) becomes a chance-constrained optimization problem in A :

Let J ⋆ α ( s ) be the optimal solution of Problem C α ( s ) . For a given number L ∈ N + , let E L := ( ˜ α (1) , ..., ˜ α ( L ) ) be an element of [0 , 1] L , defining as a set of violation probabilities, where each ˜ α ( i ) is a threshold of violation probability in Problem C α ( s ) when α = ˜ α ( i ) . For a violation probability set E L , we have a corresponding optimal objective value set { J ⋆ ˜ α (1) ( s ) , ..., J ⋆ ˜ α ( i ) ( s ) , ..., J ⋆ ˜ α ( L ) ( s ) } , where J ⋆ ˜ α ( i ) ( s ) is the optimal objective value of Problem C α ( s ) when α = ˜ α ( i ) . Let V d ,L := { ν d ,L ∈ [0 , 1] L : ∑ L i =1 ν d ,L ( i ) = 1 } be a set of discrete probability measures that defined on E L . By determining a violation probability set E L and assigning a discrete probability ν d ,L to E L , we get a probabilistic decision in which the threshold of violation probability is randomly extracted from E L obeying the discrete probability ν d ,L . The corresponding expectation of the optimal objective value is ∑ L i =1 J ⋆ ˜ α ( i ) ν d ,L ( i ) . Another discrete probability measure optimization problem with chance constraint is formulated as

We have the following proposition regarding the optimal values of Problems ˜ B α ( s , L ) and ˜ V α ( s , L ) . Proposition 2. For every L ∈ N + and s ∈ S , the optimal value of Problem ˜ V α ( s , L ) is equal to the one of Problem ˜ B α ( s , L ) .

Proof of Proposition 2. For an arbitrary L ∈ N + and an arbitrary s ∈ S , let ( ˜ µ d ,L , ˜ C L ) be an optimal solution of Problem ˜ B α ( s , L ) , where ˜ C L = ( a (1) , ..., a ( i ) , ..., a ( L ) ) . Notice that ( ˜ µ d ,L , ˜ C L ) is feasible for Problem ˜ B α ( s , L ) and thus we have

Define the optimal value of Problem ˜ B α ( s , L ) by ˜ J α ( s , L ) and thus

Define a set of violation probabilities as

where ˜ α ( i ) = 1 -P ⋆ ( s , a ( i ) ) . Let ν d ,L ∈ V d ,L be a probability measure that satisfies ν d ,L ( i ) = ˜ µ d ,L ( i ) , i ∈ [ L ] . Then, by replacing ˜ α ( i ) = 1 -P ⋆ ( s , a ( i ) ) and ν d ,L ( i ) = ˜ µ d ,L ( i ) into (8), we have

which implies that ( ν d ,L , E L ) is a feasible solution of Problem ˜ V α ( s , L ) . Let ˜ J v α ( s , L ) be the optimal value of Problem ˜ V α ( s , L ) . Then, we have

On the other hand, for an arbitrary L ∈ N + and an arbitrary s ∈ S , let ( ¯ ν d ,L , E L ) be an optimal solution of Problem ˜ V α ( s , L ) , where E L = ( ¯ α (1) , ..., ¯ α ( i ) , ..., ¯ α ( L ) ) . We have

For E L , define a set of decision variables as ̂ C L = { ˆ a (1) , ..., ˆ a ( L ) } , where ˆ a ( i ) is an optimal solution of Problem C α ( s ) with α = ¯ α ( i ) . Note that we have P ⋆ ( s , ˆ a ( i ) ) ≥ 1 -¯ α ( i ) and Q ⋆ ( s , ˆ a ( i ) ) = J ⋆ ¯ α ( i ) ( s ) . Define a discrete probability vector ˆ µ d ,L = ¯ ν d ,L . Since it holds that

we have that ( ˆ µ d ,L , ̂ C L ) is a feasible solution of Problem ˜ B α ( s , L ) . Therefore,

By (12) and (16), we have J b ( s , L ) = J v ( s , L ) ,

˜ α ˜ α which completes the proof.

Based on Theorem 1, Propositions 1 and 2, we give the proof of Theorem 2 as follows:

Proof of Theorem 2. Wefirst show that the optimal objective value ˜ J w α ( s ) of Problem W α ( s ) satisfies

For (18), it can be directly obtained since ˜ J w α ( s ) = ˜ J b α ( s , L ) with L = 2 and the feasible region of Problem ˜ B α ( s , L ) is a subset of the feasible region of Problem PMO with L = 2 , which leads to ˜ J w α ( s ) = ˜ J b α ( s , L ) ≤ V ⋆ α ( s ) . It remains to prove (19). We prove (19) in the following steps:

(a) We apply Proposition 1, supporting hyperplane theorem (p. 133 of [27]), and Caratheodory's theorem [17] to prove that ˜ J v α ( s , L ) ≥ V ⋆ α ( s ) with L = 2 ;

(b) By Proposition 2, we obtain ˜ J w α ( s ) = ˜ J b α ( s , L ) = ˜ J v α ( s , L ) ≥ V ⋆ α ( s ) , which is (19).

Since (b) is obvious if (a) holds, we here focus on proving (a).

Define a set H ( s ) := [0 , 1] × R . Let (˜ α, -J ∗ ˜ α ( s )) ∈ H ( s ) be a pair of violation probability threshold ˜ α and the negative of the corresponding optimal value of Problem C α ( s ) with α = ˜ α . Let conv ( H ( s )) be the convex hull of H ( s ) .

Construct a new optimization problem as

To attain (17), we will show

Let ( ˜ α ⋄ h ,α ( s ) , -J ⋄ h ,α ( s ) ) be an optimal solution of Problem H α ( s ) . We will show that J ⋄ h ,α ( s ) ≥ V ⋆ ( s ) for any s ∈ S , and J ⋄ h ,α ( s ) ≤ ˜ J v α ( s , L ) with L = 2 , which leads to (a).

First, we show that J ⋄ h ,α ( s ) ≥ V ⋆ ( s ) for any s ∈ S . For any L ∈ N + , let ¯ θ L := ( ¯ ν d ,L , ¯ α (1) , ..., ¯ α ( L ) ) be an optimal solution of Problem ˜ V α ( s , L ) and we have

By the definition of convex hull, we know that

Due to (20), ( α mean ( ¯ θ L ) , -J ⋆ mean ( ¯ θ L ) ) is a feasible solution of Problem H α ( s ) and thus we have

Note that (23) holds for an arbitrary L ∈ N + , which includes the one that satisfies ˜ J b α ( s , L ) = V ⋆ α ( s ) (by Proposition 1, there exists one L that attains the equality). Therefore, we have

Then, we show that J ⋄ h ,α ( s ) ≤ ˜ J v α ( s , L ) with L = 2 . To attain this, we first show that (˜ α ⋄ h ,α ( s ) , -J ⋄ h ,α ( s )) is one boundary point of conv ( H ( s )) .

Suppose on the contrary that (˜ α ⋄ h ,α ( s ) , -J ⋄ h ,α ( s )) is an interior point. Thus there exists a neighborhood of (˜ α ⋄ h ,α ( s ) , -J ⋄ h ,α ( s )) such that is within conv ( H ( s )) . Suppose that B ε ( (˜ α ⋄ h ,α ( s ) , -J ⋄ h ,α ( s )) ) ⊂ conv ( H ( s )) , where ε > 0 . For any ˜ ε < ε , we have that (˜ α ⋄ h ,α ( s ) , -J ⋄ h ,α ( s ) -˜ ε ) ∈ B ε ( (˜ α ⋄ h ,α ( s ) , -J ⋄ h ,α ( s )) ) ⊂ conv ( H ( s )) . Since 1 -˜ α ⋄ h ,α ( s ) ≥ 1 -α , ( ˜ α ⋄ h ,α ( s ) , -J ⋄ h ,α -˜ ε ( s ) ) is a feasible solution of Problem H α ( s ) . However, -J ⋄ h ,α ( s ) -˜ ε < -J ⋄ h ,α ( s ) holds and it contracts with that ( ˜ α ⋄ h ,α ( s ) , -J ⋄ h ,α ( s ) ) is an optimal solution of Problem H α ( s ) . Thus, ( ˜ α ⋄ h ,α ( s ) , -J ⋄ h ,α ( s ) ) is a boundary point.

By supporting hyperplane theorem (p. 133 of [27]), there exists a line L that passes through the boundary point ( ˜ α ⋄ h ,α ( s ) , -J ⋄ h ( s ) ) and contains conv ( H ( s )) in one of its closed half-spaces. Note that L is a one-dimensional linear space. Therefore, we can also say ( ˜ α ⋄ h ,α ( s ) , -J ⋄ h ( s ) ) is within the convex hull of L ⋂ H ( s ) , conv ( L ⋂ H ( s )) , namely, ( ˜ α ⋄ h ,α ( s ) , -J ⋄ h ) ∈ conv ( L ⋂ H ( s ) ) . By Caratheodory's theorem [17], we have that ( ˜ α ⋄ h ,α ( s ) , -J ⋄ h ( s ) ) ∈ conv ( L ⋂ H ( s )) is within the convex combination of at most two points in L ⋂ H , namely, ∃ ν ⋄ m ∈ V d , 2 , ∃{ ˜ α (1) m , ˜ α (2) m } ∈ [0 , 1] 2 such that

It holds that

Thus, ( ν ⋄ m (1) , ν ⋄ m (2) , ˜ α (1) m , ˜ α (2) m ) is a feasible solution of Problem ˜ V α ( s , L ) when L = 2 . Therefore, we have

From J ⋄ h ,α ( s ) ≤ ˜ J v α ( s , L ) and J ⋄ h ,α ( s ) ≥ V ⋆ α ( s ) (by (24)), we have ˜ J v α ( s , L ) ≥ V ⋆ α ( s ) . Since ˜ J b α ( s , L ) = ˜ J v α ( s , L ) by Proposition 2, we have ˜ J b α ( s , L ) ≥ V ⋆ α ( s ) , which leads to ˜ J b α ( s , L ) = V ⋆ α ( s ) since ˜ J b α ( s , L ) ≤ V ⋆ α ( s ) also holds. Note that ˜ J w α ( s , L ) = ˜ J b α ( s , L ) , we have (17). The rest of Theorem 2 can be shown by applying Theorem 1, which completes the proof of Theorem 2.

D Proof of Theorem 3

As preparation for proving Theorem 3, we first give the following proposition on the optimal solution of Problem FPO .

Proposition 3. Let z ⋆ α ( s ) = ( a ⋆ (1) ( s ) , a ⋆ (2) ( s ) , w ⋆ ( s ) ) be an optimal solution of Problem FPO . Let ˜ α (1) = 1 -P ⋆ ( a ⋆ (1) ( s )) and ˜ α (2) = 1 -P ⋆ ( a ⋆ (2) ( s )) . We have

Proof of Proposition 3. By repeating the proof of Theorem 1, we can show that ˜ V d α ( s ) equals to the optimal value of the following problem:

Based on Proposition 3, we give the proof of Theorem 3.

Proof of Theorem 3. If α = 0 , the optimal solution of Problem ˜ V α ( s , L ) with L = 2 has to set ˜ α (1) = ˜ α (2) = 0 , which leads to

Then, by Proposition 2 and Theorem 2, we have

which can be attained by an optimal solution of Problem C α ( s ) referring to a deterministic policy.

E Proof of Theorem 4

Proof. Define the discounted return of a specific trajectory with γ unsafe ∈ (0 , 1) by

Define a function G ⋆ ( s , a ) by

Here, π ⋆ sec is an optimal solution of Problem ECRL . From the definition of G ( τ ∞ ) in (31), we can rewrite G ⋆ ( s , a ) in the following way:

The continuity of each Pr π ⋆ sec s 0 , ∞ { I ( g ( s i )) | s 0 = s , a 0 = a } is guaranteed by Assumption 2 and the continuity of g ( · ) (pp. 78-79 of [25]), which naturally leads to the continuity of G ⋆ ( s , a ) . With G ⋆ ( s , a ) , a probability measure optimization problem is defined as follows:

By just repeating the proof of Theorem 1, we can obtain that the optimal objective value of Problem B sec α ( s ) equals the one of Problem ECRL for any s ∈ S . A flipping-based version of Problem B sec α ( s ) is written by

The continuity of G ⋆ ( s , a ) holds and it is bounded within [0 , 1] . Thus, Theorem 4 can be proved by following the same process of proving Theorem 2 after replacing P ⋆ ( s , a ) by G ⋆ ( s , a ) .

F Proof of Theorem 5

Proof. Define a violation probability function V joint ( θ , s ) by

Note that the constraint V joint ( θ , s ) ≤ α is equivalent to

By using Boole's inequality, we have

Due to (35), ˜ V T unsafe ( θ , s ) ≤ α, ∀ s ∈ S implies V joint ( θ , s ) ≤ α, ∀ s ∈ S . By replacing s by s k , we obtain (4) by setting C ( θ , s ) = ˜ V T unsafe ( θ , s ) -α. Then, ˜ V T unsafe ( θ , s ) -α is a conservative approximation of joint chance constraint (3).

Then, we will show that we can find γ unsafe and T to make the following equality holds:

Define ˜ V T unsafe by

Define V T,γ unsafe unsafe ( θ , s ) by

Define the error between V T,γ unsafe unsafe ( θ , s ) and H unsafe ( θ , s ) by

Note that ˜ ϵ inf V ( T, γ unsafe , θ , s ) is positive for any θ ∈ Θ , s ∈ S , γ unsafe ∈ (0 , 1] and it decreases monotonically as T increases. It increases monotonically as γ unsafe increases to 1.

On the other hand, define the error between V T,γ unsafe unsafe ( θ , s ) and ˜ V T unsafe ( θ , s ) by

The error ˜ ϵ T V ( T, γ unsafe , θ , s ) decreases monotonically as γ unsafe increases to 1. It decreases monotonically as T decreases.

We give the error between H unsafe ( θ , s ) and ˜ V T unsafe ( θ , s ) as follows:

For any given θ , s , it is able to decrease T and meanwhile increase γ unsafe to simultaneously achieve:

increasing ˜ ϵ inf V ( T, γ unsafe , θ , s ) ;

decreasing ˜ ϵ T V ( T, γ unsafe , θ , s ) .

Then, there is small enough T and large enough γ unsafe to ensure that ϵ V ( T, γ unsafe , θ , s ) > 0 . Besides, since Θ and S are compact and functions H unsafe ( θ , s ) , ˜ V T unsafe ( θ , s ) , and V T,γ unsafe unsafe ( θ , s ) are continuous (yielded by Assumption 2), T and γ unsafe such that, if γ unsafe > γ unsafe and T < T , ϵ V ( T, γ unsafe , θ , s ) > 0 , ∀ θ ∈ Θ , s ∈ S . Then, we have

H unsafe ( θ , s ) -α ≤ 0 ⇒ ˜ V T unsafe ( θ , s ) + ϵ V ( T, γ unsafe , θ , s ) -α ≤ 0 ⇒ ˜ V T unsafe ( θ , s ) -α ≤ 0 . Thus, by replacing s by s k , we obtain (3) by setting C ( θ , s ) = H unsafe ( θ , s ) -α if γ unsafe > γ unsafe and T < T. Then, H unsafe ( θ , s ) -α is a conservative approximation of joint chance constrain (3).

G Proof of Theorem 6

As a preparation for proving Theorem 6, we first give the following proposition for the optimal solution of Problem PFPRL .

Proposition 4. Let ζ ∗ m = ( ν ∗ m (1) , ν ∗ m (2) , θ (1) ∗ , θ (2) ∗ ) ∈ D α be an optimal solution of Problem PFPRL . Let ˜ α (1) = 1 -F d ( θ (1) ∗ ) and ˜ α (2) = 1 -F d ( θ (2) ∗ ) . We have θ (1) ∗ ∈ Θ ∗ ˜ α (1) , θ (1) ∗ ∈ Θ ∗ ˜ α (2) .

Proof of Proposition 4. Suppose that θ (1) ∗ / ∈ Θ ∗ ˜ α (1) . Then, J ( θ (1) ∗ ) < J ∗ ˜ α (1) holds. Let ˆ θ (1) ∈ Θ ∗ ˜ α (1) . Then, F d ( ˆ θ (1) ) ≥ 1 -˜ α (1) = F d ( θ (1) ∗ ) and J ( ˆ θ (1) ) = J ∗ ˜ α (1) > J ( θ (1) ∗ ) . We have

Thus, ˆ ζ m = ( ν ∗ m (1) , ν ∗ m (2) , ˆ θ (1) , θ (2) ∗ ) is a feasible solution of Problem PFPRL and has a larger objective function value than ζ ∗ m which contradicts to ζ ∗ m ∈ D α . Therefore, we have θ (1) ∗ ∈ Θ ∗ ˜ α (1) . Follow the above procedures, we could also prove that θ (2) ∗ ∈ Θ ∗ ˜ α (2) .

Then, we give the proof of Theorem 6 as follows:

Proof of Theorem 6. For J ∗ α = J w α , it can be obtained by repeating the proof of Theorem 2.

Let ζ ∗ m = ( ν ∗ m (1) , ν ∗ m (2) , θ (1) ∗ , θ (2) ∗ ) ∈ D α be an optimal solution of Problem PFPRL . Let ˜ α (1) = 1 -F d ( θ (1) ∗ ) and ˜ α (2) = 1 -F d ( θ (2) ∗ ) . By Theorem 6 and Proposition 4, we have

Since J ∗ α is a strictly convex function of α on ( α, α ) , we have

which completes the proof.

H Proof of Theorem 7

Proof of Theorem 7. Since Problem LP is a special case of Problem PSPRL , Theorem 6 implies the existence of an optimal solution in ˜ D α ( Z S ) with no more than two non-zero elements.

Since ˜ θ i is the optimal solution of Problem PDPRL with α = ˜ α i , Problem LP has the same optimal value with the following optimization problem:

Define another optimization problem as:

Let ̂ J k α and ̂ D α be the optimal solution and optimal solution set of Problem ̂ K ˜ α .

Note that Z S = { ˜ α i } S i =1 is extracted according to uniform distribution. By applying Theorem 6 of [35], we have

˜ J k α ( Z S ) ≤ ̂ J k α ;

˜ J k α ( Z S ) → ̂ J k α with probability 1 as S →∞ .

Then, if we show that ̂ J k α = J ∗ α , it leads to ˜ J k α ( Z S ) →J ∗ α with probability 1 as S →∞ . By Proposition 4, we have

where θ ( i ) ∗ is one optimal solution of Problem PDPRL with α = ˜ α ( i ) , i = 1 , 2 . We further have

Thus, the probability measure ˜ ν flip c defined by

is a feasible solution of Problem ̂ K ˜ α , which implies that

On the other hand, by applying Theorem 6, we have that Problem ̂ K ˜ α has a flipping-based optimal solution ˆ ν flip c ,

which implies that ̂ J k α equals an objective value of a feasible solution of Problem PSPRL . Thus,

which completes the proof.

It leads to

Due to 46 and (50), we have ̂ J k α = J ∗ α ,

I Proof of Theorem 8

Proof. First, we show that ˜ θ α s ( s , θ k , D N ) is a feasible solution of Problem CPOS with probability larger than 1 -exp { -2 N ( α s -˜ α s ) 2 (1 -γ unsafe ) 2 } . Define two functions by

Here, we simplify the notation by omitting s and θ k since it claims the same conclusion for all s and θ k . Besides, define two probability α trf s and ˜ α trf s transformed from α s and ˜ α s by

Note that α trf s -˜ α trf s = ( α s -˜ α s )(1 -γ unsafe ) .

Let ˆ θ bad be an infeasible solution of Problem CPOS . Then, we have

The probability of ˆ θ bad being a feasible solution of Problem S -CPOS is Pr { ˜ F ( ˆ θ bad , D N ) ≥ 1 -˜ α trf s } which satisfies that

where Y i is defined by

According to Hoeffding's inequality [23], (52) implies

Here, (53) means that the probability of ˆ θ bad being a feasible solution of Problem S -CPOS is smaller than exp { -2 N ( α s -˜ α s ) 2 (1 -γ unsafe ) 2 } , which implies that a feasible solution of Problem S -CPOS has a probability larger than 1 -exp { -2 N ( α s -˜ α s ) 2 (1 -γ unsafe ) 2 } to be a feasible solution of Problem CPOS . The optimal solution ˜ θ α s ( s , θ k , D N ) of Problem S -CPOS is also included.

When ˜ θ α s ( s , θ k , D N ) is feasible for Problem CPOS , it is feasible for Problem CLPS . By applying Theorem 5, we have that ˜ θ α s ( s , θ k , D N ) is also feasible for Problem LPS and thus the policy admitted by ˜ θ α s ( s , θ k , D N ) satisfies the joint chance constraint in Problem CCRL .

J Conservative approximation by affine chance constraint

Firstly, we will show that an affine chance constraint exists to approximate the joint chance constraint conservatively. The approximate problem also has a flipping-based policy in the optimal solution set. Since the problem with affine chance constraint can be transformed into the generalized safe exploration ( GSE ) problem [42], MASE and shielding methods [2, 28] can be applied to solve it, which gives the approximate solution of Problem CCRL .

Let H acc ( θ , s ) be a unsafety function defined by

Notice that H acc ( θ , s ) satisfies

Thus, the constraint H acc ( θ , s ) ≤ φ is equivalent to

which is a special case of affine chance constraint [13]. The theorem of conservative approximation of joint chance constraint based on affine chance constraint is as follows:

Theorem 9. Suppose that S is compact and Assumption 2 holds. Define a function C acc ( θ , s ) as follows:

If φ ≤ α , C acc ( θ , s , φ ) is a conservative approximation of joint chance constraint (3) .

Proof. From the definition of H acc ( θ , s ) as (54), we have

Due to Boole's inequality (p. 14 of [10]), we further have

Thus, by (57), the following holds

Since the left side of (58) is equivalent to

(58) implies that C acc ( θ , s , φ ) is a conservative approximation of joint chance constraint (3).

MDP with affine chance constraint can be written by

where the constraint of Problem A α ( s ) is a conservative approximation of (1), which can be proved following the same flow of Theorem 9. The following theorem for Problem A α ( s ) holds:

Theorem 10. A flipping-based policy exists in the optimal solution set of Problem A α ( s ) .

Proof. Define a function H ⋆ acc ( s , a ) by

Here, π ⋆ acc is an optimal solution of Problem A α ( s ) . The continuity of H ⋆ acc ( s , a ) is guaranteed by Assumption 2 and the continuity of g ( · ) (pp. 78-79 of [25]). With H ⋆ acc ( s , a ) , a probability measure optimization problem is defined as follows:

By just repeating the proof of Theorem 1, we can obtain that the optimal objective value of Problem B acc α ( s ) equals the one of Problem A α ( s ) for any s ∈ S . A flipping-based version of Problem B acc α ( s ) is written by

Since the continuity of H ⋆ acc ( s , a ) holds and it is bounded within [0 , 1] , Theorem 10 can be proved by following the same process of proving Theorem 2 after replacing P ⋆ ( s , a ) by H ⋆ acc ( s , a ) .

K Details of Numerical Example

We present the details of our numerical example. The system dynamics is described by

Here, d t is the sampling time, the system state is s k := [ x k y k ] ⊤ representing the position of the point, the action is a k := [ u k v k ] ⊤ representing the velocity on each direction, and the disturbance vector is d k := [ δ k ζ k ] ⊤ representing the system disturbance. Both δ k , ζ k are random variables with zero means and standard deviations as 0 . 6 . The initial point is s 0 = [0 0] ⊤ . The goal point is s g = [15 15] ⊤ . The instantanuous loss function at step k is ℓ ( s k ) := ∥ s k -s g ∥ 2 2 . For a given time-horizon T, we consider the joint chance constraint Pr {( ∧ T k =1 s k / ∈ O 1 ) ∧ ( ∧ T k =1 s k / ∈ O 2 )} ≥ 1 -α, where the dangerous regions O 1 and O 2 are defined by O 1 := { s : ∥ s -s o1 ∥ 2 ≤ 2 . 5 } , s o1 = [7 . 5 10] ⊤

A value of

Sampling

Solving optimization

problem

Otrain

Data set

Train neural

networks

Deterministic

policy

and O 2 := { s : ∥ s -s o2 ∥ 2 ≤ 2 . 5 } , s o2 = [10 5] ⊤ . This numerical example investigates whether and when optimal flipping-based policy outperforms optimal deterministic policy under the same violation probability constraint. Therefore, instead of validating the algorithms of optimizing the policy, we implemented a heuristic method to obtain an optimal deterministic policy for each violation probability limit α . Then, following Algorithm 1, we obtained the optimal flipping-based policy for each α. The framework of heuristically obtaining the optimal deterministic policy is summarized in Figure 6. The heuristic method to obtain an optimal deterministic policy includes two steps. First, for a given initial state s ( i ) ini , we solve the following optimization problem ( O train ):

Here, A is a two-dimensional identity matrix and the extended dangerous regions ˜ O ext 1 and ˜ O ext 2 are defined by ˜ O ext 1 ,k := { s : ∥ s -s o1 ∥ 2 ≤ 2 . 5 + 0 . 6 β √ k d t } , s o1 = [7 . 5 10] ⊤ and ˜ O ext 2 ,k := { s : ∥ s -s o2 ∥ 2 ≤ 2 . 5 + 0 . 6 β √ k d t } , s o2 = [10 5] ⊤ . Here, β is a coefficient to regulate the violation probability. Note that the disturbance obeys Gaussian distribution with zero covariance and the same deviation, and the confidence region for any given probability can be described by a circle. Thus, by regulating β , we can ensure the probability confidence of the obtained solution. For a given β , if we solve problem ( O train ) for any s ( i ) ini , i = 1 , ..., N and extract the first one ˆ a ( i ) 0 of the solution sequence, we can obtain a set D β N := { ( s ( i ) ini , ˆ a ( i ) 0 ) } N i =1 . Then, we can use D β N to train a neural network-based policy with the state as input and the action as output. We obtain neural network-based policies with different violation probability thresholds by varying β from 1 to 2 . 2 with 0 . 05 as an increment. In the test, we use the inverse distance as a metric for evaluating performance, which is defined by r ( s k ) := 1 / ( ∥ s k -s g ∥ 2 2 +0 . 1 ) . We tested each neural network with five times simulation sets. In each simulation set, one thousand simulations were conducted to calculate the violation probability and the mean reward.

L Details of Safety-Gym Experiment

We used a machine with Intel(R) Core(TM) i7-14700 CPU, 32GB RAM, and NVIDIA 4060 GPU. We present the details of our experiments using Safety Gym. Our experimental setup differs slightly from the original Safety Gym in that we deterministically replace the obstacles (i.e., unsafe regions). This modification ensures that the environment is solvable and that a viable solution exists. Details of our experiments using Safety Gym are as follows. To ensure the generalization of the algorithm, we

Note: Same training parameters as in training process except for training scale.

used the original SafetyPointGoal2-v0 environment. In the initial stage, we identified parameters with good convergence properties according to specified criteria. Using these parameters, we trained the CPO and PCPO algorithms at 10 cost limit intervals within the range of 40-180, continuing until the policy network converged. In the testing experiments, we utilized the saved parameters from these converged networks, loading them into the policy network and sampling data under different seeds. Finally, we selected the policy network at an appropriate cost limit as the base network for the Flipping-based policy. When the two base networks output different actions at each step, we chose different actions according to the flip probability, sampling the results under various seed environments to obtain the final results for the Flipping-based policy.

Collision Probability Analysis. We monitored whether the agent encountered collisions under each single-step condition across all tests. Subsequently, we computed the collision probabilities for T = 3 , 10 , 30 by assessing the presence of collisions across every T consecutive step. The findings are presented in Figure 4. It is important to acknowledge that, despite utilizing a trained and converged stable policy network during the collision testing phase, the collision probability and

cost limit curves might exhibit some instability, attributed to the relatively small sample size of 25 trajectories. This variability is likely because the CPO and PCPO algorithms were not primarily designed to minimize collision probabilities. Nonetheless, the curves demonstrate a clear positive correlation, affirming the validity and reliability of our experimental outcomes. Our analysis further supports that the flipping-based method effectively enhances reward performance without increasing collision risks.

We also conducted experiments for two more baselines, P3O and CUP on PointGoal2. Figure 7 presents the experimental results, showing that the flipping-based policy can also enhance the performance of P3O and CUP. Additionally, experiments were conducted for the four baselines-CPO, PCPO, P3O, and CUP-under the CarGoal2 environment, with the results summarized in Figure 8. The findings in CarGoal2 are consistent with those in PointGoal2.

Figures 9 and 10 summarize reward and cost profiles of the training processes for each baseline algorithm on PointsGoal2. The training results further confirm that combining a performance policy with a safe policy yields a flipping-based policy that outperforms the policy trained by the original algorithm, while adhering to the required cost limit.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract, we explicitly state that our contribution is introducing a flippingbased policy for safe reinforcement learning, accompanied by theoretical foundations for optimality and practical implementation. Safe reinforcement learning forms the central scope of our paper. Further, in the introduction, we delineate the scope within the initial two paragraphs and subsequently detail the contribution in the third paragraph.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We give a separate "Limitations" part in Appendix A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided the full set of assumptions and a complete proof. The complete proofs are included in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have fully disclosed all the information needed to reproduce the numerical example's results of the paper in Section 5.1 and Appendix K, and the main experimental results of the paper in Section 5.2 and Appendix L.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided the code in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the numerical example's details in Appendix K and the experimental details in Appendix L.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in the appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide the error bars to show the statistical significance of the experiments, which are given in Figure 3.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the information on the computer resources at the beginning of Section 5.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conducted the research conforming in every respect with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In the final paragraph of the Introduction and the Conclusion, we discuss the societal impacts of our work. Our approach enhances the performance of existing safe reinforcement learning (RL) algorithms while adhering to the same safety constraints. This advancement promotes the adoption of safe RL in safety-critical decision-making

applications, such as healthcare, economics, and autonomous driving. Besides, we have not yet found the negative societal impacts of the work. The details is given in Appendix A.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not include any data or models with a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We conducted experiments using the Safety Gym environment [34]. The experimental framework of Safe RL algorithms was facilitated by OmniSafe [24]. These details are disclosed at the beginning of Section 5.2. Both Safety Gym and OmniSafe are accessible online, and further information can be found in the aforementioned references.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets. We use the existing toolbox or data sets for the experiment to validate our theory.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not include any research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not include any research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is dealing with a chance constrained MDP problem, where the authors constrain the probability of staying in the safe set throughout an episode. The authors show that the original problem can be recast using a flipping-based policy, where the optimal decision to satisfy the constraint is conditioned on the current state (regardless if it is in the safe set or not). The authors proceed to develop a practical algorithm that can be applied to standard RL algorithms such as CPO and demonstrate the utility of their approach. In particular they show that the flipping based policy can outperform significantly a deterministic policy\n\nSTRENGTHS:\n1. The paper is rigorous and the maths seem to be correct, however, I have a few questions and clarifications and to be honest I didn’t have time to validate the proofs.\n1. The “secret” of many practical safe RL research is that, in fact, that many algorithms are shown to train well a deterministic policy. The authors demonstrate that there is a possibility to train a truly stochastic policy as well. I find this line of research very valuable.\n\nWEAKNESSES:\n1. Some of the theoretical results would benefit from explanations and intuitions behind them. For instance, why do they work? See questions\n1. I found the discussion of the practical algorithm confusing (but keep in mind time limitation for reviews). For example, the problem LP is not clear. See questions.\n1. The paper is heavy on the theory side, which is not necessarily a weakness, but the experimental side could be improved. For example, I am not sure how, in fact, easy it is to adapt, e.g., safe PPO to a flipping-based policy.\n\nQUESTIONS:\n1. Proposition 1 seems like a very strong result and I don’t have access [21]. Could the authors restate Theorem 1.3 from [21] and explain the intuition behind it. For example, how is it possible to cast an infinite dimensional problem to finite dimensions? \n1. What is the intuition behind the possibility of using only two policies and not L as Proposition 1 suggests? \n1. If I understand correctly the problem LP solves a number of PDPRL problems for a number of different safety levels $\\alpha_i$ and then compares it with a predefined safety level $\\alpha$. Then chooses only two values of $\\alpha_i$ for the final flipping policy. Could you explain the intuition behind it. Why does this approach work?\n1. What steps one has to take to adapt say safe PPO to a flipping-based policy?\n1. How did you reformulate the PointGoal problem as a chance constrained problem?\n\nLIMITATIONS:\nlimitations are discussed", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a number of optimality results for flipping-based policies in safe reinforcement learning. It first establishes that flipping-based policies are overall optimal for joint chance constrained MDPs, which represents a significant reduction of the original optimization formulation. Then, acknowledging that even this formulation is practically intractable, it provides a number of relaxations and principled approximations to get to a computationally tractable procedure, and show that one can find flipping-based policies that outperform the best deterministic policies that one can find.\n\nSTRENGTHS:\n1. Safe RL is a very important and challenging problem.\n2. Theorem 2 (and 4 and 6 and 7) is quite interesting and, at least to me, surprising!\n3. The authors take a methodical and principled approach to connect their main theory in section 3 with a practical algorithm that can be used for RL by the end of section 4.\n\nWEAKNESSES:\n1. Theorems 5 and 8 don’t seem very strong or useful since they don’t specify the constraints on \\gamma_unsafe and T (just say they exist), so in particular it may not hold at all for a given user-specified T, and even if it does, it doesn’t provide a way for the user to find a \\gamma_unsafe that makes the approximation conservative for their specified T.\n\nQUESTIONS:\n1. line 91: are there \\alpha subscripts missing in a couple places here? I don’t think, e.g., \\pi^* has been defined without an \\alpha subscript.\n2. Line 204: I think there’s a conditioning on s_k=s missing in the definition of F^d. Or otherwise I don’t know what the s in the subscript means.\n3. In 4.3, is there a distinction between \\Theta and \\boldsymbol{\\Theta}?\n4. In 5.2, I think you mean to say the reward INCREASED with the flipping policy\n5. Theorem 6 seems far stronger than Theorem 2, and I feel this gap is quite counterintuitive and unacknowledged by the authors, though maybe I’m missing something; if this point could be clarified, I would raise my score. In particular, Theorem 2 says that in the current state, the controller can select from just two (state-dependent) actions (with a state-dependent flipping probability) at the current time point while remaining optimal. But Theorem 6 seems to say that the entire optimal policy can be written as a (fixed, not-state-dependent) mixture of two policies. Perhaps most surprising is the fact that the flipping weight between the two policies is not state-dependent, whereas in Theorem 2 it was allowed to depend on state. Is there an easy way to see why it’s not so surprising? For instance, if the parameterized deterministic policy class is a universal approximating (deterministic) function class, then the PSPRL optimum should essentially be the same as the original CCRL optimum, right (I can see that if PSPRL can implement any policy via \\theta and then arbitrarily mix between them, and by having a continuous mixture of policies that can take the same action at some states and different actions at others, the PSPRL optimum can, e.g., implement a switching policy with a different weight for every state)? But then Theorem 6, if I’m reading it right, says there’s an optimal policy for the original problem with a switching weight that doesn’t depend on state at all! What am I missing?\n\nLIMITATIONS:\nThe experimental results certainly support the rest of the paper, but are somewhat unsurprising, since they basically just show that there can be a mixture of two deterministic policies that beats the best deterministic policy—no surprise there since mixtures of two deterministic policies is a larger class than deterministic policies. And the actual flipping policy found in figure 2 seems a bit troubling to me—in practice would one want a policy that improves upon another safe policy by with some probability taking a marginally unsafe shortcut? For, e.g., a self-driving car, I would probably say no, but maybe this is a weakness of the joint chance constraint formulation, which only conditions on safety, but not the state itself, when demanding joint safety for the next T time steps. I have to agree that, for the problem as formulated, the flipping policy in Figure 2 is indeed better than the deterministic policy. Perhaps a bit more discussion of the problem formulation is merited, as opposed to just referencing other works that use the same formulation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

comments: SUMMARY:
The study proposes a flipping-based policy for managing chance constraints in Markov Decision Processes (MDPs). It introduces a probabilistic approach where actions are selected by flipping a "distorted coin", which is helpful in handling uncertainties in safety-critical environments. The authors establish a theoretical framework, including a Bellman equation adaptation for this setup and proofs of optimality under given constraints.

STRENGTHS:
1. The flipping-based approach is a potentially impactful method for addressing uncertainty in decision-making processes.
2. The paper provides rigorous theoretical backing for the proposed method, including detailed proofs.

WEAKNESSES:
1. The algorithm involves multiple layers of optimization and probability adjustments which might be difficult to tune in real-world applications.
2. The experimental section primarily focuses on simulated environments (Safety Gym benchmarks), and the performance is not significantly improvement compared with CPO and PCPO. There are some new methods that significantly better than CPO and PCPO, e.g., https://arxiv.org/pdf/2405.05890 and https://arxiv.org/pdf/2405.01677

QUESTIONS:
1. Could we possibly use neural networks to predict action candidates and flip probabilities?
2. What is the difference between flip-based policy and reject sampling?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new policy called the flipping-based policy for Chance-Constrained Markov Decision Processes (CCMDPs), which is useful in safe reinforcement learning. The policy uses a coin flip to choose between two actions, depending on the state. The authors establish a Bellman equation for CCMDPs and show that this flipping-based policy can be part of the optimal solution. To provide a practical algorithm, they also demonstrate how joint chance constraints can be approximated into Expected Cumulative Safety Constraints (ECSCs). The paper presents a framework for integrating this policy into existing safe RL algorithms like CPO and PCPO, showing improvements in performance on Safety Gym benchmarks while maintaining safety constraints.\n\nSTRENGTHS:\n1. This paper presents an extensive detailed theoretical analysis, with the entire analytical and proof process vividly illustrated in Figure 1 and Figure 5.\n2. The concept of employing a flipping policy is quite novel, offering a simple yet potent perspective to solve the problem PMO.\n3. This paper delivers a comprehensive framework from theoretical analysis to practical general safety RL algorithms, providing mathematically insightful and practically effective solutions. The experimental section and visualizations are designed to complement the core theorems, ensuring a cohesive understanding.\n\nWEAKNESSES:\n1. The typo in the abstract: expected cumulative safety constraints (ESCSs) should be ECSCs.\n2. The typo in Algorithm 2, line 2: Remove one of the redundant as in the statement.\n3. The assumption regarding the reward and state transition function, as mentioned by the author, is not inherently natural or generalizable across various applications. This is particularly evident in decision-making environments within robotics and autonomous driving, where reward signals are often non-continuous, and state transitions can be abrupt. This limitation significantly constrains the potential impact of the paper.\n4. The paper, as indicated by its title and abstract, aims to approximate the solution to CCRL problems with ECRL. However, the text frequently shifts focus to extending CCRL to ECRL for generalization, which only serves to complicate the narrative and blur the paper's primary objective. It is recommended that the author refocus the discussion on the application of ECRL to CCRL for a clearer and more coherent presentation.\n5. The experimental section exclusively presents the results of the test process, which, although demonstrating positive outcomes, fails to provide any insights into the training process itself. It is crucial for the author to include at least some intermediate results from the training process to validate that the process does not incur significant overheads.\n6. The utilization of a flipping-based policy introduces broader confidence intervals in the results, a concern that may be particularly relevant in safety RL applications.\n7. The authors should ensure that at least two environments are considered in experiments for broader applicability.\n8. This paper should add a related work section to compare and contextualize the current research within the broader field.\n\nQUESTIONS:\n1. What is the cause of the sudden change behavior observed in the experimental results of the \"origin\" group in Figure 2?\n2. Figure 2 serves as a clear and intuitive example. However, in more complex scenarios involving more than two possible safety areas, how is the behavior of the flipping-based policy?\n3. The statement \"Gaussian distribution-based stochastic policies are essentially deterministic policies disturbed by noise for exploration\" may oversimplify the role and impact of Gaussian distribution-based policies. Is there some related work or more detailed explanations? While these policies (e.g. classical PPO and SAC) do indeed introduce stochasticity to facilitate exploration, they also fundamentally alter the optimization process during training. This change is not merely superficial but influences how the policy gradients are computed and how the agent learns to make decisions.\n\nLIMITATIONS:\nThe authors thoroughly discussed the limitations and potential negative societal impacts in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-23"}