A Survey of Constraint Formulationsin Safe Reinforcement Learning
Akfumi Wachi , Xun Shen , Yanan Sui
11

LY Corporation

22

Osaka University

Our Contributions
Provide a comprehensive survey focusing on constraint
formulations in safe reinforcement learning and introduce
representative algorithms for each formulation.
Discuss the relationships between various constraint
formulations by defining three theoretical notions:transformability, generalizability, and conservativeapproximation.
Present theoretical results demonstrating that two problems
exist, termed Identical or More General Safe RL (IoMG-SafeRL)
problems, into which other common problems can be eithertransformed or conservatively approximated.
Bridge the gaps between the safe RL problems with appropriate
algorithms by organizing existing research focusing on
constraint formulation.

33Tshinghua University

Theoretical Relations among Common Constraint
Formulations of Safe RL
Provide theoretical relations between each constraintrepresentation.
Problem 1 (Eπ + Cumulative)• C = h c, γc, ξ i• fC (π) = Vcπ (ρ) − ξ
Transformable
Transformable
Problem 2 (Eπ + Cumulative)• C = h Sunsafeh, γc, ξ iiPH h• fC (π) = Eπh=0 γc I(sh ∈ Sunsafe ) − ξ

Transformable

Problem 4 (Eπ + Instantaneous)• C = h c, {ξh}Hh=0 i • fC (π) := maxh∈[H] Eπ [ c(sh, ah) ] − ξhConservative Approximation
Problem 3 (Joint Chance Constraint)• C = h Sunsafeh, ξ ii
WH
• fC (π) := Pπh=0 sh ∈ Sunsafe − ξConservative Approximation

Figure 2. Relations among common safe RL formulations based on Eπ and theone with chance constraints.

Safe Reinforcement Learning
Safe reinforcement learning (RL) is a promising paradigm forapplying RL algorithms to real-world applications.
Safe RL is typically modeled as constrained Markov decision
processes (CMDPs).
M ∪ C := h S, A, P, H, r, γr , ρ i ∪ C,|{z}Standard MDP (M)where S := {s} is a state space, A := {a} is an action space, andP : S × A → ∆(S) is the state transition, where P(s0 | s, a) is theprobability of transition from state s to state s0 when action a istaken. H ∈ Z+ is the (fixed) finite length of each episode,r : S × A → [0, 1] is the reward function, γr ∈ [0, 1) is the discountfactor for the reward, and ρ ∈ ∆(S) is the initial state distribution.
Given a policy π ∈ Π, the value function is defined asH
X0πhVr (ρ) := Eπ 
γr r(sh0, ah0) sh = s  .

Problem 5 (Pπ + Cumulative)• C = h c, γc, ξ i hi
PH h
• fC (π) = 1 − Pπh=0 γc c(sh , ah ) ≤ ξ

Problem 6 (Pπ + Instantaneous)• C = h c, ξ i Q• fC (π) = 1 − Hh=0 Pπ [ c(sh , ah ) ≤ ξ ]Transformable

Problem 7 (Pπ + Instantaneous)• C = h c, {ξh}Hh=0QHi• fC (π) := 1 − h=0 Pπ [ c(sh, ah) ≤ ξh ]Transformable

Figure 3. Relations among common safe RL formulations based on Pπ (i.e.,almost-surely constraints).

Review on Diverse Constraint Formulations
and Representative Existing Papers
List of representative papers and algorithms associated with eachconstraint formulation of safe RL.

h0=hSince the initial state s0 is sampled from ρ, we slightly abuse thenotation and define ππVr (ρ) := Es∼ρ Vr,0(s) .
Given a constraint tuple C, a policy must be within the feasible
policy spaceb := {π ∈ Π | fC (π) ≤ 0}.
ΠThe optimal policy π ? is defined as
πarg max Vr (ρ)π∈Π

subject to fC (π) ≤ 0.

Due to the diversity of safety constraint representations and littlediscussion on their interrelations, it is not easy to understand safe
RL research systematically.
Step 1: Problem Formulation
πmax Vr (ρ)π∈Π

subject to [Safety Constraint]

Typical RL objective
Diverse Safety Constraint representations
Step 2: Policy Optimization
Either use an existing algorithm suitable for theproblem setup or develop a new algorithm
Figure 1. A typical sequence of safe RL based on constrained criteria.

IJCAI 2024

Poster Session

Figure 4. Common safe RL formulations based on the constrained criterion andassociated representative work. Y = Yes, T = Training, C = After Convergence, and A= Authors’ Implementation. Please see the paper for more details.

akifumi.wachi@lycorp.co.jp