A Survey of Constraint Formulationsin Safe Reinforcement Learning

Akifumi Wachi, Xun Shen, Yanan Sui

1

Reinforcement Learning (RL)

ActionAgentEnvironment
AlphaGo(Google DeepMind)

PolicyGran Turismo(Sony AI)

StateReward

RLHF (OpenAI)2

Safety Issues in RL
Gap
Research on RL

AlphaGo(Google DeepMind)

Gran Turismo(Sony AI)

Unsafe actions → No problem ☺

Real Applications

MedicalApplications

Autonomous Driving

Unsafe actions → Catastrophic results 

Safe RL is needed!!
3

Safe RL in This Talk
• Safe RL is a broad topic by definition.
• Garcia and Fernández (2015) classified optimization criteria into 4 groups:
1. Constrained criterion
2. Worst-case criterion
3. Risk-sensitive criterion
4. Others (e.g., r-squared, value-at-risk)• This talk focuses on safe RL based on the constrained criterion.
Garcıa and Fernández. "A comprehensive survey on safe reinforcement learning." JMLR 16.1 (2015): 1437-1480.
4Safe RL with Constrained Criterion

subject toSafety Constraint

Typical RL objective(Expected cumulative reward)

5

Potential Applications of Safe RL

Industrial Robot

MedicalAutonomous Driving© NASAChatbot

Space Exploration
6

Diverse Required Safety Levels

MediumRequired Safety LevelHigh7

Diverse Constraint Formulations

subject toSafety Constraint

Typical RL objective
Diverse constraint formulations depending on
the applications or required safety levels

Expectation or Almost surely
Cumulative or Instantaneous
8Common Constraint Representations
Problem Formulation 1 (Expected Cumulative Safety Constraint)
subject to
Typical RL objective

Safety Constraint

SameStructure
9Common Constraint Representations
Problem Formulation 1 (Expected Cumulative Safety Constraint)
subject to
Typical RL objectiveSafety Constraint• One of the most popular formulations.
• Many well-known algorithms are based on this formulation.
• CPO[1], {TRPO, PPO}-Lagrangian[2], RCPO[3], etc.
• Focus on the averaged performance → Relatively low required safety level[1] Achiam+. Constrained policy optimization. In ICML, 2017.
[2] Ray+. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 2019.
[3] Tessler+. Reward constrained policy optimization. In ICLR, 2019.

10

Common Constraint Representations
Problem Formulation 2 (Almost Surely Cumulative Safety Constraint)
subject to
Typical RL objective

Safety Constraint• Require the constraint satisfaction with probability of 1 (i.e., almost surely).
• ℙ𝜋 is used rather than 𝔼𝜋 .
• Higher required level of safety.
• Saute RL[1] algorithm is based on this formulation.
• Good theoretical properties + Empirical performance.
[1] Sootla+. Saute RL: Almost surely safe reinforcement learning using state augmentation. In ICML, 2022.

11

Common Constraint Representations
Problem Formulation 3 (Almost Surely Instantaneous Safety Constraint)
subject to
Typical RL objectiveSafety Constraint• Require the constraint satisfaction with probability of 1 at every time step.
• Very high required level of safety.

• Many algorithms are based on this formulation.
• SMbRL[1], RL-CBF[2], SafeMDP[3], SNO-MDP[4], etc.
[1] Berkenkamp+. Safe model-based reinforcement learning with stability guarantees. In NeurIPS, 2017.
[2] Cheng+. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In AAAI, 2019[3] Turchetta+. Safe exploration in finite Markov decision processes with Gaussian processes. In NeurIPS, 2016.
[4] Wachi and Sui. Safe reinforcement learning in constrained Markov decision processes. In ICML, 2020.

12

Typical Procedure of Safe RL
Step 1: Problem Formulation
subject toSafety Constraint• Diverse safety constraint representations

Step 2: Policy Optimization• Either use an existing algorithm suitable forthe problem setup or develop a new algorithm
13Issues of Previous Safe RL Research
Step 1: Problem Formulation
subject toSafety Constraint• Diverse safety constraint representations

Step 2: Policy Optimization• Most safe RL researches have pursued SOTA performance• Existing survey papers have focused on algorithms

14

Our Contributions
Step 1: Problem Formulation
subject toSafety Constraint• Diverse safety constraint representations• Constraint formulation is the first step in safe RL• Crucial to properly understand diverse constraint representations.
• Our paper provides comprehensive survey on Safe RL
focusing on problem formulation in safe RL.
15

List of representative algorithms
associated with each formulation

16

Theoretical relations between each constraint representation
17

Conclusion (Take Home Messages)• Safety is an important issue in RL• Diverse problem settings → Diverse constraint representations• Our paper provides• Comprehensive survey of safe RL literaturefrom the perspective of constraint formulations• Theoretical analysis on interrelations between each formulation

Thank you!!
Contact: wachi.akifumi [at] gmail.com
Paper: https://arxiv.org/abs/2402.02025
18