Safety Representations for Safer Policy Learning
Authors: Kaustubh Mani
Vincent Mai
Charlie Gauthier
Annie S Chen
Samer B. Nashed
Liam Paull
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning
Safe Exploration
Representation Learning
Primary area: reinforcement learning
TL;DR: This paper proposes learning state-centric safety representations as an inductive bias to mitigate conservativeness in policy learning for safety-critical applications in Reinforcement Learning.

Abstract: Reinforcement learning algorithms typically necessitate extensive exploration of the state space to find optimal policies. However, in safety-critical applications, the risks associated with such exploration can lead to catastrophic consequences. Existing safe exploration methods attempt to mitigate this by imposing constraints, which often result in overly conservative behaviours and inefficient learning. Heavy penalties for early constraint violations can trap agents in local optima, deterring exploration of risky yet high-reward regions of the state space. To address this, we introduce a method that explicitly learns state-conditioned safety representations. By augmenting the state features with these safety representations, our approach naturally encourages safer exploration without being excessively cautious, resulting in more efficient and safer policy learning in safety-critical scenarios. Empirical evaluations across diverse environments show that our method significantly improves task performance while reducing constraint violations during training, underscoring its effectiveness in balancing exploration with safety.

Published as a conference paper at ICLR 2025
SAFETY REPRESENTATIONS FOR SAFER POLICY LEARNING
Kaustubh Mani', Vincent Mai?, Charlie Gauthier's Annie Chen', Samer Nashed Liam Paull
Mila, Université de Montréal, Quebec, Canada
2 Hydro-Québe Research Center, Quebec, Canada
3 Standford University
4 Canada CIFAR AI Chair
kaustubb mani @mi quebec
ABSTRACT
Reinforcement learning algorithms typically necessitate extensive exploration of the state space find optimal policies. However, in safety-critic applications, the risks associated with such exploration can lead to catastrophic consequences. Existing safe exploration methods attempt to mitigate this by imposing constraints, which often result in overly conservative behaviours and inefficient learning. Heavy penalties for early constraint violations can trap agents in local optima, deterring exploration of risky yet high-reward regions of the state space. To address this, we introduce a method that explicitly learns catte-oonditoonel safety represen- tations. By augmenting the state features with these safety representations, approach naturally encourages safer exploration without being excessively cautious, resulting in more efficient and safer policy learning in safety-critical scenarios. Em- pirical evaluations across diverse environments show that our method significantly improves task performance while reducing constraint violations during training, underscoring its effectiveness in balancing exploration with safety.
INTRODUCTION
Reinforcement learning (RL) has achieved notable success across various domains, from game playing (Shao al. 2019 roboties (Zhang Moj 202T (Singh al. 2022). However, training RL agents directly in safety-critical environments remains challenging partly due to the presence of failure events that are deemed undesirable or unacceptable both during training and deployment. To manage these failures, RL algorithms typically rely on failure penalties or impose safety constraints (Xie) et al. 2022 Leike et al. 2017; [Achiam et al.) 2017] Stooke et al. 2020). While these methods help minimize unsafe behaviours during learning, they often result in overly cautious policies, restricting the agent's ability to explore leading to sub-optimal performance. The key challenge, therefore, lies in designing RI. algorithms that can effectively balance the risks of exploration with the reward of task completion.
One significant factor contributing conservative behaviour in RL agents is "primacy bias" Niikishin et al. 2022), where early experiences exert a lasting influence on the agent's learning trajectory. In safety-critical applications, severe penalties for constraint violations encountered early in the training process can disproportionately shape the agent's 's policy, leading to overly cautious decision-making and hindered exploration. As a result, agents learn safety representations that overestimate the risk of failure, discouraging further exploration. This often results in agents with a narrow view of the state space and locally optimal yet overly conservative policies that sacrifice performance for safety. While considerable body of research has explored some notion of safety estimation for safer policy learning (Tang et al. 2019 Chow et al. 20188) Greenberg et al.) 2022). most approaches focus on failure prevention by implicitly or explicitly restricting agent exploration. For example, methods such as Bharadhwaj et al. (2020) and Srinivasan et al. (2020) filter out actions with the likelihood of failure above specific threshold. In this work, demonstrate that by incorporating accurate safety representations into the learning process, agents can make more informed decisions, balancing exploration with safety.

---

Published as a conference paper at ICLR 2025
Specifically, we present Safery Representations for Policy Learning (SRPL), a framework that aug- ments an RL agent's ss state representation by integrating a state-conditioned safety representation derived from the agent's experiences during the learning process. SRPL is grounded in the under- standing that risk is often unevenly distributed among states. For instance, driving in the wrong lane two-lane road is inherently unsafe, independent of the specific policy the agent follows. The safety representation is captured by a steps-to-cost (S2C) model, which for any given state estimates the distribution over the proximity unsafe or cost-inducing states. Additionally, by learning safery representations that are state-centric utilizing data from the agent's 's experience (as opposed to just the current policy), we encourage the generalizability of the safety representations across tasks.
To summarize, we study three primary hypotheses addressing the key challenges outlined:
By directly integrating safety information into the state representation, the safety and efficiency of RI agents during the learning process significantly enhanced.
Safety representations be efficiently learned online using the experiences generated during RL training, resulting in improved performance without requiring prior or additional data.
When learned from an agent's entire experience that includes diversity of policies, safety representations can be generalized across various tasks, acting as an effective prior for learning new tasks.
The SRPL framework be used to augment any RL algorithm and show results for several on-policy and off-policy baselines in S We evaluate SRPL agents on several simulated robotic tasks, including manipulation, navigation. and locomotion. Our results show that by leveraging safety information, SRPL agents are significantly more samplc-efficien while being safer during learning compared to baselines. Additionally, safety information transfers well across tasks, providing useful prior for learning new policies.
PRELIMINARIES
Markov decision processes (MDPs) are defined tuple (S, A,T, R,d,)), where S is set of states, A is set of actions, R S x A x + IR is the reward function indicating the immediate reward for executing action a in state and resulting in state s', T S x A x S + [0, 1] is the forward dynamics model indicating the probability of transitioning to state s' after executing action in state S + [0, 1] represents the probability of starting in a state s E and y [0, 1) is the discount factor. The solution to an MDP is an optimal policy n² that maximizes the expected discounted cumulative reward, JTT::
H
( = Bx))d(s)
E
y R(s ar |
(1)
Here, H is the length of the horizon.
Constrained MDPs (CMDPs) defined a tuple (5, A, d, 7, C,C, where (S, A, R, d, is an MDP, C S + 1) is a cost function, and 8 is the constraint threshold or a maximum, cumulative cost that is acceptable in expectation. Intuitively, we can view the constraints imposed by C and B as reducing the set of possible policies from all policies II to those satisfying the constraints Te C II. The solution a CMDP a policy T² where
n" argmax relle J(x), = (T II: Jc(T) $ 3).
H
where Je(T)) = E Bxord(e) E
v'C(s)]
(2)
To find an optimal policy, online RL algorithms allow an agent to explore the environment while simultaneously using these trajectory rollouts optimize the policy. In deep reinforcement learning (DRL), it is typically impossible to guarantee that an agent will never execute policy æ & Ie during training. Therefore, in the absence of any prior information, the agent will likely violate
2

---

Published as a conference paper at ICLR 2025
DON
40
sabey
Saaee Viltaaion Count
8000
Onlow
sodess
10-
20-
200
Return
3
6000
-20
2000
2000
Start
-00
1000
Goal
200
403 Episodes
600
800
200
400
600
800
State (Sorted Visitatio))
Episodes
USNN with Gi Salvty
USON with GT Salety
255
200
50
State Vatation Count
200-433
Fallures
150
30-
400
300 5333
Goal stote
100
-0-
200
100
400
600
State (Serzed by Visiationn
Epicodes
Episodes
applica- tions, we perform experiments on the Island Navigation euvironment Leike et al. (2017). We assume access to the Manhattan distance from the nearest water cell as ground truth (GT) safcty information. (Col 1) shows that without this information, penalties duc to failure early in the learning process bias the agent toward overly conservative behaviour resulting in suboptimal policies that avoid water but fail to complete the task. (Col 2) compares Q-value estimates a DQN agent with and without GT safety information across states over multiple episodes. Without safety information, the agent fails to distinguish between risky and less risky states, producing uniformly low Q-values across all states. This highlights the inability of RL agents to learn good safety representations using reward signals alone. (Col 3) examines state visitation patterns over episodes, showing that while both agents initially explore the environment, reduces exploration, oscillating between two states to avoid failure but failing to reach the goal state while the agent with safety information is able to explore a larger region of the state-space. More detailed discussion in Sec. 3.1
Figure 1: motivate the benefit of learning state-conditioned safety representations in
agent without safety information quickly
the constraints during exploration. Given this, often care about both the cumulative constraint violations incurred during training as well as the expected constraint violations of the final policy.
In the remainder of this paper, will refer to states that we aim to avoid, such sink states in MDPs and cost-inducing states in CMDPs, as "unsafe" states. We' ve also used "distance to unsafe" and "steps to unsafe" interchangeably. We define a failure as event where constraint is violated, and we will use the terms "failure" and "constraint violation" interchangeably. Furthermore, following common practices in safe RL literature (Achiam et al. 2017, Stooke al.) 2020)] (Sootla et al.) 2022). we assume that unsafe states can be identified either through the termination of an episode or via the cost signal.
3 SAFETY REPRESENTATIONS FOR POLICY LEARNING
In this section, we will begin by motivating the usefulness of safety information in a toy example where we assume that this information is somehow provided. Subsequently, we will formalize our choice of safety representation and describe the SRPL framework.
3.1 MOTIVATING EXAMPLE
To demonstrate the usefulness of state-centric safety representations, we perform experiments on Island Navigation (Leike et al. 2017), a grid world environment designed for evaluating safe exploration approaches. The agent's 's goal is to navigate the island without entering the water cells. Entering a water cell is considered unsafe and leads to a failure penalty in the form of a negative reward and episode termination. The agent is only rewarded when it visits the goal state. The input to the RL agent is the image of the entire grid. Instead of experimenting with a single environment with fixed start and goal state (Leike al. 2017) where the agent can simply memorize action
3

---

Published as a conference paper at ICLR 2025
iistance-to-unsall 8(s)
NLL Grad
Trajectory Buffer
g
G
RL
training RL training
(+)
action
π
Agent
s
action
Environment
8, r
Figure 2: SPRI. Framework: SPRI. explicctyy learns safety representations for states distribution over proximity unsafe states (cost-inducing states) through steps-to-cost (S2C) model and uses this information implicitly guide policy learning towards exploring safer regions of the state space.
sequences, we create four different versions (Fig. [1 (in the appendix)) of the Island Navigation environment with different start and goal positions as well as locations of the water tiles. Each episode begins with randomly selecting an environment. Thus the agent needs to learn good state-conditioned representations of safety to solve the task as well minimize failures during learning.
In this environment, a reasonable proxy for safety associated with every cell is Manhattan distance to the nearest water cell. To investigate how this priviliged safety information can be useful to the policy, we provide the RL agent with this distance (referred to as GT safety) by adding it to the state representation. We train DQN(Mmih et al. 2015) agents with and without this safety information, along with variants of DQN that model aleatoric (c51 Bellemare et al. (2017) and epistemic (BootstrapDQN OObaand
Osband et al. (2016) uncertainty.
Fig. (col 2) shows the evolution of the Q-value distributions for all state-action pairs for a particular instance of the environment during training, for DQN agents with and without safety information. In the presence of safety information, the DQN agent is able to efficiently learn a correlation between safety information and reward, outputting low Q-values for risky states (GT safety 1) and high Q-values for safe states (GT safety = 3), as a result efficiently exploring the state-space and ultimately converging to an optimal policy. On the other hand, the DQN agent without this information fails to learn accurate internal representations of safety and instead overestimates risk for all states resulting in uniformly low Q-values (Fig. (col 2) (row 1)), further discouraging exploration and as a result converging to a sub-optimal policy (Fig. (col 1) (row 1)). Fig. [ (col 3) (row 1) shows that the policy learned by the DQN agent in the absence of safety information ensures safety of the agent by oscillating between two or three states near the start state"but but fails to consistently reach the goal
state resulting in low average return (Fig. [col I)).
This example demonstrates that having access additional information about the safety of a state can be useful for RL agents to overcome the bias created by negative experiences early in learning that result in an overestimation of risk leading to low Q-values and, as result, yield conservative policies with sub-optimal performance. However, typically this information is not provided to a learning agent and must be inferred from the agent's observation of the environment. We propose explicitly learning safety representations using agent experience during learning.
3.2 STATE-CENTRR SAFETY REPRESENTATIONS
We propose to learn state-conditioned safety representations as an inductive bias to enable safety- informed agents by leveraging the agent's prior experience. This raises two key questions: (1) What constitutes an ideal state-centric representation of safety? (2) How should we train such a representation?
An effective safety representation must capture the immediate likelihood of failure in the current state as well as reflect potential risks in future states. This representation should incorporate both the risks associated with exploration from given state, the uncertainties in the environment's 's dynamics, and the ambiguities in policy choices for action selection following the current state. A simple scalar safety representation (Bharadhwaj et al., 2020, (Srinivasan et al. 2020), lacks the expressiveness necessary to capture these complexitics. Therefore, we propose modelling safety as probability
'Episodes terminate when the agent reaches the goal state enters water cell. However, oscillating between two states causes episodes truncate the max-steps limit of 100, resulting in high state-visitation counts
4

---

Published as a conference paper at ICLR 2025
distribution over distances to cost-inducing states. To avoid learning representations that overfit data from a narrow region of the state space induced by a particular policy, we propose to learn safety representations over the entirety of the agent's 's experience instead of solicy-specifie rollouts.
Formally, we model safety as func- tion 6 S - A AH,, where S is the state space and A HH represents probability simplex over the safety horizon 3 Given the experience of an agent, the model (s)s) aims to capture the conditional probability of entering an unsafe state in exactly t (1,2...., H&) steps given that the agent in state (Fig. 3) Here, GH,(* (s) represents the probability that the agent remained safe throughout the safety horizon H, without encoun- tering an unsafe state based on the agent's past experience. This model can be learned from the set of trajectories which represent the experience of the agent, as we will describe next.
A
Agent
Goal
Water
25
A
141
Figure 3: Safety Representation: We demonstrate the S2C model's s oulput two different states (A & B3.. A being farther from the water cells (indicated blue) has its peak at (distance from the unsafe set) while B is more risky state and has its peak at 1.
3.3 SRPL FRAMEWORK
To learn the state-centric safety representation, we train neural network, referred to as the "steps- to-cost" or the S2C model, which takes the state as input and outputs the corresponding safety representation. This representation is modelled as discrete distribution, implemented the softmax output of a neural network G" parameterized by v.
The safety representation is learned alongside the RL policy by constructing separate re- play buffer Dszc in the case of on-policy algorithms, which contains trajectories т = (so, (80)))...... (Sp; (s,)) from policy rollouts. In the case of off-policy algorithms like CSC (Bharadhwaj et al. 2020) and CVPO et al. (2022), the off-policy buffer is used to store the "steps-to-cost" information corresponding states for each trajectory. At the end of each episode, every state s in the trajectory t is labelled with its corresponding "steps-to-cost" value 8 (s). which the number of actions taken before encountering an unsafe state. If unsafe state is encountered during the episode, the distance to unsafe for all states in the trajectory is set to the safety horizon length (H,) indicate safety of the state within the safety horizon. In this way, for every trajectory T E DD2cc and for every state
we have a label 8 (s) which we can use to train our S2C model by minimizing the following negative log-likelihood loss
H,
(szc((s,T);n) = Σ 118 = log(((((
(3)
where I[5(s) = t] is an indicator function that takes value 1 if (x() = and 0 otherwise.
In the SRPL framework, (Fig. 2). the output of the S2C model (s) is incorporated into the agent's state by augmenting the original state with the learned safety representation. The augmented state is defined s' {s, (s)) For high-dimensional observations, such as raw images, the safety representation is concatenated with the encoded feature representation of the observation, such that the augmented observation becomes o' (F(o), (o))), where F is the feature encoder.
Implementation details: We model the safety distribution over fixed safety horizon H, << H. relying on the assumption that information about near-term safety is more important and an agent can safely navigate the state space with this information. Instead of modelling the distribution over all time steps between [1, H.). we split this range into bins to further reduce the dimensionality of the safety representation. An ablation over the choices of bin size and safety horizon H, is included in the Appendix I For on-policy algorithms, a separate off-policy replay buffer is maintained which
HH << H, where H is the MDP time horizon. For categorical distributions, NLL loss is equivalent Cross-Entropp loss.
5

---

Published as a conference paper at ICLR 2025
TIPD-HD
SP.PPG
AdtallHaandhnn
Dafe
PoineBation1
PaintGoltt
2007
0.-
0.4
3--
5000
0--
3007
Stece
167
307
123
120
100
30
0.
0.4 0.6
3.0
1.3
Stecs 107
50g0s 266
Stepe
Stepa
Figure 4: Performance of SRPL agents (denoted SR-#) four different tasks. For these experiments. both the S2C model and the policy have been randomly initialized no prior information has been provided to the agent. SRPL agents consistently outperform their baseline counterparts on both safety during learning as well as sample efficiency. Results were obtained by averaging the training runs across five seeds. The input to the RL agent is state-based in the form of joint states or LiDAR points.
stores policy rollouts along with the distance to unsafe values. To preserve only relevant experiences about policies similar to the agent's 's current policy, the replay buffer throws away samples from older policies. More thorough implementation details are provided in Appendix A..
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We perform our experiments on four tasks three distinct environments. First is a manipulation task AdroitHandPen (Rajeswaran et al. 2017) where a 24-degree of freedom Shadow Hand agent needs to learn to manipulate a pen from a start orientation to a randomly sampled goal orientation. Dropping the pen from the hand is considered a failure constraint violation. Next, have an autonomous driving environment Safv (Li et al. 2022), where an RI. agent is learning to drive the road while avoiding traffic, each collision incurs cost and the cost threshold is set to 1 (3 = 1). Finally, we evaluate our method on the Safety Gym (Ray et al., 2019) environment on tasks PointGoall and PoinButton]. For the PointGoall task we have a point agent that is tasked with reaching a random goal position from a random start position, the environment consists of regions that are unsafe and accumulate cost. For the PointButton/ task, the agent needs to press a sequence of buttons in the correct order, pressing the wrong button incurs a cost. In addition, there are static hazards and dynamic objects that the agent needs to avoid. The goal in these environments is to do the task while incurring costs lower than a threshold (6 = 10). Additionally, we also show results on Mujoco locomotion environments (Ant, Hopper and Walker2d) in the Appendix. The goal in these tasks is for the robot to move fast possible while avoiding falling the ground which is considered a failure or a constraint violation.
Further, evaluate the transferability of learned safety representations from one task to another. For this, we use PointButtonI as the source task and PointGoall as the target task. We augment the state representation of the Safety Gym environments ensure the same state dimensionality for PointGoall and PointButton/ (More details in Appendix A.4). The S2C model is trained on the source task and frozen for the target task and we study the performance of the S2C model at transferring information in terms of safety and efficiency on the target task. Additionally, we show that the S2C model provides a good initialization for fine-tuning representations on the target task in the case of transferring both policy and safety representations.
4.2 BASELINES & COMPARISON
The SRPL framework can be integrated with any RL algorithm that accounts for risk sensitivity or operates in safety-critical environments. We evaluate its effect on feature learning by comparing several baseline algorithms with their SRPL-augmented versions. We compare against both off off-po and on-policy safe RL algorithms. On-policy baselines include Constrained Policy Optimization (CPO) (Achiam et al.; 2017). a second-order method for enforcing constraints, TRPO-PID (Stooke 2020) L.agrangian-bar based method that uses PID controller for stable learning, SauteRI. (Sootla

---

Published as a conference paper at ICLR 2025
AdrvitHandPPs
DufeMe
Polntioall
PolntButtee
Methods
Retum (7)
Pfulloros (j)
Siccess Rate
Tocal (oo
Return (t)
Cost-Rate (+le") (
Retuna (T)
Cont-Rote (+1)2)
CPO
4154 1758
6667 + 965
0.088 0.1
2715 = 366
6.26 + 3.28
1.61+-0.12 0.12
0.91 + 0.70
1.64 1:4-+-0..00
TRPO-PID
4809 2541
6629 445
0.076 + 0.37
2471 1335
2.50 2.61
1.09 0.02
0.15+ 0.73
1.13+0.0
SanteRI
3651 1783
9:00 + 2557
0.08 + 0.00
10170 = 1196
3.63 1.75
1.08 0.02
-1.45 0.56
1.09 110-+:00 0.000
CRPO
7803 + 2576
5702 1146
0:22 0.24
2500 - 732
7.49 1.00
1.08 0.13
1.47 0.68
+.08
SR-CPU
7176 1: 1392
4797 991
0.47 0.34
1997
11:03 1.28
1.49 a: 0,00
2.11 1: 1.16
1
DR-TROO-PIDD
5963 1584
5311 1. 741
0.12 1. 0.29
2001 1014
7.31 1 2.85
1.07 0.02
1,38 1 1.19
1,08 0.00
SR. SauteLL
6316 1634
0.115 0.12
8082
1.25
1.05 0.00
-1.01
=:
0.01
SR-CRPO
8800 1. 988
4626 1 447
0.53 L. 0.18
2689 569
10.40 1. 4.64
1.51 0.02
3,12
1. 0.99
1.53...000
Table I: Performance of the RL policies at the end of the training. SRPL versions of the baseline algorithms (denoted by SR-*) consistendly reduce the number of failures or cost-rate while significantly improving return. Training details for SRPL and baselines are provided in Appendix AA.2 The input the RL agent is state-based in the form of joint states LiDAR points.
ct al. 2022) a state-augmentatiinn method that stores the remaining constraint budget as part of the state information, CRPO (Xu et al. (202T) primal approach which updates the policy by alternating between reward maximization and constraint satisfaction. Off-policy baselines include Conservative Safety Crities (CSC) (Bharadhwaj et al.] 2020) which learns safety critic using safe Bellman operator and Constraint Variational Policy Optimization (CVPO) (Liu et al. (022)) which formulates the constrained MDP problem an Expectation Maximization (EM) problem.
Evaluation Metrics. For AdroitHandPen environment, we evaluate SRPI. and baselines on episodic return (sample efficiency) and total failures/costs incurred during the training of the RL agent (safety). For the SafeMetaDrive environment, we evaluate the algorithms on task performance and safety in terms of the success rate and total cost, respectively. For the Safety Gym environments, we evaluate the algorithms on the task performance and safety in terms of episodic return and episodic cost, respectively. Additionally, we measure the task performance and safety at the end of training for the Safety Gym environments in terms of average return and cost rate respectively. Cost-rate is measured by dividing the total cost incurred during training by the number of actions/steps taken in the environment.
5 RESULTS
5.1 LEARNING SAFETY REPRESENTATIONS ALONGSIDE POLICY
We study the performance of SRPL. agents when jointly training the S2C model and policy (Fig. 4 In this setting, the S2C model does not contain prior information and thus must learn state-conditioned safety representations using rollouts generated by the agent during training.
Table T shows the results for on-policy safe RL algorithms along with their SRPL counterparts. Learning safety representations alongside the policy greatly enhances the sample efficiency of the baseline algorithms on all tasks, while enabling faster constraint satisfaction or fewer failures or costs during training. From Fig. 4 we can see that SauteRL. (Sootla et al. 2022) performs the best in terms of constraint satisfaction in the case of Safety Gym (Ray et al. 2019) environments where the constraint threshold is greater than (3 > 1) and fails to do the same for tasks like AdroitHandPen and SafeMetaDrive. We hypothesize that this limitation arises from how SauteRL encodes the remaining safety budget into the state. In these environments, the safety budget remains O until a constraint violation occurs, rendering it ineffective for accurately representing safety throughout the task.
Fig. shows the results for off off-policy baselines and their SRPL counterparts Safety Gym environments PointGoal! and PointButton/ / over 2M timesteps") Fl By filtering out actions that might cause a constraint violation through the use of safety critic, CSC (Bharadhwaj et al.) 2020) enables better constraint satisfaction but results in conservative or suboptimal task performance. On the other hand, CVPO (Liu al. 2022) is significantly more sample efficient than CSC as well as the on-policy counterparts. SRPL significantly improves the sample efficiency of both CVPO and CSC baselines reaching similar or higher performance in comparison to SR-CVPO and SR-CPO in just 2M steps. 5.2 RISK-REWARD TRADEOFF
In CMDPs, safety and task performance can be treated as separate objectives, framing the problem as a multi-objective optimization task. Depending on the safety-critical nature of a system, different levels
Ooff-policy methods are generally more sample efficient and require fewer training samples
7

---

Published as a conference paper at ICLR 2025
CVPO SR. CVPO
CSC SR CSC
cost-linit
Safety Transfer
Poky Worsler
PointButton1
PointGoall
10.0
10.0
10.0
Deturn
7.5
5.0
Retum
2-
2.5
5.0
0.0
2.5
1n7
0.5
1.5
Episodic Cost
15
100
1eb
Epocdic
10
Eponte Cost
3.5
Stegs
20-
Figure 6: Transferring the Safety Representations: Transferring the policy as well as learned safety rep- resentations from PointButton] to PointGoall, Zero- shot transfer of the (frozen) safety representation leads to significant improvement
0.0
0.5
1.0
0.0
1.5
2.0
1e6
166
Figure 5: Off-policy results for CSC and CVPO and their SRPL. counterparts on Safety Gym environments over 2.M timesteps. While CSC has better constraint satisfaction also leads to suboptimal performance, CVPO has better sample efficiency which is further improved by SRPL.
terms sample effi- ciency over vanilla CPO with or without policy trans- for. Fine-tuning the safety representations on Point- Goal/ further improves the performance while leading to better constraint satisfaction for policy transfer.
of tolerance for constraint violations during learning can be set, effectively prioritizing one objective over the other. Emphasizing safety during the learning process tends to discourage exploration, which can subsequently reduce task performance, a phenomenon often referred to the risk-reward seeety-perfommacce tradeoff. To investigate the effect that learning safety representations has on this tradeoff, we conduct experiments comparing the risk-reward tradeoff capabilities of SPRL with baseline algorithms.
In algorithms that approach the CMDP problem through a Lagrangian formulation (CSC in Fig. 7 prioritizing safety corresponds to increasing the value of the initial Lagrange multiplier. A higher Lagrange multiplier value reduces exploration and enhances safety during learning. In algorithms like CPO (Achiam et al. (017) that enforce exact constraint satisfaction at each step, we adjust the constraint threshold to modulate this tradeoff. Our experiments are conducted in two distinct environments: AdroitHandPen and Ant. In Fig. 7 cach point represents the policy's 's total cost incurred during learning (x-axis) and its final performance (y-axis), measured as the average return, for various safety requirement settings (either Lagrange multiplier ellipses represent the variance across both the x and y directions. As we increase the priority of safety while learning, the number of failures reduces along with the task performance for all RL agents.
constraint threshold). The
Figure elearly illustrates that SPRL improves baseline algorithms' ability to balance task perfor- mance and safety during the learning process. This indicates that the SPRI. framework facilitates safer learning for a given level of task performance or enhances task performance for a desired level of safety. Additionally, expected, the figure shows that safety information becomes increasingly valuable as the safety-criticality of the objective rises.
5.3 SRPL AS AN EFFECTIVE PRIOR
As deseribed in Sec. 4.1 to study the generalizability of the learned safety representations across tasks, we use PointButon] as the source task and PoinGooal as the target task. We present two sets of results 1) where we transfer the safety representations but not the policy (Fig. (]left),, training policy from scratch the target task) and 2) where we transfer both the policy and the safety representations to the target task. Additionally, for both these cases, we study the effect of freezing the S2C model (i.e. safety representations) well finetuning the safety representations
the target task.
From Fig. 6 (left), we see that transferring safety representations without finetuning them on the target task (SR-CPO (safety transfer) (frozen)) improves the sample efficiency as well enables
8

---

Published as a conference paper at ICLR 2025
AdroitHandPen
Ant
8000
7000
2000
6000
failures
6000
5I-C00
Return
batum
Iatum
4000
4000
4000
2000
2000
3000
2000
2000
2000
0.5
1.0
0.5
1.0
Steps
1e7
Steps
le7
1000
SR SP-CPO
fallures
Figure 8: Alternative Safety Representations: An anal- ysis of the effect of different choices in modelling safety representations on overall performance terms of safety and efficiency of the algorithm.
Figure 7: Risk-Reward Tradeoff: SR-CPO is able to better tradeoff risk and reward in comparison vanilla CPO thanks learned safety representations.
faster constraint satisfaction on the target task. However, training the safety representations directly on the target task from scratch (SR-CPO (no transfer)) leads to better performance at convergence, due the fact that the source and the target environments are not identical in their distribution of (safety
cost- bost-inducing states. We further see that fine-tuning the safety representations transfer) (finetuned)) achieves similar performance at convergence and faster constraint satisfaction in comparison to learning representations from scratch. Fig. (rightt shows the results when transferring the policy from source to target task. We see that CPO with policy transfer is able to be more sample efficient compared to CPO trained from scratch. Transferring both the learned safety representations and the policy keeping the safety representations frozen (SR-CPO (policy transfer) (frozen)) leads to significant improvement in sample efficiency over CPO (policy transfer). Additionally, finetuning the safety representations on the target task (SR-CPO (policy transfer) (finetuned) leads to even more sample-efficient agents well as better constraint satisfaction.
6
ABLATIONS & ANALYSIS
6.1 ALTERNATIVE SAFETY REPRESENTATIONS
The choice of modelling the safety representation depends on the properties we desire our safety representations to encode. For the safety representation to be state-centric we need it to be trained in a policy-agnostic way (i.e. unlike the value function the safety representation encodes the likelihood of failure from data collected from diverse set of policies experienced during learning). In this section, we study some the alternate choices of safety representations:
(v1) modelling safety representation as the expected likelihood of entering unsafe/cost- inducing state for the current policy (Bharadhwaj et al. 2020).
(v2) policy-dependent safety representation distribution over proximity to unsafe states trained using on-policy rollouts,
(v3) learning the safety representation, as proposed here, from the past experience of the agent, across diverse set of policies.
As shown in Fig. 8 SRPL outperforms all other safety representation models. The safety representa- tion learned in (vI) does not improve the performance of the RL agent, as this information is already captured by the cost-critic in CPO (Achiam al., 2017). Although the policy-dependen safety repre- sentation (v2) enhances the base algorithm's 's performance, it does not surpass SRPL. We attribute this to two key factors: (1) SRPL utilizes data from previous policies, allowing the safety representation to encode information about broader region of the state space, while the policy-dependent safety representation confined to on-policy rollouts, a common limitation on-policy versus off off-policy reinforcement learning (Haamoja et al.) 22118); (2) by learning stateccentric features, SRPL promotes more stable dynamics during RL training, in contrast to the policy- depended representations, which fluctuates with policy updates.

---

Published as a conference paper at ICLR 2025
6.2 EFFECT OF SAFETY REPRESENTATIONS FOR HIGH-DIMENSIONAL OBSERVATIONS
To investigate the effects of learning low-dimensional state-centric safety representations from high- dimensional observations, we conducted experiments in the Safety-Gym PointGoal| environment Ray et al. (211)). We examined how different sensor modalities impact task performance and safety during learning. As shown in Table 2 increasing the dimensionality of sensor observations, from LiDAR to depth to RGB, leads to a decline in both safety and task performance. This degradation likely due to the increased complexity involved in learning effective representations from higher- emphasizes that as the dimensionality of observations rises,
dimensional data. Additionally,
learning safety representations as an inductive bias becomes increasingly critical for ensuring safe
and efficient policy learning.
Sensur
Kemore (*)
Cour-Kate (1)
Medality
CPO
SR.CPO
(5)
CPO
SR.
LINAR
6.
11.05 228
-77.0
0.00
19.07
7 RELATED WORK
Dopth
3.78 + 2-41
9
0.48
1.66
RGB
8.13 4.06
-199
20
0.334
1.7 0.374
-0.01
Representation Learning for RL: RL algorithms often need learn effec- tive policies based on observations of the environment, rather than having direct access to the true state. These observations can come from sensors like RGB cameras, LiDAR, or depth sensors. Learning rep- resentations that capture the essential aspects of the environment or task can significantly enhance efficiency and performance. To achieve this, various methods employ auxiliary rewards or alternative
Table 2: Safety representation learning improves agent performance (return) and reduces constraint violations (cost-rate) in higher- dimensional observation spaces, where representation learning is typically more challenging.
training signals (Sutton et al. |001 Jaderberg et al. 2016 Riedmiller et al. 2018 Lin et al. 2019
Bhatia et al.] 2023). p. One effective approach is learning to predict future latent states, which has proven valuable in both model- free (Munk et al. [0166] Schwarzer et al. 2020 Ota et al., 2020) and
model-based (Watter et al. 2013) Ha & Schmidhuber (2018) settings. In this paper, we 'e focused on learning representations for condiiioiiio safety that can enable more informed decision-making in safety-critical applications.
Safe Exploration: Safe exploration (Achiam et al. 2017 Liu et al. 2022 Sootla et al. 2022 Stooke
let al. 2000 Jiang et al. 22023] Gu et 2024) approaches need to contend with both the alcatoric uncertainty of the environment and the epistemic uncertainty associated with the exploration of These methods commonly achieve this by restricting exploration to parts of the state space with low epistemic uncertainty. Bayesian model-based methods (Berkenkamp) 2019), represent uncertainty within the model via Gaussian processes, favouring exploration in states with low uncertainty. (Huang et al. (023) incorporate lagrangian- methods into world models. [Fisac et al. (2019), [Srinivasan et al. (2020). Kang et al. (2022) and [Bharadhwaj et al. (2020) use safe Bellman operator (called the safety critic) to evaluate the risk of failure from a given state taking a particular action and use it to restrict exploration by filtering out actions with high risk of failure or formulating constraints according to the safety critic. (Sootla et al.) 2022 Jiang et al., (023) use accumulated cost as a proxy for risk associated with a state and use it augment the state space.
unseen parts of the
8 DISCUSSION & CONCLUSION
While SRPL offerss a valuable approach for improving safety representations in RL. agents, it comes with certain limitations, which, though typical of many RL methods, are still worth noting. First, SRPL has difficulty capturing long-horizon causal mechanisms related to safety. For example, a state the beginning of a long single-way track leading to an unsafe state is actually very risky for the agent but would be represented by the safety representation as low risk or safe, as the number of actions separating the current and final, unsafe state is large. While challenging, such examples are not typical for most embodied or otherwise high-risk agents. Second, our chosen definition of safety has practical limits. We do not consider degrees of safety, and in general defining safety, harm, danger, or any other bad outcome typically involves substantial nuance in real-world settings which most safe RL methods, including SRPL, struggle to capture completely.
In summary, this paper addresses the problem of reinforcement learning (RL) agents becoming overly conservative as a result of penalties due to safety violations early in training in safety-critica environments, leading to suboptimal policies. To tackle this, we propose framework that learns state- specific safety representations from the agent's experiences. By integrating this safety information into the state representation, our approach enables more informed and balanced dacision-making.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
The authors thank Yann Pequignot and Adriana KnatchbullHHugessen for insightful discussions and useful suggestions on the early draft. This work was supported by the DEEL Project funded by the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Consortium for Research and Innovation Aerospace in Québec (CRIAQ).
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference Machine Learning, pp. 22-31. PMLR, 2017.
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement
learning. International Conference on Machine Learning, pp. 44944449444 458. PMIR, 2017.
Felix Berkenkamp. Safe exploration in reinforcement learning: Theory and applications in robolics. PhD thesis, ETH Zurich, 2019.
Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety critics for exploration. arXiv preprint wrXiv:2010.144999 2020.
Abhinav Bhatia, Samer B Nashed, and Shlomo Zilberstein. RL.: Boosting meta reinforcement learning via RL inside RL?, arXiv preprint arXiv:2306.15909 2023.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18 (167):1-51, 2018.
Jaime Fisac, Neil F Lugovoy, Vicenç Rubics-Royo, Shromona Ghosh, and Claire J Tomlin, Bridging hamilton-Jacobi safety analysis and reinforcement learning. In 2019 International Conference Robotics and Automation (ICRA), pp. 8550-8556. IEEE, 2019.
Ido Greenberg, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Efficient risk-averse reinforcement learning. Advances in Neural Information Processing Systems, 35:3:ms,3::3339911 35:32639 2022.
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theories and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-18776 PMLR, 2018.
Weidong Huang, Jiaming Ji, Borong Zhang, Chunhe Xia, and Yaodong Yang. Safe dreamerv3: Safe reinforcement learning with world models. arXiv preprint arXiv:2307.07176. 2023.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv: 101.1....599777 2016.
Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. OmniSafe: An infrastructure for accelerating safe reinforcement learning research. Journal of Machine Learning Research, 25(285):1-6, 2024. URL http::/ httpa//.......aaaa///////2222//
html
Hao Jiang, Tien Mai, Pradeep Varakantham, and Minh Huy Hoang. Solving richly constrained reinforcement learning through state augmentation and reward penalties. arXiv preprint 1rXiv:2301111592 2023.
11

---

Published as a conference paper at ICLR 2025
Katie Kang, Paula Gradu, Jason J Choi, Michael Janner, Claire Tomlin, and Sergey Levine. Lya- punoy density models: Constraining distribution shift in learning-bast based control. In International Conference Machine Learning, pp. 10708-10733 PMLR, 2022.
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. Al safety gridworlds. arXiv preprint 1rXiv:/71.00950 2017.
Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on paltern analysis and machine intelligence, 45(3):3461-3475 2022.
Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. Advances in neural information processing systems, 32, 2019.
Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pp. 13644 13668. PMLR, 2022.
Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang. Jie Tan, and Ding Zhao. Datasets and benchmarks for offline safe reinforcement learning. Journal of Data- Data-centric Machine Learning Research, 2024.
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518(7540):529 503, 2015.
Jelle Munk, Jens Kober, and Robert Babuška. Learning state representation for deep actor-critic control. In 2016 IEEE 55th Conference Decision and Control (CDC), pp. 4667-4673. IEEE, 2016.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp. 16828-16847. PMLR, 2022.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. Advances in Neural Information Processing Systems, 29, 2016.
Kei Ota. Tomoaki Oiki, Devesh Jha, Toshisada Mariyama, and Daniel Nikovski. Can increasing input dimensionality improve deep reinforcement learning? International conference on machine learning, pp. 7424-7433. PMLR, 2020.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: /709, 10087, 2017.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforce- ment Learning. 2019.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International conference on machine learning, pp. 4354-4353. PMLR, 2018
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self relf-predictive representations. arXiv preprint arXiv:2007.05999, 2020.
Kun Shao, Zhentao Tang, Yuanheng Zhu, Nannan Li, and Dongbin Zhao. A survey of deep reinforcement learning in video games. arXiv preprint arXiv: 1912. /0944, 2019.
12

---

Published as a conference paper at ICLR 2025
Bharat Singh, Rajesh Kumar, and Vinay Pratap Singh. Reinforcement learning in robotic applications: comprehensive survey. Artificial Intelligence Review, pp. 1-46, 2022.
Aivar Sootla, Alexander I Nowen-Rivers Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté RL: Almost surely safe reinforcement learning using state augmentation. In International Conference on Machine Learning, pp. 20423-20443, PMLR, 2022.
Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha. Jie Tan, and Chelsea Finn. Learning to be safe: Deep RL with safety critic. arXiv preprint arXiv:2010.1400 14603, 2020.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by PID Lagrangian methods. In International Conference Machine Learning, pp. 9133-9144 PMLR, 2020.
Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsuper- vised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Pestems-Volunne 2, pp. 761-768, 2011.
Yichuan Charlie Tang. Jian Zhang, and Ruslan Salakhutdinov. Worst cases policy gradients. arXiv preprint arXiv:1911.0300 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In 2012 IEEF/RSI international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: locally linear latent dynamics model for control from raw images. Advances in neural information processing systems, 28, 2015.
Annie Xie, Fahim Tajwar, Archit Sharma, and Chelsea Finn. When to ask for help: Proactive interventions in autonomous reinforcement learning. Advances in Neural Information Processing Systems, 35:16918 1-9309 2022.
Tengyu Xu, Yingbin Liang, and Guanghui Lan. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In International Conference Machine Learning, pp. 11480-1149) 1491. PMLR, 2021.
Tengteng Zhang and Hongwei Mo. Reinforcement learning for robot research: A comprehensive review and open issues. International Journal of Advanced Robotic Systems, 18, 2021.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX
DQN
SR-DQN
DON with reset
DQN W/ GT risk
BootstrapDQN
40
400
20
failures
300
Return
0
200
#
-20
100
-40
0
200
400
600
800
200
400
600
800
Episodes
Episodes
Figure 9: DQN with resets: Resetting the DQN agent to overcome conservatism results in significantly more failures during training.
cost-limit 0.1
cost-limit
0.05
cost-limit 0.01
5000
cost-limit 0.1
6000
cost-limit 0.05
4000
cost-limit 0.01
5000
failures
Return
3000
4000
3000
2000
2000
1000
1000
0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Steps
1e7
Steps
1e7
Figure Conservatism As the cost-limit is reduced CPO agent struggles overcome induced by initial experiences where the agent encountered heavy penalties due to constraint violation, resulting overly conservative behaviours.
A.1 PRIMACY BIAS IN SAFE RI.
RL agents overfitting on early experience is a well-studied problem. Nikishin et al. (2022) first showed that outcomes of early experience can have long-lasting effects on subsequent learning of RL agents. Machado al. (2018) identified poor data diversity caused by limited exploration as the primary reason for overfitting. In safety-critical systems, penalties imposed on RI agents due to constraint violations early in the training can further disincentivize exploration limiting the data diversity further. As result RL agent tends to learn from samples from narrow region of the state space resulting in conservative behaviors and suboptimal performance. Nikishin et al. (2022) proposed reset the agent encourage exploration and avoid primacy bias but for safety-critical applications resetting the RL agent can lead to catastrophic consequences. As shown in Fig. 9 DQN with resets lead to significantly more failures than vanilla DON, while not resulting in significant
14

---

Published as a conference paper at ICLR 2025
improvement in performance. In this paper, we ve proposed to deal with this problem by learning state-centric representations of safety that force the agent to learn safety representations from limited data and overcome conservative behaviour.
To illustrate the extent of overfitting on negative experiences experienced by the agent early in training, we study the impact of penalties caused by constraint violation on the behaviour of the agent. We experiment on AdroitHandPen with CPO (Achiam et al.) 2017) for different cost thresholds. A lower cost limit in the case of CPO is essentially equivalent to penalizing failures during learning more and more. From Fig. We We see that as we reduce the cost limit the number of failures reduces but also the resultant policy becomes more and more conservative, ultimately for cost limit 0.01, the agent fails to learn anything about the task and settles for a local optima in which it doesn't 't move.
A.2 IMPLEMENTATION DETAILS
In this section, we aim to clarify some of the design choices and implementation details used for training SRPL agents for on-policy and off off-policy baselines. Our implementations were based on top of FSRL Liu et al. (224) and Omnisafe Ji et al. 2024) codehases.
A.2.1 LEARNING THE SAFETY REPRESENTATION
The safety representation aims to characterize a state with respect to its proximity to the unsafe states and models distribution over steps-to-cost based on the agent's 's past experience. Empirically, in the case of discrete state-spaces like Island Navigation, for a given state safety representation this is equivalent to maintaining the normalized frequencies over proximity unsafe states based on the agent's 's past experience or policy rollouts in the past.
Here face dilemma about how to learn such representation. Should we on-policy rollouts or should we use off-policy data? There is an inherent tradeoff in this choice: if we train the safety representation using on-policy rollouts ensure coverage we would need collect lots of samples which will make the algorithm highly sample inefficient and also unsafe. This is particularly infcasible in the case of off-policy baselines like CVPO (Liu al., 2022) and CSC (Bharadhwaj al. 2020), where the policy is updated at every step. On the other hand, if we learn the safety representations completely using all of the agent's past experience, at some point the representation will become irrelevant to the current policy because it is storing information about policies that might be too different from the current policy. To manage this tradeoff, we learn the safety representation using off-policy data from the agent's 's past but recent experience, we maintain a separate replay buffer in case of on- on-policy algorithms or reuse the existing replay buffer of off-policy algorithms to store steps-to-coss 6, (s) for state encountered in particular trajectory We keep fixed replay bulfer size and experiences that are generated by the older policies are discarded when the replay buffer gets full in First In First Out (FIFO) manner. This allows us ensure that the safety representation is learned with enough data to ensure coverage over the state space while also keeping the representation relevant for the current policy.
A.3 TRAINING THE S2C MODEL
Because the output of the S2C model is given to condition policy learning, there are some consid- erations while training the S2C model to stabilize the training of the SRPL agent. In the case of on-policy baselines, since the policy is frozen while collecting on-policy rollouts, the dynamics of learning the safety representation alongside the policy is more stable and we update the safety representation (i.c., the S2C) model at higher frequency than the policy. On the other hand, doing this in case of off-policy algorithms like CSC Bharadhwajj et al. (2020) and CVPO et al. (2022) really destabilizes training because the policy is being updated at very high frequency making the dynamics really complex. To solve this, we train the safety representations at significantly lower frequency than the policy. This ensures that the S2C model is frozen while the policy being updated, The idea is similar to the use of target networks proposed in Mnih et al. (2015) to stabilize training.
A.3.1 PRACTICAL CONSIDERATTIONS
There are some practical considerations in order to show results on environments like AdroitHandPen and Mujoco (Ani, Hopper, Walker2d). Training these tasks with cost-limit of 0 leads to overly
15

---

Published as a conference paper at ICLR 2025
A
Agent
Goal
Water
Figure 11: Island Navigation: Instead of experimenting with single environment with fixed start and goal state where the agent can simply memorize action sequences. We create four different versions of the Island Navigation environment with different and goal positions well locations of the water tiles.
conservative policies that learn nothing about the task as demonstrated in Fig. and Fig. [0 for cost-limit of 0, the # failures is very low but also the performance degrades. To address this issue we went with the cost limit of 0.1 for our experiments for both SRPL and vanilla versions of the baseline algorithms.
A.3.2 HYPERPARAMETER DETAILS
As described earlier we chose bin size of 4 and safety horizon H, 80 for PointGoull and Poim/Button/ / environments and bin size of and safety horizon H, 40 for all the other envi- ronments. The batch size for training the S2C model was chosen between 512 and 5000 and found that batch size of 5000 led better performance in Safety Gym environments and 512 for all the other environments. Additionally, we optimize for hyperparameters like when to update the S2C model update- freq which was set to 100 for on-policy baselines and 20000 for off-policy baselines. Additionally, learning rate of le 3 for off-policy baselines. The well-optimized hyperparameters of the baseline algorithms be found in open-source implementations such Omnisafe Ji et al. (2024 (for CPO. CRPO, TRPO-PID, SauteRL) and FSRL Liu et al. 2024 (for CVPO). For CSC Bharadhwajj (020) we used the hyperparameters specified the original paper. We faced difficulty in training on-policy baseline algorithms on SafeMetaDrive with default hyperparameters from Om- nisafe, fine-tuning hyperparameters revealed that lower target-kl kl stabilizes the baseline algorithms. torget kl 0.0005. Additionally, AdroitlanndPen and Ani tasks with the default hyperparameters. We couldn't 't stabilize the training with limited hyperparameter search. The S2C model has the same network architecture as the policy which in most cases is an MLP with two hidden layers of size 64.
used learning rate of le or le 5 for on-policy experiments and
faced difficulty in stabilizing CVPO Liu et al. (2022) on
A.4 ENVIRONMENTAL DETAILS
Island Navigation Leike al. 2017) grid-world environment explicitly designed for evaluating safe exploration algorithms. The environment consists of an agent marked A, trying navigate an island to reach a goal G while avoiding water cells (marked in blue). Entering water cell is considered failure and the agent receives a heavy reward penalty along with episode termination. The state observation consists of the image of the island at given time (as such the state is completely observable). The action space is ["left", "right", "up", ; " "doon"", so there no action to stay in place. Because its a
deterministic environment with small distance between the start state and goal state, its not difficult for RI. agents to just remember the action sequences instead of learning an accurate cate-conniiionee value function. In order to avoid such situation, instead created copies of the same environment with different positions of start state and goal state as well the location of the water cells (Fig.
16

---

Published as a conference paper at ICLR 2025
AdroitHandPen (Rajeswaran et al. 2017) is manipulation environment, where 24 24-degree of freedom Shadow Hand agent is learning to manipulate a pen from a randomly initialized start orientation to randomly specified goal orientation. The input to the RL agent is the angular positions of the finger joints, the pose of the palm of the hand, as well as the current pose of the pen and the target pose of the pen. The action space is continuous and are the absolute angular position of the hand joints (24-dimensional), The pen falling on the ground is considered failure and induces cost. Since the cost-inducing state is also non-ergodic (irrecoverable), maximum cost for an episode cannot be greater than 1. The agent is provided dense reward based on the similarity of the current pose to the target pose of the pen.
SafeMetaDrive (Li et al. 2022) is an autonomous driving environment, where an agent is tasked with navigating the traffic containing static and dynamic participants. The agent has to frequently change lanes or apply brakes to avoid incurring costs. Any collision induces cost and the cost-limit is set to 1. The observation space consists of LiDAR information about the surrounding objects, ego-vehicle's pose, and safety indicators along with route points. The agent is rewarded for following the waypoints/route as well as for keeping the lane and for driving at high velocity. The action space consists of steering, brake and throttle values.
Safety- (Ray et al. 2019) is a navigation environment, we evaluate SRPL on PointGoall and PointButton/ tasks. In PointGoal, a randomly initialized point agent trying to navigate to randomly sampled goal position while avoiding hazards in the environment which are the cost-inducing states, so that the total cost for an episode doesn't 't exceed 10. In PointButtonI, the agent is tasked with pressing the specific orange button, pressing the wrong button induces a cost and there are hazards as well as dynamic obstacles in the environment which are cost- cost-inducing. Cost threshold is set to 10. In the original version of safety gym, the agent's 's observation consists of LiDAR information about every object including hazards, buttons and goal position individually. This prevents us from doing the generalization experiment since the dimensionality of the observation for PointGoal| and PointButtonl I are different because they have different objects in the environment. To address this we aggregate the Lidar information for all objects into one LiDAR observation that for every LiDAR point provides the distance to the nearest object as well as the corresponding object label. The action space for the point agent is the two- -wo-dimensional (force applied to move the point agent and the velocity about the z-axis). The reward function for the agent is based on the distance to the goal state.
; We evaluate SRPL on three locomotion tasks (Ant, Hopper and Walker2d). The goal in all three tasks is to run as fast possible while not falling on the ground. Falling on the ground is considered a failure and induces a cost. We treat the agent fallen on the ground as a non-ergodic state and thus the maximum cumulative cost for an episode can be 1. The agent's state input is the joint position and velocity of the robot's body parts as the centre of mass-based external forces acting on the body. The action space is the torques applied to each joints. The agent's 's reward is proportional to the velocity of the agent, and the agent is penalized for applying high torque on its joints.
Mujoco environments [Todorov et al. 2012):
A.5 ADDITIONAL RESULTS
In addition to the results presented in the main paper, we perform experiments on Mujoco environ- ments (Ant, Hopper and Walker2d) and show results for CPO (Achiam al. (017) and CRPO (Xul et al. 202T) along with their SRPL counterparts. From Fig. 12 we can see for the An/ environment, CRPO significantly outperforms CPO in terms of task performance (i.e. return) but also leads to more failures during training. SR-CRPO has similar performance in terms of return but significantly lower total failures during training thus making the algorithm safer. On the other hand, SR-CPO leads to an improvement in the performance in terms of return in comparison to CPO, while incurring similar but fewer failures. Similarly, for Hopper SR-CRPO outperforms all other baselines in terms of task performance as well as leads to the least total failures during training. SR-CPO also shows improvement over CPO in terms of safety by incurring fewer failures during learning. Finally, on Walker2d, SR-CPO outperforms all the baselines in terms of task performance while incurring the same number of failures as SR-CRPO We observe a consistent improvement in either the safety or efficiency of the SRPL versions of the algorithms in comparison to their vanilla counterparts.
17

---

Published as a conference paper at ICLR 2025
CPO
CRPO
SR-CPO
SR.CAPO
Hopper
4000
Return
Return
2000
0
-2000
500
500
0.4
Steps
Steps
Steps
1600
failures
3500
fallures
1400
3000
1200
3000
1000
2000
2000
Steps
Steps
Steps
1e6
Figure We report the performance safety-informed SRPL agents (denoted SR-") on Mujoco environments (Ant, Hopper and Walker2d). For these experiments, both the S2C model and the policy have been randomly initialized so no prior information has been provided to the agent. Safety- informeef agents consistently outperform their vanilla counterparts both safety during learning well as sample efficiency. Results were obtained by averaging the training runs across five seeds. The input the RI. agent the joint state and velacity.
0.55
3050
0.50
3025
3000
3 2975
2050
0.35
2925
0.30
0.90
2900
0.25
2873
1000
200
24
20
40
24
20
Bin Size
Bin Size
Figure 14: Choice of safety horizon H, Results MetaDrive show that higher safety horizon H, leads to improvement in performance but above threshold the curve plateaus.
Figure 13:
Choice of bin size: Results on
SafeMetaDrive show that lowering the bin size better the performance of the model.
To provide more clarity into the results presented in the paper, we have added Fig. 16 which highlights the episodic cost for AdroitHandPen and SafeMetaDrive environments as well as the respective cost-limits. In our experiments we observed that setting the cost-limit to 0 for AdroitHandPen or the Mujoco environments was leading to the policy failing to learn anything, so we set the cost-limit to 0.1 (3 0.1) for AdroitHandPen as well Mujoco experiments (Fig. 12). For clarity we've also added pairwise plots for baseline algorithms with their SRPL versions on AdroitHandPen Fig. 17 Safe Fig. 18 PointGoal! Fig. 19 and PointButton Fig. 20
A.6 ADDITIONAL ABLATIONS
To further analyze the safety representations proposed in the paper, perform additional ablations that study the effect of design choices like safety horizon (H,) ) and bin size for learned distribution. We also study the generalization ability of the leamned safety representations across constraint thresholds.
A.6.1 EFFECT OF SAFETY HORIZON AND BIN SIZE
In Fig. 13 and 14 we study the effect of different choices of bin size and safety horizon the performance of the policy both terms of safety (in terms total cost) and efficiency (in terms of success rate) for SafeMetaDrive environment. In Fig. 14 we analyze the effect of varying safety horizons for a fixed bin size of bin size = 2. We can see that both success rate and total cost almost plateau post the bin size of around 50, the dimensionality of the safety representation can be
18

---

Published as a conference paper at ICLR 2025
CPO (policy transfer)
SR.CPO (policy transfer) (frozen)
cost-limit
cost-limit
12.5
7.5
10
5-
0.0
0
0.0
0.5
2.0
1.0 Stops
1.5
0.0
Stops
Stops
20-
15
10
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Stops
165
Stops
1e6
Stops
1e5
Figure Generalization across Cost thresholds: Since safety representations are learned to model the proximity to unsafe states, they can be generalized across cost thresholds. Here we show that safety representation learnt on PointButton] with fost-limit- = 10 can be transferred to PointGoall task for cost-limits other than 10 without the need for fine-tuning and results in improved sample efficiency.
the cause behind the slight deterioration in performance beyond H, 50. Fig. studies the effect of bin sizes given a fixed safety horizon H, = 40, from the figure clear that smaller bin sizes lead to better performance both in terms of safety and efficiency of the SRPL agents.
A.6.2 GENERALIZATION OF SAFETY REPRESENTATION ACROSS COST THRESHOLDS
Instead of modelling the likelihood of constraint violation, safety representation encodes the distribu- tion over steps to cost or distance to cost-inducing/unsafe states. This design choice is based on the principle that depend on the task definition (for e.g., the cost threshold definition). We perform experiments on PointGoall environment for the transferability of safety representation across constraint thresholds. For this experiment, we mimic the structure of the generalization experiments shown in Sec. [.3 We treat PointButton/ the source task and PointGoal/ as the target task where the safety representation as well as policy for both CPO and SR-CPO is trained on PointButton/ task. In order to study the generalizability of safety representations across cross thresholds we don't fine-tune the safety representation the target task for different cost thresholds thus freezing the S2C model. From Fig. [5] we can see that SR-CPO (policy transfer) (frozen safety) significantly outperforms CPO (policy transfer), thus transferring the policy and frozen safety representation across tasks enables SRPL agents to learn more samples efficiently while ensuring constraint satisfaction for different constraint thresholds.
interested in modelling state-centric features that are generalizable and don't 't
19

---

Published as a conference paper at ICLR 2025
CPO
TRPO RD
CPO
ERPO
SR CRPO
SouteAL
cast lim
SafeMetaOOive
3.-
0000
0000
2000
0.3-
2.6
0.8
3.0
0.2
04
1.6
0.8
1.0
0.2
24
0.5
08
3.0
0.
0.4
60
2.0
Dptock Cosk
2.6
22
342
Figure 16: The figure shows Episodic Cost Episodic Failure for AdroitHandPen and SafeMetaDrrvv experiments
8000
SP.CFO
TAFO-RIC
8000
SA-CROO
6000
5000
3000
6000
Return
Return
4000
Return
Return
4000
4000
2000
2000
2000
1000
1000
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
Steps
Steps
Steps
Steps
dro
SAUTRO
DRPO
0.8
0.8
0.8
Episodic Cost
Episodic Cost
Episodic Cost
Episodic Cost
0.6
0.6
0.4
0.4
0.2
0.2
0
0
Steps
le6
Steps
le6
Steps
1e6
Steps
1e6
7000
7000
Sn.cro
TRP2-1C
TRRO
6000
CRAO
st.corc
10000
Sautelt Sh.Sll
6000
Failures
5000
# Failures
5000
5000
4000
6000
3000
3000
3000
4000
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5 Steps
1.0
Steps
Steps
Steps
Figure AdroitHondPen: We plot all the baselines and their SRPL counterparts
20

---

Published as a conference paper at ICLR 2025
0.20
CRD
CARO
SALEHRI
0.20
0.15
Retur
Retur
Retur
0.10
0.10
0.05
0.05
0.00
0.5
1.0
1.0
Steps
le6
Steps
1e6
Steps
1e6
Steps
1e6
DPO CRO
TIP0-100 SAITRPO A3
CARD
SauteRt
SR CAPO
SauteR,
coal-lest
cool-and:
Episodic Cost
Episodic Cost
Episadic Cost
6-
Episodic Cost
3-
4-
2-
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Steps
1e6
Steps
1e6
Steps
1e6
Steps
1e6
crg
mio-10
cure
Sa-left
3000
3500
10000
Total Cost
Total Cost
3000
Total Cost
2500
8000
2000
2000
7000
1500
1500
6000
1000
1000
1000
5000
0.5 Steps
1.0
0.0
1.0
1.0
1e6
Steps
1e6
Steps
1e6
Steps
1e6
Figure 18: SafeMetaDrive: We plot all the baselines their SRPL counterparts
12.5
58.000
TERO
SI-CEPO
22 Sectet
10.0
Return
7.5
Return
Return
Return
5.0
2
2.5
0-
0-
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Steps
1e7
Steps
1e7
Steps
1e7
Steps
1e7
80
CPO
SE HD
125
caro
SauiAA Stideelatt.
Episodic Cost
Episodic Cost
60
Episodic Cost
100
Episodic Cost
60-
80
60
40
50
20
20
25
20
10a
10a
107
10a
10a
107
10a
10a
107
10a
10f
107
Steps
Steps
Steps
Steps
Figure 19; PointGoall; We plot the baselines and their SRPL counterparts
21

---

Published as a conference paper at ICLR 2025
0.0
Sh-Rro-nD TAPD-P0
D310
GU-PP0
18-13R9
GB Gi-Couteli
-0.5
1
Return
Returm
Returm
-1.0
-1.5
-2.0
-2
0.0
0.5
1,0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Steps
1e7
Steps
1e7
Steps
1e7
Steps
100
125
sa.cro
maro-np
100
5n-CYro
St-Souler)
contting
Episodic Cost
conttimnt
cost Iint
con-tim
Episodic
Episodic
60
Episodic
75
20
10'
10"
10"
10
10
10
10a
10"
107
Steps
Steps
Steps
Steps
Figure 20: PointButton]: plot baselines their SRPL counterparts
22


Reviews:
[{"review_id": "xRkJeViS2v", "reviewer": "Reviewer_K4qc", "summary": "This paper introduces the RIPL framework, which integrates state-conditioned risk representations into the agent’s state features. These risk representations help guide exploration more efficiently and reduce the risk of constraint violations without being overly conservative. Empirical results demonstrate that RIPL improves both task performance and safety.", "questions": "1. How sensitive does RIPL perform as we set different bin sizes for risk? How do we determine this value?\n2. I think transfering the risk model between tasks is an interesting point. Currently, the authors freeze the risk model on the target task. Would it also be possible (and more reasonable) to fine-tune it in the target environment?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of state-conditioned risk representations is intuitive and reasonable, and is a novel approach in the community of safety RL.\n2. The method is sound and shows significant improvements in multiple environments, and the transferability of the risk models are also tested.\n3. The structure of the paper is clear and the writing is easy to follow.", "weaknesses": "1. The related works in the recent two years seem to be missing, and the latest baselines that the author compared with are CSC and TRPO-PID, which were published in 2020. Are there other more recent work in the safty RL literature? If time is not enough for experimental comparison, that's fine (though preferred); It would be nice to mention the most recent advances and the advantages of the proposed method compared with them. (I am not an expert in safety RL, but would love to see clarifications on whether there is/is not more recent works)\n2. As is discussed in the limitation, the true relation between states' risks might not be genuinely captured (long-horizon dependency etc.). I agree that it is an important aspect to investigate about in future work, but how well can the current risk model reflect the risk? For example, consider providing a visualization of the output distribution of the risk model w.r.t. the constrained states. It'd be nice to have illustrations in the continuous case, similar to Fig.3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dBz7SdQN6T", "reviewer": "Reviewer_maKe", "summary": "The main idea of the paper is to augment the state space with a distance to “unsafe” regions and use this augmented state-space to learn safe RL policy. The paper presents a number of experiments and comparisons to safe RL methods as well as an ablation study showcasing different aspects of the algorithm.", "questions": "1. The paper \n   > Jiang, H., Mai, T., Varakantham, P., & Hoang, M. H. (2023). Solving Richly Constrained Reinforcement Learning through State Augmentation and Reward Penalties. arXiv preprint arXiv:2301.11592.\n\nis another example of state-space augmentation for safety.\n\n2. The authors use a time-dependent risk measure. However, we have a different risk measure - accumulated costs. We more or less know that when the accumulated costs reach a certain level then we are likely to be unsafe soon. Why not use the costs as distance? Is there a benefit of using this specific risk measure?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is very well-written\n2. The motivational example is great! I would recommend getting the message a bit more fleshed out. That is, that adding a distance measure in this example with sparse rewards made a huge difference for training. The paper tries to do the same trick in safe RL, where we often treat safety similarly to RL in the sparse reward setting - either safe or not safe.\n3. The paper has a great ablation study", "weaknesses": "1. The CPO baseline is fine, but not amazing. I think it would be good to use at least one more baseline for demonstration. There are a few implementations in https://github.com/PKU-Alignment/safety-gymnasium\n2. Ideally, it would be good to see more risk measure distances compared to the presented one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uRzORB1lRi", "reviewer": "Reviewer_jX1B", "summary": "The paper proposes a safe RL method based on augmenting the state space with a learned safety representation, which is referred to as Risk-Informed Policy Learning. This state-conditioned representation is learned during training, and captures the probability of safety violations in each of the next several timesteps. The goal of incorporating this representation is to learn policies that satisfy safety constraints without being overly conservative in terms of task performance. Experiments across 4 different tasks evaluate task performance and safety metrics, where the learned state representation is combined with several different safe RL algorithms.", "questions": "**[Q1] Experiment clarifications**\n- Please provide the average total cost per trajectory for each experiment, and how this relates to the safety constraint budget $\\beta$.\n- Please describe the implementation of the risk model in more detail. What horizon is used? What network architecture is used? Other key details?\n- Could you please provide more detail on how Figure 7 (and the RI-CPO training from scratch in Figure 5) support the claim about transferability? I also think that RI-CPO should be included in Figure 7 to more easily compare against it without needing to look back to Figure 5. \n\n**[Q2] Choice of RL algorithms**\n- Why is CPO used as the main safe RL algorithm for most of the experiments? More recent safe RL algorithms have been shown to outperform CPO, such as CVPO [3] and LAMBDA [4]. \n- The approach in [1] also seems relevant to this work, since it incorporates information about safety constraint violation through state augmentation (this work considers a more detailed, learned state augmentation while [1] uses the remaining budget).\n\n**Minor:**\n- cost function notation is inconsistent: $c$ vs. $C$ vs. $\\mathcal{C}$\n- an index variable $i$ appears in (2) but is not used anywhere\n- incorrect column / row references in lines 191-192\n- floating quotation mark in the top left of Figure 3\n- colors are not defined in Figures 3 and 9\n- incorrect, hardcoded equations throughout Appendix\n---\n\n**References:**\n\n[3] Liu et al. Constrained Variational Policy Optimization for Safe Reinforcement Learning. In ICML 2022.\n\n[4] As et al. Constrained Policy Optimization via Bayesian World Models. In ICLR 2022.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **[S1] Important topic:** Safe / Constrained RL is an important topic, and training policies that satisfy safety constraints without being overly conservative is an important goal.\n- **[S2] Compatible with many RL algorithms:** The proposed method augments the state with a learned representation, so it can be applied to many different RL algorithms. This is also demonstrated in the experiments, where it is applied to 4 different safe RL algorithms.\n- **[S3] Experiments across diverse set of tasks and several base algorithms:** Experiments consider manipulation, locomotion, and navigation tasks, and the proposed state representation is combined with 4 different safe RL algorithms. Additional ablation analyses are also included, which consider different input modalities and alternate choices for the learned safety representation.", "weaknesses": "**[W1] Confusing use of the term “risk” throughout the paper**\n- A risk measure is a function that maps a random variable to a scalar value (e.g., CVaR). Risk-sensitive / risk-aware / risk-averse RL methods typically apply a risk measure over a distribution of future returns. This is very different from the ideas proposed in this work, where a “risk function” is mapping states to a distribution over timesteps. I strongly suggest that the authors change the term “risk-informed” to something different that better reflects what is being proposed in this paper.\n\n**[W2] The definition of the risk model is inconsistent across the paper**\n- The risk model is defined as a probability distribution over the number of steps to failure (i.e., constraint violation in (2)) in several places (lines 60-61, 172-173, 212, 256-257). I believe this formulation makes the most sense, as it captures information about constraint violation similar to the state augmentation proposed in [1]. However, in other places (lines 221-226, 461, 719-720, 791-792) the risk model is described as a probability distribution over the number of steps to an unsafe state (i.e., a state where $c=1$ but not necessarily a constraint violation). \n\n**[W3] Experimental analysis is not clearly presented**\n- The safety metrics presented in the Experiments section are difficult to interpret. I cannot tell what percentage of trajectories end in safety violations in the AdroitHandPen and Ant tasks based on the #failures metric, and I cannot tell whether the safety constraint is satisfied in PointGoal1 and PointButton1 tasks based on the Cost-rate metric. In Figure 5, it does not look like the safety constraint of $\\beta=10$ is satisfied for these tasks. See [Q1].\n- There is not enough detail on how the safety representation is implemented in the Experiments. There might be some information in Appendix A.6, but it is difficult to understand. See [Q1].\n- I do not understand how the results in Figure 7 support the claims about transferability. Policy transfer works well both with and without the RI representation, and it does not look like transferring the risk model leads to better performance compared to just training RI-CPO from scratch as shown in Figure 5. See [Q1].\n- I do not understand some of the presentation choices in Figures 5-7: returns and costs are shown for different numbers of total timesteps; some RI algorithms are shown while others are not; some algorithms show performance throughout training while others are just horizontal lines.\n\n**[W4] Appendix is not well-written and does not present significant results**\n- The results in the Appendix rely on unusual assumptions and do not provide significant insights into the main problem considered in the paper.\n- Proposition 0.3 is not new. See Theorem 3 in [2].\n- Assumptions / proofs are often not explained in an appropriate amount of detail: the definition of $r(s)$ in Proposition 0.1 does not seem to be directly used in the proof; $\\eta$ introduced in lines 928-929 does not appear to be used, as (19) looks to just be a repeat of (18).\n- Several incorrect, hardcoded equation references. Strange numbering of propositions.\n\n---\n\n**References:**\n\n[1] Sootla et al. Sauté RL: Almost Surely Safe Reinforcement Learning Using State Augmentation. In ICML 2022.\n\n[2] Syed et al. Apprenticeship Learning Using Linear Programming. In ICML 2008.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KAQbt3jgG8", "reviewer": "Reviewer_DjQM", "summary": "In this paper, the authors propose a new reinforcement learning framework with a specific focus on \"safe exploration.\" The proposed framework, named RIPL, can be described as a data-augmentation method that introduces additional risk-related representations into the original state-space, leading to more efficient safe policy learning. The idea is intriguing, and the toy experiments illustrate the motivation behind it well. However, the paper faces some issues with clarity. Additionally, the experimental results are limited and not sufficiently convincing. I hope the authors can address the concerns I have outlined below.", "questions": "1. In this paper, the authors frequently claim that the \"RL agent overfits the negative samples.\" However, it is unclear whether this \"overfitting\" refers to the representations, Q-functions, or actions. In Column 1 of Figure 1, how does the training curve reveal this \"overfitting problem\"? Could the authors provide a more detailed explanation, along with additional evidence?\n\n2. In the experiments, are the environments image-based or state-based? Since all environments provide both versions, it would be helpful to clarify this in the figure/table captions.\n\n3. The experimental results provided are too limited to convincingly demonstrate the algorithm's effectiveness. Could the authors present more results from Mujoco (hopper, halfcheetah, walker2d), Adroit (hammer, door), and Safe Gym (Point/Car/Doggo goal/button/push 1/2)? If possible, including results from MetaDrive, which is arguably the most challenging environment for testing algorithms with safety considerations, would strengthen the case.\n\n4. Given that the focus of this work is \"Safe Exploration,\" it would be beneficial to emphasize the safety-related results during the exploration process. Could the authors offer more insights on this aspect?\n\n5. The key point for RIPL is the risk buffer, but it is not adequately discussed in the main text. How is the buffer updated? A very large number is assigned to the risk-batch-size. Does this imply that it never updates and contains all information throughout the training process? If so, the risk-distance probability estimation must be inaccurate since the dataset's policies vary significantly. The estimation should differ substantially for different policies.\n\n6. Since $R_H(s)$ is a distribution, its dimension varies significantly across environments with different horizons. What happens if the total horizon timestep is extremely large? In most cases, we consider infinite-horizon MDPs. How can RIPL generalize to that?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The idea presented in this paper is interesting, and the implementation looks simple and practical.\n2. The proposed framework, RIPL, is flexible and can easily be combined with many existing online safe RL algorithms.\n3. The toy experiments do a good job of showing the motivation behind the approach.", "weaknesses": "1. As the authors describe, $R_H(s)$ represents the probability that the agent will remain safe for the entire episode. However, this probability is highly dependent on the policy being used. In Appendix A, it is derived based on the specific policy, which raises a concern: how is the risk buffer collected and updated to train $R_H(s)$? If all new trajectories are collected without discarding old ones, the estimation will be inaccurate. On the other hand, if only on-policy trajectories are used, the approach is not safe and requires a large number of samples for convergence.\n2. A similar idea of learning risk-related value estimation has been proposed in [1], with several works following this approach to learn Safe Q via a safe-Bellman operator. This method offers more solid theoretical guarantees.\n3. The caption for Figure 3 is unclear and does not align with the figure, which could cause confusion.\n4. The baseline algorithms used in the experiments seem somewhat outdated, and the number of experimental results provided is too limited to fully demonstrate the advantages of the proposed algorithm.\n\n[1] Fisac, Jaime F., et al. \"Bridging hamilton-jacobi safety analysis and reinforcement learning.\" 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)