Safety through feedback in Constrained RL
Authors: Shashank Reddy Chirra
Pradeep Varakantham
Praveen Paruchuri
Venue: NeurIPS 2024 poster (2024)
Keywords: Constrained RL
Cost Inference
Human Feedback
Primary area: reinforcement_learning
TL;DR: Generating safe policies in constrained rl settings through cost estimation from safety feedback

Abstract: In safety-critical RL settings, the inclusion of an additional cost function is often favoured over the arduous task of modifying the reward function to ensure the agent's safe behaviour. However, designing or evaluating such a cost function can be prohibitively expensive. For instance, in the domain of self-driving, designing a cost function that encompasses all unsafe behaviours (e.g., aggressive lane changes, risky overtakes) is inherently complex, it must also consider all the actors present in the scene making it expensive to evaluate. In such scenarios, the cost function can be learned from feedback collected offline in between training rounds. This feedback can be system generated or elicited from a human observing the training process. Previous approaches have not been able to scale to complex environments and are constrained to receiving feedback at the state level which can be expensive to collect. To this end, we introduce an approach that scales to more complex domains and extends beyond state-level feedback, thus, reducing the burden on the evaluator. Inferring the cost function in such settings poses challenges, particularly in assigning credit to individual states based on trajectory-level feedback. To address this, we propose a surrogate objective that transforms the problem into a state-level supervised classification task with noisy labels, which can be solved efficiently. Additionally, it is often infeasible to collect feedback for every trajectory generated by the agent, hence, two fundamental questions arise: (1) Which trajectories should be presented to the human? and (2) How many trajectories are necessary for effective learning? To address these questions, we introduce a \textit{novelty-based sampling} mechanism that selectively involves the evaluator only when the the agent encounters a \textit{novel} trajectory, and discontinues querying once the trajectories are no longer \textit{novel}. We showcase the efficiency of our method through experimentation on several benchmark Safety Gymnasium environments and realistic self-driving scenarios. Our method demonstrates near-optimal performance, comparable to when the cost function is known, by relying solely on trajectory-level feedback across multiple domains. This highlights both the effectiveness and scalability of our approach. The code to replicate these results can be found at \href{https://github.com/shshnkreddy/RLSF}{https://github.com/shshnkreddy/RLSF}

Safety through feedback in Constrained RL
Shashank Reddy Chirra"! Pradeep Varakantham' Praveen Paruchuri 'Singapore Management University. 'IIT Hyderabad baenahannkc,radeeppiii pradeepy) Osmu. law..ed..us.........mm.m edu.sg, praveen. perrit..peiiit.com in
(shashankc
Abstract
In safety-critical RL settings, the inclusion of an additional cost function is often favoured over the arduous task modifying the reward function to ensure the agent's safe behaviour. However, designing or evaluating such cost function can prohibitively expensive. For instance, in the domain of self-driving, designing cost function that encompasses all unsafe behaviours (e.g., aggressive lane changes, risky overtakes) is inherently complex, must also consider all the actors present in the scene making expensive evaluate. In such scenarios, the cost function can be learned from feedback collected offline in between training rounds. This feedback can be system generated elicited from human observing the training process. Previous approaches have not been able to scale to complex environments and constrained receiving feedback the state level which can be expensive to collect, To this end, we introduce an approach that scales to more complex domains and extends beyond state-level feedback, thus, reducing the burden on the evaluator. Inferring the cost function in such settings poses challenges, particularly in assigning credit individual states based on trajectory- level feedback. To address this, we propose a surrogate objective that transforms the problem into a state-level supervised classification task with noisy labels, which can be solved efficiently. Additionally, it is often infeasible to collect feedback for every trajectory generated by the agent, hence, two fundamental questions arise: (1) Which trajectories should be presented to the human? and (2) How many trajectories are necessary for effective learning? To address these questions, we introduce novelty-bused -bused sampling mechanism that selectively involves the evaluator only when the the agent encounters a novel trajectory, and discontinues querying once the trajectories are longer novel. We showcase the efficiency of our method through experimentation on several benchmark Safety Gymnasium environments and realistic self-driving scenarios. Our method demonstrates near- optimal performance, comparable to when the cost function is known, by relying solely trajectory- feedback across multiple domains. This highlights both the effectiveness and scalability of our approach. The code to replicate these results can be found at httnss//giihub.com//sssskkko///
Introduction
Reinforcement Learning (RL) known suffer from the problem of reward design, especially in safety-related settings [4] In such a case, constrained RL settings have emerged as promising alternative to generate safe p 16 34 This framework introduces an additional cost function to split the task related information (rewards) from the safety related information (costs). In this paper, we address a scenario where the reward function is well-defined, while the cost function remains unknown priori and requires inference from feedback. This arises in cases where the cost function is expensive to design or evaluate, For instance, consider the development of an autonomous
"Corresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

driving system, where the reward function may be defined as the time taken to reach destination which easy to define. However, formulating the cost function presents significant challenges. Designing comprehensive cost function that effectively penalizes all potential unsafe behaviors non-trivial, and ensuring safety often involves subjective judgments, which can vary based on individual preferences. Even if one succeeds in devising such function, it must account for all environmental factors, such as neighboring vehicles and pedestrians. The evaluation of such a cost function in high-fidelity simulators can be prohibitívely expensive.
Feedback can be collected from either a human observer, who monitors the agent's 's training process and periodically provides feedback on presented trajectories, or from a system that computes the cost incurred on selected trajectories. Throughout this paper, we use the term evaluator to refer to the entity providing feedback, whether human or yystem-generate
Cost inference in Constrained RL settings has gained recent attention. One key thread of research has focused on learning cost from from constraint abiding expert demonstrations 28 32 However,
these expert demonstrations are not easily available in all settings. For example, consider robotic manipulation tasks where the human and the robot have different morphologies. This paper focuses on the second line of research in this area, where we generate trajectories and gather feedback from an evaluator to infer the underlying cost function. Prior works in this thread make limiting assumptions
on the nature of the cost function such as smoothness or linearity [28 32 5 and are limited to obtaining feedback at the state level [8] which is expensive to collect from human evaluators. We do not make such assumptions and can take feedback provided at over longer horizons, in some cases the entire trajectory.
Frameworks for learning from feedback must exhibit the following properties as emphasized in 18, [2]: 1) Feedback must be collected offfine in between rounds, since the agent may need to act in real time. 2) The amount of feedback collected must be minimized. 3) Binary feedback is more suitable as compared to numeric feedback as it is more intuitive to provide for humans. It has also been shown that humans provide less consistent feedback if is numeric [2] 4) Each state is assigned a binary cost value, indicating that it is inherently safe or unsafe, which is more intuitive for humans when assessing the safety of policies [3]
To this end we propose the Reinforcement Learning from Safety Feedback (RLSF) algorithm, which embodies all the aforementioned properties. The key contributions of this algorithm include:
Extends prior work to collect feedback over longer horizons. This is done by presenting the evaluator with the entire trajectory, breaking the trajectory into segments and eliciting feedback at the segment level. Inferring the costs directly by maximizing likelihood presents new challenges due to the problem of credit assignment over longer horizons. To tackle this, we present a surrogate loss that converts the problem from trajectory level cost inference to a supervised binary classification problem with noisy labels.
Introduces a novelty based sampling mechanism that reduces the number of queries by sampling novel trajectories for feedback.
Learns safe policies across diverse benchmark safety environments. We also show that the learnt cost function can be transferred to train an agent with different dynamics/morphology from scratch without collecting additional feedback.
2 Preliminaries
Markov Decision Process A Markov Decision Process (MDP) M is defined by the tuple A, H.), where S denotes the set of states, A is the set of actions, P(s'|s, ss)a) E [0. 1] is the transition probability, r(s, a) E R is the reward function, 7 E [0, 1] is the discount factor and ((s) A(8) the initial state distribution. A policy (-]ss) E ((A) is distribution over the set of valid actions for state s. We denote the set of all stationary policies as II. A trajectory 7= {(st, a:)} denotes the state-action pairs encountered by executing π in M. We use the short hand Tay to denote a trajectory segment, i.e, the subsequence of (st, a.) pairs encountered from timestep i to j. The expected value of a function f under π as J'( shorthand f()) to represent the discounted sum of f along the trajectory т. The occupancy measure
00 1' [(t, (s))).).. We also employ the F -
'The last two points are specific human evaluators

---

of policy is define p(s.a) E Ei=o function, p describes the frequency with which state-action pair visited by π.
(s), = (s, a)]l. where I[.] denotes the indicator
Constrained Markov Decision Process A Constrained MDP introduces a function c(s, a) E RR and cost threshold Cmaz E IR that defines the maximum cost that can be accrued by a policy, The set of feasible policies is defined as II, = { E 11: (T) S Cmaz A policy is considered to be safe w.r.t if belongs IIe.
3 Problem Definition
In this paper, we consider the constrained RL problem defined as,
π argmin J"(T
(I)
TEll.
We assume the threshold Cmaz is known, but the cost function is not known and must be inferred from feedback collected from an external evaluator. In many scenarios Cmaz typically known, representing predefined limit on acceptable costs or risks in the environment. However, crafting the cost function c(s, a) such that penalizes all unsafe behaviour can be infeasible.
We incorporate an additional constraint enforcing the cost function to be binary. i.e, c(s, a) E {0, 1]. This ensures that each state-action pair is inherently categorized either safe unsafe. We opt for this approach because is simpler for human evaluators to assign a binary safety value to state-actions when assessing policy safety, as emphasized in 31.
4 Method
In this section, we introduce Reinforcement Learning from Safety Feedback (RLSF), an on on-policy algorithm that consists of two alternating stages: 1) Data/Feedback collection and 2) Constraint inference/Policy improvement. In the first stage, data is collected via rollouts of the current policy for fixed number of trajectories. Next, subset of these trajectories is presented for feedback from evaluator, which are then stored a separate buffer. The second stage consists two parts: i) Estimation of the cost function from the feedback data and ii) Improvement of the policy using the collected trajectories and their inferred costs. We repeat stages (1) and (2) until convergence.
First, we highlight how the feedback is collected and propose method to infer the constraint function using this data. Next, we recognize the practical limitations acquiring feedback for every trajectory during training and detail our approach to sampling subset of trajectories for efficient cost learning of the cost function. Finally, we detail how the inferred cost function is used to improve the policy.
4.1 Nature of the Feedback
In the feedback process, the evaluator is first presented with the entire trajectory To:r- Afterward, the trajectory is divided into contiguous segments Tsz of length k. and feedback is collected for each segment. The segment length can be adjusted based on the complexity of the environment: in simpler environments, feedback can be gathered for the entire trajectory, whereas for environments with long horizons and sparse cost violations, shorter segments may be used. This approach simplifies the challenge of assigning credit to individual states. Similar methods have been adopted in other works that rely on human feedback [12 [8]. However, reducing segment length comes at an increased cost of obtaining feedback from the evaluator.
The evaluator is tasked with classifying a segment as unsafe if the agent encounters an unsafe state any point within the segment. This decision was made ensure consistent feedback from the evaluator. Alternative approaches-sute as marking a segment unsafe based on the number of unsafe states visited leaving the classification to the evaluator's subjective judgment are more prone to generating inconsistent feedback in the case of human evaluators 12
4.2 Inferring the Cust Function
Let P (Tug, saffe represent the feedback collected from the evaluator, where y = 1 if the segment was labelled safe and gafe 0 otherwise. We assume there exists underlying ground

---

truth cost function (ge(s, a) E [0, 1] based on which the evaluator provides feedback. Then, the probability that state is safe is defined as Pst (s a) [[ a) 0].
Now, let pe a) represent our estimate of pst praafe(s a) that we intend to estimate from the collected feedback. Then, by definition of how the feedback is collected, the probability that trajectory segment Taij is labelled as safe is given by,
p pa (TE)) = -II (s
(2)
We can infer prafe(s, (8, a) by minimizing the likelihood loss,
T,
[y cate logp pafe logpaafe (T(s)) + (1 - y g - p (Tase(
-F(rrugy y yaf nografei ( at) + y log (1
log
I poaae az))
Directly minimizing Eq is challenging as the term II paafe (s,, at) would collapse to when the segment length is long, causing unstable gradients. To address this issue, propose using surrogate loss function where we replace II prate (st, at) by Il -poofe (st, at)).
L = - -B(rras yrufe
log (1 (st, log ta az) + (1- - (1-y0//)) Σ
- a() = (s. - 1]loggeate (s,
+ pate = log (1 1a--ppaeeee p (s,
= [[[[ssa)d log bogpaate ( (s,a) + E(s.a)ad, log (1 asafe(s
(4)
where dg(s,a) a) = El El [[(s; az) = (s, a) Ny ny n 11] and do(s,a)
E(ras I = ny = o] represent the densities with which states oc- in safe and unsafe segments respectively.
The surrogate loss reformulates the objective from the segment level-where collapsing probabilities over long segments can be problematic to the state level, where this issue does not occur. Optimizing in breaking down the segments into individual states and assigning each state the label of the segment. Subsequently, states uniformly sampled at random, and the binary cross-entropy loss is minimized. Consequently, I,aur represents a binary classification problem in which one class contains noisy labels. While unsafe states are always accurately labeled since their presence necessitates an unsafe classification for the segment-a safe state may receive conflicting labels based on the status of the segment belongs to. Thus, with a sufficient number of samples, believe it becomes feasible to reliably differentiate between safe and unsafe states.
Proposition 1. The surrogate loss L is upper bound the likelihood loss m
Thus, minimizing T,sur guarantees an upper bound on the likelihood loss of the estimated cost function.
Having discussed the surrogate loss, we now examine the characteristics of its optimal solution. Proposition 2. The optimal solution to Eq 4 Ayields the estimate,
())
p sa (8, a) = ds,s,) ds(s,a)
(5)
Subsequently, we define the inferred cost function as c, (s, a) - mpp (1p (s, < 311 Employing Cx,s, a) in policy updates instead of ege introduces a bias in estimating the cost accrued by the

---

policy, that we analyse below. For this analysis, we assume that the feedback is sufficient, i.e, the density d(s, a) is greater than zero for every state, otherwise p" (s. a) is not defined.
Proposition 3. For a fixed policy T,, the bias in the estimation of the incurred costs is given by,
Enl' cx a)] - Ex Cya(s, a)] = -( [I[dd(s, a) > dg a)]]
(6)
where Pg (s, a) F. E (T((( 0z) = (s,a) n (g a) = 0]] is the occupancy measure of safe states visited by TT.
Proposition 3 iillustrate that (* a) misclassifies certain safe states as unsafe when their frequency in segments labeled unsafe exceeds that in segments labeled safe by the evaluator. We contend that this misclassification is likely to diminish with increased data collection shorter segment lengths. However, it is important to note that this misclassification is guaranteed to be zero only when the segment length is reduced to one, meaning feedback is provided at the state level.
Additionally, note that the bias is non- negative, meaning that the expected cost Ex (s. a)] acts as an upper bound on the true cost incurred by π. Therefore, ensuring that the policy does not exceed the threshold Cmaz on c, guarantees that it adheres to the threshold on Cgt.
Corollary 1. Any policy π that is safe w.r.t ca is guaranteed to be safe w.E. Cyt.
= In practice we represent Pe (s, a) using neural network with parameters 0. The resulting cost function is defined as co(s, a) & (s, < 2-
4.3 Efficient Subsampling of Trajectories
To reduce the burden on the evaluator and minimize the cost of feedback, we present a subset of the trajectories collected by the policy for feedback. The common approach is to break the problem into two parts: (1) define a schedule N Nqueries(i) (i) that determines the number of trajectories to be shown to the user at the end of each data collection round i. Subsequently, Nqueries (i) trajectories are sampled from those collected by the policy at data collection round i. While the ideal goal is to sample subset of trajectories that maximizes the expected value of information [1)] achieving
this is computationally intractable [2]. To address this challenge, various sampling methods have been employed, seeking to maximize surrogate measure of this value. Among these, uncertainty sampling stands out as the most prominent approach, wherein trajectories are sampled based on the estimator's 's uncertainty about their predictions 12 20 However, quantifying the uncertainty is challenging given the lack of calibration in neural network predictions. To address this challenge, ensemble methods are frequently employed where the disagreement among the models is used as an uncertainty measure. However, the training of n distinct neural networks can exact substantial resource costs, prompting consideration for alternative approaches.
In light of this, we introduce a new form of uncertainty sampling called novelty sampling. With novelty sampling, we gather all the novel trajectories after each round and present them the evaluator for feedback. Formally, we define a state as novel if its density in the feedback data collected so far A trajectory deemed novel if it comprises of least e novel states. This can be interpreted as ensuring that the edit distance
d(s) E. d(s, a)
well-known measure of trajectory distance 27 between the current trajectory and previously seen trajectories exceeds threshold e. We do not consider novelty for state-action pairs as we found that extending to this case adversely impacted the performance.
The central notion is that the model is prone to errors on novel states- those it has not encountered during training. This arises because the policy evolves over time, venturing into states that were not previously encountered during data collection rounds. Therefore, this sampling strategy effectively reduces the epistemic uncertainty of the model-error arising from insufficient training data-therebb making it a form of uncertainty sampling. Furthermore, this sampling method offers the advantage of implicitly establishing decreasing querying schedule as novelty of trajectories reduces over time as shown in Figure 6 in the Appendix.
We compute the density d(s) through count-based method, utilizing hashmap to track the frequency of state occurrences in trajectories presented to the evaluator. Employing SimHash [10] we discretize the state space using hash function Φ S {-1, 13", which maps locally similar states (measured by angular distance) to a binary code as:
Φ(s) sgn(Ag(s)
5

---

where g S + R is an optional prepossessing function and A is an matrix with i.id entries drawn from standard normal distribution. Also, note that n controls the granularity of the hash function, i.e, the number of states mapped to the same value. In our setting, g(s) as we observed no improvement when employing functions like autoencoders for feature extraction.
4.4 Policy Optimization
After the data collection round, where we sample trajectories {7} and their corresponding rewards {r(r)} using we estimate their costs c(r) using the inferred cost function. Our proposed method allows for the policy to updated utilizing any on-policy constrained RL. algorithm. In this study, we employ the PPO-Lagrangian algorithm 3 that combines the PPO algorithm 26 with a lagrangian
multiplier to ensure safety.
A detailed description of the proposed method can be found in Algorithm | Lines [6-17] describe the data and feedback collection stage, and Lines [18-23] describe the cost inference and policy improvement stage.
Algorithm Reinforcement Learning from Safety Feedback (RLSF)
1: Input: cost threshold Craz, segment length k novelty criterion
2: Initialize: policy Tp
3: Initialize: classifier co. learning rate Iro and feedback buffer D
4: Initialize: A E Ixa with entries sampled i.i.d from (0 1), o(.) = sgn ())( density map
d(.)
5: while not converged do
6: Collect trajectories (r), (r(T))
Data Collection
7: for each trajectory r' E {T] do
> Feedback Collection
8:
novel < True 3 e states {se} such that d(o(s.)) if novel then
9:
Show 7 the evaluator
11:
for each segment Tj:j+k-1 E T do
12:
Obtain feedback yrafe for the segment TJ -
D DU [((s, yoafe) v(s,a) J d(p(s)) + d(((s)) + Vs T-tk-1
14:
Update the densities
15:
end for end if
16:
17:
end for
18:
for each gradient step do
Update cost estimates
19:
Sample random minibatch b + ((s,a). ~ - 0 - Ira s (b)
20:
21:
end for
22:
Infer costs {co(T.) for all 7 {7) Update æ using (r(T)) and (co(T:)}-
23:
Policy Improvement
24:
end while
Experiments
We investigate the following questions in our experiments:
1. Does RLSF succeed in effectively learning safe behaviours?
2. Can the inferred cost function be transferred across agents in the same task?
3. How does the proposed novelty based sampling scheme compare with other methods used in the literature?
4. How accurate inferred cost function compared to the true cost function?
5. How can we address the overestimation bias of the inferred cost function as described in Section 4.2

---

5.1 Experiment Setup
Table I: Performance of different algorithms on the Safety Benchmarks. The first 7 environments represent the hard constraint case. The remaining environments illustrate the soft constraint case, with values in brackets indicating the cost threshold. Each algorithm run for 6 independent seeds. (orange) and (blue) indicate the best performance in the known costs and inferred costs settings, respectively. Algorithms with a cost violation (C.V) rate below 1% are deemed to have equal performance in terms of safety.
Cost Known (Best Run)
Cost Inferred (Mean Standard error)
Environment
PPOLag
SIMKC
SDM
SIM
RLSF (Ours)
Point Circle
Return
45.26
46,09
36 + 3.95
22.26 = 9.59
36,42 1: 1.78 1,9 :
C.V Rate (%)
0.4
0.43
11.43 + 0.69
35.21 10.09
Car Circle
Return
14.34
15.21
5.18 2.48
6.34 2.87
9.37 0.97
C.V Rate (%)
0.84
5.4
6.2 + 6.18
4.53
0.54 0.30
Biased Pendulum
Return
717.43
983.27
495.38
160.84
577.15
184.31
721.48 111.49
C.V Rate (%)
0.0
39.91 17.05
48.58 21.67
0
Return
22.62
21.05
86.96 10.69
2.15 8.58
16.09 1.44
Blocked Swimmer
C.V Rate (%)
3.91
0.01
92.8 1.65
13.33 12.11
0.01 001-:000 0.01
HalfChectah
Return
2786.71
2497.82 82
3031.7
336.48
257.34
147.35
2112.63
161.26
C.V Rate (%)
0.42
0.06
59.4 11 8.28
0.0 0.0
0,06 0.01
Return
1705.00
1555.25
1097.57
56.35
990.08 + 8.66
1408,71 27.3
Hopper
C.V Rate (%)
0.19
0.02
0.0 0.0
0.0 1.01100 0.0
0,29 0.02
Return
2947.25
2925.23
2195.94 + 134.21
993.38 + 17.69
2783.29 57,51
Walker2d
C.V Rate (%)
0.16
0.0
1.58 + 1.53
0.0 0.0
0,05 = 0.01
Return Cost (40.0)
26.16
26.10
1 1.8149
10.86 1: 4.1
24.65 1: 0.59
Point Goal
34.19
31.83
30.37 + 13.29
62.76 + 12.85
35.08 +: 1.08
Return
27.37
26.44
1.05 + 2.83
10.88 + 7.1
24.28 + 2.1
Car Goal
Cost (40.0)
41.67
35.41
34.71 + 9.87
33.33 11.26
41.25 + 2.27
Return
6.00
10.84
0.16 0.14
3.63 + 1.77
2.68 + 1,03 30.51 + 3.4
Point Push
Cost (35.0)
26.08
26.96
22.89 + 5.95
45.43 + 3.86
Return
3.07
2.68
-3.04 + 3.3
1.56 0.46
1.54 4: 0.51
Car Push
Cost (35.0)
20.53
20.95
23.25 + 7.78
36.55 +: 1.48
27.69 + 1.19
We evaluate RLSF on multiple continuous control benchmarks in the Safety Gymnasium environment [17] and Mujoco 300 based environments introduced in [22]. The Circle, Blocked Swimmer and Biased Pendulum environments constrain the position of the agent whereas the Half Cheetah, Hopper and Walker-2d environments constrain the velocity of the agent. The Goal and Push tasks are the most challenging as they contain static and dynamic obstacles that the agent must avoid while completing the Aal of the above environments reflect safety challenges that an agent could potentially face in real-world scenarios. Additionally, we conduct experiments using the Driver simulator introduced in (2I) which presents two scenarios that autonomous driving agent is likely to encounter the highway: lane changes and blocked paths. In this setup, the cost function is based on multiple variables, including speed, position, and distance other vehicles. Additionally, introduce a third scenario: overtaking on a two-lane highway. A detailed description the tasks can be found in the Appendix B
We split the environments into two settings: (1) hard constraint setting (Cmar = 0) where the safety of the policy is measured in terms of the cost violation (CV) rate defined as the number of cost violations divided by the length of the episode and (2) soft constraint setting where Cmea >> O. We utilize automated script that leverages the underlying cost function to simulate the feedback provided by the evaluator.
We compare the performance of our algorithm against the following baselines: Self Imitation Safe Reinforcement Learning (SIM) [66] SIM is state-of-the -ortte-rrt method in constrained RL that also supports the case where feedback is elicited from an external evaluator. Similar to RLSF, the method consists of two stages, a data collectionnfeeddack stage and a policy optimization stage. the first stage, trajectory is labelled as good if (r(r) 2 Ugood n. T(c(T) < Cmar), and labelled as bud [r(r) 2 Tbad U T(c(T) 2 Cmar)) where Tgood and Thad are predefined thresholds on the reward. The information l[c(T) < Cmax] is received from feedback. The idea is then to imitate the good
7

---

BateCriveBlickkng
SafeDroerraancChange
Sctedrrive/Twolan
Cost Viclation Rate
Cost Voletion
Valetion
0.4
Steps (1e1)
Steps 13e
(1ef)
Figure I: Cost Violation rate of different algorithms in the Driver environment. Each algorithm is run for 6 independent seeds, with the curves representing the mean and the shaded regions indicating the standard error.
trajectories and stay away from the bad trajectories. p the occupancy measure of the current policy т and p p are the occupancy measures of the good and bad trajectories respectively, then, is optimized as,
argmax
110B))
(8)
where =0" = and KL denotes the Kullback-Leibler divergence.
Safe Distribution Matching (SDM): SIM combines rewards with safety feedback into a joint notion of good and bad which may not be desirable when the cost function is unknown. Thus, we introduce an additional baseline that keeps these two signals separate by labelling the trajectory as good c(T) S Cmaz and bad otherwise. The policy is then updated as,
argmax + AKLL(p..1lp'
where A E [0, 1] controls the tradeoff between the two objectives.
As an upper bound, we compare the performance of our algorithm to scenarios where the cost function is known: PPO-Lagrangian (PPOLag) 3 and SIM with known costs (SIMKC) [66. The objective is not to surpass their performance, but to match it. Therefore, when reporting the results of these algorithms, we present the best- pesst-erformin seed across multiple runs.
All results presented use novelty- based sampling unless stated otherwise. We use segment length of 1 in the Driver, Goal and Push environments. This is because the Safety Goal and Push environments contain small obstacles that the agent interacts with for very brief periods of time, hence requiring more fine- mne-grained feedback. An example of this is present Figure [T in the Appendix. In the Driver environments, randomly initialized policy was highly unsafe. Thus long segment length would force the evaluator to label every segment unsafe, making cost inference infeasible. the remaining environments, the segment length corresponds to the length of the episode. Details on the number of queries generated for feedback can be found in Table in the Appendix. We grant the two baseline methods (with unknown costs) advantage by providing feedback for every trajectory generated.
5.2 Cost Inference across various tasks
Benchmark Environments Table [ presents the performance of the various algorithms on the benchmark environments. RLSF significantly outperforms the two baselines terms of reward and safety in all of the environments RLSF comes to within & 80% of the performance of the best run of PPOLag in 7/11 environments thereby underscoring its effectiveness in learning safe policies.
Driving Scenarios In the driving environments, we evaluate the performance of our algorithm against two baseline methods: naive PPO policy that solely maximizes reward and the PPO-Lag algorithm. Figure I presents these results. Notably, our method demonstrates comparable safety
eexcept the Car Goal environment where RLSF and PPOLag marginally exceed the threshold

---

Sefeeylloppereeeiity
SaletyHopprreelodity
1400
entropy_uniform
1200
Cost Viclttoo Rate
getropy_decreassng random Sandom_decreasing random_unifone
novel
Steps (1
Figure 2: Comparison of different sampling and scheduling schemes. Results are averaged over 3 independent seeds. The proposed sampling method generates on average s 1950 queries, hence for fair comparison the other methods were given a budget of 2000 queries.
performance to PPO-Lag while significantly outperforming PPO in terms of cost violations. This highlights the effectiveness of our approach in learning safe autonomous driving policies. Video clips showcasing the learned policies in the 3 scenarios are included in the supplementary material.
5.3 Cost Transfer
We also showcase the potential of using the inferred cost function to train an agent with different embodiment or morphology solve the same task from scratch, without requiring any additional feedback. We utilize the inferred cost function obtained during the training of a Point agent to train the Doggo agent for the Circle and Goal tasks. The observation space the environments consists of agent-related telemetry (acceleration, velocity) and task-related information (lidar sensors for coal/boundary/obsttclee detection). these experiments, we do not incorporate agent-related information when learning the cost function, ensuring that it can be transferred to new agent. Next, the inferred cost function remains fixed during the training of the second agent, and the task-related features are utilized for cost inference. We compare the performance of this approach with that of the PPO-Lagrangian algorithm that utilizes the underlying cost function of the environment. From Table 2 we can infer that agents trained using transferred cost function comparable in performance to agents trained using the true underlying cost function.
Table Comparison of PPOLag performance when trained with the underlying task cost function versus the transferred cost function. Results are averaged over three independent seeds.
Source Env
Target Env
PPOLag with true cost
PPOLag with transferred cost
Return C.V Rate (%)
2.69 + 0.24
2.00 0.27 0.18 0.05
Point Circle
Doggo Circle
0.63 = 0.09
Return
1.45 + 0.06
1 1.20 + 0.26
Point Goal
Doggo Goal
Cost (40.0)
40.86 4.80
37.31 + 6.77
5.4
Effect of Novelty Sampling
Next, demonstrate the effectiveness of the proposed novelty-based sampling mechanism by comparing with other popular querying methods from the literature [18] (20]. These methods
require a predefined budget for the number of trajectories presented to the evaluator. We evaluate two schedules: (1) Uniform schedule, where a fixed number of trajectories is shown for feedback after each round, and (2) Decreasing schedule, where the number of trajectories decreases in proportion to 1 each round. For each schedule, we test the following strategies for sampling the subset of trajectories to be presented to the evaluator: (a) Random sampling, which selects a random subset of trajectories, and (b) Entropy sampling, where trajectories are sampled in descending order of the
average entropy H(pp sn.f (st, az)) of the estimator. As shown in Figure entropy based sampling outperforms random sampling, and decreasing schedule outperforms uniform schedule. However, all the methods fall short compared the proposed novelty based sampling mechanism.

---

5.5 Analyzing the Inferred Cost function
SeletyPointGoil
To further evaluate the quality of the inferred reward function, we plot it alongside the true underlying cost function of the environment as presented in Figure 3] In the Point Goal envi- ronment, where feedback is collected for each state, the inferred cost function closely aligns with the true cost given sufficient data. However, in the Point Circle environment, where feedback collected at the trajectory level, the inferred cost function exhibits slight overestimation bias, as discussed in Section 4.2 Nevertheless, from Figure 3 we can conclude that our method learns cost functions that strongly correlate with the costs.
Cast
Estimated
Steps
5.6 Bias Correction
In Section 4.2 we established that when feedback is collected over segments, the inferred cost function tends to exhibit an overestimation bias. As result, optimizing for policy that deemed safe with respect to Cmaz using the inferred cost function lead to overly conservative policies. If the bias b, could be calculated apriori, then adjusting the safety threshold Cmaz would still satisfy the safety guarantees outlined in Corollary [ However, since estimating in advance is infeasible, we propose heuristic approach where a constant E RR is added to Cmaz to account for the bias. It's important note that Corollary only holds 6 S b, meaning this heuristic doesn't 't guarantee that the resulting policy will always be safe. Nevertheless, we tested this approach with different 8 values in the Car Circle environment, as this setting exhibited a higher overestimation bias (as reflected in Table T The results, shown in Figure demonstrate that adding this bonus does improve performance (with higher& values yielding better results), though it introduces the additional challenge of tuning 6.
Cost
(3,9)
Figure Comparing the inferred cost to the true cost.
6 Limitations, Future Work and Broader Impact
In this work, we consider the problem of learning cost func- tions from feedback in scenarios where evaluating defining such functions is either costly or infeasible. We present a novel framework which significantly reduces the burden the eval- uator by eliciting feedback over extended horizons, a challenge that has not been addressed in previous research. To further reduce their load, we propose a ewvelty-based sampling method that only presents previously unseen trajectories the evaluator. Through experiments on multiple benchmark environments, we demonstrate the effectiveness of our framework and the pro- posed sampling algorithm in learning safe policies. However, there are a few limitations we would like to acknowledge. First, our method relies state-level feedback in some environments, which can be expensive to obtain. Second, while we simulate feedback using the true cost function, real-world feedback is often noisy, especially when provided by human evaluators.
SafetyCarCrclle
Return 10
5- 30
6=20
U
Staps (1e4)
CafetyCerCrcle
5-
5-20
Cost Viglaiin Rate
would be valuable to conduct experiments involving real human subjects to validate the approach.
By enabling safer autonomous systems, our approach has the potential to enhance safety across range of applications, in- cluding autonomous vehicles, drones used for delivery, and industrial robots operating in environments like warehouses or manufacturing plants. These improvements could significantly reduce accidents, protect human operators, and increase the overall reliability of automated systems.
Stops (1e
Figure 4: Increasing Cmaz by 8 to correct for the overestimation bias.
10

---

7 Acknowledgments
This research/project is supported by the National Research Foundation Singapore and DSO National Laboratories under the Al Singapore Programme (AISG Award No: AISG2-RP-2000-0116 and Lee Kuan Yew Fellowship awarded Pradeep Varakantham.
References
[1] J. Achiam, D. Held, A. Tamar, and Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning. 2017.
[2] N. Ailon. An active learning algorithm for ranking from pairwise preferences with almost optimal query complexity. Journal of Machine Learning Research, 2010.
[3] D. A. Alex Ray, Joshua Achiam. Benchmarking safe exploration in deep reinforcement learning. 1rXiv:1910.01708, 2019.
[4] E. Altman. Constrained Markov Decision Processes. CRC Press, 1999.
[5] Amani, C. Thrampoulidis, and L. Yang. Safe reinforcement learning with linear function
approximation. In Proceedings of the 38th International Conference on Machine Learning, 2021.
[6] U. Anwar, Malik, A. Aghasi, and Ahmed. Inverse constrained reinforcement learning. In Proceedings of the 38th International Conference Machine Learning. 2021.
[7] M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In 30th Conference on Neural Information Processing Systems, 2016.
[8] A. Bennett, D. K. Misra, and N. Kallus. Provable safe reinforcement learning with binary feedback. In 26th International Conference on Artificial Intelligence and Statistics, 2022.
[9] Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros. Large-scale study of curio learning. In 7th International Conference Learning Representations, 2019.
[10] M. S. Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the 34th Annual ACM Symposium on Theory of Computing, 2002.
[11] R. Cheng, G. Orosz, R. M. Murray, and J. W. Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In 33rd AAAI Conference on Artificial Intelligence, 2019.
[12] P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In 3/st Conference on Neural Information Processing Systems, 2017.
[13] J. Fu, J. D. Co-Reyes, and S. Levine. Ex2:
Exploration with exemplar models for deep reinforcement learning. In 31st Conference on Neural Information Processing Systems, 2017.
[14] D. Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon. Iq-leam: Inverse soft-q learning for imitation. In 35th Conference on Neural Information Processing Systems, 2021.
[15] J. Ho and S. Ermon. Generative adversarial imitation learning. In 30th Conference on Neural Information Processing Systems, 2016.
[16] H. Hoang, T. Mai, and P. Varakantham. Imitate the good and avoid the bad: An incremen- tal approach to safe reinforcement learning. In 39th Annual AAAI Conference on Artificial Intelligence, 2024.
[17] J. Ji, B. Zhang, J. Zhou, X. Pan, W. Huang, R. Sun, Y. Geng. Y. Zhong, J. Dai, and Y. Yang. Safety gymnasium: unified safe reinforcement learning benchmark. In 37th Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[18] K. Lee, L. M. Smith, and P. Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In Proceedings of the 38th International Conference on Machine Learning, 2021.
[19] J. S. Leonard. The Foundations of Statistics. John Wiley & Sons, 1978.

---

[20] X. Liang. K. Shu, K. Lee, and P. Abbeel. Reward uncertainty for exploration in preference-basee reinforcement learning. In 10th International Conference on Learning Representations, 2022.
[21] D. Lindner, S. Tschiatschek, K. Hofmann, and A. Krause. Interactively learning preference constraints in linear bandits. In Proceedings of the 39th International Conference on Machine Learning, 2022.
[22] G. Liu, Y. Luo, A. Gaurav, K. Rezace, and P. Poupart. Benchmarking constraint inference in inverse reinforcement learning. 11th International Conference on Learning Representations, 2023.
[23] S. Poletti, A. Testolin, and S. Tschiatschek. Learning constraints from human stop-feedback in reinforcement learning. In 22nd International Conference on Autonomous Agents and Multiagent Systems, 2023.
[24] D. Sadigh, A. D. Dragan, S. S. Sastry, and S. A. Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems, 2017.
[25] W. Saunders, G. Sastry, A. Stuhlmülller, and O. Evans. Trial without enror: Towards safe reinforcement learning via human intervention. 16th International Conference on Autonomous Agents and Multiagent Systems, 2017.
[26] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms.
2017.
[27] H. Su, Liu, B. Zheng. X. Zhou, and K. Zheng. A survey of trajectory distance measures and performance evaluation. The VLDB Journal, 2020.
[28] Y. Sui, A. Gotovos, J. Burdick, and A. Krause. Safe exploration for optimization with gaussian processes. In Proceedings of the 32nd International Conference Machine Learning, 2015.
[29] H. Tang, R. Houthooft, D. Foote, A. Stooke, X. Chen, Y. Duan, J. Schulman, F. D). Turck, and Abbeel. #exploration: study count-based exploration for deep reinforcement learning. In 31 Conference Neural Information Processing Systems, 2017.
[30] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
[31] A. Wachi, W. Hashimoto, and K. Hashimoto. Long-term safe reinforcement learning with binary feedback. In 39th Annual AAAI Conference on Artificial Intelligence, 2024.
[32] A. Wachi and Y. Sui. Safe reinforcement learning in constrained markov decision processes. In International Conference Machine Learning. 2020.
[33] S. Xu and G. Liu. Uncertainty- aware constraint inference in inverse constrained reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.
[34] I.. Yang, J. Ji, J. Dai, I.. Zhang, B. Zhou, P. Li, Y. Yang, and G. Pan. Constrained update projection approach to safe policy optimization. In 36th Conference on Neural Information Processing Systems, 2022.
[35] Y. Zhang, Q. H. Vuong, and K. W. Ross. First order constrained optimization in policy space. In 34th Conference on Neural Information Processing Systems, 2020.
12

---

A Proofs
Proposition The surrogate loss L sau is an upper bound the likelihood loss L Lule,
Proof. We have,
1,a ml Fi yoafe)
Σ 1=1
log (1 - p at))
log p (stte(s): az))
1 IT p peafe
( fa featss y1-ye/
log
IT'= prase
Notice that the term,
1 1- 11 (sate
log
20
II: psase ac)
This is because
j safe (zt az)
III ua (st az)
which we prove subsequently.
Lemma 1. For any sequence of numbers { where 0s E [0, for all E I...., n we
have 1 [. 2; 2 I1 (1)x.)
Proof.
Using the AM-GM inequality we have,
n
1=
Fj
n1
Σ
EE Tq
Tj
21 1
(10)
n
n
Similarly, we have,
x2) 5 2i=1 xz})
Ei=1 (1 - Ri)
Σし (1 xia)
x4)
2
(I1)
n
72
Adding Eq 10 and Eq
we get,
Σn
E>=n(1 - Fii)
2,)
(=1
i=1
m.) 2 0
1=1
85 2
(1 - x.))
(12)
11
13

---

Proposition 2. The optimal solution to Eg 4 vields the estimate,
dg(s, a) dg(s, ds
p. (s,
Proof. The proof is obtained by differentiating w.I.t p a) and setting to O. Proposition 3. For fixed policy the bias in the estimation of the incurred costs is given by,
(x, a)] Ex Cyr(s, a)] El [T dgl]
where P7 a)) = safe states visited by n
= (s,a) n Cst(s, a) 0]] is the occupancy measure of true
Proof. The estimated cost functiion c.(s, a) = 1pp (s. a) t. from Eq
we get p. pa (s, a) <
when (s, a) > dg(s, a). Thus, e- (s, a) = [l|d a) > ds)
T
(x a) (st(s, a)) |
ETTT
Σ
Ic, (st, a1) - (s,
x=0
r
E
I[(st, at) (s, Cgc(St, ax]
t=0
T
Eraa
[[(s,, a.) = (s = (st, a+) - cor az)
1= 8,a
[[[[,, a:) (s, (acoo)))=((((((((
(00, az) (ge(
Eraa (( Bes 0] + (( (c,(8, -1]
where (s) = E ET at) = (s n cgr(s,
0]] and (s,
Eiax T o = a) n
Cgt Cge = I]].
Note that when a state unsafe, i.e, (s a) = 1, then, p. safe a) = 0 as ny a) = 0. Thus for all of these states, (s, a) 1. Hence, are left with.
[x a) - cgr(s, = E(a,a) Co (s,
= E(s.a)ppg [1d > dg]]
Corollary 2. Any policy и that is safe w.rl ex guaranteed to be safe w.rl Cyt.
Proof.
From Proposition
3 we know that Eg (cx a)] - (g a)) = ((r,))pp3 [I|ds > dgll.
Since E(s,a)-pg [[d dgI O, we have Ex|c, (s, a)] (s
Thus, (s, Cmar (ogeee a)] S Cmar.
B Environments
B.1 Benchmark environments:
Mujoco [30]-bseel environments extend traditional locomotion tasks by incorporating safety
constraints. These environments are categorized into two types: position- based and velocity- constraints. A detailed description of each is provided below:
Position Based: These environments, recently introduced as benchmarks in are explained in more detail below.
14

---

(a) Blocked Swimmer: The agent controls robot composed of three segments joined two points. The objective to move the robot to the right as quickly as possible by applying torque through rotors at the joints. The agent receives reward at each time step, proportional to displacement in the X-direction. The episode concludes after 1000 time steps. To increase the challenge, cost is imposed when the agent's s X --coordinate exceeds 0.5, constraining it to the region -00 I X < 0.5.
(b) Biased Pendulum: The agent's goal is to balance a pole on a cart. Each episode ends if the pole falls after a maximum of 1000 time steps. The agent receives reward of I when its X X-coordinate is within the range (-oc. -0.01). which monotonically decreases to 0.1 as the X -ccorrdinate approaches Beyond this, the agent receives constant reward of 0.1, encouraging movement in the leftward direction. To prevent excessive leftward movement, a cost of 1 incurred when the X-coordinate is in the region (-o0,- -0 0.015], constraining the agent from moving too far left.
2. Velocity Based:
These environments are part of the Safety Gymnasium benchmarks, introducing velocity constraint for the Half-Cheetah, Hopper, and Walker-2d agents. The agent incurs a cost of 1 whenever it exceeds a velocity threshold of Umaz/ 22, where Umaz is the maximum velocity achieved by the agent when trained with PPO for 10 steps.
Safety Gymnasium 17 has emerged as prominent benchmark for evaluating constrained re- inforcement learning algorithms, providing a challenging framework for cost inference. The envi- ronments feature multiple agents, including the Point, Car, and Doggo, listed in increasing order of difficulty in controlling the agent. The objective is to complete specific tasks using one these agents.
In the Circle task, the agent must navigate around the boundary of circle centered the origin while remaining within a safe region defined by two vertical boundaries at x = =0.7. A cost of is incurred each time the agent exceeds this boundary (|æ] 2 0.7).
In the Goal task, the agent must reach a designated goal area as quickly as possible while avoiding both static and dynamic obstacles, incurring cost of for each collision.
The Push task builds upon the Goal task by adding the challenge of pushing a block to the goal area while still avoiding collisions with obstacles in the environment.
B.2 Safe Driver
Driving-based environments are emerging as key benchmarks for addressing the problem of constraint inference [22], particularly due to the complex safety considerations involved in driving. We evaluate our algorithm using the Driver simulator introduced in [21 to learn safe behaviors from feedback. This environment encompasses two scenarios that self driving agent is likely to encounter on the highway: a blocked road and a lane change. Additionally, we introduce a third scenario that involves overtaking on two-lane highway, where the agent must safely pass slower car in its lane by utilizing the second lane. This maneuver poses risks, as traffic in this lane is moving in the opposite direction, necessitating careful execution. Below, provide brief description of the environment.
The Driver environment utilizes point-mass dynamics and features continuous state and action spaces. The state of vehicle in the simulation is represented by the tuple (x, y, 6, v). which includes the agent's position (a, y). heading é and velocity v. The observation of the ego vehicle is formed by concatenating the states of all vehicles in the scene. The environment allows for two actions (a1, Q2) where aj represents the steering input dz denotes the applied acceleration.
The state of the agents evolves as
11+1 = (x + Ex, y + dy, o + 60, v + 8v) (ōx, dy, 80, dv) (o cos Φ, sin o, 02 - av)
where is used to control the friction. Additionally, the velocity is clipped to the range [-1, 1] at each timestep.
The task is to control the ego vehicle (white in Figure 5) to reach the top of the road as soon as possible while adhering to costs on the velocity and proximity to other vehicles. The trajectories of the other vehicles are fixed, with some random noise applied to their steering and acceleration inputs.
15

---

SafeDriverBlocking
SafeDriverrWooLnee
SafeDriverLaneChange
8
Figure 5: Driver Environments
The agent receives reward r. 10(y: yt-1) each time step, incentivizing navigate through traffic rapidly. Additionally, it incurs a penalty of -1 for going off the road, driving backward, failing to stay in the center of the lane. collision results in penalty of 1000 The episode terminates when the agent reaches the top of the road or collides with another vehicle. Although these constraints can be integrated into the reward function, they can still lead to unsafe behaviors, highlighting the necessity for an additional cost function. The PPO agent is trained using this reward function.
In the CMDP settings, the cost function incorporates some of the constraints from the reward function. For instance, the agent incurs cost of when it goes off the road or drives backward Additional costs are incurred when the agent crosses the speed limit Umar or gets too close to any nearby vehicles. The ego is deemed to close
neighbouring vehicle if exp =b(crd² czd?) ba > 0.4, where a = 0.01, = 30, e1 = 10, C2 = 2 and d, and dy represent the distance of the nearby vehicle from the ego vehicle.
C Related Work
Constrained RL The Constrained Markov Decision Process (CMDP) framework 4 has emerged as valuable tool in safety-critice settings, offering clear separation between task-related infor- mation (rewards) and safety-relate information (costs). This separation simplifies environment specification and allows for easier transfer of cost functions across similar environments. While previous research has extensively addressed this problem [] 35 34 16), these studies typically assume that the cost function is known. However, this assumption can be limiting when designing the cost function is complex or costly to evaluate. Consequently, there a need infer the cost function from external data in such cases.
16

---

Cost Inference Historically, cost inference has primarily depended on two types of data: (1) evaluator feedback and (2) constraint abiding demonstrations obtained from an expert.
Previous work on learning from feedback often relies on limiting assumptions such as deterministic transitions and smoothness [28] 32 11 linearity assumptions [5] on the cost function. Moreover, none of these approaches address the problem of actively querying humans to minimize the required feedback. Although this has been theoretically explored in (8), the proposed algorithm has not yet been adapted for complex environments. Additionally, these methods are typically limited to feedback on individual state-action pairs, which can be expensive when relying on human evaluators. Our RLSF approach addresses these limitations by reducing the amount of feedback required and avoiding assumptions about the cost function.
Inferring cost functions from constraint-satisfying expert demonstrations has gained significant attention recently 6 33 22) However, acquiring such expert demonstrations can be challenging in many scenarios. An alternative form of feedback human interventions. For instance, (3 rely on the human evaluator to terminate the episode when the agent is about to engage in unsafe behavior, while [25] depend on the human to take control of the agent in such situations and guide it to safe state.
Novelty-Basee Sampling Novelty estimation has been widely explored the context of active exploration in reinforcement learning. The connection between prediction accuracy and state novelty has been extensively studied, as highlighted by [9] Additionally, state density estimation has been investigated in works such as [29] 13), where infrequently visited states are assigned an exploration bonus. Notably, 29 also uses Simhash 10 to maintain state density estimates.
Reward Inference This work also connects the analogous problem of reward inference in standard MDPs, which similarly rely on human feedback 112 18 20 and expert demonstrations
15
14.
D
Experiments
D.1
Hyperparameters
We conducted the experiments on cluster quipped with NVIDIA RTX A5000 GPUs and 96 core CPUs. The experiments took an approximate of two weeks to run. The detailed hyperparameters utilized in the experiments are presented in Table All results presented both the main paper and the appendix are based on three independent seeds, with the mean and standard error reported, unless explicitly stated otherwise.
Table 3: Hyper Parameters
Hejer
Fapper
Alter ladk soe
(OA, 250)
[156, 256 254)
(206, 255, 250)
120h 155, 230)
1125
1155 250,
1256
256, 2551
(206) 155, 250)
(206, 255 250)
[64]
Caveler Newwek
[5454]
(6.541
(3]
[32)
(13)
[21]
541
[64. 541
[66]
(1/001
0.000
0.001
11011
01001
Lagnagta.
B011
041
0011
0.1
6.11
0.01
0.50
01)1
0011
051
Jurpedee Choccter
500
360
900
9301
1000
5003
5331
5001
500
5301
bach
4290
4236
Statlak tat
D.2
Main Experiments
The training curves for the benchmark experiments and the Driver environment presented in Figure 7 and Figure Additionally information on the number of queries presented for feedback is presented in Table
D.3 Novelty based Sampling
Souure/]show the implicit decreasing schedule encountered when using novelty based sumpling in various environments. Figure shows additional comparison with other querying mechanisms in the Biased Pendulum environment. To further assess the effectiveness of novelty based sampling, categorized the trajectories collected during data collection round based on novelty. We then
17

---

SchteGrredd
Scht/Crthhnn
Stpp
Stepe Iteft
SpeaCatiin
GOD Woa
Staza
Sabryyttt Swechhaaccttyt
Saza
acoadiameee
PPOLag
SIMKC
Any
SDM
SIM
RLSF (Ours)
Cod Sparos 608
Saza
Stepo 11a"
Figure 7: Training curves depicting the performance of differentt algorithms, based on six independent seeds. The curves show the mean returns, with shaded regions representing the standard error. Parallel lines indicate the best-performing run.
18

---

Table 4: Number of trajectories presented the evaluator for feedback
Nature of Costs
Environment
Number of Trajectories shown
Episode Length
Segment Length (k)
Point Circle
889.66 + 185.27
500
500
Car Circle
1063 73.48
500
500
Position
Biased Pendulum
3939.33 + 231.89
1000
1000
Blocked Swimmer
646.83 127.56
1000
1000
HallCheetah
4112.83 + 134.26
1000
1000
Velocity
Hopper
1924.99 + 92.59
1000
1000
Walker2d
4170.33 + 174.83
1000
1000
Point Goal
7892 + 178.15
1000
Car Goal
3866.5 + 80.83
1000
Obstacles
Point Push
6409.33 77.46
1000
Car Push
3368.33 + 176.36
1000
1
Blocking
6242.3 + 778.13
100
Driving
Two Lanes
2912.37 + 443.31
100
Lane Change
3510.82 + 644.28
100
1
SafeCriverBlocking
SateBreertaaeeChange
SefeDrive/TwoLarc
4.2
02
(1eb)
Steps (1e
(1eb)
SafeCriberBBlecking
SafeDr veriaaneChangee
Sa SeDeirierr TwoLares
Coost Violation Rate
Violarton
Viglaiton
0.4
0.4
(105)
Stops (1c9)
(1o5)
Figure 8: Performance of algorithms in different driving scenarios. Each algorithm is run for independent seeds, curves represent the mean and shaded regions represent the standard error. Normalized return is given by the formula r-mm where TpPo is the return of PPO and Frandom
is the return of a random policy.
19

---

BisscdPenuulum
BavedPeendluum
182
mindom_uifoome
entropy_decreasing
Cost Volaton
0122
0.30
entropy_unifone random_decreaate novel
000
Steps (1et)
Steps (1ef)
Figure 9: Comparison different sampling and scheduling schemes. The results are averaged over independent seeds. The proposed sampling method generates average 3600 queries, hence for fair comparison the other methods were given a budget of 4000 queries.
0.973
0.088
0.587
SafetyPoiniCicle
SafetyHoparVäacily
SafetyPaindPush
Model prediction accuracy (mean + standard error) in the surrogate task (averaged over seeds) with trajectory level feedback. Predictions are made on the next 50k steps following 250k training steps.
Figure 10:
evaluated the model's accuracy on the pseudo preedd-preiitiion task for these two trajectory types, as shown in Figure [0] This analysis indicates that the model's accuracy lower for novel trajectories, corroborating the claims made in Section 4.3
D.4 Number of Queries generated
The number trajectories queried for feedback of the order of O1ee)), the exact number differs across environ- ments and individual runs due to novelty based sampling. Detailed information on the number of queries can be found in Table 4 We provide the two baseline methods with an advantage, whereby feedback is obtained for every trajectory generated by the policy, which of the order Olle)).
Bochedssmmer
0.-
D.5 Complexity of Constraint Inference
Figure highlights the difficulty of inferring constraints when violations are sparse. The brief interaction peri- ods with obstacles in the environment make it challenging assign credit unsafe states when feedback is collected over long horizons. Consequently, feedback was collected the state level in such settings. This is illustrated in Figure 12] where the overestimation bias is considerably higher in the Point Goal environment compared to Point Circle, resulting significant drop in performance.
Figure 6: Implicit decreasing schedule observed when following novelty based sampling across different environments. The results averaged over 6 indepen- dent seeds.
20

---

500K interactions
20M interactions
1.0
1.0
Cost
0.5
Cost
0.6
02
0.
0.8
200
400
600
500
2000
200
400
600
800
1000
Time Steps
Time Steps
Figure 11: Costs incurred by policy after 500K and 20M training interactions over a randomly sampled trajectory in the Point Goal environment.
D.6 Ablation on the size of the feedback buffer
We conducted an ablation study on the size of the feedback buffer, the results of which are presented in Figure 13 We can observe that the policy becomes overly conservative when the size of the feedback buffer is small (100k, which corresponds to approximately two data collection rounds). We hypothesize that this because of two interacting factors: (a) discarding past feedback can result in forgetting, which in turn leads to inaccuracies in estimating unsafe regions, and (b) there exists feedback loop which the inferred cost function influences on-policy data collection. If the inferred cost function overestimates costs in certain regions, the policy will avoid them and fail to gather the necessary feedback to correct this error. Conversely, if the costs are underestimated, the policy will visit those states and receive feedback, allowing for corrections. Therefore, smaller feedback buffer sizes can result in the learning overly conservative policies.
D.7
Need for the surrogate loss
In Section 4.2 we highlighted the challenges associated with minimizing Eq particularly due to the term I (prafe(sr,aa)) which tends to collapse to 0 with longer segment lengths, resulting investigate this further, we conducted simple experiment
unstable gradients. performed few gradient steps optimize EqBaand and plotted the gradient norm. The results, shown in Figure [4] reveal that (a) directly optimizing Eq 3 produces gradients with a large norm that either quickly vanish explode, rendering the learning process infeasible. In contrast, the surrogate objective in Eq 4 yields stable gradients allowing for convergence.
which we
21

---

SafetyPointCircle
SeletyPoirtCircle
5.10
42-
6.25
Cost Visaatin Iaale
20
5.05
k:10
1:0
530
SafetyPointGoal
SafetyPointGoal
2-
50-
Robum
Cont
100
1030
(1o7)
llc?)
Figure 12: Ablation on the segment length for feedback elicitation.
22

---

Safety
Safety
0.10
100k
1M
0.00
Cost Violation Rate
4M
Return
0.08
0.04
100k
0.02
1M
4M
20
20
Steps (1e*)
Steps (1e")
SafetyPointGoal
SafetyPointGoal
100k
1M
4M
Threshold
Retum
Cost
0-
100k
30-
-10
20
Steps (1e*)
Steps (1e*)
Figure 13: Impact of the size of the feedback buffer performance.
SafetyPointCircle
MLE Loss
Surrogate Loss
Norm of the gradient
10'
10n
10
10.
12.1
Iteration
Figure 14: Norm of the gradient when optimising the MLE loss v/s the proposed surrogate loss with trajectory level feedback.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction are backed by theoretical and empirical results shown in Section 4.2 and respectively.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the proposed approach are discussed in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification: All the assumptions are discussed in Sections and and the complete proof can be found in the Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Apart from the Safety Gymansium environments that are publicly available, all other environments are described in detail so that they can be reproduced from scratch. The hyper parameters used to train the proposed approach are given in Appendix D.I to ensure reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
25

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: All the environments used are described in Section B The code (with instruc- tions on how to run) is publicly available: https://gthub.com/hhhhseedddy/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (tttpss:/nnips.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/nips. wwtpcc/pcccpcpcccceooooooop /ttoggguSudeeeoeeuuubbiaiaiaaaai weneiisssnnPoPiiiio
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The hyperparameters used to run our propsed approach is present Section in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All results presented in the paper are averaged over multiple seeds and are presented with the standard error.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details on the compute and time taken to produce the results is available in Section[..]jn the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///eweuuissssppuitt///////a
Answer: [Yes]
Justification:
The presented research is compliant with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The social impacts of our work is discussed in Section 6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
27

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The work presented in this paper has no risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The README file present in the code credits the authors, and the license and terms of use of has been properly respected.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The Safety Driver environments described in Section B in the Appendix and instructions to run them are present in the README file included with the code.
Guidelines:
The answer NA means that the paper does not release assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not conduct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not conduct research with human subjects
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
29

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "gwX7fD7aCY", "reviewer": "Reviewer_E6xa", "summary": "Authors propose a new way of estimating the cost function for constrained RL through user feedback. They propose a surrogate loss that is used to train a model that estimates the probability that a state-action pair is safe. They then use the model to estimate the cost of a policy and adhere to the constraint while looking for the best reward.", "questions": "Could authors explain the DNN that estimates the probability of safety for a (s, a) pair?\nseems like they are directly estimating the probability? \nI am curious, if the authors have considered estimating the d_g and d_b densities (as defined in proposition 2 for the optimal solution) instead?", "limitations": "How is novelty sampling (which seems naive to me) related to uncertainty sampling? The proposed novelty sampling, roughly, ignores the trajectories that have many visited sub-sequences. \n\nI think Corollary 1. is theoretically wrong, as the authors have conveniently ignored the inherent inaccuracies in user feedback. Authors mention labeling noise as the noise that is the result of expanding an unsafe label to all the pairs in the trajectory but there is also the meta noise that the user labeling(user feedback) by itself is an error-prone process.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is relatively easy to follow for a person that is not very familiar with the field. The motive and theoretical assumptions are explained fairly well.", "weaknesses": "The experiment section lacks some details and is much harder to follow. \n\nI did not see any experiments (ablations) on the size of the feedback buffer. \n\nAlso I was hoping to see some discussion around the introduced bias in estimation with differing c_max values.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and constructive review of our paper. We appreciate your positive assessment of our work and the valuable feedback you provided. We hope to address the weaknesses and questions you raised below:\n\n1) We use a regular Multi-Layer Perceptron (MLP) to estimate the probability of safety. The model takes the state-action pair as input and outputs the probability $p^{safe}_{\\theta}(s,a)$. We request the reviewer to refer to Section C.1 in the Appendix for further details on the network architecture of the Classifier.\n\n    Yes, we did consider directly estimating $d_{g}$ and $d_{b}$, some possible methods include: \n    * SimHash can be used to discretize the state-space and store $d_{g}$ and $d_{b}$ via counts. We believe hashing states via raw feature state features is good enough for novelty estimation but the classification task would require more rich features. This is because this type of model is essentially a linear model, limiting its capacity to capture state densities at the accuracy required for cost function estimation.\n    * Exemplar methods, as proposed by Fu et al., estimate state densities through a proxy binary classification task. This task distinguishes between a single state $s^*$ and the states in the feedback buffer $B$. Let $D$ represent the dataset containing only $s^*$. Since $s^*$ may also exist in $B$, $p(D|s^*)$ is not necessarily 1, requiring the classifier to account for the density of $s^*$ in $B$. This approach enables the estimation of $p(s^*|B)$, i.e, the state density. We believe solving two proxy binary classification problems (one for each buffer) is inefficient compared to the single problem specified by Eq 4. \n    * Generative Models such as GANs [Goodfellow, et al.], VAEs [Kingma, et al.] and Normalising Flows [Rezende, et al.], etc. that implicitly store the densities in order to generate samples. The density can be extracted from Flow based models but GANs and VAEs require additional modifications to extract the density. These models are more complex and hence introduce additional constraints such as increased data requirements and computational demands.\n\n    Given the above reasons, we believe directly estimating the probability $p^{safe}_{*}$ is a more suitable approach.\n\n2) Thanks for pointing this out. We have added an ablation on the size of the feedback buffer in the attached document (Figure 3). As expected, we notice that the performance of RLSF increases with the size of the feedback buffer as feedback given in the past is not discarded.\n\n3) We agree that a discussion on this point would be insightful and will add it in the revised version. Corollary 1 is true for any threshold $c_{\\text{max}}$, since the estimated cost function has an overestimation bias. Consequently, satisfying the $c_{\\text{max}}$ threshold can lead to overly conservative policies. To counter this, ideally one could increase the threshold $c_{\\text{max}}$ to $c_{\\text{max}} + \\text{bias}$. Calculating the bias apriori without the knowledge of the ground truth cost function is infeasible. Hence, one could take a heuristic approach and increase the threshold to $c_{max} + \\delta$, with $\\delta \\in \\mathbb{R}$. We conducted additional experiments in the SafetyCarCircle environment to test this method (Figure 4 in the attached document), as we observed higher overestimation bias in this setting (Table 1 in the main paper). We observed that adding such a bonus does indeed boost the performance of the proposed method.\n\n4) Novelty-based sampling is a form of uncertainty sampling as it targets states with high epistemic uncertainty—uncertainty arising from a lack of feedback. Epistemic uncertainty is known to adversely affect prediction accuracy [Burda, et al.]. Since we cannot directly measure epistemic uncertainty, we instead analyse the correlation between novelty of trajectories and the model's prediction accuracy on states within those trajectories. This analysis, presented in Figure 5 of the attached document, serves to evaluate the effectiveness of our proposed novelty-based sampling in identifying states with high epistemic uncertainty.\nThe results demonstrate that our novelty-based sampling method does indeed identify states with high predictive errors, which is indicative of high epistemic uncertainty. By eliciting feedback for these states, we can improve the model's accuracy on these states. This further explains the superior performance of our proposed sampling method, as evidenced in Figures 1 and 7 of the main paper.\n\n5) Please note that Corollary 1 is assuming ground truth, $c_{gt}$ information is correct (and not misspecified), the feedback is “sufficient” and is based on the optimal classifier $p^{safe}_{*}$\n\n    We acknowledge that real-world scenarios may deviate from these assumptions:\n     * $p^{safe}_{\\theta}$ \\text{ may not always converge to } $p^{safe}_{*}$ due to factors such as optimization errors, model capacity, etc. \n     * Feedback may be missing for some states. \n     * Evaluation (meta) noise may occur, particularly with human evaluators. Possible methods to account for this (meta) noise are: \n\n       a) $p^{safe}_{gt}(s,a)=\\mathbb{I}[c_{gt}(s,a)=0]$ (noise in safety preferences)\n\n       b) In $y^{safe}$ (noise in labelling process)\n\n    We agree that theoretical results encompassing all these factors would be valuable. However, developing such a comprehensive analysis is non-trivial and beyond the scope of this work. Hence, we have also provided experimental results for cases where these assumptions cannot be satisfied. \n    \nCitations: \\\nBurda, Y., et al.. Exploration by random network distillation. ICLR. \\\nFu, J., et al. EX2: Exploration with exemplar models for deep reinforcement learning. NeurIPS \\\nGoodfellow, I., et al.. Generative adversarial nets.  NeurIPS \\\nKingma, D. P., et al. . Auto-Encoding Variational Bayes. ICLR. \\\nRezende, D. J., et al. Variational Inference with Normalising Flows. ICML", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w9AtDN8tW4", "reviewer": "Reviewer_w6J6", "summary": "This paper proposes a surrogate loss function, which instead of collecting a feedback over trajectory-based evaluator, it breaks long trajectories into segments and the evaluator classifies and assigns labels to segments unsafe if any individual state is unsafe during the segments. This paper also proposes a novelty sampling, which only gathers novel trajectories for feedback from evaluator based on the trajectory distance measure edit distance.", "questions": "1. On line 136, the probability of a state being safe is defined as $\\mathbb{1}[c_{\\mathrm{gt}}(s, a) = 0]$. Isn't this an indicator function instead of assigning a probability to this specific state? Or this is a typo.\n2. What does the ``noisy labels`` mean on line 150? This phrase also appears somewhere else and I am not sure what it refers to. Also for the sentence following that, ``However, if it occurs in a segment labeled unsafe, it is incorrectly labelled unsafe``. Does it mean if the segment is labelled as unsafe, some of the individual states in the segment ought to be safe but they are labeled as unsafe then?\n3. Why is it true that ``with a sufficient number of samples, the number of correct labels will outweigh the number of incorrect lables for each state``?\n4. I can't quite follow the paper but based on my understanding, there's a human evaluator to evaluate segment, and another evaluator for state?", "limitations": "Both limitations and societal impacts are discussed in the paper.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The pseudo algorithm helps to understand the paper's idea a bit more. The experiment part seems complete with the methods tested in different tasks and environments to cover the proposed algorithms and the novelty-based sampling method.", "weaknesses": "1. I found it hard to follow the paper, especially in section 4.2. Some questions are raised in the following section.\n2. I understand that Eq 4 is an upper bound for Eq3, but I doubt that the difference between Eq3 and Eq4 might be too huge, and in this case, switching to Eq4 might result in a very inaccurate estimate $p_*^{\\mathrm{safe}}$ for the following analysis in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and constructive feedback. We appreciate your insights and will address your questions and concerns below:\n\n1) Please note that $p_{gt}^{safe}(s,a)$ is the ground truth probability. Since ground truth is exactly known for a state on whether it is safe or unsafe, $p_{gt}^{safe}(s,a)$ is either 0 or 1 depending on whether $c_{gt} = 0 \\text{ or } 1$ respectively. This is the reason for using indicator function for ground truth. \n$p^{safe}(s,a)$ is our estimate of the probability and this will typically have values that are not 0 and 1 only. \n\n\n2) The surrogate loss introduces a binary classification problem where each state is assigned a pseudo-label $y^{\\text{safe}}$ based on the segment it occurred in. Safe states (w.r.t ground truth cost) can be assigned both unsafe and safe pseudo-labels depending on whether they occur in a segment that contains an unsafe state (or not). We refer to this phenomenon as “noisy labels” in the surrogate classification task.\nYes, the evaluator is tasked to classify a segment unsafe if it contains one or more unsafe states. The rest of the states (if any), although inherently safe, are assigned the pseudo-label of unsafe.\n\n\n3) Thanks to the reviewer for pointing this out and we do need to modify it as follows:\n“We posit that with a sufficient number of samples, we will be able to distinguish the safe states from unsafe states. This is because we observe that: \n(i) safe states will appear in both safe and unsafe segments; and \n(ii) unsafe states would appear only in unsafe segments and not in safe segments;”\nHence with a sufficient number of samples,  it would become feasible to distinguish safe from the unsafe states.\nWe would direct the reviewer to our strong performance on multiple safety environments (Circle, Goal and Mujoco based) that indicates this hypothesis to hold quite well in environments when the feedback is collected for the entire trajectory. \n\n\n4) There is a single evaluator (human or a program that is pre-trained/pre-coded) that classifies a segment. Each trajectory that requires evaluation (chosen based on novelty) is then broken down into segments, and each segment is passed to the evaluator for feedback. Feedback is not explicitly collected for individual states unless the segment length is 1.\n\n\n5) We theoretically quantify the difference caused by transitioning from Equation 3 to Equation 4 in Proposition 3, in terms of cost overestimation. If this difference is considered excessive (results in overly conservative policies), the segment length can be reduced (by the user). However, this adjustment comes at the cost of increased evaluation effort, thus introducing a tradeoff.\n\n    Directly optimising Equation 3 with trajectory-level feedback results in numerical instability (Figure 2, attached document). This highlights the necessity of the proposed surrogate objective.\n\n    We further investigate the effects of transitioning from Equation 3 to Equation 4 in Table 1 of the attached document. MLE corresponds to taking feedback at the state level, equivalent to solving Equation 3. The Surrogate, on the other hand, represents solving Equation 4 with feedback collected at the trajectory level.\n\n    We find that in the Circle environment, optimising the surrogate loss with trajectory-level feedback achieved performance comparable to optimising Equation 3. This suggests that the difference between Equation 3 and Equation 4 is minimal in the Circle environment. Additionally, the strong performance of RLSF compared to PPOLag in Table 1 of the main paper indicates that this minimal difference likely extends to the other Mujoco-based environments as well, demonstrating the promising applicability of our approach across a wide range of problems.\n\n    However, Table 1 in the attached document shows a divergence between the two objectives in the Goal environment. The poor performance of RLSF in this case can be attributed to the overestimation bias discussed in the paper and the complexity of the cost function in this task, as described in lines 267-275. Therefore, in this scenario, state-level feedback is necessary for good performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5TuJhTTNnp", "reviewer": "Reviewer_1oHB", "summary": "This paper presents a method for using evaluator feedback as safety constraints in a constrained RL framework. The novelty comes from the nature of the feedback and the sampling scheme. It is motivated by the fact that previous approaches have various limitations: designing cost, especially comprehensively, is expensive and even seemingly impossible a priori. Offline feedback can help, but existing approaches often don't scale, and/or are constrained to receive feedback at the state level, which is also expensive. Giving feedback at the trajectory level is an option, but also a difficult state-credit assignment problem; and even getting feedback for every trajectory is expensive. \n\nThis paper presents RLSF, a method in which binary feedback is given on segments or entire trajectories and a novel surrogate loss is presented to make this a useful learning signal, then a new \"novelty-based sampling mechanism\" is used to train efficiently.\n\nThe paper takes us through the methods, including the feedback collection process (human or procedural evaluators), the policy improvement step, the nature of the feedback (1 if every state in a given segment is safe, 0 otherwise), and how the cost function is inferred from the collected data. It also presents a surrogate loss function for turning segment-level feedback into a dense signal. \n\nThe paper then goes through experiments and results in the Safety Gym and Driver simulation environments. The tasks have various constraints in position and velocity. Experiments compare to three general baselines - self-imitation safe RL, safe distribution matching, and PPO where the true cost function is known (considered an upper bound). The methods are tested directly on the tasks, then learned cost functions are transferred to a new agent. The paper then goes through ablations removing the novelty-based sampling method. The results show that this RLSF algorithm shows significant improvement over baselines.", "questions": "- The paper cites sparse cost violation as a motivation for collecting feedback on shorter segments. Why is that? Is the idea that when cost violations are dense, it's hard to give more specialized feedback, but when they're more sparse then the evaluator can just go for it? Or is it purely a horizon issue (shorter horizons have fewer cost violations), in which case mentioning sparse cost violations at all doesn't make sense?", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "### Originality\n- I am not aware of any methods that use exactly this surrogate loss or sampling method. RLSF does borrow heavily from prior work, but the loss function, sampling method, and otherwise novel (to my knowledge) mix of existing elements does constitute an original work.\n\n### Quality\n- Experiments section, particularly ablation of the sampling section and cost transfer, is creative and shows lots of important aspects of the study. The cost transfer results are particularly promising. \n- Results speak for themselves! They show a system that works in simpler environmetns and is ready to be scaled up and tested. \n\n### Clarity\n- Well written and clear. There is a fair amount of technical content in terms of defining problems and setting up notation, and the paper handles it all well. \n- The novelty-based sampling mechanism is particularly well-explained.\n\n### Significance \nSeems good. It's good to see new ways of integrating feedback as human interaction with robots comes nearer and nearer.", "weaknesses": "### Quality\n- \"We posit that with a sufficient number of samples, the number of correct labels will outweigh the number of incorrect labels for each state\" - doesn't this require sparsity of safety violations and small segments (meaning a hefty feedback load)? Questions like this would require answers to judge if this work is useful. \n- It would be good to see more error modes - for example, the paper cites novel states as a hotbed for potential errors, but it's only one. What are the others?\n- In the experiments section, it would be good to see cost transfer compared to other cost function learning methods, not just PPO vs. RLSF vs. PPOLag.\n\n### Clarity\n- Tables should have bolded numbers, even if all the bolding will be in one column. The tables are incredibly dense right now and tiring to read. \n- Explanation of surrogate loss derivation would benefit from more justification. E.g. I see that it's true that the product term substitution makes it a state-level term instead of a segment-level term, but explain that. more clearly and why it's okay to do and still expect the optimization to work out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and positive review of our paper. We look forward to addressing your questions and suggestions below:\n\n1) By sparsity we mean the density of violations within the segment is low. This makes distinguishing safe states from unsafe states challenging, as it becomes difficult to identify which states caused the segment to be marked unsafe. Consider Figure 8 in the Appendix, in this case less than 10% of the states in that trajectory are actually unsafe. Thus, the rest of the states get an incorrect pseudo-label of unsafe. This would reduce significantly if the cost violations were more dense, say (~50%) of the states within the trajectory were unsafe, or if feedback was collected over shorter segments (which inherently reduces sparsity). To summarise, dense cost violations and shorter feedback segments simplify the cost inference problem, while sparse violations and longer segments make it more challenging.\n\n\n2) Thanks to the reviewer for pointing this out and we do need to modify it as follows:\n\n    “We posit that with a sufficient number of samples, we will be able to distinguish the safe states from unsafe states. This is because we observe that: \n\n    (i) safe states will appear in both safe and unsafe segments; and \n    (ii) unsafe states would appear only in unsafe segments and not in safe segments;”\n\n    Hence with a sufficient number of samples,  it would become feasible to distinguish safe from the unsafe states.\n    We would direct the reviewer to our strong performance on multiple safety environments (Circle, Goal and Mujoco based) that                 indicates this hypothesis to hold quite well in environments when the feedback is collected for the entire trajectory. \n\n3) We can broadly categorise the reasons for model prediction errors into four main branches: Limited Data, Noise in the Labels, Model Misspecification and Optimization Error [Burda, et al.]. The latter two sources must be addressed before the learning process. Novelty based sampling addresses the first source of error. Label Noise can be reduced by decreasing the segment length. Other methods such as eliciting extra state-level feedback on states with noisy labels are also possible, making for interesting future work.\n\n4) Please note that for doing cost transfer, there is an explicit cost function that has to be learnt that can be transferred. We would like to highlight that SIM and SDM do not learn explicit cost functions. Additionally, previous work [25,29,10,7,5] has not been scaled to handle the environments with continuous state and action spaces considered in these experiments. Therefore, our cost transfer experiments (Table 2) were limited to the aforementioned algorithms.\n\n5) We appreciate the feedback on Table 1, we will bold the numbers in the final draft.\n\n6) The main motivation for proposing the surrogate loss was due to the numerical instability that arises from multiplying the probabilities over a long horizon. To further illustrate this point, we plotted the norm of the gradient after the first 10 epochs of training the classifier using Eq 3 directly, as shown in Figure 1 (in the attached document). The gradient either collapses to 0 or explodes very quickly in the first few iterations making optimization challenging. The surrogate objective side-steps the need to multiply probabilities over long horizons and thus results in stable gradients as shown in Figure, leading to easier optimization. We acknowledge that this motivation was brief in the paper (due to space limitations), but will take this feedback into consideration and make the motivation more clear.\n\nCitations:\nBurda, Y., Edwards, H., Storkey, A., & Klimov, O. (2019). Exploration by random network distillation. In International Conference on Learning Representations (ICLR).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aCA4B7jHr7", "reviewer": "Reviewer_xFMh", "summary": "This paper studies the safe RL problem with an unknown cost function. As an on-policy approach, this work tries to learn the cost function with safety feedback from an evaluator with novelty sampling and conducts policy optimization at the same time. The authors propose that their approach can deal with feedback over longer horizons by a surrogate loss, reduce sampling queries by novelty sampling, and demonstrate the effectiveness via a set of experiments.", "questions": "1. In section 4.2, why do you formulate the safety/cost function by a probability measure? It is confusing to me. Therefore, I cannot understand why using maximum likelihood to learn the cost function. \n2. You have to show the learned cost function is close to the ground truth in the experiments. Otherwise, I cannot trust the experimental results as there are so many uncertainties/hyper-parameters in RL. \n3. A suggestion for this paper is to consider LLM as an evaluator.", "limitations": "The authors discussed their limitations in the conclusion.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow in general. \n2. The experiment results look promising. \n3. The setting of the unknown cost function in safe RL is very interesting and important.", "weaknesses": "1. The novelty of this paper might not be enough. First, it is not clear to me the difficulty of inferring the cost function in RL, current existing approaches, and how the proposed approach outperforms the base approaches. Second, novelty sampling or frequency-based sampling is well-studied and leveraged in classical RL. Third, there is nothing new in policy optimization in this paper. \n2. It is not clear to me that how close the learned cost function is to the ground truth cost function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and constructive feedback. Your insights and suggestions are appreciated, and we look forward to addressing your questions below:\n\n1) In binary classification problems, the ground truth is often represented as a Dirac measure (indicator function), and maximum likelihood estimation (MLE) leads to the widely used Binary Cross-Entropy (BCE) loss. Motivated by this, we represent the cost function as a probability measure in our work and subsequently use MLE.\n\n2) We understand the concern and have added plots showing the inferred cost function and the estimated costs in Figure 1 in the attached document. It can be seen that the cost function closely approximates the true cost in all the environments. In Hopper, there is a bit of divergence, but with enough steps, we are able to get close. \n\n\n3) Yes, this suggestion makes for interesting future work. The challenge lies in converting the MDP to text, instead eliciting feedback from multi-modal foundation models that are shown videos of the trajectory is another promising direction. \n\n4) We understand this is a subjective assessment of the reviewer and we respect that. However, we do wish to point out that there are multiple key differentiating factors between the proposed work and existing literature that the reviewer seems to have missed:\n    * [25,29,10,7,5] are restricted to state-level feedback from the evaluator. Which can be a burden on the evaluator. The proposed method does not have the same restriction. Our results on the Circle and Mujoco based environments show that it is possible to collect feedback at the trajectory level and still obtain good performance.\n    * All existing methods except [7] do not consider the problem of selecting a subset of all the trajectories taken for feedback. Eliciting feedback for every trajectory taken can again lead to a significant burden on the evaluator.\nThey do not scale to complex environments with large state-action spaces. To the best of our knowledge, our paper is the only one to experiment on complex environments like the Safety Gymnasium benchmark, which is used to evaluate the performance of Constrained RL algorithms with known cost functions.\n    * Scalable methods have been proposed in Constraint Inference from demonstrations [6,30,20]. These methods assume the existence of constraint-abiding expert demonstrations which may not be readily accessible in all scenarios. Instead of generating expert trajectories, we only need evaluation of automatically generated trajectories by human/machine. \n    * Novelty-based sampling has been introduced in the context of exploration problems [8,26]. To the best of our knowledge, we are the first to apply it in feedback-based RL settings. Additionally, we extend the concept of novelty to trajectories, inspired by its similarity to edit distance between trajectories [24]. The effectiveness of novelty based sampling as a form of sampling states that have a high prediction error is shown in Figure 5 in the attached document, and the overall gain in performance from using this sampling method is present in Figures 1 and 7.\n\n    We would refer the reviewer to Section D in the Appendix for further details on the prior work.\n\n    Our analysis of Table 1 reveals that the proposed RLSF algorithm demonstrates notable improvements over the baseline approaches across all tested environments. The baseline methods faced challenges in balancing cost estimation and reward optimization. In some cases (Point Circle, Biased Pendulum, Blocked Swimmer, Half Cheetah, Point Goal, Point Push), the baseline approaches  tended to underestimate costs, resulting in higher cost violations. In others (Car Circle, Car Push), costs were overestimated, leading to suboptimal reward outcomes.\n    The RLSF algorithm, however, showcases consistent superior performance compared to these baseline methods. Additionally, it achieves results comparable to PPOLag in several environments which underscores the potential of RLSF as a promising approach for addressing the complexities of reinforcement learning tasks without knowing the underlying cost constraints and just having demonstrations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their thoughtful and insightful reviews. We're happy to see that reviewers E6xa, 1oHB, and w6J6 found our paper well-written and easy to follow, and that all reviewers recognized the importance of our work in the context of safe reinforcement learning.

Meta-review: This paper presents a novel approach to learning cost functions for constrained reinforcement learning from trajectory-level feedback, which is an important problem in safe RL. The proposed Reinforcement Learning from Safety Feedback (RLSF) algorithm shows strong performance across a range of benchmark environments, including Mujoco-based and Safety Gymnasium tasks. Based on the reviews and the authors' responses, I recommend accepting this paper though with some reservations.

The reviewers generally found merit in the paper's approach to learning cost functions for constrained reinforcement learning from trajectory-level feedback. The main strengths highlighted by the reviewers include the novel formulation of the problem, the ability to learn from coarse feedback, and the strong performance across a range of benchmark environments, including complex Mujoco-based tasks. The novelty-based sampling mechanism for efficient feedback collection was also viewed positively.

However, some limitations were noted across the reviews, such as about the novelty of the work and the lack of comparisons to other learning-based methods for cost inference. The authors have clarified several key differentiating factors of their work, including the ability to use trajectory-level feedback, selective feedback collection for scalability, and the novel application of novelty-based sampling in feedback-based RL settings. The authors' rebuttal provided some clarifications and additional results, including ablation studies on different data sources and visualizations of the learned cost functions. These additions help address some of the reviewers' concerns, particularly regarding the effectiveness of the method and its comparison to baselines.

To improve the final version, the authors should:

1. Include more comparisons to relevant cost inference methods.
2. Provide the additional ablation studies and visualizations mentioned in the rebuttal in the main paper.
3. Clarify the technical details that were found confusing, particularly around the surrogate loss derivation.
4. Address the limitations more thoroughly, especially regarding the simplicity of some experiments and the reliance on simulated feedback.

While the concerns about limited comparisons and experimental simplicity are valid, the overall positive assessment from the majority of reviewers and the potential impact of this work on safe RL justify acceptance. In conclusion, despite some limitations, this paper presents a novel and important contribution to the field of safe reinforcement learning, demonstrating a promising approach to learning cost functions from coarse feedback.
Decision: Accept (poster)