Safety through feedback in Constrained RL

Authors: shashank  reddy chirra
pradeep  varakantham
praveen  paruchuri
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 15:30:33

Abstract: In safety-critical RL settings, the inclusion of an additional cost function is often favoured over the arduous task of modifying the reward function to ensure the agent's safe behaviour. However, designing or evaluating such a cost function can be prohibitively expensive. For instance, in the domain of self-driving, designing a cost function that encompasses all unsafe behaviours (e.g., aggressive lane changes, risky overtakes) is inherently complex, it must also consider all the actors present in the scene making it expensive to evaluate. In such scenarios, the cost function can be learned from feedback collected offline in between training rounds. This feedback can be system generated or elicited from a human observing the training process. Previous approaches have not been able to scale to complex environments and are constrained to receiving feedback at the state level which can be expensive to collect. To this end, we introduce an approach that scales to more complex domains and extends beyond state-level feedback, thus, reducing the burden on the evaluator. Inferring the cost function in such settings poses challenges, particularly in assigning credit to individual states based on trajectory-level feedback. To address this, we propose a surrogate objective that transforms the problem into a state-level supervised classification task with noisy labels, which can be solved efficiently. Additionally, it is often infeasible to collect feedback for every trajectory generated by the agent, hence, two fundamental questions arise: (1) Which trajectories should be presented to the human? and (2) How many trajectories are necessary for effective learning? To address these questions, we introduce a \textit{novelty-based sampling} mechanism that selectively involves the evaluator only when the the agent encounters a \textit{novel} trajectory, and discontinues querying once the trajectories are no longer \textit{novel}. We showcase the efficiency of our method through experimentation on several benchmark Safety Gymnasium environments and realistic self-driving scenarios. Our method demonstrates near-optimal performance, comparable to when the cost function is known, by relying solely on trajectory-level feedback across multiple domains. This highlights both the effectiveness and scalability of our approach. The code to replicate these results can be found at \href{https://github.com/shshnkreddy/RLSF}{https://github.com/shshnkreddy/RLSF}

Main paper: 
 introduction: 
 Reinforcement Learning (RL) is known to suffer from the problem of reward design, especially in safety-related settings [4] . In such a case, constrained RL settings have emerged as a promising alternative to generate safe policies [3, 16, 34] . This framework introduces an additional cost function to split the task related information (rewards) from the safety related information (costs). In this paper, we address a scenario where the reward function is well-defined, while the cost function remains unknown a priori and requires inference from feedback. This arises in cases where the cost function is expensive to design or evaluate. For instance, consider the development of an autonomous driving system, where the reward function may be defined as the time taken to reach a destination which is easy to define. However, formulating the cost function presents significant challenges. Designing a comprehensive cost function that effectively penalizes all potential unsafe behaviors is non-trivial, and ensuring safety often involves subjective judgments, which can vary based on individual preferences. Even if one succeeds in devising such a function, it must account for all environmental factors, such as neighboring vehicles and pedestrians. The evaluation of such a cost function in high-fidelity simulators can be prohibitively expensive.Feedback can be collected from either a human observer, who monitors the agent's training process and periodically provides feedback on presented trajectories, or from a system that computes the cost incurred on selected trajectories. Throughout this paper, we use the term evaluator to refer to the entity providing feedback, whether human or system-generated.Cost inference in Constrained RL settings has gained recent attention. One key thread of research has focused on learning cost from from constraint abiding expert demonstrations [28, 32] . However, these expert demonstrations are not easily available in all settings. For example, consider robotic manipulation tasks where the human and the robot have different morphologies. This paper focuses on the second line of research in this area, where we generate trajectories and gather feedback from an evaluator to infer the underlying cost function. Prior works in this thread make limiting assumptions on the nature of the cost function such as smoothness or linearity [28, 32, 11, 5] and are limited to obtaining feedback at the state level [8] which is expensive to collect from human evaluators. We do not make such assumptions and can take feedback provided at over longer horizons, in some cases the entire trajectory.Frameworks for learning from feedback must exhibit the following properties as emphasized in [8, 18, 12] : 1) Feedback must be collected offline in between rounds, since the agent may need to act in real time. 2) The amount of feedback collected must be minimized. 3) Binary feedback is more suitable as compared to numeric feedback as it is more intuitive to provide for humans. It has also been shown that humans provide less consistent feedback if it is numeric [12] . 4) Each state is assigned a binary cost value, indicating that it is inherently safe or unsafe, which is more intuitive for humans when assessing the safety of policies [31] 1 .To this end we propose the Reinforcement Learning from Safety Feedback (RLSF) algorithm, which embodies all the aforementioned properties. The key contributions of this algorithm include:• Extends prior work to collect feedback over longer horizons. This is done by presenting the evaluator with the entire trajectory, breaking the trajectory into segments and eliciting feedback at the segment level. Inferring the costs directly by maximizing likelihood presents new challenges due to the problem of credit assignment over longer horizons. To tackle this, we present a surrogate loss that converts the problem from trajectory level cost inference to a supervised binary classification problem with noisy labels.• Introduces a novelty based sampling mechanism that reduces the number of queries by sampling novel trajectories for feedback.• Learns safe policies across diverse benchmark safety environments. We also show that the learnt cost function can be transferred to train an agent with different dynamics/morphology from scratch without collecting additional feedback. 
preliminaries: 
 Markov Decision Process A Markov Decision Process (MDP) M is defined by the tuple (S, A, P, r, γ, µ), where S denotes the set of states, A is the set of actions, P(s ′ |s, a) ∈ [0, 1] is the transition probability, r(s, a) ∈ R is the reward function, γ ∈ [0, 1] is the discount factor and µ(s) ∈ ∆(S) is the initial state distribution. A policy π(.|s) ∈ ∆(A) is a distribution over the set of valid actions for state s. We denote the set of all stationary policies as Π. A trajectory τ = {(s t , a t )} denotes the state-action pairs encountered by executing π in M. We use the short hand τ i:j to denote a trajectory segment, i.e, the subsequence of (s t , a t ) pairs encountered from timestep i to j. The expected value of a function f under π as J f (π) ≜ E τ ∼π [ ∞ t=0 γ t f (s t , a t )]. We also employ the shorthand f (τ ) to represent the discounted sum of f along the trajectory τ . The occupancy measure of a policy is define as ρ(s, a) = E τ ∼π [ ∞ t=0 γ t I[(s t , a t ) = (s, a)]], where I[.] denotes the indicator function. ρ describes the frequency with which a state-action pair is visited by π. [4] introduces a function c(s, a) ∈ R and a cost threshold c max ∈ R that defines the maximum cost that can be accrued by a policy. The set of feasible policies is defined as Π c = {π ∈ Π : J c (π) ≤ c max }. A policy is considered to be safe w.r.t c if it belongs to Π c .Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitations of the proposed approach are discussed in Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem definition: 
 In this paper, we consider the constrained RL problem defined as,EQUATIONWe assume the threshold c max is known, but the cost function is not known and must be inferred from feedback collected from an external evaluator. In many scenarios c max is typically known, representing a predefined limit on acceptable costs or risks in the environment. However, crafting the cost function c(s, a) such that it penalizes all unsafe behaviour can be infeasible.We incorporate an additional constraint enforcing the cost function to be binary, i.e, c(s, a) ∈ {0, 1}. This ensures that each state-action pair is inherently categorized as either safe or unsafe. We opt for this approach because it is simpler for human evaluators to assign a binary safety value to state-actions when assessing policy safety, as emphasized in [31] .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
method: 
 In this section, we introduce Reinforcement Learning from Safety Feedback (RLSF), an on-policy algorithm that consists of two alternating stages: 1) Data/Feedback collection and 2) Constraint inference/Policy improvement. In the first stage, data is collected via rollouts of the current policy for a fixed number of trajectories. Next, a subset of these trajectories is presented for feedback from evaluator, which are then stored in a separate buffer. The second stage consists of two parts: i) Estimation of the cost function from the feedback data and ii) Improvement of the policy using the collected trajectories and their inferred costs. We repeat stages (1) and (2) until convergence.First, we highlight how the feedback is collected and propose a method to infer the constraint function using this data. Next, we recognize the practical limitations of acquiring feedback for every trajectory during training and detail our approach to sampling a subset of trajectories for efficient cost learning of the cost function. Finally, we detail how the inferred cost function is used to improve the policy.In the feedback process, the evaluator is first presented with the entire trajectory τ 0:T . Afterward, the trajectory is divided into contiguous segments τ i:j of length k, and feedback is collected for each segment. The segment length can be adjusted based on the complexity of the environment: in simpler environments, feedback can be gathered for the entire trajectory, whereas for environments with long horizons and sparse cost violations, shorter segments may be used. This approach simplifies the challenge of assigning credit to individual states. Similar methods have been adopted in other works that rely on human feedback [12, 18] . However, reducing segment length comes at an increased cost of obtaining feedback from the evaluator.The evaluator is tasked with classifying a segment as unsafe if the agent encounters an unsafe state at any point within the segment. This decision was made to ensure consistent feedback from the evaluator. Alternative approaches-such as marking a segment unsafe based on the number of unsafe states visited or leaving the classification to the evaluator's subjective judgment are more prone to generating inconsistent feedback in the case of human evaluators [12] .Let P = {τ i:j , y saf e } represent the feedback collected from the evaluator, where y saf e = 1 if the segment was labelled safe and y saf e = 0 otherwise. We assume there exists an underlying ground truth cost function c gt (s, a) ∈ [0, 1] based on which the evaluator provides feedback. Then, the probability that a state is safe is defined as p saf e gt (s, a) = I[c gt (s, a) = 0]. Now, let p saf e (s, a) represent our estimate of p saf e gt (s, a) that we intend to estimate from the collected feedback. Then, by definition of how the feedback is collected, the probability that a trajectory segment τ i:j is labelled as safe is given by,EQUATIONWe can infer p saf e (s, a) by minimizing the likelihood loss, L mle = -E (τi:j ,y saf e )∼P y saf e log p saf e (τ i:j ) + (1 -y saf e ) log 1 -p saf e (τ i:j )= -E (τi:j ,y saf e )∼P y saf e j t=i log p saf e (s t , a t ) + (1 -y saf e ) log (1 -EQUATIONDirectly minimizing Eq 3 is challenging as the term j t=i p saf e (s t , a t ) would collapse to 0 when the segment length is long, causing unstable gradients. To address this issue, we propose using a surrogate loss function where we replace 1 -j t=i p saf e (s t , a t ) by j t=i (1 -p saf e (s t , a t )).L sur = -E (τi:j ,y saf e )∼P y saf e j t=i log p saf e (s t , a t ) + (1 -y saf e ) j t=i log (1 -p saf e (s t , a t ))= -E (τi:j ,y saf e )∼P The surrogate loss reformulates the objective from the segment level-where collapsing probabilities over long segments can be problematic-to the state level, where this issue does not occur. Optimizing Eq 4 involves breaking down the segments into individual states and assigning each state the label of the segment. Subsequently, states are uniformly sampled at random, and the binary cross-entropy loss is minimized. Consequently, L sur represents a binary classification problem in which one class contains noisy labels. While unsafe states are always accurately labeled-since their presence necessitates an unsafe classification for the segment-a safe state may receive conflicting labels based on the status of the segment it belongs to. Thus, with a sufficient number of samples, we believe it becomes feasible to reliably differentiate between safe and unsafe states. Proposition 1. The surrogate loss L sur is an upper bound on the likelihood loss L mle . Thus, minimizing L sur guarantees an upper bound on the likelihood loss of the estimated cost function.Having discussed the surrogate loss, we now examine the characteristics of its optimal solution. Proposition 2. The optimal solution to Eq 4 yields the estimate,EQUATIONSubsequently, we define the inferred cost function as c * (s, a) ≜ I[p saf e * (s, a) < 1 2 ]. Employing c * (s, a) in policy updates instead of c gt (s, a) introduces a bias in estimating the cost accrued by the policy, that we analyse below. For this analysis, we assume that the feedback is sufficient, i.e, the density d(s, a) is greater than zero for every state, otherwise p * (s, a) is not defined. Proposition 3. For a fixed policy π, the bias in the estimation of the incurred costs is given by,EQUATIONwhereρ π g (s, a) = E π [ T t=0 γ t [I[(s t , a t ) = (s, a) ∩ c gt (s, a) = 0]] is the occupancy measure of safe states visited by π.Proposition 3 illustrates that c * (s, a) misclassifies certain safe states as unsafe when their frequency in segments labeled unsafe exceeds that in segments labeled safe by the evaluator. We contend that this misclassification is likely to diminish with increased data collection or shorter segment lengths. However, it is important to note that this misclassification is guaranteed to be zero only when the segment length is reduced to one, meaning feedback is provided at the state level.Additionally, note that the bias is non-negative, meaning that the expected cost E π [c * (s, a)] acts as an upper bound on the true cost incurred by π. Therefore, ensuring that the policy does not exceed the threshold c max on c * guarantees that it adheres to the threshold on c gt . Corollary 1. Any policy π that is safe w.r.t c * is guaranteed to be safe w.r.t c gt .In practice, we represent p saf e θ (s, a) using a neural network with parameters θ. The resulting cost function is defined as c θ (s, a) ≜ I[p saf e (s, a)] < 1 2 .To reduce the burden on the evaluator and minimize the cost of feedback, we present a subset of the trajectories collected by the policy for feedback. The common approach is to break the problem into two parts: (1) define a schedule N queries (i) that determines the number of trajectories to be shown to the user at the end of each data collection round i. Subsequently, N queries (i) trajectories are sampled from those collected by the policy at data collection round i. While the ideal goal is to sample a subset of trajectories that maximizes the expected value of information [19] , achieving this is computationally intractable [2] . To address this challenge, various sampling methods have been employed, seeking to maximize a surrogate measure of this value. Among these, uncertainty sampling stands out as the most prominent approach, wherein trajectories are sampled based on the estimator's uncertainty about their predictions [12, 18, 20] . However, quantifying the uncertainty is challenging given the lack of calibration in neural network predictions. To address this challenge, ensemble methods are frequently employed where the disagreement among the models is used as an uncertainty measure. However, the training of n distinct neural networks can exact substantial resource costs, prompting consideration for alternative approaches.In light of this, we introduce a new form of uncertainty sampling called novelty sampling. With novelty sampling, we gather all the novel trajectories after each round and present them to the evaluator for feedback. Formally, we define a state as novel if its density in the feedback data collected so far d(s) = a d(s, a) is 0. A trajectory is deemed novel if it comprises of at least e novel states. This can be interpreted as ensuring that the edit distance-a well-known measure of trajectory distance [27] -between the current trajectory and previously seen trajectories exceeds a threshold e. We do not consider novelty for state-action pairs as we found that extending to this case adversely impacted the performance.The central notion is that the model is prone to errors on novel states-those it has not encountered during training. This arises because the policy evolves over time, venturing into states that were not previously encountered during data collection rounds. Therefore, this sampling strategy effectively reduces the epistemic uncertainty of the model-error arising from insufficient training data-thereby making it a form of uncertainty sampling. Furthermore, this sampling method offers the advantage of implicitly establishing a decreasing querying schedule as novelty of trajectories reduces over time as shown in Figure 6 in the Appendix.We compute the density d(s) through a count-based method, utilizing a hashmap to track the frequency of state occurrences in trajectories presented to the evaluator. Employing SimHash [10] , we discretize the state space using a hash function ϕ : S → {-1, 1} n , which maps locally similar states (measured by angular distance) to a binary code as:EQUATIONwhere g : S → R d is an optional prepossessing function and A is an n × d matrix with i.i.d entries drawn from a standard normal distribution. Also, note that n controls the granularity of the hash function, i.e, the number of states mapped to the same value. In our setting, g(s) = s as we observed no improvement when employing functions like autoencoders for feature extraction.After the data collection round, where we sample trajectories {τ } and their corresponding rewards {r(τ )} using π, we estimate their costs c(τ ) using the inferred cost function. Our proposed method allows for the policy to be updated utilizing any on-policy constrained RL algorithm. In this study, we employ the PPO-Lagrangian algorithm [3] that combines the PPO algorithm [26] with a lagrangian multiplier to ensure safety.A detailed description of the proposed method can be found in Algorithm 1. Lines [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] describe the data and feedback collection stage, and Lines [18] [19] [20] [21] [22] [23] describe the cost inference and policy improvement stage. for each trajectoryτ i ∈ {τ } do ▷ Feedback Collection 8: novel ← True if ∃ e states {s e } ∈ τ i such that d(ϕ(s e )) > 0 9:if novel then 10:Show τ i to the evaluator 11:for each segment τ j:j+k-1 ∈ τ i do 12:Obtain feedback y saf e for the segment τ i j:j+k-1 . Infer costs {c θ (τ i )} for all τ i ∼ {τ }.Update π using {r(τ i )} and {c θ (τ i )}. ▷ Policy Improvement 24: end while 
experiments: 
 We investigate the following questions in our experiments:1. Does RLSF succeed in effectively learning safe behaviours? 2. Can the inferred cost function be transferred across agents in the same task? 3. How does the proposed novelty based sampling scheme compare with other methods used in the literature?4. How accurate is inferred cost function compared to the true cost function?5. How can we address the overestimation bias of the inferred cost function as described in Section 4.2? We evaluate RLSF on multiple continuous control benchmarks in the Safety Gymnasium environment [17] and Mujoco [30] based environments introduced in [22] . The Circle, Blocked Swimmer and Biased Pendulum environments constrain the position of the agent whereas the Half Cheetah, Hopper and Walker-2d environments constrain the velocity of the agent. The Goal and Push tasks are the most challenging as they contain static and dynamic obstacles that the agent must avoid while completing the task.All of the above environments reflect safety challenges that an agent could potentially face in real-world scenarios. Additionally, we conduct experiments using the Driver simulator introduced in [21] , which presents two scenarios that an autonomous driving agent is likely to encounter on the highway: lane changes and blocked paths. In this setup, the cost function is based on multiple variables, including speed, position, and distance to other vehicles. Additionally, we introduce a third scenario: overtaking on a two-lane highway. A detailed description of the tasks can be found in the Appendix B.We split the environments into two settings: (1) hard constraint setting (c max = 0) where the safety of the policy is measured in terms of the cost violation (CV) rate defined as the number of cost violations divided by the length of the episode and (2) soft constraint setting where c max > 0. We utilize an automated script that leverages the underlying cost function to simulate the feedback provided by the evaluator.We compare the performance of our algorithm against the following baselines: Self Imitation Safe Reinforcement Learning (SIM) [16] : SIM is a state-of-the-art method in constrained RL that also supports the case where feedback is elicited from an external evaluator. Similar to RLSF, the method consists of two stages, a data collection/feedback stage and a policy optimization stage. In the first stage, a trajectory τ is labelled asgood if [r(τ ) ≥ r good ∩ I[c(τ ) ≤ c max ], and labelled as bad if [r(τ ) ≥ r bad ∪ I[c(τ ) ≥ c max ],where r good and r bad are predefined thresholds on the reward. The information I[c(τ ) ≤ c max ] is received from feedback. The idea is then to imitate the good trajectories and stay away from the bad trajectories. If ρ π is the occupancy measure of the current policy π and ρ G , ρ B are the occupancy measures of the good and bad trajectories respectively, then, π is optimized as,EQUATIONwhere ρ π,G = (ρ π + ρ G )/2 and KL denotes the Kullback-Leibler divergence.Safe Distribution Matching (SDM): SIM combines rewards with safety feedback into a joint notion of good and bad which may not be desirable when the cost function is unknown. Thus, we introduce an additional baseline that keeps these two signals separate by labelling the trajectory as good if c(τ ) ≤ c max and bad otherwise. The policy is then updated as,EQUATIONwhere λ ∈ [0, 1] controls the tradeoff between the two objectives.As an upper bound, we compare the performance of our algorithm to scenarios where the cost function is known: PPO-Lagrangian (PPOLag) [3] and SIM with known costs (SIMKC) [16] . The objective is not to surpass their performance, but to match it. Therefore, when reporting the results of these algorithms, we present the best-performing seed across multiple runs.All results presented use novelty-based sampling unless stated otherwise. We use a segment length of 1 in the Driver, Goal and Push environments. This is because the Safety Goal and Push environments contain small obstacles that the agent interacts with for very brief periods of time, hence requiring more fine-grained feedback. An example of this is present in Figure 11 in the Appendix. In the Driver environments, a randomly initialized policy was highly unsafe. Thus a long segment length would force the evaluator to label every segment unsafe, making cost inference infeasible. In the remaining environments, the segment length corresponds to the length of the episode. Details on the number of queries generated for feedback can be found in Table 4 in the Appendix. We grant the two baseline methods (with unknown costs) an advantage by providing feedback for every trajectory generated.Benchmark Environments Table 1 presents the performance of the various algorithms on the benchmark environments. RLSF significantly outperforms the two baselines in terms of reward and safety in all of the environments2 . RLSF comes to within ≈ 80% of the performance of the best run of PPOLag in 7/11 environments thereby underscoring its effectiveness in learning safe policies.Driving Scenarios In the driving environments, we evaluate the performance of our algorithm against two baseline methods: a naive PPO policy that solely maximizes reward and the PPO-Lag algorithm. Figure 1 presents these results. Notably, our method demonstrates comparable safety performance to PPO-Lag while significantly outperforming PPO in terms of cost violations. This highlights the effectiveness of our approach in learning safe autonomous driving policies. Video clips showcasing the learned policies in the 3 scenarios are included in the supplementary material.We also showcase the potential of using the inferred cost function to train an agent with a different embodiment or morphology to solve the same task from scratch, without requiring any additional feedback. We utilize the inferred cost function obtained during the training of a Point agent to train the Doggo agent for the Circle and Goal tasks. The observation space in the environments consists of agent-related telemetry (acceleration, velocity) and task-related information (lidar sensors for goal/boundary/obstacle detection). In these experiments, we do not incorporate agent-related information when learning the cost function, ensuring that it can be transferred to a new agent. Next, the inferred cost function remains fixed during the training of the second agent, and the task-related features are utilized for cost inference. We compare the performance of this approach with that of the PPO-Lagrangian algorithm that utilizes the underlying cost function of the environment. From Table 2 we can infer that agents trained using transferred cost function are comparable in performance to agents trained using the true underlying cost function. [18, 20] . These methods require a predefined budget for the number of trajectories presented to the evaluator. We evaluate two schedules: (1) Uniform schedule, where a fixed number of trajectories is shown for feedback after each round, and (2) Decreasing schedule, where the number of trajectories decreases in proportion to 1 t each round. For each schedule, we test the following strategies for sampling the subset of trajectories to be presented to the evaluator: (a) Random sampling, which selects a random subset of trajectories, and (b) Entropy sampling, where trajectories are sampled in descending order of the average entropy H(p saf e θ (s t , a t )) of the estimator. As shown in Figure 2 , entropy based sampling outperforms random sampling, and a decreasing schedule outperforms a uniform schedule. However, all the methods fall short compared the proposed novelty based sampling mechanism. To further evaluate the quality of the inferred reward function, we plot it alongside the true underlying cost function of the environment as presented in Figure 3 . In the Point Goal environment, where feedback is collected for each state, the inferred cost function closely aligns with the true cost given sufficient data. However, in the Point Circle environment, where feedback is collected at the trajectory level, the inferred cost function exhibits a slight overestimation bias, as discussed in Section 4.2. Nevertheless, from Figure 3 , we can conclude that our method learns cost functions that strongly correlate with the true costs.In Section 4.2, we established that when feedback is collected over segments, the inferred cost function tends to exhibit an overestimation bias. As a result, optimizing for a policy that is deemed safe with respect to c max using the inferred cost function can lead to overly conservative policies. If the bias b, could be calculated apriori, then adjusting the safety threshold to c max + b would still satisfy the safety guarantees outlined in Corollary 1. However, since estimating b in advance is infeasible, we propose a heuristic approach where a constant δ ∈ R + is added to c max to account for the bias. It's important to note that Corollary 1 only holds if δ ≤ b, meaning this heuristic doesn't guarantee that the resulting policy will always be safe. Nevertheless, we tested this approach with different δ values in the Car Circle environment, as this setting exhibited a higher overestimation bias (as reflected in Table 1 ). The results, shown in Figure 4 , demonstrate that adding this bonus does improve performance (with higher δ values yielding better results), though it introduces the additional challenge of tuning δ.6 Limitations, Future Work and Broader Impact In this work, we consider the problem of learning cost functions from feedback in scenarios where evaluating or defining such functions is either costly or infeasible. We present a novel framework which significantly reduces the burden on the evaluator by eliciting feedback over extended horizons, a challenge that has not been addressed in previous research. To further reduce their load, we propose a novelty-based sampling method that only presents previously unseen trajectories to the evaluator. Through experiments on multiple benchmark environments, we demonstrate the effectiveness of our framework and the proposed sampling algorithm in learning safe policies. However, there are a few limitations we would like to acknowledge. First, our method relies on state-level feedback in some environments, which can be expensive to obtain. Second, while we simulate feedback using the true cost function, real-world feedback is often noisy, especially when provided by human evaluators. It would be valuable to conduct experiments involving real human subjects to validate the approach.By enabling safer autonomous systems, our approach has the potential to enhance safety across a range of applications, including autonomous vehicles, drones used for delivery, and industrial robots operating in environments like warehouses or manufacturing plants. These improvements could significantly reduce accidents, protect human operators, and increase the overall reliability of automated systems.A Proofs Proposition 1. The surrogate loss L sur is an upper bound on the likelihood loss L mle . Proof. We have, L sur -L mle = -E (τi:j ,y saf e )∼P ( which we prove subsequently.Lemma 1. For any sequence of numbers {x 1 , . . . , x n }, where x i ∈ [0, 1] for all i ∈ 1, . . . , n we have 1 -n i=1 x i ≥ n i=1 (1 -x i ).Proof. Using the AM-GM inequality we have,EQUATIONSimilarly, we have,EQUATIONAdding Eq 10 and Eq 11, we get,EQUATIONProposition 2. The optimal solution to Eq 4 yields the estimate,p saf e * (s, a) = d g (s, a) d g (s, a) + d b (s, a)Proof. The proof is obtained by differentiating Eq 4 w.r.t p saf e (s, a) and setting it to 0. Proposition 3. For a fixed policy π, the bias in the estimation of the incurred costs is given by, Proof. From Proposition 3, we know thatE π [c * (s, a)] -E π [c gt (s, a)] = E (s,a)∼d π g [I[d b > d g ]] where ρ π g (s, a) = E π [ T t=0 [I[(s t , a t ) = (E π [c * (s, a) -c gt (s, a)] = E τ ∼π T t=0 [c * (s t , a t ) -c gt (s t , a t )] = E τ ∼π T t=0 s,a I[(s t , a t ) = (s, a)][c * (s t , a t ) -c gt (s t , a t )] = E τ ∼π T t=0 s,a I[(s t , a t ) = (s, a) ∩ c gt (s, a) = 0] c * (s t , a t ) -c gt (s t , a t ) + I[(s t , a t ) = (s, a) ∩ c gt (s, a) = 1] c * (s t , a t ) -c gt (s t , a t ) = E τ ∼π E (s,E π [c * (s, a)] -E π [c gt (s, a)] = E (s,a)∼ρ π g [I[d b > d g ]]. Since E (s,a)∼ρ π g [I[d b > d g ]] ≥ 0, we have E π [c * (s, a)] ≥ E π [c gt (s, a)]. Thus, E π [c * (s, a)] ≤ c max =⇒ E π [c gt (s, a)] ≤ c max .Mujoco [30] -based environments extend traditional locomotion tasks by incorporating safety constraints. These environments are categorized into two types: position-based and velocity-based constraints. A detailed description of each is provided below:1. Position Based: These environments, recently introduced as benchmarks in [22] , are explained in more detail below.(a) Blocked Swimmer: The agent controls a robot composed of three segments joined at two points. The objective is to move the robot to the right as quickly as possible by applying torque through rotors at the joints. The agent receives a reward at each time step, proportional to its displacement in the X-direction. The episode concludes after 1000 time steps. To increase the challenge, a cost is imposed when the agent's X-coordinate exceeds 0.5, constraining it to the region -∞ ≤ X < 0.5. (b) Biased Pendulum: The agent's goal is to balance a pole on a cart. Each episode ends if the pole falls or after a maximum of 1000 time steps. The agent receives a reward of 1 when its X-coordinate is within the range (-∞, -0.01], which monotonically decreases to 0.1 as the X-coordinate approaches 0. Beyond this, the agent receives a constant reward of 0.1, encouraging movement in the leftward direction. To prevent excessive leftward movement, a cost of 1 is incurred when the X-coordinate is in the region (-∞, -0.015], constraining the agent from moving too far left.2. Velocity Based: These environments are part of the Safety Gymnasium benchmarks, introducing a velocity constraint for the Half-Cheetah, Hopper, and Walker-2d agents. The agent incurs a cost of 1 whenever it exceeds a velocity threshold of v max /2, where v max is the maximum velocity achieved by the agent when trained with PPO for 1 × 10 7 steps.Safety Gymnasium [17] has emerged as a prominent benchmark for evaluating constrained reinforcement learning algorithms, providing a challenging framework for cost inference. The environments feature multiple agents, including the Point, Car, and Doggo, listed in increasing order of difficulty in controlling the agent. The objective is to complete specific tasks using one of these agents.In the Circle task, the agent must navigate around the boundary of a circle centered at the origin while remaining within a safe region defined by two vertical boundaries at x = ±0.7. A cost of 1 is incurred each time the agent exceeds this boundary (|x| ≥ 0.7).In the Goal task, the agent must reach a designated goal area as quickly as possible while avoiding both static and dynamic obstacles, incurring a cost of 1 for each collision.The Push task builds upon the Goal task by adding the challenge of pushing a block to the goal area while still avoiding collisions with obstacles in the environment.Driving-based environments are emerging as key benchmarks for addressing the problem of constraint inference [22] , particularly due to the complex safety considerations involved in driving. We evaluate our algorithm using the Driver simulator introduced in [21, 24] to learn safe behaviors from feedback. This environment encompasses two scenarios that a self-driving agent is likely to encounter on the highway: a blocked road and a lane change. Additionally, we introduce a third scenario that involves overtaking on a two-lane highway, where the agent must safely pass a slower car in its lane by utilizing the second lane. This maneuver poses risks, as traffic in this lane is moving in the opposite direction, necessitating careful execution. Below, we provide a brief description of the environment.The Driver environment utilizes point-mass dynamics and features continuous state and action spaces.The state of a vehicle in the simulation is represented by the tuple (x, y, ϕ, v), which includes the agent's position (x, y), heading ϕ and velocity v. The observation of the ego vehicle is formed by concatenating the states of all vehicles in the scene. The environment allows for two actions (a 1 , a 2 ) where a 1 represents the steering input a 2 denotes the applied acceleration.The state of the agents evolves as,s t+1 = (x + δx, y + δy, ϕ + δϕ, v + δv) (δx, δy, δϕ, δv) = (v cos ϕ, v sin ϕ, a 2 -αv)where α is used to control the friction. Additionally, the velocity is clipped to the range [-1, 1] at each timestep.The task is to control the ego vehicle (white in Figure 5 ) to reach the top of the road as soon as possible while adhering to costs on the velocity and proximity to other vehicles. The trajectories of the other vehicles are fixed, with some random noise applied to their steering and acceleration inputs.Figure 5 : Driver EnvironmentsThe agent receives a reward r t = 10(y t -y t-1 ) at each time step, incentivizing it to navigate through traffic rapidly. Additionally, it incurs a penalty of -1 for going off the road, driving backward, or failing to stay in the center of the lane. A collision results in a penalty of -100. The episode terminates when the agent reaches the top of the road or collides with another vehicle. Although these constraints can be integrated into the reward function, they can still lead to unsafe behaviors, highlighting the necessity for an additional cost function. The PPO agent is trained using this reward function.In the CMDP settings, the cost function incorporates some of the constraints from the reward function.For instance, the agent incurs a cost of 1 when it goes off the road or drives backward. Additional costs are incurred when the agent crosses the speed limit v max or gets too close to any nearby vehicles. The ego car is deemed to close to a neighbouring vehicle if exp -b(c 1 d 2 x + c 2 d 2 y ) + ba ≥ 0.Constrained RL The Constrained Markov Decision Process (CMDP) framework [4] has emerged as a valuable tool in safety-critical settings, offering a clear separation between task-related information (rewards) and safety-related information (costs). This separation simplifies environment specification and allows for easier transfer of cost functions across similar environments. While previous research has extensively addressed this problem [1, 35, 34, 16] , these studies typically assume that the cost function is known. However, this assumption can be limiting when designing the cost function is complex or costly to evaluate. Consequently, there is a need to infer the cost function from external data in such cases.Cost Inference Historically, cost inference has primarily depended on two types of data: (1) evaluator feedback and (2) constraint abiding demonstrations obtained from an expert.Previous work on learning from feedback often relies on limiting assumptions such as deterministic transitions and smoothness [28, 32, 11] or linearity assumptions [5] on the cost function. Moreover, none of these approaches address the problem of actively querying humans to minimize the required feedback. Although this has been theoretically explored in [8] , the proposed algorithm has not yet been adapted for complex environments. Additionally, these methods are typically limited to feedback on individual state-action pairs, which can be expensive when relying on human evaluators. Our RLSF approach addresses these limitations by reducing the amount of feedback required and avoiding assumptions about the cost function.Inferring cost functions from constraint-satisfying expert demonstrations has gained significant attention recently [6, 33, 22] . However, acquiring such expert demonstrations can be challenging in many scenarios. An alternative form of feedback is human interventions. For instance, [23] rely on the human evaluator to terminate the episode when the agent is about to engage in unsafe behavior, while [25] depend on the human to take control of the agent in such situations and guide it to a safe state.Novelty-Based Sampling Novelty estimation has been widely explored in the context of active exploration in reinforcement learning. The connection between prediction accuracy and state novelty has been extensively studied, as highlighted by [9] . Additionally, state density estimation has been investigated in works such as [29, 7, 13] , where infrequently visited states are assigned an exploration bonus. Notably, [29] also uses Simhash [10] to maintain state density estimates.Reward Inference This work also connects to the analogous problem of reward inference in standard MDPs, which similarly rely on human feedback [12, 18, 20] and expert demonstrations [15, 14] .We conducted the experiments on a cluster quipped with 4 NVIDIA RTX A5000 GPUs and 96 core CPUs. The experiments took an approximate of two weeks to run. The detailed hyperparameters utilized in the experiments are presented in Table 3 . The training curves for the benchmark experiments and the Driver environment are presented in Figure 7 and Figure 8 . Additionally information on the number of queries presented for feedback is presented in Table 4 .Figure 6 shows the implicit decreasing schedule encountered when using novelty based sampling in various environments. Figure 9 shows additional comparison with other querying mechanisms in the Biased Pendulum environment. To further assess the effectiveness of novelty based sampling, we categorized the trajectories collected during a data collection round based on novelty. We then evaluated the model's accuracy on the pseudo-prediction task for these two trajectory types, as The number of trajectories queried for feedback is of the order of O(1e 3 ), the exact number differs across environments and individual runs due to novelty based sampling. Detailed information on the number of queries can be found in Table 4 . We provide the two baseline methods with an advantage, whereby feedback is obtained for every trajectory generated by the policy, which is of the order O(1e 4 ).Figure 11 highlights the difficulty of inferring constraints when cost violations are sparse. The brief interaction periods with obstacles in the environment make it challenging to assign credit to unsafe states when feedback is collected over long horizons. Consequently, feedback was collected at the state level in such settings. This is illustrated in Figure 12 , where the overestimation bias is considerably higher in the Point Goal environment compared to Point Circle, resulting in a significant drop in performance. We conducted an ablation study on the size of the feedback buffer, the results of which are presented in Figure 13 . We can observe that the policy becomes overly conservative when the size of the feedback buffer is small (100k, which corresponds to approximately two data collection rounds). We hypothesize that this is because of two interacting factors: (a) discarding past feedback can result in forgetting, which in turn leads to inaccuracies in estimating unsafe regions, and (b) there exists a feedback loop in which the inferred cost function influences on-policy data collection. If the inferred cost function overestimates costs in certain regions, the policy will avoid them and fail to gather the necessary feedback to correct this error. Conversely, if the costs are underestimated, the policy will visit those states and receive feedback, allowing for corrections. Therefore, smaller feedback buffer sizes can result in the learning of overly conservative policies.In Section 4.2, we highlighted the challenges associated with minimizing Eq 3 particularly due to the term j t=i p saf e (s t , a t ) which tends to collapse to 0 with longer segment lengths, resulting in unstable gradients. To investigate this further, we conducted a simple experiment in which we performed a few gradient steps to optimize Eq 3 and plotted the gradient norm. The results, shown in Figure 14 reveal that (a) directly optimizing Eq 3 produces gradients with a large norm that either quickly vanish or explode, rendering the learning process infeasible. In contrast, the surrogate objective in Eq 4 yields stable gradients allowing for convergence. NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The social impacts of our work is discussed in Section 6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The work presented in this paper has no risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The README file present in the code credits the authors, and the license and terms of use of has been properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. The last two points are specific to human evaluatorsexcept the Car Goal environment where RLSF and PPOLag marginally exceed the threshold 
 
 Appendix:  


Decision: This paper presents a novel approach to learning cost functions for constrained reinforcement learning from trajectory-level feedback, which is an important problem in safe RL. The proposed Reinforcement Learning from Safety Feedback (RLSF) algorithm shows strong performance across a range of benchmark environments, including Mujoco-based and Safety Gymnasium tasks. Based on the reviews and the authors' responses, I recommend accepting this paper though with some reservations.

The reviewers generally found merit in the paper's approach to learning cost functions for constrained reinforcement learning from trajectory-level feedback. The main strengths highlighted by the reviewers include the novel formulation of the problem, the ability to learn from coarse feedback, and the strong performance across a range of benchmark environments, including complex Mujoco-based tasks. The novelty-based sampling mechanism for efficient feedback collection was also viewed positively.

However, some limitations were noted across the reviews, such as about the novelty of the work and the lack of comparisons to other learning-based methods for cost inference. The authors have clarified several key differentiating factors of their work, including the ability to use trajectory-level feedback, selective feedback collection for scalability, and the novel application of novelty-based sampling in feedback-based RL settings. The authors' rebuttal provided some clarifications and additional results, including ablation studies on different data sources and visualizations of the learned cost functions. These additions help address some of the reviewers' concerns, particularly regarding the effectiveness of the method and its comparison to baselines.

To improve the final version, the authors should:

1. Include more comparisons to relevant cost inference methods.
2. Provide the additional ablation studies and visualizations mentioned in the rebuttal in the main paper.
3. Clarify the technical details that were found confusing, particularly around the surrogate loss derivation.
4. Address the limitations more thoroughly, especially regarding the simplicity of some experiments and the reliance on simulated feedback.

While the concerns about limited comparisons and experimental simplicity are valid, the overall positive assessment from the majority of reviewers and the potential impact of this work on safe RL justify acceptance. In conclusion, despite some limitations, this paper presents a novel and important contribution to the field of safe reinforcement learning, demonstrating a promising approach to learning cost functions from coarse feedback.

Reviews: review: paper_summary: paper_summary: Authors propose a new way of estimating the cost function for constrained RL through user feedback. They propose a surrogate loss that is used to train a model that estimates the probability that a state-action pair is safe. They then use the model to estimate the cost of a policy and adhere to the constraint while looking for the best reward.
strength_weakness: strength_weakness: The experiment section lacks some details and is much harder to follow. 

I did not see any experiments (ablations) on the size of the feedback buffer. 

Also I was hoping to see some discussion around the introduced bias in estimation with differing c_max values.
questions: questions: Could authors explain the DNN that estimates the probability of safety for a (s, a) pair?
seems like they are directly estimating the probability? 
I am curious, if the authors have considered estimating the d_g and d_b densities (as defined in proposition 2 for the optimal solution) instead?
limitations: limitations: How is novelty sampling (which seems naive to me) related to uncertainty sampling? The proposed novelty sampling, roughly, ignores the trajectories that have many visited sub-sequences. 

I think Corollary 1. is theoretically wrong, as the authors have conveniently ignored the inherent inaccuracies in user feedback. Authors mention labeling noise as the noise that is the result of expanding an unsafe label to all the pairs in the trajectory but there is also the meta noise that the user labeling(user feedback) by itself is an error-prone process.
paper_summary: paper_summary: This paper proposes a surrogate loss function, which instead of collecting a feedback over trajectory-based evaluator, it breaks long trajectories into segments and the evaluator classifies and assigns labels to segments unsafe if any individual state is unsafe during the segments. This paper also proposes a novelty sampling, which only gathers novel trajectories for feedback from evaluator based on the trajectory distance measure edit distance.
strength_weakness: strength_weakness: 1. I found it hard to follow the paper, especially in section 4.2. Some questions are raised in the following section.
2. I understand that Eq 4 is an upper bound for Eq3, but I doubt that the difference between Eq3 and Eq4 might be too huge, and in this case, switching to Eq4 might result in a very inaccurate estimate $p_*^{\mathrm{safe}}$ for the following analysis in the paper.
questions: questions: 1. On line 136, the probability of a state being safe is defined as $\mathbb{1}[c_{\mathrm{gt}}(s, a) = 0]$. Isn't this an indicator function instead of assigning a probability to this specific state? Or this is a typo.
2. What does the ``noisy labels`` mean on line 150? This phrase also appears somewhere else and I am not sure what it refers to. Also for the sentence following that, ``However, if it occurs in a segment labeled unsafe, it is incorrectly labelled unsafe``. Does it mean if the segment is labelled as unsafe, some of the individual states in the segment ought to be safe but they are labeled as unsafe then?
3. Why is it true that ``with a sufficient number of samples, the number of correct labels will outweigh the number of incorrect lables for each state``?
4. I can't quite follow the paper but based on my understanding, there's a human evaluator to evaluate segment, and another evaluator for state?
limitations: limitations: Both limitations and societal impacts are discussed in the paper.
paper_summary: paper_summary: This paper presents a method for using evaluator feedback as safety constraints in a constrained RL framework. The novelty comes from the nature of the feedback and the sampling scheme. It is motivated by the fact that previous approaches have various limitations: designing cost, especially comprehensively, is expensive and even seemingly impossible a priori. Offline feedback can help, but existing approaches often don't scale, and/or are constrained to receive feedback at the state level, which is also expensive. Giving feedback at the trajectory level is an option, but also a difficult state-credit assignment problem; and even getting feedback for every trajectory is expensive. 

This paper presents RLSF, a method in which binary feedback is given on segments or entire trajectories and a novel surrogate loss is presented to make this a useful learning signal, then a new "novelty-based sampling mechanism" is used to train efficiently.

The paper takes us through the methods, including the feedback collection process (human or procedural evaluators), the policy improvement step, the nature of the feedback (1 if every state in a given segment is safe, 0 otherwise), and how the cost function is inferred from the collected data. It also presents a surrogate loss function for turning segment-level feedback into a dense signal. 

The paper then goes through experiments and results in the Safety Gym and Driver simulation environments. The tasks have various constraints in position and velocity. Experiments compare to three general baselines - self-imitation safe RL, safe distribution matching, and PPO where the true cost function is known (considered an upper bound). The methods are tested directly on the tasks, then learned cost functions are transferred to a new agent. The paper then goes through ablations removing the novelty-based sampling method. The results show that this RLSF algorithm shows significant improvement over baselines.
strength_weakness: strength_weakness: ### Quality
- "We posit that with a sufficient number of samples, the number of correct labels will outweigh the number of incorrect labels for each state" - doesn't this require sparsity of safety violations and small segments (meaning a hefty feedback load)? Questions like this would require answers to judge if this work is useful. 
- It would be good to see more error modes - for example, the paper cites novel states as a hotbed for potential errors, but it's only one. What are the others?
- In the experiments section, it would be good to see cost transfer compared to other cost function learning methods, not just PPO vs. RLSF vs. PPOLag.

### Clarity
- Tables should have bolded numbers, even if all the bolding will be in one column. The tables are incredibly dense right now and tiring to read. 
- Explanation of surrogate loss derivation would benefit from more justification. E.g. I see that it's true that the product term substitution makes it a state-level term instead of a segment-level term, but explain that. more clearly and why it's okay to do and still expect the optimization to work out.
questions: questions: - The paper cites sparse cost violation as a motivation for collecting feedback on shorter segments. Why is that? Is the idea that when cost violations are dense, it's hard to give more specialized feedback, but when they're more sparse then the evaluator can just go for it? Or is it purely a horizon issue (shorter horizons have fewer cost violations), in which case mentioning sparse cost violations at all doesn't make sense?
limitations: limitations: None
paper_summary: paper_summary: This paper studies the safe RL problem with an unknown cost function. As an on-policy approach, this work tries to learn the cost function with safety feedback from an evaluator with novelty sampling and conducts policy optimization at the same time. The authors propose that their approach can deal with feedback over longer horizons by a surrogate loss, reduce sampling queries by novelty sampling, and demonstrate the effectiveness via a set of experiments.
strength_weakness: strength_weakness: 1. The novelty of this paper might not be enough. First, it is not clear to me the difficulty of inferring the cost function in RL, current existing approaches, and how the proposed approach outperforms the base approaches. Second, novelty sampling or frequency-based sampling is well-studied and leveraged in classical RL. Third, there is nothing new in policy optimization in this paper. 
2. It is not clear to me that how close the learned cost function is to the ground truth cost function.
questions: questions: 1. In section 4.2, why do you formulate the safety/cost function by a probability measure? It is confusing to me. Therefore, I cannot understand why using maximum likelihood to learn the cost function. 
2. You have to show the learned cost function is close to the ground truth in the experiments. Otherwise, I cannot trust the experimental results as there are so many uncertainties/hyper-parameters in RL. 
3. A suggestion for this paper is to consider LLM as an evaluator.
limitations: limitations: The authors discussed their limitations in the conclusion.

Comments: title: Thanks for the response
Please Refer Figure 1 in the attached PDF not main paper.
comment: Hi, thanks for the detailed response. All the changes look good. They were mainly about presentation and detail, so they wouldn't move me to change my score, but I do think they make the paper a better read.
I thank authors for addressing my questions. I am not an expert in this field, and I'm willing to raise my score from 4 to 5.
It appears there's been a misunderstanding regarding Figure 1. We're specifically referring to Figure 1 in the PDF attached to the common rebuttal, not the Figure 1 in the paper. This figure shows that the inferred cost function is indeed close to the ground truth cost function.

Regarding the differentiating factors, we respectfully disagree with the reviewer's assessment that they are insignificant. The challenge of designing cost or reward functions for every state-action pair is substantial, as there can be unintended consequences due to interaction of the costs. Our contributions are significant because they enable the integration of qualitative feedback on safety of trajectory segments, which is otherwise difficult to quantify. For example, consider a Roomba robot cleaning a house. If the owner indicates that a certain movement around the sofa was unsafe because the robot unexpectedly emerged from behind it, our approach can incorporate this feedback to prevent similar behavior in the future. This capability is crucial in creating systems that can adapt and improve based on real-world, qualitative safety feedback.
I appreciate the feedback from the authors. BCE and MLE are fine to the reviewer because they are common and standard for a binary classification task. 

Unfortunately, I don't see the current Figure 1 in the paper shows any comparison between learned cost function and ground truth, which further adds to my confusion. 

As for the multiple differentiating factors, I don't feel they are significant. Therefore, I will keep my current score.