Counterexample Guided RL Policy Refinement Using Bayesian Optimization
Authors: Briti Gangopadhyay
Pallab Dasgupta
Venue: NeurIPS 2021 Poster (2021)
Keywords: Safe Reinforcement Learning
Bayesian Optimization
Proximal Policy Optimization

Abstract: Constructing Reinforcement Learning (RL) policies that adhere to safety requirements is an emerging field of study. RL agents learn via trial and error with an objective to optimize a reward signal. Often policies that are designed to accumulate rewards do not satisfy safety specifications. We present a methodology for counterexample guided refinement of a trained RL policy against a given safety specification. Our approach has two main components. The first component is an approach to discover failure trajectories using Bayesian optimization over multiple parameters of uncertainty from a policy learnt in a model-free setting. The second component selectively modifies the failure points of the policy using gradient-based updates. The approach has been tested on several RL environments, and we demonstrate that the policy can be made to respect the safety specifications through such targeted changes.

Counterexample Guided RL Policy Refinement Using Bayesian Optimization
Briti Gangopadhyay"
Pallab Dasgupta
Department of Computer Science Indian Institute of Technology Kharagpur briti aarr__ggggopachayliitteee in
Department of Computer Science Indian Institute of Technology Kharagpur pallab$cse. .itteeeeeeiiikppp....ee ac. in
Abstract
Constructing Reinforcement Learning (RL) policies that adhere to safety require- ments is an emerging field of study. RL agents learn via trial and error with an objective to optimize reward signal. Often policies that are designed to accumu- late rewards do not satisfy safety specifications. We present a methodology for counterexample guided refinement of a trained RL policy against given safety specification. Our approach has two main components. The first component is an approach to discover failure trajectories using Bayesian Optimization over multiple parameters of uncertainty from a policy learnt in model-free setting. The second component selectively modifies the failure points of the policy using gradient-based updates. The approach has been tested on several RL environments, and we demon- strate that the policy can be made to respect the safety specifications through such targeted changes.
Introduction
Classical Reinforcement Learning (RL) is designed to perceive the environment and act towards maximizing a long term reward. These algorithms have shown superhuman performances in games
like GO [10], motivating their in safety -criticll domains like autonomous driving [23] 20 16]. These policies are mostly learnt in simulation environments such as TORCS [22] or CARLA [4
for accelerated learning. Therefore they susceptible to perform poorly in the real world where domain uncertainty arises due to noisy data. Also, the choice of a RL policy is determined by the design of the reward signal, which not constructed properly, may lead the agent towards unintended or even harmful behaviour. Hence the need for constructing policies that account for risks arising due uncertainty and do violate safety specifications.
Safe RL aims to learn policies that maximize the expectation of the return while respecting safety specifications [17]. The existing literature safe RL is broadly divided into three schools of thought namely, 1) Transforming the optimization criteria by factoring in costs derived from safety constraints [I), Using external knowledge in the form of teacher
safety shield to replace unsafe actions chosen by the policy during exploration / deployment (2), and 3) Establishing the existence of an inductive safety invariant such Lyapunov function [1I] control barrier function (25). Establishing such an invariant is often intractable for Neural Network controllers that use many parameters and work on high- ligh-dimensional state spaces.
In this work, we propose a counterexample guided policy refinement method for verifying and correcting policy that has already been optimized and calibrated. Failure traces may occur in such a policy due to the following reasons: a) Failure trajectories are extremely rare and do not contribute significantly enough reduce the expected reward, b) Uncertainty in the environment
"Code available online at httts://gittbb.cmmbbrriggggll rettnaam bo
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

which had not been accounted for during policy training. Our aim is to progressively make the system safer by finding and eliminating failure traces with targeted policy updates. Counterexample guided refinement techniques have been well studied in the formal methods community for program verification
9 and reachability analysis of hybrid systems
but not so far in the context of RI..
We study the proposed methodology on environments that work over continuous space and continuous or discrete actions. The proposed methodology is divided into two steps:
Given a policy Tolds learnt for optimizing reward in a given environment, we test against parameters with uncertainties and set of objective functions yp derived from the negation the given safety criteria. Multiple counter-example traces or failure trajectories are uncovered from Told using Bayesian Optimization.
2. Using the failure trajectories we selectively do a gradient update on Told to construct a new policy Rnew that excludes the counterexample traces under the given domain uncertainties. We specifically work with policy gradient RL algorithm, Proximal Policy Optimization (PPO), which directly optimizes the learnt policy using gradient updates.
These steps may be repeated multiple times. We show that such targeted updates ensure that Tnew is safer than Told, but not significantly different in terms of reward.
The paper organized follows. Section outlines related work, Section 3 presents the methodology to uncover failure trajectories, Section presents the methodology of policy refinement, Section presents case studies on several RL environments, and Section dection/6/provinn provides concluding remarks.
2 Related Work
The approach proposed in this paper uses Bayesian Optimization for finding adversarial tests cases in terms of trajectories for the RL policy leading to failure of the safety specification. The failures then used to make the policy safer.
Bayesian Optimization (BO) (28). is machine learning technique for finding the global max- ima/minima of an unknown function based on stochastic evaluations. BO has been extensively used in combinatorial optimization [4) hyper-parameter optimization for neural networks [29] and optimization of parameters in robotics. recent literature, BO has been successfully applied for generation of adversarial counterexamples for complex controllers 11 I8] including identification of test cases for autonomous driving systems [[5], and coverage of faults in analog and mixed cireuits [19]. Traditional Bayesian Optimization is limited finding a single global minima/maxima which can lead to the discovery of only one counterexample. Hence, extensions that find multiple In this work, we closely follow the
test cases has been proposed on top of traditional BO [5 19 formulations and extensions of [5]
Safe RL is area of emerging significance buoyed by the proliferation of RL in safety-critical control systems. A survey of RL safety problems and taxonomies can be found in [17]. The goal of safe RL is to create agents whose behaviour is aligned with the user's intentions. Achieving this via reward shaping
(21] causes a trade-off between a reward enacting task-solving behaviour and penalty for avoidane of hazards. Less penalty promotes unsafe behaviour and high penalty dilutes the objective goal of the RL.
A natural way to incorporate safety specification in RL is by imposing constraints. A substantial body of work exists this direction (2 12 1. However, there is no obvious guarantee that the framework of constrained RI. solves the agent alignment problem. Constrained RI. requires the specification cost functions for the constraints, along with the reward function, with susceptibility towards design errors [26). In the proposed method we neither change the reward function nor introduce constraint costs. Our policy corrections are affected by changing the policy gradient based on violations of the safety specifications.
Another school of work uses Preference Based RL, where the agent learns directly from an expert's preferences of trajectories instead of -esignd numeric reward. Preference -aasee RL suffers from the temporal credit assignment problem, that is, determining which states or actions contribute to the encountered preferences [31] Counterexample guided refinement has been studied in Apprentice I learning (AL.) using PCTL model checking [32]. Our approach is applicable to wider range of applications since it can handle larger state spaces than the model checking approach from [32]

---

The authors in [6 do counterexample based improvement on RNN-based strategy on POMDP using Linear Programming. Model-checking is performed on an RNN strategy induced MDP and the strategy is refined using Linear Programming based on counterexamples. The application of their work is limited to environments with discrete state space. In their setting, the POMDP model is known apriori, which is not true in our setting.
Finding Failure Traces using Bayesian Optimization
This section outlines approach for using Bayesian Optimization look for failure traces in a trained RI policy, Totd, with respect to a given safety specification. BO is machine learning approach formulated as:
argmin pEP
where f R2 + IR is continuous but lacks special structure and E P where P is a set of function parameters. BO used optimize derivative free, expensive-to-ewe to-valuutee black box functions. In its core BO uses Gaussian Process (GP) models to estimate a prior over the function f. The function value, fp). treated random variable by Gaussian Process, which enables the modelling of finite number of observations using joint Gaussian distribution. The distribution initially has an assumed mean, m(p), at O and the kernel, x(ps, Pý), squared exponential function of the parameters p. e Pe-P,1)* For every unobserved value px, the mean, m, and variance, o² can be estimated using the following equations [28]:
m(p- (p)
m(p.) KTTK' m(p))
(2(p-1)
K KK
=<= where K = k<p, p). K, = p.). and K., = p(p..p.) An acquisition function then used to sample the estimated prior, along with assumed mean and variance, for unobserved points derive posterior distribution. The acquisition function chosen in this paper is Expected Improvement, which represents the quantum of improvement while exploring and exploiting the function find global minimum. If f (p) is the minimal value of fp) observed so far and f'(p) is the new evaluation, then the expected improvement utility, El, is given by:
(I(p) E [mamm (
where, E indicates the expectation taken under the posterior distribution, given evaluations of f. The expected improvement algorithm then chooses evaluate the function at the point having the largest expected improvement, namely:
Pr+1 EngmaxE1(y
The prior model improves through repeated sampling using the acquisition function until the improve- ment becomes insignificant (indicative of a global minima) or the computation budget is exhausted.
We now explain the use of BO in our framework. The following example elucidates the problem.
The toy grid-world of Fig. 7 has two start locations at (0,0) and (1,0), and sin- gle goal location at (0,3). The agent has two actions, "left" for moving left, and "up" for go- ing up. The reward indicated beside some of the locations. and is -1 for all other locations. Suppose the safety requirement is that the agent should never reach a location having reward less than -1. which is modeled by the predicate: ((reward(state) + 0). In the absence of this safety requirement, the following trajectories are optimal in terms of total reward:
1
3
Example 3.1
State
+10
Start State
Figure A toy grid environment
(0, left
(0,1) left (0, 2) left
Trajectory-l:
(0,3)
(1,0) left (1,1) left (1,
ieft (1,3) "P, (0.3)
Trajectory-2:
Trujectory-2 violates the safety specification at (1,1), and is therefore failure trajectory. BO helps us find such failure traces automatically, by intelligently testing the RL policy with adversarial tests derived from the safety specification.
3

---

Algorithm 1: Finding Failure Trajectories
Input:
Policy Rotá, Parameter Bounds P. Objective functions ye, Agent A (Fold, p,p, A):
Function
i <-
while i < mar budgel do Sample parameters E [Plow, Phigh] based on GP models
j i+1
0j --pi Pi while not done do s Totd(o;)
0j+1 t A.step(aj)
& + {))
03 t 0941
j+1+1
end
Poud, + argiinng (uI(E)))... (Pn(E)) Update each GP model with measurements (pr, pr (p.)) Er + Er (É, with feal, < 0)
remove p. from search space
++++1
end return E
Comments:
p Set of objective functions corresponding predicates in negation of safety criteria.
& sequence of states os, actions a, over time 0, I,...
Prott Evaluation of the objective functions in yp for the 2th trajectory.
In the proposed methodology we sample the given policy, Told, for failure trajectories using Algo- rithm [ which aims to minimize a set of objective functions, (pp), derived from the specification over the given set of parameter bounds P. (p(), is manifestation of the negation of the safety criteria in terms of a set of objective functions that can be minimized. The function set y derived from the safety criteria may involve negation, conjunction, and disjunction of multiple predicates over the variables in the system i.e., & = 1 Auz2 (i) Vj12- is assumed that each predicate function 14 is smooth and continuous function over trajectories ç. Violation of any μl E yo forms the basis of violation for the overall safety criteria. Each such component, j4 is modeled as a separate GP in our methodology. This method is able to uncover more counterexamples than modeling jo with single GP as demonstrated by [8]. For cases where a set of predicates need to be jointly optimized as shown in example 3.2 normalize the variables. In each iteration, the parameter values selected by BO are evaluated against each μi and the evaluated value used by the acquisition function to push each μ4 towards its minimum. Each trajectory, containing sampled parameters ps, that violates any Pa is added in the failure trajectory set. This process continues until the entire sample space covered i.e all the possible states that can cause failure is removed from the search space, or the time budget is exhausted. We use the extension of BO provided in (15), that removes the failure area enclosed by already explored trajectory from the search space, uncover multiple failure trajectories.
Example 3.2 A safety specification for lunar lander trained using RI. might be that the lander cannot be tilted at an angle while being close to the ground. The lander coordinates are specified by (g) and the angle is given by the variable langle- Concretely, let us assume 0 < ly < 10
and -1 S langle < Using predicate logic we can represent this safety specification as ly
5- langle 2 -0.5, which in words says that if the lander is at vertical distance of less than 5m from the ground then the angle of the lander is greater than -0.5. The negation of this specification can be rewritten as &p (ly < 5) A (langle < <0.5).. There are two predicates in the specification
and p2 : .langte + 0.5 < O. This can be converted as an equivalent BO objective function min(p + (22.. Intuitively, this means that if sp < O the safety specification is violated, for example one such evaluation of the function could be ly and langle -0.8. An example is shown in Figure 2b
μ1 :y

---

Task Environment
Query with sampied parameters
Parameter Bounds
Safety Specifications
Testing Module
Bayesian Optimization
Feedback
Output
Policy Toll
Set Failure Trajectories
Fallure Trajectory
Pollcy correction using gradient update
0.21
0.27
10.81
-1
Initial state for
Set of Subtasks
Train with objective p
0 and max(Expected Reward)
task policy TTc
(b)
Figure 2: a) Overview of the policy refinement framework. The failure trajectories are obtained by executing BO on the original learnt policy. The failure trajectories are then used for learning a new sub-policy which acts as the reference for gradient updates when refining the original policy. b) An example of failure trajectory and the corresponding corrected trajectory Lunar Lander Environment.
4 Policy Refinement Methodology
Fig. 2a outlines the proposed counter-exane exammle guided policy refinement approach. each iteration, the previous policy is tested using the approach proposed in SectionBJto find set of failure trajectories. This section presents the proposed methodology for using these failure trajectories to refine the policy. We begin with set of formal definitions.
Definition 4.1 (Failure Trajectory) A failure trajectory, E,s = {so, ao S1, a1...., St. ot), is se- quence of state-action pairs that contains one or more states where the safety specification refuted iee 3s; E Eyz (s:) < 0 where P* E G. E₁ denotes the set of failure trajectories.
Definition 4.2 (Saddle point of failure trajectory) We define saddle point as the first state on failure trajectory where the prediction made by the actor network violates the safety specification.
Following example 3.1 want to change the policy for the grid-world from the before the saddle state (1,1) i.e., from (1,0) in Trajectory 2. The updated trajectory to be enforced in the policy network is (1,0) "P, (0,0) requiring minimum gradient update. Algorithm the proposed policy refinement methodology. For each failure trajectory E,i E & obtained using Algorithm T. identify the saddle point, Ssaddle, We the states preceding the saddle state, Ssaddles, where in < i, in Eyı f as the initial states a new sub-policy тe that learns the correct actions for all trajectories in E,- The objective of т. is to satisfy the specification, namely to have praa 0 on states of the new trajectory, while also maximizing the reward function. To incorporate a negative penalty for the violation of the safety specification Peat, with an importance of /3 along with the reward of a trajectory Efi while training *.-
2 formally outlines
add the evaluation of the objective functions
We considered the possibility of refining the original policy Notd by using Te safety shield, that is, to override the action of Told by the action of Te in selected states. In this approach, n. does not force Told to learn possibly more rewarding policies by exploring ways around the saddle points. In other words, forcible preemption of actions proposed by Told results in policy which is distant from Toid and potentially less rewarding. Moreover disruptive changes in optimized policy is typically not welcome in any industrial setting.
Therefore we propose to refine aotd using proximal policy gradient approach, where we use to change the policy gradient near the saddle points in a way that wotd adjusts itself locally to new policy nnww- Over time, Nnww learns avoid the failure trajectories through such local adjustments.
5

---

Algorithm 2:
Policy Refinement Algorithm
Input: Policy Motá, Failure Trajectories Function Correct_ for each Ey: Ey do
(Tola: Ey):
R; = R; + 3 qval, (Ef))
Train using (Ersssaddie;;
end
for each Eyi in Ey
Collect observation data D2 for Ey:
Collect new actions and log probabilities corresponding to Db. from new sub-policy
Set Advantage Estimate Ac,
Update Toid Trew by maximizing the PPO clip objective using Te
end
Comments:
Importance value of evaluated safety specification.
Since the adjustments are local, Told and nnww are proximal and therefore (typically) not very different in terms of reward. In order to formalize this approach, we outline the policy gradient methodology. followed by our proposed intervention to use it in the above context.
Suppose (S, A. R, is a finite horizon discounted Markov Decision Process (MDP), where the set of states, denotes the actions, and P denotes the probability of transition from state E S to state s E S by taking action E A. Further, R denotes the reward signal and 0 S 7 S 1 is the discount factor. The next state of the environment, $t+1 E S, is determined stochastically according to a joint distribution depending the state-action pair (st, Qz).
The objective of any policy gradient RL algorithm 33) is to learn the best action to take in it's policy
o, by fine-tuning set of neural network parameters O. The policy, æ, is defined by P, [A; = a,S. s, 0: = f), namely the probability of taking action a₂ state 8; under parameters 0,. Popular policy gradient algorithms follow an actor-critic architecture where the critic estimates the value function by updating the value function parameters and the actor updates the policy parameters, 0, for To (als)), in the direction suggested by the critic. We focus particular class of Policy Gradient Algorithms known as Proximal Policy Optimization (PPO) (7). The core idea behind PPO is to use a clipped surrogate objective that improves the training stability by limiting the changes made the policy at each step. The policy update objective function is given by
L
B min (rr(0), clip(r. e, +)) A.
(3)
(o(a)(st)
where re
(4)
Te. (aelse)
Following the theory of importance sampling, (0) calculates the ratio between the probability of the action under the current policy (s)) and the probability of the action under the previous policy Ta..a (ar]ss). to trace the impact of the actions. To avoid drastic changes during policy update,
the clipped surrogate function, r.glip takes the minimum of the clipped ratio and the actual ratio multiplied with the advantage, A..
In the proposed counter-example guided policy refinement approach, the machinery used for gradient update of the actor network, oold, is the same as the gradient update in original PPO algorithm except that we intend to use to update the gradient. Towards this goal we introduce the following modifications:
1. We change the clipped objective ratio r0) as follows:
(0) Tota(ar st)
(5)
(T(aa
where n, is the sub policy with correct actions for failure trajectory Eyr and Tald is the old policy that we want to refine new policy nnew-
We set the advantage factor A, to be for the following reason. During the correction process, the action predicted by no given a failure trajectory forms the mean of the probability distribution

---

fre Lomeedo Colpere
Areage Catesoo Ratur Fort
Avergg Sptodic Souar
Copa of Palcy
Urgpud Pulcc
Pargy
1381
Tearing Shys
(a)
Random Observations
Fallrre Trajectory Observations
Action Map
Action Map
Action
Action map
5.0
0
0.0
-25
4-
-5.0
-17
-1/2
R12
-/2
0
112
-822
82
Original Pendulum Policy
Updated Pendulum Policy
Original Pendulum Policy
Updated Pendulum Policy
Figure 3: a) Reward plots for different environments showing training steps and the reward obtained by the sub-policy and the original policy b) Observation action hcat-map for pendulum policy
from which the action sampled, that is, a; ~ P(Te((:)) where o; E Eyi an observed state in the trajectory. 'P can follow Categorical distribution (for discrete actions) or a Multivariate Normal distribution (for continuous actions). We only consider the oobseraation,actinn satisfaction of yo over the entire trajectory while sampling from T.. Since, these corrected trajectories are to be enforced into Vold we set the advantage factor Ar to be 1.
pairs that lead the
Section 5 provides empirical evidence in support of the proposed methodology in terms of safety and reward. We also emphasize that the refined policy Nnew obtained using our method minimally different from the old policy Told. thereby ensuring minimal trade-off of reward in favor of safety. The following metric defines the distance between the two policies.
Definition 4.3 Variation Distance between policies: Since work with model free reinforce- ment learning, we define the variation distance between two policies in terms of their trajectories. We sample random trajectories from both the policies along with the failure trajectories from policy Vold and their corresponding updated trajectories from policy Tnew- We want to measure the difference of states visited between the trajectories. The distance function Dy is given as follows where Ei, c correspond to the trajectories of oold and knew respectively.
Do(woid
Tnew
Σ
Σ
(8i)) (#f)=neml
(6)
7
STEEENNEE
5 Empirical Studies
We test our framework on a set of environments from Open Al gym 5]. The environments, safety specifications, parameter bounds, number of failure trajectories and the mean variation distance, along with the standard deviation, between the original and updated policy is summarised in Table I Fig 3a shows the plots of rewards obtained by the original policy and the sub-policy in each environment. The sub-policies have lower training time due to less number of trajectories to train on. The experiments were run on machine with AMD Ryzen 4600h 6 core processor and GeForce GTX 1660 Graphics unit. We set max budget 200 for each iteration of BO. The mean and
7

---

Table I: Description of assertions tested PPO policies along with the parameters hounds for BO query space. We also report the numher of failure trajectories uncovered and corrected for each policy and the variation distance between the updated policy and the original policy, In our experiments n=1000
Environment
Safety Criteria
Parameter Bounds
Failures
Distance
State [(-0.05, 0.05)]
1. -2.4 < position < <2.4
174.4
1.255
Mass (0.05, 0.15)
Cart-pole-v0
2. -2.0 < momentum < 2.0
+ 0.51
+ 0.195
Length pole (0.4, 0.6)
3. angle 0.2
force magnitude: (8.00, 12.00)
8: (-m,m))
80.1
10.866
Pendulum-v0
1. Reward > -300
à: (0.1)
+
+ 1.379
speed: (-1,1)
1.85
Hull angle (0,2
40.6
11.189
1. Hull Position 00
Bipedal
Velocity x: (-1,1)
2. -0.8 Hull Angle <
v3
Velocity y: (-1,1)
4.08
1.375
1. -0.4 < Landing
25 (0.10)
Position, < 0.4
40.85
2.215
Ys: (0,20)
LunarLander Continuous-v2
Pos, 0.1 (angle>-1
+
velæs: (0.3)
angle I)
5.14
0.282
velys: (0,3)
Reward >
variance of the number of failure trajectories uncovered is reported over 20 BO evaluations.
Cart-pole Environment: In this environment the pole on top of the cart is controlled by ap- plying a force of PPO policy is tested against the following assertions:
the cart and the goal is to prevent the pole from falling over. The original
1. The cart should not go beyond -2.4 (extreme left) or +2.4 (extreme right).
2. The cart maintains momentum between -2.0 and 2.0
3. The angle made by the pole should be greater than 0.2 with respect to the rest position
Pendulum Environment:
The pendulum-vO environment models an inverted pendulum swing-up problem. The pendulum starts in random position, and the goal is to swing it up so that it stays upright. We wish to ensure the pendulum does not swing twice on its pivot to stabilize i.e., the reward obtained is not less than -300. There are three sources of uncertainty as described in Table [ Figure 3b shows the observation vs action map for a set of random observations and for observations of a failure trajectory. We can observe that actions with high values have reduced in the updated policy. However, the positive and negative action distribution over the observations remain the same.
Bipedal Walker: The goal in the bipedal walker environment to navigate a two legged bot successfully to the end of terrain. This is challenging environment to test having 24 variables as a part of the state space and action variables for different joints of the bot all them taking their values from continuous domains. The robot falls the ground when the hull angle goes below -0.8 (falling towards the left) or the angle is above 2 (falling towards the right). Hence, we want to find trajectories where hull angle -0.8 hull angle 2 which forms predicates for objective function. Also, the episodes terminates the walker does not move at all (hull position < 0).
Lunar Lander: The task in this environment is to land a lunar lander vehicle smoothly be- tween the landing flags at coordinate (0.0). We consider four sources of disturbances for the system. This can occur in the real world for a variety of reason such as noisy senor readings. The disturbance (æa, ys) added the initial (x.y) coordinates where 23 E (0, 10) and ya 20). Also disturbance (velrs, velys) are added to the initial horizontal and vertical linear velocity where velu,, velys E 30. Under these disturbances want to the policy satisfy the following assertions.
1. The lander should not land beyond certain distance from the flag.
2. If the lander is close to the ground then it should not be tilted beyond certain angle.

---

Table 2: Comparison with Baselines
Environment
Policy A
Policy B
Policy
Policy D
Failures:
Failures: 179
Failures: 52
Failures: 0
Cart-pole
Training Steps: 150K
Training Steps: 900K
Training Steps: 150K
Training Steps: 1M
80K (Update)
Failures: 89
Failures: 89
Failures: 102
Failures: 0
Pendulum
Training Steps: 850K 20K (Update)
Training Steps: 1.6M
Training Steps: 850K
Training Steps: 1.8M
Failures:
Failures: 45
Failures: 145
Failures: 41
Bipedal
Training Steps: 2.8M 20K (Update)
Training Steps: 7.5M
Training Steps: 2.8M
Training Steps: 8M
Faitures:
Failures: 42
Failures: 18
Failures: 5
LunarLander
Training Steps: 400K 20K (Update)
Training Steps: 1.1M
Training Steps: 400K Training Steps: 1.2M
Continuous
For the first assertion the failure trajectories have reward greater than 200 as the rover lands suc- cessfully but lands beyond the allowed distance from the flags which violates the safety specification exhibiting that even trajectories that have positive rewards can violate safety specifications.
As exhibited by Table is able to find failure trajectories, as many as 175 for cart-pole environment, from tuned policy which had converged to high reward. As displayed by TigureBbjan the variation distance reported in Table the updation does not lead to policy significantly different from the original policy in terms of optimization goal. We compare the following baseline PPO policies with the refined PPO policy:
(A) PPO policy trained from scratch with negative penalty for property violation, (B) PPO policy trained from scratch with only counterexample traces and negative penalty after one iteration of testing with BO same Te. (C) PPO policy trained from scratch with original training traces, counterexample traces and negative penalty after testing with one iteration of BO, and (D) The refined policy knew- The observations are reported in Table 2 Even though the Policy A penalized with respect to the satisfaction of the safety property, the variable variations which occurred during deployment were not known during training. Hence, Policy A has high number of counterexamples Policy B has counterexamples that had not occurred in the original policy as it was only trained on counterexample traces and did not retain the optimized behaviour of the original policy. In some cases, Policy C has fewer counterexamples than Policy A for the same tests but the training time is substantially higher than our policy refinement method. There also cases (bipedal walker) where Policy C has high number of counterexamples even after the inclusion of counterexample traces and negative penalty, the updates are not targeted. The refined policy by our methodology did not report any counterexamples for the same property even after 200 BO iterations and works with much fewer training and update steps.
6 Conclusions and Discussion
Adversarial testing of deep neural networks is critical for deployment of learned systems in safety critical applications. In the context of deep reinforcement learning, the problem is more complex as adversarial testing needs find failure trajectories by searching over the space of states, actions, and parameters. The methodology presented in this paper targets the search for failure trajectories by posing as Bayesian Optimization problem, and then uses policy gradient update mechanism to locally retrain the policy to avoid unsafe states. We believe this is the first work on counter- counter-example example guided policy gradient update, and the method appears promising based on empirical evidence.
A limitation of our method is in providing formally guaranteed safety which can be provided via formal verification methods or existence of Iyapunov based policies on model based techniques where the MDP is known apriori. The method in its current form is also incapable of handling temporal safety specifications. we find counter-examples using Gaussian Processes to estimate a prior the safety specification function, the function must be smooth and continuous function of the trajectory.

---

7
Broader Impact
Our paper introduces methodology for testing and correcting RL policies that have already been optimized and calibrated. RL has the potential to bring significant benefits for society, beyond the boundaries of solving tasks in simulation environments. In many safety critical domains, it of paramount importance to develop RL approaches that respect safety constraints during deployment. RL policies need adapt and accommodate safety specifications of an ever changing environment without going through and entire training cycle. We believe that our work contributes to this quest, potentially bringing RI closer to high-stakes real-world deployment. Of course, any technology especially one general as RI. has the potential of misuse, and we refrain from speculating further.
Acknowledgments and Disclosure of Funding
The authors would like to thank Department Science and Technology (DST), Government of India and TCS Research Scholarship for partially supporting this work.
References
[1] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 22-31. PMLR, 06-11 Aug 2017.
[2] Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pages 2669-226622 2678, 2018.
[3] Rajeev Alur, Thao Dang, and Franjo Ivančić. Counterexample-guide predicate abstraction of hybrid systems. Theoretical Computer Science, 354(2):250 271, 2006. ISSN 0304-3975. doi: https://doi.co org/10. 2005.......51111 11.026. URL URL http://wcciineedeee..ooooe/eeeee
pi7/00004395505006666 Tools and Algorithms for the Construction and Analysis of Systems
(TACAS 2003).
[4] Ricardo Baptista and Matthias Poloezek. Bayesian optimization of combinatorial structures. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 462-471. PMLR, 10-15 Jul 2018.
[5] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1606::600601:: 2016.
[6] Steven Carr, Nils Jansen, Ralf Wimmer, Alexandru Serban, Bernd Becker, and Ufuk Topcu. Counterexample--uidde strategy improvement for pomdps using recurrent neural networks, In IJCAI, pages 5532-5539, 2019. doi: 10.24963/iccai.21117766
[7] Yinlam Chow, Ofir Nachum, Edgar lyapunov-based approach safe reinforcement learning. Advances in Neural Information Processing Systems, 2011--Deee::002288111 2018. ISSN 10495258.
and Mohammad Ghavamzadeh. A
[8] Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. arXiv, 2019. ISSN 23318422.
[9] Edmund Clarke, Оma Grumberg, Somesh Jha, Yuan Lu, and Helmut Veith. Counterexample- guided abstraction refinement. In Allen Emerson and Aravinda Prasad Sistla, editors, Com- puter Aided Verification, pages 154-169, Berlin, Heidelberg, 2000. Springer Berlin Heidelberg. ISBN
[10] A. Huang C. J. Maddison A. Guez L. Sifre G. van den Driessche J. Schrittwieser Antonoglou V. Panneershelvam M. Lanctot S. Dieleman D. Grewe J. Nham N. Kalchbrenner I. Sutskever P. Lillicrap M. Leach Kavukeuoglu T. Graepel D. Silver and Hassabis. Mastering the game of go with deep neural networks and tree search. In Nature, page 484-489, 2016.
10

---

[11] Jamal Daafouz, Pierre Riedinger, and Claude Tung. Stability analysis and control synthesis for switched systems: A switched Iyapunov function approach. Automatic Control, IEEE Transactions on, 47:1883 1887, 12 2002. doi: 10. 0..1109/TAC.2200884447777
[12] Gal Dalai, Krishnamurthy Dvijotham, Matej Vecerík, T. Hester, Cosmin Paduraru, and Y. Tassa. Safe exploration in continuous action spaces. ArXiv, abs/aaasss..// 1811088117 2018.
[13] Jyotirmoy Deshmukh, Marko Horvat, Xiaoqing Jin, Rupak Majumdar, and Vinayak S. Prabhu. Testing cyber-physical systems through bayesian optimization. ACM Trans. Embed. Comput. Syst., 16(5s), September 2017. ISSN 1539-9087. doi: 10. 10.1145/3126521. URL httpss:////oo
10.1145/3126221
[14] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference Robot Learning, pages 1-16, 2017.
[15] B. Gangopadhyay, S. Khastgir, S. Dey, P. Dasgupta, G. Montana, and P. Jennings, Identification of test cases for automated driving systems using bayesian optimization. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 1961-1967, 2019. doi: 10. 109/ 8917103.
[16] Briti Gangopadhyay, Harshit Soora, and Pallab Dasgupta. Hierarchical program-triggered reinforcement learning agents for automated driving. IEEE Transactions on Intelligent Trans- portation Systems, pages 1-10, 2021. doi: 10.1 1.2969999
[17] Javier Garefa, Fern, and o Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(42):1437-1499 2015. http://t///mm:::rr/aaaaa v16/garcia 5a.hmml
[18) S. Ghosh, F. Berkenkamp, G. Ranade, S. Qadeer, and A. Kapoor. Verifying controllers against adversarial examples with bayesian optimization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7306-7313, 2018. doi: 10.1109/ICRA. 8.018.888 8.8460635.
[19] Hanbin Hu, Peng Li, and Jianhua Z. Huang. Parallelizable bayesian optimization for analog and mixed-signal rare failure detection with high coverage. ICCAD '18, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450359504.
doi: 10.1 10.1145/2440655.22403..
URL httpp://oo..og//011444444477//33 14553240765.3240333
[20] A. Kendall, J. Hawke, D. Janz, P. Mazur, D. Reda, J. Allen, V. Lam, A. Bewley, and A. Shah. Learning drive in a day. In 2019 International Conference on Robotics and Automation (ICRA), pages 22444, 2019.
[21] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg, Scalable agent alignment via reward modeling: A research direction. arXiv, 2018. ISSN 23318422.
[22] Volodymyr Mnih Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1928-1937, New York, New York, USA, 20-22 Jun 2016. PMLR. htmll
URL http::/pweeecdnngss.lpppess////s
[23] E. Perot, M. Jaritz, M. Toromanoff, and R. De Charette. End-to-ent driving in a realistic racing game with deep reinforcement learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 474-475, 2017.
[24] T. Pham, G. De Magistris, and R. Tachibana. Optlayer- practical constrained optimization for deep reinforcement learning in the real world. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 0236-6243. 2018. doi: 10.1109//R..2008 2.8.4445444
[25] Stephen Prajna and Ali Jadbabaie. Safety verification of hybrid systems using barrier certificates. In Rajeev Alur and George J. Pappas, editors, Hybrid Systems: Computation and Control, pages 477-492, Berlin, Heidelberg, 2004. Springer Berlin Heidelberg, ISBN 978-3-540-24743-2.
11

---

[26] Alex Ray, Joshua Achiam, and Dario Amodei Openai. Benchmarking Safe Exploration in Deep Reinforcement Learning. arXiv preprint. 2019. URL https:/ddmmuutpkyyy...odddooo... safexp-short.pdf
[27] John Schulman, F. Wolski, Prafulla Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. ArXiv,
[28] B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):148-11)) 175, 2016. doi: 10.1 100////0...015..444222000.
[29] Jasper Snock, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Proceedings of the 25th International Conference Neural Information Processing Systems Volume 2, NIPS' 12, page 2951 1919, Red Hook, NY, USA, 2012. Curran Associates Inc.
[30] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
[31] Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference-bace based reinforcement learning methods. Journal of Machine Learning Research, 18: 1-46, 2017. ISSN 15337928.
[32] Weichao Zhou and Wenchao Li. Safety-aware apprenticeship learning. In Hana Chockler and Georg Weissenbacher, editors, Computer Aided Verification, pages 662-680, Cham, 2018. Springer International Publishing.
12


Reviews:
[{"review_id": "fHe7IIhC_K5", "reviewer": "Reviewer_4nks", "summary": "This paper presents a counterexample-guided policy refinement method to check and correct an already trained RL policy. The proposed algorithm (i) uses Bayesian Optimization to find failure cases in a policy, (ii) then learns a new sub-policy to locally overcome the failed cases, and (iii) finally uses a slightly modified PPO algorithm to refine/update the initial policy to eliminate the failure cases in a proximal manner. The paper is evaluated on several OpenAI Gym environments and shows they are able to find and fix several failure cases with a small mean variation distance between the policies. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Using a counterexample guided approach in RL to ensure safety is an interesting approach\n- The algorithm to proximally refine the policy is neat and well done \n\nWeakness:\n\nThis paper needs a more thorough evaluation including comparing to a few baselines, evaluating on more metrics, and on harder problems \n- With the lack of baselines, it is hard to appreciate the numbers in the evaluation. One obvious baseline would be training/fine-tuning with the combined reward function (reward + loss penalty, the total reward used for training $\\pi_c$) on all initial states (essentially eliminating the Bayesian Optimization step. \n- Also, please report the initial reward, final reward, # failure cases in the final policy.\n- It will also be interesting to see how this algorithm affects the sample complexity (which needs to take into account the sample complexity for both the Bayesian optimization and the retraining steps) and compare to other baselines/approaches' sample complexity.\n- A tricky part of any counter-example guided algorithm is that it requires multiple back and forth iterations and sometimes it does not converge. When you fix a policy for some failure cases, more and newer failure cases creep up. However, in the current evaluation, it seems like all problems are solved with just 1 iteration. Can the authors comment on this? Is this some special property of your algorithm or are the tasks too simple?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "VpAKN5xOJsU", "reviewer": "Reviewer_PvUR", "summary": "The paper proposes a training methodology to incorporate constraints into a trained RL policy. It uses Bayesian optimization to select parameters for the MDP that are most likely to produce trajectories violating the constraints and then performs PPO updates to steer the policy away from these trajectories while remaining close to the original one. The experiments are conducted on popular OpenAI gym environments.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I'm not an expert on safe RL and I found the presentation of the topic given in the paper rather confusing. In particular it's not clear to me what exactly is the specific problem with directly penalizing constraint violation in the reward function that the authors are trying to solve. The authors claim that their method avoids such an approach, yet the proposed PPO updates look very much like adding negative reward for constraint violation. Those updates also seem to be largely perpendicular to the choice of using BO for MDP parameter selection and it's unclear to me whether the authors claim each of those separately as novel contributions or only their combination.\n\nFor me the fatal flaw of this paper is the underwhelming experimental evaluation, in particular not including any baselines. I don't quite understand how the original policy in the experiments was obtained, but my impression is that it was trained without any constraint violation penalties and I would very much like to see a comparison with a policy being trained in such a setting with normal PPO. On the BO side, at the very least we should see a comparison with random sampling. Without those experiments, I don't think the paper can be accepted.\n\nEDIT:\nraised score after rebuttal, see the rationale in the discussion below", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "USAMgHxa0fX", "reviewer": "Reviewer_yDKZ", "summary": "Authors introduce a safety-oriented policy-refinement method for reinforcement learning. Authors use Bayesian optimization to find safety-violation traces from a policy generated by a vanilla RL algorithm. Those traces are latter use to produce a new trained policy that satisfies the requirements. The method is meant for continuous-action environments", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Authors present a novel method for generating safe RL agents as far as I am aware, both the process of finding counterexamples through Bayesian optimization and  their policy refinement method are interesting contributions. However, authors claim two times that their are first to refine learned RL policies through counterexamples. This is not the case since [1] already did that. However the proposed method is significantly different from that previous approach. Moreover, the proposed method is applicable to a wider range of applications since it can handle larger state spaces than the model checking approach from [1].\n\nThe paper is clear and well written, authors introduce various images and running examples that makes the approach intuitive and easy to follow. They provide sufficient proof and explanation with definitions and various experimental benchmarks. I only saw a handful minor things that needed correction.\n\nOverall is a good contribution and I recommend acceptance (upon that authors correct the two times they say to be first to present a counter-example refinement method for RL and include [1] in their references).\n\nAdditional comments:\n- Error intervals are barely visible  in figure 3, I strongly suggest to draw the plots with a different tool than tensorboard\n- What do the error intervals represent, quartiles? std?\n- Table titles should be above the table according to neurips style, also in the caption, II suggest recalling what n is\n\n[1] Zhou, W., & Li, W. (2018, July). Safety-aware apprenticeship learning. In International Conference on Computer Aided Verification (pp. 662-680). Springer, Cham.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors explicitly state their limitations in the conclusions", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "H-LfnJmiFL", "reviewer": "Reviewer_528P", "summary": "this paper studies the problem of how to train a policy \\pi which maximize rewards while retaining certain safety properties, stated as logical constraints. a BO algorithm discovers failure trajectories, which are then added to the PPO learning algorithm to steer the policy away from constraint violations.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality and Significance:\nI am a fan of this line of research, I think incorporating constraints as a way of specifying safety properties might be more clear to end-users than reward engineering or preference learning. As constraints are something (presumably) easy to understand by both algorithms and humans, it is a natural avenue to explore in human-machine communications in the realm of safety specifications.\n\n### Clarity:\nI do not think this paper is clearly written. Specifically, I spent a long time trying to understand what exactly is the query space for the BO. \n\nTo put it in lunar lander terms, is BO trying to discover a failure trajectory by:\n1. put the lander into a weird initial location (x,y) with an odd angle so that it fails?\n2. varying the mass / size of the lunar lander so that it fails?\n3. forcing the policy \\pi to make a low probability choice so it is more likely to fail?\n\nThe key phrase \"Parameter Bound\" seems to denote the space of query points for the BO, but is present in the paper only twice, and later in table1 there's a column named \"domain of uncertainty\". Are they the same thing? I'm assuming they are, but I am very lost.\n\nI would suggest use lunar lander as the motivating example, as example 3.1 is way too detached from the main body of work to be the \"opener example\". I would also make the space of BO very explicit, if my understanding is correct, I would love to see a phrase of the form \"randomly sampling mass and size of the lunar lander is unlikely to uncover a failure trajectory, but with BO we can actively query for a failure case of mass and size, and reliably sample failure trajectory with high probability. This is assuming the space of BO in lunar lander is this \"parameter bound\" or \"domain of uncertainty\", which could be wrong.\n\n### Quality:\n[edit : with the added random sampling baseline I think it is sufficient quality]\n\nThis work is (not yet) solid.\n\nIt is unclear to me from the experiments that BO is warranted for the domain in question. There should be some kind of metric showcasing how BO out performs just random sampling\n\nIt is also unclear to me that if the proposed method \"proves\" that no failure can occur, as the BO is working over a stochastic policy \\pi, it may be possible that under some roll-outs, the policy will exhibit unsafe behaviour but simply \"got lucky\". At the very least, there should be an experiment that empirically justifies that your proposed algorithm gives a policy that is safer over many roll-outs.\n\nI would suggest inclusion of a comprehensive table where, for each task (lander, walker, etc) you train three different policies:\n\\pi_a) without safety corrections, only maximizing rewards\n\\pi_b) with corrections, but sampled randomly without BO\n\\pi_c) with corrections, and sampled with BO\nThen, evaluate how \\pi_a, \\pi_b, \\pi_c performs on 10k new roll outs, notice the percentage of states that have failures. Hopefully, you'll discover that \\pi_c have a very low number of failure states (this isn't going to be 0, due to the stochastic nature of \\pi).\n\nOverall, this work does not \"prove\" the resulting policy \\pi_c is absolutely safe, and does not justify it empirically either.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see a \"limitation and negative impact\" section in the paper.\n\nI find the check-list item \"Did you discuss any potential negative societal impacts of your work? [N/A] paper presents works on safe RL with no foreseeable direct societal implications\" incorrect, as safety in RL is (at least it is touted to be) very important for societal implications as we need to trust the AI we built. In fact, the first paragraph of the intro makes a very strong case that this line of work has huge safety implications. This is _inconsistent_ with how the paper handled the checklist item with [N/A].", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has considerable support from the reviewers but not does not seem to rise to the level of an oral.

For me the formal adversarial set up is strange.  Some form importance sampling under an "unsafe" state-transition bias would yield an estimate of the probability of a safety violation. Optimizing an estimate of the probability of a safety violation, possibly with stochastically drawn model parameters, seems like a cleaner conceptual set up to me.  But the reviewers are happy and I will go along.
Decision: Accept (Poster)