Counterexample Guided RL Policy Refinement Using Bayesian Optimization

1 Introduction . Classical Reinforcement Learning ( RL ) is designed to perceive the environment and act towards maximizing a long term reward . These algorithms have shown superhuman performances in games like GO [ 10 ] , motivating their use in safety-critical domains like autonomous driving [ 23 , 20 , 16 ] . These policies are mostly learnt in simulation environments such as TORCS [ 22 ] or CARLA [ 14 ] for accelerated learning . Therefore they are susceptible to perform poorly in the real world where domain uncertainty arises due to noisy data . Also , the choice of a RL policy is determined by the design of the reward signal , which if not constructed properly , may lead the agent towards unintended or even harmful behaviour . Hence the need for constructing policies that account for risks arising due to uncertainty and do not violate safety specifications . Safe RL aims to learn policies that maximize the expectation of the return while respecting safety specifications [ 17 ] . The existing literature on safe RL is broadly divided into three schools of thought namely , 1 ) Transforming the optimization criteria by factoring in costs derived from safety constraints [ 1 ] , 2 ) Using external knowledge in the form of a teacher or a safety shield to replace unsafe actions chosen by the policy during exploration / deployment [ 2 ] , and 3 ) Establishing the existence of an inductive safety invariant such as a Lyapunov function [ 11 ] or a control barrier function [ 25 ] . Establishing such an invariant is often intractable for Neural Network controllers that use many parameters and work on high-dimensional state spaces . In this work , we propose a counterexample guided policy refinement method for verifying and correcting a policy that has already been optimized and calibrated . Failure traces may occur in such a policy due to the following reasons : a ) Failure trajectories are extremely rare and do not contribute significantly enough to reduce the expected reward , b ) Uncertainty in the environment ∗Code is available online at https : //github.com/britig/policy-refinement-bo 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . which had not been accounted for during policy training . Our aim is to progressively make the system safer by finding and eliminating failure traces with targeted policy updates . Counterexample guided refinement techniques have been well studied in the formal methods community for program verification [ 9 ] and reachability analysis of hybrid systems [ 3 ] , but not so far in the context of RL . We study the proposed methodology on environments that work over continuous space and continuous or discrete actions . The proposed methodology is divided into two steps : 1 . Given a policy πold , learnt for optimizing reward in a given environment , we test it against parameters with uncertainties and a set of objective functions ϕ derived from the negation of the given safety criteria . Multiple counter-example traces or failure trajectories are uncovered from πold using Bayesian Optimization . 2 . Using the failure trajectories we selectively do a gradient update on πold to construct a new policy πnew , that excludes the counterexample traces under the given domain uncertainties . We specifically work with a policy gradient RL algorithm , Proximal Policy Optimization ( PPO ) , which directly optimizes the learnt policy using gradient updates . These steps may be repeated multiple times . We show that such targeted updates ensure that πnew is safer than πold , but not significantly different in terms of reward . The paper is organized as follows . Section 2 outlines related work , Section 3 presents the methodology to uncover failure trajectories , Section 4 presents the methodology of policy refinement , Section 5 presents case studies on several RL environments , and Section 6 provides concluding remarks . 2 Related Work . The approach proposed in this paper uses Bayesian Optimization for finding adversarial tests cases in terms of trajectories for the RL policy leading to failure of the safety specification . The failures are then used to make the policy safer . Bayesian Optimization ( BO ) [ 28 ] , is a machine learning technique for finding the global maxima/minima of an unknown function based on stochastic evaluations . BO has been extensively used in combinatorial optimization [ 4 ] , hyper-parameter optimization for neural networks [ 29 ] and optimization of parameters in robotics . In recent literature , BO has been successfully applied for generation of adversarial counterexamples for complex controllers [ 13 , 18 ] including identification of test cases for autonomous driving systems [ 15 ] , and coverage of faults in analog and mixed circuits [ 19 ] . Traditional Bayesian Optimization is limited to finding a single global minima/maxima which can lead to the discovery of only one counterexample . Hence , extensions that find multiple test cases has been proposed on top of traditional BO [ 15 , 19 ] . In this work , we closely follow the formulations and extensions of [ 15 ] . Safe RL is an area of emerging significance buoyed by the proliferation of RL in safety-critical control systems . A survey of RL safety problems and taxonomies can be found in [ 17 ] . The goal of safe RL is to create agents whose behaviour is aligned with the user ’ s intentions . Achieving this via reward shaping [ 21 ] causes a trade-off between a reward enacting task-solving behaviour and a penalty for non-avoidance of hazards . Less penalty promotes unsafe behaviour and high penalty dilutes the objective goal of the RL . A natural way to incorporate safety specification in RL is by imposing constraints . A substantial body of work exists in this direction [ 24 , 12 , 7 , 8 , 1 ] . However , there is no obvious guarantee that the framework of constrained RL solves the agent alignment problem . Constrained RL requires the specification cost functions for the constraints , along with the reward function , with susceptibility towards design errors [ 26 ] . In the proposed method we neither change the reward function nor introduce constraint costs . Our policy corrections are affected by changing the policy gradient based on violations of the safety specifications . Another school of work uses Preference-Based RL , where the agent learns directly from an expert ’ s preferences of trajectories instead of a hand-designed numeric reward . Preference-Based RL suffers from the temporal credit assignment problem , that is , determining which states or actions contribute to the encountered preferences [ 31 ] . Counterexample guided refinement has been studied in Apprentice Learning ( AL ) using PCTL model checking [ 32 ] . Our approach is applicable to a wider range of applications since it can handle larger state spaces than the model checking approach from [ 32 ] . The authors in [ 6 ] do a counterexample based improvement on RNN-based strategy on a POMDP using Linear Programming . Model-checking is performed on an RNN strategy induced MDP and the strategy is refined using Linear Programming based on counterexamples . The application of their work is limited to environments with discrete state space . In their setting , the POMDP model is known apriori , which is not true in our setting . 3 Finding Failure Traces using Bayesian Optimization . This section outlines our approach for using Bayesian Optimization to look for failure traces in a trained RL policy , πold , with respect to a given safety specification . BO is a machine learning approach formulated as : argmin p∈P f ( p ) where f : Rd → R is continuous but lacks special structure and p ∈ P where P is a set of function parameters . BO is used to optimize derivative free , expensive-to-evaluate black box functions . In its core BO uses Gaussian Process ( GP ) models to estimate a prior over the function f . The function value , f ( p ) , is treated as a random variable by a Gaussian Process , which enables the modelling of a finite number of observations using a joint Gaussian distribution . The distribution initially has an assumed mean , m ( p ) , at 0 and the kernel , κ ( pi , pj ) , is a squared exponential function of the parameters p , e− ( |pi−pj | ) 2 . For every unobserved value p∗ , the mean , m , and variance , σ2 can be estimated using the following equations [ 28 ] : m ( p∗|p ) = m ( p∗ ) +KT∗ K−1 ( f ( p ) −m ( p ) ) ( 1 ) σ2 ( p∗|p ) = K∗∗ −KT∗ K−1K∗ ( 2 ) where K = κ ( p , p ) , K∗ = κ ( p , p∗ ) , and K∗∗ = κ ( p∗ , p∗ ) . An acquisition function is then used to sample the estimated prior , along with assumed mean and variance , for unobserved points to derive a posterior distribution . The acquisition function chosen in this paper is Expected Improvement , which represents the quantum of improvement while exploring and exploiting the function to find a global minimum . If f∗ ( p ) is the minimal value of f ( p ) observed so far and f ' ( p ) is the new evaluation , then the expected improvement utility , EI , is given by : EI ( p ) = E [ max ( 0 , ( f∗ ( p ) − f ′ ( p ) ) ) ] where , E indicates the expectation taken under the posterior distribution , given evaluations of f . The expected improvement algorithm then chooses to evaluate the function at the point having the largest expected improvement , namely : pn+1 = argmaxEI ( p ) The prior model improves through repeated sampling using the acquisition function until the improvement becomes insignificant ( indicative of a global minima ) or the computation budget is exhausted . We now explain the use of BO in our framework . The following example elucidates the problem . Figure 1 : A toy grid environment Example 3.1 The toy grid-world of Fig . 1 has two start locations at ( 0,0 ) and ( 1,0 ) , and a single goal location at ( 0,3 ) . The agent has two actions , `` left '' for moving left , and `` up '' for going up . The reward is indicated beside some of the locations , and is -1 for all other locations . Suppose the safety requirement is that the agent should never reach a location having a reward less than -1 , which is modeled by the predicate : ¬ ( reward ( state ) + 1 < 0 ) . In the absence of this safety requirement , the following trajectories are optimal in terms of total reward : Trajectory-1 : ( 0 , 0 ) left−−→ ( 0 , 1 ) left−−→ ( 0 , 2 ) left−−→ ( 0 , 3 ) Trajectory-2 : ( 1 , 0 ) left−−→ ( 1 , 1 ) left−−→ ( 1 , 2 ) left−−→ ( 1 , 3 ) up−→ ( 0 , 3 ) Trajectory-2 violates the safety specification at ( 1,1 ) , and is therefore a failure trajectory . BO helps us to find such failure traces automatically , by intelligently testing the RL policy with adversarial tests derived from the safety specification . Algorithm 1 : Finding Failure Trajectories Input : Policy πold , Parameter Bounds P , Objective functions ϕ , Agent A Function identify_failures ( πold , P , ϕ , A ) : i← 1 while i < max_budget do Sample parameters pi ∈ [ Plow , Phigh ] based on GP models j ← 1 oj ← pi while not done do aj ← πold ( oj ) oj+1 ← A.step ( aj ) ξi← ξi∪ ( oj , aj ) oj ← oj+1 j ← j + 1 end ϕvali ← argminξi ( µ1 ( ξi ) ) . . . ( µn ( ξi ) ) Update each GP model with measurements ( pi , µk ( pi ) ) ξf ← ξf ∪ ( ξi with ϕvali < 0 ) remove pi from search space i← i+ 1 end return ξf Comments : ϕ : Set of objective functions corresponding to predicates in negation of safety criteria . ξi : A sequence of states ot , actions at over time t = 0 , 1 , . . . ϕvali : Evaluation of the objective functions in ϕ for the i th trajectory . In the proposed methodology we sample the given policy , πold , for failure trajectories using Algorithm 1 which aims to minimize a set of objective functions , ϕ ( p ) , derived from the specification over the given set of parameter bounds P . ϕ ( p ) , is a manifestation of the negation of the safety criteria in terms of a set of objective functions that can be minimized . The function set ϕ derived from the safety criteria may involve negation , conjunction , and disjunction of multiple predicates over the variables in the system i.e. , ϕ : = µ | ¬µ | µ1∧µ2 | µ1∨µ2 . It is assumed that each predicate function µk is a smooth and continuous function over trajectories ξ . Violation of any µk ∈ ϕ forms the basis of violation for the overall safety criteria . Each such component , µk is modeled as a separate GP in our methodology . This method is able to uncover more counterexamples than modeling ϕ with a single GP as demonstrated by [ 18 ] . For cases where a set of predicates need to be jointly optimized as shown in example 3.2 we normalize the variables . In each iteration , the parameter values selected by BO are evaluated against each µk and the evaluated value is used by the acquisition function to push each µk towards its minimum . Each trajectory , containing sampled parameters pi , that violates any µk is added in the failure trajectory set . This process continues until the entire sample space is covered i.e all the possible states that can cause failure is removed from the search space , or the time budget is exhausted . We use the extension of BO provided in [ 15 ] , that removes the failure area enclosed by an already explored trajectory from the search space , to uncover multiple failure trajectories . Example 3.2 A safety specification for lunar lander trained using RL might be that the lander can not be tilted at an angle while being close to the ground . The lander coordinates are specified by ( lx , ly ) and the angle is given by the variable langle . Concretely , let us assume 0 ≤ ly ≤ 10 and −1 ≤ langle ≤ 0 . Using predicate logic we can represent this safety specification as ly < 5→ langle ≥ −0.5 , which in words says that if the lander is at a vertical distance of less than 5m from the ground then the angle of the lander is greater than -0.5 . The negation of this specification can be rewritten as ϕ : ( ly < 5 ) ∧ ( langle < −0.5 ) . There are two predicates in the specification µ1 : ly − 5 < 0 and µ2 : langle + 0.5 < 0 . This can be converted as an equivalent BO objective function as min ( µ1 + µ2 ) . Intuitively , this means that if ϕ < 0 the safety specification is violated , for example one such evaluation of the function could be ly = 1 and langle = −0.8 . An example is shown in Figure 2b .

Summary: Authors introduce a safety-oriented policy-refinement method for reinforcement learning. Authors use Bayesian optimization to find safety-violation traces from a policy generated by a vanilla RL algorithm. Those traces are latter use to produce a new trained policy that satisfies the requirements. The method is meant for continuous-action environments