Counterexample Guided RL Policy Refinement Using Bayesian Optimization

1 Introduction . Classical Reinforcement Learning ( RL ) is designed to perceive the environment and act towards maximizing a long term reward . These algorithms have shown superhuman performances in games like GO [ 10 ] , motivating their use in safety-critical domains like autonomous driving [ 23 , 20 , 16 ] . These policies are mostly learnt in simulation environments such as TORCS [ 22 ] or CARLA [ 14 ] for accelerated learning . Therefore they are susceptible to perform poorly in the real world where domain uncertainty arises due to noisy data . Also , the choice of a RL policy is determined by the design of the reward signal , which if not constructed properly , may lead the agent towards unintended or even harmful behaviour . Hence the need for constructing policies that account for risks arising due to uncertainty and do not violate safety specifications . Safe RL aims to learn policies that maximize the expectation of the return while respecting safety specifications [ 17 ] . The existing literature on safe RL is broadly divided into three schools of thought namely , 1 ) Transforming the optimization criteria by factoring in costs derived from safety constraints [ 1 ] , 2 ) Using external knowledge in the form of a teacher or a safety shield to replace unsafe actions chosen by the policy during exploration / deployment [ 2 ] , and 3 ) Establishing the existence of an inductive safety invariant such as a Lyapunov function [ 11 ] or a control barrier function [ 25 ] . Establishing such an invariant is often intractable for Neural Network controllers that use many parameters and work on high-dimensional state spaces . In this work , we propose a counterexample guided policy refinement method for verifying and correcting a policy that has already been optimized and calibrated . Failure traces may occur in such a policy due to the following reasons : a ) Failure trajectories are extremely rare and do not contribute significantly enough to reduce the expected reward , b ) Uncertainty in the environment ∗Code is available online at https : //github.com/britig/policy-refinement-bo 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . which had not been accounted for during policy training . Our aim is to progressively make the system safer by finding and eliminating failure traces with targeted policy updates . Counterexample guided refinement techniques have been well studied in the formal methods community for program verification [ 9 ] and reachability analysis of hybrid systems [ 3 ] , but not so far in the context of RL . We study the proposed methodology on environments that work over continuous space and continuous or discrete actions . The proposed methodology is divided into two steps : 1 . Given a policy πold , learnt for optimizing reward in a given environment , we test it against parameters with uncertainties and a set of objective functions ϕ derived from the negation of the given safety criteria . Multiple counter-example traces or failure trajectories are uncovered from πold using Bayesian Optimization . 2 . Using the failure trajectories we selectively do a gradient update on πold to construct a new policy πnew , that excludes the counterexample traces under the given domain uncertainties . We specifically work with a policy gradient RL algorithm , Proximal Policy Optimization ( PPO ) , which directly optimizes the learnt policy using gradient updates . These steps may be repeated multiple times . We show that such targeted updates ensure that πnew is safer than πold , but not significantly different in terms of reward . The paper is organized as follows . Section 2 outlines related work , Section 3 presents the methodology to uncover failure trajectories , Section 4 presents the methodology of policy refinement , Section 5 presents case studies on several RL environments , and Section 6 provides concluding remarks . 2 Related Work . The approach proposed in this paper uses Bayesian Optimization for finding adversarial tests cases in terms of trajectories for the RL policy leading to failure of the safety specification . The failures are then used to make the policy safer . Bayesian Optimization ( BO ) [ 28 ] , is a machine learning technique for finding the global maxima/minima of an unknown function based on stochastic evaluations . BO has been extensively used in combinatorial optimization [ 4 ] , hyper-parameter optimization for neural networks [ 29 ] and optimization of parameters in robotics . In recent literature , BO has been successfully applied for generation of adversarial counterexamples for complex controllers [ 13 , 18 ] including identification of test cases for autonomous driving systems [ 15 ] , and coverage of faults in analog and mixed circuits [ 19 ] . Traditional Bayesian Optimization is limited to finding a single global minima/maxima which can lead to the discovery of only one counterexample . Hence , extensions that find multiple test cases has been proposed on top of traditional BO [ 15 , 19 ] . In this work , we closely follow the formulations and extensions of [ 15 ] . Safe RL is an area of emerging significance buoyed by the proliferation of RL in safety-critical control systems . A survey of RL safety problems and taxonomies can be found in [ 17 ] . The goal of safe RL is to create agents whose behaviour is aligned with the user ’ s intentions . Achieving this via reward shaping [ 21 ] causes a trade-off between a reward enacting task-solving behaviour and a penalty for non-avoidance of hazards . Less penalty promotes unsafe behaviour and high penalty dilutes the objective goal of the RL . A natural way to incorporate safety specification in RL is by imposing constraints . A substantial body of work exists in this direction [ 24 , 12 , 7 , 8 , 1 ] . However , there is no obvious guarantee that the framework of constrained RL solves the agent alignment problem . Constrained RL requires the specification cost functions for the constraints , along with the reward function , with susceptibility towards design errors [ 26 ] . In the proposed method we neither change the reward function nor introduce constraint costs . Our policy corrections are affected by changing the policy gradient based on violations of the safety specifications . Another school of work uses Preference-Based RL , where the agent learns directly from an expert ’ s preferences of trajectories instead of a hand-designed numeric reward . Preference-Based RL suffers from the temporal credit assignment problem , that is , determining which states or actions contribute to the encountered preferences [ 31 ] . Counterexample guided refinement has been studied in Apprentice Learning ( AL ) using PCTL model checking [ 32 ] . Our approach is applicable to a wider range of applications since it can handle larger state spaces than the model checking approach from [ 32 ] . The authors in [ 6 ] do a counterexample based improvement on RNN-based strategy on a POMDP using Linear Programming . Model-checking is performed on an RNN strategy induced MDP and the strategy is refined using Linear Programming based on counterexamples . The application of their work is limited to environments with discrete state space . In their setting , the POMDP model is known apriori , which is not true in our setting . 3 Finding Failure Traces using Bayesian Optimization . This section outlines our approach for using Bayesian Optimization to look for failure traces in a trained RL policy , πold , with respect to a given safety specification . BO is a machine learning approach formulated as : argmin p∈P f ( p ) where f : Rd → R is continuous but lacks special structure and p ∈ P where P is a set of function parameters . BO is used to optimize derivative free , expensive-to-evaluate black box functions . In its core BO uses Gaussian Process ( GP ) models to estimate a prior over the function f . The function value , f ( p ) , is treated as a random variable by a Gaussian Process , which enables the modelling of a finite number of observations using a joint Gaussian distribution . The distribution initially has an assumed mean , m ( p ) , at 0 and the kernel , κ ( pi , pj ) , is a squared exponential function of the parameters p , e− ( |pi−pj | ) 2 . For every unobserved value p∗ , the mean , m , and variance , σ2 can be estimated using the following equations [ 28 ] : m ( p∗|p ) = m ( p∗ ) +KT∗ K−1 ( f ( p ) −m ( p ) ) ( 1 ) σ2 ( p∗|p ) = K∗∗ −KT∗ K−1K∗ ( 2 ) where K = κ ( p , p ) , K∗ = κ ( p , p∗ ) , and K∗∗ = κ ( p∗ , p∗ ) . An acquisition function is then used to sample the estimated prior , along with assumed mean and variance , for unobserved points to derive a posterior distribution . The acquisition function chosen in this paper is Expected Improvement , which represents the quantum of improvement while exploring and exploiting the function to find a global minimum . If f∗ ( p ) is the minimal value of f ( p ) observed so far and f ' ( p ) is the new evaluation , then the expected improvement utility , EI , is given by : EI ( p ) = E [ max ( 0 , ( f∗ ( p ) − f ′ ( p ) ) ) ] where , E indicates the expectation taken under the posterior distribution , given evaluations of f . The expected improvement algorithm then chooses to evaluate the function at the point having the largest expected improvement , namely : pn+1 = argmaxEI ( p ) The prior model improves through repeated sampling using the acquisition function until the improvement becomes insignificant ( indicative of a global minima ) or the computation budget is exhausted . We now explain the use of BO in our framework . The following example elucidates the problem . Figure 1 : A toy grid environment Example 3.1 The toy grid-world of Fig . 1 has two start locations at ( 0,0 ) and ( 1,0 ) , and a single goal location at ( 0,3 ) . The agent has two actions , `` left '' for moving left , and `` up '' for going up . The reward is indicated beside some of the locations , and is -1 for all other locations . Suppose the safety requirement is that the agent should never reach a location having a reward less than -1 , which is modeled by the predicate : ¬ ( reward ( state ) + 1 < 0 ) . In the absence of this safety requirement , the following trajectories are optimal in terms of total reward : Trajectory-1 : ( 0 , 0 ) left−−→ ( 0 , 1 ) left−−→ ( 0 , 2 ) left−−→ ( 0 , 3 ) Trajectory-2 : ( 1 , 0 ) left−−→ ( 1 , 1 ) left−−→ ( 1 , 2 ) left−−→ ( 1 , 3 ) up−→ ( 0 , 3 ) Trajectory-2 violates the safety specification at ( 1,1 ) , and is therefore a failure trajectory . BO helps us to find such failure traces automatically , by intelligently testing the RL policy with adversarial tests derived from the safety specification . Algorithm 1 : Finding Failure Trajectories Input : Policy πold , Parameter Bounds P , Objective functions ϕ , Agent A Function identify_failures ( πold , P , ϕ , A ) : i← 1 while i < max_budget do Sample parameters pi ∈ [ Plow , Phigh ] based on GP models j ← 1 oj ← pi while not done do aj ← πold ( oj ) oj+1 ← A.step ( aj ) ξi← ξi∪ ( oj , aj ) oj ← oj+1 j ← j + 1 end ϕvali ← argminξi ( µ1 ( ξi ) ) . . . ( µn ( ξi ) ) Update each GP model with measurements ( pi , µk ( pi ) ) ξf ← ξf ∪ ( ξi with ϕvali < 0 ) remove pi from search space i← i+ 1 end return ξf Comments : ϕ : Set of objective functions corresponding to predicates in negation of safety criteria . ξi : A sequence of states ot , actions at over time t = 0 , 1 , . . . ϕvali : Evaluation of the objective functions in ϕ for the i th trajectory . In the proposed methodology we sample the given policy , πold , for failure trajectories using Algorithm 1 which aims to minimize a set of objective functions , ϕ ( p ) , derived from the specification over the given set of parameter bounds P . ϕ ( p ) , is a manifestation of the negation of the safety criteria in terms of a set of objective functions that can be minimized . The function set ϕ derived from the safety criteria may involve negation , conjunction , and disjunction of multiple predicates over the variables in the system i.e. , ϕ : = µ | ¬µ | µ1∧µ2 | µ1∨µ2 . It is assumed that each predicate function µk is a smooth and continuous function over trajectories ξ . Violation of any µk ∈ ϕ forms the basis of violation for the overall safety criteria . Each such component , µk is modeled as a separate GP in our methodology . This method is able to uncover more counterexamples than modeling ϕ with a single GP as demonstrated by [ 18 ] . For cases where a set of predicates need to be jointly optimized as shown in example 3.2 we normalize the variables . In each iteration , the parameter values selected by BO are evaluated against each µk and the evaluated value is used by the acquisition function to push each µk towards its minimum . Each trajectory , containing sampled parameters pi , that violates any µk is added in the failure trajectory set . This process continues until the entire sample space is covered i.e all the possible states that can cause failure is removed from the search space , or the time budget is exhausted . We use the extension of BO provided in [ 15 ] , that removes the failure area enclosed by an already explored trajectory from the search space , to uncover multiple failure trajectories . Example 3.2 A safety specification for lunar lander trained using RL might be that the lander can not be tilted at an angle while being close to the ground . The lander coordinates are specified by ( lx , ly ) and the angle is given by the variable langle . Concretely , let us assume 0 ≤ ly ≤ 10 and −1 ≤ langle ≤ 0 . Using predicate logic we can represent this safety specification as ly < 5→ langle ≥ −0.5 , which in words says that if the lander is at a vertical distance of less than 5m from the ground then the angle of the lander is greater than -0.5 . The negation of this specification can be rewritten as ϕ : ( ly < 5 ) ∧ ( langle < −0.5 ) . There are two predicates in the specification µ1 : ly − 5 < 0 and µ2 : langle + 0.5 < 0 . This can be converted as an equivalent BO objective function as min ( µ1 + µ2 ) . Intuitively , this means that if ϕ < 0 the safety specification is violated , for example one such evaluation of the function could be ly = 1 and langle = −0.8 . An example is shown in Figure 2b .

Summary: The paper proposes a training methodology to incorporate constraints into a trained RL policy. It uses Bayesian optimization to select parameters for the MDP that are most likely to produce trajectories violating the constraints and then performs PPO updates to steer the policy away from these trajectories while remaining close to the original one. The experiments are conducted on popular OpenAI gym environments.