Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures

1 INTRODUCTION . How can we ensure machine learning systems do not make catastrophic mistakes ? While machine learning systems have shown impressive results across a variety of domains ( Krizhevsky et al. , 2012 ; Mnih et al. , 2015 ; Silver et al. , 2017 ) , they may also fail badly on particular inputs , often in unexpected ways ( Szegedy et al. , 2013 ) . As we start deploying these systems , it is important that we can reliably evaluate the risk of failure . This is particularly important for safety critical domains like autonomous driving where the negative consequences of a single mistake can overwhelm the positive benefits accrued during typical operation of the system . Limitations of random testing . The key problem we highlight is that for standard statistical evaluation , attaining confidence that the failure rate of a policy is below requires at least 1/ episodes . We now informally summarize this point , which we discuss further in Appendix A . For concreteness , consider a self-driving car company that decides that the cost of a single accident where the car is at fault outweighs the benefits of 100 million miles of faultless operation . The standard approach in machine learning is to estimate the expected return via i.i.d . samples from the data distribution ( frequently a test set ) . For tightly bounded returns , the sample estimate is guaranteed to quickly converge to the true expectation . However , with catastrophic failures , this may be prohibitively inefficient . In our current example , any policy with a failure probability greater than = 10−8 per mile has negative expected return . In other words , it would be better to not deploy the car . However , to achieve reasonable confidence that the car crashes with probability below 1e–8 , the manufacturer would need to test-drive the car for at least 1e8 miles , which may be prohibitively expensive . ∗Equal Contribution Our Contributions . To overcome the above-mentioned problems , we develop a novel adversarial evaluation approach . The central motivation behind our algorithmic choices is the fact that realworld evaluation is typically dominated by the cost of running the agent in the real-world and/or human supervision . In the self-driving car example , both issues are present : testing requires both operating a physical car , and a human test driver behind the wheel . The overarching idea is thus to screen out situations that are unlikely to be problematic , and focus evaluation on the most difficult situations . The difficulty arises in identifying these situations – since failures are rare , there is little signal to drive optimization . To address this problem , we introduce a continuation approach to learning a failure probability predictor ( AVF ) , which estimates the probability the agent fails given some initial conditions . The idea is to leverage data from less robust agents , which fail more frequently , to provide a stronger learning signal . In our implementation , this also allows the algorithm to reuse data gathered for training the agent , saving time and resources during evaluation . We note that adversarial testing is a well-established idea ( see Section 5 ) , but typically requires either a dense optimization signal or expert domain knowledge . We avoid these stumbling blocks by relying on the learned AVF , which guides the adversarially acting evaluator . We look at two settings where the AVF can be used . In the simplest setting , failure search , the problem is to efficiently find inputs ( initial conditions ) that cause failures ( Section 2.1 ) . This task has several uses . First , an adversary that solves this task efficiently allows one to identify and debug potentially unsafe policies . Second , as has been done previously in the supervised learning literature , efficient adversaries can be used for adversarial training , by folding the states causing failures back into the training algorithm ( Madry et al. , 2017 ) . The second setting , risk estimation , is the problem of efficiently estimating the failure probability of an agent ( Section 2.2 ) , which also has a simple application to efficiently selecting the most reliable agent from a finite set ( Section 4.3 ) . Empirically , we demonstrate dramatic improvements in efficiency through adversarial testing on two domains ( simulated driving and humanoid locomotion ) . In summary , we present 3 key contributions : 1 . We empirically demonstrate the limitations of random testing . We observe that with random testing , the cost of reliably obtaining even a single adversarial input can exceed the entire cost of training . Further , reliably estimating risk can exceed training costs . 2 . We describe a continuation approach for learning failure probability predictors even when failures are rare . We develop algorithms applying failure probability predictors to failure search , risk estimation , and model selection . 3 . We extensively evaluate our method on simulated driving and humanoid locomotion domains . Using adversarial evaluation , we find failures with 198 and 3100 times fewer samples respectively . On Humanoid , we bring the cost of reliable risk estimation down from greater than the cost of training to a practical budget . 2 PROBLEM FORMULATION . We first introduce our notation . Recall that we are interested in reliability assessment of a trained agent . We assume that the experimenter who performs the reliability assessment can run an experiment ( equivalently , a rollout or episode ) with the trained agent given some initial condition x ∈ X , the outcome of which is a random failure indicator C = c ( x , Z ) where Z ∼ PZ for some probability distribution PZ over some set Z and where c : X × Z → { 0 , 1 } . In particular , C is binary and C = 1 indicates a “ catastrophic failure ” . The interpretation of Z is that it collects all sources of randomness due to the agent and the environment . Unlike the case of x , Z is neither observed , nor controllable by the experimenter . We are interested in the agent ’ s performance on the environment distribution over initial conditions X ∼ PX , which we assume can be sampled quickly.1 In the real-world , evaluating c requires running the agent in the real-world and/or human supervision , which generally dominate the cost of evaluating neural networks . It is thus assumed that evaluating c on the pair ( x , Z ) is costly . In Section 4.1 , we show that even on relatively cheap simulated environments , the cost of simulating environments dominates costs of evaluating neural networks . 1 To minimize jargon , we omit standard conditions on the spaces and functions that permit the use of the language of probabilities . 2.1 FAILURE SEARCH . The objective of failure search is to find a catastrophic failure of the agent . Algorithms that search for failures , which we will call adversaries , can specify initial conditions x , observe the outcome C of running the agent on x and return as soon as C = 1 . Thus , for each round t , the adversary can choose Xt ∼ Pt ( ·|X1 , . . . , Xt−1 ) , observe Ct = c ( Xt , Zt ) , and return if Ct = 1 . Here , Zt ∼ PZ , independently of the history ( X1 , . . . , Xt , Z1 , . . . , Zt−1 ) . The naive adversary evaluates the agent on samples from PX until observing a failure . We assess adversaries by either the expected number of episodes , or the expected time elapsed , until returning a failure case . To design a more efficient adversary , the experimenter is allowed to collect historical data of the form ( X ′1 , θ1 , C ′ 1 ) , . . . , ( X ′ n , θn , C ′ n ) while they are training their agent . Here , the initial condition X ′t and θt ∈ Θ are chosen by the training procedure , where θt encodes information about the agent used in round t to generate the observation C ′t = c ( X ′ t , Z ′ t ) . 2.2 RISK ESTIMATION . As before , let PX denote the distribution over initial states X defined by the environment . The failure probability of the trained agent is p = E [ c ( X , Z ) ] , where X ∼ PX and Z ∼ PZ independently . A typical goal is to estimate p up to a fixed relative accuracy with high probability . Given some ρ > 1 , δ ∈ ( 0 , 1 ) , an algorithm is said to be ( ρ , δ ) correct if the estimate P̂ produced belongs to the [ p/ρ , pρ ] interval with probability at least 1 − δ . When P̂ belongs to the said interval , we say that it is a ρ-approximation of p. As before , the algorithm has access to historical data and can quickly sample from PX . The naive or vanilla Monte Carlo ( VMC ) estimator samples from PX and returns the average of the observed outcomes . We assess estimators by the total number of experiments required . 3 APPROACH . In this section we describe our approach to the two problems outlined above . A common feature of our proposed solutions is that they estimate the failure probability predictor ( AVF ) f∗ : X → [ 0 , 1 ] that returns the probability of failure given a initial condition ( equivalently , the value function of an adversary which receives reward 1 for failures , without discounting ) : f∗ ( x ) = P ( c ( x , Z ) = 1 ) , x ∈ X , where Z ∼ PZ . Our continuation approach to learning an approximation f ≈ f∗ , f : X → [ 0 , 1 ] is described in Section 3.3. f will be chosen so that the cost of evaluating f is negligible compared to running an experiment . The idea is to use f to save on the cost of experimentation . Our solutions build on the certainty equivalence approach ( Turnovsky , 1976 ) : First , we describe how f∗ ( if available ) could be leveraged . In cases of certainty equivalence , we would substitute f for f∗ . We then introduce additional heuristics to reduce sensitivity to the mismatch between f and f∗ . 3.1 FAILURE SEARCH . When f∗ is known , the optimal adversary has a particularly simple form ( proof left to the reader ) : Proposition 3.1 . The adversary that minimizes the expected number of rounds until failure evaluates the agent repeatedly on an instance x∗ ∈ X that maximizes f∗ : X → [ 0 , 1 ] . Having access to f ≈ f∗ , a natural approach is to evaluate the agent on arg maxx f ( x ) . There are two problems with this : ( i ) a maximizer may be hard to find and ( ii ) there is no guarantee that the maximizer of f will give a point where f∗ is large . Rather than always select the global maximizer of f , one way to increase the robustness of this procedure is to sample diverse initial conditions . We adopt a simple procedure : sample n initial conditions from PX , pick the initial condition from this set where f is the largest , and run an experiment from the found initial condition . We repeat this process with new sampled initial conditions until we find a catastrophic failure . The pseudocode is included in Appendix C as Algorithm 2 ( AVF Adversary ) .

Summary: This paper proposed an adversarial approach to identifying catastrophic failure cases in reinforcement learning. It is a timely topic and may have practical significance. The proposed approach is built on importance sampling for the failure search and function fitting for estimating the failure probabilities. Experiments on two simulated environments show significant gain of the proposed approaches over naive search.