Delivering Test and Evaluation Tools for Autonomous Unmanned Vehicles to the Fleet

The Johns Hopkins University Applied Physics Laboratory (APL) is working to develop the next generation of test and evaluation (T&E) tools for maritime, air, and ground autonomous systems. Advancement in autonomy on unmanned vehicles is outpacing test ranges’ capability for effective T&E of these systems. DoD test ranges face the challenge of being able to execute a very limited number of live tests to validate increasingly complex systems. APL is performing research and development to help solve the cost and reliability issues associated with on-range T&E of autonomous systems. Using advanced optimization techniques to intelligently explore the highly complex state space in which autonomous systems operate, the Range Adversarial Planning Tool (RAPT) team is developing tools for test ranges to identify the most relevant tests for the full scope of maritime, airborne, and ground-based autonomous systems. The principal challenge with testing and evaluating autonomy is addressing the complex, NP-Hard (non-deterministic polynomial-time hard) interactions between autonomy and the environment. Decomposing the problem only exacerbates the situation by producing an intractable set of options in various mission conditions and internal states of an autonomous system. Therefore, autonomy can be evaluated only with precise understanding of the interactions between the autonomous vehicle and the environment, enabling delineation of which situations are effective from a T&E perspective and which are not. duce a new method for intelligently generating test scenarios that inform testers on the expected performance of the autonomous system. Our approach to designing informative test cases differs from recent work in validating autonomous systems. We are not focusing on fault detection1 or model checking2 of the underlying decision engine. Instead of modeling the underlying behavior of a black-box INTRODUCTION Designing test scenarios for validation and verification of autonomous vehicles is currently an expensive and involved process. Testing requires the input of subject-matter experts who are thoroughly versed in the behaviors of both the platform and the autonomous decision engine under test. Performing live tests is also very time consuming, which severely limits the number of tests that can be performed. In this article, we introG. E. Mullins et al. Johns Hopkins APL Technical Digest, Volume 33, Number 4 (2017), www.jhuapl. edu/techdigest 280 autonomous system,3 we try to model the relationship between scenario configuration and the resulting performance metrics for the autonomy. This approach has the advantage of greatly reducing the input and output state-spaces from those that are used for traditional fault detection. The result is a model that can detect and classify anomalous behavior based on the performance metrics it exhibits. A candidate test scenario is deemed interesting and informative if it is close in the configuration space to a performance boundary. In this article, we define a performance boundary as a location in the configuration space where a discontinuity or large change in performance occurs. For example, a performance boundary may occur when a slight change in obstacle position causes an autonomous agent to transition from succeeding in its mission to failing. By identifying test cases near performance boundaries, we can assist test designers in selecting scenarios that are most likely to be informative so they can design test plans tailored to the specific autonomy in question. The method described in this article is divided into two primary approaches: the search approach and the boundary identification (ID) approach. During the search approach, we use active learning to select new test cases that are run by the autonomy simulation. Gaussian process regression (GPR)4 is used to form a model of the autonomy performance and preferentially select regions that might indicate performance boundaries. The high dimensionality of the configuration space for an autonomy under test makes it intractable to simply perform an exhaustive spread of simulations. Thus, intelligent searching of the configuration space is necessary to obtain adequate coverage of the boundary regions while minimizing the number of simulations. In the boundary ID approach, samples generated during the search approach are used to identify the performance modes in the resulting data by using unsupervised clustering algorithms. Once test cases have been classified by their performance mode, the boundaries between performance modes are identified, and the tested scenarios adjacent to boundaries can be used to aid in live test design. TEST AND EVALUATION NEED The validation and verification of autonomous systems has been an active area of research in the past few years, particularly in the use of active learning for test case generation. Here we discuss recent work in the fields of verification of autonomous systems and automatic test case generation and how it relates to the work discussed in this article. Validation of autonomous systems for critical missions has been increasing in importance as robots have become more prevalent in ordnance disposal, search and rescue, and other systems that have very low tolerance for failure.5 There has been a lot of work to develop methods that provide performance guarantees based on model-checking and formal methods.2,6 These methods require that a model—which must fully describe the autonomy performance—be generated and exhaustively tested for exceptions that break the specifications. Models that have been used in the past include finite state machines1,7 and process algebras.6 A drawback of this approach is that a model must first fully describe the autonomy, and test engineers must have full access to the model. Given the increasing complexity of autonomous systems and the black-box nature of proprietary software, these limitations prevent these methods from being applied to many systems. Learning-based methods for developing models of black-box systems for robustness testing have been the topic of many recent papers.8 These techniques often combine a model-checking strategy with an incremental model inference or learning algorithm. Typically, a finite-state machine built using Angluin’s algorithm L* has been used.9,10 Our method differs from these techniques by not attempting to model the actual behavior of the autonomy with respect to its inputs. Rather, we try to model its overall performance with respect to the stimuli it receives from its mission and environment. One method similar to our approach of boundary identification is the production of a set models used for fault detection.11 Our focus on modeling autonomy performance based on mission and environmental inputs allows us to perform black-box testing of the autonomous system. RAPT ARCHITECTURE AND SIMULATION The Range Adversarial Planning Tool (RAPT) generates test-planning products by using data from simulations of the autonomy software. Our approach relies on GPR techniques to identify the boundary conditions of the autonomy in simulation; autonomy performance metrics intelligently guide our search algorithm to identify performance boundary scenarios. Our GPR search generates scenarios that vary environmental, mission, and vehicle states in an open-architecture simulation environment. As shown in Fig. 1, the RAPT system consists of three primary components: (1) an offline state description, (2) optimization algorithms, and (3) a simulation environment. The offline state description allows test designers to define the state space for air, ground, and maritime systems against which the autonomy will be tested. The entirety of the state space cannot be executed in simulation due to the sheer size, so RAPT explores this state space using a stochastic optimization algorithm to identify test scenarios for execution in simulation. Results from these simulations inform the optimization algorithms as to which subsequent simulations will be informative and help determine the performance bounds of the autonomy. Delivering Test and Evaluation Tools for Autonomous Unmanned Vehicles Johns Hopkins APL Technical Digest, Volume 33, Number 4 (2017), www.jhuapl. edu/techdigest 281 PROBLEM DEFINITION We define the performance boundary identification problem as one of discovering test cases in the scenario configuration space that are close to discontinuities or large gradients in the performance space (i.e., where small changes in the scenario result in large changes in autonomy performance). Definitions include the following: SYSTEM UNDER TEST Several candidate systems were developed to evaluate our search and boundary ID algorithms. The first category of candidate systems consisted of mathematical test functions that accept either a twoor three-dimensional input state and output one-dimensional scores. The second category of candidate systems comprised a simple unmanned undersea vehicle (UUV) scenario with a Of ine con guration values Environment state description Mission state description Vehicle state description External simulation (TENA enabled) Output • Obstacle distribution • Predictability • Rate of change • Etc. World • Objectives • Constraints • Time line • Tactics • Kinematics • Sensor performance • Communications • Etc. Actors