# ExtendAIST: Exploring the Space of AI-in-the-Loop System Testing
## 1 Introduction  
With the broad applications of artificial intelligence on wearable devices, autonomous cars, and smart city, the traditional embedded system with predefined control strategies, including sensor, controller, and actuator, has evolved into the AI-in-the-loop system (AIS). AI-in-the-loop system is an intelligent system with capabilities of self-learning and autonomous decision because of embedded Artificial Intelligence (AI) components/modules or implementation of AI solutions in an embedded system. For instance, autonomous cars [1] perceive the surrounding driving environment by sensors such as radar and camera. Based on the perception data, autonomous cars perform tasks such as obstacles detection and tracking, traffic signals detection and recognition, and route planning through the AI modules implemented under machine learning algorithm. Autonomous cars then make decisions for these tasks and drive automatically on road via the autonomous interactions with driving environment. The safety-critical system with AI module embedded is potentially unsafe. For example, the adaptive cruise control system is an intelligent system to make cars drive automatically by interacting with the driving environment and learning the corresponding driving behaviors, such as speed up, slow down, and brake. However, the traffic accident of Tesla Model S, which crashes because of the misclassification between the sky and white truck, indicates that the AI module may be vulnerable to the perturbation and make AI-in-the-loop system unsafe. Once some faults in AI module occur without any safety control behavior from system or person, there will be either error decisionor fatal consequence. Therefore, it is crucial to detect the erroneous behaviors in AIS and assure its properties (e.g., correctness and robustness) through certain testing techniques. Some researchers have reviewed the progress of machine learning testing from different perspectives. For example, [PERSON] et al. [2] investigated the existing verification and simulation techniques for the safety of machine learning. [PERSON] et al. [3] reviewed the error detection of machine learning and the applications of conventional software testing on machine learning. [PERSON] et al. [4] studied the datasets, and black- and white-box based testing techniques for machine learning testing. [PERSON] et al. [5] discussed the safety challenge in machine learning system. [PERSON] et al. [6] introduced the testing and verification techniques for the safety of machine learning. [PERSON] et al. [7] presented a comprehensive survey on machine learning testing. However, the above-mentioned studies mainly focus on the testing or verification techniques without a systematic testing procedure from test data generation to evaluation except the review in [7], and few studies present the differences between testing conventional system and AI-in-the-loop system. Even though [PERSON] et al. [7] presented a thorough introduction of machine learning testing, including the techniques of detecting data, learning program, framework, open-source tool, and testing objects, they did not mention how to develop a new technique and the further work researchers can conduct based on each testing step. Inspired by the above enquiries, we follow a similar research method as [PERSON] [8] to propose an Extendable framework of AIS Testing, namely ExtendAIST. The ExtendAIST takes each involved testing step as extension point and implemented approaches as sub-extension points to organize the design space of AIS testing techniques practically. Additionally, this framework provides the existing techniques commonly present in individual sub-extension point. Therefore, debuggers can reuse these techniques, extend new techniques over the extension and sub-extension points, or implement new points in the space of AIS testing. The primary contributions of our paper are summarized as follows.