# ExtendAIST: Exploring the Space of AI-in-the-Loop System Testing
## 2 Motivation
### AI-in-the-Loop System Testing  
The nature of AI module design and implementation, the complex requirements, large-scale network, non-deterministic and uninterpretable learning results bring great challenges in testing AI-in-the-loop system [10; 11; 12], which distinguishes from traditional system testing. The traditional system is always a logic deterministic program with the definite output against the system under testing (SUT) and the corresponding test case. The execution behavior can be tracked and interpretable with the absolutely deterministic control flow and data flow. However, the output of AI module is non-deterministic. For example, the machine learning based AI module is a data-driven program [7] which delivers a trained model by learning from the training dataset under the learning algorithm. The predicted label of the trained model may be distinct even for the same test data, because the trained model evolves with different training datasets so that different weights and biases are trained. The testing approaches commonly used in testing ordinary system and non-AI modules are not suitable for testing the AI modules. Therefore, more novel testing coverage metrics and test data are necessary when performing the unit testing of AI-in-the-loop system. More testing properties and testing techniques should be investigated when conducting the system testing of AI-in-the-loop system for the behavior consistency. For the unit testing of traditional system and AI-in-the-loop system, the test adequacy of traditional system can be measured by control flow coverage (e.g., SC, DC, CC, and MC/DC) and data flow coverage (e.g., p-use and c-use). Except for the coverage metrics used above for non-AI modules, the test adequacy of AI modules should be measured by the structure coverage of neural network because of its non-deterministic logic, such as neuron level coverage [13; 14; 15], layer level coverage [15], and neuron pair level coverage [16]. Based on the above, all activated neurons leading to major function behaviors and corner case behaviors can be covered. The common dataset designed for different application areas sometimes consists of training and testing data. However, the common  
Figure 1: Architecture of the AI-in-the-loop system. dataset is not sufficient for each testing task. To meet the specific testing tasks of AI-in-the-loop system, new dataset including training dataset and testing dataset is generated by some test data generation strategies. The training dataset is used to train the learning program to get a trained model, and the testing dataset is used to evaluate the training effectiveness of trained model. For the system testing of traditional system and AI-in-the-loop system, the traditional system testing mainly focuses on the following functional and non-functional properties, such as correctness, real-time, reliability and safety. Due to the non-deterministic behavior of AI module, the AI-in-the-loop system suffers from the oracle problem. Therefore, more properties beyond those of traditional systems, such as the robustness [7] and stiffness [17] of the AI-in-the-loop system, should be tested by checking the behavior consistency. The robustness measures the resilience of AI-in-the-loop system towards perturbations on data or learning program. For example, if the trained model misclassifies the test data with imperceptible perturbation, then AI-in-the-loop system is said to be of low robustness. The stiffness is proposed to measure the resistance of AI-in-the-loop system by the effect of a small gradient step on one test data upon the loss on another test data. That is, the predicted label of AI-in-the-loop system is convergent for the gradient alignment between different test data. To test unique properties of the AIS, more testing techniques such as adversarial attack [18] and generative adversarial network [19] are proposed.