# ExtendAIST: Exploring the Space of AI-in-the-Loop System Testing
## 3 Extendaist
### Design
\"\(\checkmark\)\" implies that the sub-extension point is available for the related testing requirement. Take the \"Correctness\" row as example, all the coverage metrics in different granularity levels of extension point EP_TCM can be used to measure the coverage of network. All five kinds of evaluation common datasets are available for the correctness testing. In the extension point of EP_TDG, only sub-extension point EP_MT can be used to generate test data for testing correctness of AIS. When selecting the testing technique, sub-extension points EP_MUT, EP_MT, and EP_TP are appropriate to perform AIS correctness testing. \begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c c c c c c c c c c c c c c} \hline \hline

### Test Coverage Metric (EP_TCM)

#### 4.1.1 Description

Test coverage metric selection is the first step of test data generation as test data must satisfy the requirement of coverage criterion. There already exist many test coverage metrics for traditional software testing adequacy: statement coverage, condition coverage, decision coverage, and Modified Condition/Decision Coverage (MC/DC). However, all the metrics above cannot be used directly to cover the control and data flow of AI-in-the-loop system due to the structure of learning program for AI module and the non-deterministic program logic. Researchers take neuron as the basic coverage unit to conduct both major function behavior coverage and corner-case behavior coverage of neural network [15]. We divide the neural network coverage into three categories from the perspective of granularity: _neuron level coverage_, _layer level coverage_, and _neuron pair level coverage_. Therefore, ExtendAIST takes test coverage metric as an extension point EP_TCM, and neuron level coverage, layer level coverage, and neuron pair level coverage as sub-extension points named EP_NL, EP_LL, and EP_NPL, respectively. Then, testers can implement new coverage metric based on these extension points or reuse the existing techniques. #### 4.1.2 Neuron Level (EP_NL)

The neuron level coverage depends on the output value of neuron. There currently exist four kinds of neuron level coverage in this sub-extension point EP_NL: _activated neuron coverage_, _k-multisection neuron coverage_, _neuron boundary coverage_, and _strong neuron activation coverage_. * Activated neuron coverage. A neuron is considered _activated_ if the neuron output is greater than the neuron activation threshold and makes contribution to neural network's behaviors including major function and corner-case behaviors. As shown in Equation (1), the _activated neuron coverage_[13,14]\(Cov_{ANC}\) is the rate of the number of neurons activated and the number of neurons in the whole DNN. \[Cov_{ANC}=\frac{n_{activated}}{N}\] (1)
* \(k\)-multisection neuron coverage. [PERSON] et al. [15] proposed using neuron output value range \([low_{n},high_{n}]\) to distinguish the major function region and corner-case region. Then, we can measure the coverage of major function region \([low_{n},high_{n}]\) by dividing this region into \(k\) equal subsections. As shown in Equation (2), _k-multisection neuron coverage_ for a neuron \(n\), \(Cov_{k,n}\) is the rate of the number of subsections covered by \(T\) and the total number of subsections, in which \(x\) is a test input in dataset \(T\), \(\phi(x,n)\) is the output of neuron \(n\) with test input \(x\), and \(S_{i}^{n}\) is the set of values in the \(i\)th subsection. The _k-multisection neuron coverage_ for the neural network \(N\), \(Cov_{KMN}\), is based on the _k_-multisection neuron coverage of all neurons in network, which is defined in Equation (3). \[Cov_{k,n}=\frac{|\{S_{i}^{n}|\exists x\in T:\phi(x,n)\in S_{i}^{n}\}|}{k}\] (2) \[Cov_{KMN}=\frac{\Sigma_{n\in N}|\{S_{i}^{n}|\exists x\in T:\phi(x,n)\in S_{i}^ {n}\}|}{k\times|N|}\] (3)
* Neuron boundary coverage. In some cases, neuron output \(\phi(x,n)\
otin[low_{n},high_{n}]\).