# ExtendAIST: Exploring the Space of AI-in-the-Loop System Testing
## 3 Extendaist
### Design
Therefore, we take the dataset for natural language processing as a sub-extension point, EP_NALP, to design more processing method for various processing tasks. * Enron. Enron [55] is an email dataset collected from 150 senior managers of Enron, which contains about 500,000 real email messages. All emails from one manager are organized into a folder, which contains the information of message-ID, date, sender, recipient, and messages. * bAbI. bAbI [56] is a dataset from Facebook AI Research with 20 toy question-answering tasks. Each task contains 1000 training examples and 1000 test examples, which aims at reasoning or answering questions after training. * Common Crawl. The Common Crawl corpus [57] contains petabytes of data including raw web page data, metadata extracts, and text extracts by web crawling. This corpus provides web-scaled data for developing or evaluating natural language processing algorithms empirically. * SNLI. The SNLI corpus [58; 59] contains 570,000 handwritten English sentence pairs with the manual labels entailment, contradiction, and neutral. This dataset is used to evaluate models of natural language inference. #### 4.5.5 Speech Recognition (EP_SR)

This sub-extension point EP_SR allows designing datasets for learning speech recognition, with which the information omitted by human can be detected and represented by machine. * Speech Commands. Speech Commands [60; 61] is an audio dataset for detecting single keyword from a set of spoken words. This dataset focuses on the audio of on-device trigger phrases and contains 105,829 utterances of 35 words in format of WAVE files. All these utterance were recorded from 2618 speakers, and each utterance is one second or less. * Free Spoken Digit Dataset. FSDD [62] is a dataset of 2000 audio recordings of spoken English digits in format of WAVE files. These speeches were recorded by four speakers with 50 recordings of each digit per speaker. * Million Song Dataset. This dataset [63; 64] provides the feature analysis and metadata of one million popular songs, and the audio can be fetched by the code in this dataset. * LibriSpeech. LibriSpeech [65; 66] is a collection of about 1000 hours of read English speech with the labels of sentence level. Therefore, this dataset is suitable for full speech recognition. #### 4.5.6 Android Malware Detection (EP_AMD)

The sub-extension point EP_AMD allows researchers to design more malware datasets or enlarge the scales of the following existing datasets, detect the Android malware, and ensure the security of smart phones with the rapid development. * Drebin. The Drebin dataset [67; 68; 69] contains 5560 Apps from 179 different malware families, which is a lightweight dataset to identify Android malware. * Genome. The Genome dataset [70; 71] has collected more than 1200 malwares in different aspects, such as the installation methods, activation mechanism, and the nature of carried malicious payloads. * VirusTotal. VirusTotal [72] is a dataset of mobile Apps samples for analyzing suspicious files and URLs to detect types of malware including viruses, worms, and trojans. * Contagio. The Contagio [73] is a platform for collecting the most recent malware, thus it can be used to analyze and help detect unknown malware. ## 5 Discussion

Although we have investigated the possible extension, sub-extension points, and existing techniques of AI-in-the-loop system testing, the AI techniques update significantly with the increasing development of application areas and computation ability, which lead to the crucial fact that even the points provided in ExtendAIST are not adequate to develop new techniques. This case will be the main threat to the validity of our proposed ExtendAIST framework. ## 6 Conclusions

The AI-in-the-loop system is defined in this paper for discussing the main differences between testing ordinary system and AI-in-the-loop system in terms of testing coverage, testing data, testing property, and testing approach for unit testing and system testing, respectively.