# ExtendAIST: Exploring the Space of AI-in-the-Loop System Testing
## 3 Extendaist
### Design
#### 4.5.2 Image Classification (EP_IC)

The datasets for image classification are designed for object detection and classification in many application areas. We take the dataset of image classification as a sub-extension point, EP_IC, to identify more samples for training and testing models of image classification. * MNIST. MNIST [39; 40] is a dataset for recognizing handwritten digits (0-9) including 70,000 images originating from the NIST database [98]. The MNIST dataset is composed of a training dataset with 60,000 images and a test dataset with 10,000 images, each of which contains half clear digits written by government staff and half blurred digits written by students. * EMNIST. EMNIST [41; 42] is an extension dataset of MNIST for identifying handwritten digits (0-9) and letters (a-z and A-Z). Therefore, there are totally 62 classes in EMNIST, including 10 classes of digits and 52 classes of letters. However, some uppercases and lowercases cannot be distinguished easily (e.g., C and c and K and k), then the letters are merged into 37 classes. * Fashion-MNIST. Fashion-MNIST [43; 44] is a dataset with the extremely same format and size of MNIST for identifying 10 classes of fashion products, such as T-shirt, trouser, pullover, dress, coat, sandals, shirt, sneaker, bag, and ankle boots. * ImageNet. ImageNet [45; 46] is an image dataset describing the synsets in the WordNet hierarchy with on average 1000 images. * CIFAR-10/CIFAR-100. CIFAR-10 and CIFAR-100 [47] are labeled image datasets consisting of 60,000 32 \(\times\) 32 color images, 50,000 of which are training images and 10,000 are test images. CIFAR-10 is divided into 10 classes, and there are 5000 training images and 1000 test images for each class. CIFAR-100 is divided into 100 fine classes, and each class contains 500 training images and 100 test images. #### 4.5.3 Self Driving (EP_SD)

Self driving is a typical autonomous system inspecting the performance of AI-in-the-loop system. Hence, the dataset for self driving is also regarded as a sub-extension point, EP_SD, for tasks such as obstacle detection and tracking and traffic signal recognition. * Udacity Challenge. Udacity Challenge dataset [48] is a set of images for training and testing the objects detection models in the Udacity Challenge competition to measure the performance of each participating model in terms of detection capability and classification precision. The Udacity Challenge dataset contains two parts according to its classification. One consists of 9420 images in three classes: car, truck, and pedestrian; and the other consists of 15,000 images in five classes: car, truck, pedestrian, traffic light, and bycicle. * MSCOCO 2015. MSCOCO 2015 [49; 50] is a dataset gathered from the daily scenes of common objects in the real-world environment, which aims at object recognition and localization. This dataset contains 165,482 training images, 81,208 verification images and 81,434 testing images in 91 classes, such as animal, vegetable, and human. * KITTI. KITTI dataset [51; 52; 53] contains the realistic images captured from the real-world driving environments such as mid-size city, rural area, and highway. All these images are divided into five classes: road, city, residence, campus, and human. Evaluation tasks include stereo, optical flow, visual odometry, object detection, and tracking. * Baidu Apollo. The Baidu Apollo [54] open platform provides the massive annotation data and simulation for training and testing autonomous driving tasks, such as obstacle detection and classification, traffic light detection, road hackers, obstacle trajectory prediction, and scene analysis under different street views and vehicle movement images. #### 4.5.4 Natural Language Processing (EP_NALP)

Automatic natural language processing with machine learning releases researchers from the labor-intensive work of text processing. It is essential to design datasets for training and testing systems with the ability of natural language processing, which guarantees the precision of these systems.