## 4 Experimental Setup
### Data Preparation and Implementation  
**Project under evaluation**. To investigate the ranking of test cases in CAD software, we select the most widely-used open-source CAD software, Project ArtOfIllusion, as an experimental dataset. We select Project ArtOfIllusion based on the following reasons. First, to conduct the experiment, we need an open-source project; second, our approach relies on the conversion from source code to vector and the tool for such conversion, i.e., CodeZhi, is designed for Java programs; third, we tend to select a well-known CAD software in the evaluation. Based on the above three reasons, we check the CAD software written in Java in GitHub (Github, [[http://github.com/](http://github.com/)]([http://github.com/](http://github.com/)), accessed on 16 May 2022) and then select the one candidate, Project ArtOfIllusion. (Source code of Project ArtOfIllusion, [[http://github.com/ArtOfIllusion/ArtOfIllusion/](http://github.com/ArtOfIllusion/ArtOfIllusion/)]([http://github.com/ArtOfIllusion/ArtOfIllusion/](http://github.com/ArtOfIllusion/ArtOfIllusion/)), accessed on 16 May 2022). Project ArtOfIllusion is a free, multi-platform modeling, animation, and rendering suite written in Java programming language. ArtOfIllusion features a simple and streamlined interface to a broad array of powerful features, including key-frame-based and pose-based animations as well as a built-in raytracer. There are 15 modules in ArtOfIllusion, including animation, icons, image, keystroke, material, math, object, procedural, script, texture, titleImages, ui, unwrap, util, and view. There are seven modules with pre-defined test cases, i.e., module animation, math, object, procedural, raytracer, texture, and util. We use the source code and test cases in these seven modules. To increase the diversity of the datasets, we created a new dataset based on 5-fold cross validation [43]. We randomly divided all the files in the seven modules into five folds. Table 2 shows the summary of two datasets in Project ArtOfIllusion, including Dataset 1 with seven modules and Dataset 2 with five folds. Due to the limited development budget allocated for open-source projects, some of functions are not directly covered by manually-written test cases. **Defect seeding**. We employ Project ArtOfIllusion in the evaluation. To conduct the scenario of triggering defects, we follow the existing work [44] and use program mutation to automatically seed defects in the source code of Project ArtOfIllusion. Program mutation is a technique of modifying the source code in a target program to create a defective program. The purpose of program mutation is to obtain a program with seeded defects [45]. To seed defects in the source code, we use a simple way to mutate the original source code. We select all functions without return values as the mutation candidates and apply mutation \begin{table}
\begin{tabular}{c c c} \hline \hline
**Module** & **\#Functions** & **\#Test Cases** \\ \hline math & 192 & 11 \\ object & 897 & 43 \\ procedural & 577 & 12 \\ animation & 994 & 19 \\ raytracer & 0 \(\dagger\) & 2 \\ texture & 459 & 2 \\ util & 65 & 2 \\ \hline
**Sum** & **3184** & **91** \\ \hline \hline \end{tabular}  
* Module raytracer only provides test cases that examine the functional behaviors of multiple modules. Thus, there are no functions in the module.  
\end{table} 
Table 2: Dataset information with functions and test cases in Project ArtOfIllusion. # _Functions_ indicates the number of functional units in the module. # _Test Cases_ indicates the number of test cases that are manually-written for testing the module. rules that remove 30% of the statements in functions. Such kinds of functions are considered as defective, and the other functions are considered as non-defective. Such mutation is easy to be implemented and can keep the programs compilable without any manual effort.