## 5 Empirical Studies: Industrial Cases
### Results  
We used a total of five existing code-based TCP methods for model-based mutation testing. The major reason is that we wanted to compare the model-based TCP results with code-based TCP results. In the three empirical studies, Random, StTotal, and StAddtl afforded APFDs lower than those of FepTotal and FepAddtl regardless of model complexity. In most cases, the model-based TCPs did not differ markedly from those of general code-based methods. In terms of answering RQ2, FepAddtl showed the best results among those five methods excluding AVM, equivalent to those of code-based TCP. The reason that FepAddtl was the best optimal method was because it can achieve high exposure probability for the faults in the early stage of development. However, this does not mean that all test results will be the same; our experimental environment was limited. To answer RQ3, we used a TCP method employing the AVM to create a search-based algorithm. All results were obtained under the same conditions, thus, without optimization; the APFD values ranged from 0.01 to 0.25 depending on software complexity. The APFD figures fell as software complexity increased. The answer to RQ3 is that a search-based algorithm can be used for model-based TCP in the same manner as a code-based technique. In the above industrial scenarios, the AVM APFD results did not differ greatly from those of other algorithms, but the AVM was quicker. The search time, of course, depends on local optimization and the exploration algorithm employed. Fault seeding threatens internal validity. To minimize this, we used mutant operators of model-based state diagrams as seeds. To ensure reliable data collection within a reasonable time, all experiments were 100-fold iterated for each algorithm, yielding statistically robust comparisons. However, this does not mean that the APFD results of TCP methods will always be the same. We constructed an external validity threat and explored the effect thereof on TCP, but this did not verify all software models. To minimize the problem, three empirical cases were considered; we evaluated the software of the PWS, SMK, and PEPS. All models were based on UML, but only the state diagrams were used to draw behavioral diagrams. This affects test case generation and prioritization. For example, if a test suite is generated using a behavioral diagram such as a sequence diagram, this may affect the results even if the operation of both diagrams is identical. In conclusion, most search-based algorithms have the same problems, but the experimental results  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Module** & **Number of State Diagrams** & **Number of Test Cases** & **F** & \(p\)**-Value** \\ \hline Unit & 81 & 1387 & 4389.207 & \(<\)0.005 \\ Access & 13 & 1993 & 1631.736 & \(<\)0.005 \\ RKE & 11 & 1799 & 4267.017 & \(<\)0.005 \\ Start & 44 & 1385 & 3139.672 & \(<\)0.005 \\ Warning & 49 & 1755 & 4833.233 & \(<\)0.005 \\ \hline Total & 198 & 8319 & - & - \\ \hline \hline \end{tabular}
\end{table}
Table 7: ANOVA test results of passive entry start systems (PEPS) submodules.  
differ greatly if small amounts of data are compared, not always indicating the best algorithm. To deal with this issue, significance was analyzed via post-hoc comparisons of 100 experimental runs.  
Figure 6: Test case prioritization for the PEPS System.