# Model-Based Test Suite Generation Using Mutation Analysis for Fault Localization
## 4 Experimental Results
### Results  
Table 4 shows a summary of all of the experimental results. \(\tau_{R}\), \(\tau_{J}\), and \(\tau_{L}\) are the suspiciousness values of faulty statements calculated after performing 10 test suites generated using the random, Jaccard distance, and lookahead entropy methods, respectively. Suspiciousness represents the performance index of each method. A value closer to 1 indicates better performance. The suspiciousness was calculated using the tarantula suspiciousness metric [20]; avg is the average final suspiciousness value of the 100 experimental rounds, and max and min are the maximum and minimum values of the final suspiciousness values. In the table's difficulty column, W refers to weak, R to resistant, H to hard-to-kill, and N to non-mutant. The numbers inside the parentheses show the number of test cases that killed the mutants out of 2500 random test cases. In the mesh state chart, the Jaccard distance and lookahead entropy methods both achieved superior results to the random method. In the case of weak and resistant mutants, there were no significant differences between the performances of the Jaccard distance and lookahead entropy methods, but when the faulty statements were hard-to-kill mutants and non-mutants, the Jaccard distance method showed slightly better performance. In the ring state chart, the Jaccard distance method exhibited the best performance overall. In the case of hard-to-kill mutants and non-mutants, the lookahead entropy method achieved worse performance than random test suites. In the star state chart, the Jaccard distance method achieved the best performance. In the tree state chart, the Jaccard distance method exhibited the best performance, and the lookahead entropy method performed the worst. Table 5 shows the results of the performance comparison between the methods based on the mean suspiciousness value. The higher the number is, the better the performance of an indicator with a numerator was compared to an indicator with a denominator. The Jaccard distance method was 21% more efficient than the random test suites, and the lookahead entropy method was 10% more efficient than the random test suites. When it became more difficult to kill mutants, the efficiency of the lookahead entropy method decreased. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**State Chart**} & \multicolumn{2}{c}{**Difficulty**} & \multirow{2}{*}{\(\tau_{R}\)\(\tau_{J}\)\(\alpha_{S}\)} & \multirow{2}{*}{\(\tau_{L}\)\(\alpha_{S}\)} & \multirow{2}{*}{\(\tau_{R}\)\(\tau_{max}\)} & \multirow{2}{*}{\(\tau_{J}\)\(\tau_{max}\)} & \multirow{2}{*}{\(\tau_{L}\)\(\tau_{max}\)} & \multirow{2}{*}{\(\tau_{R}\)\(\tau_{  
Figures 7-10 show the changes in the suspiciousness of faulty statements when executing weak, resistant, hard-to-kill, and non-mutant test suites, respectively. Each time a test case generated by each method is executed, a trend in the suspiciousness of a faulty statement may be noted. This allows evaluation of how efficiently the test cases generated by each method increase the suspiciousness of the faulty statement. Figure 7 shows the results of the weak case in each state chart. In general, there were almost no performance differences between the Jaccard distance and lookahead entropy methods. In the random test suites, the suspiciousness was low during each execution. Figure 8 shows the results of the resistant cases for each state chart. The mesh and star state charts exhibited similar trends to the weak case; however, in the ring and tree state charts, the trend in the lookahead entropy was the same as that of the random method, in which the suspiciousness decreased as the test suites were executed. Figure 9 shows the changes in suspiciousness in each state chart for hard-to-kill cases. When the Jaccard distance was used, the suspiciousness tended to increase. However, when the lookahead entropy method was used, the suspiciousness did not increase in any state charts except for the mesh.