# Model-Based Test Suite Generation Using Mutation Analysis for Fault Localization
## 4 Experimental Results  
### Experimental Set-Up  
To confirm the effectiveness of the proposed method, tests were performed on four state charts. Figure 6 shows the four state charts used in this study. Star, ring, mesh, and tree state charts were constructed to perform experiments with a variety of forms. The star, ring, mesh, and tree forms were borrowed from different types of network organization and selected to confirm whether fault localization was affected by various organizational forms.  
The test-suite generation algorithm was used on the state charts shown in Figure 6. Faulty versions of the state charts (mutants), into which faults were inserted, were used to confirm the effectiveness of the fault detection and fault localization. State-chart information was converted into the extensible markup language metadata interchange (XMI), and a mutant was created by modifying the parsed XMI information. The mutation operator was constructed as in the work of [PERSON] et al. (2012). The mutants inserted into the faulty state charts were classified not according to their type but according to the difficulty of killing them. It is difficult to perform experiments on all faulty state-chart cases; therefore, we performed experiments on a limited number of cases. Although the mutation operators may be identical, mutant properties may vary by the guard conditions of different transitions and their priority levels. The sixth transition of the mesh state chart in Figure 6 shows an example of this. In general, transitions with a low number had high priority; therefore, transition 1 had a high priority and transition 6 had a relatively low priority. In state 1, if the guard conditions of transitions 1 and 6 were satisfied simultaneously, it passed through transition 1. Thus, the opportunity to go to transition 6 came only if transition 1's guard condition D \(\leq\) 35 \(\parallel\) E \(\leq\) 25 was false; therefore, transition 6's condition D \(\leq\) 35 always had to be false. Therefore, if a mutant such as D = 35 occurred at transition 6's guard, it became an equivalent mutant that could not be killed. For this reason, the experimental targets were selected according to the difficulty of killing mutants, rather than the mutation operators, to make performance comparisons.  
In this study, to measure the difficulty of killing a faulty statement (mutant), 2500 random test cases were performed on each faulty version of the program and the number of test cases that killed the fault was calculated. A fault was classified as \"hard-to-kill\" if it was killed by fewer than 600 test cases, \"resistant\" if it was killed by 600-1200 test cases, or \"weak\" if it was killed by more than 1200 test cases.  
The experiments were performed on faulty versions of the state charts, to which hard-to-kill, resistant, and weak mutants were introduced as faults. Furthermore, to verify whether the proposed method is valid for non-mutation faults, experiments were also performed for faults that were not generated by mutations. This experimental case can be used to verify that the proposed method is valid, even if the real fault is not a mutation fault.  
The experiments were repeated 100 times on each faulty version of the state chart, and 10 fault-localization test suites were generated during each experiment. To generate the test suites, three methods were used: random, Jaccard distance, and lookahead entropy. The random method used random TPs and random boundary test values each time a test suite was generated. The Jaccard distance and lookahead entropy methods (Section 3) were used to set the next TP and improve the test suite through the genetic algorithm. The genetic parameters used are shown in Table 3.  
Figure 6: State-chart case studies.