# The MADAG Strategy for Fault Location Techniques
## 4 Empirical Study
### Experimental Setups and Metrics  
In our experiment, the MADAG strategy starts from a failed test input with the maximum number of statements. The reason why we set the maximum number of statements of a failed test as a starting point is to reduce the possibility of missing the fault. To reduce the influence of the selection regarding the initial failed test on the validity of the experiment, [PERSON] et al. [10] and [PERSON] et al. [12] used many random inputs (20 or 30 tests) as initial failed tests and computed the average experimental results when using these initial failed tests. Although randomly selecting failed test cases can alleviate the sensitivity of the technique to the choice of the starting statements, faulty locations not covered by the statements of the chosen failed tests may be missed. Moreover, each faulty version may not have more than 20 failed tests in its subject programs. Thus, we chose an initial failed test with the maximum number of statements as the starting point. Further, we experimentally investigated the effectiveness of the MADAG strategy compared with the other test case prioritization mentioned in Section 2.1. Test case prioritization aim at prioritizing tests, reducing redundant test cases, and further narrowing the results to inspect the test outputs for developers. Simultaneously, the tests selected by the test case prioritization methods are applied to the SBFL technique to identify the fault compared with the application of all test cases to the SBFL technique, which achieves the approximate effectiveness of fault localization. A simple way to measure the effectiveness of test case prioritization methods is to observe how many test cases can be reduced. In other words, we will calculate the ratio of the selected test cases to the overall test cases, which is defined as follows:  
\[Reduction\ ratio=1-\frac{\text{selected test cases}}{\text{overall test cases}}\times 100 \tag{2}\]  
where a larger _reduction ratio_ indicates a better result. In addition to considering the reduction ratio, we use expense [9; 18; 29] as the metric to measure fault localization effectiveness. We apply the selected test cases to the SBFL technique to observe the effectiveness between the prioritized (or selected) and non-prioritized test cases. Given a ranked list produced by the SBFL technique, expense measures the mini  
Figure 5: An example program _Mid()_ with **(a)** MADAG tree and **(b)** suspiciousness metrics. mum percentage of statements in a subject program that must be examined in descending order of the ranks to locate the fault. The formula is defined as follows:  
\[Expense=\frac{\text{rank of the faulty statement}}{\text{number of executable statements}}\times 100\% \tag{3}\]  
where a smaller expense indicates a better result. We apply all test cases to the SBFL technique in order to calculate the expense, which we call the baseline diagnostic cost c. Test case prioritization and SBFL technique, then lead to a diagnostic cost \(c^{\prime}\) as same as in [PERSON] et al. (2012). Evaluating the effectiveness of the SBFL technique involves discriminating the degree of cost c and \(c^{\prime}\), so the technique achieves x% of the baseline cost as follows:  
\[x=\frac{c^{\prime}}{c}\times 100\% \tag{4}\]  
To compare each prioritization method mentioned in Section 2.1 fairly, the method can achieve 100% baseline effectiveness calculated by the number of selected tests as in [PERSON] et al. (2015). Moreover, in practice, it is difficult to control the cost to be exactly 100% of the baseline in our strategy. Thus, we evaluate the number of selected test cases when the baseline effectiveness is 100% or greater, and we do not select any more test cases. A major goal of the MADAG strategy is to minimize the number of test cases for which the SFBL technique must be used to calculate the suspicious statements and the test results must be manually inspected, but still retain the approximate effectiveness of the SBFL technique.