# Performance Analysis of Maximal Risk Evaluation Formulas for Spectrum-Based Fault Localization
## 4 Empirical Study
### Experimental Results and Analysis
An interested reader may argue that the result of \"N1 \(\dashrightarrow\) GP03 \(\dashrightarrow\) GP02 \(\dashrightarrow\) W1 \(\dashrightarrow\) GP19\" is based on the percentage of the selected faulty versions per individual program, and we compare them by taking the average of these percentages. This may be unfair because, for tcas, there is only one selected faulty version. Only tcas v1 was selected as the faulty version since the scale of tcas subject program is too small, and most of the faulty versions are of the same type. Hence, a 100% of N1 \(\dashrightarrow\) W1 in tcas may actually add some advantages to N1 compared with other individual programs. So, one may then ask what the result would be if we used the total number of selected faulty versions to compare between these risk evaluation formulas. The answer to this question is, \"We would have the same result.\" We are going to discuss this in the rest of this paragraph. Table 5 shows, for each comparison pair \(R_{1}\) and \(R_{2}\), the total number of selected faulty versions that fall into the results of \"\(R_{1}\) better\", \"\(R_{2}\) better\", and \"\(R_{1}\) same as \(R_{2}\)\". For example, in the row of CP3 (the pair of N1 and GP03), there are nine selected faulty versions that fall into \"N1 better\", five in \"GP03 better\", and 34 in \"N1 same as GP03\". Hence, N1 performs more-frequently-better than GP03. From Table 5, we have the following observations:  
1. For CP1-CP4, N1 has a higher chance to perform better than W1, GP02, GP03, and GP19. 2. For CP8, CP6, and CP10, GP03 has a higher chance to perform better than GP02, W1, and GP19. 3. For CP5 and CP9, GP02 has a higher chance to perform better than W1 and GP19. 4. For CP7, W1 has a higher chance to perform better than GP19. As a result, we have the same observation. That is, N1 \(\dashrightarrow\) GP03 \(\dashrightarrow\) GP02 \(\dashrightarrow\) W1 \(\dashrightarrow\) GP19. Since N1 and W1 are representative formulas from their original maximal formula groups \(ER^{{}^{\prime}}_{1}\) and \(ER_{5}\) respectively, we can conclude further that any maximal formula in the \(ER^{{}^{\prime}}_{1}\) group performs more-frequently-better than GP03, which in turns performs more-frequently-better than any maximal formula in the \(ER_{5}\) group, which in turns performs more-frequently-better than GP19. That is, we have \(ER^{{}^{\prime}}_{1}\dashrightarrow\) GP03 \(\dashrightarrow\) GP02 \(\dashrightarrow\)\(ER_{5}\dashrightarrow\) GP19. Here, we extend the meaning of \"\(\dashrightarrow\)\" to compare between two maximal formula groups. In other words, if GPA and GPB are two maximal risk evaluation formula groups, GPA \(\dashrightarrow\) GPB means \(R_{a}\dashrightarrow R_{b}\) where \(R_{a}\) is a risk evaluation formula in GPA and \(R_{b}\) is in GPB. Please be remindeded that all risk evaluation formulas in the same maximal formula group have the same performance. As discussed in Section 3.2, when comparing between two risk evaluation formulas \(R_{1}\) and \(R_{2}\), there are five scenarios to characterize all possible cases of \(R_{1}\) performs better than \(R_{2}\). Table 6 summarises the scenarios covered by each subject program. For example, for the case of \"flex\", among totally 50 comparison results (10 comparison pairs \(\times\) 5 mutants of flex), there are 22 comparison results of \(R_{1}\dashrightarrow R_{2}\) obtained from Scenario \(B_{(<)}A_{(>)}\), six from Scenario \(B_{(<)}A_{(=)}\), five from Scenario \(B_{(<)}A_{(<)}\), six from Scenario \(B_{(=)}A_{(>)}\), two from Scenario \(B_{(>)}A_{(>)}\), and nine comparison results indicating \(R_{1}=R_{2}\). As shown in Table 6, 51.7% cases of \(R_{1}\dashrightarrow R_{2}\) fall in Scenario \(B_{(<)}A_{(>)}\), 3.8% in Scenario \(B_{(<)}A_{(=)}\), 11.5% in Scenario \(B_{(<)}A_{(<)}\), 13.9% in Scenario \(B_{(=)}A_{(>)}\), and 19.1% in Scenario \(B_{(>)}A_{(>)}\). According to the observation, Scenario \(B_{(<)}A_{(>)}\) covers the majority of cases that \(R_{1}\dashrightarrow R_{2}\).