# Performance Analysis of Maximal Risk Evaluation Formulas for Spectrum-Based Fault Localization
## 4 Empirical Study
### Experimental Results and Analysis  
Table 3 lists all the results of these 10 comparison pairs. For example, \"N1\" in the cell of row \"tcas v1\" and column \"CP1\" indicates that \"N1 performs better than W1\". Some cells in Table 3 have \"same\" meaning that the two risk formulas in the comparison pair have the same performance. Since the performance of maximal formulas in each comparison pair varies with different programs, Table 4 summarizes the results as the percentage of \"\(R_{1}\) which performs better than \(R_{2}\)\", \"\(R_{1}\) which performs the same as \(R_{2}\)\", and \"\(R_{2}\) which performs better than \(R_{1}\)\" for each comparison \begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Program** & **eLOC** & **Faulty Versions Selected for Empirical Study** & **(Number/Total)** & **Test Suite Size** & **Description** \\ \hline flex 1.1 & 9932 & m1, m2, m3, m4, m5 & \((5\,/\,5)\) & 670 & lexical scanner \\ grep 1.2 & 7306 & m1, m2, m3, m4, m5 & \((5\,/\,5)\) & 806 & pattern match \\ sed 2.0 & 9205 & m1, m2, m3, m4, m5 & \((5\,/\,7)\) & 441 & stream editor \\ \hline Space 2.0 & 5902 & v14, v15, v18, v20, v23, v26, v33 & \((7\,/\,38)\) & 13,550 & ADL interpreter \\ \hline print\_tokens 2.0 & 342 & v5, v7 & \((2\,/\,7)\) & 4130 & lexical analyzer \\ print\_tokens 2.0 & 355 & v4, v56, v7, v8, v9, v10 & \((7\,/\,10)\) & 4115 & lexical analyzer \\ replace 2.1 & 512 & v1, v15, v30 & \((3\,/\,32)\) & 5542 & pattern match \\ schedule 2.0 & 292 & v3, v4 & \((2\,/\,9)\) & 2650 & priority scheduler \\ schedule2.0 & 262 & v6, v7, v10 & \((3\,/\,10)\) & 2710 & priority scheduler \\ tcas 2.0 & 135 & v1 & \((1\,/\,41)\) & 1608 & altitude separation \\ tot\_info 2.0 & 273 & v5, v7, v8, v15, v16, v17, v20, v23 & \((8\,/\,23)\) & 1052 & information measure \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Subject programs and test suite. pair and each individual subject program. For example, in the row of \"CP1\" and the column of flex, out of the five mutant programs, there are four mutants for which N1 performs better than W1 and there is only one mutant for which W1 performs better than N1. Hence, we can conclude that, among these five mutants of flex, N1 performs better than W1 more often. For ease of reference, we use the term _more-frequently-better_ to reflect this situation. In other words, we conclude that N1 performs more-frequently-better than W1. Formally, for two risk evaluation formulas \(R_{1}\) and \(R_{2}\), we say that \(R_{1}\) performs more-frequently-better than \(R_{2}\), denoted as \(R_{1}\dashrightarrow R_{2}\), if, among all selected faulty versions, the number of times that \"\(R_{1}\) performs better than \(R_{2}\)\" is higher than that of \"\(R_{2}\) performs better than \(R_{1}\)\". \begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline  
For CP5, we observe that GP02 performs more-frequently-better than W1 for all programs except grep, sed and schedule2. The percentage values of GP02 \(\dashrightarrow\) W1 range from 0% to 100% with an average of 66.8% whereas those of W1 \(\dashrightarrow\) GP02 range from 0% to 100% with an average of 31.4%. Hence, we can conclude that GP02 performs more-frequently-better than W1. Similarly, for CP9, we can also conclude that GP02 performs more-frequently-better than GP19. 4. W1 has a higher chance to perform better than GP19. 1. For CP7, W1 performs more-frequently-better than GP19 for all programs except flex, grep, sed and schedule2. In fact, for flex and grep, the chances for W1 \(\dashrightarrow\) GP19 and that of GP19 \(\dashrightarrow\) W1 are the same; both are 40% for flex and 40% for grep. The percentage values of W1 \(\dashrightarrow\) GP19 range from 33.3% to 100% with an average of 66.1% whereas those of GP19 \(\dashrightarrow\) W1 range from 0% to 66.7% with an average of 27.8%. Hence, we can conclude that W1 performs more-frequently-better than GP19. In summary, it is very likely to observe that W1 \(\dashrightarrow\) GP03 \(\dashrightarrow\) GP02 \(\dashrightarrow\) W1 \(\dashrightarrow\) GP19.