# Performance Analysis of Maximal Risk Evaluation Formulas for Spectrum-Based Fault Localization
## 3 A Condition with Which to Compare Risk Evaluation Formulas  
### Comparing Two Risk Evaluation Formulas  
In this section, we propose a new approach to compare between two risk evaluation formulas. Let us discuss the approach used to judge whether a risk evaluation formula is better than another in previous work. As mentioned previously, given a risk evaluation formula \(R\), the ranking list can be divided into three mutually exclusive subsets, \(S_{B}^{R}\), \(S_{F}^{R}\), and \(S_{A^{\prime}}^{R}\), in which statements are ranked with their suspiciousness. Empirical or theoretical comparison of two risk evaluation formulas \(R_{1}\) and \(R_{2}\) is \begin{table}
\begin{tabular}{c c c} \hline \hline
**Group** & **Risk Evaluation Formula** & **Expression** \\ \hline \multirow{5}{*}{\(ER_{1}^{{}^{\prime}}\)} & Naish1 (abbr. N1) & \(\begin{cases}-1&\text{if }e_{f}<F\\ n_{p}&\text{if }e_{f}=F\end{cases}\) \\ \cline{2-3}  & [PERSON] & \(e_{f}-\frac{e_{p}}{e_{p}+n_{p}+1}\) \\ \cline{2-3}  & GP13 & \(e_{f}(1+\frac{1}{2e_{p}+e_{f}})\) \\ \hline \multirow{5}{*}{\(ER_{5}\)} & [PERSON] (abbr. W1) & \(e_{f}\) \\ \cline{2-3}  & [PERSON] \& [PERSON] & \(\frac{e_{f}}{e_{f}+n_{f}+e_{f}+n_{p}}\) \\ \cline{2-3}  & [PERSON] & \(\begin{cases}0&\text{if }e_{f}<F\\ 1&\text{if }e_{f}=F\end{cases}\) \\ \hline GP02 & \(2(e_{f}+\sqrt{n_{p}})+\sqrt{e_{p}}\) \\ \hline GP03 & \(\sqrt{|e_{f}^{2}-\sqrt{e_{p}}|}\) \\ \hline GP19 & \(e_{f}\sqrt{|e_{p}-e_{f}+n_{f}-n_{p}|}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Maximal risk evaluation formula groups. then judged by the possible situations arising from six subsets: \(S_{B}^{R_{1}}\), \(S_{F}^{R_{1}}\), \(S_{A}^{R_{1}}\), \(S_{B}^{R_{2}}\), \(S_{F}^{R_{2}}\), and \(S_{A}^{R_{2}}\). For empirical study, researchers use the exact locations of the faulty statement appearing in the ranking lists of \(R_{1}\) and \(R_{2}\) to compare. So, the faulty statement will be in \(S_{F}^{R_{1}}\) and in \(S_{F}^{R_{2}}\). For \(R_{1}\) to be better than \(R_{2}\), the exact location of the faulty statement in \(R_{1}\)'s ranking list should be higher than that in \(R_{2}\). On the other hand, for theoretical comparison of two risk evaluation formulas \(R_{1}\) and \(R_{2}\), \(R_{1}\) is said to perform better than \(R_{2}\) if \"\(S_{B}^{R_{1}}\subseteq S_{B}^{R_{2}}\) and \(S_{A}^{R_{1}}\supseteq S_{A}^{R_{2}}\)[14]. Figure 2 depicts this situation. In fact, when \(S_{B}^{R_{1}}=S_{B}^{R_{2}}\) and \(S_{A}^{R_{1}}=S_{A}^{R_{2}}\), \(R_{1}\) and \(R_{2}\) have the same performance. In addition, readers may argue the fact that this may not be very accurate. For example, after the ranking, if the \"actual faulty statement\" is located at the end of \(S_{F}^{R_{1}}\) for \(R_{1}\) and is located in the front of \(S_{F}^{R_{2}}\) for \(R_{2}\), \(R_{2}\) can perform better than \(R_{1}\). The concept of consistent tie-breaking scheme is assumed in the theoretical study. This assumption of using consistent tie-breaking to compare the performance between two formulas seems reasonable without further information. Based on this idea, we propose a new condition for comparing two risk evaluation formulas \(R_{1}\) and \(R_{2}\). We first define our notations. For a risk evaluation formula \(R\), we use \(n_{B}^{R}\), \(n_{F}^{R}\) and \(n_{A}^{R}\) to denote the number of statements in \(S_{B}^{R}\), \(S_{F}^{R}\), and \(S_{A}^{R}\). Let \(n\) denote the total number of statements in a program. Please observe that \(n=n_{B}^{R}+n_{F}^{R}+n_{A}^{R}\). We now define the _expected faulty location_, \(EFL\), of a risk evaluation formula \(R\) by  
\[EFL(R)=n_{B}^{R}+\frac{1}{2}n_{F}^{R} \tag{1}\]  
where \(n_{B}^{R}\) and \(n_{F}^{R}\) are the numbers of statements in \(S_{B}^{R}\) and \(S_{F}^{R}\) respectively. This is, in fact, the _expected_ location of the faulty statement in the ranking list generated by the risk evaluation formula \(R\). Please observe that \(EFL(R)=n_{B}^{R}+\frac{1}{2}n_{F}^{R}=n_{B}^{R}+\frac{1}{2}(n-n_{B}^{R}-n_{A}^ {R})=\frac{1}{2}(n+n_{B}^{R}-n_{A}^{R})\).