# Efficient Regression Testing of Software Product Lines by Reducing Redundant Test Executions
## 5 Evaluation
### Results and Discussion
#### 5.3.3 RQ3  
How Effective Is CPLTE in Reducing the Cost of SPL Regression Testing?  
To answer RQ3, we compared CPLTE and the exhaustive execution with regard to the end-to-end time. In this comparison, only the Ekstazi_SPL selection is used for the \(A\) phase because the full  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Subject SPL**} & \multirow{2}{*}{**\#f**} & \multicolumn{3}{c}{**Number of Faults Detected**} \\ \cline{3-5}  & & \
\begin{tabular}{c} **Single-Test-** \\ **Single-Execution** \\ \end{tabular} &
\begin{tabular}{c} **Exhaustive** \\ **Execution** \\ \end{tabular} & **CPLTE** \\ \hline MobileMedia (4) & 301 & 187 & 213 & 213 \\ \hline TankWar (5) & 250 & 122 & 154 & 154 \\ \hline Prevayler (3) & 329 & 207 & 210 & 210 \\ Prevayler (5) & & 204 & 211 & 211 \\ \hline MobileRSSReader (3) & 406 & 281 & 287 & 287 \\ MobileRSSReader (5) & 406 & 282 & 287 & 287 \\ \hline Lampiro (4) & 498 & 255 & 264 & 264 \\ Lampiro (6) & 498 & 252 & 266 & 266 \\ \hline BerkeleyDB (3) & & 282 & 329 & 329 \\ BerkeleyDB (5) & 607 & 280 & 329 & 329 \\ BerkeleyDB (7) & & 284 & 334 & 334 \\ BerkeleyDB (10) & & 284 & 334 & 334 \\ \hline \hline \end{tabular} \#f: number of faults injected.  
\end{table}
Table 5: Number of faults detected by the single-test-single-execution, the exhaustive execution and CPLTE.  
selection is a naive approach. Table 6 presents the average end-to-end time (in seconds) resulting from CPLTE and the exhaustive execution (columns 2 and 3) and the average percentage of the reduced end-to-end time (Savings in column 4). From Table 6, the following observations can be made.  
First, CPLTE effectively reduces the time to perform SPL regression testing. For all the subject product families, CPLTE reduced the end-to-end time ranging from 4.3% to 69.0%. This indicates that CPLTE effectively reduces the time to execute test cases and the time to collect their execution traces compared to the exhaustive execution of test cases. This reduction of the end-to-end time is proportional to the amount of reduction of test executions because the cost of executing test cases and the cost of collecting their execution traces are the most significant part of the overall regression testing cost.  
Second, the overhead of CPLTE is small. One source of overhead in CPLTE is the \(F\)C phase (i.e., Algorithm 1 in Section 4.4). However, the overhead from the \(F\)C phase is negligible. In the case of BerkeleyDB (10), which is the largest system used in our experiment, it took only 0.11 s to perform the \(F\)C phase. Another source of overhead in CPLTE is the selective test executions (i.e., Algorithm 2 in Section 4.6). However, this overhead is insignificant, too. As Table 4; Table 6 show, the savings in terms of the end-to-end time are 2.3-9.2% lower than savings in terms of the number of test executions. Because the overhead from the \(F\)C phase is negligible and the Ekstazi_SPL selection was commonly used for the \(A\) phase in both comparison test execution methods, this result indicates that the selective test execution of CPLTE requires 2.3-9.2% more of the time to perform all the phases of regression testing. Consequently, CPLTE incurs only a small amount of overhead in identifying equivalent test execution and conducting the selective test execution and so we expect CPLTE to work well even with larger SPL systems.