# An Automatic Software Defect Repair Method Based on Multi-Objective Genetic Programming
## 4 Experimental Evaluation
### Experimental Results and Analysis  
To experimentally evaluate the MGPRepair method, this experiment selected six automated defect repair methods: jGenprog [29], RSRepair [6], ARJA [7], Nopol [30], Capgen [31], and SequenceR [32]. Among them, jGenprog is the Java version of the classic repair method Genprog, while RSRepair, Capgen, and ARJA are among the most advanced search-based repair methods. Nopol is one of the most advanced methods based on semantic constraints. SequenceR is one of the most popular methods based on machine learning. The experiment was conducted on 395 defects in the Defects4J defect dataset, reproducing these methods. The number of successfully repaired defects and the time used for each successful repair were recorded. Then, ablation experiments were performed to demonstrate the improvements brought by the optimization methods used in MGPRepair to genetic search-based repair methods. The specific details are as follows:  
1. Research Question 1: Compare the improvement in the success rate of defect repair achieved by the MGPRepair method compared to other excellent automated defect repair methods. This question has selected six representative defect repair methods to compare with MGPRepair, and these methods have been reproduced. The experimental data is shown in Table 4 below. Table 4 presents the repair results of MGPRepair and the other six repair methods on Defects4J. It details the number of defects repaired by each method in each project, as well as the total number and proportion of defects repaired across all six projects. A total of 395 defects were selected from six of the most popular projects, including Chart, Closure, Lang, Math, Time, and Mockito. MGPRepair generated reasonable and credible patches for 8, 3, 10, 27, 1, and 2 defects respectively. In total, 51 credible patches were generated, accounting for 12.91% of all defects, which is 24, 16, 26, 30, 7, and 19 more defects repaired than jGenProg, Nopol, Capgen, SequenceR, ARJA, and RSRepair respectively. MGPRepair achieved the highest number and proportion of defects repaired among these methods, especially in the Chart, Lang, and Math projects where it successfully repaired the largest \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Project** & **Project Name** & \begin{tabular}{c} **Number of** \\ **Defects** \\ \end{tabular} & \
\begin{tabular}{c} **Number of Lines** \\ **of Code** \\ \end{tabular} &
\begin{tabular}{c} **Number of** \\ **Test Cases** \\ \end{tabular} \\ \hline Chart & JFreeChart & 26 & 96 & 2205 \\ Closure & Closure Compiler & 133 & 90 & 7927 \\ Lang & Commons Lang & 65 & 22 & 2245 \\ Math & Commons Math & 106 & 85 & 3602 \\ Mockito & Mockito & 38 & 11 & 1457 \\ Time & Joda-Time & 27 & 28 & 4130 \\ \hline Total & - & 395 & 332 & 21,566 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Defects4J Dataset. number of defects. In the Time and Mockito projects, the number of successfully repaired defects was on par with the highest number repaired by the other six methods. Only in the Closure project did MGPRepair repair 2 fewer defects than the SequenceR method. In summary, it can be initially concluded that MGPRepair outperforms the other six methods in terms of defect repair success rate. This is mainly due to the fact that search-based methods are currently one of the most mature methods in the field of defect repair. The approach presented in this chapter filters and classifies repair materials based on genetic programming search. This can generate a higher-quality initial population and effectively reduce the number of iterations of the algorithm. The approach also optimizes the process of selecting offspring variants using the NSGA-II algorithm for multi-objective genetic evolution. Therefore, MGPRepair achieves a higher success rate in repair patch generation than the other six methods. However, generating a credible patch that passes the test cases does not equate to generating a correct patch.