## 5 Results
### Performance of Our Proposed Approach (RQ3)  
We evaluated our proposed approach by comparing the performance of its components when taken separately. We particularly compare our approach against ChatGPT, G3P, G3P with a seeded grammar-mapped ChatGPT-generated code, and MaOG3P using ChatGPT-generated code as the target code for similarity calculations. In our experiments, each system underwent 30 runs on every problem. We report the number of successful runs in Table 6. A run was considered successful if it found at least one correct solution that passed all training and test cases. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline \begin{tabular}{c} **Benchmark** \\ **Problem** \\ \end{tabular}  & \multirow{2}{*}{**ChatGPT**} & \multirow{2}{*}{**G3P**} & \multicolumn{2}{c}{**G3P**} & \multirow{2}{*}{**MaOG3P**} & \multirow{2}{*}{**MaOG3P**} \\  & & & **ChatGPT** & & **No Seeding** & **ChatGPT** \\ \hline NumberO & ✓ & 17 & 30 & 20 & 30 \\ \hline Small Or Large & \(\  
Overall, our proposed approach outperformed all the compared approaches. Specifically, our MaOG3P system outperformed ChatGPT, the G3P system, and the MaOG3P system using ChatGPT-generated code as a target code for similarity calculation in terms of the number of solved tasks using the predefined grammar in the benchmark suite. Moreover, our approach significantly improved the success rate of solved problems. Compared to G3P with seeded grammar-mapped programs, our proposed MaOG3P algorithm demonstrated better success rates across all problems. While our approach may not surpass ChatGPT in generating correct code for unspecified grammars, it excels in refining LLM-generated programs to adhere to predefined grammars, having fixed a total of 14 programs that were violating the predefined grammar. However, it failed to address two unsolved problems with ChatGPT in its default configuration. By seeding a program resembling the solution code, our proposed approach doubled the number of problems solved compared to the G3P algorithm. This highlights the significant enhancement in the search process facilitated by the seeding approach. Furthermore, employing multi-objective optimisation with MaOG3P improved the success rate compared to G3P with seeding, demonstrating the efficacy of similarity measures in guiding the search process towards the correct solution. We measure if there is a significant improvement in performance by calculating the \(p\)-value of the Wilcoxon rank-sum test on the best test fitness value (error-rated fitness value for MaOG3P) of each run from G3P and the proposed approach comparison experiment. The results are shown in Table 7. We use 0.05 as the significance level and highlight the value in bold for significant improvements. Overall, our proposed approach significantly improved the performance regarding the best fitness value of each run for 15 problems. We significantly improved all problems that were seeded with \"completely mapped\" code, except for the problem \"Smallest\" as G3P performed well in solving this easy problem. For problems seeded with \"partially mapped\" code, we achieved significantly better fitness values for all problems. Interestingly, we achieved significantly better fitness values for the problems \"Even Squares\" and \"Digits\", even though we did not evolve the correct solution. We did not achieve significantly better fitness value for any of the problems seeded with \"poorly mapped\" programs. Therefore, a large effort needs to be made in the future to devise advanced and more effective grammar-mapping techniques.