## 1 Introduction
Specifically, they seeded the LLM-generated code into G3P's initial population by mapping the seed to fit the predefined grammar. Their results show that they successfully improved some LLM-generated programs to fit a predefined BNF grammar. However, leveraging LLM-generated code only as seeds for the initial population causes the algorithm to lose the seeded information quickly during the evolution process. To restrict the search space and avoid programs that include bad coding practices/code smells, use unknown/untrustworthy libraries/functionalities, or do not follow a desired design pattern, we consider the definition of a \"safe\" grammar and ensure that any generated program obeys such grammar--ultimately mitigating security risks and improving code quality. While the definition of \"_safe_\" grammars is a challenge on its own that warrants several research studies (i.e., to design and prove them for different tasks), in our work, we are concerned with the capability to generate correct grammar-fitting programs. In our work, we make the following contributions:  
* We show that while our considered LLM (i.e., ChatGPT) successfully generates correct programs for most tasks in a well-known program synthesis benchmark, it struggles to generate programs that obey the predefined grammars. * We propose an evolutionary approach, i.e., Many-Objective Grammar-Guided Genetic Programming (MaOG3P), that exploits the LLM-generated program in two parts while guaranteeing that any evolved programs adhere to the predefined grammar:
* Leveraging many-objective similarity measures towards the LLM-generated code to guide the search process throughout the evolution. * Mapping the LLM-generated program to another program that obeys the predefined grammar while maintaining as much information as possible and using it as a seed in the initial population. Our MaOG3P algorithm considers four different similarity measures alongside the input-output error rate to strategically steer the search process towards more plausible program candidates and guide the evolutionary search towards solutions that exhibit structural similarities to the well-performing LLM-generated code. The idea behind using the similarity measures is that the LLM-generated code is likely to be close to the correct code. While MaOG3P is inspired by [20; 21], such approaches only evaluate the ability to leverage similarity measures towards the correct program (obtained using an oracle) to guide the evolutionary process--thus are not applicable in practice. Furthermore, as seeding is beneficial for evolutionary algorithms (both in single [39] and multi-objective [40] settings) by acting as a search accelerator/catalyser, there are various ways to source \"_good_\" quality seeds. For instance, [PERSON] et al. [41] provide programs that solve tasks different from the one at hand (which could have previously been generated by a human or another evolutionary process) to the initial population (in addition to programs generated randomly). In our work, we follow a similar seeding principle (i.e., adding a program to the initial population). However, our seeded program is generated by an LLM on the same task, following its grammar mapping (to align with the grammar in the grammar-guided genetic programming). Evaluating our approach on the general program synthesis benchmark suite 1 [42] demonstrates that MaOG3P effectively evolves the correct grammar-fitting programs for several tasks. Nevertheless, there are still improvements to be made as, in its default configuration, MaOG3P does not reach the success rate of the LLM when ignoring the grammar-fitting constraint. The remainder of the paper is organised as follows: Section 2 summarises the relevant background and related work. In Section 3, we introduce our novel MaOG3P approach. Section 4 outlines the details of our experimental setup. The results and discussions from our experiments are presented in Section 5. Finally, Section 6 concludes this work and explores avenues for future studies.