HyGenar: An LLM-Driven Hybrid Genetic Algorithm for Few-Shot Grammar Generation
Authors: Weizhi Tang, Yixuan Li, Chris Sypherd, Elizabeth Polgreen, Vaishak Belle
Accepted at: acl25

HyGenar: An LLM-Driven Hybrid Genetic Algorithm for Few-Shot Grammar Generation

Weizhi Tang, Yixuan Li, Chris Sypherd, Elizabeth Polgreen, Vaishak Belle
University of Edinburgh
Weizhi.Tang@ed.ac.uk , yixuan.li.cs@ed.ac.uk , c.n.sypherd@sms.ed.ac.uk ,
elizabeth.polgreen@ed.ac.uk , vbelle@ed.ac.uk

Abstract

Grammar plays a critical role in natural language processing and text/code generation by enabling the definition of syntax, the creation of parsers, and guiding structured outputs. Although large language models (LLMs) demonstrate impressive capabilities across domains, their ability to infer and generate grammars has not yet been thoroughly explored. In this paper, we aim to study and improve the ability of LLMs for few-shot grammar generation, where grammars are inferred from sets of a small number of positive and negative examples and generated in Backus-Naur Form. To explore this, we introduced a novel dataset comprising 540 structured grammar generation challenges, devised 6 metrics, and evaluated 8 various LLMs against it. Our findings reveal that existing LLMs perform sub-optimally in grammar generation. To address this, we propose an LLM-driven hybrid genetic algorithm, namely HyGenar, to optimize grammar generation. HyGenar achieves substantial improvements in both the syntactic and semantic correctness of generated grammars across LLMs ¹.

1 Introduction

Grammar inference, also known as grammar induction, consists of inferring a grammar from a set of examples (Horning, 1969; De la Higuera, 2010; Stevenson and Cordy, 2014b; D’Ulizia et al., 2011). It has been studied and used in various fields, such as natural language processing, where it can reduce the effort required to generate syntactic or semantic models automatically (Kai et al., 2024; D’ulizia et al., 2011), and software engineering, where inferred grammars can guide reverse engineering and automated parser generation (Stevenson and Cordy, 2014a). By relying on characteristic examples (De la Higuera, 2010), grammar inference enables automated discovery of underlying syntactic or structural patterns.

Backus-Naur Form (BNF) is used to define the grammar of formal languages (Chomsky, 1956; Backus, 1959; Backus et al., 1960), typically Context-Free Grammars (CFGs) (Chomsky, 1956; Aho, 2007). It has also been used in various ways such as in parser generators like ANTLR4 (Parr, 2013) and Yacc (Johnson and Hill, 1978) or in constraining output structure of Large Language Models (LLMs) (Willard and Louf, 2023; Beurer-Kellner et al., 2024).

Although LLMs have exhibited remarkable capabilities across diverse domains (Gaur and Saunshi, 2023; Imani et al., 2023; Pan et al., 2023; Tang and Belle, 2024; Li et al., 2024; Jiang et al., 2024a), their capacity for grammar inference, and partic-

![A diagram showing a set of positive examples, a set of negative examples, and a robot generating a BNF grammar, with arrows indicating acceptance and rejection.](page_1012_320_384_384.png)

Figure 1: Given a small set of positive and negative examples, LLMs should infer and generate a grammar that accepts all positives and rejects all negatives.

¹The code is open-source and available at https://github.com/RutaTang/HyGenar.
ularly for generating grammars in BNF, has not yet been well explored. This paper focuses on investigating and improving the ability of LLMs to infer a CFG and generate it in BNF from a given set of positive and negative examples. A correctly generated grammar should accept all positives and reject all negatives. Typically, grammar inference requires a large set of characteristic examples that can uniquely identify a CFG (De la Higuera, 2010). However, we instead explore whether LLMs can infer a CFG from fewer examples without imposing any constraints on the examples, based on their experience and knowledge acquired through training on large-scale corpora. We refer to this process as “few-shot grammar generation”, emphasizing both the grammar inference of CFGs with fewer examples and their generation in BNF, and we use “grammar” to denote a CFG represented in BNF. An example is shown in Figure 1.

To explore the capacity of LLMs for few-shot grammar generation, we first construct a dedicated dataset of 540 challenges, each with only 3 positive and 3 negative examples, and use it to evaluate the performance of 8 LLMs, encompassing both open- and closed-source models of varying parameter sizes, including those specialized for code generation. We then propose an LLM-driven hybrid genetic algorithm, namely HyGenar, which adapts the principles of genetic algorithms with the integration of LLM-driven population initialization and mutation. We devised and adopted 6 metrics to comprehensively evaluate and analyze their performance from the perspectives of syntax and semantic correctness, over-fitting, over-generalization, and utility. The results show that, while most LLMs demonstrate unsatisfactory performance, our proposed algorithm significantly enhances the grammar generation ability across most of the evaluated LLMs.

We summarize the main contributions of this paper as follows:

1. We constructed a dedicated dataset of 540 challenges for few-shot grammar generation and comprehensively evaluated 8 LLMs;

2. We designed 6 metrics for measuring the ability of LLMs in this task and performed an extensive analysis based on them;

3. We proposed a novel method, HyGenar, to improve the grammar generation performance of LLMs and showed that it achieves significant improvements across LLMs.

2 Background

2.1 Context-Free Grammar

A context-free grammar (CFG) consists of terminals, non-terminals, a start symbol, and production rules (Hopcroft et al., 2001; Aho, 2007). It can be formally defined as a quadruple \( \mathcal{G} = (V, \Sigma, \Pi, S) \), where \( V \) is a finite set of non-terminal symbols, \( \Sigma \) is the set of terminals, \( \Sigma \cap V = \emptyset \), \( \Pi \) is a finite set of production rules, \( \Pi \subseteq V \times (V \cup \Sigma)^* \), and \( S \in V \) is the start symbol of the grammar. Elements of \( (V \cup \Sigma)^* \) are known as sentential forms.

The language generated by \( \mathcal{G} \), denoted \( \mathcal{L}(\mathcal{G}) \), comprises all strings derivable from \( S \) using the rules in \( \Pi \): \( \mathcal{L}(\mathcal{G}) = \{ \sigma \in \Sigma^* \mid S \Rightarrow^* \sigma \} \). For \( \alpha, \beta \in (V \cup \Sigma)^* \), we say \( \alpha \) directly derives \( \beta \) in one step as \( \alpha \Rightarrow \beta \), and define \( \alpha \Rightarrow^* \beta \) as \( \alpha \) deriving \( \beta \) in zero or more steps if there exists a finite sequence of \( \gamma_0, \ldots, \gamma_n \in (V \cup \Sigma)^* \) where \( n \geq 0 \), such that \( \alpha = \gamma_0 \Rightarrow \gamma_1 \Rightarrow \cdots \Rightarrow \gamma_n = \beta \).

2.2 Backus-Naur Form

Backus–Naur Form (BNF) is a notation used to define the grammar of formal languages, typically CFG (Chomsky, 1956; Backus, 1959; Backus et al., 1960). In this paper, LLM generates grammars in BNF.

A context-free grammar \( \mathcal{G} = (V, \Sigma, \Pi, S) \) is given in BNF notation as a list of grouped production rules, where each rule group is written:

\[
<v_i> ::= \alpha_1|\alpha_2|\ldots
\]

where \( v_i \) enclosed between the pair "< >" is a nonterminal symbol in \( V \), and \( \alpha_1, \alpha_2 \ldots \in (\Sigma \cup V)^* \) is the list of sentential forms that can be derived in one step from \( v_i \). This represents the set of all production rules with \( v_i \) on the left-hand side, i.e., \( v_i \rightarrow \alpha_1, v_i \rightarrow \alpha_2, \ldots \).

We extend the definition of a CFG to define a grammar in BNF to be \( G = (V, \Sigma, \Pi, S, R) \), where \( V, \Sigma, \Pi, \) and \( S \) are defined as before. \( R \) is a set of sets of production rules, denoted \( \{ r_1, \ldots, r_n \} \). Each \( r_i \in R \) is a set of production rules where the left-hand side of all rules is the \( i^{th} \) non-terminal symbol \( v_i \), i.e., \( r_i \) is \( (v_i \times (\Sigma \cup V)^*) \cap \Pi \). Since each non-terminal symbol has a corresponding rule set, \( n = |V| = |R| \). The language of the BNF grammar \( G \) is denoted by \( \mathcal{L}(G) \) and is equal to the language of the original CFG, \( \mathcal{L}(\mathcal{G}) \).
We say that a grammar \( G \) is a valid grammar, and that \( valid(G) \) evaluates to true, if it has a correct BNF syntax, and if \( R \) is as defined above, and if all nonterminal symbols have at least one rule in their corresponding rule set.

2.3 Grammar Inference

Grammar inference aims to learn grammar automatically from a set of examples (Horning, 1969; De la Higuera, 2010; Stevenson and Cordy, 2014b; D’Ulizia et al., 2011). In this paper, we focus on inferring a CFG, in BNF notation, from a small set of positive and negative examples.

Given a set of positive examples \( \mathcal{P} \) and negative examples \( \mathcal{N} \), consisting of strings that must be, respectively, accepted and rejected, the objective is to infer a BNF grammar \( G \). The generated \( G \) should satisfy \( \mathcal{P} \subseteq \mathcal{L}(G) \) and \( \mathcal{N} \cap \mathcal{L}(G) = \emptyset \), which ensures \( G \) accepts all positive examples and rejects all negative examples.

3 Related Work

3.1 Grammar Generation

Grammar inference has been widely studied and applied across various fields, such as natural language processing (Kai et al., 2024; D’Ulizia et al., 2011), bio-informatics (De la Higuera, 2010), pattern recognition (Pedro et al., 2013; Richetin and Vernadat, 1984; De la Higuera, 2010), and software engineering (Schröder and Cito, 2022; Stevenson and Cordy, 2014b). Previous works have also proposed various approaches for grammar inference (Rodrigues and Lopes, 2007; Cohen et al., 2017; Li et al., 2023; D’ulizia et al., 2011; Chen, 1995). However, few works are directly related to exploring the ability of LLMs for few-shot grammar generation, which, to reiterate, is to infer grammars from a small set of positives and negatives while generating them in BNF.

3.2 Code Generation

LLMs demonstrate the ability of code generation (Jiang et al., 2024b; Huang et al., 2023; Dehaerne et al., 2022), with various approaches proposed to improve it (Shinn et al., 2023; Madaan et al., 2023; Huang et al., 2023; Jiang et al., 2023b; Chen et al., 2023). We consider grammar generation to share notable similarities with code generation, since in grammar generation it is not only required to infer grammars but also to generate grammars in BNF. Thus, following a similar approach to Reflexion (Shinn et al., 2023) and Self-Refine (Madaan et al., 2023) to enhance code generation, we propose a method as one of the baselines for evaluation.

4 Grammar Generation Ability of LLMs

In this section, we describe the dedicated dataset constructed to evaluate the ability of LLMs in grammar generation, introduce 6 metrics we use in evaluation, explain experiments, and analyze results in detail. We detail each in the following subsections.

4.1 Dataset

To evaluate the capacity of LLMs for grammar generation, we present a dedicated dataset.

During dataset construction, for each \( k \in \{1, 2, \ldots, 9\} \), we prompted GPT-4o (OpenAI et al., 2024) to generate 10 distinct reference grammars, where each reference grammar \( G^{ref} \) has \( k \) nonterminal symbols and thus \( |R| = k \). This gives a total of 90 reference grammars. We used \( G^{ref} \) to prompt GPT-4o to produce 6 different challenges with each challenge consisting of a set of positives \( \mathcal{P} \) and negatives \( \mathcal{N} \) where \( |\mathcal{P}| = 3 \) and \( |\mathcal{N}| = 3 \), in a way that \( \mathcal{P} \subseteq \mathcal{L}(G^{ref}) \) and \( \mathcal{N} \cap \mathcal{L}(G^{ref}) = \emptyset \). However, GPT-4o often failed to produce challenges with valid reference grammars, positives, and negatives as \( k \) increased. We manually corrected erroneous reference grammars, positives, and negatives by using a BNF parser which takes a grammar and outputs whether a grammar is in valid BNF, and whether positives are accepted, and negatives are rejected\footnote{Refer to Appendix A for the details of dataset construction.}.

Following this process, we obtained a dataset of 540 challenges, each consisting of 3 positives and 3 negatives. Figure 1 shows an example challenge and a corresponding solution.

4.2 Metrics

Let \( C \) be a set of \( N \) challenges where each is a tuple \( (G^{ref}, \mathcal{P}, \mathcal{N}, G^*) \) consisting of a reference grammar, and a set of positive examples and negative examples, respectively, and the corresponding candidate grammar \( G^* \) generated by an LLM. We evaluate the quality of generated grammars using 6 key metrics:

Syntax Correctness (SX) The syntax correctness metric \( SX \) quantifies the proportion of guesses
that conform to the valid BNF syntax defined in Section 2.2. We define an indicator function as:

\[
\mathbb{I}_{SX}(G^*) = \begin{cases}
1 & \text{if } valid(G^*), \\
0 & \text{otherwise.}
\end{cases}
\]

\( SX(C) \) is defined as: \( \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}_{SX}(G_i^*) \).

**Semantic Correctness (SE)** \( SE \) captures the proportion of guesses that are semantically correct. We define an indicator function as follows, noting that if \( G^* \) is not in valid BNF, then \( \mathcal{L}(G^*) = \emptyset \):

\[
\mathbb{I}_{SE}(G^*, \mathcal{P}, \mathcal{N}) = \begin{cases}
1 & \text{if } \mathcal{P} \subseteq \mathcal{L}(G^*) \land \mathcal{N} \cap \mathcal{L}(G^*) = \emptyset \\
0 & \text{otherwise.}
\end{cases}
\]

\( SE(C) \) is given by: \( \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}_{SE}(G_i^*, \mathcal{P}_i, \mathcal{N}_i) \).

**Estimating Grammar Quality** Given a set of positive and negative examples, there are many possible valid and semantically correct solutions that are undesirable. For instance, the following grammar would be an undesirable solution for Figure 1. Here the \( \mathcal{L}(G^*) = |P| \), and is overfitted to the examples:

<stmt> ::= "add(1,2,3)" | "merge(x,y)" | "fibonacci(9)"

Equally undesirable is a grammar that over-generalizes from the examples, and defines a significantly larger language than the reference grammar, because this is highly likely to contain invalid strings. There are no standard metrics to measure over-fitting or over-generalization in grammar generation, so we devise 4 metrics based around the number of production rules used in parsing the positive examples.

First, let \( \Pi_{\mathcal{P}} \subseteq \Pi \) be the set of production rules that are used in the left-most derivations of all positive examples in \( \mathcal{P} \). That is, the set of rules in \( \Pi \) which occur in a sequence of rules \( S \rightarrow \alpha_1 \rightarrow \ldots \rightarrow \alpha_n \rightarrow p \) where \( p \in \mathcal{P} \), and all rules expand the left-most non-terminal in \( \alpha_1, \ldots, \alpha_n \).

We report metrics across only solved challenges, i.e., a challenge where \( G^* \) is syntactically and semantically correct. Our four metrics are defined as follows\(^3\):

• \( Diff(C) \), calculates the difference between the number of production rules in \( G^* \) used in parsing the positive examples and the number of production rules used by \( G^{ref} \) for a given challenge, i.e., \( |\Pi_{\mathcal{P}}^{ref}| - |\Pi_{\mathcal{P}}| \). A grammar that uses substantially fewer production rules has probably overfitted to the examples, and a grammar that uses substantially more production rules may have over-generalized. We report the average of \( Diff \) across all solved challenges as \( Diff^\circ \).
• \( OF \) estimates over-fitting by counting the percentage of solved challenges on which \( G^* \) uses fewer than half the number of production rules used by \( G^{ref} \), i.e., the number of times \( |\Pi_{\mathcal{P}}^{ref}| - |\Pi_{\mathcal{P}}| > \frac{|\Pi_{\mathcal{P}}^{ref}|}{2} \).
• \( OG \) estimates over-generalization by counting the percentage of solved challenges on which \( G^* \) uses more than half the number of production rules used by \( G^{ref} \), i.e., \( |\Pi_{\mathcal{P}}^{ref}| - |\Pi_{\mathcal{P}}| < -\frac{|\Pi_{\mathcal{P}}^{ref}|}{2} \).
• \( TU \) calculates the proportion of production rules that are used in parsing the positive examples for a given challenge, i.e., \( \frac{|\Pi_{\mathcal{P}}|}{|\Pi^{+}|} \), indicating the utility. We report the average of \( TU \) across all solved challenges as \( TU^\circ \).

**4.3 Baselines**

To establish baselines, we adopted two approaches, Direct Prompting (DP) and Optimization of the BNF Parser for LLM-Friendly Feedback (OPF).

In DP, we directly prompted LLMs with positive and negative examples, asking them to produce grammars that accept all positives and reject all negatives\(^4\).

In OPF, inspired by Reflexion (Shinn et al., 2023) and Self-Refine (Madaan et al., 2023), we further optimized the BNF parser by enabling it to provide more LLM-friendly error messages as feedback, aiming to improve the performance of grammar generation for LLMs\(^5\).

**4.4 Experiment Settings**

For a comprehensive evaluation, we selected a total of 8 LLMs, ensuring a diverse selection of both open- and closed-source LLMs, along with LLMs with varying parameter sizes and LLMs specifically designed for code generation.

Specifically, we selected two closed-source LLMs, *GPT-4o* (OpenAI et al., 2024) and *GPT-3.5-Turbo* (Brown et al., 2020a). For the other 6

\(^3\)Refer to Appendix B for the details of formal definitions.
\(^4\)Refer to Appendix C for the details of DP.
\(^5\)Refer to Appendix D for the details of OPF.
open-source LLMs, we note them in the notation: {model_name}:{parameter_size}-{model_type}.
We selected the following LLMs: Llama3:70b-Instruct (Grattafiori et al., 2024), Qwen:72b-Chat (Bai et al., 2023), Gemma2:27b-Instruct (Team et al., 2024), Mistral:7b-Instruct (Jiang et al., 2023a), Codestral:22b (MistralAI, 2024), and Starcoder2:15b-Instruct (Lozhkov et al., 2024). Among the selected LLMs, Starcoder2:15b-Instruct and Codestral:22b are LLMs for code generation.

For the DP baseline, we set the temperature to 0 and the maximum token to 2000.

For the OPF baseline6, we set the temperature to 0.3, the maximum token to 2000, and the maximum number of turns, max_turns, to 5.

4.5 Results & Analysis

The results of SX and SE for the 8 LLMs are presented in Table 1. We observed that GPT-4o, GPT-3.5-Turbo, and Gemma2:27b-Instruct achieve relatively high SX, while the other LLMs generally fare worse in DP baseline. Compared to the DP baseline, applying OPF leads to a significant enhancement in SX for both Mistral:7b-Instruct and Codestral:22b, yielding an 18% improvement for each of them, suggesting that parser feedback can help increase SX. Nevertheless, for most LLMs, OPF yields only slight gains in SX. Furthermore, it is worth noting that for Starcoder2:15b-Instruct, SX decreases by 16% after applying OPF. This indicates that feedback from the parser can sometimes lead to performance degradation if LLMs fail to interpret the feedback correctly.

Although several LLMs attain high SX, their SE remains low in DP, with an average of 39%. For example, GPT-3.5-Turbo has 94% SX but only 37% SE. With OPF, LLMs like Mistral:7b-Instruct and Codestral:22b achieve notable SE improvement with enhanced SX, illustrating that improved SX can positively influence SE. However, for most LLMs, OPF yields only marginal SE gains, particularly in the case that their SX is already high and OPF fails to contribute significant SX improvement.

Noticeably, although OPF helps improve SX, a significant gap still persists between SX and SE for most LLMs. For example, even after OPF, GPT-3.5-Turbo maintains a 57% gap between SX and SE, and Starcoder2:15b-Instruct has a 40% gap.

Since the ultimate objective is to improve SE, a more advanced approach is needed to address this limitation.

Furthermore, as other four metrics shown in Table 2, on average, for most of LLMs, the lower Diff°, OF, and OG indicate negligible over-fitting or over-generalization issues, and the higher TU° means LLMs are not predisposed to generate irrelevant production rules. Nevertheless, as the number of production rules increases, Diff° and OF exhibit slight increases, indicating a tendency toward mild over-fitting, while no significant over-generalization issues are observed 7.

In addition to the quantitative metrics, we also conducted a qualitative analysis by examining the generated grammars. We compared the grammars containing syntactic errors generated in the DP but corrected after the OPF. We primarily found four significant issues causing lower SX: unsupported symbols such as injecting quantifiers like “*” and “?” or character classes like “[a-z]”, erroneously introduced and misplaced brackets such as wrapping two terminals with round and square brackets, failing to wrap non-terminal with angle brackets, and omission the separators “|” between sentential forms. While the the first two issues are rampant across most LLMs, the third issue is mainly found in Mistral:7b-Instruct and the last one is sporadic. For most LLMs, OPF can occasionally mitigate these issues, but insignificantly. Furthermore, we also observed LLMs show an ability to recognize keywords in the examples, treating them as complete terminals rather than decomposing them into multiple terminals as individual characters. For example, they treat “if” and “SELECT” as complete terminals rather than splitting them into multiple characters. This may be benefited from the common sense acquired by LLMs from the corpus (Brown et al., 2020b).

6Refer to Appendix D for the reason of setting the temperature greater than 0 for OPF.

7Refer to Appendix F.3 and F.4 for more details.

5 LLM-Driven Hybrid Genetic Algorithm

The results of the baselines demonstrate the unsatisfactory performance of LLMs in grammar generation. To address this, we propose an LLM-driven hybrid genetic algorithm, namely HyGenar, a novel algorithm inspired by the concept of genetic algorithms. The following sections detail HyGenar and elaborate on our experiment settings, results, and analysis.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">Syntax Correctness (SX)</th>
    <th colspan="3">Semantics Correctness(SE)</th>
  </tr>
  <tr>
    <th>DP</th>
    <th>OPF</th>
    <th>HyGenar</th>
    <th>DP</th>
    <th>OPF</th>
    <th>HyGenar</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>93</td>
    <td><b>97</b> ↑<sub>4</sub></td>
    <td>96 ↑<sub>3</sub> ↓<sub>1</sub></td>
    <td>84</td>
    <td>85 ↑<sub>1</sub></td>
    <td><b>93</b> ↑<sub>9</sub> ↑<sub>8</sub></td>
  </tr>
  <tr>
    <td>GPT-3.5-Turbo</td>
    <td>94</td>
    <td>95 ↑<sub>1</sub></td>
    <td><b>99</b> ↑<sub>5</sub> ↑<sub>4</sub></td>
    <td>37</td>
    <td>38 ↑<sub>1</sub></td>
    <td><b>61</b> ↑<sub>24</sub> ↑<sub>23</sub></td>
  </tr>
  <tr>
    <td>Llama3:70b-Instruct</td>
    <td>57</td>
    <td>61 ↑<sub>4</sub></td>
    <td><b>75</b> ↑<sub>18</sub> ↑<sub>14</sub></td>
    <td>41</td>
    <td>42 ↑<sub>1</sub></td>
    <td><b>61</b> ↑<sub>20</sub> ↑<sub>19</sub></td>
  </tr>
  <tr>
    <td>Qwen:72b-Chat</td>
    <td>47</td>
    <td>49 ↑<sub>2</sub></td>
    <td><b>76</b> ↑<sub>29</sub> ↑<sub>27</sub></td>
    <td>20</td>
    <td>21 ↑<sub>1</sub></td>
    <td><b>38</b> ↑<sub>18</sub> ↑<sub>17</sub></td>
  </tr>
  <tr>
    <td>Mistral:7b-Instruct</td>
    <td>1</td>
    <td><b>19</b> ↑<sub>18</sub></td>
    <td>1 ↓<sub>18</sub></td>
    <td>0</td>
    <td><b>8</b> ↑<sub>8</sub></td>
    <td>1 ↑<sub>1</sub> ↓<sub>7</sub></td>
  </tr>
  <tr>
    <td>Gemma2:27b-Instruct</td>
    <td>91</td>
    <td>92 ↑<sub>1</sub></td>
    <td><b>98</b> ↑<sub>7</sub> ↑<sub>6</sub></td>
    <td>56</td>
    <td>57 ↑<sub>1</sub></td>
    <td><b>79</b> ↑<sub>23</sub> ↑<sub>22</sub></td>
  </tr>
  <tr>
    <td>Starcoder2:15b-Instruct</td>
    <td>76</td>
    <td>60 ↓<sub>16</sub></td>
    <td><b>98</b> ↑<sub>22</sub> ↑<sub>38</sub></td>
    <td>30</td>
    <td>20 ↓<sub>10</sub></td>
    <td><b>44</b> ↑<sub>14</sub> ↑<sub>24</sub></td>
  </tr>
  <tr>
    <td>Codestral:22b</td>
    <td>53</td>
    <td>71 ↑<sub>18</sub></td>
    <td><b>80</b> ↑<sub>27</sub> ↑<sub>9</sub></td>
    <td>44</td>
    <td>52 ↑<sub>8</sub></td>
    <td><b>67</b> ↑<sub>23</sub> ↑<sub>15</sub></td>
  </tr>
</table>

Table 1: The results of syntax and semantic correctness for LLMs grammar generation are presented as percentages (%). For each LLM, the best syntax and semantic correctness are highlighted with bold font. Blue arrows ↑↓ represent performance differences relative to the DP baseline, while red arrows ↑↓ indicate differences relative to the OPF baseline.

5.1 Methodology

HyGenar consists of four main components: Fitness, Selection, Crossover, and Mutation. It begins by prompting an LLM to generate an initial population of candidate grammars from positive and negative examples. In each generation, the Fitness function scores each candidate and the Select function chooses a subset of the population. To form a new population, the Cross function operates on two randomly selected candidates from this subset to generate a new candidate, which is then modified by the Mutate function and added to the new population, until the maximum population size is reached. The new population then advances to the next generation. We provide pseudocode in Algorithm 1 in Appendix E. We detail each component as follows:

Fitness Given a generated grammar \( G^* \), if it is syntactically incorrect, it is assigned a score of −1. For a valid grammar, we define two indicator functions:

\[
\mathbb{I}_A(G^*, p) = \begin{cases}
1 & \text{if } p \in \mathcal{L}(G^*), \\
0 & \text{otherwise.}
\end{cases}
\]

\[
\mathbb{I}_R(G^*, n) = \begin{cases}
1 & \text{if } n \notin \mathcal{L}(G^*), \\
0 & \text{otherwise.}
\end{cases}
\]

The Fitness function \( \text{Fitness}(G^*, \mathcal{P}, \mathcal{N}) \) is then defined as:

\[
\sum_{p_i \in \mathcal{P}} \mathbb{I}_A(G^*, p_i) + \sum_{n_i \in \mathcal{N}} \mathbb{I}_R(G^*, n_i).
\]

Selection Let \( \mathbb{G} = \{G_1^*, G_2^*, \ldots, G_k^*\} \) be a population of candidate grammars. Each \( G_i^* \in \mathbb{G} \) is assigned a fitness score \( s_i \in S \) by the Fitness function, where \( S = [s_1, s_2, \ldots, s_k] \) is a sequence of their corresponding scores. We define the Select function as:

\[
\text{Select}(\mathbb{G}, S) = \{G_{\sigma(1)}^*, G_{\sigma(2)}^*, \ldots, G_{\sigma(\frac{k}{2})}^*\},
\]

where \( \sigma : \{1, 2, \ldots, k\} \to \{1, 2, \ldots, k\} \) is a permutation of the indices such that: \( s_{\sigma(1)} \geq s_{\sigma(2)} \geq \cdots \geq s_{\sigma(k)} \). Hence, half of the candidates from the population are selected in decreasing order of their fitness scores.

Crossover The crossover function splices the production rules from two grammars together, at a randomly chosen splicing point. Let \( G_a^* \) and \( G_b^* \) be two candidate grammars, with rules \( R_a^* \) and \( R_b^* \) respectively. If both \( R \) are empty, we return one grammar at random. If one grammar has a non-empty \( R \), we return that grammar. Otherwise, we apply the crossover function with probability \( \rho \) (the crossover rate).

The crossover operation works in the following way. Let \( \ell = \min(|R_a^*|, |R_b^*|) \). First sample a crossover point \( w \sim \text{Uniform}(\{1, 2, \ldots, \ell\}) \). Let \( R_a^* = r_1^a, r_2^a, \ldots \) be the rules from \( G_a^* \) and \( R_b^* \) be the rules from \( G_b^* \). We take the first \( w - 1 \) rule sets from \( R_a^* \) and then prefix these to the last \( n - w \) rule sets from \( R_b^* \), where \( n = |R_b^*| \). This generates a new rule set \( R' = \{r_1^a, \ldots, r_{w-1}^a, r_w^b, \ldots, r_n^b\} \).

Crossover returns a new grammar \( G' = (V', \Sigma', \Pi', S_a, R') \), where \( V' \) and \( \Sigma' \) are all non-terminal and terminal symbols in \( R' \), \( \Pi' \) is all rules in \( R' \), and \( S_a \) is the start symbol from \( G_a^* \).

Mutation We use two mutation methods: mutation by LLM, and local mutation. We chose
whether to mutate a grammar at all, with probability \( \mu \) (the mutation rate). If the grammar has no production rules, we apply the LLM mutation. Otherwise, we apply local mutation with a probability 0.5 and LLM mutation otherwise.

LLM-Driven Mutation The LLM-driven mutation uses \( G^* \), \( \mathcal{P} \), and \( \mathcal{N} \) to prompt\(^8\) an LLM to produce a new grammar. In this approach, we expect that, with the knowledge and experience obtained by training on vast corpora, LLMs can heuristically provide more novel and dramatic modifications such as introducing new terminals or non-terminals, adding or removing production rules, or reshaping the structure of grammars, which is hard to approach with local mutation.

Local Mutation The local mutation is designed to produce incremental, targeted alterations to the grammar while preserving the majority of its original form. It is less flexible than LLM-driven mutation and it is unable to introduce new non-terminals or to drastically restructure grammars. However, it can not only potentially find a grammar candidate but also provide insights for LLM-driven mutation. Given a grammar \( G^* \), with a set of sets of production rules \( R^* \), local mutation comprises the following steps:

• **Rule set selection** We sample an integer \( i \sim \mathrm{Uniform}(\{1, \ldots, |R|\}) \). This index \( i \) chooses the rule set \( r_i \in R^* \) that will be mutated.

• **Shuffle** The Shuffle mutation shuffles the order of symbols on the right-hand side of a production rule. That is, each rule is a mapping from a non-terminal \( v \) to a sequence of non-terminal and terminal symbols \( (V \cup \Sigma)^* \), and we shuffle this sequence randomly. For example, the rule set:
\[
<e> ::= <e> "*" <e> | <e> "/" <e>
\]
with infix operators, may be shuffled to
\[
<e> ::= "*" <e> <e> | "/" <e> <e>,
\]
switching to prefix operators.

Shuffle is applied to all production rules in \( r_i \).

• **Space Insertion** The SpaceInsert mutation inserts a randomly chosen number of whitespace terminal symbols "\_." into the right-hand side of a production rule. Given a rule \( v_i \rightarrow \alpha \), Shuffle chooses the number of whitespace terminals to be inserted by sampling \( I \sim \mathrm{Uniform}(0, |\alpha|) \), where \( |\alpha| \) is the number of symbols in \( \alpha \). Each space is inserted before or after a randomly chosen symbol in \( \alpha \). As an example, the rule set:
\[
<e> ::= <noun> <verb>
\]
may be changed to
\[
<e> ::= <noun> "\_." <verb>.
\]
For each production rule in \( r_i \), we randomly decide whether SpaceInsert should be applied to that rule with a low probability\(^9\).

The Shuffle alteration heuristic is motivated by two key insights. First, shuffling the right-hand side of production rules may yield a grammar that accepts more positive examples and rejects more negative ones. Second, although Shuffle may rarely yield a better candidate for a complex target grammar, the new variant of the grammar produced from Shuffle is expected to provide alternative perspectives and new insights for LLMs to help generate subsequent grammars in future generations.

The SpaceInsert alteration was introduced because some LLMs tended to omit explicit space symbols between symbols in an alternative, resulting in degraded grammar generation, even if they were prompted to pay attention to space inclusion\(^{10}\). We expect that incorporating SpaceInsert will offer insights for LLMs of the explicit inclusion of spaces to thereby enhance performance.

5.2 Experiment Settings

In addition to a set of positive and negative examples and an LLM, HyGenar takes four parameters: *population size* (grammars per generation), *generations* (number of evolution iterations), *crossover rate* (probability of crossover), *mutation rate* (probability of mutation). In our experiments, we set these to 10, 5, 0.7, and 0.3, respectively.

We selected the same 8 LLMs as the DP and OPF baselines, setting maximum tokens to 2000 and temperature to 0.7. In HyGenar, a nonzero temperature is necessary for diversity. To ensure it does not significantly impact results and show its robustness, we further repeat the experiments 5 times with *GPT-3.5-Turbo* and *GPT-4o*.

5.3 Results & Analysis

As shown in Table 1, HyGenar substantially boosts SX for most LLMs. For example, *Qwen:72b-Instruct* gains 29% over DP and 27% over OPF.

\footnotetext{8Refer to Prompt Template 2 in Appendix E for the prompt we designed for LLM-driven mutation.}
\footnotetext{9We fix it to 0.1.}
\footnotetext{10Refer to Prompt Template 1 in Appendix C for details.}
<table>
  <tr>
    <th>Methods</th>
    <th>Diff°</th>
    <th>OF</th>
    <th>OG</th>
    <th>TU°</th>
  </tr>
  <tr>
    <td>DP</td>
    <td>1.12</td>
    <td>3.83</td>
    <td>0.63</td>
    <td>88.74</td>
  </tr>
  <tr>
    <td>OPF</td>
    <td>1.10</td>
    <td>4.72</td>
    <td>1.31</td>
    <td>90.76</td>
  </tr>
  <tr>
    <td>HyGenar</td>
    <td>1.19</td>
    <td>4.44</td>
    <td>0.92</td>
    <td>91.27</td>
  </tr>
</table>

Table 2: The averages of \( Diff^{\circ} \), \( OF(\%) \), \( OG(\%) \), and \( TU^{\circ}(\%) \) across all LLMs.

Even for LLMs already improved by OPF, such as Codestral:22b with an 18% improvement, HyGenar adds an additional 9%. Meanwhile, Starcoder2:15b-Instruct, which experiences a 16% drop under OPF, achieves a 22% improvement compared to DP and a 38% improvement over OPF, with HyGenar. On average, it improves \( SX \) 13.88% compared to DP and 9.88% over OPF.

While enhancing \( SX \) is essential, the ultimate objective is to improve \( SE \). As shown in Table 1, our method significantly boosts \( SE \) for all LLMs except Mistral:7b-Instruct. For example, with HyGenar, GPT-4o rises from 84% with DP and 85% with OPF to 93%, and GPT-3.5-Turbo, noted for low semantic accuracy, increases by 24% over DP and 23% over OPF. Notably, although the contribution from the enhancement of \( SX \) is essential to \( SE \), HyGenar does not rely solely on enhancing \( SX \) to achieve significant improvement of \( SE \), as five LLMs demonstrated higher \( SE \) increases than their \( SX \). Across the selected LLMs, the average \( SE \) improvement is 16.5% compared to DP and 15.13% compared to OPF.

We further analyzed the performance as the number of non-terminals and production rules increases. For non-terminals, we partition the dataset into 3 groups: \( C_1 \) (1–3 non-terminals), \( C_2 \) (4–6 non-terminals), and \( C_3 \) (7–9 non-terminals). For production rules, we split the dataset into another 3 groups: \( P_1 \) (1–6 production rules), \( P_2 \) (6–15 production rules), and \( P_3 \) (greater than 16 production rules). We observed that the performance of LLMs decreases as the number of non-terminals and production rules increases. Nevertheless, HyGenar still substantially improves both \( SX \) and \( SE^{11} \).

Furthermore, as shown in Table 2, \( OF \) does not significantly increase in HyGenar, indicating that the substantial improvements observed in both the \( SX \) and \( SE \) for HyGenar are not attributed to over-fitting\(^{12}\).

Moreover, we also conducted a qualitative analysis of how HyGenar improves \( SX \) and \( SE \). For \( SX \), HyGenar significantly reduces the issues of unsupported symbols injection and misplaced brackets. However, it fails to address the issue of unwrapped non-terminals, which is also the issue mainly happened in *Mistral:7b-Instruct*. Unlike OPF, which benefits from more explicit syntax error feedback, our approach lacks such direct syntax corrective guidance, meaning that if an LLM inherently struggles to generate syntactically correct grammars, our method may fail to produce valid candidates and process evolution, thereby lowering both \( SX \) and \( SE \). Nevertheless, as long as at least a few candidates are generated in correct syntax, HyGenar can optimize their generations during the evolutionary process to mitigate the aforementioned issues and improve \( SX \). For \( SE \), attributing the significant improvement is complex. However, we still observed two phenomena. First, after applying HyGenar, terminals that were not previously considered in the grammars generated by DP or OPF have been introduced. Second, the semantic errors that were caused by the absence of space terminals in DP or OPF have been alleviated.

Due to the relatively high temperature of 0.7 used for HyGenar, we repeated the experiments 5 times with *GPT-4o* and *GPT-3.5-Turbo*\(^{13}\) to ensure robustness. The averages of \( SX \) for *GPT-4o* and *GPT-3.5-Turbo* are 95.8% and 98.6%, with standard deviations 0.4% and 0.49% respectively, while the averages of \( SE \) are 93.2% and 61.6% with standard deviations 0.4% and 0.49% respectively. These results indicate that setting the temperature to 0.7 has a negligible impact on performance and show the robustness of HyGenar.

6 Conclusion

To explore the few-shot grammar generation ability of LLMs, we constructed a dedicated dataset consisting of 540 challenges, devised and adopted 6 metrics, and evaluated 8 various LLMs. Due to their unsatisfactory performance, we introduced HyGenar, an LLM-driven hybrid genetic algorithm for grammar generation. Our results indicate that HyGenar significantly enhances both syntax and semantic correctness compared to the two baselines. We believe this work provides valuable insights into LLM-based grammar generation and highlights the potential of LLM-driven hybrid genetic algorithms

11 Refer to Tables 4 and 5 in Appendix F.1 and F.2 for details.
12 Refer to Tables 6, 7, and 8 in Appendix F.3 for the more details.
13 Refer to Table 3 in Appendix F.5 for details.
in this domain.

7 Limitations

We discuss several limitations and concerns in this work, revealing potential challenges, constraints, and confusion.

First, although the results indicate that GPT-4o exhibits remarkable SX and SE, it is important to note that these results may be attributable to the use of GPT-4o during dataset construction. Nonetheless, even though GPT-4o already demonstrates excellent performance, HyGenar can still enhance it significantly.

Second, as demonstrated, our method does not outperform OPF for Mistral:7b-Instruct in SX and SE due to its inherent failure to generate syntactically correct grammars. Nevertheless, our approach yields significant SX and SE improvements for all other LLMs. We also propose to combine syntactical feedback and HyGenar to mitigate this limitation and further improve the performance.

Third, one may argue that given any finite set of positive and negative examples, it could always be possible to construct a regular grammar rather than a CFG that can accept all positives and reject all negatives. However, such an approach may function more like a classifier rather than a grammar and may lack applicability in subsequent tasks, such as constructing an abstract syntax tree.

Finally, in this work, we primarily focus on LLM-based few-shot grammar generation without comparing algorithms that are not LLM-based. The reasons behind this are that most algorithms require a large set of characteristic examples to uniquely determine the target grammar (De la Higuera, 2010). Instead, we do not impose such constraints on our example set and hypothesize that the experience and knowledge acquired from corpus can enable LLMs to handle few-shot grammar generation tasks. Consequently, those algorithms may not be directly applicable. In addition, since we focus on the exploration and improvement of the ability of LLMs in few-shot grammar generation, we construct two LLM-based baselines for fair comparison.

References

2007. Compilers: principles, techniques, tools, 2nd ed edition. Pearson/Addison Wesley, Boston.

John W Backus. 1959. The syntax and the semantics of the proposed international algebraic language of the zurich acm-gamm conference. In ICIP Proceedings, pages 125–132.

John W Backus, Friedrich L Bauer, Julien Green, Charles Katz, John McCarthy, Alan J Perlis, Heinz Rutishauser, Klaus Samelson, Bernard Vauquois, Joseph Henry Wegstein, et al. 1960. Report on the algorithmic language algol 60. Communications of the ACM, 3(5):299–311.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingen Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen technical report. Preprint, arXiv:2309.16609.

Luca Beurer-Kellner, Marc Fischer, and Martin T. Vechev. 2024. Guiding llms the right way: Fast, non-invasive constrained generation. ArXiv, abs/2403.06988.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020a. Language models are few-shot learners. Preprint, arXiv:2005.14165.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020b. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc.

Stanley F Chen. 1995. Bayesian grammar induction for language modeling. arXiv preprint cmp-lg/9504034.

Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. ArXiv, abs/2304.05128.
N. Chomsky. 1956. Three models for the description of language. IRE Transactions on Information Theory, 2(3):113–124.

Mor Cohen, Avi Caciularu, Idan Rejwan, and Jonathan Berant. 2017. Inducing regular grammars using recurrent neural networks. ArXiv, abs/1710.10453.

Colin De la Higuera. 2010. Grammatical inference: learning automata and grammars. Cambridge University Press.

Enrique Dehaerne, Bappaditya Dey, Sandip Halder, Stefan De Gendt, and Wannes Meert. 2022. Code generation using machine learning: A systematic review. IEEE Access, 10:82434–82455.

Arianna D’Ulizia, F. Ferri, and P. Grifoni. 2011. A survey of grammatical inference methods for natural language learning. Artificial Intelligence Review, 36:1–27.

Arianna D’Ulizia, Fernando Ferri, and Patrizia Grifoni. 2011. A survey of grammatical inference methods for natural language learning. Artificial Intelligence Review, 36(1):1–27.

Vedant Gaur and Nikunj Saunshi. 2023. Reasoning in large language models through symbolic math word problems. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5889–5903, Toronto, Canada. Association for Computational Linguistics.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Alionsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esioibu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Niall, Gregoire Milalon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Jun-teng Jia, Kalyan Vasudean Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kam-badur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiaod Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbally, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seo-hyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhsale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Colot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gouget, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyu Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xin-feng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmeen Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boeseneng, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bhamrane, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Chingshiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh,
Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcuate, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Gunas Lakshminarayanan, Hakan Inan, Hamid Shojanazari, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Illias Leoniadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gay, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reitzenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A. Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihr Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Prithish Urvraj, Qian Liang, Rachad Alal, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battye, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillion, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharad Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuvigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihaiulescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoqiu Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

John E Hopcroft, Rajeev Motwani, and Jeffrey D Ullman. 2001. Introduction to automata theory, languages, and computation. Acm Sigact News, 32(1):60–65.

James Jay Horning. 1969. A study of grammatical inference. Stanford University.

Dong Huang, Qingwen Bu, Jie M. Zhang, Michael Luck, and Heming Cui. 2023. Agentocoder: Multi-agent-based code generation with iterative testing and optimisation. ArXiv, abs/2312.13010.

Shima Imani, Liang Du, and Harsh Shrivastava. 2023. MathPrompter: Mathematical reasoning using large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 37–42, Toronto, Canada. Association for Computational Linguistics.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023a. Mistral 7b. Preprint, arXiv:2310.06825.

Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024a. A survey on large language models for code generation. Preprint, arXiv:2406.00515.

Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024b. A survey on large language models for code generation. ArXiv, abs/2406.00515.

Xue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. 2023b. Self-planning code generation with large language models. ACM Transactions on Software Engineering and Methodology.

S. C. Johnson and Murray Hill. 1978. Yacc: Yet another compiler-compiler.

Jushi Kai, Shengyuan Hou, Yusheng Huang, and Zhouhan Lin. 2024. Leveraging grammar induction for language understanding and generation. Preprint, arXiv:2410.04878.
Yixuan Li, Federico Mora, Elizabeth Polgreen, and Sanjit A Seshia. 2023. Genetic algorithms for searching a matrix of metagrammars for synthesis. arXiv preprint arXiv:2306.00521.

Yixuan Li, Julian Parsert, and Elizabeth Polgreen. 2024. Guiding enumerative program synthesis with large language models. In Computer Aided Verification. CAV 2024, pages 280–301, Cham. Springer Nature Switzerland.

Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtiar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wending Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhuo Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muenninghoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Cheng-hao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostafa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2024. Starcoder 2 and the stack v2: The next generation. Preprint, arXiv:2402.19173.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Processing Systems.

MistralAI. 2024. Codedstal. https://mistral.ai/news/codedstal/.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmid, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltuscu, Haiming Bao, Mohammad Bavarian, Jeff Belgium, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Justin Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonh, Hee-woo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Renichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Shahar Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillett, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael
Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Liangming Pan, Alon Albalaik, Xinyi Wang, and William Yang Wang. 2023. Logic-Im: Empowering large language models with symbolic solvers for faithful logical reasoning. (arXiv:2305.12295). ArXiv:2305.12295 [cs].

Terence Parr. 2013. The Definitive ANTLR 4 Reference, 2nd edition. Pragmatic Bookshelf.

Ricardo Wandré Dias Pedro, Fátima L. S. Nunes, and Ariane Machado-Lima. 2013. Using grammars for pattern recognition in images. ACM Computing Surveys (CSUR), 46:1 – 34.

M. Richetin and F. Vernadat. 1984. Efficient regular grammatical inference for pattern recognition. Pattern Recognit., 17:245–250.

Ernesto Rodrigues and Heitor Silvério Lopes. 2007. Genetic programming for induction of context-free grammars. Seventh International Conference on Intelligent Systems Design and Applications (ISDA 2007), pages 297–302.

Michael Schröder and Jürgen Cito. 2022. Grammars for free: toward grammar inference for ad hoc parsers. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, page 41–45, Pittsburgh Pennsylvania. ACM.

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems.

Andrew Stevenson and J. Cordy. 2014a. A survey of grammatical inference in software engineering. Sci. Comput. Program., 96:444–459.

Andrew Stevenson and James R. Cordy. 2014b. A survey of grammatical inference in software engineering. Science of Computer Programming, 96:444–459.

Weizhi Tang and Vaishak Belle. 2024. Tom-Im: Delegating theory of mind reasoning to external symbolic executors in large language models. arXiv preprint arXiv:2404.15515.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Husseinot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafiti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksiej Seyvryn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozinska, Dustin Herbrison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucińska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Letícia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidson, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofiz Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengcheng Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardesher Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotrutz, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenelly, Robert Dadashi, and Alek Andreev. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118.

Brandon T. Willard and Rémi Louf. 2023. Efficient guided generation for large language models. Preprint, arXiv:2307.09702.
A  Dataset Construction

To evaluate the capacity of LLMs in few-shot grammar generation, we present a dedicated dataset. We explain the details of the construction process in this section.

For clear explanation, let \( \mathcal{G}^{ref} = \bigcup_{k=1}^K \mathcal{G}_k^{ref} \) be a set of reference grammars, where \( \mathcal{G}_k^{ref} \) is a set of reference grammars in which each reference grammar \( G_k^{ref} \) having exactly \( k \) non-terminals and thus its \( |R| = k \). Each set of reference grammars \( \mathcal{G}_k^{ref} = \{ G_{k,1}^{ref}, G_{k,2}^{ref}, \ldots, G_{k,n}^{ref} \} \) contains \( n \) reference grammar with \( k \) non-terminals.

Initially, we constructed \( \mathcal{G}^{ref} = \bigcup_{k=1}^9 \mathcal{G}_k^{ref} \).
For each number \( k \), we prompted GPT-4o to produce \( n = 10 \) reference grammars to yield \( \mathcal{G}_k^{ref} = \{ G_{k,1}^{ref}, G_{k,2}^{ref}, \ldots, G_{k,10}^{ref} \} \), with the prompt template demonstrated in Prompt Template 3. In Prompt Template 3, \( k \) means the placeholder of the number of non-terminals, and \( n \) means the number of reference grammars needed to be produced. However, GPT-4o failed to consistently generate reference grammars in the correct syntax or with the correct number of non-terminals, especially as the number of non-terminals increased. To ensure that the generated reference grammars are syntactically correct and have the correct number of non-terminals, we used a BNF parser to do verification. It takes a \( G^{ref} \) and checks whether \( valid(G^{ref}) \) is true and whether it has the required number of non-terminals. Any reference grammar that is not valid or has a wrong number of non-terminals were manually corrected. For duplicated reference grammars, we prompted GPT-4o to generate an alternative. This resulted in 90 reference grammars (i.e., \( |\mathcal{G}^{ref}| = |\bigcup_{k=1}^9 \mathcal{G}_k^{ref}| = 90 \)), which are the reference grammars used to generate positive and negative examples subsequently.

For each reference grammar \( G_i^{ref} \in \mathcal{G}^{ref} \), we prompted GPT-4o to generate 6 various challenges. For each challenge, GPT-4o is prompted by Prompt Template 6 and Prompt Template 4 to produce a set of 3 positive examples (\( \mathcal{P} \subseteq \mathcal{L}(G^{ref}) \)), and a set of three negative examples (\( \mathcal{N} \cap \mathcal{L}(G^{ref}) = \emptyset \)), respectively. In both Prompt Template 6 and Prompt Template 4, \( m \) means the number of examples needed to be generated, and \( reference\_grammar \) means the given reference grammar by which the generated examples should be accepted or rejected. However, we observed that GPT-4o frequently failed to produce valid positive and negative examples, leading to either \( \mathcal{P} \not\subseteq \mathcal{L}(G^{ref}) \) or \( \mathcal{N} \cap \mathcal{L}(G^{ref}) \neq \emptyset \), or both. The number of failures tends to increase as the number of non-terminals of \( G^{ref} \) increases. To ensure the correctness of the generated challenges, we used a BNF parser to verify whether all given positive examples and negative examples can be accepted and rejected, respectively, by their corresponding \( G^{ref} \). Erroneous positive and negative examples were manually corrected. Ultimately, we obtained a dataset consisting of a total of 540 challenges.

We visually summarize the dataset construction procedure in Figure 2.

B  Grammar Quality Metrics

This section covers the formal definitions of the grammar quality metrics.

First define \( \Pi_\mathcal{P} \subseteq \Pi \) to be the set of production rules that are used in the left-most derivations of all positive examples in \( \mathcal{P} \). That is, the set of rules in \( \Pi \) which occur in a sequence of rules \( S \to \alpha_1 \to \ldots \to \alpha_n \to p \) where \( p \in \mathcal{P} \), and all rules expand the left-most non-terminal in \( \alpha_1, \ldots, \alpha_n \).

Let \( \Pi^* \) be the production rules in \( G^* \) and \( \Pi^{ref} \) be the production rules in \( G^{ref} \). Let us define \( Diff(G^{ref}, G^*) = |\Pi^{ref}_\mathcal{P}| - |\Pi_\mathcal{P}| \). The average difference in production rules used over the whole set of \( k \) solved challenges, \( C' \), is given by:

\[
Diff^\circ = \frac{1}{k} \sum_{i=1}^k Diff(G_i^{ref}, G_i^*).
\]

We define two indicator functions, which indicate when a grammar uses substantially fewer rules than the reference grammar, and substantially more rules than the reference grammar:

\[
\mathbb{I}_{OF}(G^{ref}, G^*, \mathcal{P}) =
\begin{cases}
1 & \text{if } |\Pi^{ref}_\mathcal{P}| - |\Pi_\mathcal{P}| > \frac{|\Pi^{ref}_\mathcal{P}|}{2} \\
0 & \text{otherwise}.
\end{cases}
\]

\[
\mathbb{I}_{OG}(G^{ref}, G^*, \mathcal{P}) =
\begin{cases}
1 & \text{if } |\Pi^{ref}_\mathcal{P}| - |\Pi_\mathcal{P}| < -\frac{|\Pi^{ref}_\mathcal{P}|}{2} \\
0 & \text{otherwise}.
\end{cases}
\]

The metric to estimate whether the generated grammars overfit the examples, is then given by

\[
OF(C') = \frac{1}{k} \sum_{i=1}^k \mathbb{I}_{OF}(G_i^{ref}, G_i^*, \mathcal{P}_i).
\]
Figure 2: The Dataset Construction Process: (1) GPT-4o is prompted with Prompt Template 3 to generate a set of reference grammars; (2) A set of reference grammars \( \mathcal{G}^{ref} = \bigcup_{i=1}^k \mathcal{G}_i^{ref} \) are generated by LLMs; (3) A BNF parser is used to check the correctness of each generated reference grammar; (4) Erroneous reference grammars are reported to humans; (5) Reported reference grammars are modified and corrected manually; (6) GPT-4o is prompted with Prompt Template 6 and Prompt Template 4 to generate challenges for each reference grammars; (7) Challenges are generated by LLMs, in which each challenge consists of 3 positive and 3 negative examples; (8) A BNF parser is used to verify whether positive and negative examples are accepted and rejected by their corresponding reference grammar respectively; (9) Erroneous challenges are reported to humans; (10) Reported challenges are corrected manually; (11) The final dataset consisting of 540 challenges are obtained.

The metric to estimate whether the generated grammars overgeneralize the examples, \( OG \), is given as:

\[
OG(C') = \frac{1}{k} \sum_{i=1}^k \mathbb{I}_{OG}(G_i^{ref}, G_i^*, \mathcal{P}_i).
\]

In addition, the \( TU \) metric, for a given challenge, to measure the percentage of \( |\Pi^*| \) taken up by \( |\Pi_\mathcal{P}| \), indicating the utility of \( G^* \), is given as:

\[
TU(G^*, \mathcal{P}) = \frac{|\Pi_\mathcal{P}|}{|\Pi^*|}
\]

for which lower \( TU \) indicates a bunch of irrelevant or nonsensical production rules of \( G^* \) while higher \( TU \) indicates the opposite. The average utility over \( C' \) is given by:

\[
TU^\circ = \frac{1}{k} \sum_{i=1}^k TU(G_i^*, \mathcal{P})
\]

C Direct Prompting

For the DP approach, Prompt Template 1 is used to prompt LLMs to directly generate a grammar with a given set of positive and negative examples. In Prompt Template 1, *positive_examples* and *negative_examples* are placeholders for a set of positive and negative examples. In addition, it also specifies a list of requirements LLMs should take care of and obey when generating grammars.

D Optimization of BNF Parser for Providing LLM-Friendly Feedback

For the OPF approach, we use the same prompt template used in DP, as shown in Prompt Template 1, to prompt LLMs to generate an initial grammar. Then, Prompt Template 5 is used in the iterations of the feedback loop to construct prompts from feedback offered by the BNF parser to LLMs. In Prompt Template 5, *positive_examples* and *negative_examples* are the placeholders for a set of positive and negative examples, *bnf_grammar* means the previously generated erroneous grammar, and *parser_feedback* is the placeholder for feedback provided from the BNF parser.

Furthermore, for each feedback given by the BNF parser, in addition to giving essential feedback such as notifying the line number for the place the error occurs, we optimize the BNF parser to also provide LLM-friendly feedback to LLMs, such as the possible reasons for the error or ways to fix it. We have shown some of them in Parser Feedback 1 and Parser Feedback 2.

In addition, OPF includes a parameter called *max_turns*, which specifies the maximum number of feedback iterations. If an LLM can generate a valid grammar based on earlier feedback, the algorithm stops early; otherwise, it continues until reaching the specified maximum.

Moreover, it is worth noting that this approach does not aim to follow every component of Reflexion or Self-Refine strictly. For instance, it does not
maintain a long-term context or external memory. Instead, it uses only the most recent feedback in each turn to guide self-refinement. Concretely, in each feedback iteration, an LLM is provided with the previously generated erroneous grammar, corresponding positive and negative examples, and the latest feedback, to produce revised grammars. Therefore, since, in each iteration, an LLM may produce similar and even the same grammar as the previous ones, especially when they fail to fix the previous errors leading to the same feedback provided by the parser, we set the temperature to 0.3 to expect to enable LLMs to generate more diverse grammars even for encountering the same feedback, to optimize the performance. This approach highlights the optimization from the perspective of the parser to provide more LLM-friendly feedback. However, due to limited space and the trivial-yet-complex optimization process, for the details of the optimization of the BNF parser, please refer to our source code and the comments.

E LLM-Driven Hybrid Genetic Algorithm

The pseudocode of HyGenar is presented in Algorithm 1, with detailed descriptions of its primary functions including Fitness, Select, Cross, and Mutate provided in Section 5. In Algorithm 1, we note Fitness function as FITNESS, Select function as SELECT, Cross function as CROSS, and Mutate function as MUTATE.

As shown in Algorithm 1, it takes seven parameters: \( \mathcal{P} \) and \( \mathcal{N} \) means a set of positive and negative examples respectively, \( k \) indicates population size, \( g \) represents generations which means the number of iterations of evolution, \( \rho \) means the crossover rate, \( \mu \) means the mutation rate, and \( LLM \) means an LLM which takes a prompt and returns a response. In addition, PROMPTGENERATOR means to generate a prompt from the prompt template shown in Prompt Template 1. MAXFITNESSSCORE is a constant indicating the highest fitness score any candidate grammar can achieve and due to each challenge in the constructed dataset only having 3 positive and 3 negative examples, the highest fitness score is 6. We thus set MAXFITNESSSCORE to 6.

In addition, for LLMMut, we have shown the prompt template used to prompt LLMs to mutate a given grammar in Prompt Template 2, in which bnf\_grammar means the placeholder for a candidate grammar while positive\_examples and negative\_examples means a set of positive and negative examples respectively.

F Additional Results

In addition to the results shown in Table 1, we have shown 7 more results and discussed them respectively in subsections F.1, F.2, F.3, F.4, and F.5.

F.1 Results for \( C_1, C_2, \) and \( C_3 \)

Table 4 presents the results categorized into three subsets: \( C_1, C_2, \) and \( C_3 \). The subset \( C_1 \) includes challenges where the reference grammars have \( 1 \sim 3 \) non-terminals, \( C_2 \) are those with \( 4 \sim 6 \) non-terminals, and \( C_3 \) consists of challenges with \( 7 \sim 9 \) non-terminals. Therefore, it aims to demonstrate and analyze the performance of LLMs as the number of non-terminals increases.

As the results are shown in Table 4, as the number of non-terminals increases, both \( SX \) and \( SE \) decrease across all LLMs. While DP and OPF exhibit suboptimal and unsatisfactory performance, HyGenar consistently demonstrates and contributes substantial improvements across most LLMs, even as the number of non-terminals increases. For example, in the case of GPT-4o, HyGenar increases the \( SE \) by 21% compared to DP and OPF on \( C_3 \). Similarly, with GPT-3.5-Turbo, compared to DP and OPF, HyGenar improves the \( SE \) by 30% and 28% on \( C_2 \) and 20% and 19% on \( C_3 \), respectively.

F.2 Results for \( P_1, P_2, \) and \( P_3 \)

Table 5 demonstrates the results grouped into three subsets: \( P_1, P_2, \) and \( P_3 \). The subset \( P_1 \) consists of challenges where the reference grammars have \( 1 \sim 6 \) production rules, \( P_2 \) includes those with \( 7 \sim 15 \) production rules, and \( P_3 \) consists of challenges with the number of production rules greater than 16. Therefore, it aims to show and analyze the performance of LLMs as the number of production rules increases.

Similar to the results of \( C_1, C_2, \) and \( C_3 \), as the number of production rules increases, both \( SX \) and \( SE \) decrease across all LLMs. Nevertheless, HyGenar can still steadily improve both \( SE \) and \( SX \), even as the number of production rules increases.

F.3 Results for Diff, OF, and OG Metrics

To investigate whether LLMs generate grammars in an overfitted manner and whether HyGenar improves performance through overfitting, as well as
Algorithm 1 HyGenar

procedure GENERATEGRAMMAR(\( \mathcal{P}, \mathcal{N}, k, g, \rho, \mu, LLM \))
    population \( \leftarrow [] \)  ▷ Initialize population as an empty list
    \( G^*.best \leftarrow \text{null} \)  ▷ Keep track of overall best grammar
    fitness\_best \( \leftarrow -1 \)  ▷ Track highest fitness found so far
    for \( i \leftarrow 1 \) to \( k \) do
        prompt \( \leftarrow \) PROMPTGENERATOR  ▷ Use prompt template from Prompt Template 1
        \( G^* \leftarrow LLM(prompt^t) \)  ▷ Use LLM to get an initial candidate
        score \( \leftarrow \) FITNESS\((G^*, \mathcal{P}, \mathcal{N})\)  ▷ Compute fitness
        if \( score = \text{MAXFITNESSSCORE} \) then
            return \( G^* \)  ▷ Return early if perfect score is achieved
        population \( \leftarrow \) population \( \parallel [G^*] \)  ▷ Add candidate to population
    for \( i \leftarrow 1 \) to \( g \) do
        fitnessScores \( \leftarrow [] \)
        for \( G^* \in \) population do
            score \( \leftarrow \) FITNESS\((G^*, \mathcal{P}, \mathcal{N})\)
            fitnessScores \( \leftarrow \) fitnessScores \( \parallel [(score, G^*)] \)  ▷ Add a tuple of score and grammar
            if \( score > \) fitness\_best then
                fitness\_best \( \leftarrow \) score
                \( G^*.best \leftarrow G^* \)
        if fitness\_best = MAXFITNESSSCORE then
            return \( G^*.best \)  ▷ Return the best grammar found
        \( \mathbb{G} = [G^* | (score, G^*) \in \text{fitnessScores}] \)
        \( S = [\text{score} | (score, G^*) \in \text{fitnessScores}] \)
        selected \( \leftarrow \) SELECT\((\mathbb{G}, S)\)
        population\_new \( \leftarrow [] \)
        while \( |\text{population\_new}| < k \) do
            \( G^*_a, G^*_b \leftarrow \) RANDOMCHOICE(selected)
            \( G^* \leftarrow \) CROSS\((G^*_a, G^*_b, \rho)\)
            if UNIFORM\((0, 1) < \mu \) then
                \( G^* \leftarrow \) MUTATE\((G^*, \mathcal{P}, \mathcal{N}, LLM)\)
            population\_new \( \leftarrow \) population\_new \( \parallel [G^*] \)
        population \( \leftarrow \) population\_new  ▷ Proceed to the next generation
    return \( G^*.best \)

to examine whether LLMs and our method produce overly generalized grammars, we employed the three evaluation metrics: \( Diff \), \( OF \), \( OG \). The results are presented in Table 6, 7, and 8, respectively. The notation “N/A” indicates inapplicability. Since these three metrics are only applicable when a grammar possesses correct semantics, “N/A” thus signifies that no grammar in the evaluation set exhibits correct semantics.

Through the \( Diff \) metric, as shown in Tabel 6, we observe that the number of production rules used in derivations of the generated grammars and the reference grammars does not differ significantly on average. However, as the number of production rules in the reference grammar increases, the \( Diff^\circ \) exhibits a slight upward trend. The \( Diff^\circ \) is almost always positive across most LLMs and methods, which indicates that, in most cases, the number of production rules used by the generated grammars is lower than that of the reference grammars.

Furthermore, Table 7 presents the \( OF \) metric. For some models, particularly *GPT-3.5-Turbo*, an increasing number of production rules corresponds to a certain degree of overfitting. Nevertheless, on average, most LLMs do not exhibit significant overfitting. Additionally, we observe that in HyGenar, the \( OF \) metric does not show a significant difference compared to the baselines, indicating that HyGenar does not improve performance through overfitting.
Additionally, we present the OG metric in Table 8. We observed that, for some models, such as Qwen:72b-Instruct, as the number of production rules in the reference grammar increases, they tend to generate overly generalized grammars. Nevertheless, on average, most LLMs do not tend to generate overgeneralized grammars.

F.4 Results for TU Metrics

To investigate whether LLMs generate irrelevant production rules, we employ the TU metric. For example, given the challenge shown in Figure 1, a generated grammar might be:

<stmt> ::= <func> "(" <args> ")"
<args> ::= <expr> | <expr> "," <args>
<expr> ::= <char> | <number>
<func> ::= <char> <func> | <char>
<char> ::= "a" | ... | "z"
<number> ::= "0" | ... | "9"
<hello> ::= "hello"
<world> ::= "world"

in which:

<hello> ::= "hello"
<world> ::= "world"

are two irrelevant production rules.

As shown in Table 9, on average, across both baselines and in HyGenar, TU remains relatively high, indicating that LLMs do not tend to produce irrelevant production rules. However, as the number of production rules increases, TU shows a tendency of declination. Nevertheless, this does not imply that LLMs generate more irrelevant rules. Considering the results from OG, we think this decrease may more likely be attributable to the generated grammar becoming more generalized.

F.5 Results for Robustness Evaluation

Since HyGenar requires setting the temperature greater than 0 which we set to 0.7, we repeated 5 independent experiments for both GPT-4o and GPT-3.5-Turbo to ensure the temperature does not affect the results significantly.

The results are demonstrated in Table 3, in which each row means the results of one experiment. The averages of syntax correctness of GPT-4o and GPT-3.5-Turbo are 95.8% and 98.6% and the standard deviations are 0.4% and 0.49%, respectively. The averages of semantic correctness of GPT-4o and GPT-3.5-Turbo are 93.2% and 61.6% and the standard deviations are 0.4% and 0.49%. Therefore, it indicates that although we set the temperature to 0.7 in HyGenar, the fluctuation of both syntax correctness and semantic correctness are very slight and the performance across multiple experiments stays steady. Thus, the results demonstrated the robustness of our proposed method, HyGenar.

Prompt Template 1: Generate a Grammar Directly with a Given Set Positive and Negative Examples

Given a set of positive and negative examples, generate the Backus–Naur Form (BNF) grammar that accepts all positive examples and rejects all negative examples.
1. Only generate the standard BNF grammar;
2. The generated BNF grammar MUST accept all positive examples and reject all negative examples;
3. Each terminal symbol MUST be quoted with double quotes and MUST NOT escape double quotes or pipeline in terminal symbols;
4. Pay special attention to whether spaces, line breaks, or other special symbols are required between each symbol, and if so, these need to be explicitly specified, e.g. <term> ::= "1" "+" "2" can handle "1+2" but not "1 + 2" while <term> ::= "1" " " "+" " " "2" can handle "1 + 2" but not "1+2";
5. The entry point of the generated BNF grammar MUST be the non-terminal symbol in the first production rule;
6. Only the generated BNF should be wrapped in a pair of triple backtick;
7. Do NOT output any additional texts, comments, or explanations.

===Positive Examples====
{positive_examples}
===Negative Examples====
{negative_examples}
<table>
  <tr>
    <th>Experiment</th>
    <th><i>SX</i></th>
    <th><i>SE</i></th>
  </tr>
  <tr>
    <th colspan="3">GPT-4o</th>
  </tr>
  <tr>
    <td>1st</td>
    <td>95</td>
    <td>93</td>
  </tr>
  <tr>
    <td>2nd</td>
    <td>96</td>
    <td>93</td>
  </tr>
  <tr>
    <td>3rd</td>
    <td>96</td>
    <td>94</td>
  </tr>
  <tr>
    <td>4th</td>
    <td>96</td>
    <td>93</td>
  </tr>
  <tr>
    <td>5th</td>
    <td>96</td>
    <td>93</td>
  </tr>
  <tr>
    <th colspan="3">GPT-3.5-Turbo</th>
  </tr>
  <tr>
    <td>1st</td>
    <td>98</td>
    <td>62</td>
  </tr>
  <tr>
    <td>2nd</td>
    <td>98</td>
    <td>62</td>
  </tr>
  <tr>
    <td>3rd</td>
    <td>99</td>
    <td>62</td>
  </tr>
  <tr>
    <td>4th</td>
    <td>99</td>
    <td>61</td>
  </tr>
  <tr>
    <td>5th</td>
    <td>99</td>
    <td>61</td>
  </tr>
</table>

Table 3: Results of Syntax and Semantic Correctness for HyGenar with <i>GPT-4o</i> and <i>GPT-3.5-Turbo</i> on Grammar Generation by Conducting 5 Independent Experiments (%)

Parser Feedback 1: Invalid Production Rule

This error is likely due to not satisfying one of the following requirements:
1. A rule MUST start with a non-terminal definition;
2. A non-terminal symbol MUST be in angle brackets, e.g. <non-terminal>;
3. A non-terminal definition must be followed by ' ::= ' to indicate the start of the right-hand side;

Prompt Template 2: LLM-Driven Mutation

Modify the following BNF grammar slightly to improve its acceptance of the positive examples and rejection of the negative examples.

===BNF Grammar===
{bnf_grammar}

===Positive Examples===
{positive_examples}
===Negative Examples===
{negative_examples}

Only output the modified BNF grammar wrapped in triple backticks.

Prompt Template 3: Generate Grammars

Generate a list of random standard Backus-Naur Form (BNF) grammar with the following constraints:
1. Each generated BNF grammar MUST be SELF-CONTAINED and VALID, which means it should be able to recognize a valid string;
2. Each generated BNF grammar MUST have exactly \( \{k\} \) lines;
3. Each generated BNF grammar MUST be unique;
4. Each generated BNF grammar MUST be separated by a newline in addition to the linebreak;
5. For each generated BNF grammar, the entry point MUST be at the first line;
6. Only generate \( \{n\} \) BNF grammars;
7. Only output BNF grammars WITHOUT any additional text or code block, like "```".

Prompt Template 4: Generate Negative Examples with a Given Grammar

Generate a list of negative examples with the following constraints:
1. Each example MUST be separated by a newline in addition to the linebreak;
2. Only output examples WITHOUT any additional text or code block, like "```";
3. Only output \( \{m\} \) examples;
4. Each example MUST be generated based on the given BNF grammar;
5. Each example should be greatly related to the given BNF grammar, but ensure it is NOT a valid string for the given BNF grammar.
For example, given the following BNF grammar:
<term> ::= "0" | "1" | "2"
you should output negative examples like:
6
*
9
Then, the given BNF grammar is:
{reference_grammar}
<table>
  <tr>
    <th>Challenge Set</th>
    <th><i>SX<sub>DP</sub></i></th>
    <th><i>SX<sub>OPF</sub></i></th>
    <th><i>SX<sub>HyGenar</sub></i></th>
    <th><i>SE<sub>DP</sub></i></th>
    <th><i>SE<sub>OPF</sub></i></th>
    <th><i>SE<sub>HyGenar</sub></i></th>
  </tr>
  <tr>
    <th colspan="7">GPT-4o</th>
  </tr>
  <tr>
    <td>\( C_1 \)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>99</td>
    <td>99</td>
    <td>100 \( \uparrow 1 \) \( \uparrow 1 \)</td>
  </tr>
  <tr>
    <td>\( C_2 \)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>93</td>
    <td>95 \( \uparrow 2 \)</td>
    <td>100 \( \uparrow 7 \) \( \uparrow 5 \)</td>
  </tr>
  <tr>
    <td>\( C_3 \)</td>
    <td>79</td>
    <td>92 \( \uparrow 13 \)</td>
    <td>87 \( \uparrow 8 \) \( \downarrow 5 \)</td>
    <td>59</td>
    <td>59</td>
    <td>80 \( \uparrow 21 \) \( \uparrow 21 \)</td>
  </tr>
  <tr>
    <td>All</td>
    <td>93</td>
    <td>97 \( \uparrow 4 \)</td>
    <td>96 \( \uparrow 3 \) \( \downarrow 1 \)</td>
    <td>84</td>
    <td>85 \( \uparrow 1 \)</td>
    <td>93 \( \uparrow 9 \) \( \uparrow 8 \)</td>
  </tr>
  <tr>
    <th colspan="7">GPT-3.5-Turbo</th>
  </tr>
  <tr>
    <td>\( C_1 \)</td>
    <td>98</td>
    <td>97 \( \downarrow 1 \)</td>
    <td>100 \( \uparrow 2 \) \( \uparrow 3 \)</td>
    <td>72</td>
    <td>71 \( \downarrow 1 \)</td>
    <td>93 \( \uparrow 21 \) \( \uparrow 22 \)</td>
  </tr>
  <tr>
    <td>\( C_2 \)</td>
    <td>98</td>
    <td>99 \( \uparrow 1 \)</td>
    <td>100 \( \uparrow 2 \) \( \uparrow 1 \)</td>
    <td>28</td>
    <td>30 \( \uparrow 2 \)</td>
    <td>58 \( \uparrow 30 \) \( \uparrow 28 \)</td>
  </tr>
  <tr>
    <td>\( C_3 \)</td>
    <td>84</td>
    <td>90 \( \uparrow 6 \)</td>
    <td>96 \( \uparrow 12 \) \( \uparrow 16 \)</td>
    <td>11</td>
    <td>12 \( \uparrow 1 \)</td>
    <td>31 \( \uparrow 20 \) \( \uparrow 19 \)</td>
  </tr>
  <tr>
    <td>All</td>
    <td>94</td>
    <td>95 \( \uparrow 1 \)</td>
    <td>99 \( \uparrow 5 \) \( \uparrow 4 \)</td>
    <td>37</td>
    <td>38 \( \uparrow 1 \)</td>
    <td>61 \( \uparrow 24 \) \( \uparrow 23 \)</td>
  </tr>
  <tr>
    <th colspan="7">Qwen:72b-Chat</th>
  </tr>
  <tr>
    <td>\( C_1 \)</td>
    <td>73</td>
    <td>76 \( \uparrow 3 \)</td>
    <td>96 \( \uparrow 23 \) \( \uparrow 20 \)</td>
    <td>52</td>
    <td>53 \( \uparrow 1 \)</td>
    <td>76 \( \uparrow 24 \) \( \uparrow 23 \)</td>
  </tr>
  <tr>
    <td>\( C_2 \)</td>
    <td>48</td>
    <td>48</td>
    <td>77 \( \uparrow 29 \) \( \uparrow 29 \)</td>
    <td>6</td>
    <td>8 \( \uparrow 2 \)</td>
    <td>26 \( \uparrow 20 \) \( \uparrow 18 \)</td>
  </tr>
  <tr>
    <td>\( C_3 \)</td>
    <td>20</td>
    <td>23 \( \uparrow 3 \)</td>
    <td>56 \( \uparrow 36 \) \( \uparrow 33 \)</td>
    <td>1</td>
    <td>2 \( \uparrow 1 \)</td>
    <td>11 \( \uparrow 10 \) \( \uparrow 9 \)</td>
  </tr>
  <tr>
    <td>All</td>
    <td>47</td>
    <td>49 \( \uparrow 2 \)</td>
    <td>76 \( \uparrow 29 \) \( \uparrow 27 \)</td>
    <td>20</td>
    <td>21 \( \uparrow 1 \)</td>
    <td>38 \( \uparrow 18 \) \( \uparrow 17 \)</td>
  </tr>
  <tr>
    <th colspan="7">Llama3:70b-Instruct</th>
  </tr>
  <tr>
    <td>\( C_1 \)</td>
    <td>88</td>
    <td>90 \( \uparrow 2 \)</td>
    <td>97 \( \uparrow 9 \) \( \uparrow 7 \)</td>
    <td>78</td>
    <td>77 \( \downarrow 1 \)</td>
    <td>94 \( \uparrow 16 \) \( \uparrow 17 \)</td>
  </tr>
  <tr>
    <td>\( C_2 \)</td>
    <td>54</td>
    <td>60 \( \uparrow 6 \)</td>
    <td>76 \( \uparrow 22 \) \( \uparrow 16 \)</td>
    <td>31</td>
    <td>35 \( \uparrow 4 \)</td>
    <td>61 \( \uparrow 30 \) \( \uparrow 26 \)</td>
  </tr>
  <tr>
    <td>\( C_3 \)</td>
    <td>28</td>
    <td>34 \( \uparrow 6 \)</td>
    <td>52 \( \uparrow 24 \) \( \uparrow 18 \)</td>
    <td>15</td>
    <td>14 \( \downarrow 1 \)</td>
    <td>29 \( \uparrow 14 \) \( \uparrow 15 \)</td>
  </tr>
  <tr>
    <td>All</td>
    <td>57</td>
    <td>61 \( \uparrow 4 \)</td>
    <td>75 \( \uparrow 18 \) \( \uparrow 14 \)</td>
    <td>41</td>
    <td>42 \( \uparrow 1 \)</td>
    <td>61 \( \uparrow 20 \) \( \uparrow 19 \)</td>
  </tr>
  <tr>
    <th colspan="7">Gemma2:27b-Instruct</th>
  </tr>
  <tr>
    <td>\( C_1 \)</td>
    <td>99</td>
    <td>100 \( \uparrow 1 \)</td>
    <td>100 \( \uparrow 1 \)</td>
    <td>91</td>
    <td>92 \( \uparrow 1 \)</td>
    <td>98 \( \uparrow 7 \) \( \uparrow 6 \)</td>
  </tr>
  <tr>
    <td>\( C_2 \)</td>
    <td>97</td>
    <td>97</td>
    <td>99 \( \uparrow 12 \) \( \uparrow 2 \)</td>
    <td>49</td>
    <td>49</td>
    <td>84 \( \uparrow 35 \) \( \uparrow 35 \)</td>
  </tr>
  <tr>
    <td>\( C_3 \)</td>
    <td>76</td>
    <td>79 \( \uparrow 3 \)</td>
    <td>93 \( \uparrow 17 \) \( \uparrow 14 \)</td>
    <td>26</td>
    <td>29 \( \uparrow 3 \)</td>
    <td>54 \( \uparrow 28 \) \( \uparrow 25 \)</td>
  </tr>
  <tr>
    <td>All</td>
    <td>91</td>
    <td>92 \( \uparrow 1 \)</td>
    <td>98 \( \uparrow 17 \) \( \uparrow 6 \)</td>
    <td>56</td>
    <td>57 \( \uparrow 1 \)</td>
    <td>79 \( \uparrow 23 \) \( \uparrow 22 \)</td>
  </tr>
  <tr>
    <th colspan="7">Mistral:7b-Instruct</th>
  </tr>
  <tr>
    <td>\( C_1 \)</td>
    <td>1</td>
    <td>25 \( \uparrow 24 \)</td>
    <td>3 \( \uparrow 2 \) \( \downarrow 22 \)</td>
    <td>0</td>
    <td>17 \( \uparrow 17 \)</td>
    <td>2 \( \uparrow 2 \) \( \downarrow 15 \)</td>
  </tr>
  <tr>
    <td>\( C_2 \)</td>
    <td>1</td>
    <td>20 \( \uparrow 19 \)</td>
    <td>1 \( \downarrow 19 \)</td>
    <td>1</td>
    <td>6 \( \uparrow 5 \)</td>
    <td>0 \( \downarrow 1 \) \( \downarrow 6 \)</td>
  </tr>
  <tr>
    <td>\( C_3 \)</td>
    <td>1</td>
    <td>11 \( \uparrow 10 \)</td>
    <td>0 \( \downarrow 1 \) \( \downarrow 11 \)</td>
    <td>0</td>
    <td>1 \( \uparrow 1 \)</td>
    <td>0 \( \downarrow 1 \)</td>
  </tr>
  <tr>
    <td>All</td>
    <td>1</td>
    <td>19 \( \uparrow 18 \)</td>
    <td>1 \( \downarrow 18 \)</td>
    <td>0</td>
    <td>8 \( \uparrow 8 \)</td>
    <td>1 \( \uparrow 1 \) \( \uparrow 7 \)</td>
  </tr>
  <tr>
    <th colspan="7">Codestar1:22b</th>
  </tr>
  <tr>
    <td>\( C_1 \)</td>
    <td>82</td>
    <td>96 \( \uparrow 14 \)</td>
    <td>99 \( \uparrow 17 \) \( \uparrow 3 \)</td>
    <td>82</td>
    <td>92 \( \uparrow 10 \)</td>
    <td>98 \( \uparrow 16 \) \( \uparrow 6 \)</td>
  </tr>
  <tr>
    <td>\( C_2 \)</td>
    <td>53</td>
    <td>77 \( \uparrow 24 \)</td>
    <td>86 \( \uparrow 33 \) \( \uparrow 9 \)</td>
    <td>36</td>
    <td>45 \( \uparrow 9 \)</td>
    <td>69 \( \uparrow 33 \) \( \uparrow 24 \)</td>
  </tr>
  <tr>
    <td>\( C_3 \)</td>
    <td>23</td>
    <td>39 \( \uparrow 16 \)</td>
    <td>57 \( \uparrow 34 \) \( \uparrow 18 \)</td>
    <td>15</td>
    <td>19 \( \uparrow 4 \)</td>
    <td>33 \( \uparrow 18 \) \( \uparrow 14 \)</td>
  </tr>
  <tr>
    <td>All</td>
    <td>53</td>
    <td>71 \( \uparrow 18 \)</td>
    <td>80 \( \uparrow 27 \) \( \uparrow 9 \)</td>
    <td>44</td>
    <td>52 \( \uparrow 8 \)</td>
    <td>67 \( \uparrow 23 \) \( \uparrow 15 \)</td>
  </tr>
  <tr>
    <th colspan="7">Starcoder2:15b-Instruct</th>
  </tr>
  <tr>
    <td>\( C_1 \)</td>
    <td>97</td>
    <td>68 \( \downarrow 29 \)</td>
    <td>100 \( \uparrow 3 \) \( \uparrow 32 \)</td>
    <td>67</td>
    <td>42 \( \downarrow 25 \)</td>
    <td>84 \( \uparrow 17 \) \( \uparrow 42 \)</td>
  </tr>
  <tr>
    <td>\( C_2 \)</td>
    <td>73</td>
    <td>65 \( \downarrow 8 \)</td>
    <td>99 \( \uparrow 26 \) \( \uparrow 34 \)</td>
    <td>14</td>
    <td>12 \( \downarrow 2 \)</td>
    <td>31 \( \uparrow 17 \) \( \uparrow 19 \)</td>
  </tr>
  <tr>
    <td>\( C_3 \)</td>
    <td>58</td>
    <td>48 \( \downarrow 10 \)</td>
    <td>94 \( \uparrow 36 \) \( \uparrow 46 \)</td>
    <td>11</td>
    <td>7 \( \downarrow 4 \)</td>
    <td>17 \( \uparrow 6 \) \( \uparrow 10 \)</td>
  </tr>
  <tr>
    <td>All</td>
    <td>76</td>
    <td>60 \( \downarrow 16 \)</td>
    <td>98 \( \uparrow 22 \) \( \uparrow 38 \)</td>
    <td>30</td>
    <td>20 \( \downarrow 10 \)</td>
    <td>44 \( \uparrow 14 \) \( \uparrow 24 \)</td>
  </tr>
</table>

Table 4: Averages of Syntax and Semantic Correctness Grouped in \( C_1 \), \( C_2 \), and \( C_3 \) (%)
<table>
  <tr>
    <th>Challenge Set</th>
    <th><i>SX<sub>DP</sub></i></th>
    <th><i>SX<sub>OPF</sub></i></th>
    <th><i>SX<sub>HyGenar</sub></i></th>
    <th><i>SE<sub>DP</sub></i></th>
    <th><i>SE<sub>OPF</sub></i></th>
    <th><i>SE<sub>HyGenar</sub></i></th>
  </tr>
  <tr>
    <th colspan="7">GPT-4o</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>99</td>
    <td>99</td>
    <td>100 \( \uparrow 1 \) \( \uparrow 1 \)</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>96</td>
    <td>95 \( \downarrow 1 \)</td>
    <td>100 \( \uparrow 4 \) \( \uparrow 5 \)</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>81</td>
    <td>93 \( \uparrow 12 \)</td>
    <td>89 \( \uparrow 8 \) \( \downarrow 4 \)</td>
    <td>62</td>
    <td>64 \( \uparrow 2 \)</td>
    <td>82 \( \uparrow 20 \) \( \uparrow 18 \)</td>
  </tr>
  <tr>
    <td>All</td>
    <td>93</td>
    <td>97 \( \uparrow 4 \)</td>
    <td>96 \( \uparrow 3 \) \( \downarrow 1 \)</td>
    <td>84</td>
    <td>85 \( \uparrow 1 \)</td>
    <td>93 \( \uparrow 9 \) \( \uparrow 8 \)</td>
  </tr>
  <tr>
    <th colspan="7">GPT-3.5-Turbo</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>98</td>
    <td>97 \( \downarrow 1 \)</td>
    <td>100 \( \uparrow 2 \) \( \uparrow 3 \)</td>
    <td>69</td>
    <td>67 \( \downarrow 2 \)</td>
    <td>93 \( \uparrow 24 \) \( \uparrow 26 \)</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>99</td>
    <td>100 \( \uparrow 1 \)</td>
    <td>100 \( \uparrow 1 \)</td>
    <td>18</td>
    <td>22 \( \uparrow 4 \)</td>
    <td>42 \( \uparrow 24 \) \( \uparrow 20 \)</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>86</td>
    <td>90 \( \uparrow 4 \)</td>
    <td>96 \( \uparrow 10 \) \( \uparrow 6 \)</td>
    <td>23</td>
    <td>24 \( \uparrow 1 \)</td>
    <td>46 \( \uparrow 23 \) \( \uparrow 22 \)</td>
  </tr>
  <tr>
    <td>All</td>
    <td>94</td>
    <td>95 \( \uparrow 1 \)</td>
    <td>99 \( \uparrow 5 \) \( \uparrow 4 \)</td>
    <td>37</td>
    <td>38 \( \uparrow 1 \)</td>
    <td>61 \( \uparrow 24 \) \( \uparrow 23 \)</td>
  </tr>
  <tr>
    <th colspan="7">Qwen:72b-Chat</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>72</td>
    <td>74 \( \uparrow 2 \)</td>
    <td>96 \( \uparrow 24 \) \( \uparrow 22 \)</td>
    <td>42</td>
    <td>43 \( \uparrow 1 \)</td>
    <td>69 \( \uparrow 27 \) \( \uparrow 26 \)</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>51</td>
    <td>51</td>
    <td>76 \( \uparrow 25 \) \( \uparrow 25 \)</td>
    <td>12</td>
    <td>13 \( \uparrow 1 \)</td>
    <td>24 \( \uparrow 12 \) \( \uparrow 11 \)</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>21</td>
    <td>25 \( \uparrow 4 \)</td>
    <td>59 \( \uparrow 38 \) \( \uparrow 34 \)</td>
    <td>6</td>
    <td>7 \( \uparrow 1 \)</td>
    <td>21 \( \uparrow 15 \) \( \uparrow 14 \)</td>
  </tr>
  <tr>
    <td>All</td>
    <td>47</td>
    <td>49 \( \uparrow 2 \)</td>
    <td>76 \( \uparrow 29 \) \( \uparrow 27 \)</td>
    <td>20</td>
    <td>21 \( \uparrow 1 \)</td>
    <td>38 \( \uparrow 18 \) \( \uparrow 17 \)</td>
  </tr>
  <tr>
    <th colspan="7">Llama3:70b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>86</td>
    <td>90 \( \uparrow 4 \)</td>
    <td>97 \( \uparrow 11 \) \( \uparrow 7 \)</td>
    <td>73</td>
    <td>73</td>
    <td>92 \( \uparrow 19 \) \( \uparrow 19 \)</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>63</td>
    <td>69 \( \uparrow 6 \)</td>
    <td>86 \( \uparrow 23 \) \( \uparrow 17 \)</td>
    <td>38</td>
    <td>40 \( \uparrow 2 \)</td>
    <td>67 \( \uparrow 29 \) \( \uparrow 27 \)</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>26</td>
    <td>30 \( \uparrow 4 \)</td>
    <td>47 \( \uparrow 21 \) \( \uparrow 17 \)</td>
    <td>15</td>
    <td>15</td>
    <td>29 \( \uparrow 14 \) \( \uparrow 14 \)</td>
  </tr>
  <tr>
    <td>All</td>
    <td>57</td>
    <td>61 \( \uparrow 4 \)</td>
    <td>75 \( \uparrow 18 \) \( \uparrow 14 \)</td>
    <td>41</td>
    <td>42 \( \uparrow 1 \)</td>
    <td>61 \( \uparrow 20 \) \( \uparrow 19 \)</td>
  </tr>
  <tr>
    <th colspan="7">Gemma2:27b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>99</td>
    <td>100 \( \uparrow 1 \)</td>
    <td>100 \( \uparrow 1 \)</td>
    <td>87</td>
    <td>88 \( \uparrow 1 \)</td>
    <td>98 \( \uparrow 11 \) \( \uparrow 10 \)</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>99</td>
    <td>99</td>
    <td>100 \( \uparrow 1 \) \( \uparrow 1 \)</td>
    <td>48</td>
    <td>47 \( \downarrow 1 \)</td>
    <td>84 \( \uparrow 36 \) \( \uparrow 37 \)</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>77</td>
    <td>80 \( \uparrow 3 \)</td>
    <td>94 \( \uparrow 17 \) \( \uparrow 14 \)</td>
    <td>33</td>
    <td>37 \( \uparrow 4 \)</td>
    <td>58 \( \uparrow 25 \) \( \uparrow 21 \)</td>
  </tr>
  <tr>
    <td>All</td>
    <td>91</td>
    <td>92 \( \uparrow 1 \)</td>
    <td>98 \( \uparrow 17 \) \( \uparrow 6 \)</td>
    <td>56</td>
    <td>57 \( \uparrow 1 \)</td>
    <td>79 \( \uparrow 23 \) \( \uparrow 22 \)</td>
  </tr>
  <tr>
    <th colspan="7">Mistral:7b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>2</td>
    <td>26 \( \uparrow 24 \)</td>
    <td>3 \( \uparrow 2 \) \( \uparrow 3 \)</td>
    <td>1</td>
    <td>15 \( \uparrow 14 \)</td>
    <td>2 \( \uparrow 1 \) \( \downarrow 13 \)</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>0</td>
    <td>19 \( \uparrow 19 \)</td>
    <td>1 \( \uparrow 1 \) \( \downarrow 18 \)</td>
    <td>0</td>
    <td>5 \( \uparrow 5 \)</td>
    <td>0 \( \downarrow 5 \)</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>0</td>
    <td>13 \( \uparrow 13 \)</td>
    <td>0 \( \downarrow 13 \)</td>
    <td>0</td>
    <td>3 \( \uparrow 3 \)</td>
    <td>0 \( \downarrow 3 \)</td>
  </tr>
  <tr>
    <td>All</td>
    <td>1</td>
    <td>19 \( \uparrow 18 \)</td>
    <td>1 \( \downarrow 18 \)</td>
    <td>0</td>
    <td>8 \( \uparrow 8 \)</td>
    <td>1 \( \uparrow 1 \) \( \downarrow 7 \)</td>
  </tr>
  <tr>
    <th colspan="7">Codestar:22b</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>82</td>
    <td>97 \( \uparrow 15 \)</td>
    <td>99 \( \uparrow 17 \) \( \uparrow 12 \)</td>
    <td>79</td>
    <td>89 \( \uparrow 10 \)</td>
    <td>97 \( \uparrow 18 \) \( \uparrow 8 \)</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>51</td>
    <td>73 \( \uparrow 22 \)</td>
    <td>80 \( \uparrow 29 \) \( \uparrow 17 \)</td>
    <td>35</td>
    <td>38 \( \uparrow 3 \)</td>
    <td>66 \( \uparrow 31 \) \( \uparrow 28 \)</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>29</td>
    <td>47 \( \uparrow 18 \)</td>
    <td>64 \( \uparrow 35 \) \( \uparrow 17 \)</td>
    <td>21</td>
    <td>30 \( \uparrow 9 \)</td>
    <td>41 \( \uparrow 20 \) \( \uparrow 11 \)</td>
  </tr>
  <tr>
    <td>All</td>
    <td>53</td>
    <td>71 \( \uparrow 18 \)</td>
    <td>80 \( \uparrow 27 \) \( \uparrow 9 \)</td>
    <td>44</td>
    <td>52 \( \uparrow 8 \)</td>
    <td>67 \( \uparrow 23 \) \( \uparrow 15 \)</td>
  </tr>
  <tr>
    <th colspan="7">Starcoder2:15b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>96</td>
    <td>68 \( \downarrow 28 \)</td>
    <td>100 \( \uparrow 4 \) \( \uparrow 32 \)</td>
    <td>54</td>
    <td>34 \( \downarrow 20 \)</td>
    <td>74 \( \uparrow 20 \) \( \uparrow 40 \)</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>65</td>
    <td>59 \( \downarrow 6 \)</td>
    <td>99 \( \uparrow 34 \) \( \uparrow 40 \)</td>
    <td>16</td>
    <td>10 \( \downarrow 6 \)</td>
    <td>26 \( \uparrow 10 \) \( \uparrow 16 \)</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>67</td>
    <td>55 \( \downarrow 12 \)</td>
    <td>95 \( \uparrow 28 \) \( \uparrow 40 \)</td>
    <td>20</td>
    <td>15 \( \downarrow 5 \)</td>
    <td>31 \( \uparrow 11 \) \( \uparrow 16 \)</td>
  </tr>
  <tr>
    <td>All</td>
    <td>76</td>
    <td>60 \( \downarrow 16 \)</td>
    <td>98 \( \uparrow 22 \) \( \uparrow 38 \)</td>
    <td>30</td>
    <td>20 \( \downarrow 10 \)</td>
    <td>44 \( \uparrow 14 \) \( \uparrow 24 \)</td>
  </tr>
</table>

Table 5: Averages of Syntax and Semantic Correctness Grouped in \( P_1, P_2, \) and \( P_3 \) (%)
<table>
  <tr>
    <th>Challenge Set</th>
    <th>\( Diff^{\circ}_{DP} \)</th>
    <th>\( Diff^{\circ}_{OPF} \)</th>
    <th>\( Diff^{\circ}_{HyGenar} \)</th>
  </tr>
  <tr>
    <th colspan="4">GPT-4o</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.22</td>
    <td>0.22</td>
    <td>0.21</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>1.38</td>
    <td>1.32</td>
    <td>1.18</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>4.37</td>
    <td>3.95</td>
    <td>3.36</td>
  </tr>
  <tr>
    <td>All</td>
    <td>1.76</td>
    <td>1.63</td>
    <td>1.65</td>
  </tr>
  <tr>
    <th colspan="4">GPT-3.5-Turbo</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.42</td>
    <td>0.37</td>
    <td>0.27</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>2.43</td>
    <td>2.34</td>
    <td>2.21</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>3.38</td>
    <td>3.42</td>
    <td>4.30</td>
  </tr>
  <tr>
    <td>All</td>
    <td>1.40</td>
    <td>1.43</td>
    <td>1.81</td>
  </tr>
  <tr>
    <th colspan="4">Qwen:72b-Chat</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.12</td>
    <td>0.08</td>
    <td>-0.02</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>0.32</td>
    <td>0.38</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>2.33</td>
    <td>3.00</td>
    <td>3.02</td>
  </tr>
  <tr>
    <td>All</td>
    <td>0.04</td>
    <td>0.50</td>
    <td>0.68</td>
  </tr>
  <tr>
    <th colspan="4">Llama3:70b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.39</td>
    <td>0.38</td>
    <td>0.44</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>1.56</td>
    <td>1.62</td>
    <td>1.50</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>2.52</td>
    <td>2.42</td>
    <td>2.85</td>
  </tr>
  <tr>
    <td>All</td>
    <td>1.00</td>
    <td>1.00</td>
    <td>1.21</td>
  </tr>
  <tr>
    <th colspan="4">Gemma2:27b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.55</td>
    <td>0.54</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>1.77</td>
    <td>1.74</td>
    <td>1.50</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>4.03</td>
    <td>4.27</td>
    <td>4.40</td>
  </tr>
  <tr>
    <td>All</td>
    <td>1.64</td>
    <td>1.74</td>
    <td>1.93</td>
  </tr>
  <tr>
    <th colspan="4">Mistral:7b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>1.00</td>
    <td>-0.59</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>N/A</td>
    <td>1.88</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>N/A</td>
    <td>3.17</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>All</td>
    <td>1.00</td>
    <td>0.44</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th colspan="4">Codestral:22b</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.18</td>
    <td>0.16</td>
    <td>0.10</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>1.26</td>
    <td>1.17</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>1.86</td>
    <td>2.34</td>
    <td>2.96</td>
  </tr>
  <tr>
    <td>All</td>
    <td>0.72</td>
    <td>0.85</td>
    <td>0.94</td>
  </tr>
  <tr>
    <th colspan="4">Starcoder2:15b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.39</td>
    <td>0.41</td>
    <td>0.20</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>0.40</td>
    <td>0.62</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>2.90</td>
    <td>3.13</td>
    <td>4.11</td>
  </tr>
  <tr>
    <td>All</td>
    <td>1.02</td>
    <td>1.22</td>
    <td>1.32</td>
  </tr>
</table>

Table 6: Averages of \( Diff^{\circ} \) Grouped in \( P_1 \), \( P_2 \), and \( P_3 \)
<table>
  <tr>
    <th>Challenge Set</th>
    <th><i>OF<sub>DP</sub></i></th>
    <th><i>OF<sub>OPF</sub></i></th>
    <th><i>OF<sub>HyGenar</sub></i></th>
  </tr>
  <tr>
    <th colspan="4">GPT-4o</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>4.03</td>
    <td>0.68</td>
    <td>1.28</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>13.49</td>
    <td>11.54</td>
    <td>11.31</td>
  </tr>
  <tr>
    <td>All</td>
    <td>5.08</td>
    <td>3.50</td>
    <td>4.17</td>
  </tr>
  <tr>
    <th colspan="4">GPT-3.5-Turbo</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>35.71</td>
    <td>34.29</td>
    <td>24.24</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>17.02</td>
    <td>16.67</td>
    <td>24.47</td>
  </tr>
  <tr>
    <td>All</td>
    <td>9.05</td>
    <td>9.85</td>
    <td>11.89</td>
  </tr>
  <tr>
    <th colspan="4">Qwen:72b-Chat</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>25.00</td>
    <td>21.43</td>
    <td>19.05</td>
  </tr>
  <tr>
    <td>All</td>
    <td>2.8</td>
    <td>2.65</td>
    <td>3.92</td>
  </tr>
  <tr>
    <th colspan="4">Llama3:70b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>1.67</td>
    <td>4.76</td>
    <td>0.95</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>6.45</td>
    <td>6.45</td>
    <td>10.17</td>
  </tr>
  <tr>
    <td>All</td>
    <td>1.35</td>
    <td>2.21</td>
    <td>2.12</td>
  </tr>
  <tr>
    <th colspan="4">Gemma2:27b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>9.33</td>
    <td>9.59</td>
    <td>6.11</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>16.18</td>
    <td>16.00</td>
    <td>14.41</td>
  </tr>
  <tr>
    <td>All</td>
    <td>6.00</td>
    <td>6.19</td>
    <td>5.88</td>
  </tr>
  <tr>
    <th colspan="4">Mistral:7b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>N/A</td>
    <td>12.05</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>N/A</td>
    <td>33.33</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>All</td>
    <td>0</td>
    <td>7.32</td>
    <td>0</td>
  </tr>
  <tr>
    <th colspan="4">Codestral:22b</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>4.76</td>
    <td>6.56</td>
    <td>10.84</td>
  </tr>
  <tr>
    <td>All</td>
    <td>0.84</td>
    <td>1.42</td>
    <td>2.49</td>
  </tr>
  <tr>
    <th colspan="4">Starcoder2:15b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>16.00</td>
    <td>6.25</td>
    <td>4.88</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>12.20</td>
    <td>12.90</td>
    <td>15.87</td>
  </tr>
  <tr>
    <td>All</td>
    <td>5.49</td>
    <td>4.63</td>
    <td>5.04</td>
  </tr>
</table>

Table 7: Averages of \( OF \) Grouped in \( P_1, P_2, \) and \( P_3 \) (%)
<table>
  <tr>
    <th>Challenge Set</th>
    <th>\( OG_{DP} \)</th>
    <th>\( OG_{OPF} \)</th>
    <th>\( OG_{HyGenar} \)</th>
  </tr>
  <tr>
    <th colspan="4">GPT-4o</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>1.11</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td>All</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.60</td>
  </tr>
  <tr>
    <th colspan="4">GPT-3.5-Turbo</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>1.19</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>2.13</td>
    <td>0.00</td>
    <td>1.06</td>
  </tr>
  <tr>
    <td>All</td>
    <td>0.50</td>
    <td>0.00</td>
    <td>0.91</td>
  </tr>
  <tr>
    <th colspan="4">Qwen:72b-Chat</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>1.32</td>
    <td>2.56</td>
    <td>3.20</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>2.70</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>8.33</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>All</td>
    <td>1.87</td>
    <td>1.77</td>
    <td>2.45</td>
  </tr>
  <tr>
    <th colspan="4">Llama3:70b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>All</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th colspan="4">Gemma2:27b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>0.00</td>
    <td>1.37</td>
    <td>1.53</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>All</td>
    <td>0.00</td>
    <td>0.33</td>
    <td>0.47</td>
  </tr>
  <tr>
    <th colspan="4">Mistral:7b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.00</td>
    <td>11.11</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>N/A</td>
    <td>0.00</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>N/A</td>
    <td>0.00</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>All</td>
    <td>0.00</td>
    <td>7.32</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th colspan="4">Codestral:22b</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.70</td>
    <td>1.25</td>
    <td>1.14</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>1.85</td>
    <td>1.67</td>
    <td>0.97</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>All</td>
    <td>0.84</td>
    <td>1.07</td>
    <td>0.83</td>
  </tr>
  <tr>
    <th colspan="4">Starcoder2:15b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>2.24</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>12.00</td>
    <td>0.00</td>
    <td>4.88</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>All</td>
    <td>1.83</td>
    <td>0.00</td>
    <td>2.10</td>
  </tr>
</table>

Table 8: Averages of \( OG \) Grouped in \( P_1, P_2, \) and \( P_3 \) (%)
<table>
  <tr>
    <th>Challenge Set</th>
    <th>\( TU_{DP}^{\circ} \)</th>
    <th>\( TU_{OPF}^{\circ} \)</th>
    <th>\( OG_{HyGenar}^{\circ} \)</th>
  </tr>
  <tr>
    <th colspan="4">GPT-4o</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>100</td>
    <td>100</td>
    <td>99.81</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>99.60</td>
    <td>99.55</td>
    <td>99.67</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>93.04</td>
    <td>92.80</td>
    <td>91.39</td>
  </tr>
  <tr>
    <td>All</td>
    <td>97.93</td>
    <td>97.81</td>
    <td>96.96</td>
  </tr>
  <tr>
    <th colspan="4">GPT-3.5-Turbo</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>99.64</td>
    <td>99.33</td>
    <td>97.93</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>90.91</td>
    <td>89.71</td>
    <td>89.11</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>78.56</td>
    <td>79.74</td>
    <td>77.57</td>
  </tr>
  <tr>
    <td>All</td>
    <td>93.43</td>
    <td>93.04</td>
    <td>90.32</td>
  </tr>
  <tr>
    <th colspan="4">Qwen:72b-Chat</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>92.95</td>
    <td>94.66</td>
    <td>90.47</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>76.56</td>
    <td>74.29</td>
    <td>81.31</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>79.79</td>
    <td>77.88</td>
    <td>77.53</td>
  </tr>
  <tr>
    <td>All</td>
    <td>88.56</td>
    <td>88.79</td>
    <td>86.14</td>
  </tr>
  <tr>
    <th colspan="4">Llama3:70b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>100</td>
    <td>100</td>
    <td>99.80</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>88.48</td>
    <td>89.06</td>
    <td>93.03</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>97.73</td>
    <td>98.24</td>
    <td>87.33</td>
  </tr>
  <tr>
    <td>All</td>
    <td>96.58</td>
    <td>96.71</td>
    <td>95.41</td>
  </tr>
  <tr>
    <th colspan="4">Gemma2:27b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>99.82</td>
    <td>99.82</td>
    <td>99.25</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>92.67</td>
    <td>94.21</td>
    <td>95.80</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>93.06</td>
    <td>94.02</td>
    <td>90.61</td>
  </tr>
  <tr>
    <td>All</td>
    <td>96.50</td>
    <td>97.07</td>
    <td>95.78</td>
  </tr>
  <tr>
    <th colspan="4">Mistral:7b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>50.00</td>
    <td>68.34</td>
    <td>83.33</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>N/A</td>
    <td>64.67</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>N/A</td>
    <td>64.17</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>All</td>
    <td>50.00</td>
    <td>67.01</td>
    <td>83.33</td>
  </tr>
  <tr>
    <th colspan="4">Codestral:22b</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>98.83</td>
    <td>98.11</td>
    <td>98.36</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>88.76</td>
    <td>90.31</td>
    <td>89.58</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>87.33</td>
    <td>85.27</td>
    <td>83.37</td>
  </tr>
  <tr>
    <td>All</td>
    <td>94.54</td>
    <td>93.66</td>
    <td>92.41</td>
  </tr>
  <tr>
    <th colspan="4">Starcoder2:15b-Instruct</th>
  </tr>
  <tr>
    <td>\( P_1 \)</td>
    <td>97.86</td>
    <td>96.44</td>
    <td>96.64</td>
  </tr>
  <tr>
    <td>\( P_2 \)</td>
    <td>83.29</td>
    <td>87.57</td>
    <td>81.92</td>
  </tr>
  <tr>
    <td>\( P_3 \)</td>
    <td>84.91</td>
    <td>85.39</td>
    <td>80.48</td>
  </tr>
  <tr>
    <td>All</td>
    <td>92.40</td>
    <td>91.96</td>
    <td>89.83</td>
  </tr>
</table>

Table 9: Averages of \( TU^{\circ} \) Grouped in \( P_1, P_2, \) and \( P_3 \) (%)
Parser Feedback 2: Lack of Alternatives
This error is likely due to the reason that the right-hand side is not defined after ' ::= '.

Prompt Template 5: Feedback Prompt in OPF
Given a set of positive and negative examples, generate the Backus–Naur Form (BNF) grammar that accepts all positive examples and rejects all negative examples.
1. Only generate the standard BNF grammar;
2. The generated BNF grammar MUST accept all positive examples and reject all negative examples;
3. Each terminal symbol MUST be quoted with double quotes and MUST NOT escape double quotes or pipeline in terminal symbols;
4. Pay special attention to whether spaces, line breaks, or other special symbols are required between each symbol, and if so, these need to be explicitly specified, e.g. <term> ::= "1" "+" "2" can handle "1+2" but not "1 + 2" while <term> ::= "1" " " "+" " " "2" can handle "1 + 2" but not "1+2";
5. The entry point of the generated BNF grammar MUST be the non-terminal symbol in the first production rule;
6. Only the generated BNF should be wrapped in a pair of triple backtick;
7. Do NOT output any additional texts, comments, or explanations.

===Positive Examples===
{positive_examples}
===Negative Examples===
{negative_examples}

===Generated BNF===
{bnf_grammar}

===Feedback===
The generated BNF grammar has incorrect syntax and please consider fixing it by referring to the feedback.
Here is the feedback from the BNF parser:
{parser_feedback}

Prompt Template 6: Generate Positive Examples with a Given Grammar
Generate a list of positive examples with the following constraints:
1. Each example MUST be separated by a newline in addition to the linebreak;
2. Only output examples WITHOUT any additional text or code block, like "~~~";
3. Only output {m} examples;
4. Each example MUST be generated based on the given BNF grammar;
5. Pay attention to whether the whitespaces are allowed between symbols.
For example, given the following BNF grammar:
<term> ::= "0" | "1" | "2"
you should output positive examples like:
0

1

2

Then, the given BNF grammar is:
{reference_grammar}

Metareview score: 3
Metareview: This paper explores the ability of LLMs to infer context-free grammars in Backus-Naur Form (BNF) from only a few positive and negative examples. The authors construct a dataset of 540 grammar induction task where each task involves learning a CFG that can accept the positive examples and reject the negative ones. Eight different LLMs are evaluated on this task. The authors compare various methods, of which the main proposal is HyGenar, based on a genetic algorithm.

The reviewers acknowledge the originality and the technical quality of the paper. No major concerns have been raised in this respect. The main controversy driving the high variance in scores is in the issue of whether the paper really fits the conference. One reviewer argues that, since the paper is focused on code generation and does not really use the kind of CFG used in NLP, it does not fit well in ACL. The authors have argued their case, citing previous programming language-oriented papers recently accepted at ACL, but the reviewer argues that those have a clearer relation to CL, in the sense that, even if the main focus is on programming languages, some lessons can be learned that can be applied in natural languages - which is much less clear here due to the focus on BNF grammars and learning from positive and negative examples.

While this is definitely a debatable issue and both sides have a point, I tend to lean towards the reviewer's position: although I wouldn't discard this paper outright, I believe that if the authors wish to publish it at an *ACL conference, they at least need to successfully convince the reader of how someone from the computational linguistics community can benefit from the insights here - which is not the case in the current version. I am recommending it for Findings as the paper itself is technically good, Findings does not have the space constraints of the main track so there is no big harm if the paper doesn't totally fit the conference theme, and NLP researchers that are also interested in compilers may be interested in it.
Summary of reasons to publish: The paper is original (both because of its focus, and because we don't often see techniques like genetic algorithms at *ACL venues), technically sound, substantial and well-written.