SynCode: LLM Generation with Grammar Augmentation
Authors: Shubham Ugare
Tarun Suresh
Hangoo Kang
Sasa Misailovic
Gagandeep Singh
Venue: Accepted by TMLR ()

Abstract: LLMs are widely used in complex AI applications. These applications underscore the need for LLM outputs to adhere to a specific format, for their integration with other components in the systems. Typically the format rules – e.g., data serialization formats such as JSON, YAML, or Code in Programming Language – are expressed as context-free grammar (CFG). Due to the hallucinations and unreliability of LLMs, instructing LLMs to adhere to specified syntax becomes an increasingly important challenge.

We present SynCode, a novel framework for efficient and general syntactical decoding with LLMs, to address this challenge. SynCode ensures soundness and completeness with respect to the CFG of a formal language, effectively retaining valid tokens while filtering out invalid ones. SynCode uses an offline-constructed, efficient lookup table, the DFA mask store, created from the DFA (Deterministic Finite Automaton) of the language’s grammar for efficient generation. SynCode seamlessly integrates with any language defined by CFG, as evidenced by experiments focusing on generating JSON, SQL, Python, and Go outputs. Our experiments evaluating the effectiveness of SynCode for JSON generation demonstrate that SynCode eliminates all syntax errors and significantly outperforms state-of-the-art baselines. Furthermore, our results underscore how SynCode significantly reduces 96.07% of syntax errors in generated Python and Go code, showcasing its substantial impact on enhancing syntactical precision in LLM generation.

Published in Transactions on Machine Learning Research (05/2025)
SynCode: LLM Generation with Grammar Augmentation
Shubham Ugare
University of Mlinois Urbena- Chanaa-namppign, USA
Tarun Suresh University of Illinois Urbona- Caanna--aappiin, USA
Hangoo Kang
University of Blinois Urbona- Chanmpogg, USA
Sasa Misailovic
University of Illinois Urbona- Caana--ammppinn USA
Gagandecp Singh
University of Mlinois Urbona- Urbona-Champinn Champoign and VMware Reseorch, USA
Reviewed on OpenReview: https: // /tte:///pppeeveeeee net/ forum? id= Hd=HoUZggAPoH.
Abstract
LLMs are widely used in complex AI applications. These applications underscore the need for LLM outputs to adhere lo specific format, for their inlegration with other components in the systems. Typically the format rules e.g., data serialization formats such as JSON, YAML, or Code in Programming Language are expressed as context-free grammar (CFG). Due to the hallucinations and unreliability LLMs, instructing LLMs to adhere to specified syntax becomes an increasingly important challenge.
We present SYNCODE, a novel framework for efficient and general syntactical decoding with LLMs, to address this challenge. SYNCODE ensures soundness and completeness with respect to the CFG of formal language, effectively relaining valid tokens while filtering out invalid ones. SYNCODE uses an offline-cnnstuuced, efficient lookup table, the DFA mask store, created from the DFA (Deterministic Finite Automaton) of the language's grammar for efficient generation. SYNCODE seamlessly integrates with any language defined by CFG, as evidenced by experiments focusing on generating JSON, SQL., Python, and Go outputs. Our experiments evaluating the effectiveness of SyNCODE for JSON generation demonstrate that SYNCODE eliminates all syntax errors and significantly outperforms state-of-the-art baselines. Furthermore, our results underscore how SYNCODE significantly reduces 96.07% of syntax errors generated Python and Go code, showcasing its substantial impact on enhancing syntactical precision in LLM generation.
Our code available at https: wtw.tthu.coouuuoooooooollaaaa//a Additional resources are available at ittps://strructurellnn com.
1 Introduction
Recent rescarch has shown that transformer-basee large language models (LLMs) can play pivotal role within compound AI systems, where they integrate with other software tools ZZhaaria el. al. 2024 Mialon et al., 2023). For example, OpenAl's code interpreter (OpenA] 2024) generates and executes Python pro- grams automatically while responding to user prompts. Similarly, Wolfram Alpha (wolfram) 2024) translates user queries about mathematical questions into a domain-specific language (DSL) for utilizing various tools. LLMs utilized in various other applications to translate natural language text into formal languages,

---

Published in Transactions on Machine Learning Research (05/2025)
such inputs to logic solvers (Pan et al.
2023 Olausson et al. 2023) and theorem provers [Wu et al. 2022
Yang et al. 2023), among others. In all these applications, the L.LM output is expected to follow certain syntactic structure. However, challenges such as ballucination and non-robustness make LLMs unreliable for
such automated systems (Liang et al. 2023). Moreover, recent theorctical (Hahn 2020 Yang et al., 2024)
and empirical (Ebrahimi et al. 2020; Bhattamishra el al., 2020: Delétang et al. 2023 research suggests that
language models based on transformers show difficulty in learning basic formal grammars.
The interaction between software tools and LLMs commonly occurs through data serialization formats like JSON or YAML, or code in domain-specific or general-purpose programming languages, such as Python or Go. Despite advancements in techniques such as fine-tuning and prompt engineering, which enhance the model's ability, these approaches fall short of fully addressing the challenge of syntactical accuracy in generated output. This problem is especially prominent in two common scenarios: (1) using open-source models, which are typically relatively small, and (2) generating text for formal languages with relatively modest representation in the LLM's training data.
Modern LLMs generate text sequentially, from left to right, one token at time. For each prefix, the model computes a probability distribution over predefined vocabulary to predict the next token. The LLM's decoding algorithm dictates how these probabilities are used to generate the token sequence. Very re- cently, researchers have proposed new techniques for grammar-guided generation to enhance the syntactical accuracy of LLMs by modifying the decoding algorithm. Although they ensure that the model consis- tently selects tokens that adhere to a specified formal language (Scholak et al., 2001 Poesía et al. 2022; Gerganov and et. al.) 2024 Willard and Louf 2023), the existing approaches for grammar- grammar-guided gener- ation either suffer from high error rates, resulting in syntactically incorrect output or impose significant run time overhead in the inference:
Issues with syntactical accuracy: The language grammar consists of the terminals, fundamental building blocks of the language (e.g., keywords, operators). Typically, a lexer creates lexical tokens from the imput, each token associated with a terminal from the grammar. The LLM tokens form part of the model's fixed vocabulary, defined before training, and do not directly correspond to lexical tokens asso- ciated with any specific grammar. This discrepancy, known as token misalignment, presents a significant challenge in ensuring procise grammar-guided generation (Pocsia et al. 2022). Thus, formally showing the soundness of the algorithm poses a challenge for ensuring the precision of the approach.
Issues with high computational overhcad: Typically, the computational complexity of additional operations performed for syntactical generation is lower than the standard LLM generation operalions needed for propagating the input through LLM layers. However, these syntactical generation opera- tions are typically exccuted soquentially on a CPU, contrast to the GPU-accclerated LLM generation, adding to the run time. Achieving low inference overhead faces two primary challenges for syntactical L.M generation. First, the algorithm should facilitate offline computations that minimize the overhead during inference. Second, it should effectively utilize available hardware resources and offload additional computations to modern hardware, such as GPUs, to enable parallel computation.
Issues with generality: Prior works are restricted to specific LLM decoding schemes (Scholak et al. 2021 Lundberg et al. 2023). major challenge for generality is designing a composable algorithm that can integrate with any decoding strategy such as greedy, beam scarch, and different types of temperature sampling.
Our goal is to make grammar-guidet generation precise and efficient by imposing formal grammar constraints on LLM generations, ensuring the output adheres strictly to the predefined syntax.
SynCode. SYNCODE is an efficient and general approach for generating syntactically correct output. SYN- CODE takes a context-free grammar (CFG) represented with extended Backus Naur form (EBNF) rules and ensures that the LLM output follows the provided grammar. SYNCODE algorithm is general and can be composed with any existing LLM decoding algorithm, including greedy, beam search, and sampling.
During the LLM decoding stage, where LLM selects the next token, SYNCODE employs strategic two-step approach. In the initial step, it leverages partial output to generate sequences of terminals that can follow
2

---

Published in Transactions on Machine Learning Research (05/2025)
the partial output called accept sequences. This reduction to the level of terminals closer abstraction to language grammar than LLM tokens simplifiess the problem. Simultaneously, SYNCODE computes a remainder from the partial output, representing the suffix that may change its terminal type in subsequent generations. In the second step, SYNCODE algorithm walks over the DFA using the remainder and uses the mask store to compule the mask (a boolean array to filter the vocabulary) specific to each accept sequence. By unifying masks for each accept sequence SYNCODE gets the set of syntactically valid tokens.
To ensure the efficiency of SYNCODE's syntactic generation, we propose novel data structure called DFA mask store which is pre-computed offiine. DFA mask store is a lookup table derived from Deterministic Finite Automata (DFA) representing the terminals of the language grammar. SYNCODE algorithm can efficiently compute the syntactically valid next LLM tokens by leveraging this mask store. Moreover, the SYNCODE algorithm offers the additional benefit parallelizing a substantial portion of the syntactical LLM generation computations by offloading them to a GPU.
We demonstrate that the SYNCODE algorithm is sound ensuring it retains all synlactically valid tokens at every generation step. SyxCODE is also complete under specific conditions affirming it rejects every syntactically invalid token.
The SYNCODE framework seamlessly integrates with any language defined by deterministic CFGs and scales efficiently to generate code for general-purpose programming languages (GPLs). We evaluate SyNCODE's ability to guide the Llama-2-7B-chat and Gemma2-2B-ii models with the JSON grammar to generate valid JSON completions to prompts from the JSONModeEval (NousReseacch 2024 dataset. We empirically show that LLMs augmented with SyNCODE do not generate any syntax errors for JSON and that guiding Gemma2-2B-it generation with SYNCODE achieves 100% JSON schema validation accuracy. We evaluato SYNCODE on generating SQL queries from the text in Spider (Yu et al., 2018 and show that SYNCODE improves both compilation rate and execution accuracy. Further, we evaluate the augmentation of SYNCODE with a diverse set of state-of-the-art L.Mss for the code completion tasks using problems from the HumanEval and MBXP datasets (Athiwaratkun et al., 2023). Our experiments, conducted with CFGs for a substantial subset of Python and Go, demonstrate that SYNCODE reduces 96.07% of the syntax errors for Python and Go on average. The remaining syntax errors persist because the LLM fails halt generation before reaching the maximum generation limit defined in our experiments.
Contributions. The main contributions of this paper are:
We present parsing-based technique for decoding of LLMs by designing novel algorithms that allow us to efficiently generate syntactically correct output.
We implement our approach into a scalable and general framework named SYNCODE that can work with any formal language with user-provided context-free grammar.
We present an extensive evaluation of the performance of SYnCODE in generating syntactically correct
output for JSON, SQL and two general-purpost programming languages Python and Go.
2 Background
In this section, provide the necessary background on LLMs and formal language grammar.
Notation. Let the alphabet Σ be finite set of characters. We use to denote an empty string. Given a set S, we use 5t to denote the set of all i-length sequences that can be formed by selecting elements from S, and S* = UIeN St. Thus Σ' represents the set of all strings over characters in E, including the empty string c. Further, we use E+ denote (E* - c). Given two strings W1, W2 E Σ", we use W1.u22 to denote string obtained by concatenating w2 to wi. All symbols used in the paper are listed in Appendix A.I
2.1 Language Models
Current language models (LM) operate on vocabulary C E' of tokens. A tokenizer takes an input prompt Co Ex, which is sequence of characters, as input and converts Co into sequence of tokens 4, 12,.... lk-
3

---

Published in Transactions on Machine Learning Research (05/2025)
#ef fnctorialin: int) -> int; """Rturnss the factoriat of
LLM
Prompt C
DFA Mask Store
mask m
ReeReturss if
Lexer
else: return factoriatin-11
return
C
S
Decoding Algorithm
Parser
Incremental Parser
Compieted Code
Figure In the SYNCODE workflow, the LLM takes partial output C. and generates a distribution for the next token 44+1. The parser processes Ck to produce accept sequences A and remainder These values are used by the DFA mask store to create token mask, eliminating syntactically invalid tokens. The LLM iteratively generates token fx+1 using the distribution and the mask, appending it to Cj to create the updated code Ck+1- The process continues until the LLM returns the final code Cn based on the defined stop condition.
Figure 2 shows a typical tokenization method, where common words (e.g., def )) have their own token (even with space in front), while rare words (e.g., incr sacr_list] ) are split into multiple tokens. In order to generate the next token, the LM M : V - RIVi takes as input the sequence of tokens t1, ... .,....,,,. tk, and outputs a vector of scores & over the vocabulary: = M(1,, ft). The softmax function softman( (2) exp(z:) (exp((z))) transforms into a probability distribution over the vocabulary V.
Decoding. Building upon this, the language model M is recurrently applied to generate sequence of tokens f1, When choosing the (k+1))th token, the prob- ability distribution for the next token is obtained through softmax(M (2,, 4x)). Various approaches for token selection from this distribution have been explored in the literature such as greedy decoding, sampling, and beam scarch. Each technique is repeated until the prediction of special end-of-sequence loken, EOS or the fulfillment of another stopping criterion. This iterative process is equivalent to sampling from a distribution over V potentially resulting in multiple fcasible decodings.
String
def iner incr_list(1; list):
"Return List elements incremented by 1.
Tokens
def iner
list
list
An
Nt
Return
list
elements
increment
ed
by
1.
Figure 2: Tokenization of a string.
Constrained Masking. In the context of decoding, we encounter scenarios where excluding specific tokens at particular positions becomes crucial (e.g., excluding harmful words). This implies we can disregard these tokens and procoed with decoding based on the remaining set. An algorithm for such masking relies on a function fm to generate the mask m based on the exact use case. In the mask E {0, 1391, '1' indicates Decoding methods mentioned earlier can be applied to
viable token, and '0' signifies discarded O softmar(2), where O represents element-wise multiplication. The resultant vector should be scaled by 1/ (m x softman(2)); to restore correct probabilities. Algorithm I presents the steps for masked decoding. In SYNCODE, we use the constrained masking technique to exclude syntactically invalid tokens.
2.2 Formal Language Grammar
A formal language syntax is represented by defining a grammar. A formal grammar is essentially a set of production rules that describe all possible strings in a given language. A grammar consists of terminal and nonterminal symbols, where terminal symbols are the actual characters or tokens in the language, while nonterminal symbols are placeholders used to define patterns or structures within the language.
The syntax for most programming languages can be defined using context- free grammar (CFG). CFG is a formal grammar that consists of production rules that can be applied to nonterminal symbol regardless of

---

Published in Transactions on Machine Learning Research (05/2025)
its context. In CFG, each production rule is of the form E with E single nonterminal symbol, and A a string of terminals and nonterminals (8 can be empty)- Regardless of which symbols surround it, the single nonterminal E' on the left-hand side can always be replaced by 3 on the right-hand side.
Terminals. We use I to denote the set of ter- minals the grammar. Regular expressions are used to describe the terminals. For instance, A regular expression A * is used for integer literal: This regular expression describes quenee of one or more digits (0 to 9). We use to denote & regular expression and L(p) C E" to denote the language recognized p. Regular ex- pressions are often associated with the creation of Deterministic Finite Automata (DFAs). DFA is theoretical construct. used to recognize pat- terns specified by regular expressions.
Algorithm Masked LLM Generation
Inputs: M: LLM, T: tokenizer, Co: imput prompt string, fm: function that generates mask, Amari maximum generated tokens, D: any decoding algorithm Output: string Cn
1: function MASKED T, Sm. Co)
Tour Tokenize Co)
3: for iE (1.... Rmmar) do
scores + M(T cur)
5:
m + J (1' T)
Definition 1 (DFA). A deterministic finite au- tomaton (DFA) D is 5-tuple, (Q, 6, qo F), consisting of finite set of states Q, finite set of input symbols called the alphabet E, transi- tion function
scores +- scores
7:
f <- D scores)
8:
if & EDS then
break
10:
Tow f- append (Twny L:)
Q x E + Q, an initial state go E Q and a set of accept states FC FQQ.
11:
Cn Detokenize TeTrr)
12:
return Cn
Let W = 0102 an be string over the alphabet
E. The DFA computation 8" :Q E- + Q on a string w defined as 8* (ro, w) = Tn when r++1 = 8(1s, 0141, for .,...,n 1. The automaton accepts the string if 8* (go, w)
Lexer. We assume lexical analysis with 1-character lookahead and no backtracking. This assumption is crucial for the elficiency of SYNCODE algorithm.
Definition 2 (Lexer). The function Lex is defoned to take partial outpul Cj E* impul and produce sequence h, 1, of lerical tokens where E".
3 Overview
3.1
Illustrative Example
start: expr
Consider an example grammar Figure that uses the Lark EBNF syntax for defining the gram- production rules. The grammar represents Domain-Spectin Language (DSL) consisting of arithmetic expressions with basic operations like ad- dition, subtraction, multiplication, and division over integers and floating point numbers. also in- cludes support for parentheses to specify precedence and allows functions like exponential (math exp), square rool (math sqrt), sine (math sin), and co- sine (math_com cos) to be applied to expressions.
expr "L" term
expr tern
torm: factor
tem "g" factor
"y" factor
11
factor: INT FLOAT "(" expr "]" function "(" expr ")"
13
function: "nsth_exp" "rath_sqrt" "rath_sin"
"sath_cos"
14
15
INT: /(0-91+/
16
FLOAT: /10-91+). (0-9)4/
18
Vignore
The symbols in the grammar such as erpr and factor that can expand into other sym- bols through the application production rules are called non-terminals. or INT cannot be further expanded and are called terminals. Let the r (lpar; rpor, add, sub, mult, dāu, int, float, math eap, math sqrt, math sin, math cos} represent the set
Figure 3: Example grammar for illustration.
Symbols such as
5

---

Published in Transactions on Machine Learning Research (05/2025)
of all terminals of the grammar. The terminal int is defined by the regular expres-
sion [0-9]=: and fooa is defined by the regular expression [0-9]+_[0-9]++ We use terminals
lpar, rpar, add, sub, mult, die, math exp, math sqrt, math sin, math__on cos, to denote the strings (
math_exx
math_sqra math_ sin
math_ cos respectively.
Task. Consider an LLM that is used to translate a natural language text to an expression in the DSL defined above. Since LLMs are typically not good at mathematical calculations, it is common in- stead let the LLM generate intermediate outputs in certain syntax, and an interpreter of the DSL then computes the LLM's output into accurate re- sults (Mialon et al. 2023). Figure presents the prompt we use for our illustrative example, contain- ing 2 question-answer pairs before the actual question that we want the LLM to answer. Providing question- answer examples before asking the actual questions is called fow-shot prompting (2-shot in this case) and significantly improves the model's accuracy (Brown et al. 2020).
Question: Can you sin degrees cos 60 degrees? wath sith_sin meeth_oo(66)
Question: what is exponent of addition of first prine numbers?
Answer: math expl2 3 + 5 7 11)
Question: what is the area of equilateral triangle with each side 2.27? Answer:
Figure 4: Prompt for the example which is provided as input to the LLM.
Standard LLM Generation. As described in Section 2 the stan- dard LLM first tokenizes the input and then iteratively predicts the next token from its vocabulary V. Figure 5 presents the output from the LLaMA-7B model and our SYNCODE when given the Fig. 4 prompt. The output of the model not a valid program in the DSL:; it uses funetions math area and math side that do not exist in the grammar. Further, LLaMA-7B does not stop after generating the answer to question and continues to generate more irrelevant question-answer pairs. SYNCODE on the other hand guarantees the syntactic validity of the LLM's output by excluding syntactically invalid choices when gencrating each token. For example, after gen- erating math SYNCODE excludes Laaeea and other choices from
Dhansand
arealmaith side 271)
Question:
the equation
Question: what
math sqriS SQUiE (2227) (2227)"022))) (2.27)
Output from LLM without and with SYNCODE. The colors rep- resent the tokenization of the output. the LLM's vocabulary. The LLM opts for _sqrt which is the top syntactically valid choice and continues
Figure 5:
the generation from math_sqrt,
Constrained Decoding, Let G denote the grammar in our example and L(G) C Σx denote all syntactically valid strings in the grammar. Ideally, we want the final LLM output Cn to be in L(G). Strings such as math_ math_exp(2 3 7 11) and math sin (30) math_ cos (60) belong to L(G) as they are syntactically valid. Let Ck denote the LLM's partial output during the k-th iteration of LLM generation. Suppose GG) denotes all prefixes of L(G), ie., all strings that can be extended to a syntactically valid sin output. math_sinn30 and math_sin sin (30) + math are in LG)) they can be extended to be syntactically valid. By ensuring that at each intermediate step, the invariant that the LLM partial generation C₁ is in the set (G) is maintained, we can guarantee that upon completion of the generation process, Cn will indeed be syntactically valid, i.e., C₂ E L(G). This ensures that an intermediate output such as math_arre area which is not in GG) is never generated by the model.
3.2 SynCode Algorithm
A key challenge in syntactic generation is token misalignment, where LLM tokens do not directly correspond to lexical tokens from the grammar. The main reason for the high error rate in syntactic generation in prior works is the lack of formalization in their approaches (Section G Our work addresses this challenge by providing an algorithm that is provably sound - retains all syntactically valid tokens and is complete under specific conditions senittioons-rjeettnnng every syntactically invalid token at every generation step.
Another significant challenge for efficiency is developing a novel algorithm that facilitates offiine computations that minimize the overhead during inference. SYNCODE tackles this challenge by creating a novel structure
6

---

Published in Transactions on Machine Learning Research (05/2025)
called the DEA mask store offiine. For a given grammar G and vocabulary V, this mask store is constructed once and can be used across all generations. DFA mask store maps states of DFAs (corresponding to terminals in the grammar G) to boolean masks m E {0, 10 over the vocabulary. This approach also benefits from parallelizing a substantial portion of the syntactical LLM gencration computations by offloading them to a GPU during inference.
Furthermore, it is challenging to ensure generality with efficiency. Many prior works are restricted to syntactic generation with specific type of decoding (Scholak et al.] 2021 Lundberg et al.) 2023). At k-th LLM iteration, for partial LLM output Ck, let Vi S V denotes the subset of vocabulary such that for any token E V& the intermediate generation continues to maintain the invariant Cx.t (G)). Our formulation for computing V from V is highly general and can be integrated with any decoding algorithm, such as greedy, sampling, or beam-search. Any algorithm that could potentially be applied to V can instead be applied to Vg. The mask store allows more efficient computation of a subset of tokens Vi-
SYNCODE works in two steps: first, it parses Cs and computes the unparsed remainder r E E' along with the acceptable terminal sequences A (formally defined in Section 4.2) In the second step, SyxCODE utilizes r, A, and the mask store. This step involves traversing the DFA and performing few lookups within the DFA mask store to obtain a subset of tokens Vi. In the following sections, we claborate on these steps using our illustrative example.
Parsing Partial Output. SYNCODE'S parsing of partial output Cs begins with lexing Cg. We assume our lexer has a 1-character lookahead and no backtracking. This assumption ensures that LLM's future generations do not alter the lexical types of any previous lexical tokens except for the final lexical token. The remainder denotes the suffix of Cg that may still change its lexical type in subsequent iterations. We define two cases for assigning r:
Case is when Ck contains an unlexed suffix and here we assign r = u. For example, C2 firr__ssrt(3 (3) * (2. is lexed as math_sqrt, ( 3 *
2. where math_sqrt
( are lexical tokens of type math_ ssrt, lpar, int, rpar, mult, Ipar, respectively. Here (2 followed by - is unlexed suffix which we assign as the remainder r.
3 )
Case 2 is when Cs ends with a complete lexical token, where is assigned the value of the final lexical token. Hence, C2 (oth_ssrr(3)
(2 lexed as math_ sqrt | C 3
2. 2. Where bath_sqrt
are lexical tokens of type math sqrt, lpar, int, rpar, mult, lpar, respectively. Although 2 is the complete final lexical token with type int, it is assigned as the remainder since in the subscquent iteration it may even change its lexical type to float.
( 3 >
In both cases, our lexer assumption ensures that the portion of Cg excluding the remainder will retain its lexical tokenization in subsequent LLM iterations. The assumption is crucial to enable incremental parsing and ensures that that the remainder r is always small, both of which contribute to reducing time complexity.
Accept Sequences. Given a sequence of lexical tokens I,,...l; we use a bottom-up LR parser to compute what types of lexical tokens are acceptable next according to the grammar. If at a certain point in the generation, we have lexical tokens math_sqrt p c) 3 > * ( 2.27 then the immediate next lexical function of the parsed partial
token can be of type rpar, add mult. We define accept sequence output (excluding the remainder) as sequence of terminals such that those lerminals can follow the currently parsed output (Definition 7p. For instance, in the case Cs = mlataasgrttet (3) * (2.27 {rpar), {add} and (mult) all are 1-length accept sequences. {add, int) and {add, flout) are some of the 2-1ength accept sequences for this example that can follow the current partial output. In Section 4.2 we show how we efficiently compute accept sequences of length and 2 using an LR(1) parser, leveraging its immediate error detection property (Aho and Johnson 1974). Further, we discuss how an LR(x) parser can be used to compute accept soquences of length x efficiently. However, in practice, SyNCODE can effectively operate with shorter accept sequences while still ensuring the soundness of syntactical generation (see Theorem thereby avoiding the high memory needed for LR(s) parsers for large values of

---

Consider accept sequence A {float, rpar). Figure displays the DFAs float of Published in Transactions on Machine Learning Research (05/2025)
DFA Mask Store. SYNCODE parsing step partitions partial output C₁ into lexically fixed part CF and remainder r. The accept sequences A are computed using the parser state on parsing C and denote the terminals that can follow CF Thus the problem of obtaining subset V₁ of tokens that will lead to syntactical continuation can be reduced to aligning accept sequence A E A with the string r.t obtained by concatenating remainder r and LLM token the vocabulary. One approach is to iterale through LLM vocabulary and verify this alignment for each token individually. However, this method is inefficient due to the need for matching |V] tokens with |A| terminal sequences. In SYNCODE algorithm, the precomputed DFA mask store is crucial for allowing efficient computation of acceptable tokens Vg- Next, show how the mask store maps the states of DFAs of the terminals and sequence of terminals to masks over the vocabulary to enable this process.
Given a remainder r and any accept sequence A E A, we want to check for a token t E V, if r.t aligns or partially matches with A. We formally define this notion of partial match in Definition We establish a connection between the match of terminal sequence and a string through the DFAs corresponding to the terminals.
Figure 6 presents DFA for the terminal int. In this DFA, q is the start state, and ofth is accept state, Further, we say that qoth, gime are live states since there is a path from those states to an accept state and the state uind is not. a live state.
Consider the partial output C Saath_sqrteee (2 As described above, in this case, the output is split in the parsed part math_sqrt ( and the last lexical token 2 which is the remainder. {int, add), {int, rpar), {foatt] are some of the accept sequences. For each of these accept sequences, we want to compule tokens E V such that appending and / i.e. 2 partially matches the accept sequence.
92
9]
else
4
41
[0-9]
9]
D
int
corresponding to the terminals in A. If begin from the initial state go
Ditut and change the current DFA state according to the characters in r, in our example with r r= 2 the resulting state of the DFA is of foot, We observe that any token / E V is acceptable if continuing the DFA walk from of ends a live state. We also allow transition from the end state and start state of DFAs of subsequent terminals in the accept sequence as shown by the dotted arrow. The partial match of r.f and A can thus be equivalently checked by doing a walk over the DFAs. Tokens such as 11 -.
Figure 6: DFA for terminal int.
and .27) are some of the tokens whore initiating a walk from of flout lcads to reaching one of the live states. For example, by consuming 27) we reach q," which is live state. Conscquently, SYNCODE approves 27) as valid continuation from C2 Seath_sqrt(3) (3)
(2
Our key insight for achieving efficiency is that for each DFA state, we can precompute LLM tokens that will lead to a transition to a live state starting from that state. Precomputing these sets can significantly reduce the computation require during inference. Further, these precomputed set of LLM tokens can be stored as boolean masks for efficiently combining them during inference. Given a DFA state q and any sequence terminals of length a, the mask store maps Ma(g, A) m, where E {0, 1 is the mask over vocabulary. During the inference time, for each accept sequence A E A, we first consume and walk over the first DFA
else
10 9)
else
else
(0 9)
[0-
Diooat
D rpar
Figure 7: DFAs for accept sequence A = {foat rpar).
8

---

Published in Transactions on Machine Learning Research (05/2025)
in the accept sequence. We then use the map Ma on the current DFA state to get the mask ma of valid tokens for A. Hence, for each accept sequence A E A, we require walk over DFA and a lookup in the mask store to obtain mA.
Finally, we combine these masks obtained for each acccept sequence to get the masks of all syntactically valid tokens by computing their union UAEa ma. In practice, these masks can be stored as tensors and can be combined efficiently using a small number of tensor union operations. We show in Theorem [ that this combined mask overapproximates the set Vk, ensuring the soundness of our approach. Further, we show that for the LR. parser with larger lookahead, our approach is complete and ensures the combined mask is exactly VA (Theorem
Bringing It All Together. In our example, SYNCODE improves the LLM's output by guiding the gen- eration. Initially, the LLM produces math as C1. Next, SYNCODE excludes LLMs top choices such as _area tri and _P| from the vocabulary, leading the decoding algorithm to select _sgrt Further, even the 12th iteration where the LLM outputs Cn = math_ (eathssrr(33)4 (2.27 SYnCONE filters out the LLM's preferred choice from the vocabulary. Instead, the LLM opts for *, eventually generating C, aath_sqrt&in (3) /4 (2.27) (2.27) } which is syntactically correct ie. C, E L(G) and also semantically accurate.
3.3 Time Complexity
At each decoding step in SYNCODE, the most resouroe-intensive tasks are computing accept sequences and generating the mask using and A. In Section 4.6 demonstrate that our implementation, leveraging LR(1) parsing, efficiently constructs 1 and 2-length accept sequences. We show that the complexity of SYNCODE at each decoding step is O(Tu [Al), where Tu represents the time needed for boolean mask union operations. Typically, |A| is small (<10 on average in our experiments) and in the worst case, it equals the size of set of all terminals [T] in the grammar. For our largest Python grammar, [T] 94. Modern hardware, especially with GPUs, can perform these vectorized union operations efficiently (Paszke et. al. 2019b), making the SYNCODE algorithm efficient in practice.
4
Syntactically Correct Generation
This section describes our main technical contributions and the SYNCODE algorithm.
4.1
Syntactical Decoding Problem
Given a language with grammar G, let L(G) S Ex denote the set of all syntactically valid outputs according to the grammar G. For grammar G, Lp( represents the set of all syntactically valid partial outputs. If string w1 belongs to (G), then there exists another string such that appending w2 to results in a string that is in the language defined by G. Formally,
Definition 3 (Partial Outputs). For grammar G, L,(G) S Σ" denotes all syntactically valid partial outputs. Formally, if w1 E Lp) then Ewzz E E' such that wT.wy E L(G)
For a grammar G and a partial output Cj belonging to the set of prefix strings (G),,, the syntactical decoding problem aims to determine the set V& of valid tokens from finite vocabulary V such that appending any token t E VR to C₂ maintains its syntactic validity according to the grammar G.
Definition 4 (Syntactical Decoding). For grammarr G, given partial output Ck E L,p(G) and finite token vocabulary V C E", the syntactical decoding problem is to compute the set V2 C V such that for any I E Vn, Cx.l E (G)
We next present SYNCODE's key aspects to solve this problem:
In the initial step, it parses C and computes the unparsed remainder r E E* along with the acceptable terminal sequences A (Section 4.2
9

---

Published in Transactions on Machine Learning Research (05/2025)
In the second step, SyxCoDE utilizes r, A, and the precomputed mask store. This phase involves travers- ing the DFA and performing a few lookups within the DFA mask store to obtain the set of syntactically valid tokens t capable of extending C (Section 4.3)
Consequently, SYNCODE efficiently computes the set of syntactically valid tokens. We show the soundness and completeness of our approach in Section 4.4
We further discuss the theoretical complexity of SYNCODE in Section 4.6 and the SYNCODE framework
in Section 4.7
4.2 Parsing Partial Output
In this section, we describe the remainder r and accept sequences returned by the parsing step.
Remainder. SYNCODE uses lexer to convert Cg sequence of lexical tokens 11.2....4 E*. Each lexical token 4, is associated with a terminal lype Tı where E [prr)) (Pr, the regular expression for terminal 74). We assumee our lexer uses a 1-character lookahead without backtracking. This ensures that the lexical types of previous tokens in Cs remain unchanged, except for the final token. The remainder r represents the suffix of C₁ that could potentially change its lexical type in future iterations. Thus the remainder r is assigned such that it is either unlexed because it does not match any terminal, or has been lexed but might undergo a different lexing subsequent iterations when Cs is extended by the LLM by appending tokens. This assumption is crucial for enabling incremental parsing and ensures that the remainder r remains small, which contributes to reducing overall time complexity. SYNCODE assigns the remainder according to the following two cases:
Case 1: Cs = 1..2...14 Assuming standard lexer with 1-character lookahead and no backtracking, all lexical tokens 4, lu.2......4 ..,22 remain unchanged upon extending Cg. However, the final lexical loken l5 may change. For example, in Python partial output in the k-th LLM iteralion, if the final lexical token is l re and the language model generates the token urn in the next iteration, the updated code results in the final lexical token becoming 1 = return This transition reflects transformation from an identifier name to Python keyword in the subsequent iterations. Thus, r is assigned the value 1,, i.e., r =fret for k-th iteration in our example.
Case 2: C+ = 1444222 .........: Here, u E E is the unlexed remainder of Ck- In this case, considering the 1-character lookahead of the lexer, the types of 11,12,...,44 do not change upon extending Ck. Consequently, r is assigned value u of the suffix that remains unlexed.
SYNCODE parsing step partitions partial output Cs into lexically fixed part C. and remainder r. Given a sequence A = To, TI...., we simplify notation by using L(A) = L(Pro Pn Pr.y) throughout the rest of the paper.
Definition 5 (Partial Parse). Given the partial output Cs E", the partial parse function pporse D' I x E" returns terminal sequence A and remainder such that C Cg. and Ch is parsed AD, t.e. cP E L(A
Accept Sequences. A sentence is sequence of terminals. A grammar G describes a (possibly infinite) set of sentences, that can be derived by using the production rules of the grammar. We use LG) S r to denote the valid sequences of terminals that can be derived from the rules of G. Further, (G) denotes all syntactically valid partial sentences of terminals. Formally,
Definition 6 (Partial Sentences). We define set of all synlactically valid partial sentences I. C r such that A E (G) if and only if DY E T such that E L
Note that L(G) and LG( are defined over alphabet Σ, whereas LF(G) and L.(G) over terminals r. Nevertheless, if a program C is parsed to obtain terminal sequence A, then E L(G) is equivalent to A E I (G). The SYNCODE parsing algorithm obtains A = Tı T... Ty by parsing Ck corresponding to the parserd part of partial output C. Given a partial sentence An, an accept sequence is sequence over r such that when appended to A the result is still partial sentence.
10

---

Published in Transactions on Machine Learning Research (05/2025)
Definition 7 (Accept Sequence). Given partial output Ck E Lp(G), and AD, pparse(Ci)), A1 I is accept sequence if ^ _ E (G)..
Consider a Python partial program C2 def is and let def. name, lpar and rpar be the terminals Python grammar. we get {def} is = pparse def is where A = {def} and r =s A1 = {name, lpar, rpar) is an accept sequence in this case as the sequence of terminals A partial sentence. The parser state on parsing the partial output. C. can be utilized to compute a set of accept sequences denoted as A. The soundness and completeness of the SYNCODE algorithm depend on the length of these accept sequences in A. In theory, using longer accept sequences enhances the precision of the SynCODE algorithm at the cost of increased computational complexity. In Section 4.5 we show our method for obtaining and 2-length accept sequences that are efficient and precise practice.
{def. name, lpar, rpar} is valid
4.3 Grammar Mask
This section outlines the utilization of the set of acceptable terminal sequences and the remainder r the creation of boolean mask using the DFA mask store which is subsequently used for constraining the LLM output. The DFA mask store is constructed offline and makes SYNCODE efficient during the LLM generation. Given partial output Ck, our objective is to identify tokens E V such that appending them to C₁ leads to syntactical complction. Given remainder and set of sequences A. the goal is to determine whether r.l partially matches the regular expression derived from any of the sequences in A. To characterize the notion of strings partially matching a regular expression, we next introduce the function pmatch.
Definition (pmatch)) The function pmatch lakes word w E E, a regular expression and relurns boolean. pmatch(w, p) true if either of the following conditions holds:
w2 E E+ such that = w1. and L(p) 1. Jwn E ,,,w2
2. Dwi E E' such that w.w, E L(p)
Thus pmatch(w, p) is true when either a prefix of w matches P or w can be extended to match The consequence allowing pmatch to be defined such that it is true even when prefix matches, that SYN- CODE will conservatively accept all tokens for which the prefix matches the accept sequence. Hence, we overapproximate the precise set of syntactically valid tokens. We make this choice ensure that SYNCODE is sound for any length of accept sequencos. Next, give definitions related DFAs. These definitions are useful for describing the construction of the DFA mask store and proving properties related to its correctness in the SYNCODE algorithm. In particular, we first define the live states of DFA. We say state is live if there is a path from to any final states in F. Formally,
Definition 9 (DFA live states). Given a DFA D(Q, O,Q,E, 6, go. F), let live(Q) S Q denote the set of live states such that
q (Elive(Q) isf 3w E* s.l. 8" (w, F
We use D2 (Q,, E,, Sy, 1b, F.) to denote a DFA corresponding to a terminal T. Next, we establish the definition dmatch for DFA, which an equivalent concept to pmatch with regular expressions. dmatch is recursively defined such that its computation can be performed by walking over the DFAs of a sequence of terminals.
Definition 10 (dmatch). Given a DFA D(Q, E, F), a string w E E", a DFA state 9 E Q and any sequence of terminals A [T 7/42 Ts+d), dmatch(w, A) true, if either of the following conditions hold:
8* (w E live(() or
Twr E E', W2 E E such that w1 wr.wz w, 8* S'(1, 9) E F and A = 0 or
Eer E E', W2 E such that w1 = w, 8* (0( 9) E F, T+t: [[J+2.... T/+4)) true where 40
and dmatch(wy, (20c 40
rit1 the start state corresponding to the DFA
for Tf+1
11

---

Published in Transactions on Machine Learning Research (05/2025)
Given accept sequence A (72.. A, objective is to compute the set of tokens
E V such that pmalch(r.I, holds, where PA = (p.. PP++)) is the regular expression obtained by concatenating regular expressions for terminals. If A" denotes the sequence {TS+2.... (T Lemma [ simplifies this problem to finding dmatch(r.t, ,A²). Furthermore, utilizing Lemma 2 this can be further reduced to computing y = (r, and dmatch((, AP). It's important to note that dmatch((, q, AP) does not depend on Ck and can be computed offline. While the computation of q for dmatch(t, q, AP) is relatively inexpensive, evaluating dmatch((, q, A²) can be computationally expensive both offline and online, as it requires considering numerous potential accept sequences offline, and where it needs to iterate over all tokens in V online. We observe that if we consider sequences of smaller lengths, we can efficiently precompute the set of tokens satisfying dmatch(t, g, AP) for all q and A offline. We later establish the soundness of SYNCODE when using accept sequences of length at least (Theorem [ and completeness for accept sequences of the length greater than maximum length of tokens in the vocabulary (Theorem 2 Typically, LLM tokens are small in size, allowing us to obtain these guarantees.
Lemma 1. Given A [T54I, T542. ... Tyta), ..Fa)) AP = dmatch(u w. % AA)) + pmatch(wu,P.) Px).
and = (pf+1. (F.aa).......aa)))
Lemma 2. If q (r, (6) and no prefir of is in M(T) i.e. Jun E', uy E Ex such that un .02 = and (w 40) E Fr then dmatch(t,,, q, A) + dmatch((.t, A).
The proofs both the lemmas are in Appendix A.2
Illustrative Example: Consider the scenario with C2 def is r is and an accept sequence A [name, (par, rpar] in A, where name, lpur, and rpar are terminals in r. Our objective is determine all t E V such that def is tt forms valid partial program. This can be achieved by finding tokens t that satisfy pmatch( is It, Px), where Pa = Ja-z, A-Z, l" Let's consider a token t = pprime (): We observe that is is_prime() O: can be decomposed into is is_prime (name), (4par), ) (rpar), and : Consequently, it partially matches as defined by pmatch. In Figure g we present the DFAs for A used in computing dmatch. The reduction dmatch(r.t.q. ,par, rpar) dmatch( is is_prime () 4gaane :lar, rpar) simplifies
successively to dmatch( () |0: 10000r,, .rpar), then to dmatch() ): 40 .), and finally to dmatch(]: | (,1"""))) As dmatch :
01 q1te is a final state, according to condition 2 of Definition 10 a mask over vocabulary
1 (ime) holds true. Next, we define
Definition 11 (Vocabulary mask). Given vocabulary V S Σ", m 13, is a mask over the vocabulary. We also use set(m) C V to denote the subset represented by m.
DFA Mask Store For integer a, define DFA table Ma as the mask store over the DFA states with a lookahead. Given the set of all DFA states On = Uter Qr, the table stores binary masks of size [V), indicating for token string t, for any DFA state E Qn and a sequence of terminals A, if dmatch(t, q, A₂) = true. The lookahead parameter a signifies the number of subsequent terminals considered when generating the mask stored in the table. Choosing a larger value for a enhances the precision of SYNCODE algorithm, but it comes at the cost of computing and storing a larger table. We next formally define the DFA mask store,
lovg.AuZ....
alse
DEL
par
[nn2 A-Z_.]
D name
Dipar
D
rpar
Figure 8: DFAs in accept sequence A = name, lpar, rpar} for example. The start state, final states, and dead states are in gray, green, and red respectively. The dashed arrows link the final states of one DFA to the starting state of the next DFA, adhering to condition 3 in Definition I0 This illustrates the sequential traversal across DFAs during the computation of dmatch.
12

---

Published in Transactions on Machine Learning Research (05/2025)
Definition 12 (DFA mask store). For an integer a, the DFA mask store Ma is function deftned Ma Qn x r + 10,1) where Qn = U+er Qr represents the set of all DFA states and a is a set of a-length terminal sequences. Then Ma(g.A) m is binary mask such that t E set(m) if dmatch(t,
For our illustrative example if m - MM2 { rpart) then t _ O): should be contained in set(m).. The grammar mask for a set of accept sequences A can be computed by combining masks for each A E A. The DFA mask store Mu maps each DFA state to all tokens such that they pmatch without considering any following accept sequence (O-length sequence). In this case, the table maps each state with single mask denoting the tokens that match the regular expression of the corresponding DFA.
Computing Grammar Mask The mask store is constructed offline by enumerating all DFA states Qn. considering all possible terminals in r, and all tokens in V. The DFA mask store depends on the set of terminals I and the model's vocabulary V. As a result, a unique mask store is created for each grammar and tokenizer combination, and to enhance efficiency, we cache and reuse this table for future inferences.
Algorithm 2 presents our approach for comput- ing the grammar mask during LLM generation. It computes grammar mask based on the sets of current accept sequences A, and the remain- der string (r). It iterates over A, considering each sequence A. The algorithm initializes an empty mask m. Tt. iterates over each accept- able sequence, considering the first terminal Ti in each. It computes the resulting state q, by processing Ti from an initial state % and the remainder string r. If or is in live state, the al- gorithm updates the grammar mask by unifying the mask cached in Ma.
Algorithm 2 Computing Grammar Mask Inputs: A: set of accept sequences, r: remainder
1: function GRAMMARMASK((A1)
2:
m f 0
3:
for A E A do
4:
9. + A
5:
q. + 840' .r)
if gr live(Qn)) ) then
IT + len(A)
m (Mi(gr, A[1 :))
return m
4.4 Soundness and Completeness
This section establishes the soundness and completeness of the SYNCODE algorithm. Algorithm 3 presents the LLM generation algorithm with SYNCODE. It takes as inputs an LLM represented by M, a tokenizer denoted by T, an input prompt string Co, the maximum number of generated tokens Amaa and a base decoding strategy D. The algorithm begins by tokenizing the input prompt using the tokenizer. It then iteratively generates tokens using the LLM, decodes the current token sequence, and performs parsing to obtain acceptable terminal sequenees A, and a remainder (line B. A grammar mask is applied to the logit scores based on these values (line 7 The algorithm subsequently selects the next token using the decoding strategy, and if the end-of-sequence token (FOS) is encountered, the process terminates. The final decoded output is obtained, incorporating the generated tokens, and is returned as the result of the Meneeaee algorithm.
Given partial output C E (G), SYNCODE generates corresponding mask m. Tf, for a token & E V, the concatenation Cg.t results in a syntactically valid partial output, i.e. Cx.t E (G), our soundness theorem ensures that t is indeed member of the set defined by the generated mask m. The subsequent theorem formally states this soundness property.
Theorem 1. Let C E Lp(G) be the partial output and any integer d 1, let Aa rd contain all possible accept terminal sequences of length and r e E" denote the remoinder. If m GrammarMask(AAn7 then for any t V, if Ck.t E Lp(G) then E set(m)
The proof of the theorem is in Appendix A.2
Next, we give a definition that establishes a partial order on sets of terminal sequences, where given two sets Ai and A2, we say sets Ai x A2 if every sequence in A has a prefix A Definition 13 (=). We define partial order & set of terminal sequences P(I*) such that Ai A when AΛ₂ A,EA1 E A, E T s.t. A2 = A
13

---

Published in Transactions on Machine Learning Research (05/2025)
We further state the lemma that shows the relation in the grammar masks generated by two accept se- quences satisfying relation x.
Algorihhm 3 SYNCODE Generation
Inputs: M: LLM,, T: tokenizer, Ca: input prompt, Timari
Lemma 3. Given A and A2 are set of accept sequences such that Ai & A2 and mi GrammarMaski r) and m2 GrammarMask(Ap1 then sel(m2) S sel(m1)
maximum generated tokens, D: decoding strategy
1: function MASKED
T, Co Ymmar D)
2: Tour + Tokenize(T, ,C,
3:
for i E (1.... Mmaz)) do
4: scores + M( cur)
The proof of the lemma is in Appendix A.2
5:
C2 decode(T (T, .''w
6:
A, r + Parse(Ci C2)
Theorem proves soundness for accept sequences
7:
m 4- GrammarMask(AAr1
Aa of length d, while Lemma 3 extends this proof to any set of accept sequences A where A き Ad- Our implementation, employing sequences of vary- ing lengths, can be proven sound based on this ex-
8:
scores 4- m @ scores
9:
+ D(scores)
10:
if f. = EOS then
11:
break
tension.
12:
T Tew + append L:)
The completeness theorem ensures that, under spec- ified conditions, each token t E set(m) guarantees Ck-l as syntactically valid partial output. An im- plementation of SYNCODE with short length of accept sequences although sound, may not guarantee completeness. To illustrate, let's take the example
13:
output <- decode(T, Tewr)
14: return oulput
where A = T/+2 E A with simple singleton regular expressions Pry+1 -[ and Pry+a (
In this
case, our algorithm conservatively treats all tokens V as syntactically valid, whenever
(C is a prefix
of those tokens (e.g.: (CC) [O] (Q)) even though some tokens may not meet syntactic validity. However, by assuming that the accept sequences are long enough, we establish the completeness of the approach.
Theorem 2. Let Ck E Lp( be the partial output, let Aa S rd contain all possible accept terminal sequences of length and r E E* denote the remainder. Suppose for any t E V,d > len(t) and m = GrummarMask(Aus such that I E sel(m) then Ck.l T.p(G)
The proof of the theorem is in Appendix A.2 While our completeness theorem ensures the SYNCODE consistently extends syntactically correct partial outputs, it does not guarantee termination with a correct and complete output. The focus of the theorem is on gencrating syntactically valid partial outputs, and the theorem does not address whether the process converges to a syntactically correct whole output. Termination considerations go beyond the completeness theorem's 's scope.
4.5 SynCode Implementation
Base LR. parser: Bottom-up LR parsers, including LR(1) and LALR(1) parsers, process terminals generated from the lexical analysis of the code sequentially and perform shift or reduce operations (Aho et al. 1986). LR(K) parsers have the immediate error detection property, ensuring they do not perform shift or reduce operations if the next input и terminals on the input tape is erroneous (Aho and Johnson 1974] Consequently, every entry in the parsing table corresponding to к terminals that maps to a shift or reduce operation indicates that the terminal is acceptable. This property allows us to use LR(1) parsing tables to efficiently compute accept sequences at any intermediate point, making them preferable for SYNCODE applications. Thus, computing acceptable terminals with LR(1) parsers has a complexity of O(T)). Although LALR(I) parsers are more commonly used due to their smaller memory requirements and faster construction, computing acceptable terminals with them requires iterating over all terminals leading to complexity of O(Tp (I')) due to the need for multiple reduce operations before confirming the validity of each terminal. Furthermore, while for и > 1, LR(k) parsers can compute accept sequences of length immediately, they incur extremely high memory requirements. Additionally, while we can use LL(K) parsing tables to compute the next & accept terminals, LR(x) parsers offer a higher degree of parsing power. Therefore, we employ LR parsers in SYNCODE. Our evaluation indicates that LR(1) parsers suffice for climinating most syntax errors, making them practical choice for SYNCODE. We discuss how the implementation of how parsing is performed incrementally to obtain the accept sequences and remainder in the Appendix A.3
14

---

Published in Transactions on Machine Learning Research (05/2025)
Accept Sequences: In our implementation, we focus generating accept sequences of length 1 or 2, they can be efficiently obtained from (R)(1) parser. While this approach incurs some loss of precision, it leads to sound but incomplete syntactical decoding. Further, our evaluation demonstrates that this strategy is efficient and precise in practical scenarios. We note that pmatch r.t with a 2-length sequence is equivalent to dmatch with a 1-length sequence, as stated in Lemma [ Consequently, in our work, we precompule mask stores Ma and Mr. On parsing the partial output Ck, the parser state of LR(1) parsers can be used to directly obtain syntactically acceptable terminals for the current completion (Ao) and the next completion (A1). We utilize Au and A1 to construct the accept
sequences sequences A, considering two cases:
Case 1: C2 = l1. ly: Let Tf represent the type of the final lexical token. In many instances, a token may be extended in the subsequent generation step, such as when an identifier name grows longer or additional words are appended to a comment. In those cases if A = T/. Ty, we include all 2-length sequences 7 for each i. As previously discussed, the type of the final lexical token may change from Ty. Consequently, when Au = {Tf. .2...., -9), we add 1-length sequences A, for each terminal sequence { from Ao; excluding Ty. This method ensures the generation ion of sequences accounting for potential extensions of the same token and changes in the type of the final lexical token.
Case 2 C2 =
In this scenario, the current terminal is incomplete, leading to lack of information about subsequent terminals. Consequently, when A = (2........ ,2,,...Tn,, we define A as a set of sequences: {Ai, A2, : where each A, corresponds to a single terminal sequence {Ti} from A Specifically, A1 = { A2- = {ra): and so forth.
4.6 Time and Space Complexity
In this section, we analyze the time complexity of the SYNCODE algorithm. We focus the cost of creating the mask at each iteration of the LLM generation loop. The key computations involved in this process are the parsing carried out by the incremental parser to compute A and the lookup/ cokup/unification /uniicctiion operations performed through the DFA mask store.
The incremental parser parses O(1) new tokens at each iteration and computes A. Let TA represent the time taken by the parser to compute the accepted terminals and Tp denote the time the parser takes parse a new token and update the parser state. Hence, in each iteration, the parser consumes O(Ta + Tp) time to generate A. The DFA mask store lookup involves traversing |A] DFA sequences, with the number of steps in this walk bounded by the length of the remainder As A can have maximun of T sequences, the DFA walk consumes O(T]) len(r)) time. We employ a hashmap to facilitate efficient lookups at each DFA node, ensuring that all lookups take constant time. Consequently, this step takes O(Ir)) time. Let To denote the time taken for computing the union of binary masks. With potentially [T] union operations to be performed, the mask computation lakes OTT, : (T]) time. Therefore, the overall time complexity at each step during generation is given by O(TA + Tp + [I'] (en(r) + TT--- -rT).
For an incremental LR(1) parser, the complexity of our algorithm at each step of LLM token generation is O(I len(r) + - (T]). With our lexer assumption, we ensure that the remainder r is small, allowing us to further simplify our complexity analysis to O(Tu (TD) by treating len(r) as constant.
Offline cost: The cost of computing the mask store Ma offline involves considering all DFA states g E QR, all possible terminal sequences of length a, and all tokens V. Given that we need to traverse the DFA for len(t) steps for each entry in the store, the time complexity for computing the mask store is (maater((en(())) (len())))QQlllllli Typically, len(t) is small, allowing us to simplify this to I(IIII)VV.. In our implementation, the use of Mo and Mj results in cost of O(IaalIVV))))) The size of [Qal depends on the complexity of regular expressions for the terminals, which may vary for each grammar. However, as demonstrated in our evaluation section, these mask stores can be computed within 10 minutes for each combination of grammar and LLM. This computation is a one-time cost that can be amortized over all generations performed for the given LLM and grammar.
Space complexity: The mask store Ma consists of masks of size V for each DFA state q E Qn and for all possible terminal sequences of length a. Consequently, the size of the mask store is Oi[Qal |VI- (TP).)))
15

---

Published in Transactions on Machine Learning Research (05/2025)
Output
Load unconstrained original nose!
Un Modell (model Pnicrosoft/phi-27
leport jur
pronet "Return 1son object to represent country India with nane, capital and population!" output Ine.in (propp))))pt))
www.eetttienttyy..o poory courtry Jefo "apptak"
SynCade
Output
Tree syncode Seport Syncode
11a Syncode(model "microsoft/phi 2", gramaar""Json)
Load the Syncode augaented code!
"nane"; *India", "pasulation"s "2,,55,427,754
prorpt
object represent country India
poputation?" output www.tta.inerrrreeeeeeeee (nn_rrapeeerr(rroppt)
Figure 9: The upper section displays erroneous output from a standard LLM generation, failing to produce the intended JSON format. The lower segment showcases the fix achieved through the of the SYNCODE framework.
In our implementation, the use of Mo and Mi reduces the size to O([Qa) |V]- [F]), In our evaluation, the size of the mask store is approximately 1-2 GB, as shown in Table
4.7 SynCode Framework
Figure g shows how SYNCODE framework can be used in practice by selecting grammar. We next discuss other important features of the framework.
Adding a New Grammar. Our Python-based SYnCODE framework is shipped with several built-in grammars such JSON, Python, Go, etc. A user can apply SYNCODE for arbitrary grammar by providing the grammar rules in EBNF syntax with little effort. The grammar needs be unambiguous LALR(1) LR(1) grammar for using the respective base parsers.
Ignore Terminals. Our EBNF synlax adopted from Lark allows one to provide ignore lerminals part of the grammar. Lark ignores those terminals while parsing. In the case of Python, this includes comments and whitespaces. SYNCODE handles these ignore terminals by adding a trivial 1-length accept sequence for each these ignore terminals.
Parsers. SYNCODE supports both LALR(1) and LR(L) as base parsers. We adapt Lark's (Lark] ] LALR(1) parser generator for SYNCODE. Since Lark does not implement the LR(1) parser generator, implemented the LR(1) parser generator top of the Lark. The generation of LR(1) parser which is performed offiine may take longer time compared to the LALR(1) parser (e.g., up to 2 mins for our Python grammar), however, it is more efficient at inference time in computing the accept sequences. Further, since the Lark-generated parser is non-incremental, we build the incremental parser on top of by caching the parser state as described in Appendix A.3
Non- Fragments of PLs. SYnCODE can handle non-context-free fragments of PLs, such inden- tation in Python and end-of-scope markers in Go. To support languages with indentation, such as Python and YAML, SYNCODE has mechanism that tracks acceptable indentation for the next token, effectively masking tokens that violate indentation constraints at a given point. This indentation constraint fcature can be enabled with any new grammar. Similarly, for handling other custom parsing rules beyond CFGs, users can add additional constraints to the generation by overriding specific SynCoDE functions. For instance, in Go, semicolons are optional and may be automatically inserted the end of non-blank lincs. Implementing such constraints in SyXCODE programmatically requires minimal effort. However, SYNCODE currently does not support enforcing semantic constraints. (e.g, if variable in program defined before it is used.)
16

---

Published in Transactions on Machine Learning Research (05/2025)
5 Experimental Methodology
Models. In our evaluation, we select a diverse set of state-of-the-art open-weight LLMs of varying sizes. Since closed-source LLMs, such as GPT-4 or Gemini, do not expose generation logits through their APIs, applying a constrained generation approach in SYNCODE is not feasible. Therefore, we focus enhanc- ing smaller, open-source models in our evaluation. We select the state-of-tho-art models Llama-2-7B- (Touvron et al., 2023b) and Gemma2-2B-4 (Team el al. 2024 for our JSON evaluation. For text-2-SQL generation experiments, we use 1lama-2-7B-chat,, Llama-3.2 3.2-1B, Alama-3.2-3B, and (emmma-2-2B-2 Fur- thermore, we chose models such as LLaMA-7B (Touvron et al. 2023ap, WizardCoder-1B (Luo et al. 2023 and CodeGen-350M (Nijkamp al.) 2023 for code completion.
Datasets. We focus evaluation on generating JSON, SQL, Python, and Go outputs. We choose JSON as it is supported by the baselines (Gerganov and et. al. 2024 Willard and Louf 2023), which allows to compare against them. We selected Python since it is extensively present in the training data employed for LLM training and fine-tuning. Conversely, we opted for Go due to its lower standard LLM accuracy and a relatively smaller presence the training data. We consider JSON-Mode- (NousResearch) 2024) dataset for text to JSON generation and HumanEval and MBXP (Athiwaratkun et al. 2023) dataset for evaluating Python and Go code generation. We display examples of prompts from these datasets in Appendix A.7
JSON-Mode-Evar (NousRescarch, 2024). It consists of 100 zero-shot problems. Each problem prompt follows the chat format with a system prompt specifying a JSON schema and a user prompt requesting the LLM to generate a JSON object that contains specified contents.
Spider text-2-SQL. Spider Yu et al. varying difficulty levels: easy (250), medium (440), hard (174), and extra hard (170).
2018 texi-to-SQL dataset consists of 1,034 problems of
Multilingual HumanEval (Athiwaratkun et al. 2023). is an extension of the original HumanEval collection (Chen et al. [Ll12,,2), which comprises 164 Python programming problems, include other languages like Go. Each problem in the dataset consists of a function definition, and text descriptions of the function as part of the function docstring.
MBXP (Athiwaratkun et al. 2023). It is extended from the MBPP (Austin et al. 2021 dataset for Python to support other languages such as Go. The dataset consists of 974 problems with the same format as HumanEval.
Grammars. For Python, we used the readily available grammar from the Lark repository. For Go, we converted an existing LL(*) grammar from (ANTLR, implementation to LR(1) grammar for our use. We write the CFG for these languages using the Extended Backus-Naur Form (EBNF) syntax. We use a substantial subset grammar for Python and Go synlactic generation with SYNCODE. The grammar has commonly used features of the language such as control flow, and loops, and excludes some features such Python's support for lambda functions. Adding support for more features would require more engincering effort but it will not change the overall technique. The grammars we used are available in Appendix A.8 The JSON grammar consists of 19 rules and 12 terminals. The Python grammar we used contains 520 production rules and 94 terminals, whereas the Go grammar comprises 349 rules and 87 terminals.
Evaluating Syntax Errors. For evaluating the errors in the generated output in each of the languages, we use their respective standard compilers.
Experimental Setup. We run experiments on 48-core Intel Xeon Silver 4214R CPU with NVidia RTX A5000 GPUs. SYNCODE is implemented using PyTorch (Paszke et al. 2019a). HuggingFace transformers library Wolf et al. 2020 and Lark library [Lark]
v0.1.1, GUID- ANCE Lundberg et al. 2023) v0.1.16 and GCD (Geng et al. 2023) in our study. The algorithmic differences in the basclines and SYNCODE are discussed in Section 7 We perform warmup run for each experiment where we measure inference time to ensure that one-time precomputation time not included in the infer- ence runtime. For fair comparison with baselines, SYNCODE uses opportunistic masking (Beurer-Kellner
Baselines. We evaluate three state-of-the-arl baselines OUTLINES Willard and Louf 2023
17

---

Published in Transactions on Machine Learning Research (05/2025)
et al. 2024), an optimization used in LLAMA.CPP and GUIDANCE. Instead of computing the full logit vector mask upfront, the model generates token and only computes the mask if the proposed token is incorrect.
6 Experimental Results
In this section, we evaluate SYNCODE on generating various formal languages. We compare SYNCODE with state-of-tho-art basclines and perform various ablation studics.
SYNCODE allows the model to generate a special EOS token (indicating the end of generation) only when the output belongs L(G). In practice, however, LLM generation typically stopped after fixed maximum number of tokens, 7mmor- Therefore, terminating with the EOS token within this limit not always guaranteed potentially resulting in syntax errors.
6.1 Effectiveness of SynCode for JSON Generation
Effectiveness of SYNCODE in generating JSON with original and explicit prompts. Column gener- ation time (with standard deviation of the mean) in seconds.
Table 1:
Model
Tool
Syntax Errors
Validation Accuracy (%) Original
Generation Time (8)
Original
Explicit
Explicit
Original
Explicit
SynCode
0
66%
84%
3.08 10.1
3.04
20.06
Standard
41
2%
58%
3.61
3.15
Llama-2-7B-chat
GUIDANCE
13
11
57%
63%
5.14 #0.16
4.22
=0.08
OUTLINES
14
62%
56%
38.07 20.27
41.79 20.2
GCD
2
62%
64%
6.08 10.3
4.01
SynCode
100%
20.00
20.05
Standard
59
59
41%
41%
4.32 20.16
5.82 10.06
Gemma2-2B-it
CUIDANCE
96%
96%
6.09
+0.29
5.56
#0.24
OUTLINES
2
67%
90%
1.99
2.75
GCD
96%
95%
19.12
20.08
8.82
20.13
We observed issucs when using Llama-2-7B-chht with Outlines v0.1.1 and therefore, we older version v0.0.46.
We evaluate the effectiveness of SYNCODE in guiding LLMs with the JSON grammar to generate syntactically correct JSON. We run the inference with Llama-2-7B-on and Gemma2-2B-it with SYNCODE, OUTLINES, GUIDANCE, GCD, and standard generation the 100 problems from the JSON-Mode-Foo dataset. We select these models for the JSON experiment they supported by all considered basclines.
We set max new tokens Amar = 400. We also report an evaluation of augmenting the prompts with an explicit request to output only JSON. We present an example of these explicit prompts in Appendix [A.7] We evaluate the correctness of JSON generated by an LLM by first evaluating whether the JSON string can be parsed and converted to a valid JSON object. We further evaluate whether the generated JSON is valid against the schema specified in the prompt. Although the SynCODE does not enforce the specific schema the JSON output for each task, believe it is important rescarch question to check whether the reduced syntax errors due to SyNCODE can also lead to improved schema validity.
Table [ presents our evaluation results. We report results for both the prompts taken directly from the dataset (denoted as 'Original') and after augmenting these prompts with an explicit request to output JSON (denoted as "Explicit"). In the 'Validation Accuracy" column, we compute the percentage of valid completions against their respective schemas. In the 'Generation Time (s)" column, we report the average time taken to generate a completion to a prompt from the dataset. Guiding Llama-2-7B-chat and Gemma2- 2B-it it with the JSON grammar via SYNCODE eliminates syntax errors in generated JSON. On the other hand, standard generation results in syntactically incorrect JSON for 98% and 59% of completions to the original prompts for the Slama-2-BB-hhal and Gemma2-2B-ir models respectively. A majority of these errors are due to the generation of nat ural language before and after the JSON. Explicit prompts somewhat mitigate this issue, but still results in syntactically invalid outputs to 41% and 59% of these prompts for standard Llama- 2-7B-chat and Gemma2-2B-it generation respectively, primarily due to crrors such as unmatched braces and unterminated string literals. OUTLINES, GUIDANCE, and GCD face similar problems with closing braces and terminating strings.
18

---

Published in Transactions on Machine Learning Research (05/2025)
Notably, SYNCODE significantly improves the JSON schema validation accuracy of Gemma2-2B-it com- pletions over standard generation, from 41% to 99% and 41% to 100% for original and explicit prompts respectively. Furthermore, SYNCODE outperforms GUTLINES,GUIDANC and GCD in validation accuracy of Llama-2-7B-chat completions by 4%, 9%, and 4% respectively for original prompts and 28%, 19%, and 20% for explicit prompts. The remaining schema validation errors with SYNCODE are semantic errors, in- cluding data type mismatch between the generation JSON and schema, missing fields required by the schema, and adding extra fields not allowed by the schema. SYNCODE is faster than all baseline grammar-guides generation methods for -lama-2-BB-che
and all but OUTLINES for Gemma2-2B-it. The low generation time can largely be attributed to the fact that many of its completions to
with OUTLINES for Tlama-2-7B-chee prompts are empty JSON (35% of original and 7% of explicit) which takes few tokens to generate but often does not conform to the schema.
Interestingly, we observe that for Llama-2-7B-chat, SYNCODE also reduces the average generation time over standard generation. We attribute this finding to the fact that without grammar-guide generation, the model generates syntactically invalid output, such natural language, in addition to JSON and thus generates more tokens in response to the same prompt than with SYNCODE. Thus, augmenting LLMs with SYNCODE can significantly improve syntactical correctness and runtime efficiency.
6.2 Effectiveness of SynCode for SQL Generation
This study demonstrates that SYNCODE improves text-to-SQL generation by enforeing grammar constraints, ensuring that generated SQL queries are syntactically accurate. We mcasure the execution accuracy of generated SQL queries using tests provided in Spider Yuu et al. 2018 benchmark. We evaluate the following models for SQT generation: Tlama-3.2-1B, T.lma-3.2-3B (base models) and Tlama-2-7B-chatt Gemma-2-2FB it (instruct-tuned models). We observe that despite explicitly prompting to only generate the SQL query, the instruct-tuned Cemma-2-2B-in model often enclosed generated SQL queries within markers, such >ssl Thus, we consider another baseline for Gemma-2-2B where we extract the SQL query
substring within these markers, handling cases where the output format is either -- "SQL query) or s= (SQL query?
For evaluation, we use the Spider (Yu et al. 2018) text-to-SQL dataset, which consists of 1,034 problems of varying difficully levels: easy (250), medium (440), hard (174), and exlra hard (170). We prompt models with schema information and text queries, instructing them to generate SQL queries only. Using greedy decoding and \n|n is used as an additional stopping condition for all experiments.
Table 2: Comparison of SyNCODE and unconstrained generation on SQL generation.
Accuracy (%)
Model
Method
Execute (%)
Tokens
Time (s)
Easy
Medium
Hard
Extra
Overall
Standard
0.0
0.0
0.0
0.0
0.0
0.0
221.43
9.883
Geuma2222.i
Standard+
44.8
18.9
21.9
17.1
25.4
77.6
221.43
9.893
SYNCODE
45.8
18.7
20.6
78.2
135.17
5.876
Standard
34.4
22.0
12.1
4.1
20.4
32.6
44.74
1.148
Llama-2-7b-chat
SynCoue
40.0
27.3
13.8
5.9
21.6
41.6
50.33
1.483
Standard
40.8
24.8
20.7
10.6
25.6
51.1
48.00
0.509
Llama-3.2-1
SynCode
16.8
28.2
23.0
10.6
28.8
59.0
56.36
0.916
Standard
38.0
29.5
28.2
12.9
28.6
67.4
47.78
0.846
1lama-3.2-3B
SynCode
47.2
34.8
32.8
19.4
34.9
81.4
47.63
1.161
Table 2 presents a comparison of SYNCODE and unconstrained generation across key metrics. The Accuracy (%) column shows the percentage of correctly generated SQL queries across different. difficulty levels. Execute (%) reflects the percentage queries successfully executed without runtime errors SQLite. The Tokens column reports the average number of tokens generated, and Time(s) shows the average generation time. Standard+ row for Gemma2 denotes the result for the additional baseline where we extract the SQL query from the full generation using regex matching.
19

---

Published in Transactions on Machine Learning Research (05/2025)
Table Comparison SYNCODE with OUTLINES generation on SQL generation.
Model
OUTLINES
SYNCOPE
Accuracy
Avg
Time (5)
Accuracy
Avg Time (8)
Llama-3.2-1B
0.70
Llama-3.2-3B
0.17
11.16
0.31
1.03
Gemma-2-2b-it
0.21
45.82
0.25
5.81
We observe that SYNCODE achieves better performance over the basclines in terms of both exccution percent- age and execution accuracy. For example, with the Llama-3.2-3B model, SYxCODE achieves an execution success rate of 81.4%, compared to 67.4% for unconstrained generation. Further, the execution accuracy improves from 28.6% to 34.9%. In the case of the Gemma2-2B-it model, we observe that SYnCODE shows moderate improvement over the Standard+ accuracy. However, it shows significant gain in the speed (1.7x) of generation and a reduction in the number of tokens generated. Although the Gemma2-2B-ii model has a good execution percentage without any runtime errors. The instruct-tuned models tends to use large number of tokens that are not part of the query. In applications where the goal is to use LLMs to generate SQL queries without additional explanations, the result with Gemma2- 2- shows that SYNCODE is useful in improving the efficiency of LLM generation along with the improvements in accuracy.
Comparison to Outlines: We perform additional experiment on the first 100 exampels in the Spider dataset on the task of SQL generation. Table 8 presents the accuracy and Avg. time taken for various models. Our results show that SYNCODE is significantly faster than OUTLINES in SQL generation.
6.3 Effectiveness of SynCode for GPL
Table 4: Number of programs with syntax errors for standard and SYNCODE generation (1 shows how much SYNCODE reduces the occurrence of the syntax errors compared to Standard generation.
Dataset
Model
Python SYNCODE
Go
Standard
Standard
SynCODE
CodeGen-350M
271
15
95%
573
49
91%
HumanEval
WizardCoder-1B
36
92%
1031
50
95%
LLaMA-7B
291
99%
725
10
90%
CodeGen-350M,
212
99%
MBXP
WizardCoder.11B
28
2
93%
243
14
94%
LLaMA-7B
148
97%
414
1
99%
We run inference with CodeGen-350M, WizardCoder-1B. and LLaMA-7B with SYNCODE and with the standard no- no-masking approch. We do not compare SyxCODE with the other baselines as none of these works support genoral-purpose programming language grammars. We experiment with both Python and Go programming languages, evaluating performance on zero-shot problems from the HurnaEvval and MBXP datasets. For each dataset, we generate n 20 and n samples per problem with the LLM, respectively, We run the LLM-generated code completion against predefined set of unit tests. For cach unit test, we record the error type when running the generated program against that test case. We use the hyperparameters temperature = 0.2 and top p 0.95. Table presents our results for Python and Go. The columns standard and SYNCODE represent the total number of generated programs with syntax errors for the respective approaches. The column + designates the percentage reduction in syntax errors from the standard generation to the SYNCODE generation. In this evaluation, across both HumanFval and MBXP datasets, we generate a total of 4154 samples for cach language. On average, of all standard generated samples, 6% and 25% have syntax errors for Python and Go, respectively.
Notably, our experiments reveal that SYNCODE reduces the number of syntax errors by over 90% over the bascline in most experiments. Morcover, SYNCODE reduces the number of syntax errors to less than 1% of the total samples. Interestingly, we observe significantly more Syntax errors in standard LLM-generated Go code than in Python code, likely because the LLMs are trained more extensively on Python code than Go. Thus, SYNCODE can be especially effective for Go and more underrepresented programming languages, where LLMs are more likely to generate syntax errors due to a limited understanding of the language. SYNCODE can bridge this gap by guiding the LLM to sample only the syntactically valid tokens during decoding.
20

---

Published in Transactions on Machine Learning Research (05/2025)
Table Functional correctness on HumanEval problems
Metric
Architecture
Python
Go
Standard
SYNCoDE
Standard
SYNCODE
CodeGen-350M
6.8% +0.1
6.9% +0.1
3.6% +0.0
3.6%
+-0.0
passib1
WizardCoder-1B
20.0% +0.2
20.0%
#0.3
9.37 40.3
9.5%
4:0 2
LLaMA-7B
11.2%
11.5%
+0.3
3.8% +0.2
4.25%
+0.2
CodeGen--30M
10.4%
1:.3
10.0%
#0.3
5.6%
20.2
6.1%
#0.3
passt11
WizardCoder-1B
27.6%
28. 18.4%
+0.7
12.5%
+0.3
13.7%
+0.4
LLaMA-7B
17.1% 49.4
18.9% #0.5
8.8%
40.3
8.8%
+0.3
Table 6: DFA Mask store creation time and memory
Python
Go
Model
[V]
Time(s)
Memory
Time(s)
Memory
CodeGen-350M
51200
602.26
1.87BB
603.03 03
1.58GB
WizardCoder-IB
49153
588.28
183GB
588.84
1,54GB
LLaMA-7B
32000
382.26
1.17GB
380.19
1.06GB
We further analyze the errors in Python and Go code generated by the LLMs augmented with SYNCODE, example of which is presented in Appendix A.6 All of the errors were because the LLM failed to generate complete program within the maximum token limit. Recall, SYNCODE provides guarantees of completeness for syntactically correct partial programs. However, it does not guarantee convergence to a syntactically correct and complete program.
Functional Correctness for Code Generation. We investigate whether augmenting L.MSs with SYN- CODE improves the functional correctness of the generated code. We evaluate functional correctness using the pass@k metric, where l samples sample passes set of unit tests, and the fraction of solved problems is calculated. Table 5 reports results for pass@1 and pass@10 for generated code completions to problems from the Human dataset. We observe that augmenting LLMs with SYNCODE has a slight improvement in functional correctness over standard generation. This observation indicates that for these state-of-the-ar models, syntactic correction can result in a small improvement in the logical correctness of the code.
generated per problem, and problem is considered solved if any
6.4 Mask Store Overhead
We analyze the time and memory overhead involved in generating a DFA mask store using SYnCODE. The DFA mask store for Llama-2-7B-chat took 113.72 seconds to create and consumes 181 MB of memory. Additionally, we report the creation time and memory overhead of DFA mask stores for models used for Python and Go in Table 6 Each row shows the SYNCODE store generation time in seconds, and memory in GBs, for a particular LLM and grammar. The [V] column represents the total vocabulary size of the tokenizer of the particular LLM. We see that generating the store requires less than 2GB of memory and several minutes across the evaluated models and grammars. This overhead is minimal for practical SYNCODE use cases, as the mask store a one-time generation task. Thereafter, the mask store can be efficiently loaded into memory and used for repeated inference. We see smaller mask store generation time and memory with Llama-2-7B-che and JSON grammar with 18 terminals opposed to LLaMA-7B, WizardCoder-1B and CodeGen-350M with Python and Go graminar with 94 and 87 terminals respectively since the size of the mask store is proportional to the number of terminals in the grammar.
7 Related Work
Our work locuses on enhancing the syntactical accuracy LLMs by using a constrained decoding algorithm. Prior research has explored two other primary directions to enhance LLMs' accuracy in generating formal
language: 1) Fino-tuning or prompt engincering (Bassamzadch and Methani 2024 Weyssow et al.) 2024).
which demands substantial data, compute resources, and time, often without any formal guarantecs. 2
Modifications the L.L''s architecture tokenization (Murty et al. 2023 Dong al. 2023 Zhu et al.
21

---

Published in Transactions on Machine Learning Research (05/2025)
Table 7: Overview of various constrained decoding methods
Regex
CFG
Precompated
GPL.
Max CFG
Input format
LMQL Beurer-Kellner et al.
2023
X
x
50-100
LMQL DSL
GUIDANCE Lundberg al.
2023
/
X
50-100
Python DSL
OUTLINES
Willard and Loul
2023
/
/
50-100
Lark EBNF
PICARD Scholak
2021
/
X
50-100
Haskell
SYNCHROMESH (Possa
2022
/
X
$
ANTLR
1.AMA.CPP (Gerganov and et.
2024
/
X
50-100
GBNF DST,
GCD Geng cl al. 2023
50-100
GF GBNF DSL
DOMINO SScurer-Kellnde al. 2024
50-100
SYxCODE (ours)
V
/
500
Lark EBNF
t Implementation issucs $ Syuchromesh is closed-source and the information about DSL grammars is unavail-
able
GF: Grammatical Framework, GBNF is DSL defined by LLAMA
2024), although these techniques have not yet achieved performance comparable to the current state-of- the-art standard LLMs. However, both fine-tuning and architectural changes are complementary to the grammar-guides decoding approach that focus in our work, and any gains through those techniques will improve the overall quality of LLM generation.
There are several recent works on constrained LLM generation (Wei et al.
2023
Beurer-Kellner et al. 2023
Lundberg et al., 2023: Willard and Loul]
2023 Scholak et al. 2021 Poesia et al.
2022
Gerganov and el. al.
2024 Geng et al. 2023 Heurer-Kellned et al. 2024 Agrawal et al.
2023 Melcer et al., 2024). This includes
recent works that have used language-scrver (tools built for communication between IDEs and programming
language-specific tools like static analyzers and compilers) suggestions to enforce language-specifie semantic
2023 Wei et al. 2023). These techniques do not guarantee syntactical accuracy and rely on the availability and efficiency of language servers. More recent works such
constraints during decoding (Agrawal et al.
as (Ugare et al. 2025 Banerjee et al.
2025
Suresh et al. 2025 were built on top of SYNCODE.
Structured LLM Generation. We focus our further discussion on comparison to the techniques that constrain LLM for structured generation according to a formal language. We compare SYNCODE with prior works in terms of precision and efficiency of the algorithms and generality and scalability of frameworks. Table 7 presents the various recent techniques for structured LLM generation. The columns 'Regex' and "CFG' indicate regular expression and CFG constraining features, respectively. The 'Precomputed' column denotes techniques that precompule certain structures to enhance generation efficiency. The 'GPL' column specifies if the tools support general-purpose PLs. "Max CFG' displays the number of production rules in the largest supported Grammar by these tochniques. We obtained these numbers by examining the built-in gramunars that were provided in the correspouding libraries. Finally, the "Input Format' column indicates the format used to specify generation constraints. In addition to the improvement over the baselines presented in the evaluation, our work focuses on rigorously formalizing the correctness of our CFG-guided generation approach.
Recent works such as GUIDANCE [Lundberg et al. 2023) and LMQI. [Beurer-Kellner et al. 2023) mitigate the unpredictability of LLM responses by using template or constraint- based controlled generation techniques. These libraries feature a templating engine where prompts are expressed with holes for the generation to fill. IMQL. Kellner et al.
2023) supports general regular expression constraints, but not. CFG constsaints.
GUIDANCE (Lundberg et al. (023) supports CFG-guided generation. It uses Earley parsing (Earloy) 1970)
for constrained decoding. Similar to other related works, it incurs high inference overhead
it checks the
syntactical validity of the entire model vocabulary at each step. It uses trie similar to (Poesia et al.) 2022 Willard and Louf, 2023 Beurer-Kellner et al. 2024). As shown in our evaluation it incurs higher overhead for JSON generation than SYNCODE. It iterates over the vocabulary in order of the next token probability to efficiently compute the next token. However, this leads to a lack of generality and it cannot be directly combined with an arbitrary decoding strategy.
OUTLINES WWillard and Louf, 2023) is library originally focused on regular expression-guidel generation and recently extended to support grammar-guided generation. During LLM generation, OUTLINES employs
22

---

Published in Transactions on Machine Learning Research (05/2025)
incremental Lark-based LALR. parser to determine the next acceptable terminals based on the grammar. Its approach differs from SynCode in how it processes acceptable tokens: Outlines computes an expensive union of regular expressions from all terminals during inference, converting this into a DFA, then validates tokens against this combined structure. In contrast, SYNCODE treats each sequence of acceptable terminals separately during decoding steps, avoiding the costly union operation. As shown in our evaluation (Section 6.1 and Section 6.2 SyNCODE performs better than OUTLINES on generating with JSON and SQL grammar and it currently lacks support for large GPL grammars.
LPAMA.CPP (Gerganov and et. al. 2024), has also recently introduced support for grammar guudded gener- ation. This approach models a nondeterministic pushdown automaton with N stacks to maintain possible parse states. LLAMA.CPP defines a new grammar syntax and implements a simplified basic parser in C+++ While this implementation in C++ reduces some parsing overhead compared to heavier LR(1) parsers im- plemented in Python on top of Lark for SYNCODE, it is algorithmically inefficient. This inefficiency again is due to the requirement iterate over the entire vocabulary and update stack states during inference. Moreover, the non-standard grammar synlax and limited support for general grammar fealures restrict its evaluation to simpler grammars such as JSON. We anticipate that LLAMA.CPI CPP and OUTLINES would perform even slower on grammars with more rules, terminals, and complex regular expressions, such as those found in Python and Go. As shown in our evaluation, SYNCODE is more efficient and results in fewer syntax errors.
SYNCHROMESH [Poesia et al. 2022 is a proprietary a tool from Microsoft that supports CFG-guided syn- factic decoding of LLMs. Similar to OUTLINES, it creates a union of regular expressions of terminals during LLM generation. Further, Synchromesh uses a non-ineremental parser for parsing. Both of these lead to lower time complexity. Synchromesh uses techniques like Target Similarity Tuning for semantic example selection and Constrained Semantic Decoding to enforce user-defined semantic constraints and works on DSLs. In contrast, our work, SYNCODE focuses exclusively on syntactic generation.
PICARD (Scholak et al. 2021 uses a specific decoding strategy that maintains a beam of multiple candidate outputs and promptly rejects the candidates that violate the syntax. It utilizes an incremental monadic parser and was developed specifically to support SQT. generation. Introducing a new grammar into PICARD necessitates considerable effort, as it lacks support for a grammar-defining language to provide grammar rules.
Recent work Domino (Beure-KKllner et al., 2024) provides CFG-guided LLM generation, sharing concep- tual similarities with SYNCODE while differing in key technical aspects. Domino avoids traversing the entire vocabulary during inference by precomputing a prefix tree corresponding to cach NEA state of the gram- mar's terminals. This structure serves a purpose similar to SYNCODE's DFA mask store, but we believe our approach offers superior efficiency. SYNCODE's mask store leverages boolean mask operations that can be performed highly efficiently on modern hardware architectures, particularly GPUs, where parallelized bitwise operations provide significant performance advantages (Paszke et al., 2019b). This architectural difference contributes to SYNCODE's exceptional inference speed compared to alternatives. Domino de- fines the minimally invasive property, which is cquivalent to SYNCODE's soundness property. However, a critical distinction between the two approaches lies in their approximation strategies: Domino applies an ander-approximation approach, permitting only tokens that precisely align with the parser's lookahead, while SYNCODE adopts a conservative over-approximation approach, allowing tokens as long as their prefixes match the parser lookahead. This fundamental difference has important theoretical implications. Due to Domino's under-approximation strategy, it requires do parser lookahead to achieve soundness guarantees, whereas SYNCODE ensures soundness for any lookahead depth. This gives SYNCODE theoretical advantage in providing stronger correctness guarantees without the computational costs of extensive lookahead. Fur- thermore, SYNCODE demonstrates superior scalability in handling complex grammars. The largest grammar that Domino currently supports is a highly simplified C grammar with approximately 70 rules, incurring roughly 25% overhead. In contrast, SYNCODE efficiently handles substantially more complex gramunars with lower computational overhead, as demonstrated our experimental results. Unfortunately, Domino's imple- mentation code is not publicly available yet, preventing direct experimental comparison with SYNCODE. Nevertheless, our theoretical analysis and empirical results with other baselines strongly suggest that SYN- CODE represents a significant advancement in CFG-guided LLM generation, combining stronger theoretical guarantees with practical efficiency.
23

---

Published in Transactions on Machine Learning Research (05/2025)
Fixed Schema Generation. Many recent works perform constrained LLM decoding to ensure that the generated output follows fixed schema of JSON XML (Zheng et al. 2023 Beurer-Kellner et. al. 2024 Willard and Louf 2023 Sengottuvelu and et. al. 2024). When employing a fixed schema, many intermediate points in the generation process offer either a single syntactical choice (c.g., key in the JSON schema) or present only a handful of distinct options. In cases where only one choice exists, the generation of the next token through the LLM can be entirely skippod. Alternatively, when there are multiple but limited choices, techniques like speculative decoding can be used to expedite the generation process (Chen et al, 2023 Leviathan et al. 2023). SYNCODE does not focus generation problems with fixed schema, it solely focuses on CFG-guided generation. We made the same observation as in (Beurer-ellner et. al. 2024 techniques such as speculation are not useful for CFGs where the schema is not fixed.
8 Conclusion
Existing methods for guiding LLMs to produce syntactically correct output have been notably slow and restrictive. In this paper, we present SYNCODE, an efficient and general framework to enhance LLMs' ability to generate syntactical output for various formal languages. During decoding, SYNCODE incrementally parses the partially generated oulput, computes the unparsed remainder and acceptable terminal sequences, and then leverages the remainder, accept sequences, and pre-computed DFA mask store to compute a mask to constrain the LLM's vocabulary to only syntactically valid tokens. We evaluated SYNCODE on generating syntactically correct JSON, SQL, Python, and Go code with different combinations of datasets, models, and tasks. SYNCODE eliminates syntax errors JSON completions and significantly improves JSON schema validation over the baselines. Furthermore, SYNCODE reduces the number of syntax errors in generated Python and Go code by 96.07% on average compared to standard generation. We believe that our approach will pave the way for more efficient and higher-quality structured LLM generation in real-world applications.
References
Lakshya A Agrawal, Aditya Kanade, Navin Goyal, Shuvendu K. Lahiri, and Sriram K. Rajamani. 2023. Guiding Language Models of Code with Global Context using Monitors. arXiv:2306.1078 [cs.CL]
A. V. Aho and C. Johnson. 1974. LR. Parsing, ACM Comput. Surv. 6, 2 (jun 1974), 1241 https:
7/doi. org/oo/ 10. 1145/3566288 356629
Alfred V. Aho, Ravi Sethi, and Jeffrey D. Ullman. 1986. Compilers, Principles, Techniques, and Tools. Addison- Addison-Wesley.
ANTLR. -. ANother Tool for Language Recognition.
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ra- manathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. 2023. Multi-lingual Evaluation of Code Generation Models. arXiv:2210.1433 [cs.LG]
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.0773 PIs.
Debangshu Banerjee, Tarun Suresh, Shubham Ugare, Sasa Misailovic, and Gagandeep Singh. 2025. CRANE: Reasoning with constrained LLM generation. arXiv:2502.09061 [Cs.PL] https://arxiv. http:::/aaxxvvorr/aaaaa 2502.
oww//ogg/////./oooooooe//.../. /beg/abb/200 222/22..
09061
Nastaran Bassanzadch and Chhaya Methani. 2024. Comparative Study of DSL Code Generation: Finc-
Tuning vs. Optimized Retrieval Augmentation. arXiv:2407.02742 [cs.SE] https://arxx http:///aaxxv.or/a/////0/// org/abs//407,,
02742
24

---

Published in Transactions on Machine Learning Research (05/2025)
Luca Beurer-Kellner, Mare Fischer, and Martin Vechev. 2023. Prompting Is Programming: A Query Lan- guage for Large Language Models. Proc. ACM Program. Lang. 7, PLDI, Article 186 (jun 2023), 24 pages.
https://boi.orr//0.. 1tt.///0/0g/0.11//8813330000 1145/3591000
Luca Beurer-Kellner, Mare Fischer, and Martin Vechev. 2024. Guiding LLMs The Right Way: Fast, Non- Invasive Constrained Generation. arXiv:2403.6698: [cs.LG]
Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. 2020. On the Ability and Limitations of Transformers to Recognize Formal Languages. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), Bonmie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association https://doi. org/10 10. 18653/v1/2020 emnlp-manlee..eeeee main.
for Computational Linguistics, Online,
576
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:2005.144 14165 [cs.CL]
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating Large Language Model Decoding with Speculative Sampling. ArXiv preprinl (2023).
Mark Chen, Jerry Tworek, Hcewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrrov Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Fe- lipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantamu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained Code. arXiv:2107.0331 [es.T.G]
Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A. Ortega. 2023. Neural Networks and the Chomsky Hierarchy. arXiv:2207.n .00098
[es.LG] http::://aartv..r/bbbbs////
02098
Yihong Dong, Ge Li, and Zhi Jin. 2023. CODEP: Grammatical Seq2Seq Model for General-Purpost Code Generation (ISSTA 2023). Association for Computing Machinery, New York, NY, USA, 188 198. https: //doi. 100 0et...11553399966663830 3598444
Jay Earley. 1970. An efficient context-free parsing algorithm. Commun. ACM 13, (feb 1970), 94-102. 1ttp://00.111////0.03322222
Javid Ebrahimi, Dhruv Gelda, and Wei Zhang. 2020. How Can Sell-Attention Networks Recognize Dyck-n Languages?. In Findings of the Association for Computational Linguistics: EMNLP 2020, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 4301-4306.
https:
//doi 1805:// findings-ennlp..ee
Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. 2023. Grammar--onssrrinned decoding for structured NLP tasks without finetuning. In Proc. of EMNLP.
Georgi Gerganov and ct. al. 2024. laama.cpp cpp: Port of Facebook's LLaMA model C/C++. https:
//gthhb.. hatam/ggidaaneeaaggueeeeee
Michael Hahn. 2020. Thcoretical Limitations of Self-Attention Neural Sequence Models. Transactions
of the Association for Computational Linguistics 8 (Dee. 2020), 156-171. https://do1.org/10..111
tacl 00306
25

---

Published in Transactions on Machine Learning Research (05/2025)
Lark. Lark parsing toolkit for Python.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast Inference from Transformers via Speculative
Decoding, arXiv:2211.17199 [cs.LG]
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Decpak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Cc Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hud- son, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaud- hary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2023. Holistic Evaluation of Language Models. arXiv:2211.09110 [cs.CL]
Scott Lundberg, Marco Tulio ArXiv preprinteia Ribeiro, and et. al. 2023. Guidance-A/Guidddneee A Guid- ance Language for Controlling Large Language Models. https://gi wwt.:/cic//ccccceeee/eo al/ guidance
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. WizardCoder: Empowering Code Large Language Models with Evol-Instruct. arXiv:2306.0855 [cs.CL]
Daniel Melcer, Nathan Fulton, Sanjay Krishna Gouda, and Haifeng Qian. 2024. Constrained Decoding for Fill-in-Uhe-Middle Code Language Models via Efficient Left and Right Quotienting of Context-Sensitive Grammars. arXiv2440..17988 (cs.PT.) https:///rxxv.org/ abs///bs..2222/////////////aa 17988
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023. Augmented Language Models: Survey. arXiv:2302.07842 [cs.CL]
Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D Manning. 2023. Pushdown Layers: Encoding Recursive Structure in Transformer Language Models. In The 2023 Conference Empirical Methods in Natural Language Processing. httpps://ppenrevier http://wwww....eeeeoooorrddddaae
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An Open Large Language Model for Code with Multi-Tury Program Synthosis.
arXiv:2203.13-17 [cs.LG]
NousResearch. 2024.
json-mode-eval.
Theo Olausson, Alex Gu, Ben Lipkin, Cedegao Zhang, Armando Solar-Lozama, Joshua Tenenbaum, and Roger Levy. 2023. LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
v1/2023. wwn.p-meeaaaaaeeee 313
OpenAL. 2024. OpneAl Tools.
https://plaaforr.oppeee http:/tffooooomm.a....mm apppnaa..cc//ccccs//ssssttttssts
Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023. Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning. arXiv:2305.12295 [cs.CL] https:
//arxiv. abs/2305 12295
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVilo, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019a. PyTorch: An Impera- tive Style, High-Porformance Deep Learning Library. In Advances in Neural Information Pro- cessing Systems 32. Curran Associates, Inc., 8024-8035. http://papers.neuu meurips.cc/papee. cc/paper/ learning-----nn------ Mibrarr .pff
26

---

Published in Transactions on Machine Learning Research (05/2025)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019b. PyTorch: An Imperative Style, High- Pegh-Peeffrmance Deep Learning Library. arXiv:1912.01703 [cs.T.G]
Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gul- wani. 2022. Synchromesh: Reliable Code Generation from Pre-trained Language Models. In International Conference on Learning Representations. htttps://ppereeval net/foorumidd
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. 2018. Meta-Learning for Semi-Supervised Few-Shot Classification. arXiv:1803.00676 [cs.LG]
Torsten Scholak, Nathan Schucher, and Damitry Bahdanau. 2021. PICARD: Parsing Incrementally for Constrained Auto-Rogressiv Decoding from Language Models. In Proceedings of the 2021 Conference on Empirical Melhods in Natural Language Processing, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 9895-9901. httpp:///abl.org//...16/////////
Rahul Sengottuvelu and al. 2024. jsonformer. https://gtthub..co///gg///
Tarun Suresh, Debangshu Banerjee, Shubham Ugare, Sasa Misailovic, and Gagandeep Singh. 2025. DINGO; Constrainod Inference for Diffusion LLMs. arXiv:2505.2306 [cs.LG] attps://arxiv.ort /a////bs/22000 2505.
23061
Gemma Team, Morgane Ríviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, An- lon Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Sean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Ali- cia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette- Choo, Danila. Sinopalníkov, David Weinberger, Dimple Vijaykumar, Dominika Rogozitiska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Dhand, Ivan Nardini, Jacinda Mcin, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joans Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gor- don, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kil- patrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Tverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsce Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wemming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joclle Barral, Zoubin Ghahramani, Raia. Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin,
Plucciskk- Harleen Batra, Harsh
27

---

Published in Transactions on Machine Learning Research (05/2025)
Kathleen Kenealy, Robert Dadashi, and Alek Andreev. 2024. Gemma 2: Improving Open Language Models at a Practical Size. arXiv:2408.00118 [es.CT]
http://arxii..orraa/////////////
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a.
LLaMA: Open and Efficient Foundation Language Models.
arXiv:2302.1397 [cs.CL]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Tgor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Nlyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.0925 [cs.CL]
09288
Shubham Ugare, Rohan Gumaste, Tarun Suresh, Gagandeep Singh, and Sasa Misailovic. 2025. IterGen: Iterative Semantic-aware Structured LLM Generation with Backtracking. arXiv:2410.07295 [cs.SE]
https://arxivv httpa/////axxvooo/aa///////s 07295
Yuxiang Wei, Chunqiu Steven Xia, and Lingming Zhang. 2023. Copiloting the Copilots: Fusing Large Language Models with Completion ion Engines for Automated Program Repair. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 223. ACM. https://doi.
1145/111111111 3611664.33366777
Martin Weyssow, Xin Zhou, Kisub Kim, David Lo, and Houari Sahraoui. 2024. Exploring Parameter- Finc-Tuning Techniques for Code Generation with Large Language Modols. arXiv:2308.104m [cs.SE]
https: //arxiv.../vv http:///aaarvv.oraaaaa//aaaa wtwg/ba////a////eeee 2208. 10462
Brandon T. Willard and Rémi Louf. 2023. Efficient Guided Generation for Large Language Models.
arXiv:2307.0080 09702 [Ls
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Process- ing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Qun Liu and David Schlangen (Eds.). Association for Computational Linguistics, Online, 38 45. https://doi. http::///ooooo/11///// org/10. 10. 11885//v///00/ emnlp camlp-demos.@ 6
wolfram. 2024.
http://mrrtting.ssseeoooaoaaaat c /ttp:///oo1rrancc.m222///////// volframalphas thee-s-ss--tttt to--toobi tor-tttobiingg ing- wtwp:putttttiaaannooleeeeeeeeeee www.blddgeeupprppeerrr--oooaaaae
Wolfram Alpha.
Yuhuai Wu, Albert Qiaochu Jiang, Wenda Li, Markus Norman Rabe, Charles E Staats, Mateja Jamnik, and Christian Szegedy. 2022. Autoformalization with Large Language Models. In Advances in Newral Information Processing Systems, Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho
(Eds.). https://openreviii wttp://wwww....aeeeeoeeoooeoeea.
Andy Yang, David Chiang, and Dana Angluin. 2024. Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languages. arXiv:2310.1380 [cs.FL] https://arxxv.org/abbb///
Kaiyu Yang, Aidan M. Swope, Alex Gu, Rahul Chalamala, Poiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. 2023. LeanDojo: Theorem Proving with Rotrieval-Angmented Lan- guage Models. arXiv:2306.1566 [cs.L.G] httpp:///axdivorg/bbb/////// 15626
28

---

Published in Transactions on Machine Learning Research (05/2025)
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A Large-Scale Human-Tabeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (Eds.). Association for Computational Linguistics, Brussels, Belgium, 1911-3921. https://doi. org/ 10. 10.11653/v1/D18-1222
Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quiney Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. 2024. The Shift from Models to Compound AI Systems. httts://bair..eree wwwerkeeey.edublllee http://lgg////////2//// /2////22222/2222 aata://cooooudadaeeeeeeeee
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2023, Efficiently Programming Large Language Models using SGLang. arXiv:2312.07104 [cs.AT]
Qihao Zhu, Qingyuan Liang, Zeyu Sun, Yingfei Xiong, Lu Zhang, and Shengyu Cheng, 2024. Gram- mar Grammar-Integrater Pretrained Encoder-Decoder Neural Model for Code. In Proceedings of the IEEE/ACM, 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE '24). As- sociation for Computing Machinery, New York, NY, USA, Article 76, 13 pages. https://doi.orgg
1145/3597503 3639125
29

---

Published in Transactions on Machine Learning Research (05/2025)
A
Appendix
A.1
List of Symbols
G
Formal Grammar
[(G)
Language grammar
Lp(G)
Prefix language of a grammar lexical tokens
I
4
i-th lexical token in the parsed output A terminal in the grammar Terminal type of i-th lexical token Set of all terminals in the grammar Language terminals for grammar G
T
("(G)
(G)
Prefix language of terminals Parser
P
A
Sequence of terminals
T
Tokenizer in LLM
V
Vocabulary of an LLM
V
Subset of vocabulary containing acceptable tokens at k-th LLM generation iteration Regular expression for a terminal 7
Ar
P
Regular expression corresponding to i-th lexical token Partial order over set of terminal sequences Remainder from SYNCODE, parsing the partial output Partial output at k-th iteration of LLM generation
<
C
C
Parsed prefix of partial output C₁ at k-th iteration of LLM generation Set of accept sequences
A
Ma
DFA lookup store function for terminal sequences of length
dmatch
Match with DFA walk as defined in Section 4
pmotch
Partial match with regular expression Partial parsing function Boolean mask Deterministic finite automaton States in DFA
pparse
m
D
Q
Σ
Set of characters i.c. alphabet for DFA Transition function in a DFA Extended transition function in a DFA Start state of DFA Set of final states in DFA Live states of the DFA Set containing all DFA states for DFAs of all terminals in the grammar Set of terminals acceptable for current lexical token Set of terminals acceptable as for next lexical token
8*
4o
F
live
Qn
Ao
A
Lex
Lexer function
len
Length of a sequence
T
Current of tokens
S
Map for storing parser state
30

---

Published in Transactions on Machine Learning Research (05/2025)
A.2 Proofs for Theorems
Lemma 1. Given A 742. Tf4d) AP (TF+2... 17.22..-7H TT.d) and PA ( .Pfta)) dmatch(w: w. %0. .PP - pmatch(w,,N)) Px).
Proof.
(a) First prove dmatch(w, To
T112 AP) pmatch(w; PN) We prove this using induction
the length of w.
For i = 0, pmatch((, PA) is trivially true. Now, we assume that for w of length i < k, dmatch(u w. 90') ,P) = pmatch(w, (,scch((w,p.. px). We consider w of length and dmatch(w, %0 Tr+, A²).
We consider conditions from Definition 10
condition true, 8y 8,,1110 (w, (w,90 +)**)) E live(Q Let q1 (w g0 By Definition Ewi s.t. 8 (W1,91) E Fry+1- Hence,
("..w1 10 11)) E Fr/+1
w.w1 [(PTy+))
We assume that each terminal L(T,) non-empty. Hence,
Ju E L(xx) = w.W1.102 E L(ps)
Hence, by condition 2 from Definition 8 pmatch(w,Pa))
If condition 2 is true, Bwn, u2 such that w1-W2 w, 3r (wi, 40 111) E F and A² 0)- Here, wi E [( ). Since A" 0, PA = Pi. and hence, wi L(px). Hence by condition from Definition Amaathhw,,P.. Px)-
condition true, 3w,, u2 such that wi w2 = w, and dmalch(w2, 4n T542 (Tr+3... ..aa....T(4a))) = true.
90 2112) E FrJ11
t/ (wi, 10 2 E Prt+1 wi (((ps--1)
Since length of w2 < k by our induction hypothesis, pmatch(w2, pAr) = brue. By Definition are two possibilities. Suppose 3ul- = W3.W4 such that wz E L(pxe).
g there
w1.wg E L(pa) = pmatch(w, px) true
Alternatively, Buy such that u2. L(par)
wi W3 L(px) = pmatch(w, PA) true
Hence, our induction proof is complete and pmatch(w, PA) = true
(b) Next we prove pmatch(w, PA) length i of
dmatch(w, 9000 T++1 AP) We prove this using induction on the
For i 0, dmatch(w, 9 T/+1_ AP) is trivially truc.
Now, we assume that for of length i < k, pmotch(w, px) = dmatch(w, to g AP)
Now we consider w of length & and pmatch(w, (ww PA).
By Definition g there are two possible conditions
Case 1: 3w, E E",wy Wy E E+ such that w = wi w2 and wi E L(px) Hence, Auy wt such that wi = wg-ws and wi E (((tyr)) and w4 E [(pA)). By induction hypothesis,
pmatch(w,,wz, w4 PAR) dmatch(w.,, f7 T/43 Tf+a))
Since W3. W02 and
W3 E [(Pr(+1)
5, (w3, 40 Frym
31

---

Published in Transactions on Machine Learning Research (05/2025)
Hence, by condition 3 in Definition 10 dmatch(w, 40 r ,A)
Case B0e such that wawn (ps))
Hence, Huz, Wa w.w, w2.w3 and wy E L(pryss) and ws E L(ps)
Now there are two possibilities, cither is prefix of w2 wy is prefix of w2
Supoose w is prefix of w2, then t (w,40 90 +) E live(Qr) and hence by Definition 10 dmatch(w, do 1 ,P) Alternatively, w2 is prefix of w then Bwg 8.t. w = w2w4
Hence, W1 W3 L(pry+1) and thus pmatch(ws, PAr)
By induction hypothesis dmatch(ww w4, 40- {T143, T(...T(+d)) and since w W2.u0 wa and ,, (W2, 90 ))+ E Fryes. We get dmatch(w, qu T1+,, AP)
Lemma 2. g (, 16) and no prefia of r is in [(T) ie. Fun E E E such that .1.02 and 67 [1, 10) E F, then dmatch(t, q, A) + dmatch(r.t, Lu. A).
Proof.
(a) First, we prove dmatch(t, q, A) = dmatch(r..s 1o 1,, From Definition I0 either of the 3 conditions hold true for dmatch(t, q, A).
condition true then
8 (t, live(Qr) = (r.r, 95) E live(Qr) = dmatch(r.t, T.
condition true, Reo, w2 such that wi w12222 = t, (w1,, E F and A = 0- Therefore, (r.w1, 4) E F = dmatch(r.t, 16.
If condition is true, Dor, u2 such that wa.w2 = L, (wi, E and dmatch(ws W2, 40' (r2...Ta)) = true. Therefore,
(r.wi,, a) E F dmatch(r.t, 4.,A)
Therefore, in all cases, dmatch(rt, go A) must hold.
(b) Now, we prove dmatch(rt, 86 A) = dmatch(t. g. A).
From Definition 10 either of the 3 conditions hold true for dmatch(y.d, g6. A).
If condition is true then
87, (r) live(Q+) - (1,1, E live(Qr) - dmatch(l, q,
If condition 2 is true, Swi, we such that w1 w1.w2 r.t, (; 97) F and A (). Since prefix of r is accepted by L(T); Euy s.t. Wyll4 t and
(, 9) E
smotch((,,g.h
If condition 3 is true, Eor, we such that ww.w2 = r.I,, (wn, % F and dmatch(w2, (2, 46' {(2... m()) = true. Since no prefix of r is accepted by L(T), 3wz s.t. waw4 = and
(3 9) E F = dmatch(t,q.k
Therefore, in all cases, dmatch(t, q, A) must hold.
32

---

Published in Transactions on Machine Learning Research (05/2025)
Theorem 1. Let Cs Lp(G) be the partial output and any integer 2 1, let Aa S ldd contain all possible accept terminal sequences of length d and E Ex denote the remainder. If Srammar-MaskAA, then for any t E V, if C2.t E Ly(G) then t E set(m)
pparse(Ck) where A T2,72...77 and let T1. pparse(Cx,t) where A = =1, 72 7..-7.--77-.. Tf+g
Proof. Let r,A
Hence, we can split r.t such that for w E 2", r. w and w (T.
There are two possible cases:
Case 1: g d
w L(T
...
By our assumption on Aa there must exist A2 = Tf+1 ... Ts4d s.t. Tf+1 ..41..Tf44 is prefix of A2. Hence,
W Lp(A2)
pmalch((.I,AA))
Case 2: 220 d Since we assume that Aa contains all possible accept sequence of length d, A2 = Jf+1 Tftd must be contained in Ad Hence, Dwi, W2 E such that w = w2. and
w1 E L(A2)
w E Lp(A2)
pmatch(r.t,Aa)
In both cases, pmatch(r.t,A.) A2). Using Lomma
dmatch(r.t,go) "1 (T/42..../44
Using Lemma
if q = , (r, 00 T111)
dmatch(r.1.. 7/1 (T/42... = dmatch(t, q, (T/42...
Here from Definition 12 if Ma-1(g, fT T/+a])++++ then t E set(mz).
Since m2 C m, we have our result L E sel(m).
Lemma 3. Given A1 and A2 are set of accept sequences such that A1 & A2 and m2 = GronmarrMask(A117 and m2 GrammarMMakk A2,r) r) then sel(mzg S sel(mı)
Proof. Since VAg E Ag2Ai E A,3A3 E 1 s.t. Ag = AI.A3, Hence
pmatch(w,px.
pmatch(w,Pp
Hence, for the mask set(m2) C set(m1)
Thcorem Let Cn E (G) be the partial output, let Aa S. D contain all possible accept terminal sequences of length and E E. denote the remainder. Suppose for any t V,d len(t) and GrammarMask(A.. such that & set(m) then Cz.l E (G)(G)
33

---

Published in Transactions on Machine Learning Research (05/2025)
For the simplicity of presenting the proof, assume that d > 2.
Since t E set(m) for some A = [T T222 Tftdl E A
dmatch(t, fTf+.................... ... ry+al) = dmatch(r.t, t0 (7542-...7/a)
pmatch(r. P1r1a (..Pr))))))
By Definition there are two possible cases:
Jer E """ u2 E t such that rit = w2 and w1 Prp4z "r142 Prrpp)
We show that this case is not possible since our terminal sequence A1 long enough that no prefix of r.l cannot be in Prr (.-rrrra)
We can infer that len(w1) < len(r.t) = len(w1) < len(r) len(t)
Further, from the assumption d >> len(t), we have
len(w1) < <d len(r)
Firstly, note that & ((Pry.--PH..) by the definition of remainder r Note that we assume no terminal contains empty string i.c. o L(pr) Hence, every string in ((T74a....PPr should have length at least d - -1
Clearly, r is prefix of w1. Let wy E E" r.w3 = wi and hence len(ws) > d - Hence,
len(r) + - < len(w1)
len(r) d len(w,) + len(r)
This is not possible and hence such W1 cannot exist.
Biy E E' such that r.t.w, E L(PT441 -P;--- ... ...PTra))
By Definition 5 we have AE, pparse(Cs) 8.1 Ck CR., A 72, T.. CH
L(pr. ...... )r
Let A = Tf+1, Tf+2....77 T/td
Since, Ca.l = r.I, C E L(A9) and r.d.w, E L(Ai), we have
Ch.rt.w I(AP.A,)
Cg.t.wi L(A.A))
By Definition 7 of accept sequence, A E (G), Hence
Cadhuun (G) Cedt (p(
Thus, our proof is complete and C,. E (p)
34

---

Published in Transactions on Machine Learning Research (05/2025)
A.3 Incremental Parsing Algorithm
Our parsing algorithm achieves incrementality in LLM generation by utilizing map S to store the parser state. This map associates a list of lexical tokens with the corresponding parser state after parsing those tokens. Frequently, in subsequent LLM generation iterations, the count of lexical tokens remains the same- eether the next vocab- ulary token is appended to the final lexical to- ken; or it increases. Although uncommon, there
Algorithm 4 Incremental Parsing
Inputs: Cs: partial output, S: state map
1: function PARSE(CA)
4, 11.22............. .2....... + Ler(Cr)
Y. S. x- RestoorSStttcSS.L
P - --niiialize(s))
parsed 1-11221 --1
6:
for E E4..1441 .... do
are cases where the number of parsed lexical to- kens may decrease during iterations. For exam- ple, in Python, an empty pair of double quotes, " is recognized as a complete lexical token rep- resenting an empty string. On the other hand,
7:
Nexl(P,4)
if P.state = Error then break
10:
porsed + parsed 4
11:
Ao A1
"" serves as a prefix to a docstring, constituting
12:
A1 + Follow(P) (P)
an incomplete parser token. Consequently, the addition of single double quote reduces the overall count of lexical tokens in these iterations.
13:
S: P.state
14:
Store (S, parsed, Si)
We observe that while the total count of lexer tokens at the end may undergo slight changes
15:
C = pursed then
16:
r r= 1
during these iterations, the majority of prefixes 17:
A
{T,, [0],), (Ty, [, ... ([A)00)), [A]]]]]]]]]) (Ao[1]].... ....
of the parsed lexical tokens remain consistent.
18:
Hence, we establish a mapping between lists of prefixes of lexical tokens and the corresponding 20:
19:
else
r == C. parsed
parser state after parsing those tokens. Subse- 21:
A+ [A100] [A
quently, when parsing new list of lexer tokens, 22: return A,r
we efficiently determine the maximum length prefix of the lexer token list that is already present in S. This incremental approach significantly reduces the complexity of our parsing algorithm.
While it could be feasible to introduce incrementality in the lexing operation, our experiments revealed that lexing consumes insignificant time in comparison to parsing. a result, we opted to focus only on performing parsing incrementally.
Our incremental parsing algorithm uses standard non-incremental base parser P that maintains parser state and supports two functions Next and Follow. The Next function accepts the next lexer token and then updates the parser state. The Follow function returns list of acceptable terminals at the current parser state. These functions are present in common parser generator tools (Lark] |NTLR D.
The Algorithm 4 presents our incremental parsing algorithm. The algorithm utilizes a lexer to tokenize the partial output. The function RestoreState is used restore the state of the parser to the maximal matching prefix of lexical tokens that exist. in S. The main loop iterates through the tokens and maintains parser state map. For each token, it updates the parser state, stores the mapping in S, and retrieves the next set of acceptable terminals. The process continues until the end of the partial output. The algorithı relurns accept sequences A and remainder r.
A.4 Ablation Studies
In this section, we perform an ablation study for incremental parsing and max new tokens.
Incremental Parsing. We compare the runtime efficiency of utilizing incremental parsing over re-running parsing from scratch in SYNCODE. We run inference on CodeGen-350M with SYNCODE using incremental parsing and parsing from scratch on Python prompts from the HumanEval datasct. We generate n samples and control the max new tokens in the code completion. Our results are presented in Figure 10b] where the x-axis represents the max new tokens and the y-axis represents the average generation time, in
35

---

Published in Transactions on Machine Learning Research (05/2025)
Table & SYNCODE on few-shot prompting
Architecture CodeeGnn335M
Error Type
Standard
SynCoDE
Syntax Indentation
53
0
100%
15
3
80%
WizardCoder-IB
Syntax
40
2
95%
Indentation
22
95%
Llama-7B
Syntax
110
0
100%
Indentation
88%
seconds, with and without incremental parsing. As shown in the ligure, the average generation time when re-parsing from scratch increases significantly as the maximum length of code that the LLM can generate increases. On the other hand, the average generation time increases slowly with incremental parsing. For max new tokens 300, SYNCODE with incremental parsing achieves 9x speedup over running parsing from scratch. Our results collectively demonstrate that augmenting SYNCODE with incremental parsing dramatically improves generation time, especially when generating longer completions.
Non-Incremental Parsing
175
Standard
Incremental Parsing
12
SynCode
Time (seconds)
Time (seconds)
10
125
100
75
50
0
50
100
150
200
250
300
100
150
200
250
300
350
400
Max new tokens
Max new tokens
(b) Average generation time with and without incremen-
(a) Average generation time for different new lokens
(al parser
Figure 10: Ablation studies on CodeGen-350M model.
Max New Tokens. We conduct an ablation study into the relationship between the maximum length of code that the LLMs can generate and generation times. We used Python prompts from the HumanEval dataset and leveraged CodeGen-350M to generate the code completions, both with and without the augmentation of the SYNCODE. As shown in Figure [0a] we increase the max new tokens, we observe corresponding increase in generation time.
A.5 Few-Shot Prompting
Few-shot prompting (Ren et al. 2018 refers to the idea that language models do not need to be specifically trained for a downstream task such as classification or question answering. Rather, it is sufficient to train them on broad text-sequence prediction datasets and provide context the form of examples when invoking them. We study the performance of utilizing SYnCODE on few-shot prompting code generation tasks. We selected Python few-shot examples from the MBXP dataset and generated code completions with CodeGen-350M, LLaMA-7BB and WizardCoder-11 with SynCODE and the standard no-masking generation. We present our results in Table g. The columns standard and SynCode represent the total number of errors of particular Error Type of -LM-ggnerated code completions to problems in a particular datasel, when utilizing that respective generation approach. The column 1 represents the percentage reduction from the standard column to the SynCode column. As shown the table, SYnCODE exhibits effectiveness not only in zero-shot but also in the context of few-shot prompting tasks. This signifies the versatility of SyNCODE in enhancing code generation across different, prompt engineering techniques.
36

---

Published in Transactions on Machine Learning Research (05/2025)
def max_fill(grid, capacity):
"""You given rectangular grid of wells. Each represents single well, and each in represents single unit of water. Each well has corresponding bucket that can used to extract water from it, and all buckets have the same capacity. Your task is to use the buckets to empty the wells. Output the number of times you need to lower the buckets." "." if len(grid) 2:
return 0 if len(grid)
return
if len(grid) - 2:
return grid[0][1] grid[0][0]
if len(grid) 3:
return grid[0][1]] grid[0][0] grid[1][1]
more lines
len(grid) = 9:
return grid[0][1] grid[0][0] grid[1][4]
grid[2][[] grid[
Figure 11: Syntactically Incorrect SYXCODE Program
A.6 SynCode Syntax Errors
Figure 11 presents an example of when the SYNCODE augmented LLM fails to generate a complete program within the maximum token limit for a problem from the HumanEval dataset. While the code is a syntac- tically correct partial program, it not a syntactically correct complete program. Recall, that SYNCODE guarantees completeness for syntactically correct partial programs but does not guarantee termination with syntactically correct complcte program.
37

---

Published in Transactions on Machine Learning Research (05/2025)
A.7
Prompts Used in the Evaluation
<>> (INST] <<BY>>>
You are helpful assistant that ansxers in JSON. Here 's the joon schema must adhere to: <achema>
('title': 'Person', 'type'; 'objact', "propertiea"s ('rirataane's ('type': 'string', description "The person's first nane "), 'leetName' '; ('type' 'string's 'description's "The person's last "), 'age': ('éescription': 'Age in years which must be equal or greater than zero.', 'type's ' 'intager', 'mininun' 0H,, 'required"; ('firstlane', lastliae'', 'age?])
</schema>
<</SY>>>
Please generate JSON output for person's profile that includes their first last nane, and age. The first name should be 'Alice', the last name 'Johnson and the age 

Listing 1: Example original JSON Prompt from the J8ON-Mode-Eval dataset (Nousces 2024). The prompt consists of a system message that specifies a schema and a user message requesting JSON output given certain parameters.
<a> 
 <<8Y8>>
You are helpful assistant that anszera in J8ON. Here aa the jaon schema you nuat adhere to:
('title?: 'Parson', 'type': 'objact', 'propertiea', ('firstNaan's ts ('type': 'string', description's "The person's first nane ",, 'lastName' ' ('type 'string', 'description 'The person's last nane. "), 'age': ('description'; 'Age in years which nust be equal or greater than zero.', 'type': '; 'integer', 'mininum' ; 0H, 'required"; ('firstKane', lastlaan' 'age
</achema>
<</SY>>>
Please generate JSON output for persen's profile that includes their first last nane, and age. The first nane should be 'Alice', the last nane 'Johnson and the age Output only


Listing 2: Example JSON prompt from the JoON-Mode-Eval dataset NousResearch augmentation with an explicit request to only output JSON.
2024 after
db_id: concert_tinge
ab_info: stadiun stadius_id location name capacity highast losest avarage ainger { singer_id nane country name song_release__eat age is_nale concert concert_id concert_nane theme stadium_id year singer_in_concert concert_id singer_id seccort.atadiure id stadium.stadian singer_in_concers singer singer.singer. # canger_tt.coccatt.co
concert.concert..
11
question: How many singers have? Only output the SQL query. SQL:
12
Listing 3: text-2-SQL prompt.
def bas_close. Das_cclos_eeeeente elenents (nunbers: List [rleat], threshold: float) bool: *.* Check if in given list of numbers, any numbers closer each other than given threshold.
mas_closs,elenents clanentsenennss.... 2.0, 3.0] 0.5)
False >>> has, clascclooeeeeeeeesse ([1.0, 2.8, 3.0, 4.0, 6.0, 2.0], 0.3)
True
Listing 4: Example Python prompt from the HumanEval dataset
(Athiwaratkun et al. 2023
package
import
"encoding/json
11 You 're an expert Golang progranner given nunhers, any 'reflect"
38

---

Published in Transactions on Machine Learning Research (05/2025)
given threshold.
// >>> has_close,
010.entttttn hes_clloe_ollaeettt(11..... 2.0, 3.01, 0.5)
11
// False
1/ >>> aeo_clooe_eeemonte ([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
13
1/ True
//
15
Tune cas_close alamenta (numbera 111itat,, threshold float64) bool {
Listing 5: Example Go prompt from the HumanEval dataset (thhwaratkun et al.
2023
A.8
Grammars Used in the Evaluation
A.8.1
JSON Grammar
?start: value
?value: object
array
UWESCAPED_STTALE STRING
number
"true"
<> true
"false"
false
'mull"
null
10
array
[value ("," v]lue)*] *]"
12
object "(" [psir ("," +air)*] "}"
13
pair UNESCAPED, B STRING ":" value
14
18
MESCAAEDDSTRRIG A
17 DIGIT: "D".."9"
"a".. "E"I"A".. IDIGII
19
INT: DIGII
20
SIGNED INT: [" IET
DECINAL: INT "." INT? "." INT
22
24
EXP: ("a"I"E") SIGRED INT
28
NUMBER: FLOAT INT
27
SIGNED : ["+"|" [- XUMBER
NS: /[ \t\f\r\a]/
29
Xignora
Listing JSON Grammar
A.8.2 SQL Grammar
expr "1"7
sep_expr: query_expp
set exp_ "UNION"1 ['DISTINCT"1] set_expr >> anion_distimet aet aet_expr "UNION"1 "ALL"1 set exp_ -> union_all
"INTENSECT"S ['DISTINCT"1] set_expr >> aet expr "EXCEPT"1 ['DISTINCT"1] set +> except_diatine set_expr "EXCEPT"1
set_expr except_all
10
query query_expr: expr:
"ORDER"1 "BY*i (order brrder_b__exp by_e ",*)+ order_by "LIMIT"A linit_com [ "DFFSET"1 skip_rous 1
12
select: "SELECT"1 ",")+] fron_ expr [ "WHERE"1 where_expt "GRDUP"1 "BY"i [[groupby_exp: ",")*] croupby_expt "HAVING"1 having_expr] "WINDOWY" window_expr
[(select_exp] ",*)+] select_expr "FRON"S [(fren_ezpr
14
15.
chere_expr: expression
17
select_ 0: expression_nath [
"A8" eliss select.espreessin
19
?fron_oxpr fron_itan -> froz_ expressin
21
order order_by_ oxpr: order >> order aider_by_oxpee oxpression
39

---

Published in Transactions on Machine Learning Research (05/2025)
baving_expr? bool_ expreexpreesien
25
groupby_oxpr: expression -> group_by
27
vindov_expr: [vindov_expt ",") _vindov_nane "AS"i (iindoo_definition
28
table, join join
mable_naee nane [ "AS"i alias >> table
30
croas_join -> jooa___oi__xpraaaee
32
subquery table_name: nane
34
subquery: "(' (query_axpr join cross_join) ")" [ *A8"s alias
36
37
cross_joit fron,
"JOIX"1 tron_iten
38
Join: from_iten JOIN_ EXPR fren_ited Item [ "ON"1 bool_expression ] -> expresssnn
39
JOIH_EXPR. _JOX__TPPE N3); "JOIB"1
41
JOIE TYPE: "INNER"1 10 JOIN_ DIRECTINCTIONN (US "QUTER"1)? JOIN DIRECTIROT
:IRECTICETO:: "FULL"S "LEFT"1 "RIGHT"1
43
Fexpression, expreseiin.,product
45
expression_nath "*" expression_ product -> expreaslon_ade
46
expreseion_aeth
expression. producct expressicn_ssu
"CASE"i (xhen_then)) "ELSE"1 expression_nath "END": case_expression
48
"CAST"1 "(" expression_ nateassion_nahhheeeeee math "AS"1 TYPENAME ")* -> as_typa
TYPENAME AGGREGATION expression_nata ")" [vindox_fora) -> sql_aggregation
"CAST"1 "(" literal
literal_cast
50
"RANK"i ")" vindov_form
expk_exptession
"DeMse_raK"") "(" ")" vindov_fors -> donse__enk_eeppreesico
58
"COALESCE" "(" [(espression__avh) ",")+] expression, ")*
coalesce, coalesce_expreenti
54
aubquary >> abquury_expreeate.
50
vindow_form: "OVER"i "(" ("PARTITION"4 "BY'i (partition_by ",") partition_by) [ 1'DDEE"' "BY ggo_range__cauee *)"
(order +")*
57
58
partition_by: expression_nath
60
rox nox_range_clan raagga_cauuse: ( ROWS RANGE frane_extent
frame_extent: trame_betxeen
frame_preceding
62
"rane_betuess "BETWERX"1 trane_bound "AND"1 frame frame_bound
63
trame_boundi frame frame_ proceding
frane_folloving *CURRENT"1 "ROV"1
frame_preceding: UNBOUNDED PRECEDING IKT_NUMBER PRECEDING
65
trame_folloving, UNBOUNDED FOLLDWING IKT INT_NUMBER FOLLDWING
RANGE: "RANGE"1
67
ROWS: "ROWS"1
UNBOUNDED:
69
PRECEDING: "PRECEDING"s
70
FOLLOWING:
72
ohen_then than: "WHEX"1 nool_exprassion "THEN"1 expression,nath
order: expression_ ["ASC"1] order
74
expression_math "DESC*1 >> order_desc
76
77
Texpreasion petpraasion_prouuctt. expression, parens
expression_product *** expressien_parens expression_ne
79
expreaston_product "/" expreaston_parens expression_ds div
81
rexpression_parenn expression
expressicn_parens expression
expression,
53
"(" expreasion_parens "/" expression ")" expression_de div
84
expreasion_parens expression ")" expression_ade
"(" expression_parens expression ")" expression_sub
96
[nane
88
fexpression: column_ nane +> columa_nane
91
SELECT CONSTRAIXT "ALL"1 "DISTINCT"A
SELECT_CONSTRANE
93
TYPENAME:
"object"i
"yarchar"1
95
"integer";
"inti6"1
97
"smallint"1
08
"int32"1
"int84"i
100
"int"i
"bigint"i
40

---

Published in Transactions on Machine Learning Research (05/2025)
"floati6"i
"float32"1
"f1oat64"1
"Ilost"i
108
"bool"i
"datetine64"1
108
"tineatamp"s
"tine"i
"data"i
111
"cateegLzy""
"string"i
113
AGCREGATION.6 ("SUX("3 *AVG("i "MIN("S
"MAX ("i
"COUNT ("i "DISTINCT"
"COUNT (*1)
alias: -> alias_atring
115
>xindov_name: name
limit_count
linit_count
skip_roza: INT NUMBER
118
bool_expression:
pol_parentheees
bool_expression "AND"i bool_parentheses bool_and
120
hool.exprassion. "OR"i bool_parentheses -> bool_or
bool _ool_pareee _al__paretthesa
conparison_type
132
*( bool_expreseion "AND"i comparison, type *)" -> bool_and
'(" bool_expreseion "OR'i comparison_type *)" bool_or
"EXISTS"1 subquery exista
126
comparison_typee equale mot_equels greater_then less_than Saatter_than_o__eee
less_ taan_than__or or_equal betveen
not_in_expp subquery_in subquery
ia_mull is_aot_mmll lipe_expr not, lik_
128
equals: expression_uath "-" expression_nata is_aull: expression, "IS"i "NULL"i
ie_not_null: expression_nath "IS"1 "XOT"1 "NULL"1
131
not aot_equals: expression_nath ("e>" "!=") expression_nath
greater_ than: expression,nas '>" expression_natt
133
less than: expresaion, nath expreasten_satl math
(reater_theu_or.equal) expression_math ">a" expression_nath
136
Less_them_or_equat expression_uath nath "c-" expreseion_natà _ati
between: expression, " expression,
expression,
138
// 'LIXE' and 'NOT LIKE
like_exps expression,
"LIKE"1 expression,
140
not _i__i_kkeexpp_____ expression_math nath "XoT"1 "LIKE"1 expresaion_nath
142
// 'IN" and 'NOT IN'
in_expr: expression,
"IN"1 "(" [expresston, "," expression, ")*
subquery_ini expression_aath "IX"i subquery
145
not not_in_expr: expression_nath "NOT*; "IE"i "(" caapression_mat ","]+ expression_nata nath *)"
subquary_not_ expreasion *NDT": "In"1 aubquary
147
number_axpt +> number
/'([='))+']>"/ string
tinestaap_express -> tinestasp.oxpressic
152
boolean:
"TRUE"1 -> true
"FALSE"1 false
154
?nunber_expr product
?product:
INT, HUNBER +> integer FLOAT float
158
INT NUMBER: [01-9]
181
STAR: "*"
sindow definition:
timestanp_ expression:: 'NOV*1 "(" ")" -> datetime nog
164
"TODAY"1 "(" ")* date_today
166
date: YEAR NONTH DAY
(4)-
168
MONTH: /[0-9] (2)/
/[0-)]22)/
time:
HOURS ":" MINUTES ":" SECONDS
171
BOURS:
[0--]] (2)/
MINUTES: (2}/
178
SECONDS: /10-91
ESCAPED_STRING
STRING
ESCAPED ESCAPED_STRING STRING: "\"" -T.SSTRR_IINAN IENER *\""
178
Ximport CHAME
180
Kinport connon NS
Kimport COMMENT
41

---

Published in Transactions on Machine Learning Research (05/2025)
Ximport
kimport conmon... FLOAT
Xignore NS
180
%ignore SQL_CONMENT
Listing SQL Grammar
A.8.3 Python Grammar
single single_input:
imput: _NL I simple_atat compound_atat _BLL
start: (_3L stmt)+
eval_inpat: tastlist _NL+
!decorator: "q" dotted_nane "(" [argumenta] ")" ] NL
decorators
decorated: decorators (clasadel funcder asyne_runcder)
ssyac_funedef- funcdef: "async" funcdef
10
fuacdef: "det" BAME "(" paraneters? ")" D*>x" test] ":" ( auite .ML.)
12
pparsnetere: paranvalue ('," paranvalue) [")" starparane kvparaae]]
starparans
14
keparans
16
starparans:
sypedparan? ('," paranvalue) (")* kxparans]
kwparans: "**" typedparan
17
?paranvalue: typedparan [* test]
19
Ptypedparam: NAME [*:* test]
Ivarargsliat: (vipdef [*=* teat] ("," vipdet [t* teat])* [".* "*" [vipdef] (""" vipdet ['-" test])+ [",* ["es" vfpdet ["]*]]] "ee" vipdef [","]]]
"." [vtpdef] (*, vfpdet ['s* test])+ ["," ["xx" vfpdef [", [',"]]]
23
vrpder [")+])
25
vipdef: NAME
Tatnt: (simple_atas aant compound_ane _tnnt ) ["aof"]
Pamall_ (axpr_atie) pasttttttto. del_stat pasa_atat flox_atat inport_atat glabal_atat 28
!?simple_ste stat: small_stat (*;' small_stat)= [";"] _XL
(annassign sugassiga (yield itestlist) nonlocal_stat assert_atat)
?expr_stat: reatlist_star_eet
31
(*e"
32
annassign: ";" ["=" test]
33
17tastliat_star_rrg
(teat/star_en) ("," -xxp=))) [",]]
33
laugassign: ("+=" "-=" "*=" *g-* "/=" I "X=" "g=" *|=" "<<=" ">>et
"*x** "//=*)
35
// For normal and annotated assignments, additional restrictions enforced by the interpreter
del_etat: "del" exprlist
37
pass_stmt: "pass"
38
flov_atat:: treak_atnt. contimue_stat return_atmt raise_atat ylald_stm stat
break_stat: "break"
40
continue_stat. "continua"
return_atnt: "return". [teatlist]
42
yield_stat: yield_expr
43
raise_stmt: "raise"
("frox" test]]
44
import_stnt: import_ name inport_fron
45
import_name: "inport" dotted_as_nanee
// mote belov: the ("." "). is
because "..." ". 18 tokenized ELLIPSIS
47
import_fron: "fron" (dota? notted_nane dots) "inport" (tan "(" import_as_nanea ") import_as_uanes)
48
!dots: "."
48
inport_as_came:
[*as"
50
dotted_as_nane, dottad_namd nane ['as" NANE]
51
'import_aa_nanes:
ae.tt_ss__ seport_ss_aamn ("," inport_ae_nane)> [","]
dotted_as_cames:: dotted_as__est ('," dotved_as_nszee+
53
dotted_nane: NAME (".* NAME)
global_etnt: "global"
56
sonlocal.stat. "ponlocal" NANE ("," -AME) test [", test]
assert_atnt: 58
57
58
compound_stat: if_stnt while_etat for_stnt try_stat vith_stat funedef clasedef
decorated asyac_stat async_stnt: "asyac" (funcdef sith stat for_stat) if_stat: "if" ";" suite ("elif" *;* suite) ("else" suite]
59
61
while stale "shile" test ":" suite ("alse* ";* suite] ["else" auita]
for stat: exprliat
42

---

Published in Transactions on Machine Learning Research (05/2025)
try_stat: ("tzy" ";" ((except_clan clause suite)+ [*elso" ";" suite] ("finally" ":" suite] I 'finally" ":" suite))
64
vitb_atat: "with" with_iten ("," sith_iten)+ ";" auite
vith_iten: best ["as" expr]
68
// на compile.c nakes sure that the default except clause is last claupt_ "except" [test ["as" NAME]]
68
suite: aimple_stat .II. _IDDENT stat+ .DEDENT
70
?test: or_test ["if" or or_test "elae" test] lanbdef
71
fiest_nocond: or_test lanbdef_nocomt
lambdef: "lambda" [varargeliet] ";" test lambdef_noconde "lanbda" [varargalist] ";" test_mocond
for_test: 73
("or" and_test)
75
Tand_teat: not_test (*and" not_test).
77
Inot_test: "not" not_teat >> not comparison
78
fcomparison: expr (_comp_op expr)
80
star_
expr
Taxpr: xor_expr ("I" xxp_)xpp)>
82
1xor_expr, and_expe expr (**" and and_expr)*
fand_expr: shift_expr (*2 shift_expr)-
Tehift_expr: arith_expr (_shift_oop arith_expr)*
86
farith_expr: tern (_add_op term).
Pterm: factor (_aul_op factor)*
87
?factor: _factor_op factor povar
89 !_fcctor_pp """]"="[""""
90
__add_op: "*"|"""
_abift_op: "<<"
92 _aul_op: """""""|"""""""""""""""""""""""" 11111111111"""""""/1""""""""
// <> isn 't actually valid comparison operater Python. It's here for the
94
// aake of a futurer......................... import deacribed in PEP 401 (abich really worka :-)
_comp_op: """""""""=""]"=="""""" |4t
"in"""ie"(""" "not"
98
97 ?pover: expr
factor]
sswiit_expr: ["avait"] aton_expr
99
expr: expr "(" [argunants] *)" funccall
101
aton_expr "[" aubacriptlist "]" getiten ston_expr aton
getattr
103
?atom:
"(" [xpeld_expr i cosp] *)" tuple
108
"[" pestllis_compp "J list
*{" [diatorsatmaker] "3" BANE >> var
108
"string*
"(" test ")*
allipsis
"Bane"
113
"True" -> const_true
"Falae" const_false
115
??testlict_comp
staareeprr) [comp_for (",* (tret] [")"] 1."]
isubscriptlist: subscript ("," subscript)* [","]
subscript: teat [teat] [teat] [sliceop]
sliceop: "." [test]
120
lexprliat: texprlatar expr) ('," (xxpr(atar_oxpr)) [',"]
testllit:: beat ('," teat)* [","]
122
/dictorsetnszer: ( ((test *;" best "e** expr) (comp_for ("," (test "," test | *e*" ezpr)
(*,"))) ((test star_expr) (comp_for ("," (test starrexpr))+ ("))))
124
clasadef: "class" NANE ["(" [argunenta] ")"] ":" auite
argusents: atarargs
("," argvalue)+ ["," starargs kvarge]]
126
kvargs
128
test comp_for
Iatararga: *** teat (*," "x" teat)x ("," argvalue)x ['," keargs]
131
kxargs: test
133
Targvalue: test ["a" teat]
comp_iter: comp_for I comp_if asyne_for
async_for: "aayne" "for" exprlist "in" or test [canp_iter]
comp_for: "for" exprliat "in" eest [comp_iter] comp_if: "if" test_nocond [comp_iter] 138
140
// not ueed in grannsr, but may appear in "pode" passed from Parser to Compiler encoding.decl;
43

---

Published in Transactions on Machine Learning Research (05/2025)
yield_expr: expr: "yield" [yield_arg]
yieid_arg: "from" teat teatliat
148
number: DEC, NUMBER HEX_NUMBEN OCT NUMBER
PLOAT, NUMBER
148
string: STRING STRING
151
// Tokens
NAME: (aa-at-2_
153
CONMENT: /#. (An ++)+/ LONG_STRING
NN.: /(ArPAn(NE ]+)+/ COMMENT)+
135
STRING: /[ubt]?r? (?<!\V "."""1"" ((761))))))))))))))))))))))))))) (2111))442111))5
158
DEC_NUMBER, /01 /01
[EX_NUHBER.C 2: [od
160
OCT XUMBER 2: [0a[0-]]*/1
BIN KUMBER [0b[0-1]+/1
162
FLOAT_ FLGAT_BUNBER HUNBER :2 /4ch://
Ximport connon V8_INLIKE
106
Adeclare INDENT DEDENT
167
Xignora MS_ INIINE
Xignore /WA Ar1AAr7Am/ // LINE_CORT
100
Xignore COMMENT
Listing 8: Python Grammar
44

---

Published in Transactions on Machine Learning Research (05/2025)
A.8.4 Go Grammar
start: package, clause esa (import_decl eas)* Orunction_decl "?oc"?)+
nathod_dem _acc
declaration)
peckage_clause: "package" NANE
import_decl: "inport" (import_apec | "(" (import_spec cas)+ ")")
import_apec: (*," I NAME)? inport_path
import_path: string.
11
declaration: const_decl type_decl | var_decl
13
:acst_dacl: "conat" (penat_apac [const_apas eas) ")")
15
const_epec: ideatifier_ list (type_? "h" expreseion.1iet??
18
identifier_ XAME ("," BANE)
20
expression_ list: exprassion ('," expression)+
22
type_decl; "type" (type_apec "(" (type_spec eos)* ")*)
sype_ alces_decl type_der
26
alias_decl NAME "=" type.
27
dep NANE type_paranetere? type_
29
type_paaaaeteen
eppe_paranetor_del ("," sype_paranottr_deel)) "]"
31
32
dec_ identifier,list type_element
34
type_ _pe_eeenent sype_tern (*)* type_term)
38
type_tora *.*? type_
37
38
// Function declarations
39
function, decl: : "func" NANE typa_paranetars? signature ("(" l1at? (*)" "eof
41
zethod_decl: "func*
signature block?
43
44
recaiver: paraneters
46
var dac_l "var" (var_apac "(" (var_apec cos)* ")
48
1ar_apec: identifier_list (type_ (*er expre)sion,liat)? expression__ist)
50
block: 'I' statoment_list? *
51
statement_list: ((";" E03?) statenent e0s)+
53
decleration labeled_stnt simple_stnt go_stat continue_stat goto_stat fellthrough,stat block if_stat svitch_stat select_stat for_stnt defer_stat
break_stat
55
simple_stat: stat inc_dec_stat assigoment expresaion short_var_dect 56
58
send_ stat: expression "<-" expression
80
inc inc_dec_stat: expression (*44* *--"))
61
62
assi asaignnent: expression assign_op expression expresaion_ 11at tz* expression list
63
sesign_op: Tem" F.nt "I-" nne Tem" */-" "X=* necwr ">se" "g-" "g" expression,
65
expreseion,liet
67
68
labaled_atmi
statement?
68
70
return_stat: "return" expreseion_list?
72
break_stat: "break" NAME?
74
continue_stat: "continue" NANE?
45

---

Published in Transactions on Machine Learning Research (05/2025)
goto_stmt: "goto"
77
78
aultthrough_att: "fallthrough"
80
defer_stat: "defer" expression
82
11 it_atat: "it" [ eexpression eca expression simple_atme atat eas expreasion) block ("else" if_etmt block))?
83
84
switch_atat: expr_ syp_sswich_sta stat type_auitch_sttt
88
expr_svitch_stan "switch" (expression? simple_stnt? cos expression?) "("
expr prr_case_claaaee ")'
87
expr_case_clausee
statement,
00
sppr_ "case" expreseion "default"
922
swssttitt_stttt "svitch" (yype_ exipc_ eaa type,eritch.guard simpla_stat eas aype_auitch__aue _garrd "{" sype_case_clane *}
98
type sup_
";-")?
"type" ")"
96
cape_caee_caueer type_exitch_came ";" statement_.iet?
98
appa_auttch_caze case: "cass" type_ "default'
100
type_list: (type_ "nil") ('," (type_ "mil" )
select_etat: "select" "(" clausee "}*
103
conn_clause: clause:
ttatement_liat?
105
cosn_case: (send_etat recv_atat) "default"
10T
recy_stat (expression_lict "s" identifier, ";=*)? expression
110
stn_t 'for" [for_clause] block
112
for for_clauae clause:: simple_atat (eos expression eos sinple_atnt)? range_clause
114
range_clause: lespression_lis: '** expression, list ":-*) "range' expression
go_stat: expresssso
117
type literal_type var_or_typa, tappttppaaaapaaeeee type_arga? "(" type_
119
channel_type
typa_arga
124
var or wtt.or_tte_naee... NAME
HAME
NAME NAME ..* "(" type. ")*
126
arrey_type: "[" array_length "]" element_type
array_lenget expression
elanent_type type.
131
pointer_type type_
138
interface_type "interface" "(" ((nethod_spec type_olenent/ cos)+ *)"
slice_type: "[" "J" elenent_type
138
// It's poasible to replace 'type" with more restricted type_lit 1ist and also pay attention mарs
139
nap_type: "map" "[" type_ *}" element_type
thannel_typa chan" "chan" "<- "<-" "chan" ) alement_type
142
sethod_epec: NAME paraneters result NANE
144
fuaction_type: "fune" signature
signature: paransters result?
149
result: paraneters type_
151
paraneters: "(" paraneter_decl ("," daramster_decl) r,"? ")" "(" ">*
46

---

Published in Transactions on Machine Learning Research (05/2025)
// separatee
name, type, (c) type // paranater_dece decl: identifier, list? "?.."? type.
157
// Although folloving is overapprozinete it's an easy vay to aroid reduce/reduce conflicts parameter_ decl: (type_ "... type. NAME type.)
158
141 expression: prinary. _xprr
(*+* expression (*t" 4/" |
"g" "<-t) expression
" "ee" "5s" 's' "&"") expression
164
expression (*4* expression (*ex* "Is" "c" '<"" ">" | ">**) expresaion
" won) expression
166
expression "EX" expresaion
expression "Il" expression
100
primary_expr: operand primsry_expr ('.' (NAME "(" type_ ")*) Index slice_ orgumente) type-
170
// aiving operand higher precedence type_ hack avoid reduce/reduce conflicta operand.3: literal NAME "(" expression ")" // removed HANE type_arge?
172
literal: basic_lit composite_lit function_lit
176
basic_lit: "mil" integer string- FLOAT_LIT CHAR_LIT
177
integer: DECINAL_LIT BINARY_LIT I DCTAL_LIT LIT
179
DECIWAL. /01 [1-9)
BEX_LIT /0x[\da-()+/1
182
OCTAL_LIT.22 /0010-7]+/1
BINARY_LITS [00b0/-1)+/1
184
FLOAT LIT.2:
LIT: 1/1
188
composite_ literal_cype value
180
literal iiteral_type: struct_tyse srray_type "[" "..." "J" olenent_tyye
slice_type map_type
"interface* "4" "}"
190
iitersl_value (element_liet
"}"
element_liee list: koyad_eleners ('," element))
195
keyad,element, (key ":")? clement
197
key: expression Aiteral_valus
clement: expression literal, value
struct_type: "struct" "(" (field_decl cos)* "}"
202
field_decl: (identifier_11et type_ uubedded_fielde string_?
204
string-: RAM_STRING_LM EITERPRETED_STRING..O.
STRING LIT: /',07%
STA.APRETEED_TRING...C /".+?'/i
208
enbedded_ "*77 (BANE NAME type_arge?
211
function, lit:: "func" signature //
214
index: " expreasion "]"
216
slice_: "I" expression? *;* expression? expression? ";" expression "." expression) "J"
218
type_ assertioo:: "." "(" type_ "
argumanta: "(" { expresssion 1sat? .....? ", "?? ")*
231
";" EOS // (this.closing@err closingBrackett)))
223
/[a-2A-2_]\ve/ EOS: .BL I *;* "/+" .+? 14/"
225
CONMENT AVP/V/
228
_NL: MARPAAEE 1+)+/ COXNEXT)
230
Xignore /it ]/
Nignore /WD \EJ=NEPNE/
LINE
47

---

Published in Transactions on Machine Learning Research (05/2025)
Listing 9: Go Grammar
48


Reviews:
[{"review_id": "fGWZWxUgdu", "reviewer": "Reviewer_G81y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:** \n\n(1): SynCode ensures that all valid tokens are retained and all invalid tokens are filtered out at each generation step, guaranteeing syntactic correctness. \n\n(2): The framework minimizes runtime overhead by leveraging precomputed masks and efficient hardware utilization, such as GPU acceleration. \n\n(3): SynCode can be integrated with any language defined by a CFG and works seamlessly with various LLM decoding strategies, including greedy decoding, beam search, and different sampling methods.\n\n(4): The paper is overall well-writen and easy-to-follow.\n\n**Weaknesses:** \n\n(1): I think the most important weaknesses for the SynCode is that the underlying idea of the framework (i.e., impose restriction on the LLM generation process through masking out all of the irrelevant tokens) is widely used in the previous works (e.g., Beurer-Kellner et al., 2024 DOMINO, Willard and Louf, 2023 Outlines, ...). Note that the authors have discussed the difference in the Related Work section and Table 6. From my point of view, those ideas may have very minor difference. Hence I would suggest the authors to highlight the key technical difference and the contribution.\n\n(2): Generating a DFA mask may require additional processing time and memory.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces SynCode, a framework that enhances the accuracy when making LLMs generate some specific kinds of code language. SynCode augment the LLMs with an additional (pre-computed) look-up table, named Deterministic Finite Automation (DFA) mask store to impose restriction on the LLM generation process. The DFA mask store maps states of the DFA to boolean masks over the LLM's vocabulary. These masks indicate which tokens are syntactically valid at each step of the generation process.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I would suggest the authors to highlight the key contribution of SynCode and its key difference with previous works (e.g., DOMINO and Outlines), in terms of performance, computation and storage overhead. Empirical evidences are highly preferable.\n\n2. The reviewer suggests the authors to add more case study to show what kind of errors that SynCode may occur. What about in comparison with DOMINO, Outlines and other similar approaches based on constraint decoding and code language grammar induction?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RlYcYUnDLt", "reviewer": "Reviewer_cP3J", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nI appreciate the effort the authors took to explain the method with examples. Besides, the overall paper is well-written, and should be a good read for a more general audience. The experiments and proofs validate the authors claim too.\n\nWeakness\n\nMy only concern is that how the pre-calculation step scales. While the authors claim that the size of terminals is reasonable in practice, it will be interesting to see how the time complexity scales with larger $|\\mathcal{A}|$. Another question I feel the authors should answer is how the space complexity is (and additionally, how they scale).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a method for constraining LLMs with grammars by precalculating DFA mask store for any language with CFG. They showcase the method on different languages such as JSON, Python, and GO. The authors also provide proof of completeness of their method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "duIhP6btFs", "reviewer": "Reviewer_uKiz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# strengths\n- Demonstrates large margins of improvement on syntactic correctness in very practically relevant problem settings for LLM generation (Python, Go, SQL, and JSON output).\n- Shows no major trade-offs in runtime performance and only modest costs in offline compute and memory needs.\n\n\n# weaknesses\n- The paper includes no statistical results on variability.\n\t- Many differences are small, specifically for runtime speed and functional correctness in Python and Go.\n- The method cannot be applied to languages that do not expose scores.\n- (very minor) Readers need modest background in programming languages and parsing to follow the text.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper contributes a technique for efficiently filtering LLM decoding outputs guided by a context-free grammar and demonstrates this technique substantially increases LLM output in terms of syntactical correctness, while maintaining or increasing the rate of output decoding.\n\nThe new technique supports offline precomputation and application to general languages (demonstrated with Python and Go, along with more specialized JSON syntax and SQL). The GPU-based implementation of masking is fast, with evaluation experiments showing the filtered decoding process can be faster than the baseline LLM.\n\nThe results show substantial increases to syntactic correctness in JSON output, SQL, python, and Go programming when applied to open models that permit querying of scores. The outputs retain good performance in terms of the correctness of outputs as well.\n\nThe new knowledge presented is around the details of the technique as the work is primarily a technical contribution.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# critical\n- Add reports of statistical variability in the results over multiple runs.\n\t- The key speedup results are very large and likely not subject to noise, but additional runs and variability reports would strengthen many of the other reported outcomes (like speed and correctness).\n\n# strengthen\n- Add functional testing results to SQL.\n- Clarify the SQL testing design.\n\t- Table 2 reports the percentage of syntactically correct queries by difficulty type. But it only reports a single value for the runtime execution success. Should this also be divided by difficulty? Or did I misunderstand the setup?\n\t- \"For example, with the Llama-3.2-3B model, SynCode achieves an execution success rate of 81.4%, compared to 67.4% for unconstrained generation\"\n\t\t- Is this success rate among syntactically valid samples, or all samples?\n- Section 6.1 reports on llama.cpp but uses the CPU, unlike the GPU used for all other results. This is misleading and should be omitted, rather than including the unfair comparison. Ideally the experiments would include llama.cpp using a GPU, but if not, it would be fairer to simply not include the result.\n- Section 6.4: \"We see smaller generation time and memory with Llama-2-7B-chat and JSON grammar as opposed to LLaMA-7B, WizardCoder-1B, and CodeGen-350M with Python and Go grammars since the size of the mask store is proportional to the number of terminals in the grammar.\"\n\t- Where are these results reported? I think I missed them.\n- Is SynCode going to be released? I did not see information on that.", "broader_impact_concerns": "None. Parsers to control decoding could theoretically be used to guide LLMs to produce socially harmful outputs, but this capability is not particularly new to the space of using LLMs or unique to the SynCode implementation.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers provided positive evaluations, recommending accept or leaning accept, with suggestions largely focused on empirical completeness and clarity of comparison:
- Reviewer G81y emphasized the novelty in proving both soundness and completeness, but requested clearer differentiation from DOMINO and Outlines.
- Reviewer cP3J asked for clarification on scalability and complexity of the DFA mask store, which the authors addressed by reporting time/memory requirements for Python and Go grammars.
- Reviewer uKiz appreciated the practical value but suggested adding statistical variability, clarifying SQL execution metrics, and removing unfair comparisons with CPU-based baselines.

The authors have adequately revised the paper to address all major concerns. The authors are recommended to include the clarifications addressed during the response period into the camera-ready version.
Decision: Accept (poster)