Title: Adaptable Logical Control for Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Language Generation

Probabilistic Models

Probabilistic Methods

Large Language Models
Abstract: Despite the success of Large Language Models (LLMs) on various tasks following human instructions, controlling model generation to follow strict constraints at inference time poses a persistent challenge. In this paper, we introduce Ctrl-G, a neuro-symbolic framework that enables tractable and adaptable control of LLM generation to follow logical constraints reliably. Ctrl-G combines any production-ready LLM with a Hidden Markov Model (HMM), guiding LLM outputs to adhere to logical constraints represented as deterministic finite automata. We show that Ctrl-G, when a TULU2-7B model is coupled with a 2B-parameter HMM, outperforms GPT4 in text editing: on the task of generating text insertions/continuations following logical constraints, our approach achieves over 30% higher satisfaction rate in human evaluation. When applied to medium-size language models (e.g., GPT2-large), Ctrl-G also beats its counterparts on standard benchmarks by large margins. Additionally, as a proof-of-concept study, we use Ctrl-G to assist LLM reasoning on the GSM benchmark, foreshadowing the application of Ctrl-G, as well as other constrained generation approaches, beyond traditional language generation tasks.

Adaptable Logical Control for Large Language Models

Abstract

Despite the success of Large Language Models (LLMs) on various tasks following human instructions, controlling model generation to follow strict constraints at inference time poses a persistent challenge. In this paper, we introduce Ctrl-G, a neuro-symbolic framework that enables tractable and adaptable control of LLM generation to follow logical constraints reliably. Ctrl-G combines any production-ready LLM with a Hidden Markov Model (HMM), guiding LLM outputs to adhere to logical constraints represented as deterministic finite automata. We show that CtrlG, when a TULU2-7B model is coupled with a 2B-parameter HMM, outperforms GPT4 in text editing: on the task of generating text insertions/continuations following logical constraints, our approach achieves over 30% higher satisfaction rate in human evaluation. When applied to medium-size language models (e.g., GPT2large), Ctrl-G also beats its counterparts on standard benchmarks by large margins. Additionally, as a proof-of-concept study, we use Ctrl-G to assist LLM reasoning on the GSM benchmark, foreshadowing the application of Ctrl-G, as well as other constrained generation approaches, beyond traditional language generation tasks.

1 Introduction

Large language models (LLMs) have achieved remarkable performance on a wide range of challenging language generation tasks including translation [4, 48, 41], summarization [49], and open-domain creative generation [45, 38]. Nevertheless, many downstream applications benefit from fine-grained control of LLMs to follow logical constraints, e.g., avoid using bad words for detoxification [9, 1] or inserting text that is coherent with contexts for document revision [16]. Despite the recent advancement of LLM finetuning techniques such as instruction-tuning [5, 42, 35] and preference optimization [28, 33], LLMs still fail to reliably follow logical constraints [37, 20].

LLM

HMM

3. inference-time guidance

conditioning on logical constraints…

DFA

2. constraint specification

❄

❄

outputs satisfying the constraints

The major difficulty of achieving constrained generation from LLMs lies in the intractability of conditioning LLMs on logical constraints [34]. One recently proposed framework called GeLaTo [47] uses tractable generative models, which can be conditioned on logical constraints efficiently, to guide autoregressive generation from LLMs. Though GeLaTo guarantees that the logical constraints

from CtrlG import *

prefix = 'First they defeated a …'

suffix = 'are few humans left …'

dfa_list = [

DFA_all_of('alien mothership',

'far from over'),

DFA_word_count(25, 30),

]

dfa = DFA_logical_and(dfa_list)

lp = CtrlGLogitsProcessor(

dfa, hmm, prefix, suffix)

llm.generate(logits_processor=lp)

5 lines of code!

'First they've defeated a small

squad [BLANK] are few humans

left, and despite their magical

power, their numbers are

getting fewer.'

User: given the following

context, generate infilling text

for [BLANK] using key phrases

'alien mothership', 'far from

over'; generated text must

contain 25 - 30 words.

'First they've defeated a

small squad of aliens, then a

larger fleet of their ships.

Eventually they've even

managed to take down the

alien mothership. But their

problems are far from over.

There are few humans left,

and despite their magical

power, their numbers are

getting fewer.'

will be satisfied, it only works for the keyword-inclusion constraint. Significantly generalizing the GeLaTo framework, we propose Ctrl-G (shorthand for controllable generation while mimicking the keyboard shortcuts Ctrl-C and Ctrl-V) for reliable , scalable and adaptable control of LLMs to follow logical constraints. Ctrl-G consists of three major steps (see Fig. 1): (1) distillation : given a LLM, we distill a Hidden Markov Model as its white-box approximation; (2) constraint specification : we construct a deterministic finite automaton (DFA) to (compactly) represent the desired logical constraint; (3) inference : we condition the HMM on the DFA-specified constraint and compute this conditional probability to steer LLM generation towards satisfying the constraint.

Ctrl-G 3 has three major advantages compared to its counterparts: (1) the desired logical constraints are guaranteed to be satisfied [47]; (2) once we have the distilled HMM, it can be applied to arbitrary constraints without retraining; (3) Ctrl-G works for any constraints specified as DFAs, which can be easily constructed for various applications by leveraging existing algorithms.

We evaluate Ctrl-G on the task of text editing: in the domain of story writing, we evaluate models' ability to generate suggestions for text insertions/continuations under combinations of logical constraints (e.g. keyphrase inclusion and length control; see Fig. 2). Human evaluation shows that Ctrl-G, where a TULU2-7B model [13] is combined with a 2B-parameter HMM, outperforms prominent LLMs including GPT3.5 and GPT4 [27] by over 30% in overall satisfaction rate (i.e., percentage of the generated text that is not only fluent but also satisfies the constraints). We note that as the constraints become more complex, while the generation quality of GPT4 declines, Ctrl-G consistently produces high-quality text, highlighting its strong generalizability to complex constraints. Even when no constraint is present, Ctrl-G still matches with the generation quality of GPT4 in text insertion.

In addition, we demonstrate the extensive adaptability of Ctrl-G on two commonly used benchmarks: commonsense generation [18] and text infilling [7]. When applied to variants of the GPT2 models, Ctrl-G outperforms prior constrained generation approaches by producing outputs of substantially higher quality while achieving 100% constraint satisfaction.

To further explore the potential of Ctrl-G, as a proof-of-concept, we conduct an empirical study on the Grade School Math (GSM) benchmark [6]; here, we use Ctrl-G to assist the LLM reasoning process by enforcing keyphrase-inclusion constraints. Performance improvement suggests the potential of Ctrl-G in applications of a scope broader than the traditional constrained generation tasks.

2 Preliminaries

In this section, we briefly summarize the background for (logically-)constrained generation and the basics for Hidden Markov Models. Notations introduced here will be used throughout the paper.

Constrained generation For simplicity, we assume that the lengths of token sequences generated by LLMs are always bounded by some number n and denote the LLM distribution as p lm ( x 1: n ) 4 . Given

logical constraint α , our goal is to generate from p lm ( x 1: n | α ) , which decomposes autoregressively:

that is, given that we have generated the first t -1 tokens x <t , we want to generate the next token x t from p lm ( x t | x <t ) · p lm ( α | x t , x <t ) . The first term p lm ( x | x <t ) is just the next-token distribution of the LLM, but the marginal probability p lm ( α | x t , x <t ) , which characterizes how likely the constraint α will be satisfied in the future , cannot be efficiently computed; specifically,

that is, we need to marginalize over all possible future sequences x >t such that, together with x ≤ t , satisfy α . For example, say α is the constraint that the phrase 'in the park' must appear at the end of the generated text; to compute the desired marginal probability, we need to enumerate over all future token sequences with this phrase at the end, and there are exponentially many of them.

Prior work To solve the problem of constrained generation, one line of work proposes search-based decoding algorithms like NeuroLogic Decoding [22, 21], which explicitly performs heuristic search to find high-probability token sequences that would (partially) satisfy the logical constraint; however such methods scale poorly because the search space grows exponentially with respect to the sequence length. The other line of works including GeDi [15], FUDGE [44] and NADO [25] train auxiliary neural classifiers to approximate the intractable term p lm ( α | x t , x <t ) ; however, they do not guarantee that the constraints will be satisfied and the classifiers need to be retrained for different constraints. Some other methods use approximate inference techniques (e.g., sequential Monte Carlo sampling) to approximate the intractable conditional distributions [30, 11, 17], which provide no guarantee on the convergence rate and often suffer from the high-variance of sampling.

From GeLaTo to Ctrl-G A recent framework called GeLaTo [47] uses tractable generative models, in particular, Hidden Markov Models (HMMs), to guide LLM generation to satisfy the given logical constraints. Specifically, GeLaTo first (1) distills an HMM p hmm ( x 1: n ) to approximate the LLM distribution p lm ( x 1: n ) and then (2) computes p hmm ( α | x t , x <t ) as an approximation for p lm ( α | x t , x <t ) . Compared to its counterparts, GeLaTo guarantees that the constraints will be satisfied. Nevertheless, two major questions remain unanswered, limiting its downstream applications:

GeLaTo only handles the keyword-inclusion constraint and it is unclear whether p hmm ( α | x t +1 , x 1: t ) can be tractably computed for other logical constraints;

despite the success of GeLaTo on language models at the scale of 0 . 1 billion parameters, it is unclear whether the assumption p hmm ( α | x ≤ t ) ≈ p lm ( α | x ≤ t ) would still hold for the more recent LLMs (e.g., Llama2), which have over 100 times more parameters.

We propose Ctrl-G as a generalization of GeLaTo and give positive answers to both questions.

Hidden Markov Models A Hidden Markov Model (HMM) [32] represents a joint probability distribution over n observed variables x 1: n and n hidden variables z 1: n . Specifically, for language modeling, x t represents the token at position t and z t is the corresponding hidden state; z t takes values in { 1 , 2 , . . . , h } , where h is the number of hidden states . An HMM models the joint distribution:

in particular, the parameters of an HMM are given by the initial probability p ( z 1 ) , the emission matrix p ( x t | z t ) and the transition matrix p ( z t +1 | z t ) ; the number of parameters of HMMs grows quadratically with respect to h . To perform inference on HMMs efficiently, we leverage the Markov property : p ( x ≥ t | z t , x <t )= p ( x ≥ t | z t ) . For example, we can efficiently compute p ( x ≤ t ) = ∑ z t p ( x ≤ t , z t ) by the following recurrence relation, referred to as the forward algorithm [32]:

3 Tractable probabilistic reasoning over logical constraints

The Ctrl-G pipeline consists of three steps (Fig. 1): (1) distillation : we train an HMM on samples drawn from the LLM to minimize their KL-divergence; (2) constraint specification : we construct a

'gets' or 'cold'),

A

B

'gets'

'gets'

≠

'cold'

'gets'

*

'gets' or 'cold'

≠

initial state

accept state(s)

C

dfa_graph = {

'edges': [

(A, A,

'gets'),

(A, B, 'gets'),

(B, A,

(B, B, 'gets'),

(B, C, 'cold'),

(C, C, *),

],

'initial_state': A,

'accept_states': {C},

}

≠

≠

the weather gets cold in winter

A

A

A

B

C

C

C

Accepted by the DFA:

the weather gets warm in winter

A

A

A

B

A

A

A

Rejected by the DFA:

(a) DFA as graph.

A

B

'gets'

'gets'

≠

'cold'

'gets'

*

'gets' or 'cold'

≠

initial state

accept state(s)

C

dfa_graph = {

'edges': [

(A, A,

'gets'),

(A, B, 'gets'),

(B, A,

'gets' or 'cold'),

(B, B, 'gets'),

(B, C, 'cold'),

(C, C, *),

],

'initial_state': A,

'accept_states': {C},

}

≠

≠

the weather gets cold in winter

A

A

A

B

C

C

C

Accepted by the DFA:

the weather gets warm in winter

A

A

A

B

A

A

A

Rejected by the DFA:

(b) Examples of DFA state transition.

A

B

'gets'

'gets'

≠

'cold'

'gets'

*

'gets' or 'cold'

≠

initial state

accept state(s)

C

dfa_graph = {

'edges': [

(A, A,

'gets'),

(A, B, 'gets'),

(B, A,

'gets' or 'cold'),

(B, B, 'gets'),

(B, C, 'cold'),

(C, C, *),

],

'initial_state': A,

'accept_states': {C},

}

≠

≠

the weather gets cold in winter

A

A

A

B

C

C

C

Accepted by the DFA:

the weather gets warm in winter

A

A

A

B

A

A

A

Rejected by the DFA:

(c) Specifying a DFA for Ctrl-G

8 8 8 (compact) deterministic finite automaton (DFA) M representing the desired logical constraint α (i.e., M accepts x 1: n if and only if x 1: n satisfies α ); (3) inference : for each step of the autoregressive generation from the LLM, we compute p hmm ( α | x t , x <t ) as an approximation for p lm ( α | x t , x <t ) and then sample the next token from

here, given that α is represented as M ,

For step (1) (distillation) we follow the procedure proposed by [47], and we describe step (2) and step (3) in Sec. 3.1 and Sec. 3.2, respectively. In the end of this section, we briefly discuss the distinction between pure logical reasoning and probabilistic reasoning over constraints.

3.1 Logical constraints as DFAs

Deterministic finite automata (DFAs) [24, 31, 10] are computation models that accept or reject some given strings. Figure 3a shows an example DFA encoding the constraint that the phrase 'gets cold' must appear: it accepts all strings containing this phrase and rejects the others. The DFA consists of 3 different states labeled A , B and C , where A is the initial state and C an accept state . The states are connected by edges marked with sets of words (tokens, to be precise), which fully specify the transition function of the DFA. A DFA decides whether a given string satisfies the constraint by consuming it left-to-right while transitioning from state to state accordingly; in the end, the DFA accepts the string if it is in an accept state. See Figure 3b for an example.

Definition 3.1. A deterministic finite automaton (DFA) is a tuple M =( Q, Σ , δ, q 0 , F ) , where Q is a finite set of states , Σ a finite set of symbols (i.e., tokens of an LLM), δ : Q × Σ → Q a transition function , q 0 an initial state , and F ⊆ Q a set of accept states . A string of tokens w 1 w 2 . . . w n is accepted by M if there exists a sequence of states q 0 , q 1 . . . q n s.t. δ ( q i , w i +1 )= q i +1 for 1 ≤ i ≤ n, q n ∈ F .

One question naturally arises: how can we come up with DFA representations for logical constraints? We first note that in the real world, we can always assume that the lengths of the generated token sequences are bounded by a constant ; hence DFAs can represent any logical constraints defined on this bounded set and the important question is whether we can do this efficiently . For many common logical constraints, we can efficiently construct their DFA representations via existing algorithms. For example, given a string consisting of n tokens, to encode the constraint that the string must appear, we can construct a DFA of size O ( n ) by adapting the well-known Knuth-Morris-Pratt (KMP) algorithm [14] for string matching (e.g., Fig. 3a). One can also easily specify compositional logical constraints via DFAs by taking their intersection (logical and), union (logical or), complement (logical negation) or concatenation, which we illustrate throughout the rest of this paper.

3.2 An efficient algorithm for marginalizing HMMs over DFAs

Now assume that we have a constraint α encoded as a DFA M with k states Q = { 1 , 2 , · · · k } and m edges, and we are given a distilled HMM with h hidden states. To sample the next token from Eq. 1, we need to compute p hmm ( α | x t , x <t ) , which is the marginal probability over all strings accepted by M (see Eq. 2). In the following, we describe a tractable algorithm for computing this probability.

In autoregressive generation, M starts from the initial state and transitions according to the transition function as each new token is generated; we denote the state of M after sampling the first t tokens x ≤ t as s t . In addition, we use the uppercase S t to denote the random variable representing the state of M after sampling the first t tokens: e.g., S n ∈ F denotes the event that the token sequence x 1: n is accepted by M . Dropping the subscript 'hmm' from p hmm ( α | x t , x <t ) , we compute

The denominator p ( x t , x <t ) can be easily computed by the forward algorithm [32]; so we compute

the first step follows from the law of total probability and the second step follows from the Markov properties of HMMs and DFAs, as well as the fact that s t is fully determined by x ≤ t . Again, the term p ( z t , x t , x <t ) can be computed by the forward algorithm and we reduce the problem to computing the boxed term. We compute p ( S n ∈ F | z t , s t ) for all 1 ≤ t ≤ n , 1 ≤ z t ≤ h and 1 ≤ s t ≤ k via the following recurrence relation:

here edge ( s t , s t +1 ):= { w : δ ( s t , w )= s t +1 } denotes the set of tokens w that transition M from s t to s t +1 . The base case of the recurrence relation is given by p ( S n ∈ F | z n , s n )=1 if s n ∈ F and 0 otherwise. We refer readers to the appendix for its derivation. Algorithm 1 shows the pseudo-code for sampling from p ctrl-g ( x 1: n | α ) autoregressively, using the recurrence relations above.

̸

Runtime analysis of Algorithm 1. To sample from Ctrl-G, the computation overhead (i.e. in addition to the LLM inference cost) is dominated by the computation of p ( S n ∈ F | z t , s t ) for all t , z t and s t as shown in Eq. 4. Since ∑ x t +1 ∈ edge ( s t ,s t +1 ) p ( x t +1 | z t +1 ) does not depend on t, we can precompute and cache their values, resulting a one-time cost of O ( mh | Σ | ) . Then, note that for s t , we only need to consider the s t +1 where edge ( s t , s t +1 ) = ∅ . Hence, fixing t and z t , when we compute p ( S n ∈ F | z t , s t ) for all 1 ≤ s t ≤ k , we only need to (1) enumerate through 1 ≤ z t +1 ≤ h and (2) for each z t +1 , we only need to go through each edge exactly once. There are m edges in total, so it follows that the cost is O ( n · h · h · m ) = O ( nmh 2 ) . The total time complexity is O ( nmh 2 + mh | Σ | ) , which simplifies to O ( nmh 2 ) given that | Σ | <nh in practice.

3.3 Logical reasoning vs. probabilistic reasoning

Some recent work as well as open source projects have proposed to use regular expressions (regex) to achieve structured generation from LLMs [23, 43, 50]. Regex are equivalent to DFAs in terms of the logical constraints they can represent, but the aforementioned approaches only perform pure logical reasoning over regex, which is not suitable for many constrained generation tasks. For example, consider the task of generating a sentence that ends with the phrase ' in the park':

guidance [23] (logical reasoning): silhouette of suspected ... an heavily secured.in the park

Ctrl-G (probabilistic reasoning): A man and a woman are walking in the park

Even though both generations end with ' in the park', it is clear that the output from guidance is not desirable as it forcefully appends the phrase to some irrelevant text. The reason is that guidance, by performing pure logical reasoning, only discard the next tokens x t that would make α unsatisfiable, while the probabilities of the other next tokens remain unchanged; in contrast, Ctrl-G performs probabilistic reasoning by estimating p lm ( α | x t , x <t ) , i.e., we estimate how likely each next token x t would eventually lead to α being satisfied. Ctrl-G subsumes the other approaches in the sense that if we set p hmm ( α | x t , x <t ) = 1 for all non-zero values, that is, if we remove all probabilistic information, then it degenerates to pure logical reasoning.

4 Evaluating Ctrl-G on constrained generation benchmarks

4.1 Commonsense Generation

Following prior work [21, 25], we first evaluate Ctrl-G on the Commonsene Generation (CommonGen) benchmark [18]. Each test example of CommonGen provides 3 to 5 concepts (keywords) as input and the goal is to generate a natural sentence that incorporates all keywords, allowing for any of their inflections. For example, given 'car' , 'snow' and 'drive' as concepts, both 'a man drives a car on a snow covered road' and 'the car drove through the snow' are considered acceptable.

DFA construction For CommonGen, given one keyword, say, 'snow', we adapt the Aho-Corasick algorithm [2] to construct a DFA enforcing the constraint that at least one of its inflections (e.g., 'snow', 'snowing' or 'snowy') must appear. To encode the constraint that multiple keywords must appear, we can simply take the intersection of the individual DFAs [10]; see appendix for an example.

Experiments & results We use the GPT2-large checkpoint (only finetuned for domain adaptation) released by [47] as our base model and we follow the same pipeline to distill an HMM with 32768 hidden states: we sample 4M examples from the base model and train the HMM for 40 EM steps, each consisting of 100K examples. We compare Ctrl-G against FUDGE [44], NADO [25], NeuroLogic A*esque decoding [21] and GeLaTo [47]; GeLaTo uses the same base model as Ctrl-G. The results are summarized in Table 1, where the Constraint column shows the percentage of the outputs containing all concepts. Compared to all baselines, Ctrl-G achieves not only 100% constraint satisfaction rate but also substantially higher generation quality measured by automatic evaluation metrics [29, 19, 40, 3].

Runtime comparison From an algorithmic perspective, GeLaTo only handles keyword constraints hence it is a special case of Ctrl-G. Nevertheless, Ctrl-G also runs significantly faster than GeLaTo, as shown in Table 2. The GeLaTo implementation only tensorizes the HMM inference component, while the component that reasons about the constraints runs sequentially on CPU. In contrast, by representing DFAs as (weighted) adjacency matrices, Ctrl-G tensorizes the inference procedure for both HMMs and DFAs and runs on GPUs with full parallelization. Besides, both GeLaTo and Ctrl-G runs significantly faster than A*esque, which explicitly performs heuristic search.

Generalization to more keywords To evaluate the generalization performance of Ctrl-G, we construct test examples containing 6 to 9 concepts (CommonGen+): we randomly select 100 examples with 5 concepts from the dev split of CommonGen, and then augment them with additional keywords

BLEU-4

19

24

29

34

# of concepts

5

6

7

8

9

(a) BLEU-4

ROUGE-L

36

39

42

45

# of concepts

5

6

7

8

9

(b) ROUGE-L

Constraint

0

25

50

75

100

# of concepts

5

6

7

8

9

Ctrl-G

FUDGE

NeuroLogic A*

(c) Constraint satisfaction rate

sampled from their reference sentences. As shown in Fig. 4, Ctrl-G achieves 100% constraint satisfaction rate while preserving high generation quality across all settings.

4.2 Text infilling

Wealso evaluate Ctrl-G on a text infilling benchmark [7] constructed from the ROC stories corpus [26]. Each test example consists of a short story with some fragments masked out, each of a specified granularity; the goal is to fill in the masks. Here is an example: 'Jill wanted to knit her [WORD] a sweater. [SENTENCE] She finished [NGRAM] for her boyfriend's birthday. Jill was [WORD].'

DFA construction The underlying logical constraint for the task of text infilling is similar to that of CommmonGen. We can view the non-masked parts, e.g., 'Jill wanted to knit her' and 'a sweater.' from the example above, as keyphrases, and the task reduces to generating a piece of text such that all keyphrases appear in the given order. In this setting, given k text fragments, we first construct M 1 , . . . , M k using the KMP algorithm [14]; then, we concatenate them to represent the constraint that they must appear in the given order. Though DFA concatenation is intractable in general [46], we observe that the KMP DFAs can actually be concatenated in linear time. See appendix for details.

Experiments & results We use the GPT2-small checkpoint (only finetuned for domain adaptation with no supervision on the task of text infilling) released by [7] as the base model for Ctrl-G and compare against the ILM model, which is a GPT2-small model trained on this text infilling benchmark with full supervision. By applying the mask function from [7], we construct 4 test sets with different masking ratios (i.e., different percentage of masked characters) by changing the hyper-parameters. We measure the BLEU and ROUGE scores of the completed stories (i.e., including both the masked and unmasked parts) with respect to the original stories. The ILM model adopts sampling for decoding, so we run the ILM inference for 10 times to report the means and standard deviations. The results are summarized in Table 3. Based on [7], ILM is trained on a distribution with a masking ratio of approximately 15%, explaining why it achieves the best performance on the test set with 13% masking

ratio. Note that the performance gap between Ctrl-G and ILM improves almost monotonically as the masking ratio increases, again illustrating the strong generalization performance of Ctrl-G.

5 Scaling up Ctrl-G for interactive text editing

Human-AI collaborative writing has been a long studied topic in the Human-Computer Interaction (HCI) community [12, 36]. One prior work [16] proposed CoAuthor, a graphical user interface for querying LLMs to generate continuation/insertion suggestions in arbitrary positions of a document. However, when using CoAuthor to ask for LLM suggestions, users are unable to specify their preferences. We propose to extend the CoAuthor system by allowing users to have fine-grained control over the suggestions generated by LLMs: for example, users can control the topic of the generated content by instructing LLMs to incorporate certain keyphrases, and they can also ask for more concise/detailed suggestions by controlling their lengths. For this application, we apply Ctrl-G to the TULU2-7B model and compare against prominent LLMs including GPT3.5 and GPT4.

5.1 Experiment setup

Dataset construction We construct an evaluation dataset consisting of 800 test examples, each based on one story passage extracted from the CoAuthor dataset [16]. These stories are jointly written by humans and the GPT3.5-turbo-instruct model, falling under ten different topics. For each story, we randomly split it into prefix , infix and suffix ; we mask out the infix and view it as a gold reference. We consider two scenarios when evaluating the models: continuation and insertion . For continuation, we only provide prefix to the model, and the model is supposed to generate one suggestion for continuation; for insertion, we provide both prefix and suffix to the model and the model is required to generate a piece of text that is coherent with both prefix and suffix . Additionally, we consider imposing combinations of the following two constraints:

Keyphrase : suggestions should include one to three given keyphrases.

Word Count : suggestions should contain a to b words where 1 ≤ a ≤ b ≤ 32 .

We consider all combinations of the following settings: insertion or continuation, w/ or w/o keyphrase constraint, w/ or w/o word-count constraint, resulting in 8 different settings. For each setting, we sample 100 stories from the CoAuthor dataset and create 100 test examples (e.g., Fig. 2).

Scaling up Ctrl-G We adopt the TULU2-7B [13] model, which is an instruction-tuned variant of the Llama2 [39] model with 7 billion parameters, as the base model for Ctrl-G. We further finetune the base model on 3000 examples extracted from the WritingPrompt dataset [8] for the task of text continuation, following the prompt 'Continue the given text:' along with a story prefix. After finetuning, we use the same prompt to sample 5 million examples from the base model and train an HMMwith 32768 hidden states (approx. 2 billion parameters). Note that for the task of text insertion, the base model only sees the prefix , while the suffix is incorporated as a part of the constraint α ; i.e., the HMM is fully responsible for guiding the base model to generate a piece of text that will be coherent with the suffix. For generation, we sample 128 examples from p ctrl-g with temperature 0 . 7 and pick the one with the highest likelihood given by the base model as the final output.

Baselines We compare Ctrl-G against prominent LLMs including the GPT3.5 model and the GPT4 model. To generate output from the GPT models, we adopt the prompt provided by the OpenAI documentation for text insertion/continuation, with constraints specified in the instructions. See appendix for the specific prompt templates. In addition to the GPT models, we also compare Ctrl-G against pure instruction-tuning: specifically, we construct 1000 training examples for the task of text insertion based on the WritingPrompt dataset and further finetune the TULU2-7B model for text insertion, following the prompt 'Generate the text at [INSERT_TEXT] tag:\n{prefix}[INSERT_TEXT]{suffix}.' For all baselines, for the purpose of fair comparison, we generate 128 samples for each test example and select the one with the highest probability as the final output.

Human evaluation To evaluate the quality of the generated outputs, we conduct human evaluation through the Amazon Mechanical Turk (MTurk) platform. For each test example, we generate the outputs from TULU2 (prompt only), GPT3.5, GPT4 and Ctrl-G respectively, and ask annotators to rate their quality on a scale from 1 to 5. For each test example, we present the generated outputs from all models, along with their original context, to the annotators side-by-side and ask them to evaluate their quality; specifically, we ask the annotators to answer the following questions:

Q1. is the paragraph coherent and grammatically correct?

Q2. is the paragraph consistent and semantically reasonable?

Q3. based on your answers to Q1&Q2, what is your rating for the overall quality?

Note that we only ask human annotators to evaluate the coherency and fluency of the generated text and they are not aware of the required logical constraints. We ask three annotators to evaluate each output and compute their inter-annotator agreement score. See appendix for more details.

5.2 Results

The evaluation results are summarized in Table 4, showing the quality score, 5 constraint satisfaction rate, and overall satisfaction rate. In particular, the overall satisfaction rate denotes the percentage of test examples that (1) satisfy the constraint and (2) attain average quality scores > 3 . For continuation, in terms of generation quality, GPT4 beats all other models; this is no surprise, as gigantic models like GPT3.5 (with 175B parameters) and GPT4 have significant advantage in generating high quality text continuations. However, despite the high generation quality, the success rates for GPT3.5 and GPT4 are relatively low (the highest 59%) while Ctrl-G always satisfy the specified constraints; hence in terms of the overall satisfaction rate, Ctrl-G beats all baselines by large margins when constraints are present. For the case of insertion, the 'implicit' soft constraint here is that the generated parts need to be coherent with the given suffix, which is challenging for autoregressive models; in this case, in terms of pure generation quality, Ctrl-G beats/matches with the performance of GPT4 in all settings; for insertion, the success rate of all baselines becomes even lower compared to continuation, while Ctrl-G achieves 100% success rate in all settings. In terms of overall satisfaction rate, Ctrl-G again beats all baselines. The other observation is that the generation quality of GPT4 decreases as the logical constraints become more complex, while the generation quality of Ctrl-G stays relatively consistent across all settings, demonstrating strong generalization performance.

5.3 Runtime analysis

We provide an empirical analysis on the runtime of Ctrl-G, with TULU2-7B as the base model. In addition to the computation cost of the base LLM, the major cost of Ctrl-G lies in the computation of p hmm ( α | x ≤ t ) , with a time complexity of O ( nmh 2 ) (Thm. 3.2); here n is the maximum sequence length, m is the size (i.e. # of edges) of the DFA, and h is the number of HMM hidden states. First, fixing the sequence length n , we change the size of the DFA and verify that the time for generating each token scales roughly linearly with respect to the DFA size (Fig. 5 left). Then, fixing a DFA of

time/token (seconds)

0

1/400

1/200

3/400

1/100

DFA size

0

450

900

1350

1800

Ctrl-G (HMM-4096)

Ctrl-G (HMM-32768)

time/token (seconds)

0

1/400

1/200

3/400

1/100

sequence length

0

120

240

360

480

600

TULU2-7B

Ctrl-G (HMM-32768 + TULU2-7B)

time/token (seconds)

0

1/400

1/200

3/400

1/100

DFA size

0

450

900

1350

1800

Ctrl-G (HMM-4096)

Ctrl-G (HMM-32768)

time/token (seconds)

0

1/400

1/200

3/400

1/100

sequence length

0

120

240

360

480

600

TULU2-7B

Ctrl-G (HMM-32768 + TULU2-7B)

size ≈ 900 , we change the sequence length n and measure the time for generating each token from Ctrl-G and the base LLM respectively. The gap between the two lines in Fig. 5 (right) shows the computation overhead introduced by Ctrl-G, which stays constant with respect to the sequence length. On the other hand, however, due to the attention mechanism, the time for generating each token from the base LLM scales linearly with respect to n . Hence, the computation cost will be dominated by the base model when generating long sequences. The runtime measurements are conducted on an NVIDIA-A100 GPU with 80GB memory.

6 Perspectives: improving LLM reasoning abilities via logical constraints

In this section, we explore the use of Ctrl-G on a non-traditional constrained generation application. As a case study, we apply Ctrl-G to assist the reasoning process of the TULU2-7B model on the grade school math (GSM) benchmark. As we naively apply chain-of-thought prompting, we observe that for 293 out of the 1319 test examples, the model fails to use all numbers provided in the problem statement; this leads to a much lower accuracy on the 293 examples compare to that on the complete test set. For such 293 test examples, we apply Ctrl-G to the TULU2-7B model to enforce the constraint that all numbers from the problem statement must be generated as part of the chain-of-thought reasoning process. We sample 16 outputs from the TULU2-7B model and do a majority vote; with Ctrl-G, the model achieves 28.3% accuracy, which is 3.4% higher than the marjoity-vote accuracy without Ctrl-G.

1 1 Our proof-of-concept study on the GSM benchmark illustrates one potential use case of Ctrl-G beyond traditional language generation tasks. Specifically, we demonstrate the possibility of 'approximating' soft control (i.e., better reasoning ability in this setting) via logical constraints. For future work, we motivate the application of Ctrl-G, as well as other constrained generation approaches, on a broader scope of downstream tasks: e.g., helping LLM detoxification by conditioning on a set of bad words/phrases not appearing, improving the reasoning ability of LLMs by conditioning on generating longer reasoning sequences, and controlling the topic of the generated content by conditioning on the occurrence of certain keyphrases.

7 Conclusion

We propose Ctrl-G, a versatile framework that enables reliable and flexible inference-time control of LLMs; given any production-ready LLM, Ctrl-G distills an HMM as its approximation and uses it to guide the LLM to generate outputs that comply with any logical constraints specified as DFAs. We show that Ctrl-G, where a 7B-parameter TULU2 model is combined with a 2B-parameter HMM, beats significantly larger LLMs like GPT4 on the task of generating text insertions/continuations with logical constraints. On commonly used constrained generation benchmarks like CommonGen, Ctrl-G beats other constrained generation approaches, as well as supervised training, by large margins. In addition to the dominant paradigm of prompt engineering, our work opens up new avenues for achieving tractable, reliable and fine-grained inference-time control of LLMs.

Acknowledgments

This work was funded in part by the DARPA ANSR program under award FA8750-23-2-0004, the DARPA PTG Program under award HR00112220005, NSF grant #IIS-1943641, NSF CAREER Award #2339766, and AFOSR MURI via grant #FA9550-22-1-0380.

References

Kareem Ahmed, Kai-Wei Chang, and Guy Van den Broeck. A pseudo-semantic loss for autoregressive models with logical constraints. Advances in Neural Information Processing Systems , 36, 2024.

Alfred V Aho and Margaret J Corasick. Efficient string matching: an aid to bibliographic search. Communications of the ACM , 18(6):333-340, 1975.

Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In European Conference on Computer Vision (ECCV) . Springer, 2016.

Yujin Baek, Koanho Lee, Dayeon Ki, Hyoung-Gyu Lee, Cheonbok Park, and Jaegul Choo. Towards accurate translation via semantically appropriate application of lexical constraints. arXiv preprint arXiv:2306.12089 , 2023.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetuned language models, 2022.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.

Chris Donahue, Mina Lee, and Percy Liang. Enabling language models to fill in the blanks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 2492-2501, 2020.

Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation, 2018.

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 3356-3369, Online, November 2020. Association for Computational Linguistics.

John E Hopcroft, Rajeev Motwani, Rotwani, and Jeffrey D Ullman. Introduction to automata theory, languages and computability, 2000.

Edward J Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models. In The Twelfth International Conference on Learning Representations , 2023.

Daphne Ippolito, Ann Yuan, Andy Coenen, and Sehmon Burnam. Creative writing with an ai-powered writing assistant: Perspectives from professional writers. arXiv preprint arXiv:2211.05030 , 2022.

Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a changing climate: Enhancing lm adaptation with tulu 2. arXiv preprint arXiv:2311.10702 , 2023.

Donald E Knuth, James H Morris, Jr, and Vaughan R Pratt. Fast pattern matching in strings. SIAM journal on computing , 6(2):323-350, 1977.

Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 4929-4952, 2021.

Mina Lee, Percy Liang, and Qian Yang. Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. In CHI Conference on Human Factors in Computing Systems , CHI '22. ACM, April 2022.

Alexander K Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K Mansinghka. Sequential monte carlo steering of large language models using probabilistic programs. arXiv preprint arXiv:2306.03081 , 2023.

Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 1823-1840, Online, November 2020. Association for Computational Linguistics.

Chin-Yew Lin and Eduard Hovy. Automatic evaluation of summaries using n-gram cooccurrence statistics. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) , 2003.

Albert Lu, Hongxin Zhang, Yanzhe Zhang, Xuezhi Wang, and Diyi Yang. Bounding the capabilities of large language models in open text generation with prompt constraints, 2023.

Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, et al. Neurologic a* esque decoding: Constrained text generation with lookahead heuristics. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 780-799, 2022.

Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. NeuroLogic decoding: (un)supervised neural text generation with predicate logic constraints. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 4288-4299, Online, June 2021. Association for Computational Linguistics.

Scott Lundberg, Marco Ribeiro, Richard Edgar, and Harsha-Nori. Guidance: a guidance language for controlling large language models., 2024.

Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics , 5:115-133, 1943.

Tao Meng, Sidi Lu, Nanyun Peng, and Kai-Wei Chang. Controllable text generation with neurally-decomposed oracle. In Advances in Neural Information Processing Systems 35 (NeurIPS) , 2022.

Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 839-849, 2016.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve

Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics (ACL) , 2002.

Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-based constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems , 35:9538-9551, 2022.

Michael O Rabin and Dana Scott. Finite automata and their decision problems. IBM journal of research and development , 3(2):114-125, 1959.

Lawrence Rabiner and Biinghwang Juang. An introduction to hidden markov models. IEEE ASSP Magazine , (1), 1986.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2023.

Dan Roth. On the hardness of approximate reasoning. Artificial Intelligence , 1996.

Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization, 2022.

Shuming Shi, Enbo Zhao, Wei Bi, Deng Cai, Leyang Cui, Xinting Huang, Haiyun Jiang, Duyu Tang, Kaiqiang Song, Longyue Wang, et al. Effidit: An assistant for improving writing efficiency. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , pages 508-515, 2023.

Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Frederick Wieting, Nanyun Peng, and Xuezhe Ma. Evaluating large language models on controlled generation tasks, 2023.

Yufei Tian and Nanyun Peng. Zero-shot sonnet generation with discourse-level planning and aesthetics features. arXiv preprint arXiv:2205.01821 , 2022.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2015.

Ke Wang, Xin Ge, Jiayi Wang, Yu Zhao, and Yuqi Zhang. Easy guided decoding in providing suggestions for interactive machine translation. arXiv preprint arXiv:2211.07093 , 2022.

Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, and Daniel Khashabi. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks, 2022.

Brandon T Willard and Rémi Louf. Efficient guided generation for large language models. arXiv e-prints , pages arXiv-2307, 2023.

Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) , 2021.

L Yao, N Peng, W Ralph, K Knight, D Zhao, and R Yan. Plan-and-write: Towards better automatic storytelling, association for the advancement of artificial intelligence. Cited on , page 92, 2019.

Sheng Yu, Qingyu Zhuang, and Kai Salomaa. The state complexities of some basic operations on regular languages. Theoretical Computer Science , 125(2):315-328, 1994.

Honghua Zhang, Meihua Dang, Nanyun Peng, and Guy Van den Broeck. Tractable control for autoregressive language generation. In International Conference on Machine Learning , pages 40932-40945. PMLR, 2023.

Jinpeng Zhang, Nini Xiao, Ke Wang, Chuanqi Dong, Xiangyu Duan, Yuqi Zhang, and Min Zhang. Disambiguated lexically constrained neural machine translation. arXiv preprint arXiv:2305.17351 , 2023.

Yusen Zhang, Yang Liu, Ziyi Yang, Yuwei Fang, Yulong Chen, Dragomir Radev, Chenguang Zhu, Michael Zeng, and Rui Zhang. Macsum: Controllable summarization with mixed attributes. Transactions of the Association for Computational Linguistics , 11:787-803, 2023.

Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. arXiv preprint arXiv:2312.07104 , 2023.

A Derivation of Eq. (4)

B DFA operations

≠

≠

≠

M

1

M

2

A

B

'gets'

'gets'

'cold'

'gets'

*

'gets' or 'cold'

≠

C

D

E

'winter'

'winter'

≠

*

CD

AD

BD

'gets'

'cold'

'gets'

AE

BE

'gets'

'cold'

CE

'winter'

'gets'

'winter'

'winter'

A

B

'gets'

'gets'

'cold'

'gets'

D

E

'winter'

'winter'

≠

M

1

M

2

A

B

'gets'

'gets'

'cold'

'gets'

*

'gets' or 'cold'

≠

C

D

E

'winter'

'winter'

≠

*

CD

AD

BD

'gets'

'cold'

'gets'

AE

BE

'gets'

'cold'

CE

'winter'

'gets'

'winter'

'winter'

A

B

'gets'

'gets'

'cold'

'gets'

D

E

'winter'

'winter'

≠

A

B

'gets'

'gets'

'cold'

'gets'

*

'gets' or 'cold'

≠

C

D

E

'winter'

'winter'

≠

*

CD

AD

BD

'gets'

'cold'

'gets'

AE

BE

'gets'

'cold'

CE

'winter'

'gets'

'winter'

'winter'

A

B

'gets'

'gets'

'cold'

'gets'

D

E

'winter'

'winter'

≠

M

1

M

2

(a) Two DFAs M 1 and M 2

(b) Intersection of

M

1

and

M

2

≠

≠

≠

(c) Concatenation of

M

1

and

M

2

9 9 9 Proposition B.1. Let M 1 be a DFA such that for each accept state s , δ ( s, w ) goes to a dead state for all w ∈ Σ . Then M 1 can be concatenated with any other DFA M 2 by merging the accept states of M 1 with the initial state of M 2 ;

here a dead state denotes a DFA state that is (1) not an accept state and (2) only transitions to itself. Instead of formally defining what it means by 'merging' the initial state of M 2 with the accept states of M 1 , we refer readers to Figure 3c for such an example.

C Human evaluation

Table 5 presents the aggregated results for all questions from the Human Evaluation. Each question was answered by three workers, and we compute their inter-annotator agreement. Each worker evaluated the outputs generated by four different models for the same prefix (and suffix) within each batch. We converted these evaluations for each batch into rankings and then used the Kendall Coefficient of Concordance to assess the correlation between the rankings assigned by each worker. The average coefficient was 0.449, indicating a moderate level of agreement among the annotators.

You will be presented with short paragraphs (A, B, C and D) extracted from some stories. Read each paragraph and rate them by answering the following questions.

Paragraph A

Dad, the ultimate judge and jury, is now being judged. Based on dad's values? Of course not! Why would anyone put their own needs and desires asideto stroke dad's ego? It is high time dad's It is high time dad's brainstorm a new approach tohandle this growing trend of judgment.idea howto handle this growing trend of judgement.

Paragraph B

Dad, theultimate judge and jury, isnowbeing judged. Based on dad's values?Of course not!Why would anyone put their own needs and desires aside to stroke dad's ego? It is high time dad's values be judged? idea how to handle this growing trend of judgement.

Questions

Fluency: Is Paragraph A well-formed and fluent?

O 5: Yes, it is well-formed and fluent.

3:Somewhat,there areafewinterruptions,butunderstandable.

O4:Between 3 and5

2:Between 1and3

O 1: No, the writing is completely broken.

Coherence: Is Paragraph A consistent and logical?

5: Yes,it is consistent and logical.

3:omewhat,thereareafewroughtransitions,butverall

O 4:Between 3 and 5

consistent.

O 1: No, the story makes no sense.

O2:Between 1 and 3

(3)Overall:Based onyour answers to (1) and (2),isParagraphA overall well-written and logical?

5: Perfect, it is well-written and logical.

3:Okay,it has some issues,but is verallacceptable.

4:Between 3 and 5

O2:Between 1and3

O 1: Poor, the writing is broken and the story makes no sense.

Questions

Fluency: Is Paragraph B well-formed and fluent?

O 5: Yes, it is well-formed and fluent.

O3:Somewhat,thereareafewinterruptions,butunderstandable.

O4:Between 3and 5

2:Between 1and3

O1: No, the writing is completely broken.

Coherence: Is Paragraph B consistent and logical?

5: Yes, it is consistent and logical.

O3:Somewhat, there are a fewrough transitions,but overall

O4:Between3 and5

consistent.

O 1: No, the story makes no sense.

2:Between 1 and 3

Overall: Based on your answers to (1) and (2), is Paragraph B overall well-written and logical?

O 5: Perfect, it is well-written and logical.

3: Okay,ithas some ssues,but is overallacceptable.

O 4: Between 3 and 5

2:Between1and3 O1:Poor,the writing is broken and the story makes no sense.

Table 6: Prompt templates for querying the GPT3.5 and GPT4 models on the task of text editing.

Continuation:

Below is the opening of a story. Continue the narrative by writing the next few sentences that includes the specified keywords. Your continuation should naturally follow the themes, tone, and setting established in the opening. Aim to write a compelling and coherent continuation that could lead the story forward. Your answer must consist of at least (WordRangeStart) words and no more than (WordRangeEnd) words. Please make sure to incorporate the given keywords in to your answer. Keywords: (Keyword).

Story: (Prefix)

Insertion:

Fill in the text at the [INSERT] in the following story with an appropriate sentence that includes the specified keywords. Feel free to use your knowledge, guesses, or interpretations to craft your answer, but ensure it is relevant to the context provided by the prefix and suffix. Your answer must consist of at least (WordRangeStart) words and no more than (WordRangeEnd) words. Please make sure to incorporate the given keywords in to your answer. Keywords: (Keyword). Story: (Prefix)[INSERT](Suffix)

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract accurately summarize the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: It is clearly stated that our proposed method only works for constraints that can be specified as DFAs.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Proof for Theorem 3.2. is given in the main paper and the full derivation of Equation (4) is presented in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The code for reproducing the main results is released on github and the model checkpoints are released on huggingface hub.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code for reproducing the main results is released on github and the model checkpoints are released on huggingface hub.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All hyper-parameters are given in the released code repository.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: For human evaluation, we computed inter-annotator agreement scores. For the text infilling experiments, which involve sampling, we reported standard deviation of the evaluation results over multiple runs.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The spec of the GPUs used for running our experiments is given.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our application has no non-trivial societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release risky models or data.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets used in the paper are properly credited. Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The released code repository is well-documented.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: See appendix for screenshots.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We did not collect any personal information about the human annotators and they have never been exposed to any form of risks.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an approach called Ctrl-G to control LLM generation, specifically, constraining LLM’s output to deterministically follow certain logical constraints, such as maintaining a certain keyword in the generated text.\nThe approach has two main parts, the first is a Hidden Markov Model (HMM) that serves as a “prediction” model to guide the generation. The data used to train the HMM is sampled from the LLM. Essentially, the HMM is like a distillation model of the LLM that tries to capture the next-token generation “search space” of the LLM on a specific task. The second part is a series of deterministic finite automata (DFAs), each designed to represent a certain logical constraint $\\alpha$. The DFAs serves as a “checker” to determine the acceptance or rejection of an output from LLM. Combining the HMM and the DFAs, the proposed approach computes the conditional probability $P_hmm(\\alpha | x_t, x_<t)$, i.e., how likely $x_t$ is leading to $\\alpha$ being satisfied, to sample the next token.\n\nSTRENGTHS:\nThe idea of using a HMM model to approximate a white-box model for exploring the LLM’s token generation space is novel. Despite that the idea of using HMM was published in a previous work [Zhang et al. Tractable control for autoregressive language generation], this paper proposes to improve by using DFAs for modeling various other task-specific logical constraints.\n\nWEAKNESSES:\nThe evaluation requires more details and/or more experimental results to justify the contribution of the proposed approach. The limitation of the proposed approach is not clearly laid out and requries more discussions. Please see questions and limitations below.\n\nQUESTIONS:\nIn section 4, the proposed approach is compared with FUDGE [Yang and Klein, 2021], NeuroLogic A*esque decoding [Lu et al., 2022], and ILM model [Donahue et al. 2020]. The background and motivation of comparing with these baseline methods is not clear. Why choose these models for comparison? \n\nIn section 5, the proposed approach is compared with plain GPT-3.5-Turbo and GPT4-Turbo. The comparison should also be made using GPT models and the proposed DFAs method as an output checking method. In fact, there are other simpler constrained parsing techniques, for example, the method used in [Constrained Language Models Yield Few-Shot Semantic Parsers](https://aclanthology.org/2021.emnlp-main.608) (Shin et al., EMNLP 2021). \n\nIn section 5, the HMM is distilled from TULU-2-7B model. Is there a specific reason for choosing this model? A 7B model seems relatively small. The llama-2 series and Google Gemma has open-sourced models with larger (> 20B) parameter sizes. \n\nSince distillation of a HMM is the key in this approach, the HMM’s training quality should be a concern. The paper mentioned that the HMM was trained on millions of samples generated from a baseline LLM (not fine-tuned on the task-specific data), but if those samples are of bad-quality, how can the quality of the obtained HMM be assured? \n\nThe paper claimed a 100% constraint satisfaction achieved with the proposed approach. Following the previous three questions, it is unclear if this 100% is achieved with the HMM model or the constrained parsing with DFAs. The proposed approach would be more valuable if the author can demonstrate the contribution is mainly from the HMM.\n\nLIMITATIONS:\nAs mentioned in the questions, the experiment only shows the HMM being trained on a 7B model. Is it possible to generalize the approach to larger models, eventually to be used with models with trillions of parameter like GPT? For example, is the sampling amount a barrier for distillation from larger models? If so, what is the size limit here? More discussion should be made on the benefits of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
This paper proposes Ctrl-G to enable flexible lexically constrained generation with high accuracy and efficiency. Ctrl-G first distills an HMM from an unconditioned language model and then formulates the lexical/logical constraints with deterministic finite automata (DFAs). The inference algorithm takes both the HMM and the DFA as input and computes the desired conditional probability for guiding 48 LLM generation towards satisfying the given constraints. Experiments in keyphrase generation, text infilling, interactive text editing, and mathematical reasoning indicate that Ctrl-G achieves competitive performances for various sizes of LMs.

STRENGTHS:
1. The proposed method is flexible and manageable for constrained generation. It achieves competitive performance on various tasks, and it can scale up to control larger models with high efficiency.
2. The proposed method is theoretically sound and is clearly presented. The illustrations help in understanding the DFA formulation.

WEAKNESSES:
1. The experimental comparison for small model tasks is relatively weak. The paper discussed several related works in the introduction and method part (e.g., GeLaTo and NADO), but the works are not included in the experiments. The evaluation metrics BLEU and ROUGE-L measure text overlaps and cannot comprehensively represent the generation quality. Qualitative results are not provided.
2. The experiment part of the paper is a little bit hard to follow. Section 5 and 6 represent key improvements, but the experimental results are displayed in the appendix (Table 4 and 5). Some sentences/sections are not finished.

QUESTIONS:
See weakness.

LIMITATIONS:
The authors discuss limitations in the experiment sections but do not discuss potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
It is a challenge to control LLMs' generation to adhere to logical constraints. Ctrl-G integrates LLMs with a Hidden Markov Model (HMM) and deterministic finite automata (DFAs) for flexible and tractable control, such as keyword and length constraints. Ctrl-G combined with TULU-2-7B model outperforms GPT-3.5 and GPT-4 in human evaluations for interactive text editing, achieving a 30% higher satisfaction rate and 100% constraint satisfaction. Additionally, Ctrl-G improves performance on the Grade School Math (GSM) dataset by guiding the reasoning process with logical constraints.

STRENGTHS:
1. The paper presented an interesting method on logically-controlled text generation.
2. Comprehensive experimental results and analysis are provided. The results are strong and the findings are backed by well-designed human evaluation. Runtime analysis on efficiency of the method is provided. 
3. The paper is well-written and presented.

WEAKNESSES:
There are no major weaknesses.

QUESTIONS:
1. Can the framework potentially handle highly creative tasks that might require more fluid and less structured outputs?
2. Can Ctrl-G potentially generalize to domains outside of those tested, such as scientific text generation or code synthesis? No need to provide results, but insights on this would be helpful.

LIMITATIONS:
Yes, the authors adequately addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the problem of generating sequences from llms while following a constraint \\alpha, providing a solution for the case where \\alpha can be represented as a regular language (e.g., a constraint on containing certain substrings).  The method allows application of any regular-language constraint, and is somewhat similar to/inspired by a previously proposed method, Gelato, in which the LLM is distilled into an HMM.\n\nThe method is evaluated with domain-finetuned TULU-7B language models, and compared to some other constrained sequence generation methods, FUDGE and Neurologic A* esque decoding: these results are favourable. It is also compared to baselines such as directly using very large language models (e.g. GPT4), with the constraint given simply as part of the input. These results are unfortunately missing (the appendix is missing), but apparently GPT4 struggles to follow constraints (whereas ctrl-G always follows the constraints, by design). It is not clear to me whether the method was compared to Gelato.\n\nA small investigation is done to see whether constrained decoding can help when solving math problems, specifically by forcing the solution to contain all numbers from the question, and the method is deployed on the GSM benchmark. The performance is improved relative to not using the method, but not compared to other methods.\n\nSTRENGTHS:\n1. A method for constrained generation that always follows the constraints, by construction\n2. The method possibly allows richer constraints than a previous one? Unfortunately I am not entirely sure about this, it is not clearly enough stated in the work\n3. Comparison with some previous methods\n\nWEAKNESSES:\n1. Weak evaluation (see questions/comments), in particular no comparison to gelato (?), no inference time comparison ( I get the impression that inference with this method is very slow), key results put in appendix, GSM experiment more a demonstration of usefulness of constrained decoding than of ctrl-G in particular (no comparison to other other constrained decoders)\n2. Some hard to follow details and overview. Missing in particular: 1) a complete description of how inference is done in this framework, and 2) more detailed description of differences and similarities between this method and Gelato (both use an HMM distilled from the LLM, but Gelato then uses only that while this method still uses the LLM as well - why does this method have both an LLM and its distilled HMM - I will appreciate more intuition on what is happening here). Also, it is not clear how/whether this method differs from gelato with respect to inference complexity (time) or potential constraint space (i.e. type of constraints that can be encoded)\n3. Confusing amount of attention given to problem of expressing \"regular\" constraints (e.g., contains/doesnt contain substring, has maximum length 3) as DFAs, I would expect this is trivial, but somehow the authors treat it as not, referencing multiple algorithms and dedicating several paragraphs and figures (e.g. 3: example of a DFA, 4: operations on DFAs) to it. 2 figures and over 40 lines (127-135, 176-196, 222-235) dedicated to question of creating DFAs for rather simple constraints, all would be better spent on describing main alg and bringing in main results (which are currently in missing appendix).\n\nQUESTIONS:\nquestions/comments:\n\n0. several references made to materials in appendix, but no appendix! even if it were there, main materials should not be in appendix - pseudo code and greater description of ctrl g would be much more appreciated in main, as well as several results referred to from main text!\n1. figure 2 never referenced in main text\n2. lines 38-39 unfinished/mangled sentence\n3. line 39-40: possibly unclear sentence? it reads like gelato provides multiple algorithms in general, but for the case of keyword constraints it only provides one, in which case it is unclear what this alg is, what the situation is for other types of constraints, and whether ctrl-G covers the same or a larger set of constraints as can be represented with gelato. generally, the comparison between ctrl-g and gelato is not clear enough on the similarities and differences and in fact on what gelato is exactly. end of this sentence (in line 41) also unclear.\n4. lines 87-91: it seems (also from lines 39-41) that the paper faults Gelato for only providing one algorithm for handling certain constraints. but doesn't this paper also have only one algorithm? this criticism doesnt make sense\n5. lines 87-91: gelato is critiqued for possibly not being efficiently computable, but my impression is that this is true for ctrl-g too (with complexity nmh^2, where h is the distilled HMM size and reaches 32k even in this paper). can you compare the two explicitly?\n6. lines 87-91: the claim that it is unclear whether gelato would scale - why is that? is it because of the distillation to an hmm, and an assumption that this will become a worse approximation as the original model grows larger? if so, does that drawback not hold also for this method?\n7. lines 99-111: the description says a DFA is used to encode the constraint, but the DFA does not seem to appear in the equations, and it is not clear how it factors in\n8. line 121 typo: \"fo\"\n9. line 125 consider making it clearer that q0 here is \"the\" q0\n10. 127-135 this feels overly mystical for the problem of designing a dfa to express simple constraints. Unless I am missing something (in which case I would appreciate explanation), I think the whole discussion of designing DFAs for the constraints does not need more details than: \"this method allows for any constraint that can be expressed with a DFA\", with maybe a reference to textbooks on dfas if someone is not experienced. similarly figs 3 and 4 are not necessary.\n11. line 138 is this DFA only partial? if not, surely m=k * |\\Sigma| - why not just write that?\n12. 146 what subscript?\n13. 146-151 here i could do with being walked through what is happening in simpler terms/more detail/justifications/something - this bit is harder for me\n14. thrm 3.2 - give some justificution. did you compute this? (if so, prove). is it taken from a paper? if so, refer.\n15. line 166: \"we set\": i get what youre saying but rephrase because you cant set this distribution as you please. maybe you mean used 0/1 or only evaluated whether or not phmm(a|..)>0\n16. section 4 again bewildering amount of space dedicated to design of dfas for ultimately very simple constraints. line 170, and then lines 176-196 (also, 186-189 seem to be a definition not a theorem), all for this simple task. Also, if must refer to various algorithms (e.g. Aho-Corasick, line 182), then need to also provide relevant sources (ie cite).\n17. lines 202-204 im a bit confused: if each epoch has its own samples, what separates epochs? i dont understand. i see similar phrasing used in gelato but i dont follow this. are you changing anything between epochs? if not, this sounds like 1 epoch over 4m samples? if yes, explain what is being changed\n18. line 226 \"inflection of keywords\" - huh?\n19. 222-235 (and 235-241 kind of): more space spent on needless discussion of dfa operations. i respect that maybe sometimes these operations are computationally expensive but it seems all constraints considered in this paper do not bump against this anyway. it would make more sense to mention this only if it somehow limited the experiments or practical applications of the method.\n20. line 231: \"dead states\" not been defined\n21. line 259: seems like a fairer comparison against ILM would have it trained with the right percentage of masked out tokens each time, or even a varying percentage if want to see how it performs when only allowed one model\n22. line 300 \"word range\" -  maybe you mean sentence length range? word count range?\n23. line 331: saying ctrl-G is a 7B param model: you are ignoring the parameter count of the distilled HMM, which seems to me quite large (32k states!). What is the vocabulary size of Tulu2? If I assume 50k (i.e. similar to GPT2), that's 32 * 50M = 1.6B parameters for the emission matrix, and another 32 * 32 M = ~1B parameters for the transition matrix.\n24. line 339-341 hard to understand what being compared to what, rephrase\n25. 350-352: everything following \"we can\" - it is possible, but this is speculation, and not backed by the results, so be more careful in presentation.\n\nLIMITATIONS:\nweak eval - missing comparisons and results", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-18"}