COLLIE: Systematic Construction of Constrained Text Generation Tasks
Authors: Shunyu Yao
Howard Chen
Austin W. Hanjie
Runzhe Yang
Karthik R Narasimhan
Venue: ICLR 2024 poster (2024)
Keywords: constrained text generation
large language models
compositional benchmark
Primary area: datasets and benchmarks
TL;DR: We propose a new framework for the systematic construction of challenging instances for constrained text generation.

Abstract: Text generation under constraints have seen increasing interests in natural language processing, especially with the rapidly improving capabilities of large language models. However, existing benchmarks for constrained generation usually focus on fixed constraint types (e.g. generate a sentence containing certain words) that have proved to be easy for state-of-the-art models like GPT-4. We present COLLIE, a grammar-based framework that allows the specification of rich, compositional constraints with diverse generation levels (word, sentence, paragraph, passage) and modeling challenges (e.g. language understanding, logical reasoning, counting, semantic planning). We also develop tools for automatic extraction of task instances given a constraint structure and a raw text corpus. Using COLLIE, we compile the COLLIE-v1 dataset with 1,132 instances comprising 13 constraint structures. We perform systematic experiments across five state-of-the-art instruction-tuned language models and analyze their performances to reveal shortcomings. COLLIE is designed to be extensible and lightweight, and we hope the community finds it useful to develop more complex constraints and evaluations in the future.

Published as a conference paper at ICLR 2024
COLLIE: SYSTEMATIC CONSTRUCTION OF CONSTRAINED TEXT GENERATION TASKS
Shunyu Yao" Howard Chen" Austin W. Hanjie*
Runzhe Yang" Karthik Narasimhan
Department of Computer Science, Princeton University (shunyuy, hc22, hjwang, runzhey, karthikn @pri innenonnnn... edu
ABSTRACT
With the rapid improvement of large language models capabilities, there has been increasing interest in challenging constrained text generation problems. However, existing benchmarks for constrained generation usually focus on fixed constraint types (e.g- generate a sentence containing certain words) that have proved to be easy for state-of-the- based framework that allows the specification of rich, compositional constraints with diverse generation levels (word, sentence, paragraph, passage) and modeling challenges (e.g. language understanding, logical reasoning, counting, semantic planning). We also develop tools for automatic extraction of task instances given a constraint structure and raw text corpus. Using COLLIE, compile the COLLIE- vI dataset with 2,080 instances comprising 13 constraint structures. We perform systematic experiments across five state-of-the-ar instruction-tuned language mod- els and analyze their performances to reveal shortcomings. COLLIE is designed be extensible and lightweight, and we hope the community finds useful develop more complex constraints and evaluations in the future.
models like GPT-4. We present COLLIE, grammar-
INTRODUCTION
Large language models (LLMs) are increasingly capable of generating coherent and fluent text when provided with high-level prompts (OpenAl 20033))), Such capabilities have raised the bar for automated text generation, allowing us to explore more nuanced ways of utilizing LMs. One such
line of inquiry is constrained text generation, whereby the LM is asked to adhere to particular topic (Keskar et al. 2019, Dathathri et al. 2020). or avoid using certain words (Lu et al. 2021 2022). However, these works seratch the surface of broader phenomenon text, as evidenced by their use in more structured tasks like problem solving (Yao et al. 2022). code
LMs do not just generate
generation (Chen al. 2022b) and even tool through API calls (Schick al. 2023).
This raises a natural question - 'what is the next iteration of text generation benchmarks that can evaluate these advanced capabilities in LLMs'? We posit that one direction incorporating logical and compositional challenges via constrained text generation. Existing benchmarks for constrained generation, however, focus only on particular constraint types, require tailored pipelines collect data and annotations, and/or can only evaluate a specific aspect of LM strengths (Lin et al. 2020 Chen et al. 2022a). They also suffer from challenges in scalable dataset construction.
In this paper, we propose COLLIE, a grammar grasmar-baseed framework that enables systematic construction of compositional constraints over diverse generation levels (e.g., words, sentences, paragraphs) and semantic requirements (e.g., language understanding, logical reasoning, counting). Operationally, COLLIE allows researchers to 1) easily specify constraint templates, and then automatically 2) extract constraint values from language corpora, 3) render them into natural language instructions, and 4) evaluate model generations against the constraint instructions.
Existing benchmarks for constrained generation focus only particular constraint types and formats (e.g., "generate a sentence with words..."). These limitations mean that benchmarks become quickly obsolete LLMs progress. In contrast, the modular and extensible design of COLLIE allows the
'Equal contribution. Project site with code and data: https: //cc//cc// 1ill1eee1 Collie is herding dog that help guide domesticated animals like llamas and alpacas.
wetimmarrk.itttii. io

---

Published as a conference paper at ICLR 2024
Constraint Grammar
count(f, word, 9) S A pos(f, word -3)
Instruction Rendering
Generate sentence with most 10 words where the third last word 'mankind'.
53.1
$3.3
Extraction $4.1
Conformance
Model
United stand, humanity overcome any challenge for the betterment words) Figure I: Our COLLIE framework for constraint structure specification, ground truth extraction, instruction rendering, and evaluation. First, the user specifies the constraint structure without specific target value (expressed in *). Second, the constraint structure is used to extract ground truth examples from text corpora that contain the target values. Third, the constraint structure and target values rendered into natural language instruction. Finally, the model's generation is evaluated against the constraint and the ground truth. The model (gpt- -3. 5-turbb)) violates the constraints by exceeding word limits and leaving the word 'mankind' the end instead of the specified position.
Evaluation $5
The only thing that will redeem mankind
Text Corpora
Flueney
is cooperation. (9 words)
of mankind.
broader NLP community to contribute additional constraints that can co-evolve with LLM capabilities over time, while also providing a convenient endpoint for users that only want to evaluate their model without developing their own constraints. The flexibility of such grammar-based framework may not only be useful for evaluation, but also in practice (c.g. word constraints, words blacklist, etc.).
We construct the dataset COLLIE-v] with 2,080 constraint instances across 13 different types, using three different corpora: Wikipedia (Foundation) 2022), CC-News (Hamborg et al. 2017). and Project Gutenberg (Brooke et al., 2015). We perform zero-shot evaluations of five state-of-the-ane LLMs of varying sizes including GPT-4 (OpenAT, 2023a) and PaLM (Anil et al. 2023). While GPT-4 comparatively performs the best, still achieves average constraint satisfaction rate of only 50.9%.
We find that challenges correlate with position - for instance, instructing models to begin a sentence with specific word leads to 100% success rate for GPT-4, while asking models end sentence with a particular word results in a success rate of 40%-60% These insights can help us diagnose LLMs, which in turn can improve LLM capabilities, and further advance the benchmark itself.
To summarize, we make the following contributions: (1) We introduce COLLIE, a framework for systematic generation of compositional constraints, that is flexible and extensible. (2) We use COLLIE to curate a new dataset COLLIE-VI comprising of 13 constraint structures. (3) We perform a comprehensive evaluation of five state-of-the-art LLMs of varying sizes and provide useful insights for both model and benchmark development in the future.
2 RELATED WORK
Constrained text generation (CTG). Early work in controllable text generation used control codes to steer the generation towards desired topics or to reduce undesirable content, by controlling for broad
attributes such sentiment toxicity (Hu et al. 2017 Keskar et al. 2019 Dathathri al. 2020
Krause el al., 2) Other work on constrained decoding provides to the language model collection of lexical items as constraints to be included excluded in the final generated text (Hokamp and Liu
2017; Hasler et al. 2018: Dinu et al., 2019: Hu et al., 2019 Lin et al. 2020 Lu et al.) 2021 2022
Li al. 2022b). Recent advances instruction tuning LLMs (Ouyang et al. 2022) have brought major improvements to controllability. These advancements have made it challenging to use existing controllable generation datasets fully assess the capabilities of modern LLMss InstructCTG /Zou
et al. 2023) is a concurrent work that also constructed dataset with text constraints. However, it mainly focuses on synthesizing types of simple CTG instructions for tuning small language models such as T5 (Raffel et al. 2020). whereas COLLIE serves to construct much more challenging and open-ended CTG tasks to evaluate and diagnose sfart-of of-the--rttttt LLMs like GPT-4 (OpenAl (023b). Lastly, there is a line of work that sets up CTG in more practical downstream applications, such as controllable summarization (Zhang et al. 2023). The flexibility of COLLIE allows these "functional constraints" to be incorporated for more usefulness, which leave for future work.
2

---

Published as a conference paper at ICLR 2024
Grammar-based compositional tests. Building benchmarks with data synthesized from grammars has been explored previously in the context of question answering (Weston et al. 2013), instruction following (Chevalier-Bossee Boisvert et al. 2019; Ruis et al. 2020 and visual reasoning JJhnsoo et al.) 2017). These benchmarks showcased the utility of grammars to systematically generate a comprehen- sive set of test cases or specify some fixed constraints. In contrast, COLLIE aims enable flexible, and dynamic constraint construction that can co-evolve with models. Furthermore, previous datasets were synthetic with limited linguistic diversity and practical applicability to real-world scenarios. In contrast, since our COLLIE framework extracts values and examples from natural language corpora to construct the constraints, represents a more realistic challenge for modern LLMs.
Systematic and scalable language benchmarks. The emergence of increasingly powerful general- purpose language models has created a need for scalable benchmarks that can systematically and comprehensively evaluate them. A few recent examples include HELM (Liang et al. 2022). BIG-
Bench (Srivastava et al. 2022), MMLU (Hendrycks et al. 2020). TaskBench500 (Li et 2022a). and
Natural Instructions (Wang et al. 2022). However, building such benchmarks require considerable human effort, and may become obsolete when stronger models enter the arena. We provide new perspective in this race between model capabilities and challenging benchmarks: leverage compositionality to construct automatic and scalable benchmarks with minimal human effort that can co-evolve with model capabilities to remain challenging and relevant.
3 COLLIE FRAMEWORK TO CONSTRUCT CONSTRAINED TEXT GENERATION
COLLIE allows researchers to easily 1) specify textual constraint structures via grammar, then automatically 2) extract constraint values from text corpora, 3) render constraints into natural language instructions, and 4) evaluate generations with respect to constraints.
Grammar, Two observations about text constraints motivate a grammar characterization: 1) they involve different levels of text, e.g. character, word, sentence, paragraph; and 2) many of them specify either the count or position at a certain text level (existence is equivalent to count > 0).
Let capitalized letters (S, M, T) denote non-terminal variables, and other symbols o 4 denote terminals. A full constraint specification within our grammar S (Eq. [ consists of two parts: generation level (1evel word, sentence, a paragraph, or a document, and multi-constraint M (Eq, D). which is a log- ical composition of one more base-constraints C. A text T' (Eq. 4 within these constraints can either be the full generated text Ç. or a part of it when qualified with pos(-). For example, pos (pos(f, paragraph, 3), sentence, 1)) means "the last sentence of the 3rd paragraph of the gener- ated text". For terminal variables, we define level & of text (Eq. 5) string number relation or @ (Eq, 5). and string or number value Tar or Unum (Eq, 7. A represents the logical 'and' operator, and V represents the logical 'or'. With these definitions, construct the following grammar:
E) specifying whether the generated text & should be
s + (I)vel(E) = =6) A)AAA M M C CMAM C. M
(constraint specification)
(1)
(multi-constraing
(2)
C + count (T, f, Uat E Unum pos T,, l, Unum) Usr
(base-constraint)
(3)
T + & (osssTT (,,, E, Unum)
(text)
(4)
f + char| word sentence paragraph passage
(level)
(5)
-
-
(relation)
(6)
Oar E Σ
Unum E Z
(value)
At the core of our grammar, we consider two (symmetrical) types of base-constraints C' (Eq.
1. Count constraints. count (T, var) @ Unum constrains the occurrences of particular level-f string Var. For example, count (T, word, 'happy") S means "T should contain the word 'happy" no more than 3 times". In contrast, count (T", l') @ Voum constrains the occurrences of level-é strings in each level-E' unit of text T. For example, count (T, char, sentence) = 50 means "each sentence of text T should have exactly 50 characters".
2. Position constraints, pos(T, l, Unum) o Usr specifies that a particular part of the text T should equal (or not equal) the given string Uar- For example, pos(T, word, 3) 'happy' means "the 3rd word should be 'happy' in text T". We also allow negative indices for reverse counting, e.g. pospos( pospooTTTToo char, -1) A means "the last letter should not be 'x" in text T"
3

---

Published as a conference paper at ICLR 2024
Note that the grammar above can easily be extended to accommodate more types of base-constraints (e.g. part of speech, sentiment) by implementing the corresponding semantic checks - we leave this to future work. Also for convenience, specifications that only differ in their values (e.g. generate a sentence with exactly x words, x E N), and constraint to refer particular constraint specification with concrete values (e.g. generate sentence with exactly 5 words).
constraint structure to refer to family of constraint
Examples and conceptual challenges. Our grammar can express a wide range of constraints through logical compositions of base-constraints across different text levels. Table TableIiilla Willustrates some structures across generation levels, identified by names such as para0l for paragraph generation, etc.
In addition to the generation levels, count and pos across different levels introduce variety of challenges. For exam- ple, word01 1 and sent 01 challenge token-based language models to count characters; pass01 requires high-level se- mantic planning for models to generate a coherent passage under constraints; sent 04 and para02 challenge mod- els to generate text with presence or absence of particular words; sent 03, para03, and para04 require count- ing at multiple levels; and word02, word03, sent 02, para and pass01 combine counting and positional challenges different levels, which can be considered most demanding conceptually. We empirically assess the diffi- culty of constraint structures in Section Example COLLIE usage is presented in Figure 2
import Constraint, Targethivel, Count, Relation
Constraint parget_lever Targeeteee ("word"), trans transtormation--oont// (), reletion-Relatin (1==)),
text TThis is a good sentence." print check (text, 51) True
Figure 2: Example COLLIE code for simple number of words constraint.
Table 1:
List all constraint structures used in COLLIE-VI, with (simplified) example values. Multi-constraint M
ID
Example instruction
word0l Generate a word with least 15 letters.
count(E, char, word) 155
word02 Generate a word with 10 letters, where
count(E, char, word) 10 A pos((, char, 1) - A pos(f, loof,cchae char, 3) 'r' A pos (C, char, 9) = 'e'
letter is 's', letter 3 is 'r', letter 9 is 'e'. word03 Generate a word with at most 10 letters and
count char, word) < A
ends with "r".
(O char,-1)
sent01 Please generate sentence with exactly 82 characters. Include whitespace into your character count.
count char, sentence) 82
sent02 Generate a sentence with 10 words, where word 3 is "soft" and word 7 is "beach" and word 10 is "math".
count word, sentence) = 10
(os word, "soft" A
pos word, 7) "beach"
pos word, 10) "math"
Sent03 Generate a sentence with least words, and each word less than six characters. sent04 Generate a sentence but be sure to include the words "soft", "beach" and "math".
count(E, word, sentence) 2 20 count char, word) < 6
count word, 'soft') > 0
count(E, word, "heach") 0
count word, 'math") > >
para01 Generate a paragraph where each sentence begins with the word "soft".
pos(pos({E, senteese,,,settece,, 1), word,
'soft' A
pos(pos(E, sentence,, 2), word, 1) = 'soft' A...
para02 Generate a paragraph with at least 4 sentences, but do not use the words "the", "and" or "of".
sentenne, ) 2 4
count(E. word, 'the' 0
count(E, word, 'and') A
count(E, word, 'of')
para03 Generate a paragraph with exactly sentences, each with between 10 and 15 words.
count(E, paragraph) count(f, sentence) 2 10 A
count(E, word, sentence) S 15
para04
Generate a paragraph with at least 3 sentences, each with least 15 words.
sentence, paragraph) paragahhhh 2 3
count(f, senndneentenee) 15
para05
Generate a paragraph with 2 sentences that end in "math" and "rock" respectively.
(f paragraph)) = 2
pos(pos(C Sentenne,,, 1), word, -1) = "math" A
pos pos(pos(E sentence, 2), word, -1) = "rock"
passO1
Generate a passage with 2 paragraphs, each ending in "I sit." and "I cry." respectively.
count paragraph, passage) 2
pos (pos({, paragraph, 1), sentence, -1) "I sit."
pos pps(pos(E, paragraph, 2), sentence, -1) =rycry""
4

---

Published as a conference paper at ICLR 2024
In conjunction with the grammar, we develop set of compiling tools to help construct datasets with minimal human efforts. Concretely, the pipeline of dataset construction involves 4 stages (Figure DD:
1. Specify constraint structures. Researchers specify constraint structures (c.g. Table D and optionally with a value range (c.g. "generate a sentence with z words", and 5 S x S 10). This is the only stage that involves manual effort.
2. Extract constraint values from corpora. We design an automatic extraction algorithm that runs through a given text corpus to find strings that fit constraint structure with some value ranges. For example, given the constraint structure count(f, word, 0) = x with value range 5 I x S. 10, the extraction algorithm returns sentences in the corpus that have 5-10 words, with associated word counts, This ensures each constraint has at least one natural solution. More details are in Section I
3. Render natural language instructions. Each constraint can be rendered into natural language instruction (Table via ruled-based translation, thanks to the compositionality grammar of COLLIE, For example, constraint count char, 'v') 2 count (ç, char, ' can be synthetically rendered into the instruction "Please generate a word with exactly 2 character "v" and exactly character 'i" r. is also possible to improve the instruction fluency or naturalness by adding additional rules to the synthetic translation, or use L.MSs to polish instructions. More details are in Section A.
Evaluate generations. Given text E generated by model, we use a parser to evaluate against a constraint specification S and derive a True/False value, indicating if & satisfies S. We use an average success rate as the main metric to evaluate constraint conformance. We can also compare the fluency of & against the corpus-extracted "groundtruth" text, and render more fine-grained natural language feedback indicating which base-constraints are met and which not (see Section A.2
4 COLLIE-VI DATASET
We construct COL.LIE-VI using constraints structures from Table [ which contains 2,080 constraint instances from 13 constraint types, with 1,435 unique constraint prompts. The broader NIP commu- nity can contribute to future dataset releases by adding additional constraints, metrics, data sources. The curated constraint set can co-evolve with models to become more challenging and comprehensive as model capabilities improve.
4.1 CONSTRAINT SPECIFICATION AND EXTRACTION
Constraint specification. We begin by defining 13 constraint structures. We chose these 13 structures to span various generation levels (word, sentence, paragraph and passage generation) and challenges (counting, position). In total, we have 3 word-level, 4 sentence-level, 5 paragraph-level, and I passage-level constraint structures. Of these 13 constraint structures, 5 are single-level and the remaining 8 are multi-level constraints. See Table for the exact constraint structures
Constraint extraction. While constructing constraint structures is straightforward using our grammar, choosing constraint targets is challenging for two reasons: (1) Not all targets will admit conforming natural language string. For instance, the constraint, "Generate a two word sentence beginning with the word The." has no grammatically acceptable answer. (2) Even if constraint admits a possible answer, it may not admit plausible answer. For instance, "Generate sentence with 1928 words" is possible, but any such sentence very unlikely appear in regular discourse.
To address both challenges, we sample constraint target values from natural language corpora, which we denote as the data source. Given a constraint structure C and documents D {d1,.... ). we chunk each document into a series of strings d; = {$1, 3m), where each can be a sentence, paragraph, or passage as required by C. Each string 8; undergoes -purce- automated filtering and pooteprocessing to remove artifacts, which we detail in Section B.4, Given C and 8,, we extract target values such that is satisfied. In most cases, the satisfying target values can be directly extracted using our provided utilities. For example, for constraints with structure "sentence with words", we can directly apply word tokenization and counting to the example string 8;. In cases in which direct extraction is not possible, (e.g. "do not include word w"), specify range of possible targets (c.g. {the, and, of) to sweep over. All in all, our approach ensures that (1) there exists a natural language string that satisfy each constraint and target pair, and (2) the targets follow
5

---

Published as a conference paper at ICLR 2024
Guten
CC News
Wiki
wordn2
wondo]
(b)
Passage
0.66
Para/Passage
11.80
wordos g7
Paragraph
0.20
Sents/Para
swn001
113
Words/Sent:
26.00
sent03
demt02
1.00
278
Passage
0.80
Para/Passagr:
(6.80
sent04
267
Paragraph
Sents/Fara + [1.60
pers0]
26
Sentence
0.34
Words/Sent:
274
206
Paragraph
Passage
0.42
Para/Passage
Sents/Para
3.20
[5.30
Sentence
0.30
Words/Sent-
24.60
pass01
0.00
0.25
0.50
0.75
Number
Fraction Filtered
Mean Number
20
Figure 3: Data statistics. (a) Number of constraints from each constraint structure. (b) Fraction of strings removed by automated filtering. (c) Length statisties for different levels for each data source.
plausible distribution induced by natural language corpora. Our extraction system is extensible, and can operate on new constraints and data sources with minimal modifications.
Extensibility Adding additional data sources to the extraction pipeline similarly easy, requiring a text delimiter, and optional string filtering and post-processing functions. As a case-study on the extensibility of COLLIE, we demonstrate how extend constraints include POS-tags such as "Generate sentence with verbs". Details are in section B.
4.2 DATA SOURCES
To adequately cover diverse styles and content, extract constraint targets from three distinct data sources: Wikipedia (Wiki) (Foundation) 2022), Common Crawl News (CC-News) (Hamborg et al.
2017), and the Project Gutenberg Corpus (Guten) (Brooke et al, 2013). We provide an overview of these data sources below and leave source-specific filtering and post-processing details Section B.4
Wiki. Wikipedia (Wiki) (Foundation 2022 consists of over million English Wikipedia articles. included this data source for the diverse subject matter present the corpus.
CC-News. The Common Crawl News corpus (CC-News) (Hamborg et al. 2017) consists of 708,241 English language news articles published between Jan 2017 and December 2019. include CC-News to include interview dialogues, well popular culture and current events.
Guten. The Project Gutenberg corpus (Guten) (Brooke et al., 2015) consists of over 50,000 docu- ments that include fiction, histories, biographies, and other works that in the public domain in the United States. We include this corpus for its variety in genres (e.g. non-fiction, fiction, plays, etc.) and style from different time periods.
4.3 DATA VALIDATION AND STATISTICS
We extract constraints from 300 randomly sampled documents from each source. After extracting the target values, we sample up to 100 targets for each constraint structure on each data source. We remove any string targets by that begins or ends with any character that is We randomly sample 5 out of these 100 targets and their supporting examples to qualitatively verify their validity. Since the extraction process relatively fast, we modify filters and post-processors there are systemic issues and re-run the extraction phase. We provide statistics of the final number of constraints from each constraint structure in Figure D a). Some constraints (e.g. number of sentences per paragraph) are tightly clustered around the mean, and thus do not induce many valid constraint targets. The fraction strings filtered for each data source and level is presented in Figure b). The automated filtering removes large fraction of the strings in most cases, as high recall is important to ensure quality. The high fraction of omitted passages is due to the removal of passages < paragraphs in length. Mean lengths for each level and data source is presented in Figure Ce).
letter or number.

---

Published as a conference paper at ICLR 2024
Zero-shot performance
Constraint satisfaction rate GPT-4
(c) Constraint satisfaction rate GPT-3.5
werdol
wordo1
GPT4
50.9%
werd52
10.9%
word52 10.1%
Mer000
worded 10.0%
GPP335
27.9%
305652
0.4%
sest52 0.0%
PuLM
506600
40.5%
18.1%
pera01
87.1N
Vicuna77
16.4%
pera30
pemm00
53.3%
pamc2
pamio
850.4%
Alpaca-71
16.3%
pera25
KB9 5%%
parvoS 10.0%
poss01
+51.0%
pass01 11.0%
(d) Constraint satisfaction rate PaLM (6) Constraint satisfaction rate Vicuna-7B (1) Constraint satisfaction rate Alpaca-7B
wordo1
wordo]
wordo1
0.3%
wond02
0.0%
word02 10.0%
woreh1
115.0%
wondi
0.0%
wordh2 1.00%
sen122
0.0%
aen02
0.0%
sent02 0.0%
502622
15.1%
206003
1.0%
seat03
2.8%
200001
57.7%
H61.2%
[ame02
13.00%
peraf?
20.0%
par47
235.9%
pamod 18.0%
pertl
parsox
pero25 10.0%
perat5
00.%%
parc05 10.0%
305901 10.0%
pas501
0.0%
pa6001
10.0%
Figure 4: Model comparison. (a) Overall model performance summarized by weighted average across all constraint groups (b) -(f) Constraint satisfaction rates generated texts by GPT-4, GPT- 3.5, PaLM, Vicuma-7B, and Alpaca-7B across various constraint groups. Error bars represent standard error. Constraint group names are in Table Sample sizes are reported in Figure [|
RESULTS
Our main experiments in this paper focus on a zero-shot prompting setup with the following language models (LMs): 1) larger and closed-source LMs such as OpenAl GPT (Brown et al. 2020 ApenAl
2023b (gpt-3. 5-turbo, 4pt-4) and Google P-LM-2 (Anil et al. (023) (text bison- 001);
2) smaller and open-source LMs such as Alpaca-7B (Taori et al.) 2023), Vicuna-7B (Chiang et al. 2023). We performed additional one-shot prompting and find GPT performances similar zero-shot performance, see Section C.1 By default, we use sampling temperature of 0.7, and sample multiple
trials (20 for GPT/PaLM, for Alpaca/Vicuna). All experiments were run in July, 2023.
Zero-shot performance comparison. As evidenced in Figures aa), GPT-4 consistently surpassed other models in zero-shot constrained text generation performances, achieving more than twice the constraint satisfaction rate than other non-GPT models. The overarching performance trend observed shows GPT-4 leading the pack, followed by GPT-3.5 and PaLM with a large gap, and then followed closely by the smaller models, Vicuna-7B and Alpaca-7B.
Constraints all models can follow. Certain tasks, specifically word01 (generating a word with at least letters), sent 04 (generating sentence containing words X, Y, Z), and para01 (generating a paragraph with each sentence starting with the word X), posed minimal challenge to the majority of contemporary language models. These tasks demonstrate the proficiency of current models at simple constraints ensuring existence, as depicted in Figure 9 f).
Constraints partially solved by GPT-4 only. However, notable distinction arose when tasks incorporated more counting/position constraints and requested longer generations. Tasks such as word03, p4 p5ra05, and passO1 were only partially addressed by GPT-4, with constraint satisfaction rates ranging between 40% and 70%. Despite GPT-4's partial success in these tasks, other models failed deliver any satisfactory performance.
Constraints remaining very challenging, Furthermore, some tasks proved challenging across all models. Tasks word02, sent01, sent and para 0 present challenges terms of arbitrary position constraints and mixed counting levels (see Section 5.I for detailed analysis), indicating areas that necessitate further advancements language model technology. Moreover, the average pass @ 20 rate of GPT-4 was above 63% across all constraints, significantly higher than the 32% achieved by GPT-3.5, as depicted in Figure 5 Although GPT-4 demonstrated a significant performance advantage, its constraint satisfaction rate of 63% is far from perfect. This suggests considerable scope for improvement in controllable text generation with language models. These findings underscore the opportunities and challenges in the continued evolution of language models.
7

---

Published as a conference paper at ICLR 2024
Position effect
90
GP14
Overall pass@k
Rate of constraint satisfaction (%6
100
GPI4
GPT-3.5
GP335
80
Vicura-78
(%)
Alpaca 78
60
40
20-
20
1 5.6 1011112111111111711 161920
para0:
word03
paraOn
passO1
Number of samples (k)
char) word) word) char) (last word)
Figure 5: @ass@k performance. We sam- ple the model-generated text 20 times for all instruction prompts in the dataset. The curves represent the average pass rate across all instruction prompts up to l samples. The shaded areas indicate the standard errors.
Figure 6: Position effect. Satisfaction rates of LMs tasks involving pos(5, level, i). The tasks word02 and sent 02 impose constraints on characters and words at arbitrary positions. The task para01 constrains the first word. The tasks word0 3, para05, and passOl 01 constrain the last characters, words, and sentences.
Counting level effect
100
constraint satisfaction
GPT4
GPT-3
80
PaLM
Vicuna-
Alpaco 78
word01 entichar word)
pers03 in cattword para) ent(soon para)
in sentence) ontichar
sont)
Figure 7: Counting level effect. Satisfaction rates for LMs tasks involving count level, sp). Task word01 1 sets a minimum word length of a. Task sent requires exactly a characters in a sentence. Task sent 03 asks sentence contain at least words, with each word no longer than letters. Task para04 asks a paragraph to consist of at least b sentences, each containing minimum of words. Task para03 further imposes an upper limit on the number of words per sentence.
5.1 ANALYSIS
Performance consistency across data sources. We observe a high degree of consistency in the performance of models on given constraint structure, regardless of the data source. This uniformity evident across all models, highlighted in Figure (g). This indicates that the ability of language model to adhere to the logic of constraints takes precedence over the specific target values or the distribution the data.
Position effect. As depicted in Figure 3 the (O level, i) function, constraining the i-th sub-string (letter, word, or sentence), exhibits varying levels of difficulty depending on the value of Models generally perform well when the positional constraint is applied the first sub-string = 1, task para0l). However, only GPT-4 displays partial success with the last positional constraints = -1, tasks word03, para pass01). Notably, all models encounter difficulties when generating text that satisfies positional constraints at arbitrary positions i. Additionally, we find that the position effect exhibits a lower sensitivity to constraint levels.
Counting level effect. Counting characters within a word is easier than within a sentence for models, illustrated in Figure Furthermore, tasks demanding exact equality (task sent 01) prove more
8

---

Published as a conference paper at ICLR 2024
(a)
Constraint satisfaction rate
(b)
GPI-4 Interactive generation
GPT-4 (4th round)
70-
word01
word02
8.2%
100.0%
Constraint satiefaction nate (%6
03
61.7%
65.0%
66.0%
word03
100.0%
sent01
23.0%
80
sent02
0.7%
sent03
86.1%
155
sent01
99.6%
51.3%
para01
100.0%
-00
100.0%
para03
35.2%
para04
-67.3%
45
para05
88.2%
pass01
71.9%
40
round
2nd round
3rd round
round
Figure 8: GPT-4 interactive generation performance. (a) Constraint satisfaction rate of GPT-4 generated texts in the 4th round across various constraint groups. (b) GPT-4 overall performance in different feedback rounds. The 1st round is zero-shot, and the 2nd 4th rounds are with feedback.
challenging than those requiring range (task para03), and are considerably more difficult than tasks specifying just an upper or lower bound (tasks word01, sent03, para0
Increased difficulty with logical composition. The incorporation of logical compositions into constraints considerably increases their difficulty. Task sent03 serves as an example of this, adding an extra constraint at the sentence level compared to task word0l. Despite the assumption that the added constraint should be manageable for all models, performance on task sent uniformly trails behind that task word0l, as shown in Figure This highlights the intricacy and challenge introduced by logical compositions within constraints.
Performance enhancement through feedback and interaction. We utilize COLLIE to generate automated natural language feedback (e.g., "Your task to generate a word with exactly 2 character 'v' and exactly 3 character 'i'. However, you generate word with 3 character "v" and 4 character "i""), and engage LLMs in a generation-feedhace dialogue. In Figure g we observe a significant 20% improvement in GPT-4 performance after the second round of feedback. However, the model's performance plateaus at 66% even after three additional rounds of feedback, comparable to pass@ 5 using i.i.d. sampling. The extent of performance improvement varies across tasks, with word03's constraint satisfaction rate increasing from 62.1% to 10%. Conversely, word02, sent 01, and s2 tasks remain challenging for the model. These findings suggest that there still room for improvement, highlighting the difficulty of our dataset, and emphasizing the need for further research on better ways to incorporate natural language feedback.
CONCLUSION
In this work, we present COLLIE, grammar-based framework for specifying textual constraints. COLLIE simplifies the process of creating constrained generationn datasets by enabling researchers to focus on specifying high level constraint structures, while COLLIE automatically extracts constraint values, renders natural language instructions, and assesses model performance. To demonstrate the utility of the COLLIE framework, we construct COLLIE-VV with 1,132 constraints from 13 different types, extracted from different data sources. We evaluate five state-of-the. art LLMs of various sizes on COLLIE-VI and find that it provides fine-grained insights into model capabilities and shortcomings. We hope that model developers can use COLLIE-vI to develop more capable models, while future releases of COLLIE continue to adapt to the capabilities and needs of future models and users.
LIMITATIONS AND SOCIETAI. IMPACTS
Although care was taken to design the filtering and processing functions, such automated approaches are never perfect and remaining artifacts in corpora might lead to unnatural reference texts or con- straints. Further filtering (e.g.. by grammar checkers, parsers, humans) could improve the dataset quality. Our representative constraint structures were selected to encompass diverse constrained generation challenges, but with all generation benchmarks, they cannot capture all dimensions

---

Published as a conference paper at ICLR 2024
and nuances of model capabilities. Benchmarks are highly influential in shaping model develop- ment, the capabilities and limitations which may disproportionately impact different communities. Our benchmark is exception. However, by providing extensible, easy-to-use framework for constraint development, we hope COLLIE will enable diverse stakeholders to engage with dataset building, helping ensure that future model capabilities serve diverse interests and needs.
ACKNOWLEDGEMENTS
We thank Xiao Liu for Vicuna/Alpace APIs that supported our preliminary experiments, and Princeton NLP Group for helpful discussion and feedback in general. We acknowledge support from a Princeton SEAS Innovation grant and the National Science Foundation under Grant No. 2239363. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and not necessarily reflect the views of the National Science Foundation.
REFERENCES
R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al. Palm 2 technical report. arXiv preprint 2rXiv:2305.14400 2023.
Bird. Nltk: the natural language toolkit. Proceedings of the COLING/ACL 2006 Inteructive Presentation Sessions, pages 69-72, 2006.
J. Brooke, A. Hammond, and G. Hirst. Gutentag: an nip-driven tool for digital humanities research in the project gutenberg corpus. In Proceedings of the Fourth Workshop on Computational Linguistics for Literature, pages 42-47, 2015.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Process- mips cC/ pwwpr//020/haaaaeeeeeppp / html
S. Gray, B. Chess, Clark, C. Berner, S. McCandlish, A. Radford,
ing Systems (NeurIPS), 2020. URL 12f.cc442ff4444 Abst
H. Chen, H. Li, D. Chen, and K. Narasimhan. Controllable text generation with language constraints.
In preprint, 2022a. URL https://p////c. /ttp///avvvvoor//////2....//
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Ponde de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, I.. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. Pet- roski Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert- Voss,, W. Hebgen Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. In preprint, 2022b. URL
https: arxiv wrg. abg//bb//100
M. Deovalier-Boisvert Bahdanau, S. Lahlou, L. Willems, C. Saharia, T. H. Nguyen, and Y. Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. In International Conference on Learning Representations (ICLR), 2019. URL https : htt///ixvvv. /ttt::s//axxxi.oo///aa//.
1810 0.000.00 08272)
W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%** * chatgpt quality, March 2023. URL htRll//cpe///mssssrrr///////2222
S. Dathathri, A. Madotto, J. Lan,
J. Hung, E. Frank, P. Molino, Yosinski, and R. Liu. Plug and play language models: A simple approach controlled text generation. In International Conference on Learning Representations (ICLR), 2020. URL ht https::/cpenreviii net/pdE?id- H1 eeEEEBKDS
10

---

Published as a conference paper at ICLR 2024
G. Dinu, P. Mathur, M. Federico, and Y. Al-Onaizan. Training neural machine translation to apply terminology constraints. In Association for Computational Linguistics (ACL), 2019.
W. Foundation. Wikimedia downloads, 2022. URL httpt::tt//nggeee
F. Hamborg, N. Meuschke, C. Breitinger, and B. Gipp. news-please: A generic news crawler and extractor. In Proceedings of the 15th International Symposium of Information Science, pages 218-223, March 2017. doi:
E. Hasler, A. de Gispert, G. Iglesias, and B. Byrne Neural machine translation decoding with terminology constraints. In North American Association for Computational Linguistics (NAACL), 2018.
D. Hendrycks, C. Burns, Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
C. Hokamp and Q. Liu. Lexically constrained decoding for sequence generation using grid beam search. In Association for Computational Linguistics (ACL), 2017.
J. E. Hu, H. Khayrallah, R. Culkin, P. Xia, T. Chen, M. Post, and B. Van Durme. Improved lexically constrained decoding for translation and monolingual rewriting. In North American Association for Computational Linguistics (NAACL), 2019.
Z. Hu, Z. Yang. X. Liang, R. Salakhutdinov, and E, P. Xing. Toward controlled generation of text. In International Conference on Machine Learning (ICML), 2017.
J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. Conference on computer vision and pattern recognition (CVPR), 2017. URL [Rppttss///eeexplrreeeeee
axg/document. 8099698
N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, and R. Socher. CTRL: A transfsomete language model for controllable generation. preprinz, 2019. URL htttttttt//aaiv..cco abs 1 1909.99... 05858
B. Krause, A. D. Gotmare, B. McCann, N. S. Keskar, S. Joty, R. Socher, and N. F. Rajani. GeDi: Generative discriminator guided sequence generation. In Findings of the Empirical Methods in Natural Language Processing (EMNLP Findings), 2021. URL /ttps::/aclantee
brq/2021. findings-enn emn 1e.mmnpp.... p.p.p... 424
Q. Lhoest, A. Villanova del Moral, Y. Jernite, A. Thakur, P. von Platen, S. Patil, J. Chaumond, M. Drame, J. Plu, L. Tunstall, Davison, M. Saško, G. Chhablani,
Malik, S. Brandeis,
T. Le Scao, V. Sanh, C. Xu, N. Patry, A. McMillan-Major, P. Schmid, S. Gugger, C. Delangue, T. Matussière, L.. Debut, S. Bekman, P. Cistac, Goehringer, V. Mustar, Lagunas, A. Rush, and T. Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. URL
21
B. Z. Li, J. Yu, M. Khabsa, L. Zettlemoyer, A. Halevy, and J. Andreas. Quantifying adaptability in pre-trained language models with 500 tasks. In North American Association for Computational Linguistics (NAACL), 2022a. URL /httpss///axiiv. org/ /tss//1112 03204
Li, Thickstun. 1. Gulrajani, P. Liang, and T. B. Hashimoto. Diffusion-Im improves controllable text generation. In Advances in Neural Information Processing Systems (NewrIPS), 2022b. URL https:://penns
ew net Forum? (ttp:mmmmd=3s911000 jLyk Lyy
Liang, R. Bommasani, T. Lee, D. Tsipras. Soylu, M. Yasunaga, Y. Zhang, Narayanan, Wu. A. Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:221 1.09/ 10, 2022.
B. Y. Lin, W. Zhou, M. Shen, P. Zhou, C. Bhagavatula, Y. Choi, and X. Ren. Commongen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Empirical Methods in Natural l Langugge Processing (EMNLP Findings), 2020.
11

---

Published as a conference paper at ICLR 2024
X. Lu. P. West, R. Zellers, R. L. Bras, C. Bhagavatula, and Y. Choi. Neurologic decoding: (un)supervised neural text generation with predicate logic constraints. In North American Associ- ation for Computational Linguistics (NAACL), 2021.
pertts://aclanthool.e org/
2021 naac -main. 339.pt pf
X. Lu, S. Welleck, P. West, L. Jiang, J. Kasai, D. Khashabi, R. L.. Bras, L. Qin, Y. Yu, R. Zellers, N. A. Smith, and Y. Choi. Neurologic a auusque decoding: Constrained text generation with lookahead heuristics, In North American Association for Computational Linguistics (NAACL), 2022. 2022. URL
https::/aclaaneee w ogy. org./or naac -ma mm.
OpenAl. Gpr-4 technical report. preprint, 2023a. URL http:://axxv.orr//85555
08774
OpenAL. Gpt-4 technical report, 2023b.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurlPS), 2022.. URL net https://openrev wttpeeewwwwweeeoetteeeecoo TTBCACCC
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, Li, and P. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21 21(1)) 15581, 2020.
L. Ruis, J. Andreas, M. Baroni, D. Bouchacourt, and B. M. Lake. A benchmark for systematic generalization in grounded language understanding. In Advances Neural Information Processing Systems (NeurIPS), 2020. URL https: org/ abs /200.,203...m 05161
T. Schick, J. Dwivedi- Yu, R. Dessi, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. In preprint, 2023.
URL https: http:///axxvv..oo wttp:///////......oooooo http:////aaa...oo////////22000.
04761
A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro,
A. Gupta, A. Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint XXiv:22060006155 2022.
R. Taori, Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, Liang, and T. Hashimoto. Stan- ford alpaca: An instruction-follooinn llama model. https://githhnn com/ ttt.sttttsuuuutae lab/ stanford_aa a ppcca 2023.
Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, et al. Super-natuurlinsttuutiins Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085-5109, 2022.
Weston, A. Bordes, Chopra, M. Rush, Van Merriënboer, A. Joulin, and Mikolov. Towards ui-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05699 2015.
Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Cao. React: Synergizing reasoning and acting in language models. In International Conference on Leurning Representutions (ICLR), 2022. URL http::tt:://e/ //tpv//axivvvv//////////.. org/ abs /2210. 03629
Y. Zhang, Y. Liu, Z. Yang. Y. Fang, Y. Chen, D. Radev, C. Zhu, M. Zeng, and R. Zhang. Mac- sum: Controllable summarization with mixed attributes. Transactions of the Association for Computational Linguistics, 11:787-803, 2023.
W. Zhou, Y. E. Jiang, E. Wilcox, R. Cotterell, and M. Sachan. Controlled text generation with natural language instructions. arXiv preprint arXiv:2304.1429, 2023.
12


Reviews:
[{"review_id": "pAA8lNaBa4", "reviewer": "Reviewer_nWAg", "summary": "This paper introduces a grammar based framework called COLLIE for constrained text generation at varying levels of specifications. Additionally it proposes a development tool for automated extraction of task instances given a constraint structure and text corpus. Furthermore, the paper constructs a dataset from three sources - Wikipedia, CCNews, and Project Gutenberg using the previously mentioned framework and calls it COLLIE-v1. The resultant dataset COLLIE-v1 is constructed using manually crafted constraints and is used to analyze LLM performances as well as highlight their shortcomings. The paper focuses on five of the most prominent LLMs, namely GPT-3.5,GPT-4,PaLM, Vicuna-7B and Alpaca-7B.\n\nInterestingly, the COLLIE framework enables flexible, extensible and dynamic constraint construction that can co-evolve with the upcoming LLMs and help in understanding their shortcomings to better solve them.\n\nCOLLIE can thus help in not only evaluating and benchmarking LLMs but also help in constraint text generation independently.", "questions": "Section 5: Performance enhancement through feedback - It might help to list down the details of how the feedback is being used.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Well written paper with evaluation on competitive LLM baselines.\n    \n2. Combination of rule based and neural based generation enables NLP grounded generations\n    \n3. Open-sourcing of code and the related dataset for promoting further research.\n    \n4. Comprehensive analysis to highlight the shortcoming of current LLMs that needs to be addressed.", "weaknesses": "1. Some important details for instruction rendering should be moved to the main paper.\n    \n2. The paper mentions that the technique can be used for constraining words, word blacklisting, however a qualitative analysis is missing for the same in the current version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qmWxOmaztO", "reviewer": "Reviewer_PuAQ", "summary": "The authors propose a framework to automatically extract samples from large unlabeled text corpora for constrained text generation. Specifically, they manually craft rules/constraints and find satisfying texts as references. For example, a constraint can be \"the third last word being mankind\". They then evaluate off-the-shelf LLMs on this extracted dataset.", "questions": "In the limitations, the authors mentioned potential problems with filtering and processing functions. Could the authors elaborate on what issues might exist for the extraction process?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors evaluated different off-the-shelf LLMs and showed that they don't fully solve this task.", "weaknesses": "1. The idea is not very novel, and similar ideas have already been explored. For example, [1] also constructed a dataset using similar constraints for instruction fine-tuning.\n2. The dataset may not be very useful. Specifically, because the rules are too vague/arbitrary, the extracted ground truth is not useful for the evaluation process: the authors only use them for comparing fluency. In addition, since the rules can be arbitrarily designed, this compiled dataset does not hold much value, because many similar datasets can be compiled with different engineering details.\n\n[1] Controlled text generation with natural language instructions. https://proceedings.mlr.press/v202/zhou23g/zhou23g.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7grLGMKxQz", "reviewer": "Reviewer_6Jty", "summary": "This paper presents a framework, COLLIE, that allows researchers to build constrained text generation benchmark using different combinations of generation levels and modeling challenges (tasks), and a benchmark, COLLIE-v1, which is constructed using that framework and consists of 20,80 data instances comprising 13 structures.\nThe value of this paper lies in that it provides a method which allows future work to construct data of their interest in a scalable manner, and the analyses that the this paper conducts provide insights to researchers who are focusing on developing LLMs with better logical, reasoning, and compositional capacities.", "questions": "Please see weakness", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper provides a method which allows future work to construct data of their interest in a scalable manner.\n2. The analyses that the this paper conducts provide insights to researchers who are focusing on developing LLMs with better logical, reasoning, and compositional capacities.\n3. This paper is generally well-written and easy to follow.", "weaknesses": "There is no great weaknesses that I can find - but there is a minor one:\n\nAlthough I understand that the authors are focusing on more \"basic\" units, such as tokens, sentence, etc, so this paper can be more practical and useful for downstream applications, such as pretraining and evaluating LLMs, most of current work on constrained text generation seem to focus on text summarization, including controllable text summarization (e.g., MACSum Zhang et al, 2023). However this paper does not mention any of this, and does not discuss any potential that the proposed method can be applied to those constrained text generation (summarization) tasks. I personally suggest that the authors provide some discussion or explain on this issue.\n\nZhang, Yusen, Yang Liu, Ziyi Yang, Yuwei Fang, Yulong Chen, Dragomir Radev, Chenguang Zhu, Michael Zeng, and Rui Zhang. \"Macsum: Controllable summarization with mixed attributes.\" Transactions of the Association for Computational Linguistics 11 (2023): 787-803.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vq04LCvSvv", "reviewer": "Reviewer_p51c", "summary": "The authors propose COLLIE, a grammar-based framework for probing the performances of LLMs on constrained text generation tasks. COLLIE realizes the constraints on different levels (word, paragraph, passage) and imposes requirements on various aspects, e.g., character count, word count, and positions of a specific text. Experiments are conducted on instructions built from three sources and on various strong LLMs (GPTs, PaLMs, and etc.). The results suggest that GPT-4 generally outperforms other compared models but still performs poorly on a subset of the tasks. \n\nOverall, while the concept of this task is simple, the challenges posed to existing LLMs are non-trivial, clearly pointing out the incapabilities of these models. I believe COLLIE has the potential to drive further developments of existing models on more nuanced tasks.", "questions": "See weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written and easy-to-understand.\n* COLLIE is a conceptually simple but scientifically non-trivial method, and is beneficial to further development of LLMs.\n* The experiments and analyses are comprehensive, including evaluations on current state-of-the-art models.\n* The authors also provide complementary code which helps verify their approach.", "weaknesses": "I have one question regarding to the instructions obtained from different sources. Since the grammar is translated into natural language instruction to prompt the model, how do the instructions from different source datasets differ from one another?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)