Evaluating Large Language Models on Controlled Generation Tasks
Authors: Jiao Sun
Yufei Tian
Wangchunshu Zhou
Nan Xu
Qian Hu
Rahul Gupta
John Frederick Wieting
Nanyun Peng
Xuezhe Ma
Venue: EMNLP 2023 Main (2023)
Keywords: Evaluation
Large Language Model
Analysis
TL;DR: We test LLM on controlled text generation tasks and conclude that LLMs struggle at meeting fine-grained hard constraints.

Abstract: While recent studies have looked into the abilities of large language models in various benchmark tasks, including question generation, reading comprehension, multilingual and etc, there have been few studies looking into the controllability of large language models on generation tasks. We present an extensive analysis of various benchmarks including a sentence planning benchmark with different granularities. After comparing large language models against state-of-the-start finetuned smaller models, we present a spectrum showing large language models falling behind, are comparable, or exceed the ability of smaller models. We conclude that *large language models struggle at meeting fine-grained hard constraints*.

Evaluating Large Language Models on Controlled Generation Tasks
Jiao Sun Yufei Tian Wangchunshu Zhou Nan Xu
Qian Hu Rahul Gupta' John Wieting Nanyun Peng2 Xuezhe Ma
UUniversity of Southern California UUniversity of California, Los Angeles ETH Zurich 4 Amazon 5 Google DeepMind
[jiaosun, nanx, yuszheaaaeaa.ae..c.. edu Cyufeit, violetpeng]@cs... ucla. edu wangchunshu zhou@iff ethz. ch {huqia, gupra)eammazo. com
jwietingégoogle. com
Abstract
Control
Evaluation
While recent studies have looked into the abili- ties of large language models in various bench- mark tasks, few studies have looked into the controllability of large language models on gen- eration tasks. We present systematic and ex- tensive analysis of the controllability of large language models on ten benchmarks, includ- ing a new simple yet challenging numerical planning benchmark with different granulari- ties. After comparing large language models against oatte-of-the-starr finetumed smaller mod- els, we present spectrum showing when large language models fall behind, are comparable, exceed the ability of smaller models. We conclude that large language models struggle meeting fine-grained hard constraints.
constrained
content generation
sentiment, topic, keyword
Amazon Review CommonSen M202
off-the-shelf model,
repetition. diversity. coherence
story
prefix
ROC writing prompts
generation
rationale
generation
correct answer
CoS-E ECOA
Increased accuracy
numerical planning
prefix & number of
NPB
MSE, success rate
lexioal
paraphrase generation
sementic 6 syntax
PeraNMT QQPPoS
overlapping.
syntax match
good
poor
Figure I: We test large language models on five con- trolled generation tasks with various control factors us- ing automatic evaluation methods. We show spectrum of abilities of large language models on such tasks and conclude that large language models struggle at fine- grained hard constraints such as numerical planning.
1 Introduction
Text generation models should generate texts that meet controllable constraints humans wish (Zhang et al., 2022). For example, one can avoid the blandness caused by repetitive patterns by controlling the syntax of generated sentences (lyyer et al., 2018; Qian et al., 2019). In a customized di- alogue system, one should be able to control the persona of the utterance (Smith et al., 2020). Previ- ous works either finetume generation models such BART (Lewis et al., 2019) specific tasks for better controllability (e.g., controlled paraphrase generation (Sun al., 2021)) or design constrained decoding strategies (e.g.. look-back decoding strat- egy by Xu ct al. (2023a)) for controlled generation. Large Language Models (LLMs) have re- cently shown great potential in various genera- tion tasks. For example, Jiao et al. (2023a) shows that ChatGPT with GPT-4 an engine achieves commercial-level machine translation quality. Laskar et al. (2023) find that annotators prefer summaries generated from ChatGPT over state-of-the-art summarization models. However,
few works investigate the controllability of large language models. Towards this end, we aim to study and understand the controllability of large language models to answer the question: Are large language models better than fineluned smaller mod- els controllability generation tasks?.
The main contribution of this work is conduct comprehensive analysis of LLM's controllabil- ity on five tasks and ten generation benchmarks, including controlled story generation, controlled free-form generation with sentiment and topics, controlled paraphrase generation, and controlled rationale generation as in Figure 1. We further design new simple yet challenging benchmark named Numerical Planning Benchmark (NPB), where the task is to satisfy numerical constraints from four granularities (word-, syllable-, sentence- and paragraph-level) and under different content controls (e.g., prefix and ending). For evaluation, we use automatic metrics, which are imperfect yet convenient and reproducible.!
"The first four authors contribute equally.
hhttps://gi http:://gsttub...c/mmmiiaaaaaa2 23sun//

---

After an in-depth examination, we categorize LLM's controllability on a spectrum: from lagging behind and being on par with to surpassing smaller finetuned models. Our findings indicate that large language models have difficulties adhering to spe- cific hard constraints, such as numerical planning.
Granularity
Task Illustration
Generate sentence using exactly 5 words/syllables.
Word/Syllable
Complete sentence "This is a story" using exactly 5 words/syllables.
Complete sentence "This is a story" using exactly words/syllables, including the last word as "town".
We first introduce the numerical planning task and the associated evaluation as this is new, intu- itively simple, yet challenging task ($2). For the rest, we rank them by the task difficulty indicated in Figure from easy to hard: constrained content generation ($3), story generation ($4), rationale generation ($5) and paraphrase generation ($6).
Sentence
Generate a paragraph with 5 sentences,
Paragraph
Generate article with 5 paragraphs...
Table Task illustration for the Numerical Planning Benchmark. We test LLMs' numerical planning abil- ity under various constraints (word counting and end word) and granularities (word, syllable, sentence, and paragraph). Due to space limitations, we only show the full constraints under the word granularity here.
2 Numerical Planning
Can LLMs count from two to ten?
Task Description. We introduce the Numerical Planning Benchmark (NPB) an intuitive task that tests the basic numerical planning ability of LLMs. The high-level task descriptions can be found in Ta- ble 1. We are inspired by real-world scenarios such as creative writing. For example, writers may wish to generate sentences or poems with a specific struc- ture, such as a fixed number of words or syllables in each line, aiming to adhere to particular forms (e.g,, sonnets, where each line contains exactly 10 11 syllables (Tian and Peng, 2022)). Meanwhile, humans may also want full control over the start and end of each line for rhetorical purposes such as alliteration and rhyming. Inductively, we formulate our numerical planning benchmark from four dif- ferent granularities: generating a piece of text that contains a predefined number of words, syllables, sentences, or paragraphs given a plausible pair of prefix (start) and suffix (ending) as constraints. The prefix is given to LLMs such that they are only queried to generate the continuations.
of Romance Books and Reddit Short Stories. ? For word-level planning tasks (word and sylla- ble count), we randomly select sentences from the above datasets. Then, we select the last word in each sentence as the suffix. Depending on how many additional words we query the LLMs to gen- erate, we select the first few words in each sentence as the prefix (if we simply ask LLMs to generate freely without a prefix, the outputs lack diversity). Our prompt is written Complete a sentence that starts with (prefix) using exactly (N) additional words (including the last word [last word)). The sentence must end with the word flast word). Sen- tence: (prefix), and LLMs will continue. In the few-shot setting, we provide the task description and three examples. For each example, we also pro- vide explanations to help LLMs better understand our task. For example,
##Prefix: This is a story about a young girl's ##Last word: town
Evaluation Metrics. We use success rate (SR) and mean squared error (MSE) as automatic eval- uation metrics. As our control is two-fold, we separately calculate the success rates of 1) gener- ating the continuation with the correct counts and 2) generating the continuation with the proper end- ing. We also calculate the MSE between our input numbers and output numbers.
##N: 5
##Output: This is story about a young girl's redemption in a small town.
##Explanation: We generated "redemption in a small town". contains exactly words and ends with the last word 'town'.
We query the LLMs to generate outputs from N = 2 to N = 10 words. Each number N has 100 evaluation samples. For paragraph-Ievel tasks, the prefix and suffix are the first and last sentences in the corresponding paragraphs. For all experi-
Evaluate with LLMs. We evaluate ChatGPT and Alpaca-7b on our NPB benchmark in zero-shot and few-shot settings. Each request used to query the LLMs corresponds real case in the datasets
"huugingfacc.ooddaatttt//aaii wwtpe//cctoaaaaaaaaaaaaaaaaaaaae wwwp:e///wwweeeeeeoooooooueaaaee hooks, www.lagg...o//aassccctoooorrtrrr
"huggingfacc
con-contollgen

---

0.7
0.7
0.6
0.6
0.4
0.4
0.3
0.3
2 6 nums output
0.1
2
0.1
0.0
nums output
4
0.0
10
nums input (without last word constraint)
6 7
10
7
12
5
numss input
Figure 2: Histogram visualization in the distribution (frequency, z-axis) of input numbers (x-axis) and output numbers (y-axis) for word count planning. Left: querying ChatGPT to generate continuation of given prefix with N words. Right: querying ChatGPT to generate a continuation with N words of a given prefix that ends with a given word. Small red dots mark those bars where output numbers equal input numbers. These bars represent the fine-grained success rates. For either case, there is significant drop when the input number reaches six.
SR count
SR
SR -
MSE count
Upon further inspection, we find that LLMs try to mimic the style or features (such as length) the in-context examples and are, therefore, more likely to generate outputs with the wrong word counts once the input number N cannot be found in the examples. Our results resonate with Yin al. (2023); Kung and Peng (2023); Sinha al. (2023) that LMs do not truly understand task definitions via in-context learning.
Model
last word
both
GPT-2 (fine-tuned)
0.64
0.60
1.62
Alpaca-7bzs
0.17
0.31
0.09
9.19
Alpaca-7bICL
0.14
0.34
0.07
9.76
Vicunazs
0.08
0.09
0.03
27.68
VicunarCL
0.13
0.30
0.04
13.43
Falconzs
0.13
0).42
0.08
11.60
Falcon-7bici
0.11
0.34
0.03
13.72
ChatGPT
0.41
0.74
0.36
3.64
ChatGPT 0.37
0.78
0.34
4.95
Figure 2 is fine-grained visualization of the input and output numbers distribution by zero-shot ChatGPT. Specifically, compare LLMs' numeri- cal planning abilities with (c.g., complete sentence with "redemption in a small town" using exactly words, including the last word as "happy") and without additional suffix constraint (e.g., complete sentence with "redemption in a small town" us- ing exactly 5 words). LLMs can generate more freely without suffix constraints to meet the nu- merical constraint. However, ChatGPT doesn't al- ways translate to higher success rate. We find out that only when N is small (i.e., and 3), ChatGPT achieves higher success rate if explicitly told the last word of the target sentence.
Table 2: Success rates for the word count planning task. Surprisingly, few-shot in-context learning (ICL.)
underperforms zero-shot (zs) on numerical planning.
ments, our decoding strategy is top p (p = 0.95) sampling with temperature T = 0.3 unless other- wise specified.
Result. We report the model performance of LLMs and fine-tuned GPT-2-large model on the task of word count planning Table 2. Due to space limitations, we compile the results of the remaining tasks in Appendix A. First, is clear LLMs are poor at numerical planning, although it is an extremely simple task for humans. Given its extremely poor performance, incapable of doing numerical planning. Secondly, LLMs learn to incorporate literal constraints, such the last word, via few-shot in-context learning. Interestingly. few-shot in-context learning dete- riorates the performance of numerical planning.
consider Alpaca
Finally, would like to point out few be- haviors. First, although the general trend is that LLMs" numerical planning ability drops N in- creases, N is a clear exception (performs worse) among various experiments we repeated. Second, by checking the failure cases, we find that

---

ChatGPT always generates shorter continuations than required. Moreover, we see a sudden drop in model performances (from above ~0.6 to ~0.44 when the input number increases from 5 to We encourage future research to investigate these behaviors.
Model
Topic
Sentiment
Keyword
Diffusion-LM
68.9
83.7
93.2
GPT-2 (1.5B, line-tuned)
63.4
76.5
88.9
T5 (3B, fine-tuned)
67.3
83.9
94.8
LLaMA-7Bzs
45,3
58,4
83.5
LLaMA-7BICL
63.5
85.1
93.0
Alpaca-7825
58.9
78.4
91.2
Alpaca-78ICL
65.2
86.9
94.8
3 Content-Controllee Generation
Vicuna-782s
61.0
80.5
91.6
Vicuna-7BICL
65.8
87.4
94.3
Falcon-782s
61.9
81.0
92.1
Task Description. We consider three types of content constraints: topic, sentiment, and keyword. The detailed task definitions and dataset can be found in Appendix B.
Falcon-7BICL
66.0
87.7
94.2
ChatGPTzs
66.4
84.5
97.3
ChatGPTICL
88.4
90.3
98.1
Table 3: Results content-constrained text generation.
We use the success rate the evaluation metric to measure how well LLMs follow the content constraints. Specifi- cally, we use GPT-3.5 (Ouyang al., 2022) based topic/sentiment classifiers with in-context learn- ing using five examples per category to evaluate whether the generated texts belong to the specified topic or sentiment class. We consider an LLM to succeed in one example if the predicted class of the generated text is identical to the input constraint, For deysord-constrained generation, keyword coverage metric that measures the per- centage of input keywords included in generated texts.
Evaluation Metrics.
et al., 2019) and task-specific classifiers as a com- petitive non-LLM baseline
The results are shown in Table 3. We find that Alpaca significantly outperforms LLaMA in the zero-shot setting. This is intuitive since natural language instruction of constraints resem- bles instruction tuning data. However, this perfor- mance gap significantly reduced when in-context learning used. We think this because the role of instruction tuning is mainly to adapt an LLM to human-friendly prompt formats instead of in- creasing the LLM's capability. We also find that ChatGPT achieves competitive performance with- out in-context learning and outperforms Diffusion- LM, a competitive supervised baseline, by large margin. Moreover, the performance of ChatGPT can be further improved by adding in-context ex- amples to the prompt. This suggests that LLMs' ability to follow content constraints expressed in natural language depends on three confounding fac- tors: instruction tuming or supervised fine-tuning, in-context learning, and model capacity.
Results.
the
Evaluate with LLMs. For the content con- strained generation with LLMs, we follow Zhou et al. (2023) and use natural language instructions to prompt LLMs. Specifically, we use a prompt template of "Write a sentence about (topic name]" for conttrien generation, "Write an Ama- review with (level number] star about ran- dom thing. The number of stars ranges from one to five. One star is the most negative, and five stars are the most positive" for sentiment constraints, and "Write a sentence using the following key- words: (keywords)" for keyword constraints.
Story Generation
In addition to zero-shot evaluation, also eval- uate LLMs in the in-context learning setting by appending the following demonstration template: " "Below are some examples for the task: Input: (input 11, Output: foutput 1); Input: (input 2), Output: foutput 2/... " We use 5 in-context examples per class following the practice in Zhou et al. (2023).
Task Description. Given the beginning text a story, open-ended story generation aims to de- code texts that are coherent with previous topics, and informative without undesired repetitions (Su et al., 2022; Su and Xu, 2022; Xu et al., 2023b). Despite the impressive success on generating flu- and accurate sentences for low-entropy tasks such as summarization or translation, large-scale language models (LLMs) still suffer from serious degeneration problems, such as undesired repeti- tions (Holtzman et al., 2020: Su et al., 2022) and
We compare various LLMs including ChatGPT, LLaMA, Alpaca, Vicuna, and Falcon in our exper- iments. We also report the results of Diffusion- LM (Li et al., 2022b) based on BERT-large (Devlin

---

LM
Method
rep-2]
rep-3]
rep-4]
diversityt
coherenect
Evaluation Metries. Following open-ended story generation literature (Su al., 2022; Li al., 2022a; Xu et al., 2023b), we adopt the following automatic metrics to evaluate generation quality: 1) rep-n to measure sequence-level repetition accord- ing to the portion of duplicate n-grams (Welleck et al., 2019); 2) diversity to assess the overall model repetition by considering rep-n different n-gram levels; coherence measured the cosine simi- larity between prefix and continuation embeddings represented by SimCSE (Gao et al., 2021). We do not report MAUVE (Pillutla et al., 2021) score due to the concern that MAUVE may not accurately reflect human preferences considering contradicted results between MAUVE and human evaluations observed in prior work (Su and Xu, 2022).
ROC
Human
1.74
0.32
0.04
0.97
0.48
Nacleus
1.80
0.35
0.12
0.97
0.33
GPT-2-XL
Typical
0.4
pssampling
1,00
0.34
SinCTO
3.10
0.46
0.23
0.96
0.32
Look-back
7.24
0.92
0.14
0.92
0.47
Vicuna
2.36
0.45
0.97
0.60
LIM
Falcon
1.87
ChatGPT
1.18
0.10
0.02
0.98
0.52
Writing
Human
15.61
3.78
0.80
0.31
Nucleus
5.40
2.41
1.72
0.91
0.34
GPT-2XL
Typical
3.60
1.51
1.10
0.94
0.30
p-sampling
6.17
2.88
2.16
0.89
0.35
SimCTG
0.36
Look back
7.94
1,25
0.33
0.91
0.52
Vicuna
8.27
2.59
1.14
0.88
0.49
Falcon
11.20
7,79
6.94
0.76
0.53
ChatGPT 5.99
1.15
0.35
0.92
0.52
Table 4: Performance of different decoding strategies and LLMs for open-ended story generation. Vicuna stands for Vicuna-7B, Falcon for -nnsttutt
Evaluate with L.Ms.. Chatbots that fine-tune LLMs on instructions are also evaluated: Vicuna- 7B (Chiang et al., 2023), Bisoon-7-Inntruune (Al- mazrouei et al., 2023) and ChatGPT. We prepend the following instruction before the story prefix as prompt: 1) ROC: "Please continue writing this story within 4 very short sentences: <prefix>", 2) WP: "Please continue writing this story within 256 words: <prefix>"".
unnatural topic drifts (Li et al., 2022a), under open- ended settings.
Datasets. We evaluate different generation meth- ods on two popular benchmark story datasets: ROCStories and Writing Prompts. ROCStories (ROC) (Mostafazadeh et al., 2016) is corpus comprising commonsense stories written by crowd- sourced workers within short sentences. Given the first sentence as prefix. generation methods are required to produce four continuing sentences. Writing Prompts (WP) is a challenging task for inspiring continuations with abstract, high-level story prompts submitted by online users and con- tinuations by others Reddit (Fan et al., 2018). Following prior literature (Xu al., 2023b), utilize the first 32 tokens the prefix and ask for continuation with 256 tokens. Since we prompt different language models or decoding algorithms without extra fine-tuning, we directly sample 1,000 development and 1,000 testing instances from both ROC and WP.
Results. As shown in Table 4, both Vicuna-7B and ChatGPT are able to continue writing more fluent and coherent stories on both ROC and WP compared with other decoding methods based on GPT2-XL. Falcon-7B-Instruct obtains consistently lower diversity than other baselines, while ChatGPT achieves more robust performance in terms di- versity and coherence on both datasets.
Rationale Generation
Task Description. Free-form rationales are known to aid model interpretability by providing additional world knowledge or commonsense rea- soning steps (Kim, 2015; Lipton, 2018; Alvarez- Melis and Jaakkola, 2018). Wei et al. (2022) show that rationales can improve large language models' ability to solve complex reasoning tasks. Extractive rationales in question-answering tasks are based on the input passage extract related information to answer the question. Conversely, free-form ra- tionales in the question guestion-answering tasks are open-
Baselines. We evaluate the pre-trained LLM,, GPT-2-XL (Radford et al., 2019), with both search (SimCTG (Su et al., 2022) and Look-back (Xu al., 2023b)) and sampling decoding methods (Nucleus sampling (Holtzman et al., 2020), Typical decoding (Meister et al., 2022) and a s7-samllin (He-
Phttps://chal www.aat.oppeniicoocmm
"We adopt generation parameters for different LLMs sug- gested from their respective documents or APIs. We leave evaluation on more configurations in our repository: https: //github. com/sunj ia 233un/ //m/mmmmm//// controlgen.
witt et al., 2022)).

---

I-0
0.87
They share the same sets of questions and options. Based on the findings from Sun al. (2022), both CoS-E and ECQA tend to leak the correct answer in the rationale, while ECQA rationales contain the background necessary to answer the questions. We conduct our analysis on question-ansee answer pairs from the test set. Based on the evaluation acc(I+R-22) O)- acc(I-O), since we are evaluating the same set of question-answer pairs, ace(I-xO) (O]+ is always the same. Therefore, we only compare acc(I+R=+O) with different LLMs.
I+RCoS-E +00
0.92
I+RCCOA +0
0.99
Model
Leakage
Non-Leakage
I+RAlpaca-78
++
0.91
0.86
I+RLLaMA-7B
0.87
0.79
I-Ryicuna-78
+0
0.95
0.74
I+RFalcon778 ++
0.83
0.65
I+RChatGPT ++
0.98
0.93
Table 5: Rationales generated by ChatGPT are on par with coswssrowdder rationales ECQA with FlanT5 XXL (Chung et al., 2022b) as the backbone model. Rul- ing out leakage results in at least 5% accuracy drop.
Evaluate with LLMs. We prompt LLMs to pro- vide background knowledge that can help answer the question and control whether to leak the cor- rect options in rationales. We use ChatGPT the example for illustration:
ended and condition on purely the question and options. (Sun et al., 2022) studies how different the quality of rationales would impact rationales' utilities in terms of improving the model perfor- mance and claims that crowdsourced rationales superior to generated rationales. Sun al. (2022) finetunes T5-base for both rationale generation and question answering. With the power of LLMs, we want to revisit the problem and see whether the utility of generated rationales conditioned on the question and options has been improved.
Leakage. We have ChatGPT take the role of A teacher who is trying to explain to students the rationale behind choosing the correct option for multiple-choice question. Then prompt it with Question: (question) Options: (concatenated options) Explain the rationale behind choosing the correct option "[correct answer)".
The role of ChatGPT becomes A teacher who trying to explain to students the rationale behind a multiple-choice question. However, you do not want to leak the correct answer directly. and prompt it with Question: qquestion/ Options: /concatenated options) Explain the rationale behind choosing the cor- rect answer. Do not mention the correct answer "[correct answer)" explicitly.
Non-leakage.
Evaluation. We follow previous works and use the performance gap before and after adding ratio- nales in the input to measure the utility of ratio- nales, written as acc(I+R-O) acc(I+O), where stands for question and options as input, R stands for rationales, and O stands for one of the op- tions as output. For the backbone model for ques- tion answering, we use flanT5-XXL (Chung et al., 2022a) instead of T5-base it can handle longer sequences and is better at reasoning.
We highlight the difference between the two modes with underline. When prompting LLaMA and Al- paca, we remove the role description and only use the prompts. Through analysis, aim to answer two questions: Are L.L--generaed rationales par with crowdsourced rationales? 2) How much would leakage impact the utility of rationales?
Sun et al. (2022) shows that two factors are mainly affecting the utility of rationales. One is leakage, which means that the correct answer is explicitly written in the rationales, and one can choose the correct answer among all the options by rationales without knowing the questions. The other is background knowledge, which is the ad- ditional background knowledge or reasoning step that can help answer the question.
Result. Compared to T5, FlanT5 has better rea- soning abilities (Chung et al., 2022b) and is more capable of understanding instructions. Therefore, use FlanT5 instead of using T5 as the back- bone model for question answering, which can theoretically examine the utility of rationales bet- ter ruling out the incapability of models. Simply given the question and the option strings, Table 5 shows that F5 XXL has an accuracy of 0.87
Datasets. CoS-E (Rajani et al., 2019) and ECQA (Aggarwal al., 2021) are the most popular free-form rationale datasets through crowdsoureing. ECQA builds on CoS-E and improves the quality of the CoS-E dataset from various aspects, including completeness, comprehensivenesss coherence, etc.

---

(while T5 in (Sun al., 2022) scores 0.57 under the same setting). We then show the performance with crowdsourced rationales from both ECQA and CoS-E. With crowdsourced rationales from ECQA, the model almost solved the task and reached a performance of 0.99. With CoS-E rationales, the accuracy is 0.92. Our finding echoes with Sun et al. (2022) that ECQA rationales are better quality.
Evaluation Metries. We use two sets of evalua- tion metrics evaluate the quality of generated paraphrases. We use lexical-overlapping-aaee scores to evaluate the semantic preservation and tree-edit distances to evaluate the syntactic confor- mation. For lexical-overlapping-- higher is better. For tree edit distance, the lower is better, indicating that the newly derived syntax matches more closely with the expected syntax. In this work, we prune the constituency parse trees at level of 2 and only compare the high-level syntac- tic structure. TED-R means the tree edit distance between the candidate cundidate-generated sentence with the ground-truth paraphrase as the reference. TED-E compares the candidate sentence against the exem- plar that only provides the syntax.
scores, the
We then evaluate the utility of LLM-ggnnrated rationales under both the Leakage and Non-leakage scenarios, As the majority of crowdsourced ratio- nales contain leakage (Sun et al., 2022), we con- sider it fair to compare LLM-generated rationales under the Leakage scenarios against crowdsourced rationales. We have two major findings:
ChatGPT generated rationales are on par with ECQA rationales from crowdsoureing.
Evaluate with LLMs. We provide three ways prompt for the controlled paraphrase generation:
We quantify the influence of leakage in measur- ing the utility of rationales: whether or not having leakage in rationales could result an accuracy difference of at least 5%.
Direct. We prompt LLMs directly without speci- fying any constraints. The prompt written as Paraphrase (source sentence). Please only have the paraphrase in the response.
Control. Under this mode, sentence for the syntactic control signal. The prompt is written Paraphrase "[source sen- tence]" so that it uses the syntactic structure from "/exemplar]"; please only have the paraphrase in the response.
the exemplar
6 Controlled Paraphrase Generation
Task Description. Syntactically-controll para- phrase generation can benefit a wide range of NLP applications such as dialogue generation (Gao et al., 2020), improving the robustness of mod- els (Huang and Chang, 2021) or metrics (Aggarwal et al,, 2022), and diversifying other generation tasks such diverse question generation. Syntactically- controlled paraphrase generation is challenging be- cause it requires satisfying two folds of control signals: semantic preservation and syntactic con- formation. By definition of paraphrases, the gen- eration should have exactly the same semantics as the input text. With syntax as part of the input, generated paraphrases should also conform with the indicated syntax. The input syntax come from variety of sources.
We observe that under the Control mode, the gener- ated paraphrases would sometimes take the syntac- tic information from the exemplars and the seman- tic meaning from exemplar sentences. To solve this, introduce the third mode Control with syntax explanation. We first extract the constituency parse structure from the exemplar sentence using Stan- ford CoreNLP, prune the parse tree at the height of two (i.c., parse at H2), and then ask ChatGPT to generate a natural language explanation of the pruned syntactic parse, which we refer to as syntax explanation. The generated syntax explanation will be part of the input.
Control with Syntax Explanation. The prompt is written as Paraphrase "[source sentence]" that the senfence has a syntactic structure of "[pruned syntax]". [generated explanation for the syntax.) Please only have the generated para- phrase, not its parse, in the response.
Datasets. We evaluate on ParaNMT-small (Chen et al., 2019), derived from ParaNMT (Wieting and Gimpel, 2018), and QQP-Pos (Kumar et al., 2020). Our train/dev/test split follows previous works (Ku- mar et al., 2020: Sun et al., 2021). Each instance is a tuple of (source sentence, exemplar, ground-truth paraphrase), ), where the exemplar shares the same syntax with the ground-trui -ttnt paraphrase.
Table 7 shows examples of generated explana- tions for constituency parse trees pruned at height

---

TED-R
TED-E.
BLEUT
METEORT
ROUGE-IT
ROUGE-21
ROUGE-L
(I=2)
(1I=2)
Direct
10.8
26.2
44.2
18.6
44.9
1.4
1.5
ParaNMT
Ctrl
14.3
30.7
51.4
25.8
50.7
1.3
1.2
-Small
Syntax exp.
13.6
27.3
46.4
20.2
47.0
1.4
AESOP
22.9
32.7
54.4
29.8
56.4
0.9
0.5
QQPPos
Direct
6.7
25,2
39.8
15.6
41.5
1.8
1.8
Ctrl
10.5
25.6
43.0
19.8
45.2
1.4
1.4
Syntax exp.
9.0
26.5
42.8
17.8
14.2
1.8
1.8
AESOP
47.3
49.7
73.3
54.1
75.6
0.4
0.3
Table 6: Performance comparison with ground-truth syntactic control for AESOP (Sun et al., 2021) and fine-shot ChatGPT. With coarse syntactic control from shallow height of pruning, AESOP, the state of the finetuned small model, outperforms five-shot ChatGPT across all semantic preservation (BLUE, ROUGE Scores, and METEOR) and syntactic conformation metrics (TED-R and TED-E the height of two) by large margin. means higher is better, while means lower is better. By comparing ctrl with syntax explanation, we show that ChatGPT is better at mimicking syntactic structure from an exemplar than utilizing the syntactic information directly from the syntax.
Pruned Parse at H=2
Explanation
syntactic conformation metrics (TED-R and TED- E at the height of two). In addition, we find that ChatGPT's performance is the best under the set- ting of Control, where for control signals. Compared with the setting Con- trol with syntax explanation, Table shows that ChatGPT is good at mimicking syntactic structures from sentences instead of directly incorporating the syntactic parses. Besides ChatGPT, we also tried A Alpaca (Taori et al., 2023) and LLaMA (Touvron et al., 2023) the controlled paraphrase genera- tion task. However, they repeat input sentences and struggle to generate meaningful content. Therefore, do not include them here for comparison.
This represents sentence structure with a noun phrase and a verb phrase its constituents.
(ROOT (S (NP) (VP)) )))
exemplar sentences
with fragment that includes subordinate clause followed by period.
This
(ROOT (FRAG (SBAR)
())
This sentence structure represents an subord
(ROOT (SBARQ
interrogative sentence inate clause before main clause.
(WHADVP (SQ) )))
This is parse tree for a sentence containing a main verb and subject. with a possible adverb and complement structure.
(ROOT (SQ (VBP) (RB ) (NP (VP
Table 7: Examples of generated explanations for pruned constituency parse trees by ChatGPT.
Related Works
two by ChatGPT. We prompt ChatGPT from zero shots to five shots for our experiments, find that ChatGPT's performance peaks with five shots as expected, and compare the performance of five- shot ChatGPT with AESOP (Sun et al., 2021). The backbone of AESOP is the BART-base model, 140m-parameter model finetuned with specialized input and output format tailored for the controlled paraphrase generation task. To the best of our knowledge, AESOP remains the state-of-the-art paraphrase generation model on both ParaNMT- small and QQPPos datasets.
LLM Evaluation. While the advancement more potent large language models drives our work, our focus aligns more with recent studies evaluat- ing LLMs' performance on academic NLP bench- marks. We roughly categorize these studies ei- ther general specific NLP tasks. For general NLP tasks, Qin al. (2023) shows that ChatGPT performs well on many tasks involving reasoning capabilities but not on sequence tagging. Ahuja et al. (2023) evaluate LLMs on various multilingual NLP tasks. For specific tasks, Jiao et al. (2023b) shows that ChatGPT has achieved competitive per- formance machine translation. Gao al. (2023) ChatGPT for event extraction and shows that only matches with around half percent of spe- cialized event extraction models. To the best of the authors' knowledge, we are the first to study the controllability of LLMs and the tasks in our work
Result. Table shows the performance compar- ison between five-shot ChatGPT and AESOP. We show that AESOP surpasses ChatGPT across all evaluation metrics for both semantic preservation metrics dexical-overlappe based metrics includ- ing BLEU, ROUGE scores, and METEOR) and

---

have not been previously studied. Instead of having single conclusion on if LLMs perform well at cer- tain task, we provide a spectrum showcasing how LLMs' abilities vary according to different control granularities.
number N has been averaged over 100 different evaluation samples reduce noise. However, none of these experiments show that LLMs can do better than fine-tuned GPT-2.
In-Context Learning. We try to give more demonstration of NPB in our prompts and we sur- prisingly found that this does not help once the input number N cannot be found in the examples. Our results resonate with Yin et al. (2023); Kung and Peng (2023) that LLMs do not truly understand task definitions via in-context learning.
8 Discussion: Why and How
We believe that our work makes a substantial con- tribution to the field of benchmarking LLMs' con- trollabiltiy, especially considering the prevalence of LLMs these days. That being said, we do have few hypotheses investigate why LLMs fail numerical planning and how we could potentially increase their controllability.
How to Improve. We encourage future work to explore from two different directions: 1) chainoreeegrraph-offthohggt reasoning, and 2) bridging LLMs with non- -on-autoregressvvee generation abilities (e.g.. NADO (Meng et al., 2022)). For the first one, can try both simple chain/trec/grappe of-thought prompting or even pretrained LLMs with heeaan--ofttoougghssrrtthhhpppp pairs, as it shows promises for mathematical reasoning (Zhou et al., 2022). However, this will not fundamentally solve the planning issue. It is straightforward that auto- regressively generating the next tokens will lead to the problem of models not "looking back" and therefore not adhering to the fine-grained control signals. Therefore, we encourage researchers also investigate multi-step planning and iterative revisions with LLMs, or, more fundamentally. chal- lenge the autoregressive architecture of LLMs.
Tokenization. On one hand, tokenization indeed makes the task of numerical planning more chal- lenging than without, by separating the generative process (i.e., subword-level generation) and the nu- merical planning process (i.e., counting complete words). However, we posit that tokenizers not nec- essarily impact the ability of word planning, as is a standard practice that a subword starting with a special token will indicate the start of new word (e.g., "G" in BPE tokenizer," which has been used by many LLMs such as GPT and RoBERTa). Nor are we aware of evidence that the subwords of tokenizer roughly correspond to units of syllables. For example, Tian et al. (2023) shows that smaller models such as GPT-2-large fine-tuned on syllable- related data can achieve a success rate of close to 90% on the same syllable-plamnin task. On the other hand, the best performance of ChatGPT is 37%.
Conclusion
Decoding Methods. The reported results are based on sampling with a temperature of 0.3. More- over, we have experiments showing that our con- clusion is robust to the change of decoding mech- anisms, where we try other decoding methods be- yond sampling with T 0.3.
We test the controllability of large language mod- els on five tasks and ten benchmarks, including a numerical planning benchmark that is easy for hu- mans while challenging for LLMs. From there, we draw a spectrum by comparing the performance between LLMs and smaller specialized models. LLMs are able to generate human-level rationales and conform with coarse control signals, such as sentiment, topic and keyword incorporation. How- ever, they struggle at fine-grained hard constraints, such as numerical planning and paraphrase gener- ations. We hope that our work inspire down- stream applications on when adopt LLMs. For example, we find that LLMs are good at generating rationales, and these automatic rationales could be used to further boost LLMs' performance through chain-of of-thought reasoning.
Specifically, we tried 1) greedy decoding, 2) beam search with beam size 8, and 3) sampling with temperature T {0.3, 0.7, 1.0}. For the prior two, most of the generated outputs are highly simi- lar, plain, and lack diversity. As for sampling with T = (0.3, 0.7, 1.0). the success rate decreases as T increases. We think T 0.3 is reasonable bal- ance between diversity and quality. We believe that our results convey meaningful signals since each
httppp:s//iiigiiaaaee.ooooo/rrrs course/ thaptert//?ffurttthyteers pair- tokeiiiztooeiiiiiinn

---

Acknowledgement
Methods in Natural Language Processing, pages 936- 945, Copenhagen, Denmark. Association for Compu- tational Linguistics.
The authors thank anonymous reviewers for their constructive feedback and suggestions that helped improve the draft, especially reviewer rXWW. Jiao and Yufei are supported by Amazon fellowships.
Mingda Chen, Qingming Tang. Sam Wiseman, and Kevin Gimpel. 2019. A multi-task approach for dis- entangling syntax and semantics in sentence represen- tations. pages 2453-2464, Minneapolis, Minnesota. Association for Computational Linguistics.
Limitations
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
This work is subject to couple of limitations. First, all of our experiments involved heavy prompt en- gineering effort. Although we have attempted to choose the best performing prompts, there might be room for better prompts which could influence the reported evaluation metrics. Second, automatic evaluations are imperfect. Last, we have not pro- posed solutions after identifying tasks where LLMs struggle. We leave this for future work.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022a. Scaling cnstructionfineturne language mod- els.
References
Arshiya Aggarwal, Jiao Sun, and Nanyun Peng. 2022. Towards robust NLG bias evaluation with syntactically-dive prompts. In Findings of the As- sociation for Computational Linguistics: EMNLP 2022, pages 6022-6032, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Hyung Won Chung. Le Hou, Shayne Longpre. Bar- ret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022b. Scaling instruction-finetunee language mod- cls. arXiv preprint arXiv:2210.
Shourya Aggarwal, Divyanshu Mandowara, Vishwa- jeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021. Explanations for Common- senseQA: New Dataset and Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pages 3050-3065, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- mologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Kabir Ahuja. Harshita Diddee, Rishav Hada, Milli- cent Ochieng. Krithika Ramesh, Prachi Jain, Ak- shay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023. Mega: Multilingual evaluation generative ai.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hicrarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al- shamsi, Alessandro Cappelli. Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Hes- low, Julien Launay. Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with tfth-offthhe-art performance.
Jun Gao, Huan Zhao, Changlong Yu, and Ruifeng Xu. 2023. Exploring the fcasibility of chatgpt for event extraction.
Silin Gao, Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020. Paraphrase augmented task-oriented dialog genera- tion. ArXiv, abs/2004.00742.
David Alvarez-Melis and T. Jaakkola. 2018 Towards robust interpretability with self-explaining neural net- works. In NeurIPS.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2017. Guided open vocabulary im- age captioning with constrained beam search. In Proceedings of the 2017 Conference on Empirical

---

John Hewitt, Christopher Manning, and Percy Liang. 2022. Truncation sampling as language model desmoothing. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 3414 3427, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2022a. Contrastive de- coding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.1500
Xiang Lisa Li, John Thickstun, Ishaan Guirajani, Percy Liang, and Tatsunori Hashimoto. 2022b. Diffusion- LM improves controllable text generation. In Ad- vances in Neural Information Processing Systems.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration.
Kuan-Hao Huang and Kai-Wei Chang. 2021. Generat- ing syntactically controlled paraphrases without us- ing annotated parallel pairs. ArXiv, abs/2101.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen. Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A constrained text gen- eration challenge for generative commonsense rea- soning. In Findings of the Association for Compula- tional Linguistics: EMNLP 2020, pages 1823-1840, Online. Association for Computational Linguistics.
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1875-1885, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Zachary C Lipton. 2018 The mythos of model inter- pretability: In machine learning, the concept of in- terpretability is both important and slippery. Queue, 16(3):31-57.
Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Neuro- Logic decoding: (un)supervised neural text genera- tion with predicate logic constraints. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 4288-4299, Online. Association for Computational Linguistics.
Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023a. Is chatgpt a good translator? yes with gpt-4 as the engine.
Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu, 2023b. Is chatgpt a good translator? yes with gpt-4 as the engine.
Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. 2020. The multilingual Amazon reviews cor-
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2022. Typical decoding for natural lan- guage generation. arXiv preprint arxiv:2202.00 000666.
pus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4563-4568, Online, Association for Computational Linguistics.
Tao Meng, Sidi Lu, Nanyun Peng, and Kai- Wei Chang. 2022. Controllable text generation with neurally- decomposed oracle. In Advances in Newral Informa- tion Processing Systems, volume 35, pages 28125- 28139. Curran Associates, Inc.
Been Kim. 2015. Interactive and interpretable machine learning models for human machine collaboration. D. thesis, Massachusetts Institute of Technology.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He. Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849, San Diego, Californa. Association for Computational Linguis- tics.
A. Kumar, Kabir Ahuja, Raghuram Vadapalli, and Talukdar. 2020. Syntax-guided controlled genera- tion of paraphrases. Transactions of the Association for Computational Linguistics, 8:330-344 345.
Po-Nien Kung and Nanyun Peng. 2023. Do models really learn to follow instructions? an empirical study of instruction tuning. ACL 2023.
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq R. Joty, and J. Huang. 2023. A systematic study and compre- hensive evaluation chatgpt on benchmark datasets.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishknn. Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder. Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdel rahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019 Bart: Denoising sequence-to-sequeee ural language generation, translation, and compre- hension. Annual Meeting of the Association for Computational Linguistics.
pre-training for nat-
Krishna Pillutla. Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid

---

Harchaoui. 2021. Mauve: Measuring the gap be- tween neural text and human text using divergence frontiers. Advances in Neural Information Process- ing Systems,
Yixuan Su, Tian Lan, Yan Wang, Dami Yogatama, Ling- peng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. arXiv preprint arXiv:2202.06417
Matt Post and David Vilar. 2018. Fast lexically con- strained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Pa- pers), pages 1314-1324, New Orleans, Louisiana. Association for Computational Linguistics.
Yixuan Su and Jialu Xu. 2022. An empirical study on contrastive search and contrastive decoding for open-ended text generation. arXiv preprint arXiv:2211117997.
Jiao Sun, Xuezhe Ma, and Nanyun Peng. 2021, AESOP: Paraphrase generation with adaptive syntactic control. In Proceedings of the 2021 Conference Empiri- cal Methods Natural Language Processing, pages 5176-5189, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Lihua Qian, Lin Qiu, Weinan Zhang, Xin Jiang, and Yong Yu. 2019. Exploring diverse expressions for paraphrase generation. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3173-3182, Hong Kong, China, Association for Computational Linguisties.
Jiao Sun, Swabha Swayamdipta, Jonathan May, and Xuezhe Ma. 2022. Investigating the benefits of free- form rationales. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5867-5882, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a pereras- natural language process- ing task solver?
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin. Percy Liang. and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An castruction-folooiinn Ilama model. https: // gi thub. tannn/tattu-labba wattsstaaa/stttaoor.tttttt
Lianhui Qin. Sean Welleck, Daniel Khashabi, and Yejin Choi. 2022. COLD decoding: Energy-based con- strained text generation with langevin dynamics. In Advances in Neural Information Processing Systems.
Yufei Tian, Anjali Narayan-Chen, Shereen Oraby. Alessandra Cervone, Gunnar Sigurdsson, Chenyang Tao, Wenbo Zhao, Tagyoung Chung, Jing Huang, and Namyun Peng. 2023. Unsupervised melody-to-lyre generation. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9235-9253 Toronto, Canada. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188)9
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain your- self! leveraging language models for commonsense reasoning. In Proceedings of the 2019 Conference of the Association for Computational Linguistics (ACI22019).
Yufei Tian and Nanyun Peng. 2022. Zero-shot sonnet generation with discourse-level planning and aesthet- ics features. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, pages 3587-3597, Seattle, United States. Association for Computational Linguisties.
Machel Reid, Victor Zhong, Suchin Gururangan, and Luke Zettlemoyer. 2022. M2D2: A massively multi- domain language modeling dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 964-975, Abu Dhabi, United Arab Emirates. Association for Com- putational I Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 230.
Koustuv Sinha, Jon Gauthier, Aaron Mueller, Kan- ishka Misra, Keren Fuentes, Roger Levy, and Adina Williams. 2023. Language model acceptability judge- ments are not always robust to context. In Proceed- ings of the st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6043-6063, Toronto, Canada. Association for Computational Linguistics.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. CoRR, a 1903.
Sean Welleck, Iia Kulikov, Stephen Roller, Emily Di- nan, Kyunghyun Cho, and Jason Weston. 2019. Neu- text generation with unlikelihood training. arXiv preprint arXiv: 04319.
Eric Michael Smith, Diana Gonzalez-Rico, Emily Di- nan, and Y-Lan Boureau, 2020. Controlling style in generated dialogue.

---

John Wieting and Kevin Gimpel. 2018. ParaNMT-5OME Pushing the limits of paraphrastic sentence embed- dings with millions of machine translations. In Pro- ceedings of the 56th Annual Meeting of the Associa- lion for Computational Linguistics (Volume 1: Long Pupers), pages 451-462, Melhourne, Australia. As- sociation for Computational Linguistics.
SR
MSE
Model
count
suffix
both
count
syllable
planning
ChatGPT
0.37
0.75
0.32
4.83
ChatGPT ICI.
0.30
0.84
0.28
6.10
Alpaca-7b
0.15
0.33
0.07
9.44
Alpaca-7h ICL
0.12
0.36
0.05
10.61
Nan Xu, Chunting Zhou, Asli Celikyilmaz. and Xuezhe ChatGPT
sentence
planning
0.38
0.625
0.29
1.69
Ma. 2023a. Look-back decoding for open-ended text generation.
ChatGPT ICL
0.36
0.66
0.27
2.05
Alpaca-7b 7b
0.19
0.19
0.07
6.56
Alpaca-7b ICL
0.17
0.26
0.10
8.04
Nan Xu, Chunting Zhou, Asli Celikyilmaz, and Xuezhe
Ma. 2023b. Look-back decoding for open-ended text generation. arXiv preprint 2rXiv:2305. 13777
paragraph
planning
ChatGPT
0.69
0.17
0.
3.24
ChatGPT ICL
0.57
0.17
0.34
4.43
Fan Yin, Jesse Vig, Philippe Laban, Shafiq Joty, Caim- ing Xiong, and you read the instructions? rethinking effective- ness of task definitions in instruction learning. ACL 2023.
Alpaca-7h 7h
Failed
Jason Wu. 2023. Did
Alpaca-7h ICL
Failed
Table Success for the syllable, sentence, and paragraph count planning tasks. LLMS are best at sen- tence count planning and worst at syllable count plan- ning.
Hanqing Zhang, Haolin Song. Shaoyu Li, Ming Zhou, and Dawei Song. 2022. survey controllable text generation using transformer-based pre-trained language models. ArXiv, abs/2201.05333 105337.
A SPB additional results
Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi. 2022. Teaching algorithmic reasoning via in-context learning.
We report the additional results of ChatGPT and Alpaca on the SPB benchmark in Table 8. Recall that the suffix for the paragraph planning task is the last sentence. In practice, LLMs are unable to follow instructions and copy the requirement prompted. Hence, when we compute the success rate for this last task, check the token overlap between the generated sentence and require- ment, and if more than 2/3 of the tokens overlup, we will consider it as a success.
Wangchunshu Zhou, Yuchen Eleanor Jiang, Ethan Wilcox, Ryan Cotterell, and Mrinmaya Sachan. 2023. Controlled text generation with natural language in- structions.
Cansen Çaglayan and Murat Karakaya. 2021. Topic- controlled text generation. In 2021 6th International Conference on Computer Science and Engineering (UBMK), pages 533-536.
Taking all four tasks in the SPB benchmark into account, we find out that Alpaca-7b have very little numerical planning ability. ChatGPT on the hother hand is best at sentence count planning, and worst at syllable count planning.
B Additional Information of Content Controlled Generation
Controlled content generation refers to the task of controlling the content of generated texts. We consider three types of content constraints:
Topic constraint. It requires the model to gen- erate texts about certain topies. Traditional methods for topic constrained generation ther append a special token for different top- ics (Caglayan and Karakaya, 2021) or use trained topic classifiers (Qin et al., 2022) to guide the generation process.
Sentiment constraint. Similar to topic constraint, this task requires the model to generate texts

---

certain sentiments. The aforementioned methods for topic constrained generation also apply to sentiment constrained generation.
Keyword constraint. Keyword constrained, or lexical constrained text generation requires the model to generate texts that contain certain key- words or tokens. Traditional methods for key- word constrained text generation generally en- force lexical constraints on the outputs by mod- ifying the search space according to the con- straints (Anderson et al., 2017; Post and Vilar, 2018; Lu et al., 2021).
Datasets. For topic constraints, set of the topics from the first hierarchy in the M2D2 dataset (Reid et al., 2022) which contains domains such as health, history, society, technol- ogy, arts, science, etc. The total number of topics is 10 in our experiments. We use the Amazon Re- view dataset (Keung et al., 2020) for sentiment constrained text generation. The sentiment is mea- sure by 1 to 5 stars. For lexical constrained text generation, use the CommonGEN dataset (Lin al., 2020) which requires the model to generate sentence using three five keywords.
sub-


Reviews:
[{"review_id": "omGVjS1wjU", "reviewer": "Reviewer_iLd9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your feedback and would like to clarify some concerns as follows:\n\n**Reason 1: Few of findings are new to the community.**\n\nMany findings seem obvious in retrospect, but this does not mean that the community is already aware of them and can use them as building blocks for future work (https://aclrollingreview.org/reviewertutorial#6-check-for-lazy-thinking). To the best of our knowledge, we propose the numerical planning task and are the first to systematically study the performance of large language models on extensive controlled generation tasks, which have never been explored before and as such are new.\n\n**Reason 2: Discussion about why LLMs struggle at fine-grained hard constraints and how to address.**\nThank you for the insightful comments. Our work focuses on analyzing the capabilities of large language models. We hesitate about sharing our speculation without knowing the details of models such as ChatGPT. For example, we speculate that the model does not perform on controlled paraphrase generation tasks due to the lack of linguistic knowledge in the pertaining. Including parsing corpus such as TreeBank in the pretraining may help boost the model performance. However, this is unverifiable. Therefore, we emphasize our contribution on the  extensive and thorough experimental design and showcase what we find out instead of risking to share hypotheses that we are not sure about.  We will have some discussion with Reviewer 2 around if tokenization would be the only reason for the poor performance on the NPB task, we invite you to take a look at our response below (Q. D).\n \n\n**Reason 3: Include Vicuna and Falcon in Table 2/3/5**\n\nThank you for your suggestion! We conduct additional experiments to include Vicuna and Falcon throughout different tasks to make our study more comprehensive. Constrained by the computing power, we use Vicuna-7B for Vicuna and Falcon-7B-Instruct for Falcon. Unsurprisingly, we find that both models underperform ChatGPT. Thus, additional results do not affect our conclusion in the paper. \n\nPlease see complementary results here, we will merge them with current results for the final version:\n\nTable 2 for Numerical Planning Benchmark (NPB):\n\n|     Model    | SE-Count | SR-Last Word | SR-both | MSE-count |\n|:------------:|:--------:|:------------:|---------|-----------|\n|  Vicuna_{zs} |   0.08   |     0.09     | 0.03    | 27.68     |\n| Vicuna_{ICL} |   0.13   |     0.30     | 0.04    | 13.43     |\n|  Falcon_{zs} |   0.13   |     0.42     | 0.08    | 11.60     |\n| Falcon_{ICL} |   0.11   |     0.34     | 0.03    | 13.72     |\n\nTable 3 for Content-Constrained Text Generation:\n\n|Model           | Topic  |   Sentiment  |     Keyword |\n| -------- | ------- |------- |------- |\n|Vicuna_{zs}  |\t61.0\t|           80.5\t      |     91.6 |\n|Vicuna_{ICL} | 65.8\t|\t87.4\t      |     94.3 |\n|Falcon_{zs} \t|61.9\t|           81.0\t      |\t92.1 |\n|Vicuna_{ICL} \t| 66.0\t|\t87.7\t      |\t94.2 |\n\nTable 5 for Rationale Generation: \n| Model             |          Leakage      |    Non-Leakage |\n| -------- | ------- |------- |\n| I + R_{Vicuna} |\t\t0.95\t        |\t 0.74   |\n| I + R_{Falcon} |\t\t0.83\t        |\t 0.65   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-organized and easy to follow. \n1. This motivation is reasonable to me and has not been systematically studied before. \n2. This paper conduct extensive experiments on a wide range of control generation tasks.", "reasons_to_reject": "1.While this paper shows many findings, few of them are new to the community.\n2.There should be more discussions about why LLMs struggle at fine-grained hard constraints and how to address these problems.\n3.It would be better to include vicuna and falcon in Table-2, Table-3, and Table-5.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper study the task of controlled generation, evaluating the performance of LLMs on existing controlled generation datasets. This paper also introduce a numerical planning task. Experimental results show that LLMs struggle at fine-grained hard constraints.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qDnpeOUw1h", "reviewer": "Reviewer_rXWW", "summary": "", "questions": "A: According to the primary research question, LLMs are being compared to “finetuned smaller models” -- are these models fine-tuned on the specific task? If yes, then you are not really comparing against any such model in Sections 3,4, and 5.\n\nB: How reasonable is it to expect LLMs to reason about “syllables”? Wouldn’t this depend primarily on the tokenizer more than anything?\n\nC: Did you try other decoding methods for the NPB experiments? I am not sure if results from a single setting can be counted as fully comprehensive.\n\nD: (feel free to not answer this if there is no space): GPT-2 XL gets perfect scores on ROC for a majority of the metrics in the story generation experiments using $\\eta$-sampling (as shown in Table 4) -- this seems peculiar and interesting, do you have an idea as to why this might be happening?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for such a detailed and insightful review. We appreciate your insightful suggestions and precious feedback!\n\n**Reason 1 and Q. A: Comparison between LLMs and fine-tuned smaller models**\n\nWe compare LLMs with task-specific baselines, where fine-tuned smaller models are included most of the time. For other tasks, we tailor based on the common practice or which performs the best in that task. To make it more rigorous, we will change the phrasing to ``Comparison between LLMs and task-specific models” for our final version.  \n\nContent-Controlled Generation (Section 3): For fine-tuned smaller models, we included the performance of Diffusion-LM in the first row of Table 3, which uses fine-tuned classifiers as score functions. We also tested fine-tuned GPT-2 and T5-3B with the results listed below. We observe that LLMs slightly underperform strong fine-tuned smaller models such as T5 in the zero-shot setting, and the performance gap is very small when LLMs are equipped with in-context examples.\n|**Model**           | **Topic**  |   **Sentiment**  |     **Keyword** |\n| -------- | ------- |------- |------- |\n| GPT-2 (1.5B, fine-tuned) | 63.4 | 76.5 | 88.9 |\n| T5 (3B, fine-tuned) | 67.3 | 83.9 | 94.8 |\n\nStory Generation (Section 4): We do not include fine-tuned models since in the open-ended setting, recent research work focuses on decoding strategies with zero training [1] because 1) decoding alone could achieve **similar good performance** as fine-tuned models [2] and 2) decoding is more efficient and feasible for improving generation of LLMs while fine-tuning methods are computationally prohibitive, which greatly limits their practical applicability [3].  \nRationale Generation (Section 5): Rationale generation is indeed an exception itself, where we directly compare LLMs with crowdsourced rationales due to two reasons: 1) finetuned T5 perform poorly at rationale generation task [4], and 2) the crowdsourced rationales do not necessarily mean high quality and are comparable to the quality of ChatGPT generation. Such contrast supports our claim well that LLM is very capable of rationale generation, approaching human performance. We will clarify the uniqueness of this task in our final version. \n \n[1] Xu, Nan, et al. \"Look-back Decoding for Open-Ended Text Generation.\" arXiv preprint arXiv:2305.13477 (2023).\n[2] Su, Yixuan, and Nigel Collier. \"Contrastive search is what you need for neural text generation.\" TMLR 2023.\n[3] Li, Xiang Lisa, et al. \"Contrastive decoding: Open-ended text generation as optimization.\" ACL 2023.\n[4] Sun, Jiao, et al. \"Investigating the Benefits of Free-form Rationales.\" EMNLP 2022.\n\n**Reason 2: Spectrum of LLM abilities**\n\nAwesome catch, we will change the notation for rationale generation in Figure 1 from red to green.\n\n**Reason 3: Selection of LLMs for comparison keeps changing**\n\nGreat point! As we respond to R1 under Reason 3, we decide on the LLM sets as {ChatGPT,  LLaMA, Alpaca, Falcon and Vicuna} throughout the paper and conduct complementary experiments to make our study more consistent and comprehensive. Unsurprisingly, we find that both Vicuna and Flacon still underperform ChatGPT. Thus, additional results do not affect our conclusion in the paper. For paraphrase generation, ChatGPT performs the best and all other models do not generate reasonable paraphrases, we therefore keep our original results untouched. We will merge the new results with the current results in our final version.\n\nTable 2 for Numerical Planning Benchmark (NPB):\n\n| **Model**\t |  **SR-Count**\t | **SR-Last word**|\t**SR-both** |**MSE-count**|\n| -------- | ------- | -------- | ------- |------- |\n| Vicuna_{zs}\t |  0.08\t\t |  0.09\t\t|  0.03\t|\t27.68|\t\n| Vicuna_{ICL}  |  0.13\t\t |  0.30\t\t|  0.04\t|\t13.43|\n| Falcon_{zs}    |  0.13\t\t |  0.42\t\t|  0.08\t|\t11.60|\n| Falcon_{ICL}  |  0.11\t\t |  0.34\t\t|  0.03\t|\t13.72|\n\nTable 3 for Content-Constrained Text Generation:\n\n|**Model**           | **Topic**  |   **Sentiment**  |     **Keyword** |\n| -------- | ------- |------- |------- |\n|Vicuna_{zs}  |\t61.0\t|           80.5\t      |     91.6 |\n|Vicuna_{ICL} | 65.8\t|\t87.4\t      |     94.3 |\n|Falcon_{zs} \t|61.9\t|           81.0\t      |\t92.1 |\n|Vicuna_{ICL} \t| 66.0\t|\t87.7\t      |\t94.2 |\n\nTable 5 for Rationale Generation: \n| **Model**              |          **Leakage**      |    **Non-Leakage** |\n| -------- | ------- |------- |\n| I + R_{Vicuna} |\t\t0.95\t        |\t 0.74   |\n| I + R_{Falcon} |\t\t0.83\t        |\t 0.65   |\n\n\n**Q. B: Whether reasonability of reasoning about “syllables” primarily depends on tokenizer**\n\nAs far as we know, the ability of syllable planning does not necessarily depend on tokenizers. Nor are we aware of evidence that the subwords of a tokenizer roughly correspond to units of syllables. For example, [1] shows that smaller models such as gpt-2-large fine-tuned on syllable-related data can achieve a success rate of close to 90% on the same syllable-planning task. On the other hand, the best performance of ChatGPT is 37%.\n\n[1] Tian, et al. Unsupervised Melody-to-Lyric Generation. https://aclanthology.org/2023.acl-long.513.pdf\n\n**Q. C: Other decoding methods for the NPB experiments**\n\nWe did try different decoding algorithms on the NPB benchmark in the early experimental stage but decided to proceed with the most reasonable setting: sampling with T=0.3.\n\nSpecifically, we tried 1) greedy decoding, 2) beam search with beam size 8, and 3) sampling with temperature T={0.3, 0.7, 1.0}. For the prior two, most of the generated outputs are highly similar, plain, and lack of diversity. As for sampling with T={0.3, 0.7, 1.0}, the success rate decreases as T increases. We think T=0.3 is a reasonable balance between diversity and quality. We believe that our results convey meaningful signals since each number N has been averaged over 100 different evaluation samples to reduce noise.\n\n**Q. D: GPT-2 XL gets perfect scores on ROC in Table 4**\n\nFirst of all, story generation is relatively easier compared with other controlled generation tasks as demonstrated in Table 1. Secondly, ROC dataset is not as challenging as Writing Prompts as shown in Table 4 because 1) ROC is an easier corpus comprising commonsense stories while Writing Prompts dataset is far more challenging for inspiring continuations with abstract, high-level story prompts (from lines 254 to 265); 2) GPT-2 released in 2019 scraped web pages as training data, where ROC released in 2016 could be included while samples from Writing Prompts were not covered based on our rough check. We will add prior discussion in our work.\n\n**Missing Reference and typos**\n\nGreat catches! We will fix all these in our final version and do another round of careful proofreading.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors present a spectrum of different controllable text generation scenarios for evaluating LLMs -- this seems like a neat, novel empirical contribution that can shed light on the ways in which LLMs lack in satisfying the constraints presented to them in their inputs.\n- Each task involves careful consideration of the design of the experiment as well as the dataset and measures used to shed light on the particular type of controllable generation -- the authors should be commended for this!\n- Within the spectrum, the authors further propose a novel dataset called the Numerical Planning Benchmark which targets the ability of LLMs to plan their generation based on precise numerical factors present in the prompt/instruction.", "reasons_to_reject": "- The research question states that LLMs are being compared to “finetuned smaller models” but there is no such comparison being made in sections 3, 4 (this one uses pre-trained GPT-2), and 5 (all rationales are generated using off the shelf pre-trained LLMs and these are compared against crowdsourced rationale generations) **Note:** *This point of weakness hinges on the answer to Question A of the next section.*\n- It is not immediately obvious what determines the position of a task on the spectrum. For instance, the rationale generation performance of LLMs was quite good and should probably be closer to “good” than “poor” in Figure 1.\n- The LLMs being compared are quite different across the 6 sections -- for instance sec 2 compares ChatGPT, and Alpaca, while section 3 compares ChatGPT, Alpaca, and LLaMa, while section 4 compares ChatGPT, Vicuna, and Falcon, ..and so on. This seems to defeat the purpose and potential of the full spectrum of tasks which is supposed to collectively demonstrate a holistic comparison between LLMs on controllable text generation. It would have been better if the authors chose a select few LMs and only compared them throughout the tasks (against task-specific smaller fine-tuned LMs, to stay faithful to the main research question).", "questions_to_authors": "A: According to the primary research question, LLMs are being compared to “finetuned smaller models” -- are these models fine-tuned on the specific task? If yes, then you are not really comparing against any such model in Sections 3,4, and 5.\n\nB: How reasonable is it to expect LLMs to reason about “syllables”? Wouldn’t this depend primarily on the tokenizer more than anything?\n\nC: Did you try other decoding methods for the NPB experiments? I am not sure if results from a single setting can be counted as fully comprehensive.\n\nD: (feel free to not answer this if there is no space): GPT-2 XL gets perfect scores on ROC for a majority of the metrics in the story generation experiments using $\\eta$-sampling (as shown in Table 4) -- this seems peculiar and interesting, do you have an idea as to why this might be happening?", "paper_topic_and_main_contributions": "This paper attempts to characterize the controllability of the text generated by LLMs, and in particular answers the question: “Are LLMs better than fine-tuned smaller models at controllability on text generation tasks?” \n\nTo this end, the authors design a comprehensive spectrum of tasks that sheds light on the extent to which their text generation mechanisms can be made to adhere to constraints ranging from specific attributes like sentiment, keywords, etc., to precise numerical planning, to the syntax and semantics of the output. This maps to 6 different experiments (11 different tasks), each of which targets a specific controllable constraint along with the evaluation of specific characteristics (e.g., repetition, diversity, lexical overlap, syntactic matches, etc.). As part of the experiments, the authors also design a benchmark called the Numerical Planning Benchmark that adds to existing work on controllable text generation by devising instructional prompts that specifically target the number of words/sentences/paragraphs in the generated output. Specifically for this experiment, the authors find that the tested LMs were generally bad at numerical planning, and that interestingly few-shot in-context learning actually hinders performance on this task. ChatGPT to a small extent does achieve success but this is limited to when the number of units to plan for (words/sentences, etc.) is small, and the last token is provided.\n\nOverall, the findings suggest that LLMs are more likely to be controlled to generate text that obey soft/coarse grained constraints such as sentiment, topic, generation of rationales, etc.,  but struggle on finer-grained constraints such as numerical planning and paraphrasing.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The finding that ChatGPT mimics the syntactic structure of the sentences in its context is related to the following paper which you might be interested in:\n\nKoustuv Sinha, Jon Gauthier, Aaron Mueller, Kanishka Misra, Keren Fuentes, Roger Levy, and Adina Williams. 2023. Language model acceptability judgements are not always robust to context. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6043–6063, Toronto, Canada. Association for Computational Linguistics. URL: https://aclanthology.org/2023.acl-long.333/", "Typos_Grammar_Style_and_Presentation_Improvements": "The title seems to be grammatically incorrect, I think you meant “Evaluating” instead of “Evaluate” -- I hope you can change this!!\n\nLine 577: “chain of thoughts” → “chain of thought”\n\n**Important:** I would seriously like to see a high level discussion on how the results and analyses demonstrate the usefulness of the spectrum. Currently, the conclusion section seems to pop out suddenly and unnaturally after the final experiment. I strongly encourage taking this into account for the additional page, if the paper is accepted/made public. I say this because there is genuine value in having a full range of tasks with varying difficulty and amounts of control, and it would be great if the paper does more to shed light on this fact."}}, {"review_id": "va40jw0zvI", "reviewer": "Reviewer_Hweb", "summary": "", "questions": "1. In the evaluation of Rationale Generation, FlanT5-XXL has an accuracy of 0.87, is this too high to show less differences between models? How to get a better balance between the shortages of backbone model and its reasoning abilities?\n2. Till now, which LLM should also be considered in these controlled generation tasks?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your suggestions and would like to clarify some details as follows:\n\n**Reason to reject: Include qualitative examples**\n\nGood point! We will add some failure examples for challenging tasks, including numerical planning and controlled paraphrase generation, in our final version.\n\n**Question: FlanT5 too strong? Balance the shortages of backbone model and its reasoning abilities**\n\nThis is a great point. We deliberately choose a strong backbone model that has strong reasoning capability to make sure that the performance difference comes from the rationale quality instead of the reasoning incompetence of the backbone model. We are confident that FlanT5 is an appropriate discriminator choice for this task because it responds properly to 1) the presence of rationales and 2) the change of rationale quality. From Table 5: Adding rationales helps boost the model performance, suggesting that FlanT5 is not too strong to solve the task directly. Based on [1], the quality of ECQA rationales is better than CoS-E, resulting in a better model performance. Therefore,  we believe FlanT5 strikes the perfect balance between reasoning and model capability for the rationale generation task!\n\n[1] Sun, Jiao, et al. \"Investigating the Benefits of Free-form Rationales.\" EMNLP 2022.\n\n\n**Q2: Recent LLMs to be checked**\n\nAlthough there are new open-source LLMs coming at a fast pace, they are arguably not exceeding the performance of ChatGPT (https://chat.lmsys.org/?leaderboard). On the other side, as we respond to R1 under Reason 3, R2 under Reason 3, we decide the LLM sets as {ChatGPT,  LLaMA, Alpaca, Falcon, and Vicuna} throughout the paper and conduct complementary experiments to make our study more consistent and comprehensive. Unsurprisingly, we find that both Vicuna and Flacon still underperform ChatGPT. Thus, additional results do not affect our conclusion in the paper. Please find our new results here:\n\nTable 2 for Numerical Planning Benchmark (NPB):\n\n| **Model**\t |  **SR-Count**\t | **SR-Last word**|\t**SR-both** |**MSE-count**|\n| -------- | ------- | ------- | ------- | ------- |\n| Vicuna_{zs}\t |  0.08\t\t |  0.09\t\t|  0.03\t|\t27.68|\t\n| Vicuna_{ICL}  |  0.13\t\t |  0.30\t\t|  0.04\t|\t13.43|\n| Falcon_{zs}    |  0.13\t\t |  0.42\t\t|  0.08\t|\t11.60|\n| Falcon_{ICL}  |  0.11\t\t |  0.34\t\t|  0.03\t|\t13.72|\n\nTable 3 for Content-Constrained Text Generation:\n\n|**Model**           | **Topic**  |   **Sentiment**  |     **Keyword** |\n| -------- | ------- | ------- | ------- |\n|Vicuna_{zs}  |\t61.0\t|           80.5\t      |     91.6 |\n|Vicuna_{ICL} | 65.8\t|\t87.4\t      |     94.3 |\n|Falcon_{zs} \t|61.9\t|           81.0\t      |\t92.1 |\n|Vicuna_{ICL} \t| 66.0\t|\t87.7\t      |\t94.2 |\n\nTable 5 for Rationale Generation: \n| **Model**              |          **Leakage**      |     **Non-Leakage** |\n| -------- | ------- | ------- |\n| I + R_{Vicuna} |\t\t0.95\t        |\t 0.74   |\n| I + R_{Falcon} |\t\t0.83\t        |\t 0.65   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The evaluation is really detailed and well designed.\n2. Figure 1 shows a very good overview of the whole paper, and other tables are quite clear.\n3. The writing is good and the structure is nice.", "reasons_to_reject": "1. The appendix should include more details, such as more examples and presentation of failed examples.", "questions_to_authors": "1. In the evaluation of Rationale Generation, FlanT5-XXL has an accuracy of 0.87, is this too high to show less differences between models? How to get a better balance between the shortages of backbone model and its reasoning abilities?\n2. Till now, which LLM should also be considered in these controlled generation tasks?", "paper_topic_and_main_contributions": "This paper evaluates LLMs on controlled generation tasks, stepping into five unexplored tasks. Among them, the authors propose a new benchmark called “numerical planning” which is hard for LLMs. \n\nContribution：\n1. Study the controllability of large language models using five unexplored tasks.\n2. Propose a new controlled generation benchmark which is hard for LLMs. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 218, one more period\nMaybe adding subtitles in Figure 2 will be better\nCheck the bold number of “diversity” in the table below of Table 4, should be the biggest one"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the controllability of large language models on generation tasks on various benchmark tasks. Reviewers found this paper well written and easy to follow. Reviewers also found the experiments extensive and well designed, covering a wide range of controlled generation tasks. While this is an analytic paper with comprehensive evaluations, it can add significant value by including an in-depth discussion on why LLM struggled at fine-grained hard constraints and how to close the gaps, as also commented by one reviewer. The authors provided some detailed response on this of which the author should consider add to the revised version of the paper.
Decision: Accept-Main